What is Spark?

Apache® Spark™ is an open-source cluster computing framework with in-memory processing to speed analytic applications up to 100 times faster compared to technologies on the market today. Developed in the AMPLab at UC Berkeley, Apache Spark can help reduce data interaction complexity, increase processing speed, and enhance mission-critical applications with deep intelligence. 

Highly versatile in many environments, Apache Spark is known for its ease of use in creating algorithms that harness insight from complex data. Spark was elevated to a top-level Apache Project in 2014 and continues to expand today.

IBM is committing to the Apache Spark project with investments in design-led innovation and broad-scale education programs to promote open source innovation and accelerate intelligence into every application.

Ready to learn more about Spark? Check out the free Spark Fundamentals online class from Big Data University and wander past the Spark Technology Center.


The Basics & Getting Started

Remember, to be eligible for prizes, submissions must an Apache Spark application that addresses a real business problem or core business concern related to customer care, marketing, risk management, or operations. Your Apache Spark application should be something that a business’s stakeholders could use and deploy in the future.

To meet the minimum eligibility requirements you must:

  1. Identify a business need that could be informed by data analysis.
  2. Find data (publicly available data or data from your own business) to analyze using Apache Spark to inform that business need.
  3. Analyze the data using Apache Spark and share your analysis code for judging (via GitHub or privately shared file). (Applications are encouraged – though not required – to be portable, with the ability to run on different cloud platforms.)
  4. Showcase your analysis output by including a visual (graphic) or textual explanation of your results, and a video demo explaining your process and outcomes. (Pro tip: we recommend explaining how your entry meets the judging criteria in your video demo or text description, such as the portability of your app.)

OPTIONAL: You may develop a working application that utilizes your Spark-analyzed data to help solve a business need, but this is not required to be eligible for a prize.


Business Need Examples

Not sure what business need to focus on? Your Spark application could:

  • Work to support marketing by improving upon an existing “propensity to buy” model to create better offers.
  • Support operations by focusing on the design strategy for optimizing shipments of raw materials or scheduling workers.
  • Support risk management by monitoring user profiles to build better models for behavior patterns for commerce websites. 

Want more ideas? Check out how five real companies are using Spark to help their businesses succeed and grow.


Spark Resources

We get it: learning a new platform or tool can be overwhelming. You need samples and resources to make your submission great. That’s why we’ve put together this list of sample applications, services, and platforms to help you make your best Spark app!

What to use Bluemix for your Spark App?

Typically, Bluemix services are available for free for 30 days. However, if you sign up as a participant in the Apache Spark Makers Build hackathon, you’ll have access to Bluemix services from now until November 1, 2016. Request your promo code here. (Note: Bluemix use is not required for participation in this hackathon.)



We understand that you may not have access to, or want to use, your own company’s data. With that in mind, you’re welcome to analyze publicly available data or your own. Here are some datasets to get you started. You can use one of these or any other data that is publicly available and that you have the rights to use.


City or Regional Public Data

Still data hungry? Here are a some of our favorite city, regional, and national public datasets to diversivy your options.

Have a question?

We’re here to help, and we welcome your questions. Please send us a note via the Discussion Board, or email