eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
Apache Spark, the hot open-source data processing engine, is outgrowing Apache Hadoop in terms of user adoption, according to a recent survey.
Databricks, the company founded by the creators of Apache Spark, released the findings of a survey of more than 1,400 respondents from the Spark community to identify how organizations are using the data analytics and processing engine.
The 2015 Spark User Survey results showed that the number of standalone deployments of Spark eclipses those on YARN as more users run Spark independent of Hadoop.
Indeed, the most common Spark deployments according to the community are: 48 percent standalone, 40 percent YARN within Hadoop and 11 percent with Apache Mesos. Spark users who do not use any Hadoop components have more than doubled in 2015 as compared to 2014, the survey said. Moreover, the survey found that 51 percent of respondents run Spark on a public cloud.
With more than 600 contributors in the last 12 months — up from 315 contributors for the prior 12 months, Spark is the most active open source project in big data, according to Databricks. Additionally, more than 200 organizations contribute code to Spark, making it one of the largest communities of engaged developers to date, the company said.
Spark has been referred to as a game changer and perhaps the most significant open source project of the next decade. It is an open source data processing engine built for speed, ease of use, and sophisticated analytics. Spark is designed to perform both batch processing and new workloads like streaming, interactive queries, and machine learning. Users say it speeds up big data processing by a factor of 10 to 100 and simplifies app development.
Spark is being used for an increasingly diverse set of applications, particularly by data scientists for machine learning, streaming and graph analysis use cases. In 2015, there are 56 percent more Spark streaming users than in 2014. The production use of advanced analytics, like MLib for machine learning and GraphX for graph processing, increased from 11 percent in 2014 to 15 percent in 2015. And 75 percent of Spark users are also using two or more Spark components, with 51 percent of Spark users are using three or more Spark components.
“The continued growth of Spark has been highly encouraging, as companies are going into production to obtain real business value, and they are doing so in a wide range of environments beyond Hadoop clusters,” said Matei Zaharia, creator of Apache Spark and CTO of Databricks, in a statement. “Databricks and our partners are 100 percent committed to the long-term growth of Spark and we’ll continue to make improvements based on this survey data and our ongoing community feedback, to make the most complete big data analytics toolkit accessible to all businesses.”
Apache Spark Continues to Gain Enterprise Traction
Zaharia told eWEEK Spark started out of a research project at the University of California Berkeley, where he was working with early users of MapReduce and Hadoop, including Facebook and Yahoo. He said he found some common problems among those users, chief among them being that they all wanted to run more complex algorithms that couldn’t be done with just one MapReduce step.
“MapReduce is a simple way to scan through data and aggregate information in parallel and not every algorithm can be done with it,” Zaharia said. “So we wanted to create a more general programming model for people to write cluster applications that would be fast and efficient at these more complex types of algorithms.”
Meanwhile, Spark users are becoming more diverse. Spark is breaking down technology barriers between data scientists and engineers, who are working collaboratively to solve data problems. Of those surveyed, 41 percent identified themselves as data engineers, while 22 percent of respondents identified themselves as data scientists. Spark users are solving a variety of problems in different languages — Scala (71 percent), Python (58 percent), SQL (36 percent), Java (31 percent) and R (18 percent) — all within the same framework.
Business intelligence appears to be the most popular use case for Spark, with 68 percent of respondents saying they use Spark for BI. However, 52 percent use Spark for data warehousing, 48 percent to build recommendation engines, 40 percent for processing application and system logs, 36 percent for user-facing services and 29 percent for fraud detection and security.
In addition Spark is helping to increase access to big data. Spark adoption is growing so quickly because users are enjoying its ease of use, performance and the fact that it is aligned for future growth in real-time and advanced analytics, Databricks said. Ninety one percent of those surveyed claim performance as their reason for using Spark, while 77 percent cite ease of programming. Moreover, 71 percent cite ease of deployment, 64 percent cite advanced analytics capabilities and 52 percent cite real-time streaming capabilities as their reason for using the technology.
“The enthusiasm for big data is matched only by the pace of innovation,” said Nik Rouda, senior analyst at Enterprise Strategy Group, in a statement. “Many organizations are shifting to a ‘Spark-first’ strategy, recognizing its advantages of analytics versatility, development familiarity, superior performance, range of data sources supported, and deployment flexibility. The market will no doubt continue to evolve, but Spark has established considerable momentum today.”