Apache Spark Developer Adoption on the Rise
Developers are increasingly adopting Apache Spark over MapReduce for big data processing, according to a new Typesafe survey.Results of a new survey indicate that the Apache Spark big data processing engine is gaining traction with a growing number of developers. Typesafe, the company behind Play Framework, Akka, and Scala, has released the findings of a survey of more than 2,100 enterprise developers, data scientists, executives and system architects, analyzing adoption patterns of Apache Spark. The survey showed that 13 percent of developers said they are using Spark in production, while 31 percent said they are currently evaluating Spark. Apache Spark is an open-source engine for large-scale data processing and analytics. It has been in development for a number of years at UC Berkeley's AmpLab, and is now being driven by Databricks, a Berkeley spin-out founded by Ion Stoica and Matei Zaharia. Zaharia is CTO of Databricks and the creator of Apache Sparks. Databricks worked with Typesafe on the survey. "This survey of over 2100 developers alone highlights that over 500 enterprises are using or planning to use Spark in production in 2015, in environments ranging from Hadoop clusters to public and private clouds, with data sources including key-value stores, databases, streaming data and file systems," Zaharia said. "Their use cases range from batch workloads to SQL queries, stream processing and machine learning, highlighting Spark’s unique capability as a simple, unified platform for data processing."
Typesafe said Spark awareness and adoption are seeing hockey-stick-like growth. Google Trends confirms this finding and the survey shows that 71 percent of respondents have at least evaluation or research experience with Spark—up to 35 percent are using it or plan to adopt soon. Of the survey respondents running big data applications in production, 82 percent indicated that they are eager to replace MapReduce with Spark as the core processing engine.