Close
  • Latest News
  • Cybersecurity
  • Big Data and Analytics
  • Cloud
  • Mobile
  • Networking
  • Storage
  • Applications
  • IT Management
  • Small Business
  • Development
  • Database
  • Servers
  • Android
  • Apple
  • Innovation
  • Blogs
  • PC Hardware
  • Reviews
  • Search Engines
  • Virtualization
Read Down
Sign in
Close
Welcome!Log into your account
Forgot your password?
Read Down
Password recovery
Recover your password
Close
Search
Menu
Search
  • Latest News
  • Cybersecurity
  • Big Data and Analytics
  • Cloud
  • Mobile
  • Networking
  • Storage
  • Applications
  • IT Management
  • Small Business
  • Development
  • Database
  • Servers
  • Android
  • Apple
  • Innovation
  • Blogs
  • PC Hardware
  • Reviews
  • Search Engines
  • Virtualization
More
    Home Database
    • Database

    Apache Spark Continues to Gain Enterprise Traction

    By
    Darryl K. Taft
    -
    September 25, 2015
    Share
    Facebook
    Twitter
    Linkedin
      big data BI

      Apache Spark, the hot open-source data processing engine, is outgrowing Apache Hadoop in terms of user adoption, according to a recent survey.

      Databricks, the company founded by the creators of Apache Spark, released the findings of a survey of more than 1,400 respondents from the Spark community to identify how organizations are using the data analytics and processing engine.

      The 2015 Spark User Survey results showed that the number of standalone deployments of Spark eclipses those on YARN as more users run Spark independent of Hadoop.

      Indeed, the most common Spark deployments according to the community are: 48 percent standalone, 40 percent YARN within Hadoop and 11 percent with Apache Mesos. Spark users who do not use any Hadoop components have more than doubled in 2015 as compared to 2014, the survey said. Moreover, the survey found that 51 percent of respondents run Spark on a public cloud.

      With more than 600 contributors in the last 12 months — up from 315 contributors for the prior 12 months, Spark is the most active open source project in big data, according to Databricks. Additionally, more than 200 organizations contribute code to Spark, making it one of the largest communities of engaged developers to date, the company said.

      Spark has been referred to as a game changer and perhaps the most significant open source project of the next decade. It is an open source data processing engine built for speed, ease of use, and sophisticated analytics. Spark is designed to perform both batch processing and new workloads like streaming, interactive queries, and machine learning. Users say it speeds up big data processing by a factor of 10 to 100 and simplifies app development.

      Spark is being used for an increasingly diverse set of applications, particularly by data scientists for machine learning, streaming and graph analysis use cases. In 2015, there are 56 percent more Spark streaming users than in 2014. The production use of advanced analytics, like MLib for machine learning and GraphX for graph processing, increased from 11 percent in 2014 to 15 percent in 2015. And 75 percent of Spark users are also using two or more Spark components, with 51 percent of Spark users are using three or more Spark components.

      “The continued growth of Spark has been highly encouraging, as companies are going into production to obtain real business value, and they are doing so in a wide range of environments beyond Hadoop clusters,” said Matei Zaharia, creator of Apache Spark and CTO of Databricks, in a statement. “Databricks and our partners are 100 percent committed to the long-term growth of Spark and we’ll continue to make improvements based on this survey data and our ongoing community feedback, to make the most complete big data analytics toolkit accessible to all businesses.”

      Apache Spark Continues to Gain Enterprise Traction

      Zaharia told eWEEK Spark started out of a research project at the University of California Berkeley, where he was working with early users of MapReduce and Hadoop, including Facebook and Yahoo. He said he found some common problems among those users, chief among them being that they all wanted to run more complex algorithms that couldn’t be done with just one MapReduce step.

      “MapReduce is a simple way to scan through data and aggregate information in parallel and not every algorithm can be done with it,” Zaharia said. “So we wanted to create a more general programming model for people to write cluster applications that would be fast and efficient at these more complex types of algorithms.”

      Meanwhile, Spark users are becoming more diverse. Spark is breaking down technology barriers between data scientists and engineers, who are working collaboratively to solve data problems. Of those surveyed, 41 percent identified themselves as data engineers, while 22 percent of respondents identified themselves as data scientists. Spark users are solving a variety of problems in different languages — Scala (71 percent), Python (58 percent), SQL (36 percent), Java (31 percent) and R (18 percent) — all within the same framework.

      Business intelligence appears to be the most popular use case for Spark, with 68 percent of respondents saying they use Spark for BI. However, 52 percent use Spark for data warehousing, 48 percent to build recommendation engines, 40 percent for processing application and system logs, 36 percent for user-facing services and 29 percent for fraud detection and security.

      In addition Spark is helping to increase access to big data. Spark adoption is growing so quickly because users are enjoying its ease of use, performance and the fact that it is aligned for future growth in real-time and advanced analytics, Databricks said. Ninety one percent of those surveyed claim performance as their reason for using Spark, while 77 percent cite ease of programming. Moreover, 71 percent cite ease of deployment, 64 percent cite advanced analytics capabilities and 52 percent cite real-time streaming capabilities as their reason for using the technology.

      “The enthusiasm for big data is matched only by the pace of innovation,” said Nik Rouda, senior analyst at Enterprise Strategy Group, in a statement. “Many organizations are shifting to a ‘Spark-first’ strategy, recognizing its advantages of analytics versatility, development familiarity, superior performance, range of data sources supported, and deployment flexibility. The market will no doubt continue to evolve, but Spark has established considerable momentum today.”

      Avatar
      Darryl K. Taft
      Darryl K. Taft covers the development tools and developer-related issues beat from his office in Baltimore. He has more than 10 years of experience in the business and is always looking for the next scoop. Taft is a member of the Association for Computing Machinery (ACM) and was named 'one of the most active middleware reporters in the world' by The Middleware Co. He also has his own card in the 'Who's Who in Enterprise Java' deck.

      MOST POPULAR ARTICLES

      Android

      Samsung Galaxy XCover Pro: Durability for Tough...

      Chris Preimesberger - December 5, 2020 0
      Have you ever dropped your phone, winced and felt the pain as it hit the sidewalk? Either the screen splintered like a windshield being...
      Read more
      Cloud

      Why Data Security Will Face Even Harsher...

      Chris Preimesberger - December 1, 2020 0
      Who would know more about details of the hacking process than an actual former career hacker? And who wants to understand all they can...
      Read more
      Cybersecurity

      How Veritas Is Shining a Light Into...

      eWEEK EDITORS - September 25, 2020 0
      Protecting data has always been one of the most important tasks in all of IT, yet as more companies become data companies at the...
      Read more
      Big Data and Analytics

      How NVIDIA A100 Station Brings Data Center...

      Zeus Kerravala - November 18, 2020 0
      There’s little debate that graphics processor unit manufacturer NVIDIA is the de facto standard when it comes to providing silicon to power machine learning...
      Read more
      Apple

      Why iPhone 12 Pro Makes Sense for...

      Wayne Rash - November 26, 2020 0
      If you’ve been watching the Apple commercials for the past three weeks, you already know what the company thinks will happen if you buy...
      Read more
      eWeek


      Contact Us | About | Sitemap

      Facebook
      Linkedin
      RSS
      Twitter
      Youtube

      Property of TechnologyAdvice.
      Terms of Service | Privacy Notice | Advertise | California - Do Not Sell My Information

      © 2021 TechnologyAdvice. All Rights Reserved

      Advertiser Disclosure: Some of the products that appear on this site are from companies from which TechnologyAdvice receives compensation. This compensation may impact how and where products appear on this site including, for example, the order in which they appear. TechnologyAdvice does not include all companies or all types of products available in the marketplace.

      ×