Close
  • Latest News
  • Big Data and Analytics
  • Cloud
  • Networking
  • Cybersecurity
  • Applications
  • IT Management
  • Storage
  • Sponsored
  • Mobile
  • Small Business
  • Development
  • Database
  • Servers
  • Android
  • Apple
  • Innovation
  • Blogs
  • PC Hardware
  • Reviews
  • Search Engines
  • Virtualization
Read Down
Sign in
Close
Welcome!Log into your account
Forgot your password?
Read Down
Password recovery
Recover your password
Close
Search
Logo
Logo
  • Latest News
  • Big Data and Analytics
  • Cloud
  • Networking
  • Cybersecurity
  • Applications
  • IT Management
  • Storage
  • Sponsored
  • Mobile
  • Small Business
  • Development
  • Database
  • Servers
  • Android
  • Apple
  • Innovation
  • Blogs
  • PC Hardware
  • Reviews
  • Search Engines
  • Virtualization
More
    Home Database
    • Database

    MongoDB Delivers Connector for Apache Spark

    By
    Darryl K. Taft
    -
    July 1, 2016
    Share
    Facebook
    Twitter
    Linkedin
      big data

      This week at its MongoDB World conference in New York, MongoDB announced the MongoDB Connector for Apache Spark, which enables developers and data scientists to get real-time analytics from fast-moving data.

      The product is now available for users to run analytics and glean insights from live, operational and streaming data. MongoDB worked closely with Databricks, the company founded by the team that created Apache Spark. And the MongoDB Connector has received Databricks Certified Application status for Spark. The certification means Databricks has ensured that the connector provides integration and API compatibility between Spark processes and MongoDB.

      The new Spark connector follows the pattern of MongoDB’s existing connector for Hadoop.

      Kelly Stirman, vice president of strategy and product marketing at MongoDB, told eWEEK there is a lot of interest in people using Spark with MongoDB. He said the way people typically use MongoDB with Hadoop is they have different operational systems and data moves through extract, transform and load (ETL) or some other process into Hadoop. So MongoDB created a Hadoop connector; now they have one for Spark.

      “People are saying with the kind of machine learning and analytics that they’re doing on data, they want to move some of that to run on the operational data as it’s being created,” Stirman said. “And that’s the demand of using Spark with MongoDB.”

      So MongoDB took its connector for Hadoop and enhanced it so that it would be compatible with Spark. “We learned a lot and decided there’s enough interest there to make an engineering investment to make a dedicated connector for Spark,” Stirman said.

      “Spark jobs can be executed directly against operational data managed by MongoDB, without the time and expense of ETL processes,” Eliot Horowitz, co-founder and CTO of MongoDB, said in a statement. “MongoDB can efficiently index and serve analytics results back into live, operational processes, making them smarter, more contextual and responsive to events as they happen.”

      Moreover, the MongoDB Connector for Apache Spark is written in Scala, Apache Spark’s native language so it offers a familiar development experience for Spark users. In addition, the connector exposes all of Spark’s libraries, enabling MongoDB data to run as data frames and data sets for analysis with machine learning, graph, streaming and SQL APIs, further benefiting from automatic schema inference, Stirman said.

      “Users are already combining Apache Spark and MongoDB to build sophisticated analytics applications,” Reynold Xin, co-founder and chief architect of Databricks, said in a statement. “The new native MongoDB Connector for Apache Spark provides higher performance, greater ease of use, and access to more advanced Apache Spark functionality than any MongoDB connector available today.”

      The connector also takes advantage of MongoDB’s aggregation pipeline, Stirman said. And it enables users to co-locate Resilient Distributed Datasets (RDDs) with the source MongoDB node to help minimize data movement across the cluster and reduce latency.

      Jeff Smith, data engineering team lead at x.ai, which produces an artificial intelligence-powered personal assistant for scheduling meetings, said x.ai uses both MongoDB and Apache Spark to process and analyze the huge amounts of data required to power an AI application.

      “With the new native MongoDB Connector for Apache Spark, we have an even better way of connecting up these two key pieces of our infrastructure,” Smith said in a statement. “We believe the new connector will help us move faster and build reliable machine learning systems that can operate at massive scale.”

      At its annual user conference, MongoDB also introduced Atlas, the company’s new database-as-a-service offering.

      Darryl K. Taft
      Darryl K. Taft covers the development tools and developer-related issues beat from his office in Baltimore. He has more than 10 years of experience in the business and is always looking for the next scoop. Taft is a member of the Association for Computing Machinery (ACM) and was named 'one of the most active middleware reporters in the world' by The Middleware Co. He also has his own card in the 'Who's Who in Enterprise Java' deck.

      MOST POPULAR ARTICLES

      Big Data and Analytics

      Alteryx’s Suresh Vittal on the Democratization of...

      James Maguire - May 31, 2022 0
      I spoke with Suresh Vittal, Chief Product Officer at Alteryx, about the industry mega-shift toward making data analytics tools accessible to a company’s complete...
      Read more
      Cybersecurity

      Visa’s Michael Jabbara on Cybersecurity and Digital...

      James Maguire - May 17, 2022 0
      I spoke with Michael Jabbara, VP and Global Head of Fraud Services at Visa, about the cybersecurity technology used to ensure the safe transfer...
      Read more
      Applications

      Cisco’s Thimaya Subaiya on Customer Experience in...

      James Maguire - May 10, 2022 0
      I spoke with Thimaya Subaiya, SVP and GM of Global Customer Experience at Cisco, about the factors that create good customer experience – and...
      Read more
      Cloud

      IGEL CEO Jed Ayres on Edge and...

      James Maguire - June 14, 2022 0
      I spoke with Jed Ayres, CEO of IGEL, about the endpoint sector, and an open source OS for the cloud; we also spoke about...
      Read more
      Big Data and Analytics

      GoodData CEO Roman Stanek on Business Intelligence...

      James Maguire - May 4, 2022 0
      I spoke with Roman Stanek, CEO of GoodData, about business intelligence, data as a service, and the frustration that many executives have with data...
      Read more
      Logo

      eWeek has the latest technology news and analysis, buying guides, and product reviews for IT professionals and technology buyers. The site’s focus is on innovative solutions and covering in-depth technical content. eWeek stays on the cutting edge of technology news and IT trends through interviews and expert analysis. Gain insight from top innovators and thought leaders in the fields of IT, business, enterprise software, startups, and more.

      Facebook
      Linkedin
      RSS
      Twitter
      Youtube

      Advertisers

      Advertise with TechnologyAdvice on eWeek and our other IT-focused platforms.

      Advertise with Us

      Menu

      • About eWeek
      • Subscribe to our Newsletter
      • Latest News

      Our Brands

      • Privacy Policy
      • Terms
      • About
      • Contact
      • Advertise
      • Sitemap
      • California – Do Not Sell My Information

      Property of TechnologyAdvice.
      © 2022 TechnologyAdvice. All Rights Reserved

      Advertiser Disclosure: Some of the products that appear on this site are from companies from which TechnologyAdvice receives compensation. This compensation may impact how and where products appear on this site including, for example, the order in which they appear. TechnologyAdvice does not include all companies or all types of products available in the marketplace.

      ×