Close
  • Latest News
  • Artificial Intelligence
  • Video
  • Big Data and Analytics
  • Cloud
  • Networking
  • Cybersecurity
  • Applications
  • IT Management
  • Storage
  • Sponsored
  • Mobile
  • Small Business
  • Development
  • Database
  • Servers
  • Android
  • Apple
  • Innovation
  • Blogs
  • PC Hardware
  • Reviews
  • Search Engines
  • Virtualization
Read Down
Sign in
Close
Welcome!Log into your account
Forgot your password?
Read Down
Password recovery
Recover your password
Close
Search
Logo
Subscribe
Logo
  • Latest News
  • Artificial Intelligence
  • Video
  • Big Data and Analytics
  • Cloud
  • Networking
  • Cybersecurity
  • Applications
  • IT Management
  • Storage
  • Sponsored
  • Mobile
  • Small Business
  • Development
  • Database
  • Servers
  • Android
  • Apple
  • Innovation
  • Blogs
  • PC Hardware
  • Reviews
  • Search Engines
  • Virtualization
More
    Subscribe
    Home Applications
    • Applications
    • Cloud

    Google Cloud Dataflow Now Available on Apache Spark

    Written by

    Jaikumar Vijayan
    Published January 20, 2015
    Share
    Facebook
    Twitter
    Linkedin

      eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.

      Google announced Cloud Dataflow last June as a managed service designed to help companies ingest and analyze huge data sets both in batch processing and in real-time streaming mode.

      In December, the company released a Cloud Dataflow Software Development Kit into the open-source community to encourage software developers to write applications that integrate easily with the managed service as well as with other execution environments.

      One of the results of that move is a version of Cloud Dataflow that runs on Cloudera’s distribution of the open-source Apache Spark engine for large-scale data processing. The new Dataflow “runner” announced Jan. 20 by Cloudera and Google will allow developers to target a Dataflow pipeline for execution on a cloud-hosted or on-premise Spark cluster as well as on Google’s managed service.

      One of the most compelling aspects of Cloud Dataflow is its support for pipeline logic that can execute both in batch and streaming mode, Josh Wills, senior director of data science at Cloudera, said in the company’s blog post announcing the new development.

      Cloud Dataflow’s streaming capabilities are more advanced than those available with Spark Streaming while its batch execution engine optimizes the performance of pipelines that do not process streaming data, Wills said.

      Cloud Dataflow combines several major technologies that Google has used internally for years for large-scale data processing, including MapReduce, the FlumeJava batch processing engine and the MillWheel stream-processing engine. “Dataflow is a synthesis of our investments,” in data processing technologies, said Eric Schmidt, a product manager with Google’s Cloud Platform team. “From a developer’s perspective, it is a programming model and a managed service,” he said.

      The Cloud Dataflow SDK that Google released last December gives developers a way to write big data applications that combine batch and stream processing capabilities without the need for separate programming models or separate infrastructures for running them.

      “What they would have to do previously is run a different SDK,” for each mode Schmidt said. “You would either have a set of users doing a static MapReduce batch job, or you would have another camp [doing streaming analytics],” he said. “We wanted to merge both batch and stream and have one combined service infrastructure” for running both, he said.

      Google released the SDK into the open-source community in December to ensure that Dataflow is ported to other execution environments, as well, he said. The Cloudera Apache Spark announcement is one example of the direction that Google has in mind for Dataflow, he said.

      One of the key questions when Google first announced Dataflow was whether developers using the programming model would be locked into Google infrastructure for running their pipelines. “Our strategy has been to extend the SDK to open source so they can extend it to other environments,” Schmidt said.

      With Tuesday’s announcement, Cloud Dataflow now can run on Google’s infrastructure, a Spark cluster or a local machine, he said.

      Google’s moves are designed to better position the company in the emerging market for services and technologies that can help enterprises extract business value from massive data sets. Over the years, many companies have gotten better at harvesting all kinds of data from transactional systems, clickstreams, system logs, machine sensors, mobile devices and other sources. But they have struggled to extract value from it, both because of the limitations of traditional database management technologies and the complexity involved in building a data processing infrastructure for big data sets.

      Jaikumar Vijayan
      Jaikumar Vijayan
      Vijayan is an award-winning independent journalist and tech content creation specialist covering data security and privacy, business intelligence, big data and data analytics.

      Get the Free Newsletter!

      Subscribe to Daily Tech Insider for top news, trends & analysis

      Get the Free Newsletter!

      Subscribe to Daily Tech Insider for top news, trends & analysis

      MOST POPULAR ARTICLES

      Artificial Intelligence

      9 Best AI 3D Generators You Need...

      Sam Rinko - June 25, 2024 0
      AI 3D Generators are powerful tools for many different industries. Discover the best AI 3D Generators, and learn which is best for your specific use case.
      Read more
      Cloud

      RingCentral Expands Its Collaboration Platform

      Zeus Kerravala - November 22, 2023 0
      RingCentral adds AI-enabled contact center and hybrid event products to its suite of collaboration services.
      Read more
      Artificial Intelligence

      8 Best AI Data Analytics Software &...

      Aminu Abdullahi - January 18, 2024 0
      Learn the top AI data analytics software to use. Compare AI data analytics solutions & features to make the best choice for your business.
      Read more
      Latest News

      Zeus Kerravala on Networking: Multicloud, 5G, and...

      James Maguire - December 16, 2022 0
      I spoke with Zeus Kerravala, industry analyst at ZK Research, about the rapid changes in enterprise networking, as tech advances and digital transformation prompt...
      Read more
      Video

      Datadog President Amit Agarwal on Trends in...

      James Maguire - November 11, 2022 0
      I spoke with Amit Agarwal, President of Datadog, about infrastructure observability, from current trends to key challenges to the future of this rapidly growing...
      Read more
      Logo

      eWeek has the latest technology news and analysis, buying guides, and product reviews for IT professionals and technology buyers. The site’s focus is on innovative solutions and covering in-depth technical content. eWeek stays on the cutting edge of technology news and IT trends through interviews and expert analysis. Gain insight from top innovators and thought leaders in the fields of IT, business, enterprise software, startups, and more.

      Facebook
      Linkedin
      RSS
      Twitter
      Youtube

      Advertisers

      Advertise with TechnologyAdvice on eWeek and our other IT-focused platforms.

      Advertise with Us

      Menu

      • About eWeek
      • Subscribe to our Newsletter
      • Latest News

      Our Brands

      • Privacy Policy
      • Terms
      • About
      • Contact
      • Advertise
      • Sitemap
      • California – Do Not Sell My Information

      Property of TechnologyAdvice.
      © 2024 TechnologyAdvice. All Rights Reserved

      Advertiser Disclosure: Some of the products that appear on this site are from companies from which TechnologyAdvice receives compensation. This compensation may impact how and where products appear on this site including, for example, the order in which they appear. TechnologyAdvice does not include all companies or all types of products available in the marketplace.