Close
  • Latest News
  • Artificial Intelligence
  • Big Data and Analytics
  • Cloud
  • Networking
  • Cybersecurity
  • Applications
  • IT Management
  • Storage
  • Sponsored
  • Mobile
  • Small Business
  • Development
  • Database
  • Servers
  • Android
  • Apple
  • Innovation
  • Blogs
  • PC Hardware
  • Reviews
  • Search Engines
  • Virtualization
Read Down
Sign in
Close
Welcome!Log into your account
Forgot your password?
Read Down
Password recovery
Recover your password
Close
Search
Logo
Logo
  • Latest News
  • Artificial Intelligence
  • Big Data and Analytics
  • Cloud
  • Networking
  • Cybersecurity
  • Applications
  • IT Management
  • Storage
  • Sponsored
  • Mobile
  • Small Business
  • Development
  • Database
  • Servers
  • Android
  • Apple
  • Innovation
  • Blogs
  • PC Hardware
  • Reviews
  • Search Engines
  • Virtualization
More
    Home Applications
    • Applications
    • Big Data and Analytics
    • Development
    • IT Management
    • Storage

    Apache Kafka Survey Reveals Growing Importance of Streaming Data

    By
    Chris Preimesberger
    -
    August 1, 2017
    Share
    Facebook
    Twitter
    Linkedin
      Kafka.logo

      As the overall volume of data from machines and human-operated devices–mostly from machines–continues to zoom up like an F-18 off an aircraft carrier, IT systems sometimes find themselves straining to keep that data moving from place to place at internet speed.

      All those moving bits are becoming heavy loads for systems old and new. Server and storage software needs to refreshed all the time in order to keep up, or else an enterprise can fall behind its competitors in delivery of services to customers.

      At the outset of the IoT and IIoT (industrial internet of things) age, data center operating systems like Nutanix, OpenStack and Mesosphere are three of the engines that are able to scale up in real time as needed by workload demands.

      One thing in common they have is an element called Apache Kafka, an open-source stream processing platform written in Scala and Java by Jay Kreps, Neha Narkhede and Jun Rao while they were working at LinkedIn. It was then open-sourced through the Apache Software Foundation. Kafka provides a unified, high-throughput, low-latency platform for handling real-time data feeds.

      Kafka Handles Huge Amounts of Data

      Unlike traditional enterprise messaging software, Kafka is able to handle all the data flowing through a company, and do it in near real time. Kafka is a high-performance publish/subscribe message bus designed for high availability and data durability, with minimal latency. Kafka acts as a central data backbone and enables loose coupling between applications (Akka), data processing (Spark) and data persistence (Cassandra) services.

      The problem of managing data in enterprises is an exercise in complexity. As companies add new applications, integrate and modernize existing systems, change to a microservices architecture and embrace a factor increase in data-producing endpoints, their infrastructure, and the engineers who support it, face an existential question: Should we pursue each new data project as a unique discrete effort, or find an overall approach with greater scalability?

      For many IT systems people, Kafka has become the answer to this question. It transforms an ever-increasing number of new data producers and consumers into a simple, unified streaming platform at the center of their organization. It allows any team to join the platform, allows a central team to manage the service, and scales to trillions of messages per day while delivering messages in real time.

      Survey Encompassed 350 Organizations

      To find out more about why and how companies are using streaming data and the impact it has on their business, data management platform maker Confluent recently surveyed 350 enterprises or organizations from 47 countries and a wide variety of industries to understand the evolving Apache Kafka user base, use cases and deployments.

      Following are some highlights from the survey and seven key data points that show where Kafka is headed in the enterprise.

      Data Point 1: Kafka use is experiencing a surge.

      Eighty-six percent of respondents reported that the number of their systems that use Kafka is increasing and one-fifth reported that the number is “growing a lot!” A majority (52 percent) of organizations have at least six systems running Kafka with 21 percent having more than 20. According to last year’s report, only 41 percent of organizations had at least six systems running Kafka, and only one-tenth had more than 20.

      Data Point 2: Kafka enables companies to create new market opportunities.

      Because data is available, shared and immediate, companies can create new products and significantly transform existing ones. As Kafka becomes deployed in more mission-critical infrastructures, a majority (54 percent) of surveyed organizations say that their business can make more accurate and/or quicker decisions thanks to Kafka, and 28 percent of respondents were able to identify new business opportunities through the use of Kafka.

      Data Point 3: Kafka is being used widely in the cloud.

      Kafka is used by organizations in some combination of virtual private clouds (34 percent), public clouds (52 percent) and on premises (57 percent). The survey also found that 60 percent of respondents use AWS as their public cloud, processing billions of messages per day.

      Data Point 4: Kafka is being used beyond data pipelines in the enterprise.

      Last year, the survey reported that a surge of companies adopted streaming platforms, while this year, the rise of microservices is leading to more diverse use cases of Kafka. Two-thirds (66 percent) of respondents use Kafka for stream processing, three out of five (60 percent) use it for data integration, and half (50 percent) are using Kafka for messaging and log aggregation.

      Data Point 5: Applications utilizing Kafka stretch far and wide.

      Similar to last year’s report, three-fourths (75 percent) of organizations have applications connected to their Kafka systems. These applications process data from sensors, websites, analytics and monitoring tools and share the information so the right teams can process the data they need to make decisions. The types of applications that a majority of respondents connect to with Kafka are asynchronous applications (57 percent) and data warehouse (51 percent). Organizations also have application monitoring (41 percent), system monitoring (30 percent), or recommendation/decision engines (30 percent) powered by Kafka.

      Data Point 6: The Kafka Connect API is bringing new data into the streaming era.

      The popularity of the Kafka Connect API has grown significantly over the past year. The Kafka Connect API, included in Kafka, makes it easy to add new datastores to your data pipelines without having to write the interfaces from scratch. There was a 15-point increase in organizations using the Kafka Connect API over last year (37 percent in 2017 versus 12 percent in 2016). A majority (59 percent) of respondents have databases connected to their Kafka clusters, with 25 percent of respondents connecting websites and 15 percent connecting sensors and device data to their clusters.

      Data Point 7: There is a Kafka engineer shortage.

      According to a Dice report, people with Kafka skills receive one of the highest salaries in the technology market. However, despite the salary and the growth of Kafka within organizations, the survey found that three-quarters (75 percent) of respondents find it at least somewhat difficult to find the right talent with Kafka skills.

      Editor’s note: This article was corrected to credit the creators of Kafka.

      Chris Preimesberger
      https://www.eweek.com/author/cpreimesberger/
      Chris J. Preimesberger is Editor Emeritus of eWEEK. In his 16 years and more than 5,000 articles at eWEEK, he distinguished himself in reporting and analysis of the business use of new-gen IT in a variety of sectors, including cloud computing, data center systems, storage, edge systems, security and others. In February 2017 and September 2018, Chris was named among the 250 most influential business journalists in the world (https://richtopia.com/inspirational-people/top-250-business-journalists/) by Richtopia, a UK research firm that used analytics to compile the ranking. He has won several national and regional awards for his work, including a 2011 Folio Award for a profile (https://www.eweek.com/cloud/marc-benioff-trend-seer-and-business-socialist/) of Salesforce founder/CEO Marc Benioff--the only time he has entered the competition. Previously, Chris was a founding editor of both IT Manager's Journal and DevX.com and was managing editor of Software Development magazine. He has been a stringer for the Associated Press since 1983 and resides in Silicon Valley.
      Get the Free Newsletter!
      Subscribe to Daily Tech Insider for top news, trends & analysis
      This email address is invalid.
      Get the Free Newsletter!
      Subscribe to Daily Tech Insider for top news, trends & analysis
      This email address is invalid.

      MOST POPULAR ARTICLES

      Latest News

      Zeus Kerravala on Networking: Multicloud, 5G, and...

      James Maguire - December 16, 2022 0
      I spoke with Zeus Kerravala, industry analyst at ZK Research, about the rapid changes in enterprise networking, as tech advances and digital transformation prompt...
      Read more
      Applications

      Datadog President Amit Agarwal on Trends in...

      James Maguire - November 11, 2022 0
      I spoke with Amit Agarwal, President of Datadog, about infrastructure observability, from current trends to key challenges to the future of this rapidly growing...
      Read more
      IT Management

      Intuit’s Nhung Ho on AI for the...

      James Maguire - May 13, 2022 0
      I spoke with Nhung Ho, Vice President of AI at Intuit, about adoption of AI in the small and medium-sized business market, and how...
      Read more
      Cloud

      IGEL CEO Jed Ayres on Edge and...

      James Maguire - June 14, 2022 0
      I spoke with Jed Ayres, CEO of IGEL, about the endpoint sector, and an open source OS for the cloud; we also spoke about...
      Read more
      Applications

      Kyndryl’s Nicolas Sekkaki on Handling AI and...

      James Maguire - November 9, 2022 0
      I spoke with Nicolas Sekkaki, Group Practice Leader for Applications, Data and AI at Kyndryl, about how companies can boost both their AI and...
      Read more
      Logo

      eWeek has the latest technology news and analysis, buying guides, and product reviews for IT professionals and technology buyers. The site’s focus is on innovative solutions and covering in-depth technical content. eWeek stays on the cutting edge of technology news and IT trends through interviews and expert analysis. Gain insight from top innovators and thought leaders in the fields of IT, business, enterprise software, startups, and more.

      Facebook
      Linkedin
      RSS
      Twitter
      Youtube

      Advertisers

      Advertise with TechnologyAdvice on eWeek and our other IT-focused platforms.

      Advertise with Us

      Menu

      • About eWeek
      • Subscribe to our Newsletter
      • Latest News

      Our Brands

      • Privacy Policy
      • Terms
      • About
      • Contact
      • Advertise
      • Sitemap
      • California – Do Not Sell My Information

      Property of TechnologyAdvice.
      © 2022 TechnologyAdvice. All Rights Reserved

      Advertiser Disclosure: Some of the products that appear on this site are from companies from which TechnologyAdvice receives compensation. This compensation may impact how and where products appear on this site including, for example, the order in which they appear. TechnologyAdvice does not include all companies or all types of products available in the marketplace.

      ×