IBM Extends Spark to Mainframe for Real-Time Insights

IBM's new z/OS Platform for Apache Spark gives data scientists and developers real-time access to mainframe data for faster analysis.

IBM big data

IBM has released its new z/OS Platform for Apache Spark to help give data scientists and developers real-time, secure access to mainframe data.

The new offering makes it easier and faster for users to access and analyze data in-place on the IBM z Systems mainframe to deliver real-time insights, IBM said.

Last year, IBM announced a commitment to Spark that included focusing more than 3,500 IBM researchers and developers to work on Spark-related projects. As part of that commitment, z Systems has also established a new GitHub organization for developers to collaborate on z/OS on Spark.

With IBM z/OS Platform for Apache Spark, Spark runs natively on the z/OS mainframe operating system so users can analyze data on the mainframe without having to do extract, transform and load (ETL) or break the connection between the analytics library and underlying file system.

"The z/OS Platform for Apache Spark allows clients to do in real time what previously took them days to weeks—gathering and analyzing various data points to enable them to take action to engage with customers and drive new sources of revenue," Kathryn Guarini, vice president of Offering Management for IBM z Systems, told eWEEK. "By integrating analytics directly into operational systems, IBM z/OS Platform for Apache Spark can help clients speed time to insights, minimize risk and save money. Prior to the IBM z/OS Platform for Apache Spark, clients would have to move z/OS data off-platform, in batches, and run analytics on it separately—a costly and time-consuming exercise that often saw insights perish before they could be acted upon."

Still viable in the world's leading data centers, IBM's z Systems handles data and transactions for major banks, insurers, retailers and other major industries, making the platform critical to enterprise IT. Adding Spark to the equation helps data scientists deliver critical analysis and insight faster and easier, IBM said. Developers and data scientists can use their expertise with programming languages such as Scala, Python, R and SQL to gain actionable insights, the company said.

"As businesses of all sizes transform into real-time digital organizations, they must be able to get a clear picture of all their enterprise data without the excessive time and risk of ETL," Rod Smith, IBM Fellow and head of Emerging Internet Technologies for the company, said in a statement. "With Apache Spark enabled natively on IBM platforms—now including z Systems—customers can perform analytics alongside the transactional systems that house key data, while drawing contextual insights from other data sources, enabling them to engage with customers and generate revenue in real time."

IBM z/OS Platform for Apache Spark consists of the Apache Spark core, Spark SQL, Spark Streaming, Machine Learning Library (MLlib) and Graphx, IBM said.

"For companies in today's fast-paced, highly personalized business environment, the ability to engage with a client in real time based on fresh, accurate insights can mean the difference between outcompeting and falling behind competitors," Guarini said. "The platform is a key driver in helping mainframe clients transform into cognitive businesses—businesses that are able to understand what their clients do and need, learn from this understanding and adjust processes based on this knowledge."

IBM is also working with three partners, DataFactZ, Rocket Software and Zementis, to create customized solutions with the new IBM z/OS Platform for Apache Spark.

DataFactZ is a new IBM partner that is working with Big Blue to develop Spark analytics based on Spark SQL and MLlib for data and transactions processed on the mainframe. Rocket Software is helping to extend z/OS to Apache Spark by enabling its clients to use its Rocket Launchpad solution on z/OS. And Zementis is complementing its in-transaction predictive analytics offering for z/OS with an execution engine for Apache Spark.

"Running Apache Spark natively on z/OS enables organizations to move their analytics closer to their data, instead of the other way around," said Bryan Smith vice president of research and development and CTO at Rocket Software, in a statement. "Customers interested in getting started with Apache Spark on z/OS can sign up for Rocket Launchpad, an engagement model designed to help organizations develop creative solutions to solve their most challenging data problems."