Using this week’s Spark Summit in San Francisco as a backdrop, Microsoft announced the general availability of Spark for Azure HDInsight, the company’s cloud-based big data analytics platform. As its name suggests, Apache Spark is taking the big data world by storm with an open-source data processing engine that prioritizes speed and ease of use.
This week, Microsoft said it is more fully embracing Spark by bringing Spark for Azure HDInsight out of beta. Tiffany Wissner, senior director of Microsoft Data Platform Marketing, described the product as “a fully managed Spark service from Hortonworks that has been hardened for the enterprise and made simpler for you to use,” in a June 6 announcement. “You can also rely on the industry’s highest availability service level agreement for Spark at 99.9 percent.”
In a bid to speed up deployments, Microsoft has embarked on several product integrations, including working with Hortonworks—the lead commercial sponsor of Hadoop, on which HDInsight is based—to enhance the YARN resource manager. The company also teamed with Cloudera to co-lead Livy, a REST-based Spark service (currently in alpha) that is intended to provide an easy way for applications to interface with Spark setups.
“Spark gives you fast big data processing with a general purpose flexible API,” said Anand Iyer, senior product manager at Cloudera, in a statement. “We see a natural tendency among our customers and partners to want to leverage Spark’s capabilities from client applications that can easily interface with Spark, and Livy makes that possible.”
Microsoft announced that the company plans to make R Server for HDInsight, complete with Spark integration, available on the cloud this summer. R is a statistical programming language that is generally favored by data scientists.
“This makes it easy to move code and projects to the cloud with a few clicks and within a few minutes without buying hardware or hiring specialized operations teams typically associated with big data infrastructure,” explained Wissner.
In June, the on-premises version of R Server for Hadoop will support native Spark execution frameworks, she added. Microsoft estimates that by combining R Server with Spark, customers will be able to train their statistical models on larger data sets and 100 times faster than open-source R and nearly two times faster than Spark’s own machine learning library (MLLib) by running R functions over thousands of nodes in a Spark cluster.
Also new is Microsoft R Client, a free R client for data scientists. In addition to enabling analytics on locally stored data, the software can be used on production instances of HD Insight with Spark, along with SQL Server R Services and R Server for Hadoop, said Wissner.
Finally, on the data visualization front, Microsoft announced that Power BI now supports Spark Streaming. By incorporating Spark’s stream processing technology, customers can now publish real-time events to Power BI’s dashboards for an up-to-the-second updates on key business metrics.