Enterprise Big Data Initiatives Move Out of the Sandbox

In 2012, enterprise adoption of big data initiatives rose faster than most had predicted. A recent Computerworld study of 300 large enterprises found the volume of data will increase by 60 percent in 2013, with 13 percent reporting they have big data initiatives already in place. Another 38 percent are implementing or likely to implement big data projects over the next year. Enterprises are forming dedicated big data teams and making big data a line item in their budgets. Requirements are becoming more clearly defined, and distinct patterns are emerging, such as scalability at the lowest cost, rapid-response query and analysis, and the ability to leverage existing standards-based tools that include SQL and business intelligence. Enterprises also expect built-in security and data availability features when handling mission-critical data.

Apache’s Hadoop gained momentum in the enterprise last year, exemplified by the October 2012 Hadoop World event with its 2,500 attendees—an increase from 500 attendees three years earlier. Banks and other financial institutions, communications operators, large retailers and others now take Hadoop seriously. However, big data initiatives are not only centered on the Hadoop platform. Business and IT challenges are being solved using a combination of different technologies that work in concert, such as private clouds, to manage data estates and a combination of both traditional databases and data warehouse environments, alongside Hadoop-based data management environments. A common theme across all enterprise big data projects is the drive to get up and running quickly without causing disruption to existing IT environments.

Over the coming year, cost will be one of the biggest hurdles when initiating big data projects. A recent analyst report showed that big data spending will hit $34 billion in 2013. Organizations in certain sectors must keep data online and available due to compliance regulations, and businesses also want to harness more raw data from multiple sources to conduct better analytics. Getting the right balance between efficiency and performance is the challenge. Data rates continue to grow, and building out capacity for future requirements is critical. Big data does not necessarily mean big budgets: The tolerance for $20,000 to $40,000 per terabyte of managed data is disappearing in the minds of enterprise decision makers.

There will be increased demand in 2013 for big data tools and applications that will be easier to use and will satisfy the business user, not just the technical or data scientist users. Many Hadoop-based technology capabilities are still immature and require unique specialized skills, with recent new-product announcements addressing this need, including announcements on Cloudera Impala and Microsoft Polybase. Some such capabilities exist today, making it easier to access the right data at the right time with the best set of tools.

Some of the established players that lack unique technology capabilities or expertise will make acquisitions in 2013. The two obvious top vendors are Oracle and IBM, which already have a diverse portfolio of products in the data management sector, but that also recognize that time to market is critical in gaining a stronger foothold with an all-encompassing big data offering for enterprise customers.