IT Turning to the Cloud to Handle Growing Big Data Demands

Intro: Organizations are drowning in big data, according to a recent survey from Qubole. The resulting report, titled “An In-Depth Look at Big Data Trends and Challenges,” reveals that the number of enterprises with more than 100 terabytes in their data lake is significantly on the rise. Customer service and sales needs—as well as the ever-increasing challenges of IT planning—are driving this demand. To better manage it all, companies are turning to the cloud for big data processing, while seeking to invest in self-servicing analytics solutions. More than 400 data professionals took part in the survey, which was conducted by Dimensional Research. The following slide show presents selected findings from the survey, with charts provided courtesy of Qubole.

The average enterprise data lake size is sharply rising, with 44 percent of organizations now reporting that they have more than 100 terabytes. In 2017, just 36 percent of organizations had more than 100 terabytes.

Nearly three-quarters of organizations use the cloud for big data processing. In 2017, only 58 percent did so.

When asked what types of big data processing they perform, 75 percent of survey respondents listed app data integration. Ad hoc analytics came second, as cited by 53 percent of respondents, followed by streaming data (40 percent) and machine learning (also 40 percent).

In terms of big data business function drivers, customer service ranks No. 1, as cited by 40 percent of respondents. IT planning finished a close second, as cited by 38 percent, followed by sales (33 percent), finance (also 33 percent) and resource planning (32 percent).

Enterprise resource planning tools serve as the top source of big data, as cited by 57 percent of respondents. Both customer relationship management/customer experience and finance solutions ranked second, as cited by 38 percent of respondents.

More than three of 10 respondents said their organization uses Spark as a big data framework. One-quarter listed homegrown frameworks, and 17 percent cited Hive and HBase.

Less than one of 10 organizations support self-serviced big data analytics. But just over three of five are planning to do so.

Three-quarters of respondents said their company is dealing with a big data talent and resource gap. To address this, 79 percent said their organization plans to boost headcount.

Just 40 percent of organizations support more than 25 users per big data administrator. By using more advanced, available platforms, administrators should be able to serve more than 100 users, according to the report.

When asked about their biggest big data challenge, 44 percent of respondents cited a “lack of experience” that “slows progress.” Other top challenges included the inability to keep up with new data sources (as cited by 42 percent of respondents), constantly evolving use cases (41 percent) and an abundance of manual tasks (38 percent).