Hadoop 2020: The Future of Big Data in the Enterprise

By Darryl K. Taft

If you follow the developments of Apache Hadoop, you know that industry experts are claiming that it will only be used for 10 percent of data. However, according to a recent 2014 State of Database Technology Survey, 13 percent of respondents are already using Hadoop in production or pilots, indicating that it’s on the upswing. By 2020, Hadoop will be used across the enterprise.

Hadoop has the potential to completely reshape the IT infrastructure of many companies. The technology may be playing a larger role on the IT road map than in the enterprise right now, but this will flip in the future. By 2020, most enterprises will have IT strategies that leverage Hadoop, and it will be their greatest infrastructure investment.

Hadoop is about one-tenth the cost of legacy data warehouse technology. Given this, the technology will drive down costs while driving up usability, creating great efficiencies and freeing up budget to be allocated elsewhere.

New entrants in the IT industry will afford to create and manage their own data warehouses because of the low-cost of Hadoop. This has yet to be adopted due to the inherent complexity of the technology, but even so, companies are beginning to form with the sole purpose of making Hadoop user-friendly. With the current speed of innovation and lower cost of entry, in just five years, we will see small and midsize enterprises investing in their own Hadoop-based infrastructure.

Big data is only as good as the quality of data you have. In order to get the most out of data, large amounts of information need to be processed in real time. A recent study found that nearly half of all organizations are developing big data strategies to achieve better decision-making and identify new trends to better engage with customers. By 2010, Hadoop’s low price tag will allow companies of all sizes to adopt big data strategies and begin to benefit from them sooner.

Startups represent the future of the industry and are already following the Hadoop path rather than the traditional architectures for cost and efficiency reasons. As they grow, innovate and gain popularity, adoption of their services and Hadoop will correlate, leading to exponential growth.

As Hadoop is used more and the capabilities of YARN become fully realized, more useful opportunities leveraging technology like Apache Spark and Storm will emerge and quickly increase its potential. Even now, real-time/operational analytics are the fastest moving part of the Hadoop ecosystem, and by 2020, Hadoop will be relied on for day-to-day enterprise operations.

The Internet of things is only possible with instant data processing and prescriptive analytics. As more things enter the data ecosystem, the burden of processing will become greater and legacy technology will not be able to keep up. Hadoop will be able to, and by 2020, it will be a mission-critical foundation for many businesses tied to the Internet of things.

The lack of built-in security in Hadoop is an obstacle that enterprises face today, but new tools are emerging that address this issue, connecting Kerberos and MapReduce components and ensuring compatibility between data. By 2020, expect these issues to resolved and highly regulated organizations to be managing their secure data with Hadoop.

As each company connects to various partners, customers, vendors and things, the ocean of data grows and becomes intertwined with data lakes and rivers. By 2020, Hadoop will power this connected world and it will be used to address uses cases that require a mix of batch, real-time, streaming and interactive scenarios.