eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
Yes, the phenomenon of so-called “big data” has clearly made it to the buzzword level. Sometimes truly important technology trends, in fact, do become buzzwords.
To begin to qualify this, we can look at the projected numbers from the analysts—which, of course, are themselves computed by using big data sets. Big data storage (and the analytics processing that uses all that raw data) is a burgeoning market that IDC, for one, has forecast to surge at a 40 percent compound annual growth rate, from $3.2 billion in 2010 to $17 billion in 2015. The hockey-stick curve will certainly continue to zoom up-and-to-the-right after that.
Microsoft recently conducted a big data survey of its enterprise customers which found that almost two-thirds (62 percent) of respondents said they currently store at least 100 terabytes of data. Take a look at how much data that you, as a single consumer, store; it’s a safe bet that it is way more than you might have imagined. Then think of how much data enterprises have to manage on a daily basis or even an annual basis.
Big data presents a formidable new frontier in which data sets can grow so large that they become awkward to work with using traditional database management tools. The need for new tools, frameworks, hardware, software and services to handle this emerging issue represents a huge market opportunity.
One big factor that is driving the relentless growth of big data volumes is the incredible explosion of connected devices in the marketplace.
For example, Apple sold 51 million new iPhones in the last quarter of 2013 alone, and that represents only 15 percent of the worldwide market. Samsung owns 31 percent, thanks to its sale of 103 million Android phones. Sales of video cameras and sensors for machine data generation in enterprises are also off the charts. All these devices are generating huge volumes of data that have to be stored and managed.
Here are some good stats to consider: There are upwards of 6 billion people living on Earth. There are now an estimated 4 billion connected devices in the world. The market saturation of all those devices is continually producing data at nearly unfathomable rates.
There are immense new business opportunities in big data. There is demand for effective big data toolsets that provide scalable, high-performance analytics at the lowest cost and in near-real time as business users increasingly demand continuous access to data. By analyzing this data, companies gain greater intelligence as well as a competitive advantage.
eWEEK covers this beat nearly every day in some form or fashion, whether it’s from the storage hardware and software angle—by far the healthiest overall sector in all of IT—or from cloud services, or from analytics, or from innovation.
Big Data Analytics and Storage: Going Nowhere but Up in 2014
There’s news in each of those subsections, and from the coverage of all those news items (there’s yet another example of big data storage), we’ve been able to filter through it all and come up with some bona fide trends.
Let’s get to a few key ones now that have appeared in our pages in the last few months.
Trend No. 1: Mobility Clearly Driving Big Data Investment
Mobile platforms with their location, communications and portability present a consumer platform custom-made of big data innovation. For one example, MapMyFitness started as a way to map running routes and has expanded to a wide variety of fitness activities as well as personal health monitoring.
Matt McLure, the vice president of MapMyFitness, has seen the company grow to 19 million users and developed a hybrid private and public cloud infrastructure to match capacity to user activity such as increased bikers in the summer and fitness enthusiasts, who are following through on fitness resolutions after the start of the New Year.
“We are at the center of the ecosystem of health and connected fitness,” McLure told eWEEK contributor Eric Lundquist. The scaling demands associated with the additional health and fitness monitoring is driving the company to use the data techniques developed by the likes of Facebook and Google.
Trend No. 2: More Private and Hybrid Data Clouds Now Being Built
Enterprises are not about to abandon their structured data infrastructures. Structured data from the likes of Oracle, IBM, HP and Microsoft underpin the operations of most big companies.
The goal of the technology data infrastructure executive is to blend those existing systems with hybrid systems incorporating unstructured, external data. However, traditional vendors should not breathe too easy. While the existing system will remain, chart after chart in customer presentations had those traditional vendors confined in existing boxes while the new money was going to new vendors and new platforms.
StubHub has a data network of 25 structured and unstructured data sources. Sastry Malladi, chief data architect for StubHub, said using open-source products is important to avoid proprietary architecture lock-ins. “Right now the biggest innovation is how to create a hybrid data system,” Malladi said.
Trend No. 3: The Internet of Things Will Propel Use of Big Data Analytics Even Faster
Paul Bachteal, the senior director of the Americas technology practice for business intelligence vendor SAS, told eWEEK’s Lundquist that “billion is the new million” when you start to consider all the data that will pour into organizations as the Internet of things moves from concept to reality.
The skills needed to build systems that capture, store, analyze and create predictive analysis are in short supply, and customers and vendors will have to be innovative in training employees for the new skills development.
Big Data Analytics and Storage: Going Nowhere but Up in 2014
Bachteal gave the example of railroad locomotives, which, once equipped with sensors tied to a data analysis system, allowed customers to more accurately anticipate parts wear to prevent equipment malfunctions.
Trend No. 4: Line-of-Business Employees Fast Becoming Primary Users
Thanks to the continual improvement in user interfaces—specifically drag-down menus, easy to follow wizards and familiarity with such operating systems as Windows and iOS—and increasing availability through the cloud, more and more non-IT staff workers are able to access big data analytics on a 24/7 basis and actually use it to the enterprise’s advantage.
Trend No. 5: New Data-Centric Apps Will Become Mandatory
The ability to leverage big data will emerge as the competitive weapon in 2014. More companies will use big data and Hadoop (a fast, new-generation batch processing system) to pinpoint individual consumers’ preferences for profitable up-sell and cross-sell opportunities, better mitigate risk and reduce production and overhead costs.
Trend No. 6: Search Emerging as the Unstructured Query Language
There were a large number of SQL initiatives for Hadoop in 2013, and 2014 will be the year that the unstructured query language comes into full focus. Integrating search into Hadoop provides a simple intuitive method for any business user to locate important information. Search engines also are the core for many discovery and analysis applications, including recommendation engines.
As big data analytics continues to become mainstreamed, the sector will require big innovation to develop along with it. New applications and user interfaces will be needed for all the various new devices (read that “wearable computing”); the business opportunities in all of this are astounding. You’d better believe that venture capitalists are all over this sector.
For example, Walmart is considering using crowd sourcing to set product prices and make image selections to accompany product descriptions. Digvijay Lamba, senior director of engineering for Walmart Labs, said the use of techniques such as crowd sourcing at the front end of the decision process completes the big data spectrum.
Existing big data systems are good at analyzing vast pools of data once developed, but are only as good as the data that enters the system. Crowd sourcing represents a way to add additional data at the front end of the big data process and will improve the analytical results. “We need to scale up the front end of the systems,” Lamba said.
Yes, big data is more than simply a buzzword—there’s a lot happening behind those two simple words. It will be interesting to a lot of IT folks to see how this all plays out over the next year or two.