Top 5 Big Data Trends Highlighted in Major Enterprise Projects

NEWS ANALYSIS: Major enterprise projects presented at the Big Data Innovation Conference show why this analysis technique is becoming more of a reality than just a buzzword.

Big data often comes across as more of a buzzword than reality. But listening to users speak about their projects at the Big Data Innovation conference in Boston last week provided insight into what the power of big data can mean to a company. Here are my top five takeaways from the conference.

1.) The hybrid data cloud. I blogged about this previously, but it is worth repeating. Enterprises are not about to abandon those structured data infrastructures. Structured data from the likes of Oracle, IBM and Microsoft underpin the operations of most big companies.

The goal of the technology data infrastructure executive is to blend those existing systems with hybrid systems incorporating unstructured, external data. However, traditional vendors should not breathe too easy. While the existing system will remain, chart after chart in customer presentations had those traditional vendors confined in existing boxes while the new money was going to new vendors and new platforms.

StubHub has a data network of 25 structured and unstructured data sources. Sastry Malladi, chief data architect for StubHub, said using open-source products is important to avoid proprietary architecture lock-ins. "Right now the biggest innovation is how to create a hybrid data system," said Malladi.

2.) Mobility is driving big data investment. Mobile platforms with their location, communications and portability present a consumer platform custom-made of big data innovation. MapMy Fitness started as way to map running routes and has expanded to a wide variety of fitness activities as well as personal health monitoring.

Matt McLure, the vice president of MapMyFitness, has seen the company grow to 19 million users and developed a hybrid private and public cloud infrastructure to match capacity to user activity such as increased bikers in the summer and fitness enthusiasts, who are following through on fitness resolutions after the start of the New Year.

"We are at the center of the ecosystem of health and connected fitness," said McLure. The scaling demands associated with the additional health and fitness monitoring is driving the company to use the data techniques developed by the likes of Facebook and Google.

3.) Big data can surround and enhance existing applications. StubHub started as a ticket exchange for sporting and entertainment events. The company now is taking a wider perspective on all the activities surrounding an event, including social commentary, lodging, dining and transportation services.

Those social network services are driving the hybrid model where huge amounts of data are captured, analyzed and driving recommendation engines. Traditional transaction systems were simply not designed for that type of user input.

4.) The Internet of Things will make current big data projects look like small stuff. Paul Bachteal, the senior director of the Americas technology practice for business intelligence vendor SAS, noted that "billion is the new million" when you start to consider all the data that will pour into organizations as the Internet of Things moves from concept to reality.

The skills needed to build systems that capture, store, analyze and create predictive analysis are in short supply, and customers and vendors will have to be innovative in training employees for the new skills development. Bachteal gave the example of railroad locomotives, which once equipped with sensors tied to a data analysis system, allowed customers to more accurately anticipate parts wear to prevent equipment malfunctions.

5.) Big innovation is coming to the front end of the data spectrum. Walmart is considering using crowd sourcing to set product prices and make image selections to accompany product descriptions. Digvijay Lamba, senior director of engineering for Walmart Labs, said the use of techniques such as crowd sourcing at the front end of the decision process completes the big data spectrum.

Existing big data systems are good at analyzing vast pools of data once developed, but are only as good as the data that enters the system. Crowd sourcing represents a way to add additional data at the front end of the big data process and will improve the analytical results. "We need to scale up the front end of the systems," said Lamba.

Big data is more than a buzzword, but creating big data systems requires new ways of thinking about decision systems, which are just now coming into the marketplace.

Eric Lundquist is a technology analyst at Ziff Brothers Investments, a private investment firm. Lundquist, who was editor-in-chief at eWEEK (previously PC WEEK) from 1996-2008 authored this article for eWEEK to share his thoughts on technology, products and services. No investment advice is offered in this article. All duties are disclaimed. Lundquist works separately for a private investment firm which may at any time invest in companies whose products are discussed in this article and no disclosure of securities transactions will be made.