Hadoop, Yahoo, 'Big Data' Brighten BI Future

Yahoo CTO Raymie Stata is a key player in leading "big data" analytics engines. With Hadoop moving out of Yahoo and IBM investing in big data, big data is looking to transform businesses and IT departments.

An increasing number of jumbo-size enterprise data sets-and all the technology needed to create, store, network, analyze, archive and retrieve them-are considered "big data." This massive amount of information is pushing the limits on storage, servers and security, creating an immense problem for IT departments that must be addressed.

So what's the tipping point? When does average-size data become big data?

eWEEK's crack at this definition, with help from research firm Gartner, goes like this: "Big data refers to the volume, variety and velocity of structured and unstructured data pouring through networks into processors and storage devices, along with the conversion of such data into business advice for enterprises."

These elements can be broken down into three distinct categories: volume, variety and velocity.

Volume (terabytes, petabytes and eventually exabytes): The increasing amount of business data-created by both humans and machines-is putting a major hit on IT systems, which are struggling to store, secure and make accessible all that information for future use.

Variety: Big data is also about the increasing number of data types that need to be handled differently from simple email, data logs and credit card records. These include sensor- and other machine-gathered data for scientific studies, health care records, financial data and rich media: photos, graphic presentations, music, audio and video.

Velocity: It's about the speed at which this data moves from endpoints into processing and storage.
Big Data: Tools, Processes and Procedures

"In simplest terms, the phrase [big data] refers to the tools, processes and procedures allowing an organization to create, manipulate, and manage very large data sets and storage facilities," analyst Dan Kusnetzky of the Kusnetzky Group, wrote in his blog. "Does this mean terabytes, petabytes or even larger collections of data?

"The answer offered by [IT] suppliers is -yes.' They would say, -You need our product to manage and make the best use of that mass of data.' Just thinking about the problems created by the maintenance of huge, dynamic sets of data gives me a headache."

In addition to volume, variety and velocity, there's another "v" that fits into the big data picture: value. Accurate analysis of big data provides value by helping businesspeople make the right decision at the right time.

Chris Preimesberger

Chris J. Preimesberger

Chris J. Preimesberger is Editor of Features & Analysis at eWEEK, responsible in large part for the publication's coverage areas. In his 12 years and more than 3,900 stories at eWEEK, he...