Hadoop, Yahoo, 'Big Data' Brighten BI Future

An increasing number of jumbo-size enterprise data sets—and all the technology needed to create, store, network, analyze, archive and retrieve them—are considered “big data.” This massive amount of information is pushing the limits on storage, servers and security, creating an immense problem for IT departments that must be addressed.

So what’s the tipping point? When does average-size data become big data?

eWEEK's crack at this definition, with help from research firm Gartner, goes like this: “Big data refers to the volume, variety and velocity of structured and unstructured data pouring through networks into processors and storage devices, along with the conversion of such data into business advice for enterprises.”

These elements can be broken down into three distinct categories: volume, variety and velocity.

Volume (terabytes, petabytes and eventually exabytes): The increasing amount of business data—created by both humans and machines—is putting a major hit on IT systems, which are struggling to store, secure and make accessible all that information for future use.

Variety: Big data is also about the increasing number of data types that need to be handled differently from simple email, data logs and credit card records. These include sensor- and other machine-gathered data for scientific studies, health care records, financial data and rich media: photos, graphic presentations, music, audio and video.

Velocity: It’s about the speed at which this data moves from endpoints into processing and storage.

Big Data: Tools, Processes and Procedures

“In simplest terms, the phrase [big data] refers to the tools, processes and procedures allowing an organization to create, manipulate, and manage very large data sets and storage facilities,” analyst Dan Kusnetzky of the Kusnetzky Group, wrote in his blog. “Does this mean terabytes, petabytes or even larger collections of data?

“The answer offered by [IT] suppliers is ‘yes.’ They would say, ‘You need our product to manage and make the best use of that mass of data.’ Just thinking about the problems created by the maintenance of huge, dynamic sets of data gives me a headache.”

In addition to volume, variety and velocity, there’s another “v” that fits into the big data picture: value. Accurate analysis of big data provides value by helping businesspeople make the right decision at the right time.

Next: Whole Sets, Instead of Subsets >>