Hadoop, Yahoo, 'Big Data' Brighten BI Future

By Chris Preimesberger  |  Posted 2011-08-15 Print this article Print

Yahoo CTO Raymie Stata is a key player in leading "big data" analytics engines. With Hadoop moving out of Yahoo and IBM investing in big data, big data is looking to transform businesses and IT departments.

An increasing number of jumbo-size enterprise data sets-and all the technology needed to create, store, network, analyze, archive and retrieve them-are considered "big data." This massive amount of information is pushing the limits on storage, servers and security, creating an immense problem for IT departments that must be addressed.

So what's the tipping point? When does average-size data become big data?

eWEEK's crack at this definition, with help from research firm Gartner, goes like this: "Big data refers to the volume, variety and velocity of structured and unstructured data pouring through networks into processors and storage devices, along with the conversion of such data into business advice for enterprises."

These elements can be broken down into three distinct categories: volume, variety and velocity.

Volume (terabytes, petabytes and eventually exabytes): The increasing amount of business data-created by both humans and machines-is putting a major hit on IT systems, which are struggling to store, secure and make accessible all that information for future use.

Variety: Big data is also about the increasing number of data types that need to be handled differently from simple email, data logs and credit card records. These include sensor- and other machine-gathered data for scientific studies, health care records, financial data and rich media: photos, graphic presentations, music, audio and video.

Velocity: It's about the speed at which this data moves from endpoints into processing and storage.

Big Data: Tools, Processes and Procedures

"In simplest terms, the phrase [big data] refers to the tools, processes and procedures allowing an organization to create, manipulate, and manage very large data sets and storage facilities," analyst Dan Kusnetzky of the Kusnetzky Group, wrote in his blog. "Does this mean terabytes, petabytes or even larger collections of data?

"The answer offered by [IT] suppliers is -yes.' They would say, -You need our product to manage and make the best use of that mass of data.' Just thinking about the problems created by the maintenance of huge, dynamic sets of data gives me a headache."

In addition to volume, variety and velocity, there's another "v" that fits into the big data picture: value. Accurate analysis of big data provides value by helping businesspeople make the right decision at the right time.

Chris Preimesberger Chris Preimesberger was named Editor-in-Chief of Features & Analysis at eWEEK in November 2011. Previously he served eWEEK as Senior Writer, covering a range of IT sectors that include data center systems, cloud computing, storage, virtualization, green IT, e-discovery and IT governance. His blog, Storage Station, is considered a go-to information source. Chris won a national Folio Award for magazine writing in November 2011 for a cover story on Salesforce.com and CEO-founder Marc Benioff, and he has served as a judge for the SIIA Codie Awards since 2005. In previous IT journalism, Chris was a founding editor of both IT Manager's Journal and DevX.com and was managing editor of Software Development magazine. His diverse resume also includes: sportswriter for the Los Angeles Daily News, covering NCAA and NBA basketball, television critic for the Palo Alto Times Tribune, and Sports Information Director at Stanford University. He has served as a correspondent for The Associated Press, covering Stanford and NCAA tournament basketball, since 1983. He has covered a number of major events, including the 1984 Democratic National Convention, a Presidential press conference at the White House in 1993, the Emmy Awards (three times), two Rose Bowls, the Fiesta Bowl, several NCAA men's and women's basketball tournaments, a Formula One Grand Prix auto race, a heavyweight boxing championship bout (Ali vs. Spinks, 1978), and the 1985 Super Bowl. A 1975 graduate of Pepperdine University in Malibu, Calif., Chris has won more than a dozen regional and national awards for his work. He and his wife, Rebecca, have four children and reside in Redwood City, Calif.Follow on Twitter: editingwhiz

Submit a Comment

Loading Comments...
Manage your Newsletters: Login   Register My Newsletters

Rocket Fuel