What Is Big Data?

Big data is comprised of structured data, which is data that fits cleanly into spreadsheets and databases, and unstructured data, which is human-generated information such as emails, videos, tweets, Facebook posts, call center conversations, closed circuit TV footage, mobile phone calls, and Website clicks. Unstructured “diverse data” now makes up more than 90 percent of the world’s data, and it is the fastest-growing form of information. Just as importantly, this “human information” is where the interesting things happen. This is where customers share their experiences with a brand and indicate their buying preferences. This is where teams of corporate lawyers look to find evidence of fraud and compliance violations.

Every 60 seconds, there are 98,000 new Tweets, 23,148 apps downloaded, 400,710 ad requests and 208,333 minutes of Angry Birds played. On a daily basis, it is estimated that we collectively produce 2.5 quintillion bytes of data, and the growth rate is so high that 90 percent of all information ever created was produced in the last two years alone. With information flowing through organizations at an enormous speed, those that recognize the value of their information early on are ultimately going to be more successful.

IDC recently predicted that the big data market is expected to grow from $3.2 billion in 2010 to $16.9 billion in 2015, representing 40 percent annual growth. The proliferation of consumer technology, particularly smartphone devices in the workplace, has meant that people are engaging with technology in new ways and as a result, producing more data today than ever before, and at a much faster rate.

The challenge is to understand 100 percent of big data, structured and unstructured, to unlock real business value through a powerful basis for analysis and action throughout the entire enterprise. By applying meaning-based computing to big data, businesses can get a view of all of their data, not only the 10 percent of “neat,” structured data, but the whole 100 percent.

Autonomy recently announced that its private cloud has surpassed 50 petabytes in size. Powered by Autonomy’s Intelligent Data Operating Layer (IDOL), the private cloud automatically recognizes concepts and patterns in the billions of structured and unstructured data files it ingests and indexes every day. Autonomy IDOL provides a common platform for a range of cloud-based Autonomy solutions, spanning marketing and revenue optimization, archiving, data protection, eDiscovery and information governance. The Autonomy private cloud uses the HP Converged Cloud to deliver a seamless, scalable and hybrid IT environment for its customers.

It is hard to grasp how large 50PB truly is; 50PB could hold 25 trillion pages of printed text. It would take about 25 billion floppy disks to store the same amount of information.

As of January 2012, the Library of Congress reported that it had collected about 285TB of Web archive data. A 50PB cloud would hold the entire contents of the Library of Congress 179 times.

An Oxford study reported that the human mind processes about 1PB per second. So, 50PB would store everything in your brain for nearly a minute.

A 50PB cloud would hold 665 years of HDTV video. It would hold the equivalent of 820,300 64GB iPhones.

77 million CDs (at 700MB) would occupy 50PB, so you could put the combined worldwide sales of all Pink Floyd albums, or the combined worldwide sales of all the works by U2 and Nirvana into a 50PB cloud.

You would have to convert 2.56 billion trees into paper to transfer the amount of data that can be held inside a 50PB cloud, according to researchers at the University of California at Berkeley.