Show Me the Money!

Hadoop and Big Data represent huge opportunity in the IT space, and money is flowing into the market. Cloudera closed a $40 million round of venture funding and Accel Partners launched a new $100 million Big Data Fund to find the next hot company to enter the space.

Use of Hadoop is growing, as is the interest in Hadoop. As a case in point, Hadoop World 2009 drew about 500 attendees and Hadoop World 2010 about 750 attendees. This year, Hadoop World 2011 doubled last years attendance, with 1.500 attendees.

The Hadoop ecosystem continues to grow, with companies such as Karmasphere, Splunk, MapR, Talend and others announced new Hadoop-related products and services in conjunction with Hadoop World.

Hadoop is increasingly being used to thwart fraud. JPMorgan Chase is using Hadoop to look at what IP addresses are being used to try to breach its security systems or to attempt to defraud the company.

The explosion of data, along with Hadoop and other Big Data technologies, has launched a new role in many companies—the data scientist. JPMorgan Chase has about 50 data scientists and is looking to hire and train more.

eBay is building its new search engine, codenamed Project Cassini, on top of Hadoop.

JPMorgan Chase offers Hadoop as a shared service to its lines of business operations.

One challenge for Hadoop is that while it is hot technology, it really lacks a killer app. What will be the killer app for Hadoop?

Another challenge for Hadoop is its enterprise readiness. Is Hadoop ready for use by the typical business user or executive? Companies are moving in to help push the idea that Hadoop is ready for business.

Hadoop started out primarily for use in data processing for search index building, but then new use cases cropped up, including data mining, analytics, live content serving, real-time apps and storage.

LimitationsSome of the limitations holding back wider adoption of Hadoop include redundancy, uniformity, ease of use, multi-data center support and interoperability.

Marketing and operations are the biggest users of Big Data. According to a Forrester, survey, 45 percent of marketing departments use Big Data, as do 43 percent of operations departments.

According to the Forrester survey, 72 percent of respondents said they use Big Data to analyze transactional data. Meanwhile, 35 percent said they use it for social media data, 27 percent for geospatial, 27 percent for clickstream data and 13 percent for image data.

According to Forrester, 47 percent of survey respondents said they use commercial or closed-source Big Data tools, while 37 percent said they use open-source Big Data tools such as Hadoop, MapReduce and Cassandra.

As application areas go, Forrester says business intelligence, ERP and CRM are the biggest recipients of Big Data. Fifty-five percent of respondents to a Forrester survey said they use Big Data for business intelligence; 28 percent said they use it for ERP and CRM.