Why Hadoop Analytics Projects Often Fall Short of Their Goals

by Chris Preimesberger

Hadoop deployments often start as “sandbox” environments. Over time, the workload expands, and an increasing number of jobs on the cluster support production applications that are governed by service-level agreements. Ad hoc queries and other low-priority jobs compete with business-critical applications for system resources, causing high-priority jobs to fail to complete when needed.

Hadoop diagnostic tools are static, providing log file information about what happened to jobs on the cluster after they have run. Hadoop does not provide the ability to monitor at a sufficiently granular level what’s happening while multiple jobs are running. As a result, it is difficult and often impossible to take corrective action to prevent operational problems before they occur.

Various Hadoop diagnostic tools provide the ability to analyze individual job statistics and examine activity on individual nodes on the cluster. In addition, developers can tune their code to ensure optimal performance for individual jobs. What’s lacking, however, is the ability to monitor, analyze and control what’s happening with all users, jobs and tasks running on the entire cluster, including use of each hardware resource.

While job schedulers and resource managers provide basic capabilities such as job sequencing, time- and event-based scheduling and node allocation, they are insufficient in ensuring that cluster resources are being used in the most efficient manner while jobs are running.

Organizations typically size their clusters for maximum peak workloads. The extra capacity that is rarely used can be very expensive and often unnecessary.

When rogue jobs, inefficient or expensive queries, or other processes running on the cluster adversely impact performance, it is often too late for Hadoop operators to take the necessary corrective actions before service-level agreements are missed.

When jobs crash or fail to complete on time, Hadoop operators/administrators have difficulty diagnosing performance problems. Hadoop does not provide a way of monitoring and analyzing cluster performance with sufficient context and detail. For example, it is impossible to isolate problems by user, job, or task and pinpoint bottlenecks related to network, memory or disk.

More jobs, different kinds of jobs, expanding data volumes, different data types, more complex queries and many other variables continuously increase the load on cluster resources over time. Often, the need for additional cluster resources isn’t apparent until a disaster occurs (for example, a customer-facing Website goes down or a mission-critical report doesn’t run). Consequences can include unsatisfactory customer experiences, missed business opportunities, unplanned capital expense requests and more.

Contention for system resources by HBase and MapReduce jobs can affect overall performance significantly. The inability to optimize resource utilization while these different types of workloads run concurrently leads many organizations to suffer the expense of deploying separate, dedicated clusters.

The ones at issue enable interactive exploration and fast diagnoses of performance-related issues on the cluster. The static reports and detailed log files provided with Hadoop schedulers and resource managers are not conducive to fast or easy problem diagnosing. Culling through voluminous data while troubleshooting can waste hours or even days. Hadoop operators need the ability to quickly visualize, analyze and understand the causes of performance problems and identify opportunities to optimize resource use.