Close
  • Latest News
  • Artificial Intelligence
  • Big Data and Analytics
  • Cloud
  • Networking
  • Cybersecurity
  • Applications
  • IT Management
  • Storage
  • Sponsored
  • Mobile
  • Small Business
  • Development
  • Database
  • Servers
  • Android
  • Apple
  • Innovation
  • Blogs
  • PC Hardware
  • Reviews
  • Search Engines
  • Virtualization
Read Down
Sign in
Close
Welcome!Log into your account
Forgot your password?
Read Down
Password recovery
Recover your password
Close
Search
Logo
Logo
  • Latest News
  • Artificial Intelligence
  • Big Data and Analytics
  • Cloud
  • Networking
  • Cybersecurity
  • Applications
  • IT Management
  • Storage
  • Sponsored
  • Mobile
  • Small Business
  • Development
  • Database
  • Servers
  • Android
  • Apple
  • Innovation
  • Blogs
  • PC Hardware
  • Reviews
  • Search Engines
  • Virtualization
More
    Home Database
    • Database

    The Next Challenge for Hadoop: Quality of Service

    By
    Darryl K. Taft
    -
    April 22, 2016
    Share
    Facebook
    Twitter
    Linkedin

      PrevNext

      1The Next Challenge for Hadoop: Quality of Service

      1 - The Next Challenge for Hadoop: Quality of Service

      For Hadoop to move forward into the next decade, the community must address one key but often overlooked thing: quality of service.

      2What Is QoS for Hadoop?

      2 - What Is QoS for Hadoop?

      Quality of service for Hadoop is the best first step toward measuring Hadoop performance. QoS provides the ability to ensure performance service levels for applications running on Hadoop by enabling the prioritization of critical jobs and addressing problems like resource contention, missed deadlines and sluggish cluster performance. By avoiding bottlenecks and contention, multiple jobs can run side-by-side, effectively and without interference.

      3Why QoS for Hadoop?

      3 - Why QoS for Hadoop?

      Many companies run into roadblocks when they try to guarantee performance because priority jobs aren’t completed on time and clusters are underutilized. Resource contention is inevitable with today’s multi-tenant, multi-workload clusters, especially as big data applications scale. Why is this a problem? On the business side, companies waste time and money trying to fix cluster performance issues that prevent them from gaining competitive advantages linked to big data initiatives or realizing the full ROI of their big data efforts. From a technological perspective, unreliable Hadoop performance means late jobs, missed service-level agreements, overbuilt clusters and under-utilized hardware.

      4Hadoop, We Have a Problem

      4 - Hadoop, We Have a Problem

      As organizations get more advanced in their Hadoop use and run business-critical applications in multi-tenant clusters, they can no longer afford to lose sight of what’s happening from behind an increasingly insurmountable class of performance challenges—especially, if they want to make the most out of their distributed computing investments. Complicated frameworks like YARN already place performance pressure on systems, and if you look into the future at new compute platforms like Mesos, OpenStack and Docker, they will all run into this same set of widely applicable problems eventually. It’s vital that organizations get ahead of these issues now.

      5Getting Around Workarounds

      5 - Getting Around Workarounds

      Once a Hadoop cluster hits a performance wall, admins need to find a resolution but are discovering that traditional best practices and manual tuning workarounds just don’t work. Over-provisioning, silo-ing and tuning aren’t solutions that last long term; plus, they are very expensive and create needless overhead. Purchasing additional nodes when hardware utilization is well below 100 percent is a costly, temporary fix that only addresses performance symptoms, not the fundamental limitations of Hadoop. Similarly, cluster isolation is costly, doubles complexity and simply isn’t a viable solution at scale. Finally, tuning by definition is a response to problems that have already occurred, and it’s impossible for a human to make the thousands of decisions necessary to tune settings in real time to adjust to constantly changing cluster conditions.

      6Going Real Time

      6 - Going Real Time

      The most effective solution for resource contention is to monitor hardware resources in real time. Monitoring the hardware resources of each node in the cluster second-by-second allows you to understand which job has control over resources and to know the priority levels of each job across the cluster. This ensures that all jobs get access to cluster hardware resources in an equitable manner and business-critical jobs can finish on time, thereby guaranteeing QoS for Hadoop.

      7QoS for Hadoop in Production

      7 - QoS for Hadoop in Production

      Companies like Trulia, Chartboost and Upsight are implementing systems that guarantee QoS for Hadoop and reaping the benefits. Trulia has successfully disrupted a decades-old industry by using and analyzing real-time data to deliver customized insights straight to consumers. With many teams writing Hadoop jobs or using Hive or Spark, Trulia has to ensure reliability in its multi-tenant, multi-workload environment. In response to delayed or unpredictable jobs that affected their customer push-notification programs, Trulia would intentionally underutilize its clusters to ensure jobs were completed on time and prevent traffic from being negatively affected. Now, Trulia uses Pepperdata to actively monitor and control all their Hadoop clusters.

      PrevNext
      Get the Free Newsletter!
      Subscribe to Daily Tech Insider for top news, trends & analysis
      This email address is invalid.
      Get the Free Newsletter!
      Subscribe to Daily Tech Insider for top news, trends & analysis
      This email address is invalid.

      MOST POPULAR ARTICLES

      Latest News

      Zeus Kerravala on Networking: Multicloud, 5G, and...

      James Maguire - December 16, 2022 0
      I spoke with Zeus Kerravala, industry analyst at ZK Research, about the rapid changes in enterprise networking, as tech advances and digital transformation prompt...
      Read more
      Applications

      Datadog President Amit Agarwal on Trends in...

      James Maguire - November 11, 2022 0
      I spoke with Amit Agarwal, President of Datadog, about infrastructure observability, from current trends to key challenges to the future of this rapidly growing...
      Read more
      Cloud

      IGEL CEO Jed Ayres on Edge and...

      James Maguire - June 14, 2022 0
      I spoke with Jed Ayres, CEO of IGEL, about the endpoint sector, and an open source OS for the cloud; we also spoke about...
      Read more
      IT Management

      Intuit’s Nhung Ho on AI for the...

      James Maguire - May 13, 2022 0
      I spoke with Nhung Ho, Vice President of AI at Intuit, about adoption of AI in the small and medium-sized business market, and how...
      Read more
      Applications

      Kyndryl’s Nicolas Sekkaki on Handling AI and...

      James Maguire - November 9, 2022 0
      I spoke with Nicolas Sekkaki, Group Practice Leader for Applications, Data and AI at Kyndryl, about how companies can boost both their AI and...
      Read more
      Logo

      eWeek has the latest technology news and analysis, buying guides, and product reviews for IT professionals and technology buyers. The site’s focus is on innovative solutions and covering in-depth technical content. eWeek stays on the cutting edge of technology news and IT trends through interviews and expert analysis. Gain insight from top innovators and thought leaders in the fields of IT, business, enterprise software, startups, and more.

      Facebook
      Linkedin
      RSS
      Twitter
      Youtube

      Advertisers

      Advertise with TechnologyAdvice on eWeek and our other IT-focused platforms.

      Advertise with Us

      Menu

      • About eWeek
      • Subscribe to our Newsletter
      • Latest News

      Our Brands

      • Privacy Policy
      • Terms
      • About
      • Contact
      • Advertise
      • Sitemap
      • California – Do Not Sell My Information

      Property of TechnologyAdvice.
      © 2022 TechnologyAdvice. All Rights Reserved

      Advertiser Disclosure: Some of the products that appear on this site are from companies from which TechnologyAdvice receives compensation. This compensation may impact how and where products appear on this site including, for example, the order in which they appear. TechnologyAdvice does not include all companies or all types of products available in the marketplace.

      ×