Close
  • Latest News
  • Cybersecurity
  • Big Data and Analytics
  • Cloud
  • Mobile
  • Networking
  • Storage
  • Applications
  • IT Management
  • Small Business
  • Development
  • Database
  • Servers
  • Android
  • Apple
  • Innovation
  • Blogs
  • PC Hardware
  • Reviews
  • Search Engines
  • Virtualization
Read Down
Sign in
Close
Welcome!Log into your account
Forgot your password?
Read Down
Password recovery
Recover your password
Close
Search
Menu
eWEEK.com
Search
eWEEK.com
  • Latest News
  • Cybersecurity
  • Big Data and Analytics
  • Cloud
  • Mobile
  • Networking
  • Storage
  • Applications
  • IT Management
  • Small Business
  • Development
  • Database
  • Servers
  • Android
  • Apple
  • Innovation
  • Blogs
  • PC Hardware
  • Reviews
  • Search Engines
  • Virtualization
More
    Home Applications
    • Applications
    • Cloud
    • Storage

    Latest Google Gmail Outage Caused by Insufficient Router Capacity

    By
    CLINT BOULTON
    -
    September 2, 2009
    Share
    Facebook
    Twitter
    Linkedin

      Google’s Gmail application was knocked out for the majority of users for 100 minutes on Sept. 1 due to human error, the company said last night after the Gmail engineering team fixed the issue.

      Google took a small fraction of Gmail’s servers offline to perform routine upgrades. Google does this regularly, sending traffic to other locations when one is offline. That’s when things got hairy, as Ben Treynor, vice president of engineering and site reliability czar for Google, explained:

      ““We had slightly underestimated the load which some recent changes (ironically, some designed to improve service availability) placed on the request routers-servers which direct web queries to the appropriate Gmail server for response. At about 12:30 pm Pacific a few of the request routers became overloaded and in effect told the rest of the system “stop sending us traffic, we’re too slow!” This transferred the load onto the remaining request routers, causing a few more of them to also become overloaded, and within minutes nearly all of the request routers were overloaded. As a result, people couldn’t access Gmail via the Web interface because their requests couldn’t be routed to a Gmail server.”“

      Through internal monitors, Treynor said the Gmail engineering team was alerted to the failures within seconds and added several request routers online to make up for the dearth in capacity and distributed the traffic across the request routers. Gmail came back online around 2:30 p.m. PDT.

      To ensure this lack of server capacity-which is ironic considering that Google allegedly powers the world’s most popular search engine with more than 1 million servers-doesn’t happen again, Google boosted request router capacity well beyond peak demand for extra juice when the application needs it.

      Treynor also said Google is improving the failure isolation in the routers, so a problem in one data center won’t affect servers in another facility. Moreover, he said that Google is taking steps to make sure that when the request routers are overloaded simultaneously, they all should just get slower instead of refusing to accept traffic and shifting their load to another data center.

      It’s also worth noting that when Gmail did go down, Google urged users to access it via the IMAP and POP mail protocols; mail processing continued to work normally because these requests don’t use the same routers at Google.

      “We know how many people rely on Gmail for personal and professional communications, and we take it very seriously when there’s a problem with the service,” Treynor added. “Thus, right up front, I’d like to apologize to all of you-today’s outage was a Big Deal, and we’re treating it as such.”

      So are the Gmail users who use Gmail for their businesses. Donald told Google Watch: “I use G-Mail to run my CPA practice. This is a serious (huge) problem.”

      Sergei added: “This is a huge problem and an outrage. I demand immediate Gmail access. What is with those people?”

      Indeed, more than 1.75 million businesses use Google Apps and some of them pay Google $50 per user, per year for the Google Apps collaboration suite, which boasts Gmail as its backbone. Users have little patience for a service that conks out on them, particularly when they are paying for the extra reliability and security. Read more about this on TechMeme here.

      The latest issue follows a big outage in February, when Gmail went down for two and a half hours due to “unexpected side effects of some new code.” But these last two issues were nothing compared with the August 2008 outage that took Gmail down for nearly 15 hours.

      MOST POPULAR ARTICLES

      Android

      Samsung Galaxy XCover Pro: Durability for Tough...

      CHRIS PREIMESBERGER - December 5, 2020 0
      Have you ever dropped your phone, winced and felt the pain as it hit the sidewalk? Either the screen splintered like a windshield being...
      Read more
      Cloud

      Why Data Security Will Face Even Harsher...

      CHRIS PREIMESBERGER - December 1, 2020 0
      Who would know more about details of the hacking process than an actual former career hacker? And who wants to understand all they can...
      Read more
      Cybersecurity

      How Veritas Is Shining a Light Into...

      EWEEK EDITORS - September 25, 2020 0
      Protecting data has always been one of the most important tasks in all of IT, yet as more companies become data companies at the...
      Read more
      Big Data and Analytics

      How NVIDIA A100 Station Brings Data Center...

      ZEUS KERRAVALA - November 18, 2020 0
      There’s little debate that graphics processor unit manufacturer NVIDIA is the de facto standard when it comes to providing silicon to power machine learning...
      Read more
      Apple

      Why iPhone 12 Pro Makes Sense for...

      WAYNE RASH - November 26, 2020 0
      If you’ve been watching the Apple commercials for the past three weeks, you already know what the company thinks will happen if you buy...
      Read more
      eWeek


      Contact Us | About | Sitemap

      Facebook
      Linkedin
      RSS
      Twitter
      Youtube

      Property of TechnologyAdvice.
      Terms of Service | Privacy Notice | Advertise | California - Do Not Sell My Info

      © 2020 TechnologyAdvice. All Rights Reserved

      Advertiser Disclosure: Some of the products that appear on this site are from companies from which TechnologyAdvice receives compensation. This compensation may impact how and where products appear on this site including, for example, the order in which they appear. TechnologyAdvice does not include all companies or all types of products available in the marketplace.

      ×