Close
  • Latest News
  • Artificial Intelligence
  • Video
  • Big Data and Analytics
  • Cloud
  • Networking
  • Cybersecurity
  • Applications
  • IT Management
  • Storage
  • Sponsored
  • Mobile
  • Small Business
  • Development
  • Database
  • Servers
  • Android
  • Apple
  • Innovation
  • Blogs
  • PC Hardware
  • Reviews
  • Search Engines
  • Virtualization
Read Down
Sign in
Close
Welcome!Log into your account
Forgot your password?
Read Down
Password recovery
Recover your password
Close
Search
Logo
Subscribe
Logo
  • Latest News
  • Artificial Intelligence
  • Video
  • Big Data and Analytics
  • Cloud
  • Networking
  • Cybersecurity
  • Applications
  • IT Management
  • Storage
  • Sponsored
  • Mobile
  • Small Business
  • Development
  • Database
  • Servers
  • Android
  • Apple
  • Innovation
  • Blogs
  • PC Hardware
  • Reviews
  • Search Engines
  • Virtualization
More
    Subscribe
    Home Applications
    • Applications
    • IT Management
    • Networking

    Lessons from Facebook’s Outage

    Written by

    P. J. Connolly
    Published September 27, 2010
    Share
    Facebook
    Twitter
    Linkedin

      eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.

      I wasn’t affected by the Facebook outage of Sept. 23, and I feel sorry for anyone who believes they were let down by the company. That’s not because the company owed these people access to their data. It’s because anyone whose happiness is based on the availability of Facebook, Twitter or any other Website either works for that company or needs to get a life. Although these companies’ business models are predicated on 24/7 availability, that’s just an unreasonable expectation, even today.

      I’ll admit that I’m guilty of abusing the Web for the purpose of feeding my almost infinite information jones. When I roll out of bed on any given morning, I head to the computer to see what’s happened in the world since the previous night, and I find myself visiting a range of sites; the normal routine includes the sites of traditional publishers such as The New York Times as well as less familiar sites such as the Saginaw News. Throw in some Web-only sites that specialize in IT news and a handful of webcomics, and only then am I ready to face the day.

      I expect these sites to be available, when I want them throughout the day, and as a rule, their track record is impressive. But I know better than to expect perfection; anyone seeking that is going to be disappointed in this lifetime, and probably the next as well. Even the best automated system is designed by human beings, and humans make mistakes. Sometimes mistakes cut you off from your family photos for a few hours and sometimes they cause the East Coast’s power grid to collapse.

      Maybe I’ve spent too much time in IT, and my expectations are therefore much lower than the millions who think that Websites “just happen.” But I know how difficult it can be to provide scalability and availability, and when one of my favorite sites is having a bad day, I may curse for a bit, but I usually stop when I think about what the poor slobs who are responding to the outage are going through. In pre-Internet days, I had a routine for responding to outages in my server room that included an important step: find my boss and tell him to keep his boss off my back until things were fixed.

      The fault with Facebook’s site was another example of why automated systems need to be tested out in all conceivable error conditions before they’re put into production use. The company had deployed a configuration verification system that was designed to check system caches for incorrect values and replace them with supposedly “good” values from a persistent store. The problem in this case was that the values from the persistent store were themselves incorrect.

      This led to a feedback loop that crippled a database cluster by throwing hundreds of thousands of queries at it every second. According to a blog post from Facebook Engineering Director Robert Johnson, the only way to fix the problem was to cut off all requests to the database cluster, which in effect shut down Facebook’s site. After disabling the automated configuration checker, the company was able to allow users back onto the site, its engineers having learned a lesson in foreseeing the unforeseeable.

      The Facebook outage lasted a couple of hours, and no user data appears to have been lost; this was by no means a disaster, just a much-needed dose of humility for Facebook’s engineering team.

      P. J. Connolly
      P. J. Connolly
      P. J. Connolly began writing for IT publications in 1997 and has a lengthy track record in both news and reviews. Since then, he's built two test labs from scratch and earned a reputation as the nicest skeptic you'll ever meet. Before taking up journalism, P. J. was an IT manager and consultant in San Francisco with a knack for networking the Apple Macintosh, and his love for technology is exceeded only by his contempt for the flavor of the month.

      Get the Free Newsletter!

      Subscribe to Daily Tech Insider for top news, trends & analysis

      Get the Free Newsletter!

      Subscribe to Daily Tech Insider for top news, trends & analysis

      MOST POPULAR ARTICLES

      Artificial Intelligence

      9 Best AI 3D Generators You Need...

      Sam Rinko - June 25, 2024 0
      AI 3D Generators are powerful tools for many different industries. Discover the best AI 3D Generators, and learn which is best for your specific use case.
      Read more
      Cloud

      RingCentral Expands Its Collaboration Platform

      Zeus Kerravala - November 22, 2023 0
      RingCentral adds AI-enabled contact center and hybrid event products to its suite of collaboration services.
      Read more
      Artificial Intelligence

      8 Best AI Data Analytics Software &...

      Aminu Abdullahi - January 18, 2024 0
      Learn the top AI data analytics software to use. Compare AI data analytics solutions & features to make the best choice for your business.
      Read more
      Latest News

      Zeus Kerravala on Networking: Multicloud, 5G, and...

      James Maguire - December 16, 2022 0
      I spoke with Zeus Kerravala, industry analyst at ZK Research, about the rapid changes in enterprise networking, as tech advances and digital transformation prompt...
      Read more
      Video

      Datadog President Amit Agarwal on Trends in...

      James Maguire - November 11, 2022 0
      I spoke with Amit Agarwal, President of Datadog, about infrastructure observability, from current trends to key challenges to the future of this rapidly growing...
      Read more
      Logo

      eWeek has the latest technology news and analysis, buying guides, and product reviews for IT professionals and technology buyers. The site’s focus is on innovative solutions and covering in-depth technical content. eWeek stays on the cutting edge of technology news and IT trends through interviews and expert analysis. Gain insight from top innovators and thought leaders in the fields of IT, business, enterprise software, startups, and more.

      Facebook
      Linkedin
      RSS
      Twitter
      Youtube

      Advertisers

      Advertise with TechnologyAdvice on eWeek and our other IT-focused platforms.

      Advertise with Us

      Menu

      • About eWeek
      • Subscribe to our Newsletter
      • Latest News

      Our Brands

      • Privacy Policy
      • Terms
      • About
      • Contact
      • Advertise
      • Sitemap
      • California – Do Not Sell My Information

      Property of TechnologyAdvice.
      © 2024 TechnologyAdvice. All Rights Reserved

      Advertiser Disclosure: Some of the products that appear on this site are from companies from which TechnologyAdvice receives compensation. This compensation may impact how and where products appear on this site including, for example, the order in which they appear. TechnologyAdvice does not include all companies or all types of products available in the marketplace.