Google Apps Outage Caused by High Load to Google Contacts

The downtime for Gmail and Google Contacts last week was caused by high load to Google Contacts. Google Apps users could not access or experienced disruptions in their Gmail accounts from 10 a.m. to 11:30 a.m. EDT. Users received a 500-series timeout error or found that Gmail pages loaded slowly. Users also could not access Contacts through Gmail from 10 a.m. to 1 p.m. EDT, which meant that users also couldn't access Google Talk or add contacts to Google Apps. To fix the Gmail degradation, Google's engineering team temporarily stopped all requests to use the Contacts feature from the Gmail interface.

The issue that degraded or knocked out service to Google's Gmail application Sept. 24 was high load on the company's Google Contacts service, according to an incident report from the Google Apps team.

Google Apps users could not access or experienced disruptions in their Gmail accounts from 10 a.m. to 11:30 a.m. EDT. Users received a 500-series timeout error or found that Gmail pages loaded slowly. Users also could not access Contacts through Gmail from 10 a.m. to 1 p.m. EDT, which meant that users also couldn't access Google Talk or add contacts to Google Apps.

To fix the Gmail degradation, Google's engineering team temporarily stopped all requests to use the Contacts feature from the Gmail interface and posted an alert banner in Gmail to warn customers that Contacts could not be displayed though the links in Gmail. The Apps Status Dashboard informed users that they could view contacts directly at www.google.com/contacts/yourdomain.com.

The high-load problem, according to a Google Apps incident report issued Sept. 25, was caused by a confluence of human and machine actions. This included a network issue in a data center, which caused additional load on the Contacts service; a very high utilization of the Contacts service; and an update to Gmail that inadvertently increased the load on the Contacts service.

To resolve the issues, the Google's engineering team deployed more capacity to the Contacts service through Google's flexible capacity server systems. The use of Google Contacts through Gmail was restored by 12:30 p.m. EDT, with Contacts, Google Talk and "add user" features returned to normal by 1 p.m. EDT. Read the full incident report here. 

Google assured Google Apps users that no data was lost, which is a big deal, particularly for customers paying $50 per user, per year for the service. However, there isn't much comfort in the fact that Google Apps has now experienced two significant outages in one month.

On Sept. 1, Google's Gmail application was knocked out for the majority of users for 100 minutes when Google's engineering team took a small fraction of Gmail's servers offline to perform routine upgrades.

Google issued an apology and an explanation on the Gmail blog for that outage, in addition to the incident report. There was no such olive branch for this Contacts snafu.

Gmail also went down in February 2009. Before that, August 2008 was the last significant downtime for Google Apps.

That was more than a year ago, and Google has added a lot more users to its suite of word processing, presentation, spreadsheet and chat apps. More than 1.75 million businesses now use Google Apps.

Also, Google's Gmail app logged 37 million unique monthly users in July, making it the No. 3 Webmail app behind Yahoo Mail and Microsoft Windows Live Hotmail.

Nucleus Research analyst Rebecca Wettemann said that while she hasn't heard specific complaints from Google Apps customers, the outages pose a challenge for Google in attracting business users. She added:

"With e-mail as the primary channel of business communication, any downtime can significantly impact both individual employee productivity and overall corporate performance. Companies need to evaluate very carefully the potential risks of any e-mail system or service provider before making a move-and any significant outages challenge Google's ability to expand in the enterprise."

Is 2009 just a bad year for Google Apps, or is it a signal of the growing pains the company is facing as it adds more users for its collaboration software?

It is unclear. What is clear is that consumers and paying customers alike are growing tired of the downtime. Reasons, excuses and account credits will only take Google so far.