The downtime for Gmail and Google Contacts last week was caused by high load to Google Contacts. Google Apps users could not access or experienced disruptions in their Gmail accounts from 10 a.m. to 11:30 a.m. EDT. Users received a 500-series timeout error or found that Gmail pages loaded slowly. Users also could not access Contacts through Gmail from 10 a.m. to 1 p.m. EDT, which meant that users also couldn't access Google Talk or add contacts to Google Apps. To fix the Gmail degradation, Google's engineering team temporarily stopped all requests to use the Contacts feature from the Gmail interface.
The issue that degraded or knocked out
service to Google's Gmail application
Sept. 24 was high load on the company's Google Contacts service, according to
an incident report from the Google Apps team.
Google Apps users could not access or experienced disruptions in their Gmail
accounts from 10 a.m. to 11:30 a.m. EDT.
Users received a 500-series timeout error or found that Gmail pages loaded
slowly. Users also could not access Contacts through Gmail from 10 a.m. to 1
p.m. EDT, which meant that users
also couldn't access Google Talk or add contacts to Google Apps.
To fix the Gmail degradation, Google's engineering team temporarily stopped
all requests to use the Contacts feature from the Gmail interface and posted an
alert banner in Gmail to warn customers that Contacts could not be displayed
though the links in Gmail. The Apps Status Dashboard
informed users that they could view
contacts directly at www.google.com/contacts/yourdomain.com.
The high-load problem, according to a Google Apps incident report issued
Sept. 25, was caused by a confluence of human and machine actions. This
included a network issue in a data center, which caused additional load on the
Contacts service; a very high utilization of the Contacts service; and an
update to Gmail that inadvertently increased the load on the Contacts service.
To resolve the issues, the Google's engineering team deployed more capacity
to the Contacts service through Google's flexible capacity server systems. The
use of Google Contacts through Gmail was restored by 12:30 p.m. EDT, with
Contacts, Google Talk and "add user" features returned to normal by 1 p.m. EDT.
Read the full incident report here
Google assured Google Apps users that no data was lost, which is a big deal,
particularly for customers paying $50 per user, per year for the service.
However, there isn't much comfort in the fact that Google Apps has now
experienced two significant outages in one month.
On Sept. 1, Google's Gmail application was knocked out
for the majority of users for 100 minutes
when Google's engineering team took a small fraction of Gmail's servers offline
to perform routine upgrades.
Google issued an apology and an explanation
on the Gmail blog for that outage, in addition to
the incident report. There was no such olive branch for this Contacts snafu.
Gmail also went down
in February 2009. Before that, August 2008
was the last significant downtime for Google Apps.
That was more than a year ago, and Google has added a lot more users to its
suite of word processing, presentation, spreadsheet and chat apps. More than
1.75 million businesses now use Google Apps.
Also, Google's Gmail app logged
37 million unique monthly users in July, making it the
No. 3 Webmail app behind Yahoo Mail and Microsoft Windows Live Hotmail.
Nucleus Research analyst Rebecca Wettemann said that while she hasn't heard
specific complaints from Google Apps customers, the outages pose a challenge
for Google in attracting business users. She added:
"With e-mail as the primary channel of business communication, any
downtime can significantly impact both individual employee productivity and
overall corporate performance. Companies need to evaluate very carefully
the potential risks of any e-mail system or service provider before making a
move-and any significant outages challenge Google's ability to expand in
Is 2009 just a bad year for Google Apps, or is it a signal of the growing
pains the company is facing as it adds more users for its collaboration
It is unclear. What is clear is that consumers and paying customers alike
are growing tired of the downtime. Reasons, excuses and account credits will
only take Google so far.