Google is busy bringing Gmail users affected by an outage back online. A storage software bug update caused the outage, which affected only .02 percent of users.
Google said a storage software update introduced the bug that made e-mail
messages and other data disappear from .02 percent of its users Feb. 27.
Assuming Gmail has 150 million or so users, some 30,000 people did not
receive new e-mail messages sent between 6 p.m. PST Feb. 27 to 2 p.m. PST Feb. 28.
However, the company said e-mail was never lost in the outage
, which affected messages, contacts, labels and other
Gmail has been restored for some users, and Google expects a complete
resolution soon. Gmail users may follow updates to the situation in the Google Apps
"I know what some of you are thinking: how could this happen if we
have multiple copies of your data, in multiple data centers?" wrote
Ben Treynor, Google vice president of engineering and site reliability czar.
"Well, in some rare instances software bugs can affect several copies
of the data. That's what happened here. Some copies of mail were deleted, and
we've been hard at work over the last 30 hours getting it back for the people
affected by this issue."
The bugs did not destroy e-mail messages and associated content because
Google backs up the data to tapes, which are offline.
However, in a situation that underscores Google's overall adherence to cloud
computing principles, because the tapes are offline, restoring data from them
takes longer than transferring user requests to another data center.
That's why, Treynor said, it's taken Google "hours to restore e-mail
instead of milliseconds."
Google experienced serious Gmail outages in 2008
, but Gmail delivered
uptime of 99.984 percent for business and consumer
users in 2010. That translates to 7 minutes of downtime per month over the last
year, or small delays of a few seconds.
That success rate led Google to remove the clause that allows for scheduled
downtime for Google Apps and pledge to give customers credit for any downtime.