Gmail Bug Squashed, Access Restored for Most Users

Google is busy bringing Gmail users affected by an outage back online. A storage software bug update caused the outage, which affected only .02 percent of users.

Google said a storage software update introduced the bug that made e-mail messages and other data disappear from .02 percent of its users Feb. 27.

Assuming Gmail has 150 million or so users, some 30,000 people did not receive new e-mail messages sent between 6 p.m. PST Feb. 27 to 2 p.m. PST Feb. 28.

However, the company said e-mail was never lost in the outage, which affected messages, contacts, labels and other data.

Gmail has been restored for some users, and Google expects a complete resolution soon. Gmail users may follow updates to the situation in the Google Apps status dashboard.  

"I know what some of you are thinking: how could this happen if we have multiple copies of your data, in multiple data centers?" wrote Ben Treynor, Google vice president of engineering and site reliability czar.

"Well, in some rare instances software bugs can affect several copies of the data. That's what happened here. Some copies of mail were deleted, and we've been hard at work over the last 30 hours getting it back for the people affected by this issue." 

The bugs did not destroy e-mail messages and associated content because Google backs up the data to tapes, which are offline.

However, in a situation that underscores Google's overall adherence to cloud computing principles, because the tapes are offline, restoring data from them takes longer than transferring user requests to another data center.

That's why, Treynor said, it's taken Google "hours to restore e-mail instead of milliseconds." 

Google experienced serious Gmail outages in 2008 and 2009, but Gmail delivered uptime of 99.984 percent for business and consumer users in 2010. That translates to 7 minutes of downtime per month over the last year, or small delays of a few seconds.

That success rate led Google to remove the clause that allows for scheduled downtime for Google Apps and pledge to give customers credit for any downtime.