A Salesforce.com database in North America, NA14, suffered a nearly day-long outage that led Chairman and CEO Marc Benioff to apologize and frustrated customers to vent on social media.
Ultimately, five hours’ worth of data updates were lost.
As of the morning of May 11, Salesforce was announcing that the disruption had been resolved, though the Salesforce Trust page showed “performance degradation” to still be an issue.
“The NA14 instance continues to operate in a degraded state,” said a company update. “Customers can access the Salesforce service, but we have temporarily suspended some functionality such as weekly exports and sandbox copy functionality. In addition, search indexing may be delayed.”
The failure was caused by a “file integrity issue” and was resolved by restoring the database from an earlier backup.
Salesforce also noted, “We have determined that data written to the NA14 instance between 9:53 UTC and 14:53 UTC on May 10, 2016 could not be restored.”
Finally, the company added, “We sincerely regret any inconvenience this disruption has caused you or your organization.”
According to The Register, the initial failure occurred at the Salesforce data center in Herndon, Va., and then was followed by a failure at a Washington, D.C., facility.
Benioff took to Twitter May 10 to offer apologies, replying to numerous complains with the same message: “I am sorry for our service disruption on NA14; please email me ceo@salesforce.com so we can call you.” He also added the Twitter handle for co-founder Parker Harris, who remained quiet.
Harris’ last Tweet was from May 5, as he prepared for a Salesforce World Tour.
Salesforce users, however, were active.
“Lost an entire day in the U.S. plus our China team also could not work,” senior director of marketing operations at Active Network, tweeted May 11.
The same morning, Mark Smith, the CEO and chief research officer at Ventana Research tweeted, “@salesforce thanks for restoring our sanity & now up after #NA14 down for day… still no email or notification on what is going on. Terrible!”
Dave DeMichele, an associate director at Salesforce, posted a photo of a T-shirt with a cloud and “I survived #NA14.”
In the early morning on May 11, Benioff was still managing the situation. He tweeted to Smith, “I just direct messaged you.”
A representative for the company didn’t offer additional information, but pointed to the Trust site as a place for ongoing updates.
There’s no official word yet from Salesforce about any remuneration or the financial impact to customers of the downtime and data loss.
Connections 2016, Salesforce’s digital marketing event, is now under way in Atlanta, through May 12. The tour will head to London May 19, then New York May 25 and Paris June 23.