Rackspace, like cloud competitor Amazon Web Services, was forced to reboot some of its servers after patching them to fix a security flaw in some versions of the XenServer hypervisor.
The cloud provider had to patch an untold number of servers in its global data centers over the weekend and then reboot them, which caused disruption to about a quarter of Rackspace’s more than 200,000 customers, according to President and CEO Taylor Rhodes. The issue was further complicated by a tight deadline—the vulnerability was first discovered early last week, and a patch wasn’t worked out with Xen engineers until late Sept. 26.
At the same time, technical details of the vulnerability were scheduled to be released Oct. 1, giving Rackspace, Amazon Web Services (ASW) and any other cloud provider facing the same issues only a few days to patch the systems and reboot them. They also had to alert customers to the situation, but not give out too much information that hackers could use to compromise data on the servers before they were patched.
“We were faced with the difficult decision of whether to start our reboots over the weekend, with short notice to our customers, or postpone it until Monday,” Rhodes said in a message sent to customers Sept. 30 and posted on the company’s Website Oct. 1. “The latter course would not allow us to sufficiently stagger the reboots. It would jeopardize our ability to fully patch all the affected servers before the vulnerability became public, thus exposing our customers to heightened risk. We decided the lesser evil was to proceed immediately.”
Holding back information about a particular flaw before it’s patched is standard procedure, he wrote, “lest we, in effect, ring a dinner bell for the world’s cyber-criminals.”
He took a jab at AWS officials, noting that he and Rackspace engineers, while notifying customers and partners on the urgent need to reboot some servers, did not mention that the problem was with the Xen hypervisor for fear of risking alerting the cyber-criminal community of the flaw.
“Another major cloud provider did attribute its reboot to security problems with Xen, which put all users of the affected versions of that hypervisor at heightened risk,” Rhodes wrote. “But we’re relieved to report that, as of now, we’ve learned of no data compromise among Rackspace customers. Now that the vulnerability has been fully remediated, the Xen community has lifted its embargo on talking about it.”
AWS started sending out letters to its customers Sept. 24 informing them that there was an issue, but assured them that the problem was not related to the Bash bug that arose last week as a threat to systems running Unix and Linux. Officials instead let them know that the problem was with the Xen hypervisor, and that a patch was being worked on.
AWS officials said the patching and rebooting impacted less than 10 percent of the company’s systems.
Rackspace engineers had to patch and reboot the cloud company’s Standard, Performance 1 and Performance 2 cloud servers, and its Hadoop Cloud Big Data service.
According to Rhodes, the Xen vulnerability would have enabled hackers who “followed a certain series of memory commands to read snippets of data belonging to other customers, or to crash the host server.” He admitted that everything did not go smoothly over the weekend—some reboots took longer than they should have, while some notifications could have been clearer. However, the end result was that no customer data was compromised, Rhodes said.