Rackspace Joined Amazon in Patching, Rebooting Cloud Servers
About a quarter of Rackspace's 200,000-plus customers were impacted when the cloud provider had to patch a flaw in the Xen hypervisor.Rackspace, like cloud competitor Amazon Web Services, was forced to reboot some of its servers after patching them to fix a security flaw in some versions of the XenServer hypervisor. The cloud provider had to patch an untold number of servers in its global data centers over the weekend and then reboot them, which caused disruption to about a quarter of Rackspace's more than 200,000 customers, according to President and CEO Taylor Rhodes. The issue was further complicated by a tight deadline—the vulnerability was first discovered early last week, and a patch wasn't worked out with Xen engineers until late Sept. 26. At the same time, technical details of the vulnerability were scheduled to be released Oct. 1, giving Rackspace, Amazon Web Services (ASW) and any other cloud provider facing the same issues only a few days to patch the systems and reboot them. They also had to alert customers to the situation, but not give out too much information that hackers could use to compromise data on the servers before they were patched. "We were faced with the difficult decision of whether to start our reboots over the weekend, with short notice to our customers, or postpone it until Monday," Rhodes said in a message sent to customers Sept. 30 and posted on the company's Website Oct. 1. "The latter course would not allow us to sufficiently stagger the reboots. It would jeopardize our ability to fully patch all the affected servers before the vulnerability became public, thus exposing our customers to heightened risk. We decided the lesser evil was to proceed immediately."
Holding back information about a particular flaw before it's patched is standard procedure, he wrote, "lest we, in effect, ring a dinner bell for the world's cyber-criminals."