VMware is re-issuing the updates for its ESX and ESXi Server products, which fixes the software time element flaws that led to the virtualization company’s users not being able to power on virtual machines and shutting down its Vmotion technology.
VMware customers and partners first began finding problems with the Update 2 patch for the ESX virtualization infrastructure products on Aug. 12. The next day, VMware issued a temporary “express patch” to help those customers that had already downloaded and installed the faulty update patch. On Aug. 14, VMware released a statement that it has re-issued the patch for ESX and ESXi and that update can be found here.
However, VMware did warn those customers who had downloaded the faulty patch, should continue to use the express patch that engineers released Aug. 13.
“We have notified our customers that we have re-issued the entire ESX/ESXi 3.5 Update 2 release (ISOs, upgrade tar and zip files, and patch bundles),” according to the Aug. 14 VMware statement. “Please note this update is only relevant to customers who did not install the impacted release of ESX 3.5 Update 2 (build number 103908) or ESXi 3.5 Update 2 (build number 103909).”
When customers first began notifying VMware of the problem with Update 2, the company wrote on its Web site that problem is “caused by a build timeout that was mistakenly left enabled for the release build.” This “timebomb” within the software meant that the ESX and ESXi products would not run after a certain date and time. Once the time and date hit Aug. 12 at 12:00 a.m., the virtual machines running on the ESX hypervisor shut down.
This also meant that Vmotion – VMware’s technology that allows the user to move virtual machines from one physical piece of hardware to another – also did not work.
Some VMware customers alleviated the problem by manually turning the time on the ESX host to either Aug. 10 or 11. VMware did warn that these temporary fixes could disrupt database and email servers along with domain administration systems.
While it’s not clear how many people downloaded the original patch, some Web sites claimed the issues were widespread and the problem appears to have first affected businesses in Asia first do to the time zone change.
The problem was enough that new CEO Paul Maritz issued an apology and explained that the code that caused the problems had been mistakenly left in the final release of Update 2 for ESX and ESXi 3.5
“We are doing everything in our power to make sure this doesn’t happen again,” Maritz wrote. “VMware prides itself on the quality and reliability of our products, and this incident has prompted a thorough self-examination of how we create and deliver products to our customers. We have kicked off a comprehensive, in-depth review of our QA and release processes, and will quickly make the needed changes.”