Updated: VMware customers and partners are experiencing disruptions after trying to download an update for the company’s ESX and ESXi 3.5 products. The update is causing a failure when trying to power on a virtual machine and additional problems with VMware’s Vmotion technology.
The problems, which were first reported Aug. 12, appear to have been limited to the Update 2 patch that VMware sent out for both its ESX and ESXi hypervisor products. While the faulty patch has caused problems with virtual machines and Vmotion-the technology that allows for live migration of virtual machines-no other products seemed to have been affected.
The cause behind the disruptions seems to be related to a licensing expiration time issue. In an e-mail sent late Aug. 12 to VMware customers, VMware engineers wrote that the problem is “caused by a build timeout that was mistakenly left enabled for the release build.”
While it’s not clear how many VMware customers were affected by the faulty updates, several Web sites claim that the disruptions were widespread, but VMware was still trying to address the issue. VMware did pull the update from the Web site’s download page to ensure that no other customers or partners downloaded the faulty patch.
VMware has posted several technical articles to help customers deal with the patch problem. Those articles can be found here. (VMware did note in its e-mail that a large amount of traffic to its Web site might cause problems downloading the article.)
On Aug. 13, VMware sent a second update that engineers have built an “express patch” to help ease some of the problems.
“There are two express patches: one for ESX 3.5 Update 2 and one for ESXi 3.5 Update 2,” according to the second VMware update. “They are specifically targeted for customers who have installed or fully upgraded to ESX/ESXi 3.5 Update 2 or who have applied the ESX350-200806201-UG patch to ESX/ESXi 3.5 or ESX/ESX 3.5 Update 1 hosts. For customers who haven’t done either, these express patches should not be applied.”
VMware also released a short statement on Aug. 13, which said that the problems with Update 2 should be fixed within 24 hours.
“We have notified customers and partners of this problem,” according to the VMware statement. “We have made this a high priority issue and have been working around the clock to resolve the problem. We have released an express patch for those customers that have installed/upgraded to ESX or ESXi 3.5 Update 2. Within the next 24 hours, we will also issue a full replacement for Update 2, which should be used by customers who want to perform fresh installs of ESX or ESXi.”
On the VMware communities blog, several of the company’s customers have posted temporary fixes that seem to solve the problems in the short term. One poster wrote about manually setting the dates for the ESX host back to Aug. 10, which seems to work. The licensing problem seems to happen when the date moves to Aug. 12.
Here is a list of instructions from the VMware communities blog:
- Turn off NTP [Network Time Protocol] (if you’re using it), and then manually set the date of all ESX 3.5u2 hosts back to 10th of August. This can be done either through the VI Client (Host -> Configuration -> Time Configuration) or by typing date -s “08/10/2008” at the Service Console command line on the ESX hosts.
- As soon as the date was reset to the 10th – problem solved.
- Note that running VMs were operating fine, this only seems to affect initial VM power-on (including from suspended state) and VMotion.
However, VMware does warn that this temporary fix does have some drawbacks that could affect database servers, mail servers and domain administration systems.
New VMware President and CEO Paul Maritz wrote an explanation of the ongoing problems and issued on apology on his blog. Maritz replaced former CEO Diane Greene last month. One of Maritz’s first big announcement after taking day-to-day control of VMware was that the company would start shipping the ESXi hypervisor for free.
Editor’s Note: This story was updated to include a statement from VMware.