Vista Validation Woes Caused by Preproduction Code

By Peter Galli  |  Posted 2007-08-29

Vista Validation Woes Caused by Preproduction Code


Vista Validation Woes Caused

by Preproduction Code"> The system shutdown that prevented Microsoft from validating some 12,000 Vista systems under its Windows Genuine Advantage program was the result of preproduction code being sent to production servers, the company said.

While activations, which were also affected by the preproduction code, were fixed in less than 30 minutes by rolling back the changes, the rollback did not solve the problem with the validation service, which was down for some 20 hours.

Microsoft, based in Redmond, Wash., confirmed Aug, 27 that the problems with processing validations started at about 3:30 p.m. PDT on Friday, Aug. 24, and continued until around 11:15 a.m. Pacific Time on Saturday.

Click here to read more about how Microsoft fixed the validation issue that withheld Vista features.

"Nothing more than human error started it all. Preproduction code was sent to production servers. The production servers had not yet been upgraded with a recent change to enable stronger encryption/decryption of product keys during the activation and validation processes," Alex Kochis, senior product manager for Microsofts Windows Genuine Advantage team, said in a blog post on Aug. 28.

"The result of this is that the production servers declined activation and validation requests that should have passed," he said.

While Microsoft expected the rollback to fix both the activation and validation issues at the same time, "We now realize that we didnt have the right monitoring in place to be sure the fixes had the intended effect," he said.

With regard to why its validation system does not assume that systems are genuine when its servers are down, Kochis said that is the default. "Its important to clarify that this event was not an outage. Our system is designed to default to genuine if the service is disrupted or unavailable," he said. Microsoft designed Windows Genuine Advantage to give customers the benefit of the doubt, he said. "But this event was not the same as an outage because in this case the trusted source of validations itself responded incorrectly," Kochis said.

Microsoft is improving its monitoring capabilities so that it will be alerted more quickly should something like this happen again. Even though the validation system stopped working at about 3:30 p.m., Microsoft only discovered the issue that evening as a result of calls to its customer service lines and posts to its forum.

Click here to read about the Vista update Microsoft released to stop its product activation technology from being bypassed.

The software maker is also making other changes, such as increasing the speed of escalations and adding checkpoints before changes can be made to production servers.

Page 2: Vista Validation Woes Caused by Preproduction Code

Confused Customers

Adding to Microsofts woes and those of its customers was the fact that customer support agents incorrectly told people whose Vista systems were not being validated that the servers might be down until Tuesday, Aug. 28, when, in fact, they had already been fixed as of late morning Pacific Time on Aug. 25.

"Were reviewing our procedures on that score as well—communicating clearly and accurately are super-important when things like this happen," Kochis said.

Without validated systems, customers are not able to access a number of Vistas features. These include the new Aero user interface; ReadyBoost, which expands virtual memory; Windows Defender, which still scans and identifies all threats but cleans only the severe ones; and optional updates via Windows Update, which still makes security and other critical updates available.

Whats the matter with Vista? Click here to see 12 reasons why its struggling, and five ways to get it back on its feet.

Adding insult to injury for customers unable to validate their systems was the message that appeared in the lower right-hand corner of the desktop area, saying "This copy of Windows is not genuine." The message remains until a successful validation is completed.

One customer angrily posted to the Windows Genuine Advantage blog, "I paid well in excess of $200 for my copy of Vista Ultimate and getting told that Your copy of Vista appears to be counterfeit and having functionality removed and not being able to install updates from the Microsoft site doesnt make for a very good experience.

"I was and still am quite mad at Microsoft. Im pretty sure Im going to switch to a Mac because of it. I certainly dont feel like Ive been rewarded for being a good citizen and actually buying my copy of Windows. All those pirates out there probably had no problems at all," the post said.

To read about a lawsuit that labeled Windows Genuine Advantage as spyware, click here.

For his part, Kochis once again expressed his regret about the situation. "I also want everyone to know that I am personally very disappointed that this event occurred. As an organization weve come a long way since this program began and its difficult knowing that this event confused, inconvenienced and upset our customers," he said.

But one customer lauded Kochis for being open and upfront about the problem in an MSDN blog.

"I know that takes a lot of guts to do, even though once its done, it seemed so simple," the customer wrote. "I highly encourage more transparency in both the WGA and Activation efforts at Microsoft. Im pretty sure it will do you guys a lot of good in the long run. There are still many people with grave concerns as Im sure you know!"

Check out eWEEK.coms for Microsoft and Windows news, views and analysis.

Rocket Fuel