Vistas Moment of Truth

Testers say the next build of Microsoft's latest OS better be good.

Its the hour of reckoning for Windows Vista. After five years of course changes, false starts, and a host of beta and CTP (Community Technology Preview) builds, Microsoft is set to deliver a broad-scale build of Vista to 2 million testers. Microsoft may drop the build—known by multiple names, including the consumer Vista CTP and Vista Beta 2—as early as May 23 at WinHEC in Seattle, according to company watchers. At about the same time the Vista build lands, Microsoft also is expected to release Office 2007 Beta 2, the next major milestone of Microsofts next-generation desktop suite.

While the exact release date is fluid, the facts are not—this cut of Vista needs to be more solid than a years worth of previous builds from Microsoft. Even the most recent Vista builds—including two interim CTPs delivered during the past couple of months to the companys squad of elite beta testers, known as TAP (Technology Adoption Program) partners—have suffered from a variety of performance and compatibility problems, according to interviews with eight testers.

"Theres too much variation in performance from one build to another," said Brandon LeBlanc, a Vista tester, in Portland, Ore., and a contributor to a number of Windows community sites, including "The changes they are continuing to make at this stage disrupt performance too much. Youd imagine they would have gotten past this stage by Beta 2."

Testers said that if the next build of Vista doesnt improve dramatically, Microsoft will have a tough time sticking to the outline the company issued in late March. Microsofts current timeline calls for the company to release the final Vista code to manufacturing this summer or fall, allowing customers under volume-licensing agreements to get their hands on the code by November. Microsoft executives have maintained that the dual launch of Vista and Office 2007 is set for January, when code for both products will be available to all customers.

Indeed, doubters of Microsofts ability to hit its timeline arent hard to find. In a research note earlier in May, Gartner said Vistas availability could be pushed back by as much as a quarter.

"Here is what any team has to get absolutely right in order to call a build Beta 2: The build has to be solid enough that testers can run it to perform their day-to-day work," said Michael Cherry, an analyst with Directions on Microsoft, in Kirkland, Wash. "If the software is not stable enough to run in day-to-day use, then it will delay testing, which delays testers finding bugs."

This sounds straightforward, but Microsoft employees and corporate testers running Vista as their primary operating system report the product is not "production-ready." Even longtime Vista testers and Windows community members, who know the intricacies of Microsoft products, as well as the Redmond, Wash., software makers own developers and testers, are having problems with the latest builds.

While its a tricky comparison between 2001s Windows XP and Windows Vista—given the differences in the size and the complexity of the code bases—testers noted that Vista doesnt seem to be as solid and ship-ready as XP was at the Beta 2 juncture.

"With Whistler [Windows XP], you could basically run the OS as a [near-production-ready] OS," said Brad Wardell, president and CEO of Stardock, a software vendor in Plymouth, Mich. "With Windows Vista, the networking issues, performance and compatibility prevent users from making it their main OS."

Among the issues Microsofts next build will need to address:

• Networking Vista networking is a sore spot with a number of Microsofts hard-core testers, not just Wardell. As is true on a variety of Vista fronts, especially security and systems management, Microsoft has tuned the product for users, not administrators.

"The network panel is a nightmare if you attempt to do anything mildly complex," said Carlos Echenique, site owner and editor in chief of the PlanetX64 and PlanetAMD64 Windows community sites, in Miami. "While the panel is great for simple setups, power users will start committing seppuku if they have to do any real troubleshooting."

Wardell agreed. "The betas of Windows Vista have had atrocious issues with networking being reliable," he said. "Without the basic features of a modern OS working, people wont run it, and that will mean a lot less feedback."

• Driver and application compatibility For many testers, compatibility is at the top of their Beta 2 wish lists.

"Application compatibility, leastways for the top 500 shipping applications," needs work, said John Obeto, managing partner and chief technology officer with Logikworx, a Marina Del Rey, Calif., solution provider specializing in systems and network security. Obeto also is a Vista tester and runs the AbsoluteVista community site.

"Most of my personal issues with Vista are currently caused by driver issues," said Michael Reyes, a principal with the community site. "Between the sound stack and video drivers having to be redone, a lot of the available library of drivers are very rough around the edges and do cause lockups," said Reyes in New York.

• Memory ceilings and handles Wardell said he has two primary issues with Vista: its memory use and the way it deals with "handles," a type of computing resource that various programs such as e-mail and desktop search use.

As for the memory issue, Wardell said its becoming increasingly difficult to add memory to boost performance. "We are now bumping up against the 2GB limit," said Wardell, adding that if Vista needs more than that to operate at a high level, there will be problems. As for the 2GB reference, Wardell noted that while 32-bit processors can access 4GB of memory per process in theory, the upper 2GB are reserved.

"Windows Vista uses considerably more memory than Windows XP—about twice as much—and there is not much reason to think this amount will significantly change by release. Realistically, until 64-bit machines become the norm, the 2GB limit is going to be a problem," Wardell said.

The handle issue could also be key. Wardell estimated that Windows XP boots using about 3,000 handles, compared with 15,000 for Vista. He has found that Windows slows down when handles hit 15,000 and 25,000.

Whether or not Microsoft gets to these myriad and substantial fixes and tweaks by the time the Beta 2 build hits should be apparent soon. But Cherry said Vista may illustrate that the company has to revamp its entire development process.

"I think the problems they [Microsoft] are having relate to not getting to a feature-complete state earlier," Cherry said. "It seems like they were still accepting changes and new additions very late in the process."