Quality, not time, has become the critical unit of measure in software development. Microsoft Corp. became a poster child for that doctrine when it announced Nov. 29 that it would start releasing Vista previews based on meeting quality milestones rather than hewing to monthly dates.
The quest-for-quality philosophy had been articulated one week earlier, however, during a roundtable discussion of application life-cycle quality goals and practices convened by eWEEK Labs. The group comprised representatives from quality tool vendors, platform provider and toolmaker Microsoft, and customers of those vendors.
Massive releases with dozens of features invite teams to settle for thresholds such as “90 percent complete” as defining software readiness. As teams adopt shorter cycles, however, it gets harder to bury a defect under the sheer bulk of a new-feature list.
“I know that my iterations done when it meets the quality bar—its becoming the key project management metric,” Ian McLeod, senior vice president for products at Segue Software Inc. in Lexington, Mass., said during the eWEEK conference call.
Other roundtable participants agreed with McLeod that quality tops their lists of development goals—and emphasized that the scope of the “Q” word includes not only accurate implementation of specifications but also the much broader need for satisfaction of user expectations in the field.
The scope and nature of the QA (quality assurance) task are further broadening as organizations rely on purchased software and network-based services as components of line-of-business applications.
“Were using more and more a component architecture, which means more complexity as applications become more distributed, more loosely coupled and more prone to changes by each of those components providers,” said Eldad Maniv, vice president for product management at Identify Software Ltd., in New York.
Enterprise buyers recognize that software quality is not a mere question of containing the chaos, but rather represents an important opportunity for competitive advantage with strong returns on investment.
Moreover, companies are not waiting for vendors to come to them with turnkey quality solutions; they are writing their own agendas for QA technology and inviting vendors to contribute on the buyers terms—or to be made irrelevant by the growing alternative of free and open-source tools.
“Were working on a concept that we call the fully connected life-cycle tool bench,” said Chris Meystrik, director of software engineering at television and Web retailer Jewelry Television (formerly Americas Collectibles Network Inc. before its relaunch last year), in Knoxville, Tenn.
“Where were focused is on the integration piece,” Meystrik said. “An engineer uses a set of tools—we dont want them piddling around in all kinds of other tools across the software life cycle because thats really defocusing and makes them inefficient.”
A vendor need not bother pitching Meystrik any tool that does not offer APIs to its functions. In fact, Meystrik called open-tool APIs “absolutely critical,” adding, “We dont buy a tool unless [the vendor is] willing to do that, or well go find an open-source tool that might not quite have the bells and whistles. Its absolutely the No. 1 thing that we look at.”
For example, Meystrik said, “Were using Microsoft Project, but were not typing and retyping things to get them in there; were force-feeding our requirements into [Project] and automatically getting our skeleton put together. Were automatically generating change-request tickets out of every piece of software, and those are tied to source code—thats really where were going.”
The result, Meystrik said, is an environment in which relevant information can be brought to his engineers where they can readily apply it. “[The engineers are] in Eclipse,” he said. “They need a perspective of what requirements look like; they might want to see what kind of test plans have failed—but they want that brought to them in the tool that they use.”
Segues McLeod said he hopes to see a more comprehensive, industrywide embrace of tool integration.
“Quality is all the elements of the ecosystem—requirements, development and test management, defect management, monitoring, and diagnostics across the deployment line and into operations. You mostly see point-to-point integrations, with the familiar problems of configuration control of the interfaces, alignment of data format, agreement on common repositories,” he said. “Some initiatives are out there, such as the application life-cycle framework project, which is a relatively new project within Eclipse thats attempting to standardize that integration.”
Jewelry Televisions Meystrik said he is confident in the returns that his efforts are yielding. “The addressing of these software quality issues is a significant strategic investment, from our companys perspective, and its all just business-driven and ROI-driven,” he said. “Weve proven over the last year how implementation of these ideas and concepts has driven straight to the bottom line, and the company sees that at the highest levels.”
The rapid pace of Meystriks TV and online retail environment is matched by few other business segments, but one of those few is Las Vegas gaming. Station Casinos Inc. is one of that segments major players, operating more than a dozen gaming and hospitality sites.
Companies Seek to Shorten
The gaming business demands continual creation of new customer experiences, with relentless attention to maximum uptime at minimum cost: “A majority of what we do to drive revenue has technology behind it,” said Station Casinos Vice President and CIO Marshall Andrew, in Las Vegas.
“Were trying to shorten the life cycle, get through the QA process, get the products out where they can help us generate revenue,” Andrew said. “Were looking for products and services and skill sets that let us meet market demands in a more timely manner.”
Andrew is particularly aware of the current high ratio of QA time to development time. “If it takes, for example, six months of programming time, it usually takes three months of QA time, with 50 percent of the programming time for QA. Were trying to shorten that QA time without sacrificing quality by using tools that shorten the QA—and during the QA process, pinpoint areas of deficiency that we can feed back to the programmer so that they dont take days or hours to find the problem,” he said.
Andrew described one recent example in which a customer-facing system had been “crashing every day” with substantial revenue impact. By applying Identify Softwares AppSight Black Box technology and also working with Microsoft technical personnel, he said, “we drilled down into some of the scenarios that would have taken us weeks and weeks to find without these tools. I was a little skeptical, but Im a firm believer now.”
Microsoft, of Redmond, Wash., was represented in the discussion by Sam Guckenheimer, group product planner for Microsoft Visual Studio Team System and Microsoft Solutions Framework.
Guckenheimer emphasized the need for an end-to-end process view in addressing application quality goals. It begins, he said, with design: “Weve asked what an architect would do to see, at design time, if an application is designed correctly and properly takes into account things like security violations.”
Development managers need to recognize, Guckenheimer said, that most developers are not and will not be security experts and that automatic analysis of potential security issues therefore needs to be incorporated into tools and processes in a pervasive way.
“We try to inspect the code automatically for security, and weve made our internal tools available in Visual Studio Team System to highlight practices in code that might lead to problems downstream,” he said.
Guckenheimer and others were quick to add that the quality process needs to be a closed feedback loop, no matter how much investment is made in good tools and practices before application deployment. Some issues will arise only in the deployment environment, but Identify Software Vice President Lori Wizdo expressed concern that customers may wrongly define quality in terms of whats produced by developers rather than whats experienced by users.
“The dynamic problems that youll encounter in production are going to be impacted by environmental factors, by system resources, by configuration issues, by any number of stresses that couldnt be tested for,” Wizdo said. “The current thinking is that you can [address application quality during testing], but its a life-cycle issue.”
Microsofts Guckenheimer said the company hopes to address that in two ways. “We generate unit tests against code and track code coverage so that the whole team can see what has and hasnt been exercised,” he said. “We take the testing that is done—load testing, running 1,000 simulated users against the application—and make that approachable and affordable to the developer to do that early on.”
Guckenheimer said he also hopes that developers will apply virtualization technology to narrow the gap between lab and production environments: “We let you have test labs in virtual machines so you can have, somewhere on a server hard disk, a file that you open up thats a virtual machine image that matches production and gets you out of the not on my machine problem.”
Significant downstream delays would thereby be reduced, Guckenheimer said. “At design time, you have a sort of validation early on as to whether the way youre building a distributed system will run. It takes a lot of time out of the staging process of What ports do I need to open? Do we need to change the authentication on this server?—these issues that arent really coding but where every blind man has a different leg of the elephant,” he said.
Broader availability of better-integrated tools requires a re-examination of developer and tester roles, said André Pino, chief marketing officer at Segue Software.
“A lot of the quality issues that still remain are the result of quality activities being done in isolation. Youve got developers doing unit testing, youve got testers doing UI [user interface] testing and functional testing, performance people doing performance testing. The problem is, you cant compartmentalize quality assurance because its impacted by too many people and too many activities.”
Pinos Segue colleague McLeod agreed, saying, “As developers do more testing, testers perhaps need to be more focused on users instead of merely validating requirements.”
Managers Should Remember User
As developer productivity gains flow from quality improvement investments, enterprise managers should therefore apply some of those savings toward the goal of better understanding and meeting user needs.
Quality improvement tools should be viewed to some degree as enablers for that larger effort and not as representing the end of the quality journey: “I havent seen any tool that can tell me I have a problem with my requirement. I havent seen a tool that will tell me how I need to change my requirement,” said Dalim Khandaker, manager for enterprise application performance and tuning at the Toronto offices of the professional services company CGI Group Inc.
Microsofts Guckenheimer agreed that key aspects of quality require user input and developer commitment to understand and improve: “Typically, requirements dont cover things like performance and user experience that well—[theyre] not something for which a requirements document is necessarily the right solution.”
The economics of application quality are compelling, the discussion participants agreed.
“There was a very interesting National Institute of Standards and Technology study a couple of years ago that looked at the overall economic impact of software quality,” said Identify Softwares Wizdo. “That report said that fully 80 percent of software development costs are spent correcting software defects. Solving that problem is fundamental to the development process.”
Boosting developer productivity, rather than merely burning developer hours, is a crucial corollary for Station Casinos Andrew. “Were finding it very hard to find skilled technicians, even going outside the state, so were trying to get more efficiency with better tools and better training,” he said. Moreover, he said, investments get amortized over time, “while salaries hit your income statement on a monthly basis.”
To speak of process technology in the same breath as enterprise finance is to raise another key subject: Sarbanes-Oxley Act compliance. The need to make quality more consistent and the development process more transparent and accountable “is just an artifact of us not doing the right things in the past,” Jewelry Televisions Meystrik said. “So now Moms standing up and saying, You guys will behave now. We just need to do it ourselves.”
“The multivendor, multiversion platform is creating a lot of challenges,” CGIs Khandaker said.
Meystrik is meeting that challenge with an eclectic strategy that keeps him in charge of his own portfolio. “We dont want one vendor to sell us one solution because, by and large, my history of managing large organizations tells me that doesnt work. We want best of breed,” he said. “Weve made some significant changes in coupling things together.”
Those are the terms on which quality-tool vendors must compete—or be ignored.
Peter Coffee can be reached at [email protected]
Check out eWEEK.coms for the latest news, reviews and analysis in programming environments and developer tools.