Software patching woes portend troubled times; risk assessment, realistic goals can ease burden.If you ask a group of enterprise it professionals how they feel about software patches, youd better be wearing flame-resistant clothing if you want to survive their replies. When eWeek Labs asked our Corporate Partner advisory board to assess the growing impact of these constant updates and software repairs on their day-to-day workload, we were unprepared for the intensity of their answers.
"Let me introduce you to the fifth horsewoman of the apocalypse," began the e-mail that we received from Corporate Partner Kevin Baradet, network systems director at the S.C. Johnson School of Management at Cornell University, in Ithaca, N.Y. As if war, famine, pestilence and death werent enough to make humanity miserable, Baradet is troubled by the constantly churning chaos in what should be stable production systems.
"The patches came up over all the land of computing," Baradet continued with biblical eloquence, "and settled in all the territory of applications and OS; they were very numerous. There had never been so many patches, nor would there be again any less."
Baradet was joined in anger, if not in oratory, by other eWeek enterprise advisers.
"Im sitting here with a broken Citrix server as we speak—broken by Microsoft patches," said Robert Rosen, CIO at the National Institute of Arthritis and Musculoskeletal and Skin Diseases, in Bethesda, Md. "Apparently, when they applied the latest security patch, it did something to Citrix. Theyve been working all week on it. Finally, I said [they should] do a bare-metal rebuild because they werent getting anywhere. Then the drive they were loading from died. Its been a fun week."
The same sense of a constant drain on IT staff time and morale was pervasive among our panel of in-the-trenches experts.
"Patching is very much like playing Russian roulette—with your entire system and corporate productivity at stake," said Steve Curcuru, resident wizard at Mugar Enterprises Inc., in Boston, and an eWeek Corporate Partner. "Ive had some patches simply apply and run silently and reliably first try, just like theyre supposed to. I just wish they were in the majority. Ive also had patches break previous patches or simply break legacy apps that have worked for years."
From the comments we received, it appears that the greatest cost of patching is in figuring out why something that used to work, before a patch that was supposed to improve things, is somehow disabled by the updates application.
"A recent IE security patch simply broke a macro that logs me in to a subscription Web site and pulls the latest news to a local tickler file," said Curcuru. "Before patch: worked just fine. After patch: Nothing happens at all—no error message and no Web site check. Debugging meant rewriting the macro and actually using a different series of commands. Why? I havent got a clue."
Curcuru added this rule of thumb: "If it aint seriously broke—dont patch it."
Is Cornells Baradet right in his grim prophecy that patches will always be with us? When eWeek Labs conducted a dialogue with the more than 80,000 developer subscribers to one of our weekly newsletters, we received vigorous input on both sides of that question.
"Software should be shipped with bugs. The zero-defect notion is mythological and theoretically unachievable," said consultant Boris Beizer, in Huntingdon Valley, Pa. "That doesnt mean shipping ill-behaved or useless software; it means being open with users about the bugs we find, sending notices or including the bug list, publishing the workarounds when we have them, and being honest and open about what we have and havent yet tested and what we do and dont plan to test in the near future. Most of all, it means treating users as adults who can, should and will make the right risk decisions if we give them the facts on which to base such decisions."
Regardless of whether one agrees with Beizer about the need to tolerate bugs, his comments do give IT buyers some guidelines on what to demand from suppliers. Any product that does not come with a list of known bugs, or "issues," if one prefers, is probably suspect. The list undoubtedly exists and should be shared with those who pay the bills.
If software source code is not included with a commercial product, the buyer is at the mercy of the vendor as to the identification and prioritization of repairs. Contracts can include provisions that mandate source code disclosure (for example, through a software escrow agent) if defined goals for quality and defect remediation are not met.
But source code access may merely encourage throwing good money after bad, if lack of timely support is a symptom of poor future prospects for either the vendor or the product: It may be better to cut ones losses and replace the product entirely, especially since locally maintained (and therefore site-specific) code will cost even more to support than when the mass-produced products problems could be shared with other users. Patches, even with delays, may suddenly look like the lesser evil.
Meanwhile, many development professionals said they feel that Beizers view is too tolerant of what they see as pervasive carelessness.
As noted by Terry Colligan, president of Tenberry Software Inc., in Fountain Hills, Ariz., what seems an almost religious dispute "may just be the difference between computer science—it is impossible to prove any program to be defect-free—and software engineering—we can economically make the defect rate so low that 99.9 percent of our customers will never experience a defect."
"Development tools and design techniques get better all the time; there is no reason to expect anything less than zero-defect code," said Software Design Engineer Bob Cuomo, at Agilent Technologies Inc., based in Loveland, Colo.
IT buyers may, therefore, need to look in the mirror to find at least one of the parties that creates the climate in which bugs and their attendant patches flourish.
"Two of the major reasons that I see defects are poor requirements definitions and lack of adequate test case preparation and testing," said Randy Cecrle, IT manager at the Nebraska Workers Compensation Court, in Lincoln. In one recent project, Cecrle reported, "I required the developer to spend considerable time in the above two areas. The payoff was that we delivered a major subsystem that had only one minor defect."
IT buyers need to be realistic with themselves as to what they want, what they need and what each is worth. Measuring the actual costs of defects is a necessary first step along that path.
Budget concerns, Cecrle warned, along with unreasonable expectations based on prototype development by PC users, create constant pressure to reduce the time and money available for these important defect- avoidance measures—a false economy, in the opinion of Andy Schneit, finance manager at Vodafone Global Platform and Internet Services (a division of U.K.-based Vodaphone Group plc.), in Walnut Creek, Calif.
Schneit said early defect detection costs less than 10 percent as much as subsequent repair. "Quality is a mind-set," he said. "So is lousy quality. And lousy quality has a hard time differentiating between major bugs and minor bugs."
Its up to IT buyers to demand that software suppliers take the time to specify precisely what changes a patch will make, to document any conflicts between the patch and any previously used workarounds, and to provide the necessary support tools for reversing a patch application.
At eWeek Labs, weve often seen what happens in the absence of such discipline. For example, in our tests, Microsofts Visual Studio .Net development tool suite would not install without making numerous "black box" updates to a Windows 2000 workstation—some of which appeared to reduce the reliability of the machine. There was no disclosure as to how these effects might be reversed, with what effects on the functions of the development tools.
Wed prefer that all such updates offer itemized statements of why theyre needed and what functions will be affected if any element is declined, but this may be an uphill battle. Microsoft Chairman and Chief Software Architect Bill Gates has asserted, for example, that "each version of Windows is designed as a single, integrated product," a view that defies the ability of buyers to treat a software platform as a collection of modules whose functions can be individually characterized in terms of their contracts with other modules.
To unseat the fifth rider of the apocalypse, IT buyers must be able to define quality with clear specifications and rigorous test cases; they must measure the costs of defects, as a baseline for comparison with other ways of building and buying code; and they must be uncompromising in their demand that software vendors return control to the buyer, both fully disclosing known defects and adequately supporting mechanisms for administering (and, if necessary, removing) software patches in the field.
Related Stories:
Tools Can Streamline OS Patching
On the Patch Patrol
Application Development 
