Patch Work Gets Harder

Management requires careful risk assessment and thorough testing.

Data center system patch management is growing in complexity and requires a concerted effort by IT administrators to control costs.

The pressing question of the day is how to maintain secure mission-critical systems while controlling operational costs in the face of a steady onslaught of software vulnerabilities. According to The CERT Coordination Center, more than 4,000 vulnerabilities were reported last year. In just the first quarter of this year, more than 900 vulnerabilities have been reported, according to CERT.

Putting it a different way, IT administrators face the task of evaluating about 10 new potential problems every day.

The vast majority of these reported weaknesses wont apply to many organizations—either because the organizations are not using the specific feature thats shown a weakness or because the perfect storm required to expose the weakness hasnt hit (yet).

eWEEK Labs evaluates four products ability to ease the tedious and complex process of sorting patches to determine which must be applied, and then installing the patches.

What the products had in common was a daunting requirement for IT departments. Before a patch can be implemented effectively—that is, the effects of the patch are less negative than the effects of the potential hack the patch guards against—it must be thoroughly tested. This means that IT departments must have reference systems configured with the same hardware and software as the production systems that are the intended patch targets.

We agree with the patch management vendors we worked with that the best practice is to test patches on a nonproduction machine to gauge the patchs potential impact on business systems. But even in our own lab, we have trouble maintaining systems as exact mirror images of one another—one or two tweaks sometimes get made in a hurry and arent documented in our configuration log.

Our advice for IT shops that dont have a formal test lab is to put together a small group of machines that function as guinea pigs for proposed patches.

Regardless of whether the enterprise has a full test lab or even a small group of guinea pig machines, it is just a practical matter of survival for IT departments to come up with a patch proving ground. The pace at which new exposures are reported means that the patch testing process needs to combine mirror-image systems with procedures for rapidly evaluating patches for potential problems.

While some of the patch management systems we evaluated can help make these determinations, none, as yet, can replace a case-by-case evaluation by an IT expert familiar with the organizations systems and business practices.

All the systems we tested can help, however, and with time and practice IT managers will likely benefit from the addition of a patch management tool to their maintenance arsenal.

Our testing also revealed several best practices that IT managers can implement to gain control over the patch management process.

During tests, we used Altiris Inc.s Deployment Manager 5.6 to image and restore most of our systems. Testing was much more accurate and efficient because we were able to restore systems quickly.

Of course, a system must start in good working order. During tests, we tried patching a Windows 2000 Server-based system that was running a hurriedly installed version of Microsoft Corp.s Exchange 2000 mail server. After applying a patch to the mail server, it "stopped" working.

After we restored the lab machines to their original state, we found that the Exchange server still wasnt working. It turns out that we had a slight problem with Domain Name System in our test network that interfered with Exchange. So, while we were primed to fault the patch, we had to lay blame where it was due—in this case, on our installation method.

We also strongly recommend a policy of making only one change at a time. Anyone in front-line support will tell you that applying more than one fix at a time is a sure recipe for taking down a system.