Sorry for the broken record treatment to all my regular readers, but I think its perhaps the most important security point in real world IT: Too many vulnerable systems are left in exposed situations, even when patches have been made available. For large networks that require active management, the answer is a patch management system like the one described in this eWEEK review.
The vendors in this category of administrative tool have made a lot of progress. And yet there is a recurring tension in patch management between the inherent sense in patching against known vulnerabilities and the desire to not disturb a system that is functioning correctly. The proper answer (here comes another lecture) is that you should test patches on a test bed of representative systems, or at least on a non-critical set of guinea pig systems on your network, as the eWEEK review suggests. These patch management systems make it easier to implement those best practices, but that doesnt answer every concern, and it still amounts to work for you to do.
So what do you do if there is a problem? Most, if not all of the systems allow you to roll the patch back. There are two problems with this approach: it can be time-consuming and complicated to determine there is a problem and to remediate it by unpatching the system; and if the problem causes the system to blue-screen, or be otherwise unstable enough that the patch management system cannot function, youll need a plan B.
Standard Plan B, of course, is to restore the pre-patch backup you certainly made. You dont need to backup the whole system. On Windows (the NT derivatives: Windows NT4, 2000, XP and 2003) you can backup "system state," which saves all the registry and critical system files likely to have been patched. If something totally murders your system you can then manually boot into the Windows Recovery Console and restore the system state backup (there is no such console on NT4 - yet another reason to upgrade from it - but you can make a recovery floppy). Its a heavyweight operation, but it should always work and it will be necessary only on rare occasions. I have a scheduled script that runs every night to backup system state and critical data files to a second hard drive in my most critical systems. Knowing I have this, and therefore knowing I can lose only at most a days work, gives me the confidence to install any patch.
I hear people complain sometimes that patches (they usually mean Microsoft patches, but other operating systems have at least as many) are more trouble than theyre worth. This is the kind of thinking that brought us the Slammer incident. Its a bad idea to apply patches blindly and without a plan, but its an even worse idea to dismiss patches for critical problems and leave your systems exposed.
As with so many other software fields, these vendors will need to keep ahead of what Microsoft itself offers. This week at TechEd in Dallas, Microsoft pledged to improve their patch distribution process, both to unify the installers for the patches and to cut the distribution channels to a minimum. For larger networks there is now Microsoft Software Update Services (SUS), which basically lets you set up your own in-house Windows Update server. Its not in the class of any of the products eWEEK reviewed, and its not supposed to be. But Microsoft needs to continue to improve the baseline of services included with Windows, and SUS is an improvement.
You enterprise types think you have it rough? Pity the poor consumer or small business users who doesnt have your experience and resources. There are no modern patch management systems for your dentist and his Windows Me system, just Windows Update, assuming he has the initiative to go there regularly. Microsofts approach over the last few years has been to make it easier for consumers to automatically apply patches; apart from allowing them to be uninstalled through the normal Add/Remove facilities, the assumption is that consumers arent going to test patches before applying them, and how could they? Its hard to know how this part of the system can improve. The enterprise can protect itself, but we still have the network equivalent of a public health problem.