eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
For Microsoft, delivering high-quality security patches in a timely manner has always been a lose-lose predicament.
If patches for major software vulnerabilities take too long, customers are at the mercy of zero-day threats. When patches are rushed out without proper quality assurance testing, they invariably become a system administrators worst nightmare.
Earlier this week, when Microsoft Corp. announced plans to re-release a “critical” bulletin because of patch quality problems, the move triggered a new round of eye-rolling among security research pros.
The bulletin, MS05-019, first released in April, contains patches that have caused major connectivity problems for network administrators.
The connectivity errors range from the inability of Exchange servers to talk to their domain controllers; failure of domain controller replication across WAN (wide area network) links; and inability to connect to terminal servers or to file share access.
Microsoft also acknowledged that networking programs that send TCP packets or UDP packets over raw IP sockets “may stop working” after the security update is applied on a computer running Windows XP SP1 (Service Pack 1).
A knowledge base article has been posted to highlight the problems, and hotfixes have been offered to provide temporary respite, but despite Microsofts insistence that the problems affect only a small number of customers, security experts said the re-release of a high-severity bulletin points to a weakness in Microsofts patch creation process.
“A hotfix for a patch? I hope it works properly, or whats next? A hotmend for the hotfix for the patch?” asked Corey Nachreiner, a network security analyst at WatchGuard Technologies Inc.
In an interview with Ziff Davis Internet News, Nachreiner said some of his companys clients have complained that the patches have broken VPN connections, a problem he described as “a big deal” for the SMB (small and medium-sized business) market segment.
Because the patch is rated critical by Microsoft, Nachreiner said he cannot recommend uninstalling the patch.
“It means that a lot of customers are scrambling to get hotfixes to keep their systems connected.”
Officials at Microsoft insisted the company is doing “far more for this one than necessary” to help ensure every customer has the most recent changes to the update.
“The problem being addressed is only present when a customer has specific MTU settings on their routers on their network, and is very rare,” a spokeswoman said.
“Due to the variation in network configurations made possible by third-party routers and TCP/IP devices, this is not a scenario that can be tested internally at Microsoft. The MSRC [Microsoft Security Response Center] is taking this additional step because of the feedback received from customers,” she added, insisting that the pool of customers affected by this issue is quite small.
However, as WatchGuards Nachreiner points out, a small pool of Microsofts massive user base is still a very significant number.
“When we are dealing with Microsoft updates, one thing we always reiterate, then reiterate some more, is to test before deploying. The guidance is always to download, test, then deploy the patches. With Microsoft, the test section of our guidance has gotten larger and larger.”
Next Page: Potential damage at an awkward time.
Potential Damage at an
Awkward Time “>
Nachreiner said faulty patches could cause major financial damage for enterprises.
“If its a patch being deployed on client machines, maybe its not that big a deal. But if youre patching servers that need to stay up to keep the business running, you can imagine the problem when something crashes. If you install that patch and all your VPN tunnels stop working properly, thats a big deal for a business.”
The patch reliability problem comes at an awkward time for Microsoft. The company has invested heavily to improve its patch creation and release mechanism and has stuck to its message that customers—business and consumers—should download and install its fixes.
The company has gone a step further, recruiting external patch testers under a formal Security Update Validation Program that gives select customers “limited and controlled access” to security updates ahead of public release.
The goal of that program is to provide a small number of dedicated external evaluation teams with access to the patches to test for application compatibility, stability and reliability in simulated production environments.
For the most part, Microsoft has come a long way. The company has not recalled a patch in some time, and significant patch re-releases have been few and far between.
Even Nachreiner, as harsh a Microsoft critic as there is, is willing to cut the software giant some slack.
“Theyve done a lot of things to convince me theyre getting smarter when it comes to dealing with software security. Ever since they announced the Trustworthy Computing initiative, theyve been slowly delivering on that. Even I have to admit that,” he said with a chuckle.
The plan to re-release the MS05-019 advisory is in itself a progressive move, Nachreiner added. “They know that a bad patch is a bigger problem. Its not easy to patch such a complicated vulnerability and the most important thing is to get it right.”
“Theres no doubt that theyre increasing their priority on security, but its always going to be a dilemma because the [Windows] operating system is so huge and complicated. Theyve just opened up the new advisories service, which is a big shift for them to even acknowledge vulnerabilities posted on the public lists.”
“Even the fact that theyll admit that those vulnerabilities exist and offer some guidance, that shows they are slowly getting it,” Nachreiner added.