eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
Patch Tuesday this month was an IT administrators worse nightmare.
Microsoft Corp. patched three “critical” flaws. Oracle Corp. plugged 49 database server holes. The Mozilla Org. fixed a dozen fairly serious Firefox and Thunderbird bugs. Apple Computer Inc. applied security stripes to Tiger. Cisco Systems Inc. corrected denial-of-service flaws in CallManager. And the list goes on and on.
According to a database maintained by the U.K.-based NISCC (National Infrastructure Security Co-ordination Centre), a total of 14 high-profile software vendors, including several Linux distributors, pushed out security updates between July 11 and 12, a deluge that caught enterprise IT administrators off guard and unprepared.
“Tuesday was a really, really rough day. We were prepared for Microsoft and Oracle, but to have Mozilla, Apple and others throw in important updates on the same day … it became a total nightmare,” said Mike Murray, director of vulnerable and exposure research at nCircle Inc.
Only Microsoft and Oracle, among big-name vendors, have a fixed patching schedule. Microsoft patches on the second Tuesday of every month, while Oracles Critical Patch Updates are always in the middle of the month, setting up a scenario in which Oracles patches will always coincide with Microsofts.
Multiple vendors patching at about the same time can have benefits, said Rick Greenwood, chief technology officer of Shavlik Technologies LLC, an enterprise patch management company.
“If its a planned event and you know that patches are coming, even if its from multiple vendors, you can set up a structure to plan. If you know Oracle and Microsoft and even IBM are releasing patches next week, you can make the arrangements for testing and deployment. Its not so much of a problem when you have advance notice,” Greenwood said in an interview with Ziff Davis Internet News.
“But whats happening now is that no one knows whats coming when. Then, suddenly, wham, you get all these patches at the same time. Thats when it becomes a nightmare trying to prioritize,” he added.
Neel Mehta, team leader of Internet Security Systems Inc.s X-Force research group, said Tuesdays mayhem underscores the need for “predictable patch release dates” from the big enterprise software vendors.
“I dont think everyone should be pushing out patches the same day, but if its within a certain time period, it helps everyone. It gives you the ability to plan ahead and have the resources ready to evaluate, test and deploy. Patch management is already too chaotic right now,” Mehta said.
Some vendors are pushing out security fixes around the same time as Microsoft in a “deliberate attempt” to duck under the radar, Mehta said. “If you look at whats going on, its clear that some companies are trying to hide their security problems by releasing on Microsofts patch day. They know the press and public will be zeroing in on Microsoft, so they release a patch and avoid the publicity.”
“Thats the opposite of the way vendors should be—up front about notifying customers about security issues. We would like to see them all pre-announce a patch release date and then make it clear exactly whats being patching and how severe the risks are,” Mehta said.
“In an ideal world, all the big vendors would be very public about patch release dates and would try to coordinate to stagger them to avoid the frenzy we saw this week. If admins are handling patches for many different environments, they can plan to test and deploy in succession rather than having to do them simultaneously.”
Chris Andrews, vice president of product management at patch management vendor PatchLink Inc., said the random nature of patch schedules adds to the industry-wide issue of how to apply security fixes in a timely manner.
“Most organizations, especially those with mission-critical operations, need to have negotiated downtime windows. There is value in having patches release around the same time, across multiple vendors, but it has to be known up front,” Adnrews said.
In emergencies, if a security flaw is very dangerous and theres an active exploit, Andrews said, patches should be released out of schedule. “Microsoft has the perfect idea. Let people know up front and if theres an emergency, theyll go out of cycle. Makes sense for everyone.”
Shavliks Greenwood called on companies like IBM, Sun Microsystems Inc. and Cisco Systems Inc. to follow Microsofts lead in pre-announcing when patches will be released. “It doesnt have to be around the same time as Microsoft or Oracle, but it would be nice to know when I have to get my Cisco devices ready for patches. That makes so much sense, I dont know why its not the industry standard.”
“IT admins dont like surprises. When you have a lot of stuff coming at you and youre not prepared for it, thats a worst-case scenario. [Tuesday] was one of those days when there were too many surprises,” Greenwood added.
“This week, we heard that companies just werent prepared. They were trying to determine which Microsoft and Oracle products should be fixed first when the other patches started coming in. It had to be stressful,” he said.
Beyond the bigger vendors on the network and server side, Greenwood said companies with widely deployed desktop products should also think about providing pre-patch notifications. “On most desktops, youll find Adobe Acrobat and RealPlayer, and these are also being patched fairly regularly. [The companies that make those products] should also be making it known when patches will be coming. Security is too important to be hidden.”
ISS X-Forces Mehta described patch management as an “ongoing dilemma” that needs to be addressed. “Patches are already too hard to deploy, with all the testing and evaluation and problems with breaking applications. If you are relying on patching to fully secure your business, youll always be scrambling.”
“I know organizations that wait up to a year to deploy patches, some of them very serious. They are looking to other things to protect them in that window of vulnerability while they test and get ready to deploy the patches,” Mehta said.
Murray, of nCircle, agreed that patch overloads further stretch the ability of IT departments to keep up with security. “Pre-notification is vital. If we had information ahead of time that Mozilla and Apple would be patching, it would be easier.
“Microsoft had the right idea. Announce a day and let everyone know. In a perfect world, everyone would coordinate on when theyll do patch releases, but I dont think well ever see the vendors coordinate that well,” Murray added.