Patching systems is always painful, but Tuesdays scenario of two major cumulative patch sets on the same day—from Oracle and Microsoft—is downright sadistic.
When will this patching madness end? Not soon. Although theres a concerted effort on the part of ISVs to develop more secure code, analysts predict that it will take generations before this scrubbed-up, buttoned-down code catches up with the Swiss cheese that is legacy applications.
“The problems will only get worse, not better, anytime soon,” said Jon Oltsik, an analyst with Enterprise Strategy Group. “Well have more software, more integration, more functionality [and] new protocols, and all that stuff will, by its nature, be insecure out of the chute.”
If youre running an Oracle Corp. database on top of a Microsoft Corp. platform, you have double the patching delight. A sizable number of people do indeed host that scenario, with Oracle being the No. 2 database to run on the Windows platform, right after Microsofts SQL Server database.
To ensure that patches dont break anything, enterprises that have the resources and the ability to take their systems offline put patched systems into a testing environment. With regression testing, they replicate their production systems and run them through technical and financial transactions to ensure that production wont grind to a halt because of a patch set.
Its the safest way to go, and youd think all enterprises would do it. “Those who test in production usually die by their methods,” said Mike Herald, a consultant at Pronto North America, an ISV that markets an ERP (enterprise resource planning) management system. “Weve got the same issues with our products.”
Heralds company learned the hard way that taking customers live on their software releases, without walking them through regression testing, translates into multiple nights of misery.
“I was in here working 10-hour days trying to repair the mess,” Herald said.
Thanks to that learning experience, Pronto now declines to roll out software in live production environments. “Weve since not done upgrades unless they name a project to the upgrade that dedicates resources, time and money,” he said. “In this case here, as a software company, its tough to say that, but weve come out and done that.”
Still, plenty of businesses forgo testing, given the laundry list of resources and the breathing room for system downtime that it requires. After all, enterprises need adequate space in server boxes to replicate their environments. They need manpower. They need to plan, coordinate and time potential disruption phases when the system can be down.
Heralds wife, for example, who works as a project manager at another company, turned down Oracles April patch set because her employer didnt make the necessary resources available to test the system outside the production environment.
“They basically turned down a patch set to 10g for one of the applications they were running, because a) they didnt have the resources to provide a separate test instance, and b) they werent willing to take the risk to what the patch test would do to what they had in production,” Herald said.
“She, as project manager, said to the business, and to IS, Were not going to do it, not on my watch. You cant give me the dedicated resources, both computer and people, to thoroughly test the patch set.”
After all, Herald said, at least his wifes employer had been living with the broken application. “What you dont want is for it to break something else,” he said.
As far as testing the Microsoft-Oracle duo, Oltsiks advice is to first test the platform—i.e., Windows—for two reasons. First, the application is only as stable as the platform on which it sits. And second, there are by far more attacks launched against Microsoft than against Oracle.
Regarding Oracle flaws, Oltsik recommends prioritizing patching according to severity. Thats not always an easy task, given the limited amount of information on severity that ISVs such as Oracle include in their alerts, and thus it requires digging into third-party sites that rate severity.
Also, Oltsik recommends keeping an eye on those who have access to the database. “I would … make sure anyone with access to the database is monitored, because theyre the most likely to go in and exploit those vulnerabilities,” he said.
For those shops doing any Oracle communications over the Internet, make sure the perimeter is secure as well. “[You dont want to] poke a hole in the firewall that lets people over the Internet find my Oracle servers,” Oltsik said.
