eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
The Internal Revenue Service is as serious about auditing its computer systems as it is about tax returns. So when the W.32Blaster worm and subsequent attacks wreaked havoc on computer systems across the world, the federal agency was prepared: The IRS used autonomic computing software to distribute the appropriate Microsoft Corp. patch to more than 5,000 servers and 125,000 desktops and laptops across the nation.
The project, which took a week, saved the IRS more than $1.5 million in tech staff labor costs, according to Jim Kennedy, program manager of IRS Enterprise Systems Management, in Austin, Texas.
IT managers have long complained about the efforts necessary to stay on top of frequent security patching. Faced with tighter budgets and smaller staffs, organizations such as the IRS are turning to autonomic computing to automatically push software patches and software updates to end users.
“We had to accomplish in a few days what we normally would have taken a few weeks to do,” Kennedy said. “There is no way we could have touched 5,000 systems in the first 9 hours if we had done this manually.”
|
Microsoft announced an RPC (remote procedure call) DCOM (Distributed COM) vulnerability in mid-July and offered a patch for the issue. Last month, word spread that a worm that leveraged the RPC DCOM vulnerability had begun to spread rapidly. Once it sets up residence on a machine, the Blaster worm immediately began scanning the Internet for other vulnerable targets.
The SANS Institute, in Bethesda, Md., estimates that more than 150,000 computer systems were hit by the Blaster worm and by Nachi, which was written to seek out systems infected by Blaster and force a download of the security patch.
With Blaster and Nachi added to the crop of other malicious attacks, total virus damage last month might have reached an estimated $2 billion worldwide, according to a report by the Computer Economics Institute, in Carlsbad, Calif.
Threat Response
Two years ago, in response to the burgeoning virus plague, the IRS established a Computer Systems Instant Response Center to run intrusion detection software that looked for malicious code signatures in the agencys network traffic. The center also monitors external sites, such as The SANS Institutes Web site, to stay informed about current and future threats and thus keep the IRS network environment as protected as possible. In mid-July, when Microsoft released the RPC patch, the Computer Systems Instant Response Center notified Kennedy and his colleagues in Enterprise Systems Management and turned the patch over to them for testing and distribution.
Kennedys group immediately began testing the patch to see if it would break any internal applications. Testing of the server patch alone took almost three weeks, he said. Then there was the matter of deployment: The IRS computing infrastructure consists of 5,000 servers and more than 125,000 laptops and desktops nationwide. Nevertheless, by the time the Blaster worm appeared, the IRS had finished its server testing and had applied the patch to most of its servers. That didnt completely lock out the threat, however—left to be done were testing and deployment of the patch to the agencys client-based systems.
The IRS had been scheduling the patch distribution, but with the Blaster virus spreading rapidly, the agency had to install the patch in a matter of days to protect its systems. The agency used IBM Tivolis Software Distribution 4.1 and Tivolis Event Management tool, in conjunction with Tivoli Remote Control remote deployment management software, to push the patch. The IRS also used Tivoli Software Distribution 4.1 to deliver Symantec Corp.s Cleanup Tool to each system to remove all traces of the worm. Everything was managed using the Tivoli Enterprise Console.
Since the patch was being distributed during office hours as well, Kennedy used Tivoli Enterprise Console to see if a user was logged on to a machine that was being patched. If a user was logged on, he or she got a pop-up window explaining the system had been patched and would reboot in 5 minutes. “We didnt even look for the worm footprint because the Symantec tool could be run whether you had the worm or not,” he said. “It saved us the work of having to look for the footprint.”
In the past, the IRS would use the sneakernet method to distribute software patches. The agency would burn CD-ROMs and mail them to each IRS facility, where an IT manager would have to go from computer to computer to install the patch. Alternatively, the agency would load the patches on an FTP site and have users download the patches.
Deployment Overdrive
Kennedy estimates that the IRS has done more than 400,000 automated software distributions using the Tivoli software during the past two and a half years. If IRS IT staff had to visit each workstation, it would have taken about 45 minutes per machine to apply the patch, reboot the system and install the cleaning tools, he said. The manual installation project would have taken 1,200 people and more than $1.5 million in salary to get the job done in the same time frame, he said. “The amount of time and resources we saved by deploying the patch automatically is tremendous,” he said. “[Autonomic software is] really a way to stay on top of software patches these days.”
Discuss this in the eWEEK forum.
Senior Writer Anne Chen can be contacted at [email protected].