Last November, an eWEEK reader who is an IT executive at a large organization was notified by his companys help desk that the companys Microsoft Corp. Exchange e-mail servers had gone offline.
Further investigation revealed that the Temp directory of the Exchange servers—along with other crucial directories and files—was suddenly missing. The result was 500GB of unavailable e-mail data.
The problem affected dozens of users and took nearly four days to solve. The entire help desk team was pulled from daily support tasks and pressed into a server-by-server, desktop-by-desktop recovery effort. By the end of the ordeal, it was determined that nearly 40 data center servers had been affected, many of which had to be rebuilt from scratch.
The problem? The machines had been infected by a user-level rootkit.
During an exclusive interview with eWEEK Labs last month, the IT executive described the attack and the step-by-step recovery efforts his company undertook. eWEEK Labs agreed not to name the IT executive or his organization.
Rootkits are widely known in the Unix and Linux community, but they are a fairly new problem in the Windows operating system world.
Indeed, at last months RSA Conference in San Francisco, a workshop focused on the emerging threat rootkits are posing to Windows. User-level rootkit hacks like the one described here are bad enough, but when it comes to a server infected with a kernel-level hack, "Nuke it from space" was the advice provided by the Microsoft employees leading the session.
The user-level rootkit that felled the IT executives servers was tailored for French language use, and thats how it evaded detection by a widely deployed anti-virus tool used at the execs company . The executive suspects that an administrative assistant given to wide-ranging Internet use was the weak link that enabled the rootkit infection once the rootkit was inside the network.
A forensic examination of Machine Zero revealed a keystroke logger with extensive records dating back several months.
Before this was discovered, however, a PC support technician responding to the administrative assistants report of a desktop slowdown committed a grave error—one that allowed the rootkit to spread from the users desktop to the servers. Unable to gain access to the system using the regular administrator account, the technician decided to use the domain administrator account to gain access to the PC. At this point, the rootkit was off to the races.
Almost instantaneously, the password grabber that was part of the rootkit used the domain administrator account to infect servers on the local network. The effect was devastating to the IT executives organization in more ways than one: E-mail was knocked offline in order for the hijacked servers to act as illicit distribution points for the "Bennifer" bomb "Gigli"—dubbed into French.
To recover from the infection, the IT executive first had the central network staff poison the DNS (Domain Name System) tables, cutting off the rootkits default connections to the outside world—in this case, several sites in France and two major American universities that, unbeknown to network managers, housed infected systems that were acting as robot controllers.
The wily rootkit didnt make recovery easy, though.
"We tried booting from ERD Commander [a utility from Winternals Software LP] to change the local password, but the root kit [later known as SpartaDoor and by Symantec Corp. as trojan.backdoor] checked the box preventing the user from changing the password," the IT executive said. "We missed that trick, costing us a lot of time."