Inside Microsofts Zotob Situation Room

When Microsoft Corp. shipped the MS05-039 bulletin on Aug. 9 to patch a "critical" flaw in the Windows Plug and Play service, there was general feeling of trepidation within the Microsoft Security Response Center.

Software engineers at the Redmond, Wash., company smelled trouble right off the bat. It had been more than a year since the Slammer and Sasser worm outbreaks and, to MSRC Program Manager Stephen Toulouse, the severity of this PnP vulnerability brought back a mixture of memories ranging from chaos and confusion to outright pride in the way those threats were handled.

During the Slammer outbreak, Toulouse was at a service station when he learned of the attacks over his car radio. He recalls buzzing pagers, screeching tires and puzzled faces as he scrambled to get to Redmond to start the process of containing the worm.

This time around, it would be different. "This has been a very disciplined week," Toulouse said in an interview from the MSRCs specially created "Situation Room" at the height of the recent Zotob worm attacks.

"This is something we had created an entire process around and we were much better prepared this time," he said. "Our process is working, and its working very well."

That process, Toulouse explained, started long before Patch Tuesday. "Whenever were dealing with critical updates, one of the things we do is really look very hard at the attack vectors. What are the ways people will try to exploit this? How easy is it to create and unleash a worm? We attack the flaw just like the attacker would, and we knew up front that this one would be trouble.

"We had three critical bulletins in August but, in the case of the Plug and Play vulnerability, we knew there was a remote, unauthenticated attack vector affecting Windows 2000. Whenever theres a remote, unauthenticated attack vector, it sends up major red flags," Toulouse said.

Click here to read more about "critical" PnP fix in the latest MS Patch Day.

As is customary, Toulouse and others within the MSRC began making the media rounds, underscoring the severity of that particular vulnerability. "At that stage, were worried about this one. Our guidance immediately after the patches are released was for Windows 2000 users to apply MS05-039 as the highest priority update. We wanted to stress that upfront. If youre running Windows 2000, you need to pay attention to this one."

Then, a hiccup on the Download Center that caused a big distraction. One of the "critical" bulletins—with patches for a code execution Internet Explorer flaw—got corrupted, breaking the digital signatures and preventing them from installing. The MSRC was forced to pull the patches, investigate the cause of the problem and rerelease the bulletin.

"As soon as we push the button and the bulletins get published, we watch to make sure everyone can get them. We had to cope with the IE problems, but everything was fine for everything else, including the PnP issue. Then, we have to watch the discussion lists to see how the security community is reacting," he explained.

The immediate chatter around MS05-039 was no surprise. On the security mailing lists, hackers were openly discussing the severity of the Plug and Play hole and the ways in which it could be exploited. Microsoft was watching and taking notes, keeping a wary eye out for the first proof-of-concept exploit to be released.

By Thursday, Aug. 11, the first sign of exploit code appeared on the FrSIRT (French Security Incident Response Team) Web site. In all, five Windows exploits were posted, including two for the PnP flaw.

The MSRC mobilized and started testing the public exploits. The code provided a footpath to create a destructive worm, and a decision was immediately made to publish a fresh advisory with new warnings about the potential for danger.

Microsofts advisory went out late on Thursday with a very blunt message to Windows 2000 users: Patch, or else. Toulouse and his colleagues, meanwhile, prepared for a long, testing weekend.

"We saw the exploit code and our Security Windows Reaction Team tested it against the patch, and we were convinced we would see an attack. It was only a matter of time," Toulouse said.

"We knew we would want to have our guidance and protection content published on Microsoft.com, so we alerted the folks there about what we were expecting. We wanted to have an advisory and a separate incident page if an attack happened over the weekend. This is a process we have tested and refined with every incident.

"We mobilized the product support folks and discussed what kinds of calls to expect in the event of an attack. We wanted to make sure everyone had their cell phones charged; pagers had batteries. We made sure everyone understood this was going to be a long weekend," Toulouse added. "If something happened, we needed to move very quickly."

Unlike Blaster and Sasser—network worms that hit Windows XP machines—this attack could not successfully impact the general public. The affected Windows 2000 operating system is already out of mainstream support and is not considered a consumer operating system.

"A lot of things have changed since Slammer," Toulouse said. "Customers are more aware of the need to move into a maintenance mindset. Customers using Windows 2003 Server SP1 [Service Pack 1] werent impacted by the vulnerability because of changes we made. This is best example of learning how to make product more resilient to attack and have it be secure by default."

On Saturday, the MSRC staff checked the lists again and found that the proof-of-concept code was being modified. "People were looking at it, changing it, making it more dangerous," he said. "Were watching these discussions, watching the PSS [Product Support Services] calls to see if people were being impacted."

eWEEK.com takes a look back at some of the biggest security attacks against Windows 95 since its launch a decade ago. Click here to read more.

In the wee hours of Sunday morning, an enterprise customer contacted the MSRC with the first positive identification of what would become the Zotob attack. Toulouse declined to name the customer.

"They came to us with a sample of a new attack that they believed was exploiting the Plug and Play vulnerability," he said. "We took the code and started our own investigation. We also passed it to our VIA [Virus Information Alliance] partners to make sure everyone can get their signatures updated to provide protection."

The MSRCs investigation confirmed that an actual attack exploiting MS05-039 was under way and would only get worse.

"Early Sunday morning, our investigators tell us to get started on our process. We werent seeing a widespread attack, and the anti-virus vendors werent seeing anything major yet. But, with everything we knew, we decided to activate our security response process."

By 10 a.m. Sunday, pagers started buzzing. The Situation Room was set up in Building 27 at Microsofts Redmond campus.

"This is considered a major incident, so we want to have all the right people in one room," Toulouse said. "The people responsible for the update were there. The product team guys were there. The internal investigators who were working through the night were there to brief us on how the code worked. Our communication staff was there along with the PSS guys. Were all in one place going over the response plan."

Next Page: Gates, Ballmer are notified.