Inside Microsofts Zotob Situation Room
Inside Microsofts Zotob Situation Room
When Microsoft Corp. shipped the MS05-039 bulletin on Aug. 9 to patch a "critical" flaw in the Windows Plug and Play service, there was general feeling of trepidation within the Microsoft Security Response Center.
Software engineers at the Redmond, Wash., company smelled trouble right off the bat. It had been more than a year since the Slammer and Sasser worm outbreaks and, to MSRC Program Manager Stephen Toulouse, the severity of this PnP vulnerability brought back a mixture of memories ranging from chaos and confusion to outright pride in the way those threats were handled.
During the Slammer outbreak, Toulouse was at a service station when he learned of the attacks over his car radio. He recalls buzzing pagers, screeching tires and puzzled faces as he scrambled to get to Redmond to start the process of containing the worm.
This time around, it would be different. "This has been a very disciplined week," Toulouse said in an interview from the MSRCs specially created "Situation Room" at the height of the recent Zotob worm attacks.
"This is something we had created an entire process around and we were much better prepared this time," he said. "Our process is working, and its working very well."
That process, Toulouse explained, started long before Patch Tuesday. "Whenever were dealing with critical updates, one of the things we do is really look very hard at the attack vectors. What are the ways people will try to exploit this? How easy is it to create and unleash a worm? We attack the flaw just like the attacker would, and we knew up front that this one would be trouble.
"We had three critical bulletins in August but, in the case of the Plug and Play vulnerability, we knew there was a remote, unauthenticated attack vector affecting Windows 2000. Whenever theres a remote, unauthenticated attack vector, it sends up major red flags," Toulouse said.
As is customary, Toulouse and others within the MSRC began making the media rounds, underscoring the severity of that particular vulnerability. "At that stage, were worried about this one. Our guidance immediately after the patches are released was for Windows 2000 users to apply MS05-039 as the highest priority update. We wanted to stress that upfront. If youre running Windows 2000, you need to pay attention to this one."
Then, a hiccup on the Download Center that caused a big distraction. One of the "critical" bulletinswith patches for a code execution Internet Explorer flawgot corrupted, breaking the digital signatures and preventing them from installing. The MSRC was forced to pull the patches, investigate the cause of the problem and rerelease the bulletin.
"As soon as we push the button and the bulletins get published, we watch to make sure everyone can get them. We had to cope with the IE problems, but everything was fine for everything else, including the PnP issue. Then, we have to watch the discussion lists to see how the security community is reacting," he explained.
The immediate chatter around MS05-039 was no surprise. On the security mailing lists, hackers were openly discussing the severity of the Plug and Play hole and the ways in which it could be exploited. Microsoft was watching and taking notes, keeping a wary eye out for the first proof-of-concept exploit to be released.
By Thursday, Aug. 11, the first sign of exploit code appeared on the FrSIRT (French Security Incident Response Team) Web site. In all, five Windows exploits were posted, including two for the PnP flaw.
The MSRC mobilized and started testing the public exploits. The code provided a footpath to create a destructive worm, and a decision was immediately made to publish a fresh advisory with new warnings about the potential for danger.
Microsofts advisory went out late on Thursday with a very blunt message to Windows 2000 users: Patch, or else. Toulouse and his colleagues, meanwhile, prepared for a long, testing weekend.
"We saw the exploit code and our Security Windows Reaction Team tested it against the patch, and we were convinced we would see an attack. It was only a matter of time," Toulouse said.
"We knew we would want to have our guidance and protection content published on Microsoft.com, so we alerted the folks there about what we were expecting. We wanted to have an advisory and a separate incident page if an attack happened over the weekend. This is a process we have tested and refined with every incident.
"We mobilized the product support folks and discussed what kinds of calls to expect in the event of an attack. We wanted to make sure everyone had their cell phones charged; pagers had batteries. We made sure everyone understood this was going to be a long weekend," Toulouse added. "If something happened, we needed to move very quickly."
Unlike Blaster and Sassernetwork worms that hit Windows XP machinesthis attack could not successfully impact the general public. The affected Windows 2000 operating system is already out of mainstream support and is not considered a consumer operating system.
"A lot of things have changed since Slammer," Toulouse said. "Customers are more aware of the need to move into a maintenance mindset. Customers using Windows 2003 Server SP1 [Service Pack 1] werent impacted by the vulnerability because of changes we made. This is best example of learning how to make product more resilient to attack and have it be secure by default."
On Saturday, the MSRC staff checked the lists again and found that the proof-of-concept code was being modified. "People were looking at it, changing it, making it more dangerous," he said. "Were watching these discussions, watching the PSS [Product Support Services] calls to see if people were being impacted."
In the wee hours of Sunday morning, an enterprise customer contacted the MSRC with the first positive identification of what would become the Zotob attack. Toulouse declined to name the customer.
"They came to us with a sample of a new attack that they believed was exploiting the Plug and Play vulnerability," he said. "We took the code and started our own investigation. We also passed it to our VIA [Virus Information Alliance] partners to make sure everyone can get their signatures updated to provide protection."
The MSRCs investigation confirmed that an actual attack exploiting MS05-039 was under way and would only get worse.
"Early Sunday morning, our investigators tell us to get started on our process. We werent seeing a widespread attack, and the anti-virus vendors werent seeing anything major yet. But, with everything we knew, we decided to activate our security response process."
By 10 a.m. Sunday, pagers started buzzing. The Situation Room was set up in Building 27 at Microsofts Redmond campus.
"This is considered a major incident, so we want to have all the right people in one room," Toulouse said. "The people responsible for the update were there. The product team guys were there. The internal investigators who were working through the night were there to brief us on how the code worked. Our communication staff was there along with the PSS guys. Were all in one place going over the response plan."
Next Page: Gates, Ballmer are notified.
By midday, senior executives including Microsoft Chairman Bill Gates and CEO Steve Ballmer were notified. The "Executive e-mail" is a key part of the response process, and it includes the use of a very specific, high-priority subject line to make sure the mail is read by the senior executives.
The security advisory that was first issued to warn of the attack was updated to confirm an attack was under way. Toulouse himself placed a warning on the MSRC Blog. A stand-alone Zotob incident page was created while Microsofts virus encyclopedia was updated to reflect the new threat.
"The stand-alone incident page is important," he said. "Once the word got out that an attack was under way, we need to have specific instructions to help people understand what was going on and how they could protect themselves. If someone got infected, they could find help to clean up."
Banner headlines were placed on the front page of Micrsoft.com. The warning was duplicated on the companys security portal and on the Windows 2000 product page. E-mail blasts were sent with links to the incident page, patch download locations and other mitigation guidance.
Zotob was still a very low threat but, with businesses opening for work Monday, there was a likelihood that things would escalate.
"Although infection rates are low, it doesnt mean its not a bad situation," Toulouse said. "We want to make sure, not only are we providing information to make sure customers arent impacted, but to make sure they know how to get back to an operational state."
By Monday morning, the variants started squirming, refining the original Zotob code to get around anti-virus detections. The internal investigation team was back at work, analyzing the code, rushing to keep up with the virus writers.
By Monday evening, the virus encyclopedia was updated to add entries for Zotob.C and Zotob.C. "They [the virus writers] were changing the executables and changing the way they scanned for networks. As we find the new variants, were updating the stand-alone incident page," Toulouse said.
Next Page: CNN gets hit.
"Were literally in a meeting going over our plans, keeping track of things, when we got word on Tuesday that CNN was reporting they had been hit. At the time, we knew a high-profile target was reporting they had been hit and they didnt know what it was. Their computers were shutting down and restarting," Toulouse recalled.
Microsoft would use the mainstream media interest to its advantage. The MSRC got in touch with CNN officials to discuss the attack and help contain the threat, but the television network would offer more value.
"We invited them to the Situation Room, and we let them help us get the word out. This attack against CNN was not a new attack. It was the same thing we were seeing since Sunday, but it became a major story because some big media companies got infected.
"At this stage, there are two things we want people to know. This affects Windows 2000 only, and the available patch provides protection. We also want them to know were working to help those who were impacted."
By this time, there were about a dozen Zotob mutants and evidence of rival virus writers deleting each others malware. The MSRC made the decision to ship an out-of-cycle update of the malicious software removal tool to offer detection and disinfection.
The utility is normally updated only on the second Tuesday of every month, but with Zotob squirming and the mainstream media reporting a major outbreak, Microsoft wanted customers to find immediate help.
"In the Situation Room, everything is happening simultaneously. While we were providing updates to our incident page, we were working on getting that information to the malware removal tool. We werent seeing, from our end, a massive sudden change in situation. It was the same low level of impact but with new variants and some different customers were being impacted," Toulouse said.
Once the malware remover shipped, all the guidance pages needed to be updated to reflect the fact that it was available and to point customers to the download location. The plan was to add the tool to Automatic Updates as part of the September patch cycle.
"At the moment, were still mobilized, but things have settled down," he said. "Were continuing to investigate the variants. As we see new variants, well add detections in the tool and make that available broadly on the next patch day."
Check out eWEEK.coms for Microsoft and Windows news, views and analysis.