Microsofts Security Response Center: How Little Patches Are Made (
Page 1 of 2 )
Tech Ed conference attendees get a behind-the-scenes look at how Redmond handles the creation of software patches—and an explanation for long delays in fixing known vulnerabilities.ORLANDO, Fla.—Anxious to shed the companys image as having a lax attitude about software security, officials at the Microsoft Security Response Center are using the Tech Ed conference here to provide a rare glimpse at the step-by-step process used to create, test and roll out security patches.
The software maker trained the spotlight on the operations of the MSRC during breakout sessions and one-on-one discussions with customers, stressing that all publicly—and privately—reported vulnerabilities are thoroughly investigated to determine whether customers are at risk.
"Were on all the [security mailing] lists, just like you are, and we investigate everything, even if its a post about a simple weird behavior in a product," said MSRC program manager Stephen Toulouse.
By monitoring the public lists and underground hacker sites, Toulouse said the company is able to keep track of discussions about vulnerabilities that may not have been reported to Microsoft.
In addition, he said, a "real person" monitoring the secure@micosoft.com e-mail address keeps close watch on all vulnerability reports sent in by private researchers.
"Once we determine its a real security issue, it goes into the triage phase. We immediately get in touch with the researcher and ask for more information so we can recreate the attack scenario internally. We also look for different attack scenarios," he said.
Researchers have complained in the past that Microsoft routinely ignores threat warnings, which contributes to the underlying distrust, but Toulouse said the companys mission is to improve its relationship and "create a community" with grey hat hackers.
"We respond immediately to the initial vulnerability report and provide the researcher with contact names, e-mail addresses and phone numbers. We make it clear we want to work closely with the researcher to pinpoint the problem and get it fixed. We commit to providing [researchers] with a progress report on the Microsoft investigation every time they ask for one," he said.
"We may not have enough information from the initial report and we may not know the configurations of the [researchers] vulnerable system, so we always work closely to get all the relevant details."
Armed with the vulnerability information, the triage team kicks into high gear. On every product team within Microsoft, a staff member is on call to coordinate with the MSRC and join the investigation, Toulouse said.
"It becomes a detailed, complicated process because we are looking at all supported versions of Windows to understand the complete angle and we also have to figure out the impact it could have on customers," he said. "Once we are talking to a product team, weve already confirmed its an issue that needs to be fixed and were already working on the bulletin that will eventually be released [on Patch Tuesday]."
In the midst of investigations, especially for flaws in network-facing products that could cause code execution attacks, he said, the MSRC continues to scour the Internet to determine whether public exploits are circulating.
Windows security ranks high on this years Tech Ed agenda. Click here to read more.
At this stage, Toulouse said, "parallel investigations" are done by the MSRC and the product teams to "widen the circle" and reproduce the attack vector.
"We are now thinking like the hacker. We look at the vulnerability from the hackers perspective. If its a buffer overflow and its Internet-facing, the alarms go off and we have to figure out if its easy to exploit. The product team is also investigating from the code point of view. They want to know if there are additional vulnerabilities in the associated code. So it becomes a complete audit," he said.
This, he admitted, leads to long delays in getting a comprehensive fix rolled out, but its a risk the company is willing to take to ensure the quality of patches. "We find that the [audit] process gives us more depth, and we keep the researcher in the loop throughout the process to let them know what were doing."
The testing process also puts a monkey wrench in the patch release timetable. Even though Microsoft has recruited external patch testers as part of a formalized Security Update Validation Program, Toulouse said the quality assurance process has become "very, very complex," especially for products that are closely tied to the operating system.
Click here to read more about Microsofts decision to use external patch testers.
"In theory, we can release an update with a patch very quickly, but thats a big mistake. One of the things customers demand is quality patches. They dont want to deal with faulty patches that break their applications and they dont want to deal with all the associated trouble," he said.
Its never a perfect science, and Toulouse said the MSRC learned the hard way in 2003 when it rolled out the MS03-026 bulletin to fix a code execution hole in a DCOM RPC (Windows Distributed Component Object Model Remote Procedure Call) interface and subsequently discovered that it was a substandard patch.
A few weeks later, the Blaster worm ripped through the Internet and Microsoft released MS03-039 with an admission that additional ports involving RPC remained unpatched.
"That was an experience that taught us a valuable lesson. Its better if we find it before the bad guys figure it out," he said.
In some cases, particularly when the Internet Explorer browser is involved, the testing process "becomes a significant undertaking," Toulouse said. "Its not easy to test an IE update. There are six or seven supported versions and then were dealing with all the different languages. Our commitment is to protect all customers in all languages on all supported products at the same time, so it becomes a huge undertaking."
"This is exactly why it can take a long time to ship an IE patch. Were dealing with about 440 different updates that have to be tested. We have to test thoroughly to make sure it doesnt introduce a new problem. We have to make sure it doesnt break the Internet. We have to make sure online banking sites work and third-party applications arent affected," he added.
Internet Explorer updates are also cumulative, meaning that they address several newly discovered vulnerabilities and all previously released patches, causing even more delays when the new fixes are bundled into older updates.
"This is why it takes so long, but thats not to say that if theres an exploit, we wont accelerate testing and get it out there as fast as we can. But if we find problems in the testing phase, it could trigger a restart and cause even more delays," Toulouse said.
Next Page: The job doesnt end on Patch Day.
