eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
For critics of Microsoft Corp.s software, 2003 was a very good year. The appearance of the Slammer and Blaster worms was evidence—if any were necessary—that things had gone badly awry at the Redmond, Wash., software giant.
In articles over the days and weeks that followed, security experts and even the companys customers took Microsoft to task for issuing too many patches and doing too little to make them easy to deploy.
Chairman and Chief Software Architect Bill Gates year-old Trustworthy Computing initiative had failed, experts concluded.
Today, many of those security experts have changed their tune and now say that Microsofts commitment to improving security, which began in earnest with the Trustworthy Computing memo, has begun to pay dividends.
Microsoft, the argument now goes, has transformed itself from an IT security laughingstock to an industry leader and advocate for secure development practices.
Holes in Windows are fewer and harder to find. Other software vendors, such as Oracle Corp., that ridiculed Microsoft, now find that they are the target of security researchers ire.
At the same time, Microsoft has gone from pariah to security industry darling: host of swank parties and mixers at the annual Black Hat hacker conference in Las Vegas; sponsor of its own researcher confab, Blue Hat; and a major employer of security talent.
The sunny reviews are no accident. Almost four years after Gates hit the Send button on the Trustworthy Computing e-mail, harnessing the work, minds and goodwill of security researchers has become a key element of Microsofts strategy for improving the quality of its products and burnishing its tarnished image.
George Stathakopoulos, general manager of the Security Engineering and Communications group at Microsoft, was an early advocate of improving relations with independent security researchers.
As a young engineer at Microsoft in the early 1990s, Stathakopoulos was part of the teams that shipped Windows 3.1.1 and Windows for Workgroups before becoming one of the original members of the Internet Explorer product group in 1995. He remembers the first security bug that was reported in IE, his companys awkward response to it and the string of viruses that followed: BubbleBoy, Melissa, ILoveYou, Code Red and Nimda.
As reports of new holes in IE poured in during the late 1990s, Stathakopoulos said, he and others often fumbled their response to them.
“We did not know how to handle [bug reports]. … I personally remember looking at a bug and saying, This is by design. It has to be this way,” Stathakopoulos said.
A visit to Black Hat during that period didnt help, Stathakopoulos said.
“It was not pleasant,” Stathakopoulos said. “This guy came out making smart-ass comments about Microsoft and then showing problems we have with our products. I remember being infuriated.”
Hours later, however, Stathakopoulos found himself wondering aloud to a colleague about the security holes: “How could we have missed that?”
Three years later, Stathakopoulos and Microsoft were not only back at Black Hat, they were hoisting drinks with attendees at a company-sponsored party—the first of many to come. “We didnt know if anyone would show up,” Stathakopoulos said.
But the hackers did show up, in large numbers and on time, Stathakopoulos said.
After an awkward few minutes, during which Microsoft and non-Microsoft attendees kept to themselves, the two groups began to mingle, with Microsoft techies tossing back drinks with renowned bug hunters such as David Litchfield, of the U.K.-based company Next Generation Security Software Ltd., who discovered the hole used by the Slammer worm, and Marc Maiffret, co-founder of eEye Digital Security Inc., in Aliso Viejo, Calif., Stathakopoulos said.
The new Blue Hat conferences grew out of the companys experience at events such as Black Hat, wrote Andrew Cushman, director of the Security Engineering and Communications group.
Unlike the Las Vegas extravaganza, Blue Hat allows Microsoft to bring Black Hat-style presentations right to the companys doorstep. Even more important, it gives high-level executives access to top security minds, said David LeBlanc, former security architect for Microsofts Office Division and now chief software architect at Webroot Software Inc., an anti-spyware company in Boulder, Colo.
The most recent Blue Hat, in October, brought Black Hat veteran Dan Kaminsky and “white hat” hackers such as Dave Maynor, of Atlanta-based Internet Security Systems Inc., and Matt Miller and Vinnie Liu, of the Metasploit Project, to Redmond to discuss their techniques for finding holes in Microsoft products.
More than 1,200 Microsoft developers attended sessions with the researchers, filling the Redmond campus largest lecture hall. On another day, the white hats lunched and gave abbreviated versions of their presentations to an audience of Microsoft executives that included Jim Allchin and Kevin Johnson, co-presidents of the companys Platform Products & Services Division, and Mike Nash, head of the companys Security Business & Technology Unit.
“I cant say Ive ever dropped a zero-day on senior management before,” Kaminsky, an independent researcher, wrote in a Microsoft-sponsored chat session following the event, referring to an undiscovered security hole in the companys software.
“I walked into a room with the head of Windows and three of the brains that made it happen,” Kaminsky wrote of his meeting with Microsoft brass. “Whats the first thing I did? Dove into obscure protocol negotiations and asked if I was actually seeing a problem. Looks like I was,” he said.
Next Page: What Blue Hat really does for Microsoft developers.
What Blue Hat Really
Does for Microsoft Developers”>
“Blue Hat is just part of a larger picture, which is a really broad effort to make Microsoft accessible,” said Adam Shostack, an independent security consultant in Atlanta who participated in the Blue Hat event in October.
“Pretty much any [security] conference you go to, theres a Microsoft presence.”
More interaction with the research community has given Microsoft a softer touch, even with so-called grey-hat hackers who dont always toe the corporate line or adhere to the companys vulnerability disclosure policies.
“Microsoft still has a long way to go, but theyre making an effort to build good relations with researchers, including myself,” said Tom Ferris, an independent security researcher in Mission Viejo, Calif., who runs the Security-Protocols.com Web site and has published details on several unpatched holes in Microsofts products.
Compared with other organizations, Microsoft representatives go out of their way to show respect to researchers, Ferris said.
“Theyre not hostile or offensive in e-mails. … Theyre always nice. They dont want to [tick] off the researcher,” Ferris said.
Thats a big change for a company that had a reputation for giving frosty receptions to people who reported bugs.
On the security front, Blue Hat hasnt yielded “aha” security moments as much as it has broadened the thinking of Microsofts developers, said Stephen Toulouse, security program manager at Microsofts Security Response Center.
“What were striving for is an outside perspective—getting developers to understand the misuse of code,” Toulouse said.
But there are still more than a few researchers who see the Blue Hat conferences as little more than shrewd PR for a company that is widely believed to produce insecure software.
“Microsoft got their ass handed to them by worms. It was a public embarrassment and bad [public relations],” said eEyes Maiffret, whose company frequently finds and reports critical holes in Microsofts products and has had a testy relationship with the company for years.
Maiffret gives Microsoft high marks for improving the quality of its code in recent years. But events such as Blue Hat are more public relations than serious security work, he said.
The experts who have been invited to the event are not the same researchers who are discovering the critical holes in the companys products, he said.
Still, experts and Microsoft insiders say that warm, fuzzy relations with the independent security community is just one part of the companys security makeover under Trustworthy Computing, but not the most important.
The whole initiative, especially Blue Hat, is really about increasing the security know-how of its developers, said Mike Howard, senior security program manager at Microsoft and an author of Microsofts Security Development Lifecycle program, which many experts credit with improving the quality of the companys code.
Microsoft has also used the power of its bulging purse to buy up or bring under contract some serious security talent.
Litchfields NGSS counts Microsoft as a customer, and Ferris claims the company offered him a position on its kernel development team, which he turned down. A Microsoft spokesperson said the company doesnt comment on hiring issues.
“Microsoft has hired an awful lot of my friends in the last few years,” said Shostack, who has never worked for Microsoft. “These are all security people, and theyre all over the company.”
“Theyre using their monopoly power. Its not all bad, but there are some who look at it in a cynical light,” said Gary McGraw, chief technology officer of Cigital Inc., in Dulles, Va., who declined to comment on whether his company, which helps vendors write secure applications, is under contract to Microsoft but admitted having worked with the company in the past.
Still, more security know-how coupled with better programming and liberal use of automated security scanning tools have eliminated many easy-to-exploit buffer overflow and string copy holes, experts agree.
“The best way to think about it is as an iceberg floating south. Its gradually getting smaller, and the bug hunters are scrambling for space,” said Litchfield in Surrey, England.
Next Page: The hacker bulls-eye could be shifting.
Hacker Bulls
-Eye Could Be Shifting”>
“The biggest thing Ive seen is that security moved from an ad hoc, piecemeal approach—bug hunting—to something well-defined thats part of an overall process,” Webroots Leblanc said. “Its something a lot of companies need to emulate.”
Given the events of the last six years, security experts say that what once was unthinkable may someday come to pass: hackers turning their attention from Microsoft to easier pickings in the software of other companies.
Database and enterprise software giant Oracle often comes up in discussions of other likely targets.
Researchers liken Oracle in 2005 to the Microsoft of 1999: a major software vendor with big ambitions, a huge, complicated product, a dearth of security expertise and an attitude problem.
“I remember sitting down with our research guys one night with Oracle and we found about five different flaws right away, and then just gave up,” Maiffret said. “It was like, whats the point.”
Vulnerabilities exist in all software, but Oracles response to eEyes reports is sending up red flags.
“Its like Microsoft five years ago. The technical expertise isnt there. You tell them its a buffer overflow, and have to completely draw it out for them, or they try to argue that its not a [security] problem, its just a crash,” he said.
Litchfield of NGSS recently published an open letter on the Bugtraq security discussion list that excoriated Oracle for its slow and shoddy software patching procedures, which he said left the companys customers vulnerable to attack and gave them a false sense of security.
Oracles October quarterly CPU (Critical Patch Update) addressed some of Litchfields earlier criticisms and does a better job of fixing security holes in the companys database software.
For example, the latest CPU fixes not only reported holes in the companys products, but also similar holes in other areas of the code, Litchfield said. However, that change in practice only brings Oracle to the point where vendors such as Microsoft were three or four years ago.
The story isnt much better at vendors like Apple Computer Inc. and Hewlett-Packard Co., not to mention the banks, retailers and other large corporations that write and use their own software, McGraw of Cigital said.
“The biggest hurdle is that developers dont know diddly about security,” McGraw said.
Ironically, he said, the lack of knowledge and training about security is especially chronic among the older and more experienced developers who came of age before the Internet and application security were high priorities, and who are often project managers with oversight of major software development projects.
“The more experienced they are the less they know and the less time they have to learn,” McGraw said.
Microsofts development process and procedures are unique, and uniquely suited to a mammoth software development shop. However, companies that want to make their software more secure will have to take many of the same steps as Microsoft to turn their ship around, McGraw said.
“Youve got to train your [developers], build a knowledge base, do analysis on existing products and fix them,” he said.
Even more importantly, companies have to get buy-in from the highest levels of management to make security a top priority, as Gatess Trustworthy Computing memo did at Microsoft, McGraw said.
“There were a lot of cynics who said that Microsoft is posturing, but the company has put its money where its mouth is and made slow, torturous progress,” he said.