How Closely is Open-Source Code Examined?

By Larry Seltzer  |  Posted 2004-02-22 Print this article Print

The common wisdom assumes that open-source products get reviewed more often and with more quality. After all, the source code is open for everyone to see, so it must be getting reviewed, right? And closed source is closed, so it's not getting reviewed, ri

The usually simmering open source vs. closed source debate boiled over recently following the leak of Windows source code on the Internet. And it boiled over here too. Some 95 percent of the response to my column on the Windows source code leak and what it might indicate about the value of closed-source code as a security technique said that I didnt get the point: Since open source is open, it gets a better code review. Anyone can get the source, look at it and find problems in it. Inherent in this argument is the assumption that closed-source projects dont get code reviews, or at least that they get inferior ones. Im not so sure this is true. In fact, theres no reason to believe that closed-source companies cant do a good code review, and not a lot of reason to assume that open-source projects are getting all the code review that people think they get.

Meanwhile, there isnt any official system for reviewing open-source code for security problems. Its one of those ad hoc, community arrangements. Unquestionably a lot of checking happens; some from the same consultants who do "black box testing" of Microsoft products, and some from other open-source developers. Recently, however, an attempt to set up a formal organization, called Sardonix, to organize these reviews, essentially failed when funding dried up after nobody showed up to do the reviews.

A SecurityFocus article on the failure hints at the reasons: people dont want to volunteer to do the boring, rote parts of a real security audit. Instead, they want to find scary vulnerabilities and exploits, and then bask in the glory of having found them. The only contributions to the project came from Berkeley grad students under the direction of a professor. This is actually a great idea for an academic-driven project, but it doesnt give me a warm feeling about the level of experience of the reviewers.

On the other hand, the people at Microsoft who do code reviews are paid to do it. How well they review code is related to their own review and their own compensation. According to Michael Howard, senior program manager in Microsofts security business and technology unit, if a vulnerability is found in code you wrote or reviewed its going to noticed, and affect your own performance evaluation. This strikes me as a pretty good incentive to be careful.

Next page: Who Does The Reviews?

Larry Seltzer has been writing software for and English about computers ever since—,much to his own amazement—,he graduated from the University of Pennsylvania in 1983.

He was one of the authors of NPL and NPL-R, fourth-generation languages for microcomputers by the now-defunct DeskTop Software Corporation. (Larry is sad to find absolutely no hits on any of these +products on Google.) His work at Desktop Software included programming the UCSD p-System, a virtual machine-based operating system with portable binaries that pre-dated Java by more than 10 years.

For several years, he wrote corporate software for Mathematica Policy Research (they're still in business!) and Chase Econometrics (not so lucky) before being forcibly thrown into the consulting market. He bummed around the Philadelphia consulting and contract-programming scenes for a year or two before taking a job at NSTL (National Software Testing Labs) developing product tests and managing contract testing for the computer industry, governments and publication.

In 1991 Larry moved to Massachusetts to become Technical Director of PC Week Labs (now eWeek Labs). He moved within Ziff Davis to New York in 1994 to run testing at Windows Sources. In 1995, he became Technical Director for Internet product testing at PC Magazine and stayed there till 1998.

Since then, he has been writing for numerous other publications, including Fortune Small Business, Windows 2000 Magazine (now Windows and .NET Magazine), ZDNet and Sam Whitmore's Media Survey.

Submit a Comment

Loading Comments...
Manage your Newsletters: Login   Register My Newsletters

Rocket Fuel