The Problem with the Methodology

By Larry Seltzer  |  Posted 2008-07-03 Print this article Print

Here's the problem: Because they had no minor version information, the authors took the position that anyone running IE7 was running the most updated, secure IE browser, and anyone running earlier versions was not. This is even though IE6 and even IE5 are currently supported browsers and patches are being issued for them. Mozilla retires support for old versions quickly and in fact just announced that support, including security updates, for Firefox 2 would terminate by mid-December, about six months after the release of Firefox 3. Microsoft, on the other hand, has committed to maintaining support for IE5, which debuted in 2000 with Windows 2000, as long as Windows 2000 is supported (7/13/2010). Same with IE6 for as long as Windows XP (4/8/2014) and Windows Server 2003 (7/14/2015) are supported.

With Firefox, Safari and Opera, on the other hand, they were able to say that a user was running the latest patch level. In other words, they held Internet Explorer to a different standard than they did for the other browsers. This is a common error made in security analysis, and the authors make it another way in the study.

Those of you who work in corporate IT know that you don't jump into installing new major versions of anything, especially critical applications like Web browsers. Microsoft is continuing to support IE5 on Windows 2000 because customers demand it, not because they might as well do it. They can't unilaterally order their customers, as Mozilla and Apple do, to upgrade. Small wonder those products don't have any significant official corporate adoption, and if they did, their policies would soon change.

I know I shouldn't stereotype like this, but who are Firefox users? My guess is that they are largely more technical users who take an active interest in their systems and have authority over those systems. Such users are going to update when new patches come out. Corporate users don't have the option of applying updates or installing new versions; IT does that for them.

Yet the study concludes from their numbers that Firefox's update mechanism must be better than the competition's because people use it. I have found the odd bug or two in Firefox's updater, but it is very good, at least for an individual user. It's not so good for a managed corporate network (note that there is an Active Directory-enabled version of Firefox available, customized by a third party). Perhaps the difference in adoption rates for IE major versions has nothing to do with updating mechanisms and everything to do with the choices that customers have available to them.

So if a customer chooses to run IE6 and keep it up to date with security patches, are they running the most secure browser they can? Certainly not, as the paper is correct to point out that IE7 is a much more secure browser. But Microsoft is supporting the browser and providing security updates, so running an updated IE6 is not the same as running, for instance, a version of it that hasn't been updated in a year. The authors' methodology also leaves them concluding that all copies of IE7, including those that have never been updated with the patches for that browser, are up to date.

Why, one might ask, does Microsoft not provide minor version information? Microsoft's David LeBlanc answers that question in his blog by saying that they consider such information to be an "information disclosure vulnerability." In other words, by giving a Web-based attacker precise version information, you are also giving them better information on how to attack that browser. Alun Jones challenges LeBlanc's characterization of minor version information in this way as a comment to the blog entry, and LeBlanc responds inline. I recommend reading it all for extra credit. (Yes, there will be a test on this.)

Finally, getting back to the Secunia data, the authors' results for that set were very different from those of the Google data:

Secunia [21] identified (for the month of May 2008) that 4.4% of IE7, 8.1% of Firefox, 14.3% of Safari (Windows only), and 15.2% of Opera users have not applied the most recent security patches available to them from the software vendor. In comparison, we discovered that 16.7% of Firefox, 34.7% of Safari (all OS), and 43.9% of Opera Web browser installations (using our Web server log-based measurements) had not applied the most recent security patches. We found that our Firefox, Safari, and Opera results were higher than those of Secunia's, differing by a factor of 2.1 (Firefox), 2.4 (Safari), and 2.9 (Opera), and attribute this difference to a probable bias for more security aware users to take advantage of Secunia's security scanner PSI than the average global community.

In these measurements IE7 users are much more likely to be up to date than other browser users. The authors are correct that Secunia users are more likely to be security-aware, but even when they try to adjust the numbers, multiplying the IE7 number by 2.1 "... to correct for the bias of Secunia's measurement within a security aware user population," IE7 still ends up looking better.

I know I would certainly like to get my hands on Google's user-agent logs. There are plenty of great studies you can do with it, and this study could have been a lot better had they not overreached. Unfortunately, we can't conclude a lot based on it.

Security Center Editor Larry Seltzer has worked in and written about the computer industry since 1983.

For insights on security coverage around the Web, take a look at Security Center Editor Larry Seltzer's blog Cheap Hack.


Larry Seltzer has been writing software for and English about computers ever since—,much to his own amazement—,he graduated from the University of Pennsylvania in 1983.

He was one of the authors of NPL and NPL-R, fourth-generation languages for microcomputers by the now-defunct DeskTop Software Corporation. (Larry is sad to find absolutely no hits on any of these +products on Google.) His work at Desktop Software included programming the UCSD p-System, a virtual machine-based operating system with portable binaries that pre-dated Java by more than 10 years.

For several years, he wrote corporate software for Mathematica Policy Research (they're still in business!) and Chase Econometrics (not so lucky) before being forcibly thrown into the consulting market. He bummed around the Philadelphia consulting and contract-programming scenes for a year or two before taking a job at NSTL (National Software Testing Labs) developing product tests and managing contract testing for the computer industry, governments and publication.

In 1991 Larry moved to Massachusetts to become Technical Director of PC Week Labs (now eWeek Labs). He moved within Ziff Davis to New York in 1994 to run testing at Windows Sources. In 1995, he became Technical Director for Internet product testing at PC Magazine and stayed there till 1998.

Since then, he has been writing for numerous other publications, including Fortune Small Business, Windows 2000 Magazine (now Windows and .NET Magazine), ZDNet and Sam Whitmore's Media Survey.

Submit a Comment

Loading Comments...
Manage your Newsletters: Login   Register My Newsletters

Rocket Fuel