Comparing Ratings Systems

 
 
By Larry Seltzer  |  Posted 2008-10-14 Print this article Print
 
 
 
 
 
 
 


 

The Jeff Jones article I mentioned earlier does a good job of comparing vendor ratings systems. Jones shows, for example, that Red Hat's severity ratings are quite similar to Microsoft's. That doesn't mean that Red Hat applies them the same way as Microsoft, although Red Hat previously complained about a Jones analysis, using NVD ratings, showing that they had a high percentage of "High" vulnerabilities. Because it's sort of a base line, Jones likes using the NVD ratings; in this blog, Jones shows that OS X, Red Hat and Ubuntu had many more and more severe vulnerabilities than Windows XP or Vista (in the first quarter of 2008). He makes a similar point about IE and Firefox in this blog.

Even though he uses it to make a point, Jones says he doesn't like the NVD/CVSS ratings system. Because of how the scoring works he thinks that it doesn't necessarily give what should be the higher-priority issues higher scores.

The other major problem with severity ratings that cause them to be overstated is when multiple platforms are affected, to different degrees, in an advisory. Many vendors, including outside parties such as Secunia, apply an overall severity rating to an advisory, which is usually the worst-case severity in the advisory. But depending on your architecture or other specifics, that severity may not apply. Microsoft is commonly guilty of this; a vulnerability which affects you may, for instance, be critical on Windows 2000, but far less severe on Windows XP or Windows Server 2003, and yet the overall advisory says "critical."

Take this Secunia advisory for the recent Apple vulnerability disclosure: It has 39 CVEs in it, but one overall rating of "Moderately Critical," which Secunia defines as:

Moderately Critical (3 of 5)
Typically used for remotely exploitable denial of service vulnerabilities against services like FTP, HTTP, and SMTP, and for vulnerabilities that allow system compromises but require user interaction.

This rating is also used for vulnerabilities allowing system compromise on LANs in services like SMB, RPC, NFS, LPD and similar services that are not intended for use over the Internet.

Pretty broad definition there. In fairness to Secunia, you can drill down on many-but not all-of the individual vulnerabilities and get more granular severity ratings. It can be hard to pick out these confusions even when you use individual CVE numbers. In this article, Red Hat says:

Lots of companies ship Apache in their products, but all ship different versions with different defaults on different operating systems for different architecture compiled with different compilers using different compiler options. Many Apache vulnerabilities over the years have affected different platforms in significantly different ways. We've seen an Apache vulnerability that leads to arbitrary code execution on older FreeBSD, that causes a denial of service on Windows, but that was unexploitable on Linux for example. But this flaw had a single CVE identifier.
It's easy to see administrators being confused about this, especially if they don't dig down into the details, and how many people do that? Everyone wants to provide a big summary severity rating, even the NVD who at least provides granular details behind them, because they believe that the consumers of this information want such ratings. Microsoft also provides some level of detail-not as much as the NVD-to let you determine what your specific exposure is, but the overall ratings loom over the whole process. For home users applying automatic updates, the automatic application of critical updates makes this a very real issue.

The best outcome would be for users to dig into the details, but that isn't going to happen. Since any attempt to make the data more accessible necessarily involves simplification and value judgements, there's likely no way to avoid the problems I've been discussing. In fact, the only vendor I'm not sympathetic for is Apple, since they choose to chicken out of the whole issue, and they don't even provide details of their own vulnerabilities. The problem as a whole will continue to plague us; it's another example of how security is complicated and will remain so.

Security Center Editor Larry Seltzer has worked in and written about the computer industry since 1983.

For insights on security coverage around the Web, take a look at eWEEK.com Security Center Editor Larry Seltzer's blog Cheap Hack



 
 
 
 
Larry Seltzer has been writing software for and English about computers ever since—,much to his own amazement—,he graduated from the University of Pennsylvania in 1983.

He was one of the authors of NPL and NPL-R, fourth-generation languages for microcomputers by the now-defunct DeskTop Software Corporation. (Larry is sad to find absolutely no hits on any of these +products on Google.) His work at Desktop Software included programming the UCSD p-System, a virtual machine-based operating system with portable binaries that pre-dated Java by more than 10 years.

For several years, he wrote corporate software for Mathematica Policy Research (they're still in business!) and Chase Econometrics (not so lucky) before being forcibly thrown into the consulting market. He bummed around the Philadelphia consulting and contract-programming scenes for a year or two before taking a job at NSTL (National Software Testing Labs) developing product tests and managing contract testing for the computer industry, governments and publication.

In 1991 Larry moved to Massachusetts to become Technical Director of PC Week Labs (now eWeek Labs). He moved within Ziff Davis to New York in 1994 to run testing at Windows Sources. In 1995, he became Technical Director for Internet product testing at PC Magazine and stayed there till 1998.

Since then, he has been writing for numerous other publications, including Fortune Small Business, Windows 2000 Magazine (now Windows and .NET Magazine), ZDNet and Sam Whitmore's Media Survey.
 
 
 
 
 
 
 

Submit a Comment

Loading Comments...
 
Manage your Newsletters: Login   Register My Newsletters























 
 
 
 
 
 
 
 
 
 
 
Rocket Fuel