Three Vulnerability Assessment Tools Put to the Test

Vulnerability assessment products catch big stuff but vary in overall ability.

Vulnerability assessment tools are rapidly evolving, and eWEEK Labs tests of three such products show that IT managers who leverage them will stay further ahead of potential security problems than they would by manual means.

Vulnerability assessment systems scan operating systems and applications for potential problems, such as the use of default passwords or configurations and open ports. This can give administrators a head start in fixing problems and will, hopefully, let IT organizations more effectively beat bad guys to the punch.

Thats assuming, of course, vulnerability systems catch every problem in every application. Our tests showed a gap between even the best vulnerability assessment tool and the weaknesses in our test network, but IT staffers charged with securing IT assets will benefit from using a vulnerability assessment tool, if only by eliminating much of the routine drudgery they face.

For this report, we tested three vulnerability assessment tools that represent the variety of offerings available: Harris Corp.s STAT (Security Threat Avoidance Technology) Scanner Professional Edition 5.14, Internet Security Systems Inc.s Internet Scanner 7.0 and Qualys Inc.s QualysGuard Enterprise Intranet Scanner service.

During tests, each vulnerability assessment tool found different problems when looking at the same machines in our test network. However, all found the major problems we expected them to find.

The QualysGuard Enterprise Intranet Scanner service consistently identified the most potential exposures and identified the most serious problems, earning it our eWEEK Labs Analysts Choice award. Harris STAT Scanner provided fine-grain control over how systems were scanned, while ISS Internet Scanner can be combined with the companys Server, Database and Wireless scanners to analyze the gamut of products found in the enterprise environment.

Diversity of test systems was the name of the game in our vulnerability assessment trials. We used a variety of Microsoft Corp. desktop and server operating systems, including Windows XP, 2000 and Server 2003, along with Novell Inc.s NetWare, a variety of Linux operating systems from Red Hat Inc. and Sun Microsystems Inc.s Solaris.

We also used the assessment products to scan network equipment, including switches from Cisco Systems Inc. and Extreme Networks Inc., as well as firewalls from WatchGuard Technologies Inc., Nokia Inc. and Cisco. We layered a variety of common business productivity tools and applications, such as instant messaging clients, to create a primordial soup of potential security exposures.

We ran each product in three heats, adding machines to the network after each heat. We did this to see how effectively new devices were discovered and how neatly they were pointed out in reports we ran from each product. We were pleasantly surprised at how efficiently the products spotted new devices and provided reports that made it easy to see the new arrivals. This is important because discovering new assets is a critical part of managing a secure network.

To test the effect of bandwidth contention on the vulnerability assessment tools, we generated a moderate amount of network traffic with an Ixia Communications 1600 Traffic Generator. The tests revealed a big difference in the ability of the products to scan our small test network, with ISS Internet Scanner suffering terribly when the network was bogged down.

After running the initial three heats, we made significant changes to the target systems by applying numerous patches and either disabling services such as FTP or simply turning off products such as Microsofts Internet Information Services. We then ran the vulnerability assessment tools to see how well they picked up on our changes and reported the newly changed state of test targets.

STAT Scanner and Internet Scanner are priced similarly, at roughly $100 per managed device or IP address. The QualysGuard Enterprise service figures out to about $320 per scanned IP address. QualysGuard Enterprise incurs almost none of the installation and training costs associated with STAT Scanner and Internet Scanner, but IT managers should still consider the higher cost of QualysGuard Enterprise a disadvantage.