Three Vulnerability Assessment Tools Put to the Test

eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.

Vulnerability assessment tools are rapidly evolving, and eWEEK Labs tests of three such products show that IT managers who leverage them will stay further ahead of potential security problems than they would by manual means.

Vulnerability assessment systems scan operating systems and applications for potential problems, such as the use of default passwords or configurations and open ports. This can give administrators a head start in fixing problems and will, hopefully, let IT organizations more effectively beat bad guys to the punch.

Thats assuming, of course, vulnerability systems catch every problem in every application. Our tests showed a gap between even the best vulnerability assessment tool and the weaknesses in our test network, but IT staffers charged with securing IT assets will benefit from using a vulnerability assessment tool, if only by eliminating much of the routine drudgery they face.

For this report, we tested three vulnerability assessment tools that represent the variety of offerings available: Harris Corp.s STAT (Security Threat Avoidance Technology) Scanner Professional Edition 5.14, Internet Security Systems Inc.s Internet Scanner 7.0 and Qualys Inc.s QualysGuard Enterprise Intranet Scanner service.

During tests, each vulnerability assessment tool found different problems when looking at the same machines in our test network. However, all found the major problems we expected them to find.

The QualysGuard Enterprise Intranet Scanner service consistently identified the most potential exposures and identified the most serious problems, earning it our eWEEK Labs Analysts Choice award. Harris STAT Scanner provided fine-grain control over how systems were scanned, while ISS Internet Scanner can be combined with the companys Server, Database and Wireless scanners to analyze the gamut of products found in the enterprise environment.

Diversity of test systems was the name of the game in our vulnerability assessment trials. We used a variety of Microsoft Corp. desktop and server operating systems, including Windows XP, 2000 and Server 2003, along with Novell Inc.s NetWare, a variety of Linux operating systems from Red Hat Inc. and Sun Microsystems Inc.s Solaris.

We also used the assessment products to scan network equipment, including switches from Cisco Systems Inc. and Extreme Networks Inc., as well as firewalls from WatchGuard Technologies Inc., Nokia Inc. and Cisco. We layered a variety of common business productivity tools and applications, such as instant messaging clients, to create a primordial soup of potential security exposures.

We ran each product in three heats, adding machines to the network after each heat. We did this to see how effectively new devices were discovered and how neatly they were pointed out in reports we ran from each product. We were pleasantly surprised at how efficiently the products spotted new devices and provided reports that made it easy to see the new arrivals. This is important because discovering new assets is a critical part of managing a secure network.

To test the effect of bandwidth contention on the vulnerability assessment tools, we generated a moderate amount of network traffic with an Ixia Communications 1600 Traffic Generator. The tests revealed a big difference in the ability of the products to scan our small test network, with ISS Internet Scanner suffering terribly when the network was bogged down.

After running the initial three heats, we made significant changes to the target systems by applying numerous patches and either disabling services such as FTP or simply turning off products such as Microsofts Internet Information Services. We then ran the vulnerability assessment tools to see how well they picked up on our changes and reported the newly changed state of test targets.

STAT Scanner and Internet Scanner are priced similarly, at roughly $100 per managed device or IP address. The QualysGuard Enterprise service figures out to about $320 per scanned IP address. QualysGuard Enterprise incurs almost none of the installation and training costs associated with STAT Scanner and Internet Scanner, but IT managers should still consider the higher cost of QualysGuard Enterprise a disadvantage.

Stat Scanner

Stat Scanner

Harris STAT Scanner Professional Edition Version 5.14 was most recently revised in June 2002 and costs $990 for a 10-node license. As with all the products reviewed here, volume discounts are available, and the prices quoted are published list prices.

STAT Scanner has a couple of “onlys” that made it stand out in the crowd: It was the only product in our tests that extended vulnerability testing to printers (specifically, Hewlett-Packard Co. printers)—a potential point of attack. STAT Scanner is also the only product we tested that was certified as compliant with Common Criteria, a tough international IT specification.

However, STAT Scanner stumbled when it came to recognizing several systems in our testbed. A NetWare 5.0 server, for example, was marked as an unknown operating system, and efforts to get the product to recognize NetWare proved futile.

STAT Scanner also had trouble correctly identifying Windows Server 2003-based systems. Although STAT Scanner did identify the Windows operating system, it used a Windows 2000 profile to scan for vulnerabilities. Thus, we got several false-positive vulnerability alerts, most pointing out that a variety of service packs had not been installed. Of course, Windows 2000 Server service packs are not applicable to Windows Server 2003, so the test systems were actually fine in that regard.

IT managers who are looking at vulnerability assessment tools need to put operating system and application coverage at the top of the evaluation chart. As stated earlier, much of the value of these products comes from their ability to free IT security staff from routine security scans. However, that wont happen if a tool misses a crucial operating system.

STAT Scanner can be scaled to incorporate multiple networks using STAT DVM (Distributed Vulnerability Management).

During tests, STAT Scanner did as good a job as any of the products we looked at in limiting the amount of network bandwidth used. We recommend that IT managers pay close attention to the bandwidth-throttling capabilities of vulnerability assessment tools because networks or target systems can be quickly overwhelmed by some of the scanning techniques they use. STAT Scanner did not interfere with any of our database, Web or mail servers and placed very little load on our network infrastructure.

ISS Internet Scanner

ISS Internet Scanner

ISS Internet Scanner 7.0, released in April, is a solid vulnerability assessment tool, but it worked more slowly than any of the other tools in our tests. The product does integrate with a wide range of security management tools, which is an important consideration.

But when we say ISS Internet Scanner ran slowly, we really mean slow: One scan of 16 nodes in our testbed took more than 20 minutes. In contrast, QualysGuard Enterprise did the same scan, looking for almost three times as many vulnerabilities, in just a few minutes. Both systems were running on a network with a low utilization rate.

In fact, when we used the Ixia 1600 Traffic Generator to lay down a base-line load of 25 percent bandwidth utilization on our network, all the products suffered heavy performance drops, but none as bad as ISS Internet Scanner. These controlled, repeatable tests revealed performance gaps that will be less obvious in a production network, but will be a consideration, nonetheless.

ISS Internet Scanner ably identified most of the machines in our tests, with the exception of misidentifying Windows Server 2003 systems as Windows XP systems. The scan reports did not generate any significant false positives.

Based on our review of several reports generated by ISS Internet Scanner 7.0 and the explanatory information provided by the product, we think most IT organizations will get immediate assistance by using the product. Internet Scanner 7.0 is also backed up by ISS X-Force, a comic-book-sounding name for a group of security experts who research vulnerabilities and assess threats and potential remedial actions.

We looked at the scalability of all the products and were impressed with ISS SiteProtector 2.0 as a way to manage Internet Scanner along with other security tools from ISS, including the RealSecure Network, RealSecure Server and Proventia appliances. Although the combination of these intrusion detection and attack-stopping tools was impressive, we hope that ISS will integrate into one system its Server, Database and Wireless scanners—tools that complement Internet Scanner and will allow IT organizations to more effectively scan enterprise systems.

Internet Scanner 7.0 and QualysGuard Enterprise will identify as many potential targets in the network as possible, even if the product license does not support the number of targets found. This is an emerging trend that will be a real benefit to IT departments, especially those involved in a merger with another company or undergoing internal consolidation.

Internet Scanner 7.0 starts at $1,223 for 10 IP addresses.

Qualysguard Enterprise

Qualysguard Enterprise

Qualysguard Enterprise gained Analysts Choice recognition for its ability to regularly identify the most important vulnerabilities across the widest range of operating systems, applications and infrastructure devices of any of the products we tested. The service costs $19,995 for 64 devices.

During tests, it correctly identified our Solaris server, which was running on a SunFire 280R box, our NetWare 5.0 server and various Red Hat Linux versions running on our MPC LLC (formerly MicronPC) Millennia hardware. The product was a bit sketchy, however, in its coverage of Windows, lumping all our Windows 2000 Server, Windows Server 2003 and Windows XP systems into the same category on its network maps.

More important, QualysGuard Enterprise correctly profiled the systems and provided us with accurate reports that provided explanations of the problems and suggestions for making fixes.

The biggest hurdle most IT managers will face when using QualysGuard Enterprise is trust. Unlike the other products in our roundup, QualysGuard is provided only as a service. QualysGuard Enterprise requires that a small appliance be installed on the network, but all the report processing and mapping happens at Qualys. (ISS and Foundstone Inc. also provide service offerings.)

However, because QualysGuard Enterprise is a service, we never had to worry about keeping our vulnerability definitions up-to-date.

During tests, eWEEK Labs was impressed with the freshness of Qualys vulnerability watch list. New vulnerabilities were clearly explained, and we appreciated the fact that, in most cases, Qualys went beyond simply republishing the manufacturers usually terse break/fix notes. This was a big difference between QualysGuard Enterprise and the other products tested.

IT managers who are evaluating vulnerability assessment tools should ask for samples of recent threat updates to judge the tools usefulness. For example, while all the products in our test identified systems that were susceptible to the SQL Slammer worm, only QualysGuard Enterprise flagged the problem with a clear, attention-getting flag. This seemingly simple function could be the difference between busy IT staffers fixing the most vulnerable systems and not.

As this issue was going to press, Qualys announced a new remediation service. There is a good argument that vulnerability assessment without remediation is a job half-done. While we agree with the sentiment, our recent work with patch management systems indicates that trying to cobble together both kinds of products would result in a complex, probably unmanageable system.

We recommend IT managers separate assessment and remediation functions. Qualys currently partners with Citadel Security Software Inc.s Hercules remediation tool, as does STAT. For now, we think this is the best way to get the job done.

Senior Analyst Cameron Sturdevant can be reached at cameron_sturdevant@ ziffdavis.com.