The Limits of Scanning

By Larry Seltzer  |  Posted 2006-10-02 Print this article Print

Opinion: A large test shows that the flood of malware is outstripping the capacity of most detection products to keep up.

Has the malware problem gotten out of control? An aggressive test of anti-virus products indicates that it has, at least by some measures. Its a tough call.

I worry less and less personally about malware, even though Im barraged by it day and night. Ive got a gateway security device that scans for it with a Kaspersky scanner and my mail server runs Sunbelt Softwares Ninja, which uses both Authentium and Bitdefender to scan everything coming through, *and* I have desktop anti-virus on almost all my systems. Ive got my belt and suspenders and my pants are nailed to my gut.

And its a good thing Ive got all this protection, because tests by independent test group AV-Test paint a dark picture of the detection capabilities of most products.

For many years there has been a standard of sorts for testing anti-virus products called the "WildList." The problem with the WildList is that its relatively small and contains only certain types of malware, and everyone knows its contents. As Andreas Marx of AV-Test puts it, "the WildList is not reflecting todays threats, but more or less historical threats only (e.g. which malware was widespread two months ago?)." So its not surprising that (according to AV-Test) most scanners can detect 100 percent of it. What you should worry about is the huge number of other threats out there.

Appliances from Ironport and Secure Computing close the malware-fix gap. Click here to read more.

AV-Test ran a huge test of backdoors (59,053), bots (70,658) and trojan horses (159,971) for a total of 289,682 malware samples. They ran them through 33 products. We have separate numbers on the bots, backdoors and trojans, but check out the table, below, for the ranked results of overall detection percentage.

How do the numbers look? If you ask me, not good. Five vendors scored over 99 percent, which has to be considered excellent in so large a test sample. Another six scored over 95 percent. Another seven were over 90 percent, and this is approximately the median, which was 90.42 percent. Half the products did worse than this; 10 were under 75 percent and four were under 50 percent. Thats pretty bad.

Several of the best products, like my own mail security product, use multiple engines. The No. 1 product, WebWasher by Secure Computing, for example, detected 99.97 percent or all but 87 out of the 289,682 samples. It uses the AntiVir engine (the No. 2 product) in combination with an engine Secure Computing developed on its own. Not all products that use multiple engines score better as a result, as they may configure those engines less aggressively.

Next page: Analysis.

Larry Seltzer has been writing software for and English about computers ever since—,much to his own amazement—,he graduated from the University of Pennsylvania in 1983.

He was one of the authors of NPL and NPL-R, fourth-generation languages for microcomputers by the now-defunct DeskTop Software Corporation. (Larry is sad to find absolutely no hits on any of these +products on Google.) His work at Desktop Software included programming the UCSD p-System, a virtual machine-based operating system with portable binaries that pre-dated Java by more than 10 years.

For several years, he wrote corporate software for Mathematica Policy Research (they're still in business!) and Chase Econometrics (not so lucky) before being forcibly thrown into the consulting market. He bummed around the Philadelphia consulting and contract-programming scenes for a year or two before taking a job at NSTL (National Software Testing Labs) developing product tests and managing contract testing for the computer industry, governments and publication.

In 1991 Larry moved to Massachusetts to become Technical Director of PC Week Labs (now eWeek Labs). He moved within Ziff Davis to New York in 1994 to run testing at Windows Sources. In 1995, he became Technical Director for Internet product testing at PC Magazine and stayed there till 1998.

Since then, he has been writing for numerous other publications, including Fortune Small Business, Windows 2000 Magazine (now Windows and .NET Magazine), ZDNet and Sam Whitmore's Media Survey.

Submit a Comment

Loading Comments...
Manage your Newsletters: Login   Register My Newsletters

Rocket Fuel