Ethics and Virus Testing

By Larry Seltzer  |  Posted 2006-08-20 Print this article Print

Opinion: How come discovering vulnerabilities and writing exploits is "research," but viruses for testing is a crime against humanity?

The anti-virus community is abuzz in controversy over the tests performed recently by Consumer Reports on anti-virus products. CR went out and did what many of us have considered in the past, but not actually done: With the help of consultants at ISE (Independent Security Evaluators), they created a test bed of 5,500 new viruses in order to test the products.

Theres an old joke about Consumer Reports, that nobody respects their work for their own field, just for others. So a carpenter will scoff at their review of circular saws, but trust them for gas grills and washing machines. Ive heard a lot of this in the discussions about virus testing.

Symantecs veteran virus-hunter Vincent "Vinny" Gullotto recently joined Microsoft to head its Security Research and Response team. Click here to read more.
Many in the anti-malware community are adamant that creating viruses is always a bad thing, and never necessary in order to test anti-virus software. In fact, they argue that its not as good a methodology as the alternatives. You can find some good links to opinion on the matter in this blog entry by Sunbelt Softwares Alex Eckelberry.

Ive been involved in many tests of anti-virus products, and its always tough. There are many ways you can go about the testing and they all have their strengths and weaknesses. The biggest problem is testing of heuristic protection, or protection against unknown viruses.

I have no specific opinion on the work by Consumer Reports; not being a subscriber I havent read the actual test results, just the methodology linked to above. But it seems to me that the abhorrence of virus creation that many are expressing is an overreaction.

Lets take what seem to me to be the two main arguments against it: 1) If you create malware, theres a chance it could escape and cause damage to innocent third parties, and 2) its not a good way to test AV.

Yes, theres a chance that malware could be released if youre not very careful. All kinds of bad things can happen if youre not careful. If you misconfigure your servers you could set them up to be open relays or bots to be used to attack others. If you write software badly, you could open up the computers of anyone who writes it to attack. I could go on, but the possibility of releasing test malware doesnt seem like an imminent threat, especially since theres no infamous history of such releases.

But what really strikes me in this regard is how it compares to exploit testing; vulnerability research and exploit development are somehow looked upon as honorable and a service to the industry, although there is some disagreement over whether researchers should quietly keep vendors informed. Anyone who follows this business knows that far more damage has been done to innocent third parties by vulnerability and exploit developers than by malware research. How come everyones so conservative now when it comes to malware?

But is it the best way to test? One person suggested that the better way to do testing of heuristics is to freeze copies of the anti-virus products without updates for some period of time, then apply the malware that came out since the last update. I actually ran some tests like this for PC Magazine once, and trust me, nobody will be happy with these results either, although I do have to confess we didnt freeze them long enough or have enough malware at the end.

And even if you do it well, all this tells you is how well a product protected against viruses back at the point at which you froze it. So the longer you freeze, making your test more accurate by giving it a bigger sample, the less accurate you make it by divorcing the results from the actual capabilities of the product at the time you report.

I might very well disagree with the test methodology and analysis in the Consumer Reports review, but the fact that they created viruses in order to do it is no reason to doubt the testing. Whether they create their own viruses or use existing ones they need to be careful in the handling of those viruses. Theres no ethical slippery slope here, theres just an attempt to test products aggressively, and thats something to applaud.

Security Center Editor Larry Seltzer has worked in and written about the computer industry since 1983. Check out eWEEK.coms for the latest security news, reviews and analysis. And for insights on security coverage around the Web, take a look at Security Center Editor Larry Seltzers Weblog.
Larry Seltzer has been writing software for and English about computers ever since—,much to his own amazement—,he graduated from the University of Pennsylvania in 1983.

He was one of the authors of NPL and NPL-R, fourth-generation languages for microcomputers by the now-defunct DeskTop Software Corporation. (Larry is sad to find absolutely no hits on any of these +products on Google.) His work at Desktop Software included programming the UCSD p-System, a virtual machine-based operating system with portable binaries that pre-dated Java by more than 10 years.

For several years, he wrote corporate software for Mathematica Policy Research (they're still in business!) and Chase Econometrics (not so lucky) before being forcibly thrown into the consulting market. He bummed around the Philadelphia consulting and contract-programming scenes for a year or two before taking a job at NSTL (National Software Testing Labs) developing product tests and managing contract testing for the computer industry, governments and publication.

In 1991 Larry moved to Massachusetts to become Technical Director of PC Week Labs (now eWeek Labs). He moved within Ziff Davis to New York in 1994 to run testing at Windows Sources. In 1995, he became Technical Director for Internet product testing at PC Magazine and stayed there till 1998.

Since then, he has been writing for numerous other publications, including Fortune Small Business, Windows 2000 Magazine (now Windows and .NET Magazine), ZDNet and Sam Whitmore's Media Survey.

Submit a Comment

Loading Comments...
Manage your Newsletters: Login   Register My Newsletters

Rocket Fuel