eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
The anti-virus community is abuzz in controversy over the tests performed recently by Consumer Reports on anti-virus products.
CR went out and did what many of us have considered in the past, but not actually done: With the help of consultants at ISE (Independent Security Evaluators), they created a test bed of 5,500 new viruses in order to test the products.
Theres an old joke about Consumer Reports, that nobody respects their work for their own field, just for others. So a carpenter will scoff at their review of circular saws, but trust them for gas grills and washing machines. Ive heard a lot of this in the discussions about virus testing.
Many in the anti-malware community are adamant that creating viruses is always a bad thing, and never necessary in order to test anti-virus software. In fact, they argue that its not as good a methodology as the alternatives. You can find some good links to opinion on the matter in this blog entry by Sunbelt Softwares Alex Eckelberry.
Ive been involved in many tests of anti-virus products, and its always tough. There are many ways you can go about the testing and they all have their strengths and weaknesses. The biggest problem is testing of heuristic protection, or protection against unknown viruses.
I have no specific opinion on the work by Consumer Reports; not being a subscriber I havent read the actual test results, just the methodology linked to above. But it seems to me that the abhorrence of virus creation that many are expressing is an overreaction.
Lets take what seem to me to be the two main arguments against it: 1) If you create malware, theres a chance it could escape and cause damage to innocent third parties, and 2) its not a good way to test AV.
Yes, theres a chance that malware could be released if youre not very careful. All kinds of bad things can happen if youre not careful.
If you misconfigure your servers you could set them up to be open relays or bots to be used to attack others.
If you write software badly, you could open up the computers of anyone who writes it to attack. I could go on, but the possibility of releasing test malware doesnt seem like an imminent threat, especially since theres no infamous history of such releases.
But what really strikes me in this regard is how it compares to exploit testing; vulnerability research and exploit development are somehow looked upon as honorable and a service to the industry, although there is some disagreement over whether researchers should quietly keep vendors informed.
Anyone who follows this business knows that far more damage has been done to innocent third parties by vulnerability and exploit developers than by malware research. How come everyones so conservative now when it comes to malware?
But is it the best way to test? One person suggested that the better way to do testing of heuristics is to freeze copies of the anti-virus products without updates for some period of time, then apply the malware that came out since the last update.
I actually ran some tests like this for PC Magazine once, and trust me, nobody will be happy with these results either, although I do have to confess we didnt freeze them long enough or have enough malware at the end.
And even if you do it well, all this tells you is how well a product protected against viruses back at the point at which you froze it.
So the longer you freeze, making your test more accurate by giving it a bigger sample, the less accurate you make it by divorcing the results from the actual capabilities of the product at the time you report.
I might very well disagree with the test methodology and analysis in the Consumer Reports review, but the fact that they created viruses in order to do it is no reason to doubt the testing.
Whether they create their own viruses or use existing ones they need to be careful in the handling of those viruses. Theres no ethical slippery slope here, theres just an attempt to test products aggressively, and thats something to applaud.
Security Center Editor Larry Seltzer has worked in and written about the computer industry since 1983.