Security Software Reviews Done Wrong

I don't do a lot of reviews anymore, but I spent about 13 years in the reviews business, testing a wide variety of products. A badly-done, badly-thought out review hits me like fingernails on the blackboard. So it is with the recent Consumer Reports story on computer security and accompanying review of computer security software (the full story is available only to subscribers).

[Editor's Note: Consumer Reports responded to this story after it was published. Please see the end of the story for the full text of their response.]

It's normally bad form to dump on another publication's work in the same field, but this particular story really bothers me. I knew about it before and I must admit that I was spurred into writing this story by a blog posting by Symantec's David Cole complaining about the review. I've had my share of bad experiences with Symantec products so I'm not inclined to give them free PR, but Cole's points are quite valid.

Conveying security information to lay people is a tricky business, much harder than with most other technology issues. Those lay people will be inclined to trust Consumer Reports, which has a sterling reputation, whether they deserve it or not. Bad information coming from such a trusted source thus becomes doubly bad and end up making things hard for everyone, even those of you in IT, as those lay people bring their false impressions with them to work.

To the review: The first, most ridiculous problem with the review is timing. This is in the September issue of CR which necessarily comes out in early August, and for which the testing was probably finished by early July, probably even earlier than that. Because of this schedule, CR reports on the 2008 editions of the security suites. But the new versions of the software suites come out in the fall in or around September. I'm scheduled to talk to two vendors this week about their impending 2009 editions. And since the entire industry has moved to a subscription model testing old versions makes even less sense. On this same subject, the ratings page in the review includes one last finger in the reader's eye as the "free suite" CR builds for comparison to the pay suites includes Avira Personal Edition Classic 7 which, a footnote adds, is "Discontinued; replaced by Free Antivirus 8, which claims enhancements." For these reasons alone, the review is essentially useless out of the gate.