Security Software Reviews Done Wrong

By Larry Seltzer  |  Posted 2008-08-16 Print this article Print

Updated: Consumer Reports review State of the Net survey was supposed be a guide to Protect yourself online, from the biggest threats to the best solutions. Consumer Reports examined spyware, firewalls, anti-virus, Web security, phishing and identity theft, but its methods proved troublesome and vendors such as Symantec are up in arms. Security consumers should be, too. Bad information on security software coming from such a trusted source in security software reviews is doubly bad.

I don't do a lot of reviews anymore, but I spent about 13 years in the reviews business, testing a wide variety of products. A badly-done, badly-thought out review hits me like fingernails on the blackboard. So it is with the recent Consumer Reports story on computer security and accompanying review of computer security software (the full story is available only to subscribers).

[Editor's Note: Consumer Reports responded to this story after it was published. Please see the end of the story for the full text of their response.]

It's normally bad form to dump on another publication's work in the same field, but this particular story really bothers me. I knew about it before and I must admit that I was spurred into writing this story by a blog posting by Symantec's David Cole complaining about the review. I've had my share of bad experiences with Symantec products so I'm not inclined to give them free PR, but Cole's points are quite valid.

Conveying security information to lay people is a tricky business, much harder than with most other technology issues. Those lay people will be inclined to trust Consumer Reports, which has a sterling reputation, whether they deserve it or not. Bad information coming from such a trusted source thus becomes doubly bad and end up making things hard for everyone, even those of you in IT, as those lay people bring their false impressions with them to work.

To the review: The first, most ridiculous problem with the review is timing. This is in the September issue of CR which necessarily comes out in early August, and for which the testing was probably finished by early July, probably even earlier than that. Because of this schedule, CR reports on the 2008 editions of the security suites. But the new versions of the software suites come out in the fall in or around September. I'm scheduled to talk to two vendors this week about their impending 2009 editions. And since the entire industry has moved to a subscription model testing old versions makes even less sense. On this same subject, the ratings page in the review includes one last finger in the reader's eye as the "free suite" CR builds for comparison to the pay suites includes Avira Personal Edition Classic 7 which, a footnote adds, is "Discontinued; replaced by Free Antivirus 8, which claims enhancements." For these reasons alone, the review is essentially useless out of the gate.

Larry Seltzer has been writing software for and English about computers ever since—,much to his own amazement—,he graduated from the University of Pennsylvania in 1983.

He was one of the authors of NPL and NPL-R, fourth-generation languages for microcomputers by the now-defunct DeskTop Software Corporation. (Larry is sad to find absolutely no hits on any of these +products on Google.) His work at Desktop Software included programming the UCSD p-System, a virtual machine-based operating system with portable binaries that pre-dated Java by more than 10 years.

For several years, he wrote corporate software for Mathematica Policy Research (they're still in business!) and Chase Econometrics (not so lucky) before being forcibly thrown into the consulting market. He bummed around the Philadelphia consulting and contract-programming scenes for a year or two before taking a job at NSTL (National Software Testing Labs) developing product tests and managing contract testing for the computer industry, governments and publication.

In 1991 Larry moved to Massachusetts to become Technical Director of PC Week Labs (now eWeek Labs). He moved within Ziff Davis to New York in 1994 to run testing at Windows Sources. In 1995, he became Technical Director for Internet product testing at PC Magazine and stayed there till 1998.

Since then, he has been writing for numerous other publications, including Fortune Small Business, Windows 2000 Magazine (now Windows and .NET Magazine), ZDNet and Sam Whitmore's Media Survey.

Submit a Comment

Loading Comments...
Manage your Newsletters: Login   Register My Newsletters

Rocket Fuel