How Intolerable Are False Positives In Spam Blocking?

By Larry Seltzer  |  Posted 2003-08-27 Print this article Print

Adamant in the past that false positives are totally unacceptable to a spam blocker, Security Supersite Editor Larry Seltzer has had a change of heart. Now he seeks a more subtle approach when calculating the trade-off.

As I work on a couple of upcoming reviews of anti-spam products for PC Magazine Ive been struggling to define my own standards for anti-spam products. In my previous reviews of such products Ive always stated not only that the number of false positives was the most important measure of the product, but that the products should be held to a standard of near-perfection.

Heres my reasoning: If you assume that the product will generate a non-trivial number of false positives—by which I mean legitimate, non-spam messages mistakenly classified as spam by the product—then you must check the products blocked mail folder periodically in order to separate out the real mail in there. Ive always thought that if you have to look through all the spam anyway, you may as well look at it in your inbox. This is the reason that after all the reviews Ive written, I still dont run an anti-spam product myself.

However, I think Ive finally lost my patience with this approach. Things have gotten so bad lately that Im willing to take some risks. I havent changed my opinion that false positives are the most important measure of an anti-spam product and that all anti-spam products will generate some of them. At the same time, if you have to check the blocked mail folder, you may as well not run the product. Thats why I think I wont bother checking the blocked mail.

Some time soon, Ill start running one of these programs and Ill just put my faith in it. Thats the answer: you really have to have faith in order to run an anti-spam product sensibly. Im going to assume that some amount of legitimate mail, perhaps from you, dear readers, will not make it through. Perhaps its a mark of the times that this situation doesnt bother me in the same way it used to. Heres my reasoning (or rationalization): Even if I dont run an anti-spam product some amount of real mail will not make it through. Im sure Ive mistakenly deleted many a real message in the process of manually deleting spam. In addition, not all false positives are created equal—many legitimate messages will not be missed. Some of them are commercial solicitations to which I once subscribed. Some of them are jokes sent by friends and relatives. Some may be real product announcements from vendors or public relations firms; this is a big part of the risk Ill run.

When I talk to people about the anti-spam products theyre running and many of them say they are very happy with them. Often Ill hear them say, simultaneously, that the product generates no false positives and that they dont have to bother checking the blocked mail folder. But make no mistake about it: If youre not checking the blocked mail folder you dont know how many false positives the product is generating. How could you? This is also at the heart of what makes anti-spam products so difficult to test. In an upcoming column Ill discuss some of the issues that it the most challenging category of software to test.

Again, it all comes down to a matter of faith. If youre not going to double-check the results of your anti-spam software youre putting yourself at the mercy of the product. By the same token, you have to be suspicious of vendor claims for the accuracy of their products. Spam blocking is becoming a mandatory software task, but its still a problematic one.

Security Supersite Editor Larry Seltzer has worked in and written about the computer industry since 1983.

More from Larry Seltzer
Larry Seltzer has been writing software for and English about computers ever since—,much to his own amazement—,he graduated from the University of Pennsylvania in 1983.

He was one of the authors of NPL and NPL-R, fourth-generation languages for microcomputers by the now-defunct DeskTop Software Corporation. (Larry is sad to find absolutely no hits on any of these +products on Google.) His work at Desktop Software included programming the UCSD p-System, a virtual machine-based operating system with portable binaries that pre-dated Java by more than 10 years.

For several years, he wrote corporate software for Mathematica Policy Research (they're still in business!) and Chase Econometrics (not so lucky) before being forcibly thrown into the consulting market. He bummed around the Philadelphia consulting and contract-programming scenes for a year or two before taking a job at NSTL (National Software Testing Labs) developing product tests and managing contract testing for the computer industry, governments and publication.

In 1991 Larry moved to Massachusetts to become Technical Director of PC Week Labs (now eWeek Labs). He moved within Ziff Davis to New York in 1994 to run testing at Windows Sources. In 1995, he became Technical Director for Internet product testing at PC Magazine and stayed there till 1998.

Since then, he has been writing for numerous other publications, including Fortune Small Business, Windows 2000 Magazine (now Windows and .NET Magazine), ZDNet and Sam Whitmore's Media Survey.

Submit a Comment

Loading Comments...
Manage your Newsletters: Login   Register My Newsletters

Rocket Fuel