Examining AMTSOs Principles

By Larry Seltzer  |  Posted 2008-11-14 Print this article Print

Principle No. 2, about bias, is in many ways the most impressive. It recognizes that tests may often be performed on contract for a vendor and that such testing can still be unbiased. It sets rules for disclosure of such relationships and potential conflicts:

[T]his disclosure should include any relationship that could potentially influence the tester, including: (i) whether the publication or tester has received revenue from a vendor or affiliate with regard to any particular test, and (ii) whether the publication or tester receives a significant portion of its overall revenue from a particular vendor.

I've always believed that this is the right way to do things and that people are able to consider such conflicts in judging the data.

No. 3 recognizes that labs may not want to release all details of methodology, but it requires certain important points to be published with results, such as configuration settings for products tested, full configurations of test systems and how the samples were obtained. But it doesn't require, for example, source code for test harness systems.

No. 4 is about looking at a variety of factors in drawing conclusions on a product. So a review that talks about detection percentage without discussing false positives, for example, might run afoul of this principle. I have to say this is one of the more subjective of the principles.

No. 5 says that you shouldn't just trust the judgments of the products being tested. You should confirm that samples detected as malware are in fact malware, and not false positives, or vice versa. This can be difficult.

No. 6, the consistency principle, refers to using the right products for the audience, so you shouldn't, for example, test corporate gateway products for a consumer audience, or consumer solutions for an enterprise audience.

No. 7 should be obvious: Don't assert conclusions which disagree with your data. When this happens it's usually a sign that the author is disappointed with the data for not showing what he wanted it to show. It helps to have a rule.

No. 8, about statistical significance, is one that could do with some more specifics, but perhaps that wasn't possible at this point. If we should demand statistical significance in the results, we should provide a numerical standard for it. Still, there are plenty of tests (I'll admit it, I've been involved in some) where we didn't have enough samples and went ahead anyway. Once again, it helps to have a rule.

No. 9, the active contact point rule, is a really good one for all involved. Perhaps it could have gone on to give a minimum response window.

Larry Seltzer has been writing software for and English about computers ever since—,much to his own amazement—,he graduated from the University of Pennsylvania in 1983.

He was one of the authors of NPL and NPL-R, fourth-generation languages for microcomputers by the now-defunct DeskTop Software Corporation. (Larry is sad to find absolutely no hits on any of these +products on Google.) His work at Desktop Software included programming the UCSD p-System, a virtual machine-based operating system with portable binaries that pre-dated Java by more than 10 years.

For several years, he wrote corporate software for Mathematica Policy Research (they're still in business!) and Chase Econometrics (not so lucky) before being forcibly thrown into the consulting market. He bummed around the Philadelphia consulting and contract-programming scenes for a year or two before taking a job at NSTL (National Software Testing Labs) developing product tests and managing contract testing for the computer industry, governments and publication.

In 1991 Larry moved to Massachusetts to become Technical Director of PC Week Labs (now eWeek Labs). He moved within Ziff Davis to New York in 1994 to run testing at Windows Sources. In 1995, he became Technical Director for Internet product testing at PC Magazine and stayed there till 1998.

Since then, he has been writing for numerous other publications, including Fortune Small Business, Windows 2000 Magazine (now Windows and .NET Magazine), ZDNet and Sam Whitmore's Media Survey.

Submit a Comment

Loading Comments...
Manage your Newsletters: Login   Register My Newsletters

Rocket Fuel