Examining AMTSOs Principles
Principle No. 2, about bias, is in many ways the most impressive. It recognizes that tests may often be performed on contract for a vendor and that such testing can still be unbiased. It sets rules for disclosure of such relationships and potential conflicts:No. 3 recognizes that labs may not want to release all details of methodology, but it requires certain important points to be published with results, such as configuration settings for products tested, full configurations of test systems and how the samples were obtained. But it doesn't require, for example, source code for test harness systems. No. 4 is about looking at a variety of factors in drawing conclusions on a product. So a review that talks about detection percentage without discussing false positives, for example, might run afoul of this principle. I have to say this is one of the more subjective of the principles. No. 5 says that you shouldn't just trust the judgments of the products being tested. You should confirm that samples detected as malware are in fact malware, and not false positives, or vice versa. This can be difficult. No. 6, the consistency principle, refers to using the right products for the audience, so you shouldn't, for example, test corporate gateway products for a consumer audience, or consumer solutions for an enterprise audience. No. 7 should be obvious: Don't assert conclusions which disagree with your data. When this happens it's usually a sign that the author is disappointed with the data for not showing what he wanted it to show. It helps to have a rule. No. 8, about statistical significance, is one that could do with some more specifics, but perhaps that wasn't possible at this point. If we should demand statistical significance in the results, we should provide a numerical standard for it. Still, there are plenty of tests (I'll admit it, I've been involved in some) where we didn't have enough samples and went ahead anyway. Once again, it helps to have a rule. No. 9, the active contact point rule, is a really good one for all involved. Perhaps it could have gone on to give a minimum response window.
[T]his disclosure should include any relationship that could potentially influence the tester, including: (i) whether the publication or tester has received revenue from a vendor or affiliate with regard to any particular test, and (ii) whether the publication or tester receives a significant portion of its overall revenue from a particular vendor.
I've always believed that this is the right way to do things and that people are able to consider such conflicts in judging the data.