Standards Come to Anti-malware Testing

Computer product testing, sadly, has been as much art as science over the years. It's not just that the products are so complicated as to defy simple, straightforward analysis, but also there are no general agreements on how products should be tested. Now that may be changing with respect to the testing of anti-malware products.

New guidelines issued by AMTSO (Anti-Malware Testing Standards Organization) set an excellent standard for high-quality testing that you can believe in. I was in the professional testing business for many years, at least 13 or 14, and was technical director at four different labs. I don't do much actual testing of products anymore, but I still follow testing issues carefully. I'm really impressed with what I'm reading in these standards.

Two "Principles" documents were released by AMTSO. The first, "AMTSO Fundamental Principles of Testing," is a set of rules and advice, mostly for testers. The nine principles:

1. Testing must not endanger the public.
2. Testing must be unbiased.
3. Testing should be reasonably open and transparent.
4. The effectiveness and performance of anti-malware products must be measured in a balanced way.
5. Testers must take reasonable care to validate whether test samples or test cases have been accurately classified as malicious, innocent or invalid.
6. Testing methodology must be consistent with the testing purpose.
7. The conclusions of a test must be based on the test results.
8. Test results should be statistically valid.
9. Vendors, testers and publishers must have an active contact point for testing-related correspondence.

Some of these are more obvious than others, but the elaboration of the principles that follows makes clear they aren't just lip service. With respect to No. 1, I've been involved with malware tests, especially for the ability to detect unknown malware, where we have discussed creating new malware purely for the test. The guidelines specifically forbid this, although it does allow the modification of existing malware characteristics. This principle also speaks about taking precautions to prevent malware from escaping the lab.