Standards Come to Anti-malware Testing

By Larry Seltzer  |  Posted 2008-11-14 Print this article Print

Security industry organization AMTSO develops high-quality guidelines to help vendors, analysts and publications test anti-malware products in a fair and thorough way.

Computer product testing, sadly, has been as much art as science over the years. It's not just that the products are so complicated as to defy simple, straightforward analysis, but also there are no general agreements on how products should be tested. Now that may be changing with respect to the testing of anti-malware products.

New guidelines issued by AMTSO (Anti-Malware Testing Standards Organization) set an excellent standard for high-quality testing that you can believe in. I was in the professional testing business for many years, at least 13 or 14, and was technical director at four different labs. I don't do much actual testing of products anymore, but I still follow testing issues carefully. I'm really impressed with what I'm reading in these standards.

Two "Principles" documents were released by AMTSO. The first, "AMTSO Fundamental Principles of Testing," is a set of rules and advice, mostly for testers. The nine principles:

  1. Testing must not endanger the public.
  2. Testing must be unbiased.
  3. Testing should be reasonably open and transparent.
  4. The effectiveness and performance of anti-malware products must be measured in a balanced way.
  5. Testers must take reasonable care to validate whether test samples or test cases have been accurately classified as malicious, innocent or invalid.
  6. Testing methodology must be consistent with the testing purpose.
  7. The conclusions of a test must be based on the test results.
  8. Test results should be statistically valid.
  9. Vendors, testers and publishers must have an active contact point for testing-related correspondence.
Some of these are more obvious than others, but the elaboration of the principles that follows makes clear they aren't just lip service. With respect to No. 1, I've been involved with malware tests, especially for the ability to detect unknown malware, where we have discussed creating new malware purely for the test. The guidelines specifically forbid this, although it does allow the modification of existing malware characteristics. This principle also speaks about taking precautions to prevent malware from escaping the lab.

Larry Seltzer has been writing software for and English about computers ever since—,much to his own amazement—,he graduated from the University of Pennsylvania in 1983.

He was one of the authors of NPL and NPL-R, fourth-generation languages for microcomputers by the now-defunct DeskTop Software Corporation. (Larry is sad to find absolutely no hits on any of these +products on Google.) His work at Desktop Software included programming the UCSD p-System, a virtual machine-based operating system with portable binaries that pre-dated Java by more than 10 years.

For several years, he wrote corporate software for Mathematica Policy Research (they're still in business!) and Chase Econometrics (not so lucky) before being forcibly thrown into the consulting market. He bummed around the Philadelphia consulting and contract-programming scenes for a year or two before taking a job at NSTL (National Software Testing Labs) developing product tests and managing contract testing for the computer industry, governments and publication.

In 1991 Larry moved to Massachusetts to become Technical Director of PC Week Labs (now eWeek Labs). He moved within Ziff Davis to New York in 1994 to run testing at Windows Sources. In 1995, he became Technical Director for Internet product testing at PC Magazine and stayed there till 1998.

Since then, he has been writing for numerous other publications, including Fortune Small Business, Windows 2000 Magazine (now Windows and .NET Magazine), ZDNet and Sam Whitmore's Media Survey.

Submit a Comment

Loading Comments...
Manage your Newsletters: Login   Register My Newsletters

Rocket Fuel