The Anti-Malware Testing Standards Organization has published a set of best practices for dynamic testing of host-based anti-malware products as well as a collection of fundamental principles to govern testing. While the standards represent a good first step, the AMTSO still has its work cut out for it.
The Anti-Malware Testing Standards Organization has published a set of best
practices and guiding principles for testing
. Call it a much-needed first step.
two documents on its Web site, setting the foundation for
the uniform testing regime the security industry needs. Both documents are the
fruits of discussions by more than 40 security experts, product testers and
members of the media from across the globe.
The first of the two is titled the "AMTSO Fundamental Principles of Testing"
and lays out a number of basic rules, such as requiring testers to validate
whether test samples have been correctly classified as malicious and mandating that
tests be open and transparent.
The second document is a set of best practices for dynamic testing of
host-based security products and stresses, among other things, the importance
of keeping logs of what happened during the test. It also offers advice on
subjects such as sample collection, measuring results and
handling user-product interaction.
While the documents do not get down and dirty into step-by-step guidelines
as to how tests should be conducted, they represent an important first step for
an organization with plenty of work to do. Founded in May, the AMTSO was
established to respond to concerns that product tests were not keeping up with
either technology innovations or the malware the products were designed to
Though its recommendations are voluntary, testing standards are needed to
give real meaning to the product reviews circulating around the Internet.
Offering direction on subjects such as choosing malware samples can make a big
"From my perspective, I think [one] of the most important things is sample
selection," said Dave Marcus, director of security research for McAfee's Avert
Labs. "What is exactly the sample set that you're using to test with? That's a
real important question. That probably causes more problems in testing than
almost anything else."