The Anti-Malware Testing Standards Organization has published a set of best practices and guiding principles for testing security software. Call it a much-needed first step.
The group published two documents on its Web site, setting the foundation for the uniform testing regime the security industry needs. Both documents are the fruits of discussions by more than 40 security experts, product testers and members of the media from across the globe.
The first of the two is titled the “AMTSO Fundamental Principles of Testing” and lays out a number of basic rules, such as requiring testers to validate whether test samples have been correctly classified as malicious and mandating that tests be open and transparent.
Malware detection goes hybrid. Find out more here.
The second document is a set of best practices for dynamic testing of host-based security products and stresses, among other things, the importance of keeping logs of what happened during the test. It also offers advice on subjects such as sample collection, measuring results and handling user-product interaction.
While the documents do not get down and dirty into step-by-step guidelines as to how tests should be conducted, they represent an important first step for an organization with plenty of work to do. Founded in May, the AMTSO was established to respond to concerns that product tests were not keeping up with either technology innovations or the malware the products were designed to fight.
Though its recommendations are voluntary, testing standards are needed to give real meaning to the product reviews circulating around the Internet. Offering direction on subjects such as choosing malware samples can make a big difference.
“From my perspective, I think [one] of the most important things is sample selection,” said Dave Marcus, director of security research for McAfee’s Avert Labs. “What is exactly the sample set that you’re using to test with? That’s a real important question. That probably causes more problems in testing than almost anything else.”
The Issue of Behavioral Detection
Then there is the issue of behavioral detection, which has long been a missing piece of anti-virus tests due to costs and complexity. With vendors adding functionality such as generic buffer overflow prevention and whitelisting, standards for incorporating that technology into tests need to be developed as well. A tall task to be sure, but one the AMTSO needs to work methodically toward.
“That’s one of the problems-they’re not looking at the newer functionalities like behavioral or generic buffer overflow prevention or access protection rules,” Marcus said. “I’m looking forward to seeing those types of best standards … guidelines as to how to test those newer types of technologies because I think people don’t get a real accurate representation of how good or bad a technology is because they’re tested solely against regular signatures.”