The Anti-Malware Testing Standards Organization recently adopted new guidelines to help improve the testing of security products.
Formed in 2008, AMTSO works to improve the reliability and accuracy of malware testing methods. At a meeting May 25, the group adopted two papers-“Whole Product Protection Testing Guidelines” and “Performance Testing Guidelines.”
The first of the papers advocates a more balanced look at the effectiveness of products, taking into account the multiple layers of detection and protection, the group said. Many tests focus only on simplistic detection rates, in some cases actually disabling part of the product’s functionality in order to isolate a single protective layer, the group added.
“This guidelines document marks an important step in developing tests which accurately measure how an entire product actually functions when exposed to threats,” said Igor Muttik, senior architect at McAfee Labs, in a statement. “Too many current tests focus on individual technologies, such as ‘On Demand Scans.’ Only by testing all of a product’s protection capabilities in a comprehensive test, can one provide a more realistic view of the security offered to computer users by contemporary security suites.”
The second paper, “Performance Testing Guidelines,” deals with the challenges surrounding the development and use of proper benchmarking methodologies.
“The Performance Testing Guidelines examines the myriad – and often subtle – complexities in conducting speed tests,” said Mikko Hypponen, chief research officer of F-Secure, in a statement. “It is very tempting to take a simplistic approach to measuring speed and footprint of an antivirus program. However, there is as much art as there is science in understanding the various elements which can skew the results for the unwary tester. This document will help testers understand these issues and allow them to take the necessary steps to minimize them and take them into account.”