The Anti-Malware Testing Standards Organization (AMTSO) has passed new guidelines dealing with developing proper testing metrics as well as ensuring tests are comprehensive.
The Anti-Malware Testing Standards Organization recently adopted new
guidelines to help improve the testing of security products.
Formed in 2008, AMTSO
to improve the reliability and accuracy of malware testing methods. At
a meeting May 25, the group adopted two papers-"Whole Product
Protection Testing Guidelines" and "Performance Testing Guidelines."
The first of the papers advocates a more balanced look at the effectiveness of products
taking into account the multiple layers of detection and protection,
the group said. Many tests focus only on simplistic detection rates, in
some cases actually disabling part of the product's functionality in
order to isolate a single protective layer, the group added.
"This guidelines document marks an important step in developing tests which accurately measure how an entire product actually functions
exposed to threats," said Igor Muttik, senior architect at McAfee Labs,
in a statement. "Too many current tests focus on individual
technologies, such as 'On Demand Scans.' Only by testing all of a
product's protection capabilities in a comprehensive test, can one
provide a more realistic view of the security offered to computer users
by contemporary security suites."
The second paper, "Performance Testing Guidelines," deals with the
challenges surrounding the development and use of proper benchmarking
"The Performance Testing Guidelines examines the myriad - and often
subtle - complexities in conducting speed tests," said Mikko Hypponen,
chief research officer of F-Secure, in a statement. "It is very
tempting to take a simplistic approach to measuring speed and footprint
of an antivirus program. However, there is as much art as there is
science in understanding the various elements which can skew the
results for the unwary tester. This document will help testers
understand these issues and allow them to take the necessary steps to
minimize them and take them into account."