The words “in the cloud” were heard numerous times at this year’s RSA security conference in San Francisco. With the number of cloud-based security products growing, the Anti-Malware Testing Standards Organization (AMTSO) has been stirred to action.
Last week, the two-year-old industry standards body adopted a paper setting forth best practices for testing in-the-cloud security products. The six-page document, available here, touches on subjects such as virtualization, connection filtering and the repeatability of the tests.
For users, this means bringing a new level of uniformity to the testing of cloud-based products on the market so that more value can be taken from product reviews.
“The most important element is that some of the main assumptions of on demand tests no longer apply,” explained Mark Kennedy, distinguished engineer at Symantec. “On demand tests to date were widely held to be reproducible. Cloud technology now makes this difficult if not impossible. Moreover, retrospective testing (freezing products and testing them against newer samples) will be extremely challenging without biasing it one way or the other.”
The advent of cloud technology now means static testing faces the same challenges as dynamic testing, he continued, such as the freshness of samples and access to the Internet.
“Many of us have argued for years that static testing was not indicative of the full range of protection provided by security suites, and that tests should move to dynamic testing,” Kennedy said. “The rise of cloud technology should accelerate this process.”
Testing cloud products is more complex than traditional standalone software and requires more resources, McAfee’s Igor Muttik added. To be reliable and fair, the tests have to be run constantly – which means more computers, more bandwidth and a reliable testing framework.
“If the testing setup fails, this can’t be seen as a product failure,” said Muttik, senior architect at McAfee Avert Labs. “To make the testing setup reliable, it itself needs to be tested.”
Stepping away from the cloud, the organization also announced plans to make analysis of anti-malware reviews public in order to allow consumers to better assess their validity. The reviews will be measured against AMTSO’s standards for testing.
“AMTSO is clear in its desire to improve the quality of tests in a way that is independent of any vendor or tester,” Kennedy said. “While it can be challenging to reach broad agreements with such a diverse set of competitors and experts, the speed at which we have done so underscores how important we all believe this group to be.”