The Anti-Malware Testing Standards Organization has adopted a set of best practices around testing cloud security offerings. The body - which is made up of officials from companies such as Symantec, McAfee and Trend Micro - has also agreed to make analysis of product reviews public to educate consumers.
The words "in
the cloud" were heard numerous times at this years RSA security
conference in San Francisco. With the number of cloud-based
security products growing, the Anti-Malware Testing Standards Organization
(AMTSO) has been stirred to action.
Last week, the two-year-old
industry standards body adopted a paper setting forth best practices for
testing in-the-cloud security products. The
six-page document, available here, touches on subjects such as
virtualization, connection filtering and the repeatability of the tests.
For users, this means bringing a new
level of uniformity to the testing of cloud-based products on the market so
that more value can be taken from product reviews.
The most important element is
that some of the main assumptions of on demand tests no longer apply,
explained Mark Kennedy, distinguished engineer at Symantec. On demand
tests to date were widely held to be reproducible. Cloud technology now
makes this difficult if not impossible. Moreover, retrospective testing
(freezing products and testing them against newer samples) will be extremely
challenging without biasing it one way or the other.
The advent of cloud technology now
means static testing faces the same challenges as dynamic testing, he
continued, such as the freshness of samples and access to the Internet.
Many of us have argued for years
that static testing was not indicative of the full range of protection provided
by security suites, and that tests should move to dynamic testing, Kennedy
said. The rise of cloud technology should accelerate this process.
Testing cloud products is more
complex than traditional standalone software and requires more resources,
McAfees Igor Muttik added. To be reliable and fair, the tests have to be run
constantly - which means more computers, more bandwidth and a reliable testing
framework.
If the testing setup fails, this
can't be seen as a product failure, said Muttik, senior architect at McAfee
Avert Labs. To make the testing setup reliable, it itself needs to be tested.
Stepping away from the cloud, the
organization also announced plans to make analysis of anti-malware reviews
public in order to allow consumers to better assess their validity. The reviews
will be measured against AMTSOs standards for testing.
AMTSO is clear in its desire to
improve the quality of tests in a way that is independent of any vendor or
tester, Kennedy said. While it can be challenging to reach broad agreements
with such a diverse set of competitors and experts, the speed at which we have
done so underscores how important we all believe this group to be.