Dynamic Testing Issues
There is a second AMTSO document: "Best Practices for Dynamic Testing." Most high-volume testing of malware is run through automated systems where files are copied from network shares to the test system. It's not the way users run their own computers. "Dynamic Testing" aims to reproduce, in every meaningful way, the actual user environment for which the product was designed. This has become more necessary over time as anti-malware products increasingly include features, such as very frequent updates, which do not function properly in a classic lab environment.Here's a good example of a problem that such testing encounters: PC users will be open to the Internet; should the test systems be? What if malware escapes from the test system, violating Principle 1 above? The document recognizes several approaches that can be valid, including building a fake Internet, known amusingly as a "Truman box." Whatever method you use, the important thing is to discuss what you did and the effects of it. Use of virtual machines is a big issue in dynamic testing. Spawning off a new VM for testing such products makes the testing far easier, but the environment is not the same as the typical PC user's. More and more malware is becoming aware of VM environments and using that information to change behavior, probably under the assumption that VMs indicate a tester. Because of this, as tempting as VMs are, AMTSO recommends real machines for dynamic testing, and that members share tools to facilitate such testing. Talk about standards groups usually evokes an academic image, but some of the best standards have come out of industry consortiums. AMTSO membership is largely composed of vendors, and they recognize that they have an interest in good testing. Don't expect that you'll start seeing results compliant with these guidelines a lot. Testing like this is difficult and expensive and few labs are set up to do it. If all goes well, more will be from now on. Security Center Editor Larry Seltzer has worked in and written about the computer industry since 1983. For insights on security coverage around the Web, take a look at eWEEK.com Security Center Editor Larry Seltzer's blog Cheap Hack.
The paper recognizes that testing like this is extremely difficult. Often, even when done fairly, it's impossible to reproduce results consistently. But it encourages testers to do what they can to make circumstances consistent and fair.