Dynamic Testing Issues

By Larry Seltzer  |  Posted 2008-11-14 Print this article Print


There is a second AMTSO document: "Best Practices for Dynamic Testing." Most high-volume testing of malware is run through automated systems where files are copied from network shares to the test system. It's not the way users run their own computers.

"Dynamic Testing" aims to reproduce, in every meaningful way, the actual user environment for which the product was designed. This has become more necessary over time as anti-malware products increasingly include features, such as very frequent updates, which do not function properly in a classic lab environment.

The paper recognizes that testing like this is extremely difficult. Often, even when done fairly, it's impossible to reproduce results consistently. But it encourages testers to do what they can to make circumstances consistent and fair.

Here's a good example of a problem that such testing encounters: PC users will be open to the Internet; should the test systems be? What if malware escapes from the test system, violating Principle 1 above? The document recognizes several approaches that can be valid, including building a fake Internet, known amusingly as a "Truman box." Whatever method you use, the important thing is to discuss what you did and the effects of it.

Use of virtual machines is a big issue in dynamic testing. Spawning off a new VM for testing such products makes the testing far easier, but the environment is not the same as the typical PC user's. More and more malware is becoming aware of VM environments and using that information to change behavior, probably under the assumption that VMs indicate a tester. Because of this, as tempting as VMs are, AMTSO recommends real machines for dynamic testing, and that members share tools to facilitate such testing.

Talk about standards groups usually evokes an academic image, but some of the best standards have come out of industry consortiums. AMTSO membership is largely composed of vendors, and they recognize that they have an interest in good testing.

Don't expect that you'll start seeing results compliant with these guidelines a lot. Testing like this is difficult and expensive and few labs are set up to do it. If all goes well, more will be from now on.

Security Center Editor Larry Seltzer has worked in and written about the computer industry since 1983.

For insights on security coverage around the Web, take a look at eWEEK.com Security Center Editor Larry Seltzer's blog Cheap Hack.

Larry Seltzer has been writing software for and English about computers ever since—,much to his own amazement—,he graduated from the University of Pennsylvania in 1983.

He was one of the authors of NPL and NPL-R, fourth-generation languages for microcomputers by the now-defunct DeskTop Software Corporation. (Larry is sad to find absolutely no hits on any of these +products on Google.) His work at Desktop Software included programming the UCSD p-System, a virtual machine-based operating system with portable binaries that pre-dated Java by more than 10 years.

For several years, he wrote corporate software for Mathematica Policy Research (they're still in business!) and Chase Econometrics (not so lucky) before being forcibly thrown into the consulting market. He bummed around the Philadelphia consulting and contract-programming scenes for a year or two before taking a job at NSTL (National Software Testing Labs) developing product tests and managing contract testing for the computer industry, governments and publication.

In 1991 Larry moved to Massachusetts to become Technical Director of PC Week Labs (now eWeek Labs). He moved within Ziff Davis to New York in 1994 to run testing at Windows Sources. In 1995, he became Technical Director for Internet product testing at PC Magazine and stayed there till 1998.

Since then, he has been writing for numerous other publications, including Fortune Small Business, Windows 2000 Magazine (now Windows and .NET Magazine), ZDNet and Sam Whitmore's Media Survey.

Submit a Comment

Loading Comments...
Manage your Newsletters: Login   Register My Newsletters

Rocket Fuel