Making Your Test Harness Work For You
Making your test harness work for you
First, tests need to support distributed, remote execution. Second, the target host and the build need to be logically divorced and merged together only at test invocation. Third, as much as possible, standardize on one host environment for high-volume, frequent "smoke test" runs.
The pitfall: If the test process is hard-coded to a host ("Version 3.4 can only be tested on Windows 2003 hosts with VS 2008 + our add-in"), the inefficiencies and complexity start to creep in. Standardizing on a host configuration for all active branches can open extraordinary optimization opportunities. Need to run a new system test? Grab next available node from the large homogenous grid, load product and go.
The concept of carefully subsetting the test suite to run a critical selection of smoke tests first is not new but it is essential to efficient automatic testing. The natural objection from developers is that the project-wide smoke test possibly will not cover the code they most recently modified.
Enter the second solution pattern: Test harnesses need to be highly parameterized but their invocation needs to be easily aliased. This is a common pattern you use all the time: Type "alias" at the prompt of grizzled Unix command-line hackers and you'll see shortcut entries encapsulating commonly used options to programs such as "ls"; on Windows, the Start Menu is nothing but a tree of shortcuts to programs and documents.
Effective automated test systems pull the same stunt: a million possible options to pick the smoke tests, the server tests, the user interface tests and so forth. However, they are easily abstracted behind a single user-specific click on the harness interface. Corollary gotcha: Do not let your harness sources live with your product sources because this encourages an artificial integration that makes building such an interface much more difficult.