Making Your Test Harness Work For You

By Usman Muzaffar  |  Posted 2010-07-06 Print this article Print

Making your test harness work for you

First, tests need to support distributed, remote execution. Second, the target host and the build need to be logically divorced and merged together only at test invocation. Third, as much as possible, standardize on one host environment for high-volume, frequent "smoke test" runs.

The pitfall: If the test process is hard-coded to a host ("Version 3.4 can only be tested on Windows 2003 hosts with VS 2008 + our add-in"), the inefficiencies and complexity start to creep in. Standardizing on a host configuration for all active branches can open extraordinary optimization opportunities. Need to run a new system test? Grab next available node from the large homogenous grid, load product and go.

The concept of carefully subsetting the test suite to run a critical selection of smoke tests first is not new but it is essential to efficient automatic testing. The natural objection from developers is that the project-wide smoke test possibly will not cover the code they most recently modified.

Enter the second solution pattern: Test harnesses need to be highly parameterized but their invocation needs to be easily aliased. This is a common pattern you use all the time: Type "alias" at the prompt of grizzled Unix command-line hackers and you'll see shortcut entries encapsulating commonly used options to programs such as "ls"; on Windows, the Start Menu is nothing but a tree of shortcuts to programs and documents.

Effective automated test systems pull the same stunt: a million possible options to pick the smoke tests, the server tests, the user interface tests and so forth. However, they are easily abstracted behind a single user-specific click on the harness interface. Corollary gotcha: Do not let your harness sources live with your product sources because this encourages an artificial integration that makes building such an interface much more difficult.

Usman Muzaffar is Vice President of Product Management at Electric Cloud. He was part of the team that founded the company. Prior to Electric Cloud, Usman worked as a software engineer at Scriptics and Interwoven (acquired by Autonomy), designing and developing content management, syndication, and distribution systems. He holds a Bachelor's degree in Molecular Biology from Northwestern University. He can be reached at

Submit a Comment

Loading Comments...
Manage your Newsletters: Login   Register My Newsletters

Rocket Fuel