How to Make Your Automated Software Tests Truly Automatic

Intelligent error collection and reporting

Finally, the most widely cited culprit for consuming valuable person-hours is dealing with the large volumes of data produced by test systems. Automated harnesses are usually great at posting gigabytes of HTML table with green and red icons to Web pages and inboxes, but they often fall far short of making that data actionable.

Actionable reporting means that for every data point presented to the user, you ask yourself the question: "If I show them this, what will they want to see next?"

Too often, the e-mail message says, "Test run #231 failed" but leaves it to the user to identify the offending test. Too often, the Web page says, "Test 144....FAILED" without any indication of how long it has been failing. Has it been failing since last March or has it been green since 2006 but only turned red this morning with the new employee's first check-in? Designing the test harness to make error collection—and just as important, error reporting—as intelligent as possible can have a dramatic impact on ease of use and productivity.

So, while the natural evolution of a test harness will lead it to being tedious to configure, slow to run and cryptic to interpret, there are straightforward strategies to remedy all three problems. And as shops that took the time to refactor the critical parts of resource selection, harness invocation and results reporting have shown, an easy, fast, efficient test harness may still not top your list of favorite leisure activities. However, it's a great deal more pleasant than filling in Form 1040.

Usman Muzaffar is Vice President of Product Management at Electric Cloud. He was part of the team that founded the company. Prior to Electric Cloud, Usman worked as a software engineer at Scriptics and Interwoven (acquired by Autonomy), designing and developing content management, syndication, and distribution systems. He holds a Bachelor's degree in Molecular Biology from Northwestern University. He can be reached at usman@electric-cloud.com.