Different Companies, Same Problem
Different companies, same problem
Again, there is a striking similarity in the problem evolution across companies. Early in the life cycle, developers write an application-specific test harness to validate the nascent product. Indeed, modern best practices-especially the test-driven development (TDD) practice in Agile software development-place such a premium on being able to continually and automatically exercise components as they are written that, in some cases, the test harness matures faster than the product.
The situation is complicated when release engineering promotes the harness (originally created only to assist the developer's edit/compile/debug cycle) to a key piece of infrastructure in the production integration and release process.
Now, suddenly, small details that could be taken for granted in the developer's workflow ("of course your database is at localhost:3306" and "of course gcc 3.3.1 is installed at /usr/tools/bin") require constant attention. There is no incentive for the developers to address the issues that are only faced in production. Conversely, there is no facility for the release engineering team to expose their process to developers.
Three challenges crystallize as the answers to the questions above. First, it's hard to get the environment, including both harness and product, set up on an appropriate box. Second, it's hard to invoke the test. Third, it's hard to mine the results for concrete next actions.
This triumvirate-resource configuration, test execution and results monitoring-is the "last mile" of automation. When ignored, the entire value of automated testing pays a steep penalty. Fortunately, for each of these systemic problems, there are simple (though not necessarily easy) solutions we can apply to great effect.
The resource selection problem is simply this: "Where do I run the test?" At most shops, the answer itself quickly leads to a sysadmin puzzle: "Where or how can I easily log into 20 systems in one click? Where do I have SSH (Secure Shell) keys configured? How will I keep my tests from stomping on my neighbor's? And how do I fire up virtual machines from an ESX server for my config?" and so on.
The correct answer: Stop asking these questions! Get your test harness to do it for you. There are three critical pieces you need to make that a reality-and one big trap to avoid.