Devising a Metric

By Rex Black  |  Posted 2010-01-26 Print this article Print

Devising a metric

For each of these questions, devise a metric. Let's start with the percentage of bugs found and removed prior to release. Assuming you run a system test and user acceptance test prior to deployment, calculate the metric as follows:

Defect removal effectiveness (DRE) = the amount of test bugs removed divided by the amount of test bugs removed + the production bugs found.

A goal of 100 percent DRE is unreasonable. Trying to find and remove all the bugs would be too expensive and time-consuming. Instead, to set your quality goal for this objective, benchmark your organization first and then set a goal that improves quality relative to this metric.

Based on his studies, Capers Jones reports that the industry average for DRE is about 85 percent. Depending on the criticality of your application, the cost of production bugs, the importance of schedule and budget constraints and other considerations, you might set your goal well above or well below this industry average.

Though overall DRE should not reach 100 percent, you want to find and remove almost all of the important bugs. So, check this by using the DRE again. First, calculate the DRE for all bugs. Then, calculate the DRE for the critical bugs-however you define critical. The following relationship should hold:

Quality focus: DRE (all bugs) < DRE (critical bugs)

As before, I suggest you benchmark your organization first, and then set goals for improving your quality focus using this metric. If the DRE for critical bugs is 5 to 15 percent above the overall DRE, that typically indicates a good quality focus, provided the DRE for critical bugs is over 95 percent.

Rex Black is President of RBCS. Rex is also the immediate past president of the International Software Testing Qualifications Board and the American Software Testing Qualifications Board. Rex has published six books, which have sold over 50,000 copies, including Japanese, Chinese, Indian, Hebrew and Russian editions. Rex has written over thirty articles, presented hundreds of papers, workshops and seminars, and given over fifty speeches at conferences and events around the world. Rex may be reached at

Submit a Comment

Loading Comments...
Manage your Newsletters: Login   Register My Newsletters

Rocket Fuel