Devising a Metric
Devising a metric
For each of these questions, devise a metric. Let's start with the percentage of bugs found and removed prior to release. Assuming you run a system test and user acceptance test prior to deployment, calculate the metric as follows:
Defect removal effectiveness (DRE) = the amount of test bugs removed divided by the amount of test bugs removed + the production bugs found.
A goal of 100 percent DRE is unreasonable. Trying to find and remove all the bugs would be too expensive and time-consuming. Instead, to set your quality goal for this objective, benchmark your organization first and then set a goal that improves quality relative to this metric.
Based on his studies, Capers Jones reports that the industry average for DRE is about 85 percent. Depending on the criticality of your application, the cost of production bugs, the importance of schedule and budget constraints and other considerations, you might set your goal well above or well below this industry average.
Though overall DRE should not reach 100 percent, you want to find and remove almost all of the important bugs. So, check this by using the DRE again. First, calculate the DRE for all bugs. Then, calculate the DRE for the critical bugs-however you define critical. The following relationship should hold:
Quality focus: DRE (all bugs) < DRE (critical bugs)
As before, I suggest you benchmark your organization first, and then set goals for improving your quality focus using this metric. If the DRE for critical bugs is 5 to 15 percent above the overall DRE, that typically indicates a good quality focus, provided the DRE for critical bugs is over 95 percent.