As your team sets goals related to defects, use appropriate metrics to measure progress toward those goals. There are trends that you will want to monitor for the whole release, and there are ones that are iteration-specific. For example, if you’re trying to achieve zero defects, you may want to track the open bugs at the end of each iteration, or how many bugs were found after development but before release. Most of us are interested in knowing how many defects have been reported after the code is in production, which is something completely different altogether. These issues will tell you after the fact how well your team did on the last release, but not how well you are doing on the current release. They may give you some indication of what processes you need to change to reduce the number of defects. Lisa’s team is more concerned with production defects found in the “new” code that was rewritten in the new architecture. They’re working hard to produce this new code with zero defects, so they need to know how well they’re doing. They expect that bugs will be found fairly often in the legacy system, where only the most critical functionality is covered by automated GUI smoke tests, and there are few automated unit and behind-the-GUI tests.
Knowing the defect rate of legacy code might be good justification for refactoring or rewriting it, but the team’s top priority is doing a good job with the new code, so they group bugs by “new” and “old” code, and focus on the “new” bugs.
Make sure your bug database can track what you want to measure. You may have to make some changes in both the database and your process to get the data you need. For example, if you want to measure how many defects were found in production after a release, you have to make sure you have environment and version as mandatory fields, or make sure that people who enter bugs always fill them in.
More on defect tracking systems can be found in Chapter 5, “Transitioning Traditional Processes.”
Because defect tracking systems are often used for purposes besides tracking bugs, be sure not to muddle the numbers. A request for a manual update to the database doesn’t necessarily reflect an issue with the existing code. Use your defect tracking tool properly to ensure that your metrics are meaningful.
Lisa’s Story
Periodically evaluate the metrics you’re reporting and see if they’re still relevant. Figure 15-14 shows two defect reports that Lisa’s team used for years. When we first transitioned to agile, managers and others looked at these reports to see the progress that resulted from the new process. Four years later, our ScrumMaster found that nobody was reading these reports anymore, so we quit producing them. By that time, rates of new defects had reduced dramatically, and nobody really cared about the old defects still hanging about in the legacy code.
—Lisa
Figure 15-14 Sample defect reports used (and no longer used) by Lisa’s team
Release planning is a good time to evaluate the ROI of the metrics you’ve been tracking. How much effort are you spending to gather and report the metrics? Do they tell you what you need to know? Does the code you release meet your team’s standards for internal quality? Is the code coverage percentage going up? Is the team meeting its goals for reducing the number of defects that get out to production? If not, was there a good reason?
Metrics are just one piece of the puzzle. Use your release, theme, or project planning meetings to refocus on delivering business value when the business needs it. Take some time to learn about the features you’re about to develop. Don’t get caught up with committing to your plans—the situation is bound to change. Instead, prepare for doing the right activities and getting the right resources in time to meet the customers’ priorities.
Summary
As your team puts together its plan for a new theme or release, keep the main points of this chapter in mind.
