“What in the world is he thinking?” you may be asking. After all, we know that software defects cause applications to fail, so how can a defect have value?
I first saw this idea expressed by a client as a slogan, “Defects are good,” that was used throughout their organization. As I continued to think about the statement, “Defects are good,” it caused me to consider how the many perceptions of defects affect how QA and testing professionals deal with software quality in general.
The point is that a defect found early, or before a customer finds it, has value. To the producer, the value lies in a higher quality product that will have a better reputation, or, when the defect is found early, the cost to fix it is substantially less than if found later in the project. To the customer, the value of a defect found by the producer is that the customer will not encounter the problems caused by the defect.
Of course, if processes, people, and tools were perfect, perhaps we could achieve “zero defects.” However, defects are a fact of life with the resources we work with at present. Another given is that defects can cause serious problems in a project if they are not managed properly. Defect correction consumes resources and can even cause new defects.
There is another, “dark” side to defects that supports why a better view of defects is needed. When defects are seen only as bad, management tends to punish the people who create them. Even Dr. Deming said that people create defects and get paid for doing so. However, Dr. Deming also saw greater value in preventing defects.
When people are punished for defects, they can do all kinds of creative things to shift the blame. This includes hiding them until a point in time when blame can no longer be established, or until the person responsible leaves the project or company. Some testers have even been known to “hoard” defect information until the end of a project so that a last-minute “discovery” can be seen as a heroic act.
Developers have also been known to revel in their fire-fighting ability as opposed to building fault-tolerant software. These developers are often rewarded as the people who burn the midnight oil, while the people who build reliable software often go unnoticed in the background where things don’t go wrong.
How Much Are Defects Costing Us?
The cost of something often reveals its value. That holds true with defects.
As of this writing, the average cost of a production defect in the US is over $16,000!
If you find 10 defects during testing, you have saved the organization around $160,000! That is substantial value from testing. However, too few testers communicate this value message to management. Therefore, the value of testing and testers remains unseen and unvalued.
What Can We Learn From Defects?
There are many things we can learn from defects, but many people never get past the basic purpose of problem resolution. Here are some of the things that can turn these lumps of coal into diamonds:
Prevention of Future Problems
If adequate information is obtained and defects are analyzed from a preventative standpoint, this is where the greatest value of a defect is seen. This kind of analysis requires that the defect’s point of origin can be identified and tracked.
Preventive analysis also requires that a correctable process is in place. Otherwise, there is no vehicle to convey how to prevent the problem next time. This type of defect analysis also requires that defects be categorized in ways that can be meaningful for future improvement. I often call this “starting a bug collection.”
Examples of categories would include:
- Computational defects
- Data validation defects
- Usability defects
- Logical decision defects
- Requirements defects
- Design defects
- Presentation defects
Once you identify the most common types of defects in your organization, you can start addressing them.
A primary way to use defect information to improve processes is by performing root cause analysis. Since this term has become somewhat misused, I am referring to the classic definition, in which you seek to discover the single reason a defect occurred.
Basically, you work backward from the problem and keep asking “why” about the events that led up to the failure.
Once you have arrived at the root cause, you have something to improve.
There is a challenge, however. That is, sometimes there is no identifiable single root cause. Instead, there may be multiple contributing causes.
Tracking of Relative Improvement
When an organization works on a series of projects, a natural question is whether or not the team is improving in terms of the overall quality of the projects they complete. Information gained from defect analysis can help quantify relative improvement.
Defect metrics are often seen as elusive things to track. This is because defects must be relative to something to yield meaningful information. The myth is that the only relative measurement must be size. Although expressing defects per line of code or function point will give a defect density, there are other ways to relate defects that do not require a sizing measure.
For example, how about defects per defect? That’s right – if you count the number of defects you find in your testing (producer view) and divide those by the number of defects found over the defined life of the product (producer and customer views), you arrive at a percentage called “defect detection percentage (DDP).”
As an example, if I find 85 defects in all phases of my testing and in the next 12 months (defined life of product) my customers find 15 additional defects, then the DDP is 85%.
With this information, I can look at past projects and compare DDP levels and should be able to see improvement or regression. I can also compare my metrics with those of other organizations to see how I am doing with others in my industry.
Knowing Project Status
When defects are tracked throughout a project, they can give developers, testers, management and customers a quantitative idea of when the product is ready to release. Unless, that is, management has already decided on a date that will be met regardless of defects.
As an example, perhaps there are five days left before the project deadline and the testers have found 200 defects to date, 10 of which are severe and will require several more days to correct. This kind of information should be of great value to management as it is a key piece of information to judge the readiness of the application.
In addition, the location of defects can reveal opportunities to focus resources. In some cases, a phased implementation may be a contingency if an area of an application is non-critical and has many defects.
In my experience, when this type of information is ignored by management in favor of meeting a deadline, a high-risk situation exists. Proceeding with implementation in the face of many unresolved defects exacts a toll in rework, lost customers, and inefficiency.
Defect Prediction and Estimation
Being able to predict defect numbers and trends usually takes a good historical basis of defects in your own organization. This is because nobody else creates defects the same as in your organization. It may take years to accumulate enough defect data to start predicting trends, which is why this use of defect information is usually seen in organizations that do a lot of measurement.
If you are not at this point, don’t feel bad. For my money, this is icing on the cake. You can get 80% or more of the value of defects from the other three categories. Plus, people have been known to abuse this particular application of defect analysis by withholding defect discoveries until they fit the prediction model and make the tester(s) appear to be a hero – “The prediction model indicated we would find one more defect in the final week of testing and here it is!”
Summary
If we look at defects in the light of these purposes, we can see how something as frustrating and annoying as defects can actually have value. I am certainly not proposing a lax attitude toward defects. They cost money, delay projects, and even place lives at risk in some cases. However, since each defect does cost money and time, let’s learn as much as we can about defects so we can prevent them or at least find them earlier in the software development process.
