Joseph Juran, one of the founding fathers of the quality movement, introduced the idea of a “Cost of Quality,” or “CoQ,” in his Quality Control Handbook in 1951. Because it was developed in a manufacturing context, it is often seen as the quality-related costs of building physical products on an assembly line.
The CoQ is a little confusing because it can be defined in different ways. To some, it is not the total cost to achieve quality, but rather the cost of NOT achieving quality. This is also called the “cost of non-conformance.” This was the view of Phillip Crosby, who wrote the book Quality is Free in 1979.
There is also the “Total Quality Costs” (TQC) view that is comprised of more than the costs of non-conformance, such as the costs of assessment and prevention. The confusion arises because many people equate TQC to the CoQ. I tend to take that view and include prevention and assessment costs in the CoQ.
As a software quality practitioner and consultant, I have often considered whether adjustments are needed to accurately apply the traditional CoQ to software quality. As a software quality practitioner and consultant, I have often considered whether the traditional CoQ needs adjustments to apply accurately to software quality. This article explains the traditional CoQ metric and considers additional costs seen in software projects but not typically in manufacturing.
For clarity, I will distinguish between the “traditional CoQ” and the “software CoQ” (which I’ll refer to as CoSQ in this article).
I’m not the first person to write about the distinction between CoSQ and CoQ. Cem Kaner wrote an article on this topic in Software QA in 1996, and Herb Krasner wrote an article for CrossTalk in 1998.
However, I hope this article helps, in some small way, to continue the thinking process of adapting this valuable metric to software projects. My goal is to promote a metric that clearly shows to management in companies why it makes sense to embrace software quality instead of ignoring it.
What is the Cost of Quality?
Like so many things in quality, the usage and meaning of this term vary depending on when and where you learned it. I learned the following computation from my days in preparation for the CSQA (Certified Software Quality Analyst) certification:
The traditional CoQ is comprised of:
The cost of defect prevention – This can include process improvement, training, root cause analysis, or any other activities to prevent defects.
The cost of defect detection – This can include inspections, testing, or other quality control (QC) activities designed to find problems before customers do.
The cost of failures – This includes both internal and external failure costs. Internal failures are the costs associated with the product before the customer receives it. External failures are the costs associated with the product after the customer receives it. In manufacturing, this also includes warranty costs. In many ways, both involve rework and waste, but they can also include the costs a failure may have on other items, people, projects, etc. These costs can also include the clean-up costs of a failure.
As a benchmark, CoQ typically ranges from 15% to 40% (source: qualityportal.com) of the cost to make something. For example, in software, that would be a percentage of the total project costs.
What Does the CoQ Indicate?
If you include prevention and assessment costs, the CoQ shows how much an organization spends to deliver products of acceptable quality to customers. However, if your view of CoQ includes just the failure costs, it is the cost of not achieving quality.
In 1979, Philip Crosby wrote one the groundbreaking books of the quality movement in the U.S., called Quality is Free. My first reaction to the book’s title was confusion because it seemed like quality costs a lot to achieve. However, as I read the book, Crosby argued that while achieving quality does cost money, not achieving it costs more. In other words, quality improvement pays for itself many times over by preventing costly quality problems.
Crosby also explained it this way: when something is done right the first time, and no rework is needed, quality costs nothing. However, when something has to be scrapped or reworked, it incurs additional costs. (I still think that it costs something extra to do something right the first time, but that’s just me!)
As a real-life example, I was at the Oklahoma City airport recently, getting a quick meal before my flight. When I opened the sack, I realized I had someone else’s order. So, I took the sack back to the counter and explained that I had the wrong order. They were very apologetic and got my order to me, but since I had already walked away from the counter, they had to discard the food I had taken the first time. This is where getting something wrong had a tangible cost to the food vendor.
Until you look at each component of the CoQ, it’s impossible to know what the percentage number really means. For example, two companies may each have a 30% cost of quality. Company “A” spends almost all of its CoQ on rework and has unhappy customers, while Company “B” spends almost all of its CoQ on prevention and assessment and has happy customers.
An Interesting Dynamic
You might initially think that increasing the cost of detection or prevention just shifts cost allocation. For example, instead of spending 10% of a product development cost on failure costs, you may think of just spending that 10% on better testing or detection. However, a dynamic at work is interesting and powerful.
For example, for every dollar that you spend in prevention and detection, you often save many more dollars in failure costs. The multiplier varies by organization and when a defect is found, but suffice it to say that this is a winning proposition. This is the case Crosby made in “Quality is Free”.
Therefore, it is feasible to reduce your CoSQ while improving quality and increasing profits. Remember that every dollar spent on failure costs is one less dollar of profit. Consider multi-million-dollar project failures, and you have not only a costly project failure, but an equivalent profit loss! (Management may say, “Oh well, it’s just a write-off to the business.” I say, “So are bonuses!”)
The Impact of Poor Quality on Software Projects
One cost impact that seems highly applicable to software is the cost of a failed or delayed project. Although this could be assigned to internal failure costs, the CoQ context typically refers to the cost of product rework.
For example, a project may be delayed by three months because of high levels of defects found during user acceptance testing. This three-month delay causes:
- other projects to be delayed because resources are consumed with the project at hand,
- the company to lose five major contracts because a competitor beats the company to the marketplace,
- the company to spend $100,000 a month for additional staff required to support the manual work processes that are automated by the new project.
Another example is of the project that is canceled after spending $44 million on requirements specification and system analysis and design. At this point, the team determines the approach is not feasible and that spending more money is too risky.1
Security Costs
Consider the cost to both customers and the company when customers’ private data is compromised. For Providence Health Systems of Portland, Oregon, the cost of a data breach has been enormous. According to Information Week, “A company official told the Portland Oregonian that he expects the case, not including litigation costs, to cost from $7 million to $9 million. That includes providing affected patients with access to credit monitoring and restoration services. There’s also the prospect of civil lawsuits by state agencies or individuals whose data was stolen, potentially costing millions more.”2
“ChoicePoint paid dearly earlier this year when it agreed to pay $15 million in fines after unwittingly selling more than 100,000 consumer credit reports to thieves posing as legitimate customers. The FTC said the company didn’t have “reasonable procedures” for screening customers.”
These costs do not include the costs to customers who had to spend many hours contacting credit reporting agencies, monitoring their credit card accounts, and resolving fraudulent transactions.
Opportunity Costs
Poor-quality software can also put a company in a position where it misses opportunities because it can’t respond quickly. Consider the example of a bank that identifies a market opportunity. They present the idea to their IT department, which estimates eight months because of a backlog of fixes from a previous project that is still experiencing problems, and because no project life cycle process is in place. As a result, projects take longer and have a high number of defects. Yet, nothing is ever done to improve the situation because “we don’t have the time or the money” or “we’re still making a profit, aren’t we?”
However, what if the cost of losing the business opportunity is over $20 million dollars in lost profit? You can improve many processes with that kind of money!
Why are the CoQ and CoSQ Helpful?
The main reason these metrics are helpful is that they are focused on money. I often tell people in my software testing training classes that if you look deep into the eyes of management, you will see dollar signs and calendars.
What I mean is that money and time are at the forefront of many software managers’ and executives’ minds. It’s hard to make a case for the goodness of quality, increased customer satisfaction, or even risk. The executives and managers ask, “Where’s the profit in it?” That’s the question you must be prepared to answer about software quality, and the CoSQ can help you do that.
Juran noted, “Because the main language of [corporate management] was money, there emerged the concept of studying quality-related costs as a means of communication between the quality staff departments and the company managers.”
How to Use This Metric
The CoSQ is a powerful metric to show management that software quality is a business opportunity, not a cost center. Yes, “quality” is a fuzzy concept and often fails to resonate, but the costs are not fuzzy or trivial. In fact, the costs are measurable, and companies that understand their CoQ often see higher profits because they reduce waste and realize opportunities.
I suggest taking a baseline snapshot of your organization’s costs for the major cost areas (The items shown are a suggested starting point. You may think of others):
Prevention costs:
- Training
- Root cause analysis
- Process improvement
Detection costs:
- All forms of software and system testing
- Reviews and inspections
- Help desk costs
Internal failure costs
- Rework costs
- Delayed project costs
- Wasted time
External failure costs
- Tech support call costs
- Rework
- Patches and post-release fixes (such as hot fixes)
- Failure costs to customers
- Lost sales
- Penalties
Another good thing to do is to start measuring the average cost per defect. I know of situations where people were surprised to learn that the average cost of a defect exceeded $5,000 in their organization. You can improve a lot of things for the cost of just 4 or 5 defects!
Use Examples from Your Organization
These shouldn’t be hard to find. Try to find situations where problems caused high dollar costs, or where problems have been prevented by improving processes or by inspections and testing.
Be careful with this! People can get very defensive when you use their projects as your poster child. Before using any example as a case study, I always talk to the people involved and get their blessing as well as their feedback on the accuracy of the numbers.
A Word of Warning
Don’t overreach in your calculations or oversell the value of software quality. The first thing that will happen when you start discussing the CoSQ metric is that people will scrutinize and discount your numbers. If your numbers are inflated and your claims seem too high, you will lose both your message and your credibility.
Instead, be conservative in your cost assumptions and don’t over-promise value. You can do this and still show the powerful value of software quality.
About Your Customers
Dr. Juran emphasized that we have both internal and external customers. I agree that we must consider both perspectives.
I have heard some people downplay the importance of internal customers because they don’t buy products or services. True, but I would argue internal customers also generate profits because they sell things, keep valuable customers, develop new products and services to sell, etc.
While the external customers are very, very important, the internal customers are the ones that make the decision whether or not to outsource IT functions
Many companies measure customer satisfaction. I used to think this was great until I read a book by Jeffrey Gittomer called Customer Satisfaction is Worthless, Customer Loyalty is Priceless. His point is that a satisfied customer thinks your product or service is OK but could easily be swayed to a competitor. A loyal customer believes in your product or service so much that they are with you for a long time. The only reason they leave is that they are forced to, because you lower the quality of your products and services.
This reminds me of a time I was presenting the findings and recommendations of a software testing assessment to an executive team at a financial services company. The company was spending over $250,000 a year just on the salaries of developers resolving daily software problems. The system moved billions of dollars daily.
At one point in my presentation, when I described the risks in their current methods, the VP of Marketing spoke up and said, “The only metric I care about is how many customers we have lost, and we haven’t lost any.” So he saw no need to improve.
About six months later, the CIO who had hired me for the assessment called. She was working for a different company and told me a major “incident” had occurred, and they had lost customers. The problem was that everyone who could have fixed the chronically defective software had left the company, burned out and frustrated (I suppose that could be a metric as well).
Conclusion
Dr. W. Edwards Deming taught that a business is made up of functions and activities that can be identified, measured, and improved. The goal of this effort is not to score higher on an assessment or to reach a certain level or to win an award. The goal is to improve how a business does its work so it becomes more profitable and healthy.
One key metric that can drive this type of improvement is the Cost of Quality (CoQ). For software, we need a broader view that includes costs typically not seen in manufacturing.
CoSQ can be a valuable tool for conveying the value of software quality to improve the business. Don’t oversell it, but use it to show the value of quality and benchmark your progress.
