"Because the main language of [corporate management] was money, there emerged the concept of studying quality-related costs as a means of communication between the quality staff departments and the company managers." 1
Joseph Juran, one of the world's leading quality theorists, has been advocating the analysis of quality-related costs since 1951, when he published the first edition of his Quality Control Handbook. Feigenbaum made it one of the core ideas underlying the Total Quality Management movement.2 It is a tremendously powerful tool for product quality, including software quality.
Quality costs are the costs associated with preventing, finding, and correcting defective work. These costs are huge, running at 20% - 40% of sales.3 Many of these costs can be significantly reduced or completely avoided. One of the key functions of a Quality Engineer is the reduction of the total cost of quality associated with a product.
Here are six useful definitions, as applied to software products. Figure 1 gives examples of the types of cost. Most of Figure 1's examples are (hopefully) self-explanatory, but I'll provide some additional notes on a few of the costs:4
Some programming groups treat user interface errors as low priority, leaving them until the end to fix. This can be a mistake. Marketing staff need pictures of the product's screen long before the program is finished, in order to get the artwork for the box into the printer on time. User interface bugs - the ones that will be fixed later - can make it hard for these staff members to take (or mock up) accurate screen shots. Delays caused by these minor design flaws, or by bugs that block a packaging staff member from creating or printing special reports, can cause the company to miss its printer deadline.
Including costs like lost opportunity and cost of delays in numerical estimates of the total cost of quality can be controversial. Campanella (1990)5 doesn't include these in a detailed listing of examples. Gryna (1988)6 recommends against including costs like these in the published totals because fallout from the controversy over them can kill the entire quality cost accounting effort. I include them here because I sometimes find them very useful, even if it might not make sense to include them in a balance sheet.
Some of these costs must be treated with care. For example, the cost of public relations efforts to soften the publicity effects of bugs is probably not a huge percentage of your company's PR budget. You can't charge the entire PR budget as a quality-related cost. But any money that the PR group has to spend to specifically cope with potentially bad publicity due to bugs is a failure cost.
I've omitted from Figure 1 several additional costs that I don't know how to estimate, and that I suspect are too often too controversial to use. Of these, my two strongest themes are cost of high turnover (people quit over quality-related frustration - this definitely includes sales staff, not just development and support) and cost of lost pride (many people will work less hard, with less care, if they believe that the final product will be low quality no matter what they do.)
Over the long term, a project (or corporate) cost accounting system that tracks quality-related costs can be a fundamentally important management tool. This is the path that Juran and Feigenbaum will lead you down, and they and their followers have frequently and eloquently explained the path, the system, and the goal.
I generally work with young, consumer-oriented software companies who don't see TQM programs as their top priority, and therefore my approach is more tactical. There is significant benefit in using the language and insights of quality cost analysis, on a project/product by project/product basis, even in a company that has no interest in Total Quality Management or other formal quality management models.12
Here's an example. Suppose that some feature has been designed in a way that you believe will be awkward and annoying for the customer. You raise the issue and the project manager rejects your report as subjective. It's "not a bug." Where do you go if you don't want to drop this issue? One approach is to keep taking it to higher-level managers within product development (or within the company as a whole). But without additional arguments, you'll often keep losing, without making any friends in the process.
Suppose that you change your emphasis instead. Rather than saying that, in your opinion, customers won't be happy, collect some other data:13
You won't get cost estimates from everyone, but you might be able to get ballpark estimates from most, along with one or two carefully considered estimates. This is enough to give you a range to present at the next project meeting, or in a follow-up to your original bug report. Notice the difference in your posture:
Along with arguing about individual bugs, or groups of bugs, this approach opens up opportunities for you (and other non-testers who come to realize the power of your approach) to make business cases on several other types of issues. For example:
Gryna (1988)14 and Juran & Gryna (1980)15 point out several problems that have caused cost-of-quality approaches to fail. I'll mention two of the main ones here.
First, it's unwise to try to achieve too much, too fast. For example, don't try to apply a quality cost system to every project until you've applied it successfully to one project. And don't try to measure all of the costs, because you probably can't.16
Second, beware of insisting on controversial costs. Gryna (1988)17 points out several types of costs that other managers might challenge as not being quality-related. If you include these costs in your totals (such as total cost of quality), some readers will believe that you a padding these totals, to achieve a more dramatic effect. Gryna's advice is to not include them. This is usually wise advice, but it can lead you to underestimate your customer's probable dissatisfaction with your product. As we see in the next section, down that road lies LawyerLand.
Quality Cost Analysis looks at the company's costs, not the customer's costs. The manufacturer and seller are definitely not the only people who suffer quality-related costs. The customer suffers quality-related costs too. If a manufacturer sells a bad product, the customer faces significant expenses in dealing with that bad product.
The Ford Pinto litigation provided the most famous example
of a quality cost analysis that evaluated company costs without
considering customers' costs from the customers' viewpoint. Among
the documents produced in these cases was the Grush-Saunby report,
which looked at costs associated with fuel tank integrity. The
key calculations appeared in Table 3 of the report:18
Savings - 180 burn deaths, 180 serious burn injuries, 2100 burned vehicles
Unit Cost -- $200,000 per death, $67,000 per injury, $700 per vehicle
Total Benefit - 180 x ($200,000) + 180 x ($67,000) + 2100 x ($700) = $49.5 million.
Sales - 11 million cars, 1.5 million light trucks.
Unit Cost -- $11 per car, $11 per truck
Total Cost - 11,000,000 x ($11) + 1,500,000 x ($11) = $137 million.
In other words, it looked cheaper to pay an average of $200,000 per death in lawsuit costs than to pay $11 per car to prevent fuel tank explosions. Ultimately, the lawsuit losses were much higher.19
This kind of analysis didn't go away with the Pinto. For example, in the more recent case of General Motors Corp. v. Johnston (1992)20, a PROM controlled the fuel injector in a pickup truck. The truck stalled because of a defect in the PROM and in the ensuing accident, Johnston's seven-year old grandchild was killed. The Alabama Supreme Court justified an award of $7.5 million in punitive damages against GM by noting that GM "saved approximately $42,000,000 by not having a recall or otherwise notifying its purchasers of the problem related to the PROM."
Most software failures don't lead to deaths. Most software projects involve conscious tradeoffs among several factors, including cost, time to completion, richness of the feature set, and reliability. There is nothing wrong with doing this type of business tradeoff, consciously and explicitly, unless you fail to take into account the fact that some of the problems that you leave in the product might cost your customers much, much more than they cost your company. Figure 2 lists some of the external failure costs that are borne by customers, rather than by the company.
|These are the types of costs absorbed by the seller that releases a defective product.||These are the types of costs absorbed by the customer who buys a defective product.|
The point of quality-related litigation is to transfer some of the costs borne by a cheated or injured customer back to the maker or seller of the defective product. The well-publicized cases are for disastrous personal injuries, but there are plenty of cases against computer companies and software companies for breach of contract, breach of warranty, fraud, etc.
The problem of cost-of-quality analysis is that it sets us
up to underestimate our litigation and customer dissatisfaction
risks. We think, when we have estimated the total cost of quality
associated with a project, that we have done a fairly complete
analysis. But if we don't take customers' external failure costs
into account at some point, we can be surprised by huge increased
costs (lawsuits) over decisions that we thought, in our incomplete
analyses, were safe and reasonable.
1 Gryna, F. M. (1988) "Quality Costs" in Juran, J.M. & Gryna, F. M. (1988, 4th Ed.), Juran's Quality Control Handbook, McGraw-Hill, page 4.2.
2 Feigenbaum, A.V. (1991, 3rd Ed. Revised), Total Quality Control, McGraw-Hill, Chapter 7.
3 Gryna, F. M. "Quality Costs" in Juran, J.M. & Gryna, F. M. (1988, 4th Ed.), Juran's Quality Control Handbook, McGraw-Hill, page 4.2. I'm not aware of reliable data on quality costs in software.
4 These are my translations of the ideas for a software development audience. More general, and more complete, definitions are available in Campanella, J. (Ed.) (1990), Principles of Quality Costs, ASQC Quality Press, as well as in Juran's and Feigenbaum's works.
5 Principles of Quality Costs, ASQC Quality Press, Appendix B, "Detailed Description of Quality Cost Elements."
6 "Quality Costs" in Juran, J.M. & Gryna, F. M. (1988, 4th Ed.), Juran's Quality Control Handbook, McGraw-Hill, pages 4.9 - 4.12.
7 The product is scheduled for release on July 1, so your company arranges (far in advance) for an advertising campaign starting July 10. The product has too many bugs to ship, and is delayed until December. All that advertising money was wasted.
8 If the product had to be shipped late because of bugs that had to be fixed, the direct cost of late shipment includes the lost sales, whereas the opportunity cost of the late shipment includes the costs of delaying other projects while everyone finished this one.
9 Note, by the way, that you can reduce external failure costs without improving product quality. To reduce post-sale support costs without increasing customer satisfaction, charge people for support. Switch from a toll-free support line to a toll line, cut your support staff size and you can leave callers on hold for a long time at their expense. This discourages them from calling back. Because these cost reductions don't increase customer satisfaction, the seller's cost of quality is going down, but the customer's is not.
10 This is the cost of cooperating with a government investigation. Even if your company isn't charged or penalized, you spend money on lawyers, etc.
11 Some penalties are written into the contract between the software developer and the purchaser, and the developer pays them if the product is late or has specified problems. Other penalties are imposed by law. For example, the developer/publisher of a computer program that prepares United States taxes is liable for penalties to the Internal Revenue Service for errors in tax returns that are caused by bugs or design errors in the program. The publishers are treated like other tax preparers (accountants, tax lawyers, etc.). See Revenue Ruling 85-189 in Cumulative Bulletin, 1985-2, page 341.
12 I am most definitely not saying that a tactical approach is more practical than an integrated, long-term approach. Gryna notes that there are two common approaches to cost-of-quality programs. One approach involves one-shot studies that help the company identify targets for significant improvement. The other approach incorporates quality cost control into the structure of the business. (Gryna, 1988, in Juran, J. M. & Gryna, F. M. (1988, 4th Ed.), Juran's Quality Control Handbook, McGraw-Hill, pages 4.2 onward.) The one-shot, tactical approach can prove the benefit of the more strategic, long-term system to a skeptical company.
13 Be sensitive to how you do this. If you adopt a tone that says that you think the project manager and the programming staff are idiots, you won't enjoy the long-term results.
14 in Juran, J. M. & Gryna, F. M. (1988, 4th Ed.), Juran's Quality Control Handbook, McGraw-Hill, pages 4.27-4.28.
15 Quality Planning and Analysis (2nd Ed.), McGraw-Hill,pages 30-32. Also, see Brown, M. G., Hitchcock, D. E. & Willard, M. L. (1994), Why TQM Fails and What To Do About It, Irwin Professional Publishing.
16 As he says in Juran, J.M. (1992), Juran on Quality by Design, The Free Press, p. 119, Costs of poor quality "are huge, but the amounts are not known with precision. In most companies the accounting system provides only a minority of the information needed to quantify this cost of poor quality. It takes a great deal of time and effort to extend the accounting system so as to provide full coverage. Most companies have concluded that such an effort is not cost effective. [¶] What can be done is to fill the gap by estimates, which provide managers with approximate information as to the total cost of poor quality and as to where the major areas of concentration are."
17 "Quality Costs" in Juran, J.M. & Gryna, F. M. (1988,4th Ed.), Juran's Quality Control Handbook, McGraw-Hill, pages 4.9 - 4.12.
18 This table is published in Keeton, W. P., Owen, D.G., Montgomery, J. E., & Green, M.D. (1989, 2nd Ed.) Products Liability and Safety, Cases and Materials, Foundation Press, page 841 and Posner, R.A. (1982) Tort Law: Cases and Economic Analysis, Little Brown & Co., page 225.
19 Grimshaw v. Ford Motor Co. (1981), (California Court of Appeal), California Reporter, volume 174, page 348.
20 Southern Reporter, 2nd
Series, volume 592, pages 1054 and 1061.
|Cem Kaner consults on technical and management issues, practices law, and teaches within the software development community. His book, Testing Computer Software, received the Award of Excellence in the Society for Technical Communication's 1993 Northern California Technical Publications Competition. He has managed every aspect of software development, including software development projects, software testing groups and user documentation groups. He has also worked as a programmer, a human factors analyst / UI designer, a salesperson, a technical writer, an associate in an organization development consulting firm, and as an attorney (typically representing customers and software development service providers). He has also served pro bono as a Deputy District Attorney, as an investigator/mediator for Santa Clara County's Consumer Affairs Department, as an Examiner for the California Quality Awards. He holds a B.A. (Math, Philosophy, 1974), a J.D. (1993), and a Ph.D. (Experimental Psychology, 1984) and is Certified in Quality Engineering by the American Society for Quality Control. He teaches at UC Berkeley Extension, and by private arrangement, on software testing and on the law of software quality.|