One challenge we frequently hear hurled at the practitioners and champions of independent verification and validation (IV&V) and quality assurance (QA) is, “How can you justify the cost?” Or sometimes, “How much did you actually save the program?” In other words, “Prove your value!” That’s a difficult request.
The idea of IV&V and QA is to find problems – large or small – while the problem can be fixed and/or its impact can be controlled. The findings of a good IV&V team range in severity from typos that may have led to minor embarrassment to functional errors that could have financial and other implications to what we refer to as the “CNN Moment.”
The cost of a typo ‘escaping’ in to production is usually – but not always – going to be trivial. One extreme example I like to think of, even though it’s actually a myth, is the legend of the Chevy Nova. As the story goes, “no va” in Spanish means “it doesn’t go” which led to Chevrolet having to pull the car from the Latin American market due to a poor perception among potential buyers based on its name. If the story were true (it isn’t) it’s the kind of thing an IV&V team may very well have spotted. What would that have been worth to the General Motors Corp. in the alternate reality where it really did happen? The cost could be calculated after the fact – pulling a model out of a market would involve a lot of dollars, not to mention a black eye for the brand which would be harder to quantify. But say early in the concept phase for the car someone from IV&V said, “You know you can’t use Nova as the model name in the Spanish-speaking markets.” Normally, no one is going to stop there and say: “Ok, IV&V just reported a finding and we’ve run numbers to calculate how much GM would have spent if they had not caught that problem. It turns out they saved us $XYZ million.”
Of course, there are the CNN Moments when the impact becomes unavoidable. One such moment is underway right now in early October 2013. HealthCare.gov, the sign up system for the Obamacare insurance program went live October 1. Despite having about three years to develop the system, it experience significant problems right out of the gate. This led to, literally, a CNN Moment for the Obama administration, the Dept. of Health and Human Services, and several government contractors involved in developing the system (http://www.cnn.com/2013/10/08/politics/obamacare-website-glitches/index.html?iref=allsearch ). It’s evident that IV&V was not part of this program. It’s further evident that the contractors themselves had no effective QA practice as they developed the system. Several third parties have conducted analyses of the web site based on the tip of the iceberg that’s visible to the public and noted a wide range of problems (here’s one such analysis: http://online.wsj.com/news/articles/SB10001424052702304441404579119740283413018 ). Some companies – competitors of the developing contractors – offered their services “for free” to help fix the system (http://www.appdynamics.com/blog/2013/10/11/appdynamics-offers-to-monitor-healthcare-gov-for-free/#! ). If HHS and the developers could wind back the clock, how much do you think they would value an IV&V program to oversee this critical system? The bottom line is that calculating the value of IV&V is hard – until you find yourself in the situation where you didn’t use it and CNN is calling asking for a statement.