The proper way to say the title of this talk is with more than a bit of distain. Too often, I have encountered a disturbingly negative attitude toward V&V and those who practice it. I think it is time for us to shoulder some of the blame and rethink our approach to engaging other scientists and engineers on the topic of modeling and simulation (M&S) quality.
V&V should be an easy sell to the scientific and engineering establishment. It hasn’t been, it has been resisted at every step. V&V is basically a rearticulation of the scientific method we all learn, use and ultimately love and cherish. Instead, we find a great deal of animosity toward V&V, and outright resistence to including it as part of the M&S product. To some extent it has been successful in growing as a discipline and focus, but too many barriers still exist. Through hard learned lessons I have come to the conclusion that a large part of the reason is the V&V community’s approach. For example, one of the worst ideas the V&V community has ever had is “independent V&V”. In this model V&V comes in independently and renders a judgment on the quality of M&S. It ends up being completely adversarial with the M&S community, and a recipe for disaster. We end up less engaged and hated by those we judge. No lasting V&V legacy is created through the effort. The M&S professionals treat V&V like a disease and spend a lot of time trying to simply ignore or defeat it. This time could be better spent improving the true quality, which ought to be everyone’s actual objective. Archetypical examples of this appraoch in action are federal regulators (NRC, the Defense Board…). This idea needs to be modified into something collaborative where the M&S professions end up owning the quality of their work, and V&V engages as a resource to improve quality.
The fact is that everyone doing M&S wants to do the best job they can, but to some degree don’t know how to do everything. In a lot of cases they haven’t even considered some of the issues we can help with. V&V expertise can provide knowledge and capability to improve quality if they are welcome and trusted. One of the main jobs of V&V should be build trust so that they might provide their knowledge to important work. In sense, the V&V community should be quality “coaches” for M&S. Another way the V&V community can help is to provide appropriately leveled tools for managing quality. PCMM can be such a tool if its flexibility is increased. Most acutely, PCMM needs a simpler version. Most modeling and simulation professionals will do a very good job with some aspects of quality. Other areas of quality fall outside their expertise or interest. In a very real sense, PCMM is a catalog of quality measures that could be taken. Following the framework helps M&S professionals keep all the aspects of quality in mind and within reach. The V&V community can then provide the necessary expertise to carry out a deeper quality approach.
If V&V allows itself to get into the role of judge and jury on quality, progress will be poor. V&V’s job is to ask appropriate questions about quality as partners with M&S professionals interested in improving the quality of their work. By taking this approach we can produce a M&S future where quality continuously improves.