One of the big problems that the entire V&V enterprise has is the sense of imposition on others. Every simulation worth discussing does “V&V” at some level, and almost without exception they have weaknesses. Doing V&V “right” or “well” is not easy or simple. Usually, the proper conduct of V&V will expose numerous problems with a code, and/or simulation. It’s kind of like exposing yourself to an annual physical; its good for you, but you might have to face some unpleasant realities. In addition, the activity of V&V is quite broad and something almost always slips between the cracks (or chasms in many cases).
To deal with this breadth, the V&V community has developed some frameworks to hold all the details together. Sometimes these frameworks are approached as prescriptions for all the things you must do. Instead I’ll suggest that these frameworks are not recipes, nor should they be thought of as prescriptions. They are “thou should,” not “thou shalt,” or even “you might.“
Several frameworks exist today and none of them is fit for all purposes, but all of them are instructive on the full range of activities that should be at least considered, if not engaged in.
CSAU – Code Scaling Assessment and Uncertainty developed by the Nuclear Regulatory Committee to manage the quality of analyses done for power plant accidents. It is principally applied to thermal-fluid (i.e. thermal-hydraulic) phenomena that could potentially threaten the ability of nuclear fuel to contain radioactive products. This process led the way, but has failed in many respects to keep up to date. Nonetheless it includes processes and perspectives that have not been fully replicated in subsequent work. PCMM is attempting to utilize these lessons in improving its completeness.
PCMM – Predictive Capability Maturity Model developed at Sandia National Laboratories for the stockpile stewardship program in the last 10 years. As such it reflects the goals and objectives of this program and Sandia’s particular mission space. It was inspired by the CMMI developed by Carnegie Mellon University to measure software process maturity. PCMM was Sandia’s response to calls for greater attention to detail in defining the computational input into quantitative margins and uncertainty (QMU), the process for nuclear weapons’ certification completed annually.
CAS – Credibility Assessment Scale developed by NASA. They created a similar framework to PCMM for simulation quality in the wake of the shuttle accidents and specifically after Columbia where simulation quality played an unfortunate role. In the process that unfolded with that accident, the practices and approach to modeling and simulation was found to be unsatisfactory. The NASA approach has been adopted by the agency, but does not seem to be enforced. This is a clear problem and potentially important lesson. There is a difference between an enforced standard (i.e., CSAU) and one that comes across as well intentioned, but powerless directives. Analysis should be done with substantial rigor when lives are on the line. Ironically, formally demanding this rigor may not be the most productive way to achieve this end.
PMI, Predictive Maturity Index developed at Los Alamos. This framework is substantially more focused upon validation and uncertainty, and IMHO it gives a bit lax with respect to the code’s software and numerical issues. In my view, these aspects are necessary to focus upon given advances in the past 25 years since CSAU came into us in the nuclear industry.
Computational simulations are increasingly used in our modern society to replace some degree of expensive or dangerous experiments and tests. Computational fluid and solid mechanics are ever more commonplace in modern engineering practice. The challenge of climate change may be another avenue where simulation quality is scrutinized and could benefit from a structured, disciplined approach to quality. Ultimately, these frameworks serve the role of providing greater confidence (faith) in the simulation results and their place in decision-making. Climate modeling is a place where simulation and modeling plays a large role, and the decisions being made are huge.
The question lingers in the mind, “what can these frameworks do for me?” My answer follows:
- V&V and UQ are both deep fields with numerous deep subfields. Keeping all of this straight is a massive undertaking beyond the capacity of most professional scientists or engineers.
- Everyone will default to focusing on where they are strong and comfortable, or interested. For some people it is mesh generation, for others it is modeling, and for yet others it is analysis of results. Such deep focus may not lead (or is not likely to lead) to the right sort of quality. Where quality is needed is dependent upon the problem itself and how the problem’s solution is used.
- These are useful outlines for all of the activities that a modeling and simulation project might consider. Project planning can use the frameworks to develop objectives and subtasks, prioritize and review.
- These are menus of all the sort of things you might do, not all the things you must do.
- They provide a sequenced set of activities, prepared in a sequenced rational manner with an eye toward what the modeling and simulation is used for.
- They help keep your activities in balance. They will help keep you honest.
- You will understand what is fit for purpose, when you have put too much effort into a single aspect of quality.
- V&V and UQ are developing quickly and the frameworks provide a “cheat sheet” for all of the different aspects.
- The frameworks flexibility is key, not every application necessarily should focus on every quality aspect, or apply every quality approach in equal measure.
10. Validation itself is incredibly hard in both breadth and depth. It should be engaged in a structured, thoughtful manner with a strong focus on the end application. Validation is easy to do poorly.
11. The computational science community largely ignores verification of code and calculations. Even when it is done, it is usually done poorly.
12. Error estimation and uncertainty too rarely include the impact of numerical error, and estimate uncertainty primarily through parametric changes in models.
13. Numerical error is usually much larger than acknowledged. Lots of calibration is actually accounting for the numerical error, or providing numerical stability rather than physical modeling.