Leadership is fundamentally about credibility.
― Rick Crossland
Under the best of circumstances we would like to confidently project credibility for the modeling and simulation we do. Under the worst of circumstances we would have confidence in modeling and simulation without credibility. This is common. Quite often the confidence is the product of arrogance or ignorance instead of humility and knowledge. This always manifests itself with a lack of questioning in the execution of work. Both of these issues are profoundly difficult to deal with and potentially fatal to meaningful impact of modeling and simulation. These issues are seen quite frequently. Environments with weak peer review contribute to allowing confidence with credibility to persist. The biggest part of the problem is a lack of pragmatic acceptance of modeling and simulation’s intrinsic limitations. Instead we have inflated promises and expectations delivered by over confidence and personality rather than hard nosed technical work.
When confidence and credibility are both in evidence, modeling and simulation is empowered to be impactful. It will be used appropriately with deference to what is and is not possible and known. When modeling and simulation is executed with excellence and professionalism along with hard-nosed assessment of uncertainties, using comprehensive verification and validation, the confidence is well grounded in evidence. If someone questions a simulations result, answers can be provided with well-vetted evidence. This produces confidence in the results because questions are engaged actively. In addition the limitations of the credibility are well established, and confidently be explained. Ultimately, credibility is a deeply evidence-based exercise. Properly executed and delivered, the degree of credibility depends on honest assessment and complete articulation of the basis and limits of the modeling.
When you distort the truth, you weaken your credibility.
― Frank Sonnenberg
One of the dangers of hard-nosed assessment is the tendency for those engaged in it to lose confidence in the work. Those who aggressively pursue credibility assessment tend to be cynics and doubters. They are prone to pessimism. They usually project doubt and focus on limitations of the modeling instead of confidence where it may be used. One of the hardest tricks of credibility assessment is pairing excellence in the execution of the work with an appropriate projection of confidence. The result is a mixed message where confidence is projected without credibility, and credibility is projected without confidence. Neither serves the purpose of progress in the impact of modeling and simulation.
One of the major sins of over-confidence is flawed or unexamined assumptions. This can be articulated as “unknown knowns” in the famously incomplete taxonomy forwarded by Donald Rumsfeld in his infamous quote. He didn’t state this part of the issue even though it was the fatal flaw in the logic of the Iraqi war in the aftermath of 9/11. There were basic assumptions about Hussein’s regime in Iraq that were utterly false, and these skewed the intelligence assessment leading to war. They only looked at information that supported the conclusions they had already drawn or wanted to be true. The same faulty assumptions are always present in modeling. Far too many simulation professionals ignore the foundational and unfounded assumptions in their work. In many cases assumptions are employed without thought or question. They are assumptions that the community has made for as long as anyone can remember and simply cannot be questioned. This can include anything from the equations solved, to the various modeling paradigms applied as a matter of course. Usually these are unquestioned and completely unexamined for validity in most credibility assessments.
This is an immensely tricky thing to execute. The standard assumptions are essential to managing complexity and making progress. That said, it is a remarkably difficult and important task to detect when the assumptions become limiting. More succinctly put, the limitations of the standard assumptions need to be thought-through and tested. Usually these assumptions can only be tested through removing everything else from the field and doing very hard work. It is so much easier to simply stay the course and make standard assumptions. In many cases the models have been significantly calibrated to match existing data, and new experiments or significantly more accurate measurements are needed to overturn or expose modeling limitations. Moreover the standard assumptions are usually unquestioned by peers. Questions are often met with ridicule. A deeply questioning assessment requires bravery and fortitude usually completely lacking from working scientists and utterly unsupported by our institutions.
Another manner for all of this to unfold is unwarranted confidence. Often this is couched in the form of arrogant perspectives where the proof of credibility is driven by personality. This proof by authority is incredibly common and troubling to dislodge. In many cases personal relationships to consumers of simulations are used to provide confidence. People are entrusted with the credibility and learn how to give their customer what they want. Credibility by personality is cheap and requires so much less work plus it doesn’t raise any pesky doubts. This circumstance creates an equilibrium that is often immune to scientific examination. It is easier to bullshit the consumers of modeling and simulation results than level with them about the true quality of the work.
The credibility of the teller is the ultimate test of the truth of a proposition.
― Neil Postman
More often than not honest and technically deep peer review is avoided like a plague. If it is imposed on those practicing this form of credibility, the defense of simulations takes the personal form of attacking the peer reviewers themselves. This sort of confidence is a cancer on quality and undermines any progress. It is a systematic threat to excellence in simulation, and must be controlled. It is dangerous because it is effective in providing support for modeling and simulation along with the appearance of real World impact.
One of the biggest threats to credibility is the generation of the lack of confidence honesty has. Engaging deeply and honestly in assessment of credibility is excellent at undermining confidence. Almost invariably the accumulation of evidence regarding credibility endows the recipients of this knowledge with doubt. These doubts are healthy and often the most confident people are utterly ignorant of the shortcomings. The accumulation of evidence regarding the credibility should have a benefit for the confidence in how simulation is used. This is a problem when those selling simulation oversell what it can do. The promise of simulation has been touted widely as transformative. The problem with modeling and simulation is its tangency to reality. The credibility of simulations is grounded by reality, but the uncertainty comes from both modeling, but also the measured and sensed uncertainty with our knowledge of reality.
The dynamic and tension with confidence and credibility should be deeply examined. When confidence is present without evidence, people should be deeply suspicious. A strong culture of (independent) peer review is an antidote to this. Too often these days the peer review is heavily polluted by implicit conflicts of interest. The honesty of peer review is hampered by an unwillingness to deal with problems particularly with respect to modification of the expectations. Invariably modeling and simulation has been oversold and any assessment will provide bad news. In today’s World we see a lot of bad news rejected, or repackaged (spun) to sound like good news. We are in the midst of a broader crisis of credibility with respect to information (i.e. fake news), so the issues with modeling and simulation shouldn’t be too surprising. We would all be well served by a different perspective and approach to this. The starting point is a re-centering of expectations, but so much money has been spent using grossly inflated claims.
Belief gives knowledge credibility.
― Steven Redhead
So what should we expect from modeling and simulation?
Modeling and simulation is a part of the scientific process and subject to its limits and rules. There is nothing magic about simulation that unleashes modeling from its normal limitations. The difference that simulation makes is the ability to remove the limitations of analytical model solution. Far more elaborate and accurate modeling decisions are available, but carry other difficulties due to the approximate nature of numerical solutions. The tug-of-war intellectually is the balance between modeling flexibility, nonlinearity and generality with effects of numerical solution. The bottom line is the necessity of assessing the uncertainties that arise from these realities. Nothing releases the modeling from its fundamental connection to validity grounded in real world observations. One of the key things to recognize is that models are limited and approximate in and of themselves. Models are wrong, and under a sufficiently resolved examination will be invalid. For this reason an infinitely powerful computer will ultimately be useless because the model will become invalid at some resolution. Ultimately progress in modeling and simulation is based on improving the model. This fact is ignored by computational science today and will result wasting valuable time, effort and money chasing quality that is impossible to achieve.
Bullshit is a greater enemy of the truth than lies are.
In principle the issue of credibility and confidence in modeling and simulation should be based on evidence. Ideally this evidence should be quantitative with key indicators of its quality included. Ideally, the presence of the evidence should bolster credibility. Instead, paradoxically, evidence associated with the credibility of modeling and simulation seems to undermine credibility. This is a strong indicator that claims about the predictive power of modeling and simulation has been over-stated. This is a nice way of saying this is usually a sign that the quality is actually complete bullshit! We can move a long way toward better practice by simply recalibrating our expectations about what we can and can’t predict. We should be in a state where greater knowledge about the quality, errors and uncertainty in modeling and simulation work improves our confidence.
If you can’t dazzle them with brilliance, baffle them with bullshit!
– W.C. Fields
Part of the issue is the tendency for the consumers of modeling and simulation work to not demand evidence to support confidence. This evidence should always be present and available for scrutiny. If claims of predictive power are made without evidence, the default condition should be suspicion. The various sources of error and uncertainty should be drawn out, and quantified. There should be estimates based on concrete evidence for the value of uncertainty for all sources. Any uncertainty that is declared to be zero or negligible must have very specific evidence to support this assertion. Even more important any claims of this nature should receive focused and heavy scrutiny because they are likely to be based on wishful thinking, and often lack any evidentiary basis.
One of the issues of increasing gravity in this entire enterprise is the consumption of results using modeling and simulation by people unqualified to judge the quality of the work. The whole enterprise is judged to be extremely technical and complex. This inhibits those using the results from asking key questions regarding the quality of the work. With the people producing modeling and simulation results largely driven by money rather than technical excellence, we have the recipe for disaster. Increasingly, false confidence accompanies results and snows the naïve consumers into accepting the work. Often the consumers of computational results don’t know what questions to ask. We are left with quality being determined more by flashy graphics and claims about massive computer use than any evidence of prediction. This whole cycle perpetuates an attitude that starts to allow viewing reality as more of a video game and less like a valid scientific enterprise. Over inflated claims of capability are met with money to provide more flashy graphics and quality without evidence. We are left with a field that has vastly over-promised and provided the recipe for disaster.
We now live in a world where counter-intuitive bullshitting is valorized, where the pose of argument is more important than the actual pursuit of truth, where clever answers take precedence over profound questions.
― Ta-Nahisi Coates