No. I don’t think so, but I’ll give my argument.

If you reject feedback, you also reject the choice of acting in a way that may bring you abundant success.

― John Mattone

bbf50ad76d36dc0cb78bfba3c5d3870cDespite a relatively obvious path to fulfillment, the estimation of numerical error in modeling and simulation appears to be worryingly difficult to achieve. A big part of the problem is outright laziness, inattention, and poor standards. A secondary issue is the mismatch between theory and practice. If we maintain reasonable pressure on the modeling and simulation community we can overcome the first problem, but it does require not accepting substandard work. The second problem requires some focused research, along with a more pragmatic approach to practical problems. Along with these systemic issues we can deal with a simpler problem, where to put the error bars on simulations, or should they show a bias or symmetric error. I strongly favor a bias.

Implicit in this discussion is an assumption of convergence for a local sequence of calculations. I suspect the assumption is generally a good one, but also prone to failure. One of the key realities is the relative rarity of calculations in the asymptotic range of convergence for methods and problems of interest. The biggest issue is how problems are modeled. The usual way of modeling problems or creating models for physics in problems produces technical issues that inhibit asymptotic convergence (various discontinutiies, other singularities, degenerate cases, etc.). Our convergence theory is predicated on smoothness that rarely exists in realistic problems. This gets to the core of the shortcomings of theory, we don’t know what to expect in these cases. In the end we need to either make some assumptions, collect data and do our best, or do some focused research to find a way.

The basic recipe for verification is simple: make an assumption about the form of the error, collect calculations and use the assumed error model to estimate errors. The assumed error form is A = S_k + C h_k^\alpha where A is the mesh converged solution, S_k is the solution on a grid k, h_k is the mesh density, C is a constant of proportionality and \alpha is the convergence rate. We see three unknowns in this assumed form, A, C and \alpha. Thus we need at least three solutions to solve for these values, or more if we are willing to solve an over-determined problem. At this point the hard part is done, and verification is just algebra and a few very key decisions. It is these key decisions that I’m going to ask some questions about.

The first thing to note is the basic yardstick for the error estimate is the difference between A and the grid solution S_k, which we will call \Delta A.  Notice that this whole error model assumes that the sequence of solutions S_k approaches A monotonically as h_k becomes smaller. In other words all the evidence supports the solution going to A. Therefore the error is actually signed, or biased by this fact. In a sense we should consider A to be the most likely, or best estimate of the true solution as h \rightarrow 0. There is also no evidence at all that the solution is moving the opposite direction. The problem I’m highlighting today is that the standard in solution verification does not apply these rather obvious conclusions in setting the numerical error bar.

The standard way of setting error bars takes the basic measure of error, multiplies it by an engineering safety factor C_s \ge 1, and then centers it about the mesh solution, S_k. The numerical uncertainty estimate is simple,  U_s = C_s \left| \Delta A  \right| . So half the error bar is consistent with all the evidence, but the other half is not. This is easy to fix by ridding ourselves of the inconsistent piece.

slide1The core issue I’m talking about is the position of the numerical error bar. Current approaches center the error bar on the finite grid solution of interest, usually the finest mesh used. This has the effect of giving the impression that this solution is the most likely answer, and the true answer could be either direction from that answer. Neither of these suggestions is supported by the data used to construct the error bar. For this reason the standard practice today is problematic and should be changed to something supportable by the evidence. The current error bars suggest incorrectly that the most likely error is zero. This is completely and utterly unsupported by evidence.

Instead of this impression, the evidence is pointing to the extrapolated solution as the most likely answer, and the difference between that solution, A, and the mesh of interest S_k is the most likely error. For this reason the error bar should be centered on the extrapolated solution. The most likely error is non-zero. This would make the error biased, and consistent with the evidence. If we padded our error estimate with a safety factor, C_s, the error bar would include the mesh solution, S_k and the potential for zero numerical error, but only as a low probability event. It would present the best estimate of the error as the best estimate.

There is a secondary impact of this bias that is no less important. The current standard approach also significantly discounts the potential for the numerical error to be much larger than the best estimate (where the current centering makes the best estimate appear to be low probability!). By centering the error bar on the best estimate we then present larger error as being equally as likely as smaller error, which is utterly and completely reasonable.

The man of science has learned to believe in justification, not by faith, but by verification.

― Thomas Henry Huxley

Why has this happened?

Part of the problem is the origin of error bars in common practice, and a serious technical difference in their derivation. The most common setting for error bars is measurement error. Here a number of measurements are taken and then analyzed to provide a single value (or values). In the most common use the mean value is presented as the measurement (i.e., the central tendency). Scientists then assume that the error bar is centered about the mean through assuming normal (i.e., Gaussian) statistics. This could be done differently with various biases in the data being presented, but truth be told this is rare, as is using any other statistical basis for computing the central tendency and deviations. This point of view is the standard way of viewing an error bar and implicitly plays in the mind of those viewing numerical error. This implicit view is dangerous because it imposes a technical perspective that does not fit numerical error.

The problem is that the basic structure of uncertainty is completely different with numerical error. A resolved numerical solution is definitely biased in its error. An under-resolved numerical solution is almost certainly inherently biased. The term under resolved is simply a matter of how exacting a solution one desires, so for the purposes of this conversation, all numerical solutions are under-resolved. The numerical error is always finite and if the calculation is well behaved, the error is always a bias. As such the process is utterly different than measurement error. With measurements there is an objective reality that is trying to be sensed. Observations can be biased, but generally are assumed to be unbiased unless otherwise noted. We have fluctuations in the observation or errors in the measurement itself. These both can have a distinct statistical nature. Numerical error is deterministic and structured, and has a basic bias through the leading order truncation error. As a result error bars from both sources should be structurally different. There are simply not the same thing and should not be treated as such.

The importance of this distinction in perspective is the proper accounting for sources and impact of uncertainty in modeling and simulation. Today we suffer most greatly from some degree of willful ignorance of uncertainty, and when it is acknowledged, too narrow a perspective. Numerical error is rarely estimated, assumed away and misrepresented even when it is computed. In the best work available, uncertainty is tackled as being dominantly epistemic uncertainty associated with modeling parameters (nominally subgrid or closure models). Large sources of uncertainty are defined by numerical error, problem modeling assumptions, model form error, and experimental uncertainty to name the big ones. All of these sources of uncertainty are commonly ignored by the community without much negative feedback, this needs to be somewhere for progress.

Science is a system of statements based on direct experience, and controlled by experimental verification. Verification in science is not, however, of single statements but of the entire system or a sub-system of such statements.

― Rudolf Carnap

 

Advertisements