Legends don’t have to make sense. They just have to be beautiful. Or at least interesting.
― Terry Pratchett
Along with the purchase of a big computer and the press releases trumpeting its massive power comes another form of systemic bullshit, the heroic calculation. Such heroic calculations provide massive insight into the World through this profoundly important scientific instrument, or so we are told. It is the largest simulation ever of these particular exotic phenomena and provides new perspectives as surely as the expensive new microscope or powerful telescope. The problem with these massive calculations is that this profound capability is a myth and largely complete and utter bullshit. The heroic calculation is an enormous pain-in-the-ass to perform, which justifies the heroism, but the utility of the calculation is completely over-stated. It is a stunt to “show” that the computer works, and sell its importance through propagating bullshit.
The fundamental cause of the trouble is that in the modern world the stupid are cocksure while the intelligent are full of doubt.
― Bertrand Russell
The calculation is usually characterized by how many computational elements it contains especially if it sounds like a lot (billions, so it must be great!), and how many floating operations it took (trillions and trillions, so of course its pretty awesome). These big numbers are completely meaningless and simply exist to sell the massiveness of the whole thing to unwitting lay people. Even more they say nothing whatsoever about the quality or utility of the calculation. These massive hero calculations are rarely assessed in any systematic way that defines any sense of uncertainty or accuracy. They are simply great because they are so fucking massive and huge. They are great because they are computed on this massive computer that cost a shitload of money too.
These calculations do have a utility in their trailblazing nature, but it is limited. These calculations shake out the kinks from using a new computing platform, or working at the scale that these calculation demand. As vehicles for doing science they are generically terrible, and the press releases touting the scientific achievements are bullshit. Instead the honest view of what is achieved is a voyage into the wilds of computing. The computation is done with shaky code, shaky compilers, and shaky operating systems on a shaky computer. They are achievements of will and patience too, but not science or engineering. Some day five years down the road calculation of the scale done for these heroic calculations become common. This is their achievement and it is important, but just not to the level we promote today.
Perseverance is the act of true role models and heroes.
― Liza M. Wiemer
These calculations are largely just marketing for the expensive computers. As vehicles for science these calculations are largely useless. Science needs to be done properly. Proper science does not mean one big massive calculation, proper science means a whole bunch of smaller calculations whose purpose is to carefully study a problem and determine the quality of the simulation. One calculation is never the answer; it is the ensemble of many calculations that make for good computational science. If you see a massive calculation touted as a scientific achievement your mind should immediately ask the question, how do I know its any good? Are those conducting said big calculations careful and self-critical?
The bigger the dream, the better the story.
― Richelle E. Goodrich
A worse side effect of the hero calculation is the distortion of the scientific process to both the public, but the scientific community at large. Far too much modeling and simulation is conducted in this hero mode. The standard for quality from the hero mode calculations is unfortunately the common approach for science to proceed. It makes for lousy science and worse engineering. It is common to see the heroic stunt calculation lauded by management as a stunning achievement despite it being an intrinsically poor conduct of simulation and modeling. These stunts rarely have any verification, validation or uncertainty quantification done on them. They are simply supposed to be accurate by virtue of their massiveness. They are simply the money shot for the massive computer we just spent so much on.
The problem is the nature of quantifying the uncertainties and accuracy of a modeling exercise conducted through simulation. These hero calculations are one-off exercises. They are not part of simulation campaigns that produce results that might be utilized effectively as part of quality work. They can be useful “what if” explorations of phenomena or hypotheses, but whatever the hero calculations find needs to be scrutinized heavily. A full spectrum of verification, validation and uncertainty quantification is needed to bring the hero calculation’s finding into repute. This process is science or engineering done right and its too boring for the sort of bullshit press releases we market our programs through.
A good example comes in the creation of models for complex systems. The common approach is to hand craft the model to utilize every bit of available resource and include every detail possible. This creates models that can only be run once or maybe a handful of times. The modeling approach does not lend itself to the creation of consistent models that have a coarser representation. This coarser representation could be used to assess accuracy (through solution verification), or conduct detailed uncertainty estimation. The detailed model simply can’t be used this way. It uses so much of the available computing resource that it can’t be refined either.
Don’t try to follow trends. Create them.
― Simon Zingerman
It may be the best single calculation that might be conducted, but its quality is unknown. The key question is what is important, the unknown quality of the single calculation (an presumed to be good one) or a sequence of presumably lower quality, but fully assessed calculations? The scientific method would come down firmly on the side of the assessed lower quality simulations. Their assessment makes them useful and reliable, while the presumably superior single calculation is an unknown entity.
In other words, scientists don’t concentrate on what they know, which is considerable but also miniscule, but rather on what they don’t know. The one big fact is that science traffics in ignorance, cultivates it, and is driven by it.
― Stuart Firestein
In the end we need to wean ourselves from our addiction to the fiction of the hero calculation. These calculations make for good press releases, but lousy science and even worse engineering. They distort modeling and simulation into something it is not, and certainly something it shouldn’t be. It’s high time to kill the hero, reject the legend and replace them with hardworking foot soldiers. We need to recognize that the hero calculation has a limited value, which is primarily associated with its trailblazing character in computing, but not its scientific value.
Superficiality is the curse of the modern world.
― Matthew Kelly