It is dangerous to be right in matters on which the established authorities are wrong.
Last week I received a question via email that prompted this post. It proposed that the title of the post is true. It talked about the benefits of pushing the envelope with high performance computing. The gist of the thought is that by pushing the envelope with computing we can effectively use the mainstream high end computing resources better. This is true without a doubt. It is a benefit of having a bleeding edge research program in any area. A better question is whether it is the most beneficial way to allocate our current efforts.
Everything in excess is opposed by nature.
The ability of such a program to impact the World positively is a much more difficult and nuanced question. For high performance computing we have seen decades long focus on the power and speed of the hardware that has fueled a growth in peak computing speed consistent with Moore’s law. Unfortunately a host of essential capabilities for realizing this computing power as a scientific capability have not been similarly supported. Without these other capabilities such as physical models, solution methods, algorithms, the computing hardware is nothing more than a very expensive way to use electricity. The very things that make computers really useful for the purposes of modeling and simulation are the things we have not invested in for these same decades.
The distance between insanity and genius is measured only by success
― Ian Fleming
The issue with it is that this benefit does not exist in a vacuum. There are limits to the financial and human resources that may be devoted to the objective of “predictive” modeling and simulation. My politically incorrect assertion is that the focus on high performance computing hardware is a suboptimal approach to achieving the end result of maximizing the capability for modeling and simulation. The devotion to progress in hardware is sapping the resources that might be applied to attacking this problem in a more balanced manner.
If we have no heretics we must invent them, for heresy is essential to health and growth.
― Yevgeny Zamyatin
This imbalance is primarily exemplified by the failure to invest in people, experiments and models. When I speak of investing in people, it goes far beyond simply paying people. Investing in people means creating systems where people can develop and grow in their capability while feeling safe and secure to take huge risks. Talented people who take risks are necessary for progress, and without such risk-taking progress stagnates. Without taking risks we cannot develop talent, the two are intertwined.
An expert is someone who knows some of the worst mistakes that can be made in his (her) subject, and how to avoid them.
– Werner Heisenberg
We have destroyed the vitality of our experimental sciences, which further amplifies the destruction of our scientific staff. Experimental science is absolutely necessary to advance science. This has the knock-on effect of undermining the creation of new, better models for science. Having the World’s fastest computer cannot replace any of these shortcomings.
It doesn’t matter how beautiful your theory is, it doesn’t matter how smart you are. If it doesn’t agree with experiment, it’s wrong.
― Richard Feynman
A big part of the problem is that aspects of modeling and simulation closer to the modeling are starved. The modeling associated with high performance computing is static and relatively free of progress. We are not progressing toward introducing new models into codes and ultimately practical use in a healthy way. The same can be said for solution methods that provide either more powerful or effective ways of solving models. These methods then provide the impetus for algorithms that systematically provide means of solution. Our current computing emphasis is geared toward efficiently delivering existing algorithmic solutions for existing methods for existing models.
History warns us … that it is the customary fate of new truths to begin as heresies and to end as superstitions.
― Thomas Henry Huxley
Inadequacies in models are simply allowed to persist while the fiction of their rescue by more powerful computing platforms a false promise. The real scientific answer to this issue is that a more powerful computer, better software, better algorithms or better methods cannot save a model that is incorrect. Despite this maxim we keep attacking modeling and simulation dominantly through computing hardware.
…a more powerful computer, better software, better algorithms or better methods cannot save a model that is incorrect.
The fastest computer has even taken a page from Cold War nationalism with the “missile gap” being replaced by the “supercomputer gap”. Chinese prowess in computing is demonstrated by their supercomputer and funding is supposed to follow the American effort to regain the crown of fastest. I, for one, am far more worried about the Chinese and Russian investments in human resources in modeling, methods and algorithms than computers. By all appearances these investments are significant. The American public should be far more worried by the encroachment of mediocrity into the research staff at our National Labs than our lack of the fastest computers.
In the republic of mediocrity, genius is dangerous.
― Robert G. Ingersoll
The management of our National research programs and devotion to intellectually questionable priorities is by far a greater threat to our National security than anything our adversaries are doing. A perfect example of the problem is the increasingly legacy nature of our computer codes. We looked at the local history of one type of code and noted that up until 1990 we had a new code every five years. Since 1990 we have simply kept the same old codes. This is 25 years with the same code with the same methods and same approach. We have basically lost an entire generation of staff. We have lost a generation of progress and research. The models and methods are frozen in time. This is a recipe for mediocrity at best, disaster at worst.
Progress is born of doubt and inquiry. The Church never doubts, never inquires. To doubt is heresy, to inquire is to admit that you do not know—the Church does neither.
― Robert G. Ingersoll
So, yes, having leading edge computing is a great, wonderful and important thing for the country. It’s true for any country desiring international leadership. Getting to properly defining what leading edge computing is actually comprised of becomes difficult. A completely naive and incorrect way to define leading edge is having the fastest or most powerful computer on Earth. Everyone knows what a fast computer is, but a powerful computer is a more subtle question. I would argue that in many respects my iPhone is more powerful and useful than virtually any supercomputer I’ve used. The problem in defining powerful is the limited utility of supercomputers. Supercomputers are important for solving scientific problems, which are necessarily limited in context.
The riskiest thing we can do is just maintain the status quo.
― Bob Iger
Moreover, this effort for leading edge computing lies in a resource constrained trade space and the focus on hardware leaves other efforts starved for funding or focus. Even this discussion leaves most of the important nuance untouched, the dependence on people and their talent. The issues around the efficacy of the HPC efforts are subtle and far more nuanced than the mere power of the computer. A powerful supercomputer is useless without talented people to use it. The problem in the United States is that people are something we have chronically and systematically under-invested in. Our universities are in decline and part of a vastly corrupt system that underserves the public at a massive cost. The consequences of this decline in education are then amplified by the destruction of the social contracts associated with post-educational work.
Unhappiness lies in that gap between our talents and our expectations.
― Sebastian Horsley
Employees are viewed as commodities and infinitely replicable, even at National Labs. The lifetime employment necessary for deep sustainable expertise has been replaced by an attitude more appropriate for Wal-Mart. Over the past couple of decades the sort of strong scientific leadership once provided by the National Labs has been replaced by Lab employees who are little more than “sheeple” who bend to political will rather than speak up and offer their expertise instead of politically correct pablum. Today the Labs simply do what they are told. Their spirit has been beat out of them. I might even be so bold and to say that the attempt to lead in scientific computing says more about our lack of scientific leadership than our commitment to it.
The smart way to keep people passive and obedient is to strictly limit the spectrum of acceptable opinion, but allow very lively debate within that spectrum….
― Noam Chomsky
If the models, methods, and codes used on our fastest computer are lacking, the computer’s value is diminished to the point of being worthless. A computer will provide results that are as good as the codes running on it. If the people running the problems on the computer are similarly lacking, the computer’s value is diminished as well. The computer is only answering questions as good as the people asking the questions. I believe that we have systematically failed to make investments in the models; methods, codes and people sufficient to make the focus on computing power pay off. We have created a new generation of legacy codes (just because its written in C++ does not keep it from being a legacy code!), with legacy models and methods and a staff that cannot fully understand the codes or calculations they are running. The fiction that all we need to do is refine the mesh and run calculations on bigger computers to predict nature continues to hold sway.
Every truth in this world stretched beyond its limits will become a false doctrine.
― K.P. Yohannan
This is a situation that the mismanagement of the labs has created. The DOE has done the same thing to its Labs that DOD did. DOD foolishly destroyed its research labs 30 years ago, and over the last 20 years, DOE has followed a similar path toward mediocrity. The national resource of these Labs is being allowed to fade and wither. We have allowed the Labs to atrophy. Our approach to high performance computing is but one example of this. The situation is even worse when you look at what we have done to our experimental sciences. Under these conditions having the lead in computing hardware will do little actually support our national security because we have failed to keep competence in the fundamentals necessary for efficacy.
Change almost never fails because it’s too early. It almost always fails because it’s too late.
― Seth Godin
In other words for the hardware to really matter in the delivery of predictive simulation and modeling, everything upstream of the machine needs to be right. This includes the people. We have failed to invest in leading edge technology in the very things that make the supercomputer valuable. We have a skewed and unbalanced view of how computing works, which allows the justification of our current programmatic path, but fails to deliver true progress.
Heretics are the new leaders. The ones who challenge the status quo, who get out in front of their tribes, who create movements.
― Seth Godin