Science is what we have learned about how to keep from fooling ourselves.

― Richard Feynman

Last week was another trip to the Multimat conference, a biannual meeting of scientists who solve the equations of multiple material flows under highly energetic and compressible conditions. I’ve always greeted the meeting with great enthusiasm and returned to work bristling with new ideas and inspiration. In many ways this community is the tip of the intellectual spear for modeling and simulation capability.  It also marks an anniversary of sorts, four years of blogging.  My very first post here was about the San Francisco edition of the meeting that coincided with my 50th birthday ( Two years ago we had a wonderful meeting in Wurzburg Germanyfad6939fd02149c8aa33953ec9789f41 ( Every meeting was wonderful and this was no exception, except in one very important and perhaps essential regard; the meeting seemed devoid of the usual exciting intellectual inspiration. What happened to the big ideas that flowed so easily in every previous meeting? Is it my imagination, or have the exciting new ideas dried up?

Do stuff. be clenched, curious. Not waiting for inspiration’s shove or society’s kiss on your forehead. Pay attention. It’s all about paying attention. attention is vitality. It connects you with others. It makes you eager. stay eager.

― Susan Sontag

This assessment might seem rather harsh, but upon reflecting on the previous meetings, it holds up under considerable scrutiny. Each previous meeting was full of moments where you are watching someone else’s talk and thinking, “I wish I’d thought of that, this is brilliant”. This is exactly what makes conferences so wonderful and important to attend; you get to cherry pick people’s best ideas accumulated at great effort all at once. In the moment these ideas seem like Athena springing fully formed from Zeus’ brow! Your colleagues get to look like the geniuses they are and present their most creative and ingenious thoughts in an intellectual banquet (, The reason for attending conferences isn’t to give talks; it is to learn new things taught by the smartest people you know. It is to meet and let ideas breed openly over laughter, food and drinks. You give talks as an act of repayment for the knowledge you are granted byimgres being in the audience. Giving talks is pretty low on the list of reasons, but not in the mind of our overlords, which starts to get at the problems I’ll discuss below. Given the track record of this meeting my expectations were sky-high, and the lack of inspiring ideas left me slightly despondent.

A few more thoughts about the meeting are worth pointing out before getting to the dialog about fresh ideas, their importance and postulates for their recent absence. The meeting is attended by a collection of computational scientists (mathematics, physics, engineering,…) dominated by the nuclear “club”. This means American, French and British with a smattering of Russians and Chinese – who couldn’t come this year for undisclosed reasons. These scientists for the most part work at their nation’s respective nuclear weapons’ labs. Occasional others attend like Israelis (an unacknowledged member of the club) along with a handful of Czechs, Italians, and Germans. As such the meeting serves as a proverbial checkup on the intellectual health of this important area of science at the West’s nuclear weapons Labs. This year’s checkup should give everyone pause, the state of health is declining. There is a real lack of creative energy surrounding the heart of our most important codes. Many important codes are built around a powerful hydro-solver that produces accurate, physically relevant solutions to the multi-material “hydrodynamic” equations. Previous meetings have seen a healthy resurgence of new ideas, but that upswing seems to have come to a staggering halt. These labs have also provided a deep well of inspired research that has benefited the broader scientific community including weather, climate, astrophysics and a broad swath of engineering use of computation.

In my opinion the reasons for this halt in creative energy are simple and straightforward. The foolhardy worldwide push for exascale computers is sucking the air out of the room. It is gobbling up all the resources and attention leaving nothing for new ideas. This completely obsessive and unwise focus on the hardware is attempting to continue – the already dead – Moore’s law. This push is draining the community of vitality, resources and focus. The reasons for the push are worth considering because they help define the increasingly hostile nature of the modern world toward science. The computers being build for the future are abysmal to use and the efforts to move our codes to them are sucking all the energy from the Labs. Nothing is left of creative work; nothing is left for new ideas. Simply put, the continued use of old ideas is hard enough if you add these generally unusable computers to the mix. The reason is simple; the new computers completely suck. They are true monstrosities (in the classic definition of the word) and complete pieces of shit as scientific hardware. They are exactly the computers we don’t want to use. The price of forcing them down our throats is the destruction of research that isn’t associated with simply making these awful computers work. Worse yet, the return on the massive investment of effort will be vanishingly small in terms of added modeling and simulation capability.


As noted this whole direction is a foolish attempt to breathe life into the already rigid corpse of Moore’s law. Now dead at every scale of computing and already a decade deceased at the level of computer chips – note the death of Moore’s law and the ascendency of cell phones is strongly correlated, and that probably is not a coincidence. The truth of our real performance on computers is far more dire and damning of this entire effort. We have been getting an ever-lower proportion of the potential performance on our computers for 25 years. Each computer has a peak performance measured on silly hardware friendly benchmarks that no one gives a flying fuck about (the dense linear algebra LU decomposition, Linpac). This silly and useless benchmark is how we crown the fastest computer! Our actual code performance on these machines is truly atrocious and gets worse every year. The dirty little secret is that its been getting ever worse every year. It was god-awful 20 years ago, and it has just gotten worse. Zero is a pretty good approximation to the proportion of the performance we get – generally much less than one percent. We mindfully ignore the situation just like one would ignore a cancer threatening to devour our lives. The attitude is generally, “look away, nothing to see here”. The exascale program is that cancer metastasized.

Part of the discussion about exascale needs to acknowledge the nature of choices and priorities in research. In isolation, the exascale program is an unambiguous good; it would be genuinely awesome to have – usable – exascale computers ( This good needs to be weighed in terms of its cost and the impact of alternatives. It needs to be viewed through the lens of reality too. If one looks at the raw cost, the opportunity cost and collateral damage, and under this examination we can see that the exascale program is a massively negative force in science (,,, ). In isolation without considering anything else, it is a clear positive. In the context of lost opportunities and effective use of available resources, the program is an unmitigated disaster. We will all be poorer for it as it lays waste to potential breakthroughs we will be denied in its wake. In today’s world we talk about things in isolation, free of nuance and trade spaces that would make for a more robust and responsible conversation. Our leaders are irresponsible in the extreme for taking down this path with no real discussion, or any debate taking place. The message in the trenches is “do what you’re paid to do and quit asking questions”.

dag006The really dirty secret is that chasing exascale as a route to scientific discovery is simply bullshit of the highest and most expensive order. We would be far better served by simply figuring out how to use the hardware we already have. Figuring out how to efficiently use hardware we have had for decades would be a difficult and worthy endeavor. The punch line is that we could get orders of magnitude in improved performance out of the hardware we’ve been using for decades. By simply figuring out how to get our codes working more efficiently on the computers already existing would meet most scientific goals without eviscerating the rest of computational science in the process. Instead we chase goals that are utterly meaningless. In the process we are destroying the research that has true and lasting value. The areas being ignored in the push for exascale have the capacity to provide far more scientific capability than even the most successful exascale program could possibly deliver. This brings me back to the meeting in Santa Fe and the lack of energy and exciting ideas. In the past the meeting has been a great survey of the active work from a creative and immensely talented group of people. As such this meeting is the proverbial canary in the coalmine. The ideas are dying right in front of our eyes.

1wakdnThis outcome is conflated with the general lack of intellectual vigor in any public discourse. The same lack of intellectual vigor has put this foolish exascale program in place. Ideas are viewed as counter-productive today in virtually every public square. Alarmingly, science is now suffering from the same ill. Experts and the intellectual elite are viewed unfavorably and their views held in suspicion. Their work is not supported, nor is projects and programs dependent on deep thinking, ideas or intellectual labor. The fingerprints of this systematic dumbing down of our work have reached computational science, and reaping a harvest of poisoned fruit. Another sign of the problem is the lack of engagement of our top scientists in driving new directions in research. Today, managers who do not have any active research define new directions. Every year our manager’s work gets further from any technical content. We have the blind leading the sighted and telling them to trust them, they can see where we are going. This problem highlights the core of the issue; the only thing that matters today is money. What we spend the money on, and the value of that work to advance science is essentially meaningless.

leland_taylor_320Effectively we are seeing the crisis that has infested our broader public sphere moving into science. The lack of intellectual thought and vitality pushing our public discourse to the lowest common denominator is now attacking science. Rather than integrate the best in scientific judgment into our decisions on research direction, it is ignored. The experts are simply told to get in line with the right answer or be silent. In addition, the programs defined through this process then feed back to the scientific community savaging the expertise further. The fact that this science is intimately connected to national and international security should provide a sharper point on the topic.  We are caught in a vicious cycle and we are seeing the evidence in the hollowing out of good work at this conference. If one is looking for a poster child for bad research directions, the exascale programs are a good place to look. I’m sure other areas of science are suffering through similar ills. This global effort is genuinely poorly thought through and lacks any sort of intellectual curiosity.

Moving our focus back to exascale provides a useful case study of what is going wrong. We see that programs are defined by “getting funding” rather than what needs to be done or what should be done. Arguments for funding need to be as simple as possible, and faster computers are naïve enough for unrefined people to buy into. It sounds good and technically unsophisticated people buy it hook line and sinker. Computers are big loud and have lots of flashing lights to impress managers, politicians and business people who know no better. Our scientists have been cowered into compliance and simply act happy to get money for doing something. A paycheck beats the alternative, and we should feel happy that we have that. The level of inspiration in the overall approach has basically fallen off a cliff, and new ideas are shunned because they just make things complicated. We are left with the least common denominator as the driving force. We have no stomach for nuance or subtlety.

mediocritydemotivatorPriority is placed on our existing codes working on the new super expensive computers. The up front cost of these computers is the tip of the proverbial cost iceberg. The explicit cost of the computers is their purchase price, their massive electrical bill and the cost of using these monstrosities. The computers are not the computers we want to use, they are the ones we are forced to use. As such the cost of developing codes on these computers is extreme. These new computers are immensely unproductive environments. They are a huge tax on everyone’s efforts. This sucks the creative air from the room and leads to a reduction in the ability to do anything else. Since all the things being suffocated by exascale are more useful for modeling and simulation, the ability to actually improve our computational modeling is hurt. The only things that benefit from the exascale program are trivial and already exist as well-defined modeling efforts.

Increasingly everything is run through disconnected projects that are myopic by construction. The ability to do truly unique and groundbreaking science is completely savaged by this approach to management. Breakthroughs are rarely “eureka” moments where someone simply invents something completely new. Instead, most good research is not made through connections to other good research. Conferences are great incubators for these connections. Well-defined and proven ideas are imported and redefined to make contributions to a new area. This requires people to work across discipline boundaries, and learn about new things in depth. People need to engage deeply with one another, which is similarly undermined today by project management and information security focus. The key thing is exposure to new and related areas of endeavor and basic learning. The breakthroughs come episodically and do not lend themselves to the sort of project management in vogue today.

It isn’t like I came back with nothing. There were a couple of new things that really fall into the category of following up. In one case there was a continuation of a discussion of verification of shock tube problems with someone from Los Alamos. The discussion started in Las Vegas at the ASME VVUQ meeting, and continued in Santa Fe. In a nutshell, we were trying to get cleaner verification results by dividing the problem into specific regions associated with a particular solution feature and the expectation of different rates of convergence for each. We found something unexpected in the process that doesn’t seem to follow theoretical expectations. It’s worth some significant follow-up.  A mysterious result is always something worth getting to the bottom of. The second bit of new intellectual blood came in direct response to my talk. I will also freely admit that my contribution wasn’t the best. I haven’t had any better luck with a good free energy at work to energize my work. The same exascale demon is sucking my intellectual lifeblood out. I simply reported on a here-to-fore unreported structural failing of solvers. In summary, we find systematic, but small violations of the second law of thermodynamics in rarefactions for modern and classical methods. This shouldn’t happen and violations of the second law lead to unphysical solutions. All of this stems from identifying a brutal problem ( ) that every general-purpose code fails at – what I call “Satan’s shock tube” with 12 order of magnitude jumps in density and pressure approximating propagation of waves into a vacuum.

We cannot live only for ourselves. A thousand fibers connect us with our fellow men; and among those fibers, as sympathetic threads, our actions run as causes, and they come back to us as effects.

― Herman Melville

Before closing I can say a thing or two about the meeting. None of the issues dulled the brilliance of the venue in Santa Fe, “the City Different”. While I was disappointed about not enjoying the meeting in some exotic European venue, Santa Fe is a fabulous place for a meeting. It is both old (by American standards), yet wonderfully cosmopolitan. There is genuine beauty in the area, and our hotel was nearly perfect. Santa Fe boasts exceptional weather in the fall, and the week didn’t disappoint. It has a vibrant art community including the impressive and psychedelic Meow Wolf. It was the Drury Plaza hotel placed in a remodeled (and supposedly haunted) old hospital. Two short blocks from the plaza, the hotel is enchanting and comfortable. We all shared two meals each day catered by the hotel’s exceptional restaurant. Having meals at the conference and together with the participants is optimal and makes for a much-improved meeting compared to going out to restaurants.

We had a marvelous reception on the hotel’s rooftop bar enjoying a typical and gorgeous New Mexico early autumn sunset with flowing drinks, old friends and incredibly stimulating conversation. American laws virtually prohibit government funds paying for alcohol, thus the drinks were courtesy of the British and French governments. One more idiotic prohibition on productivity and common sense that only undermines our collective efforts especially creatively and collaboratively. These laws have only gotten more prescriptive and limiting. We no longer can pay for meals for interview lunches and dinners, much less business meetings. None of this is reflective of best practice for any business. The power of breaking bread and enjoying a drink to lubricate human interactions is well known. We only hurt our productivity and capacity to produce valuable work by the restrictions. We are utterly delusional about the wisdom of these policies. All of this only serves to highlight the shortcomings in the creative energy evident from the rather low level of vibrancy exhibited by the lack of exciting new ideas.

Never underestimate the power of human stupidity.

– Robert A. Heinlein