No amount of genius can overcome a preoccupation with detail
—Levy’s Eighth Law
I’ve been inundated with thinking about exascale computing this week. Programming models, code, computer languages, libraries, and massively parallel implementations of algorithms. At the end of all the talk about advanced computing, I’m left thinking that something really key is being ignored moving forward. We are already indrowning in data whether we are talking about the Internet in general, the coming “Internet of things” or the scientific use of computing. The future is going to be much worse and we are already overwhelmed. If we try to deal with every single detail, we are destined to fail.
How can we move forward and keep our sanity?
Of course reality is actually much simpler, or at least the part we care about. In almost every decision of any importance, the details fade away and we are left with only an important core of significance. This is a key concept moving forward in computing, sparsity. Not everything matters and the important thing is discovering how to unmask this kernel of essential information. If we can’t the data deluge will drown us.
Fortunately some concepts have emerged recently that hold promise. The whole area of compressed sensing is structured around the capacity to unveil the important signalin all the noise and represent this importance compactly and optimally. This class of ideas will be important in managing the Tsunami of data that awaits us.
The future will give us more data than we can ever wade through, and we need principled ways to manage our view of it. In many cases we won’t even be able to get the data off the computer at all, only a part of it. If our code or calculation crashes we won’t be able to restart from exactly the same state. We are going to have to let go of the details. This should be easier because the reality is that they don’t matter, or more properly the vast majority of the details don’t. The trick is holding on to the details that do matter.