While musing about Moore’s law and algorithms last week something important occurred to me. The systematic decision to emphasize hardware and performance increases via Moore’s law over the larger gains that algorithms produce may have a distinct physiological basis. For decades, Moore’s law has provided a slow steady improvement, kind of like investing in bonds instead of risky stocks. Algorithmic improvements tends to be episodic and “quantum” in nature. They can’t be relied upon to deliver a steady diet of progress on a time scale that most program manager’s reign’s run.
I think the problem is that banking on Moore’s law is safe while banking on algorithms is risky. The program manager looking to succeed and move up may not want to risk the uncertainty of having little progress made on their watch. The risk of the appearance of scandal looms. Money can be spent with no obvious progress. Algorithmic work depends upon breakthroughs, which we all should know can’t be scheduled.
This has profound implications on what we try to do as a society and Nation. I’ve bemoaned the lack of “Moonshots” today. These would be the big risky projects that have huge payoffs, but large chances at failure. It is much better to rely upon the unsexy, low-risk, low-payoff work that is incremental. Pressure to publish is much the same. Incremental work will provide easier publishing and less chance of crashing and burning. Risky work is harder to publish, more prone to clashes with reviewers and can be immensely time consuming.
The deeper question is “what are we losing?” How can we remove the element of risk and failure as an impediment to deeper investments in the future? How can we create efforts that capture the imagination and provide greater accomplishments that benefit us all? Right now, the barriers to risky, but rewarding, research must be lowered or society as a whole will suffer. This suffering isn’t deeply felt because it reflects gains not seen and loss of possibility.