“To avoid criticism say nothing, do nothing, be nothing.” ― Aristotle
Peer review is undergoing a bit of a crisis these days. As with peer review itself a hard sharp look at a topic is a good thing. It is a key professional responsibility that is done without pay and little explicit appreciation. With the cost of academic journals skyrocketing people are rightly asking tough questions about the system where institutions pay for journals twice: once for the access to the journals, and a second time through the labor of their employees in conducting the peer review. Over the past twenty years I have also seen the standard of peer review for organizations up close. It is backsliding for the simple reason that the powers that be do not understand or appreciate peer review’s basic role. Because the consequences of a negative review have become so severe real hard-hitting peer review is rarely done, and never on the record. The powers that be have come to be completely intolerant of anything that looks like a mistake or failure.
“If failure is not an option, then neither is success.” ― Seth Godin
Critical hard-hitting peer review is necessary for the successful conduct of science. This is almost never questioned until one starts to peel away the most superficial layers of the scientific enterprise. In its purest form, peer review works, but reality soon intrudes as the issue is examined. It is broadly acknowledged that a positive, but critical review is enormously challenging. It is also utterly essential to self-improvement. Too often the delivery of reviews is off the mark and either comes off as parochial and mean-spirited, or perhaps worse yet, completely without depth and integrity.
In the cold light of day we should greet a well-done critical review as a gift. A well done review won’t just tell you how great you are, or how good you ideas are, but point out the weaknesses in your work, and suggest how to improve. If you’re just hearing how great you are in all likelihood the review isn’t being done well and the praise you are getting is bullshit. Moreover this bullshit is doing you a great disservice.
For me, I open email containing the review of a paper with dread. It is always personal, at the same time it isn’t usually nearly as bad as I fear. In the long run, my own work is greatly improved by peer review, and to be honest it would suffer greatly without it. Ultimately it is a great force for the good, it keeps me sharp, honest and exacting in the quality of my work. Nonetheless it is a difficult thing to deal with and the weaker part of me would gladly avoid it at times. My thinking self intervenes and takes it as an important part of self-improvement. Occasionally there is a deep disagreement, and the reviewer is wrong, but these points are usually contentious and not fully decided by the community. A good review is always an opportunity to learn and grow.
There are other issues with peer review worth taking note of. For example, certain people become quite renowned and unfortunately get immunized from peer review. A friend of mine had two fairly famous advisors and submitted a paper with them. The paper received no real review, the reviews just said “great work” and “publish”. It was a good paper, but in this case the process was broken. These non-reviews were a severe disservice to the community, my friend and even his famous advisors. Because the reviewers were anonymous, we don’t know who they were, but they did no one a favor. Even this paper could stand to be improved. I can say that my reviews are never that glib. On the other hand there are times I could do better as a reviewer, but I’m under the view that I could always do better.
The places I’ve worked are themselves reviewed in keeping with the basic scientific attitude that peer review is a necessity. The idea is good, and could be just as valuable for my employers as the peer review is for me as a scientist. Like many good ideas, the execution of the idea is flawed. Over time the flaws have grown until it is fair to say that the system is broken. The technical work is highly scripted, shows the best the organization has and never receives any more than a smattering of critique. The review only hits around the edges, and the marks given to the organization are always “World Class”. What criticism is received doesn’t really need to be addressed anyway. Rarely, if ever, does the review lead to anything except a declaration “we are great again this year”.
How did we get this way? This whole attitude has two sources: the lack of understanding of how science works, and the unwillingness to accept failure of any sort. There is a lack of understanding that mistakes happen when people or organizations stretch and challenge themselves. If you aren’t making mistakes you probably aren’t applying yourself, or trying very hard. We also systematically lowball all our expectations for achievement to avoid the possibility of mistakes or failure. This is a chronic condition that is slowly sucking the vitality from our research institutions. It is literally a crisis.
As most educators note, mistakes are necessary for success as they are the foundation of learning. If one isn’t making mistakes, or outright failing, they aren’t pushing their own limits. We are increasingly defining a system that takes mistakes and the possibility of failure off the table. The consequences of this are grave in that these mistakes and failures are the engine of success. This is not to say that we should allow the problems of malpractice or lack of seriousness of effort to creep into our work. I am saying that we need to encourage the possibility of failure and mistakes arising from honest, earnest efforts without the current threats of repercussions. By driving mistakes and failures from the system we guarantee mediocrity. If I look around me at the system we have created I see boundless mediocrity. We are becoming a milquetoast Nation. Gone are the bold initiatives that made our country proud. Now everyone is afraid of screwing up, which is precipitating the biggest screw-up of them all.
What needs to be done? How do we get out of this horrible bind?
We need to start by meaningfully differentiating between mistakes and failures due to incompetence from those coming from ambition. If we continue to punish ambitious efforts that fail in the manner we do, we will kill ambition. In many cases one glorious success is worth 100 noble failures. Our attitude toward any of these things in making sure that we have 100 mediocre successes that can be spun into seeming competence. We need to demand high standards, ask hard questions and focus on doing our best. We need to recognize that the lack of mistakes and failures is actually a bad thing and sign that things are not working.
We need to encourage our critics to find our flaws, demand we fix them and tell us where we can do better. We are not good enough, not by a long shot.