If you want to write a negative review, don’t tickle me gently with your aesthetic displeasure about my work. Unleash the goddamn Kraken.
― Scott Lynch
Sadly, this spirit is not what we see today either on the giving or receiving end of peer review, and we are all poorer for it.
As surely as the sun rises in the East, peer review that treasured and vital process for the health of science is dying or dead. In many cases we still try to conduct meaningful peer review, but increasingly it is simply a mere animated zombie form of peer review. The zombie peer review of today is a mere shadow of the living soul of science it once was. Its death is merely a manifestation of bigger broader societal trends such as those unraveling the political processes, or transforming our economies. We have allowed the quality of the work being done to become an assumption that we do not actively interrogate through a critical process (e.g., peer review). Instead if we examine the emphasis for how money is spent in science and engineering everything, but the quality of the technical work is focused on and demands are made. Instead there is an inherent assumption that the quality of the technical work is excellent and the organizational or institutional focus. With sufficient time this lack of emphasis is eroding the quality presumptions to the point where they no longer hold sway.
Being the best is rarely within our reach. Doing our best is always within our reach.
― Charles F. Glassman
In science, peer review takes many forms each vital to the healthy functioning of productive work. Peer review forms an essential check and balance on the quality of work, wellspring of ideas and vital communication mechanism. Whether in the service of publishing cutting edge research, or providing quality checks for Laboratory research or engineering design its primal function is the same; quality, defensibility and clarity are derived through its proper application. In each of its fashions the peer review has an irreplaceable core of a community wisdom, culture and self-policing. With its demise, each of these is at risk of dying too. Rebuilding everything we are tearing down is going to be expensive, time-consuming and painful.
Let’s get to the first conclusion of this thought process, peer review is healthiest in its classic form, the academic publishing review, and it is in crisis there. The scientific community widely acknowledges that the classic anonymous peer review is absolutely riddled with problems and abuses. The worst bit is that this is where it works the best. So at its best, peer review is terrible. The critiques are many and valid. For example there is widespread abuse of the process by the powerful and established. The system is driven by a corrupt academic system that feeds the overall dysfunction (i.e., publish or perish). Corruption and abuse by the journals themselves is deep and getting worse never mind the exploding costs. Then we have issues about teaming conflicts of interest and deeply passive aggressive behavior veiled behind the anonymity. Despite all these problems peer review here tends to still largely work and albeit in a deeply suboptimal manner.
Another complaint is the time and effort that these reviews take along with suggestions to make things better with modern technology. Online publishing and the ubiquity of the Internet is capable of radically reducing the time and effort (equals money) of publishing a paper. I will say that the time and effort issue for peer review is barking up the wrong tree. The problem with the time and effort is that peer review isn’t valued sufficiently. Doing peer review isn’t given much wait professionally whether you’re a professor or working in a private or government lab. Peer review won’t give you tenure, or pay raises or other benefits; it is simply a moral act as part of the community. This character as an unrewarded moral act gets to the issue at the heart of things. Moral acts and “doing the right thing” is not valued today, nor are there definable norms of behavior that drive things. It simply takes the form of an unregulated professional tax, pro bono work. The way to fix this is change the system to value and reward good peer review (and by the same token punish bad in some way). This is a positive side of modern technology, which would be good to see, as the demise of peer review is driven to some extent by negative aspects of modernity, as I will discuss at the end of this essay.
Honest differences are often a healthy sign of progress.
― Mahatma Gandhi
Let’s step away from the ideal context of the classical academic peer review of a paper to an equally common practice, the peer review of organizations, programs and projects. This is a practice of equal or greater importance as it pertains to the execution of technical work across the World. We see it taking action in the form of a design review for software, engineered products and analyses used to inform decisions. In my experience peer review in these venues is in complete free-fall and collapsing under the weight of societal pressures that cannot support the proper execution of the necessary practices. My argument is that we are living within a profoundly low-trust world, and peer review relies upon implicit expectations of trust to be executed with any competence. This lack of trust is present on both ends of the peer review system. When the trust is low, honesty cannot be present and instead honesty will be punished.
First let’s talk about the source of critique, the reviewers. Reviewers have little trust and faith that their efforts will be taken seriously if they find problems, and if they do raise an issue it is just as likely that they, the messenger, will be punished instead. As a result reviewers rarely do a complete of good job of reviewing things, as they understand what the expected result is. Thus the review gets hollowed out from its foundation because the recipient of the review expects to get more than just a passing grade, they expect to get a giant pat on the back. If they don’t get their expected results, the reaction is often swift and punishing to those finding the problems. Often those looking over the shoulder are equally unaccepting of problems being found. Those overseeing work are highly political and worried about appearances or potential scandal. The reviewers know this to and that a bad review won’t result in better work, it will just be trouble for those being reviewed. The end result is that the peer review is broken by the review itself being hollow, the reviewers being easy on the work because of the explicit expectations and the implicit punishments and lack of follow through for any problems that might be found.
Those who can create, will create. Those who cannot create, will compete.
― Michael F. Bruyn
If we look to those being reviewed, the system only gets worse. The people being reviewed only see downside to engaging in peer review, no upside at all. Increasingly any sort of spirit or implied expectation of technical quality has been left behind. Peer review is done to provide the veneer of quality regardless to its actual presence. As a consequence any result from peer review that doesn’t say this work is the best and executed perfectly is ripe to be ignored (or punish those not complying with the implied directive). Those being reviewed have no desire or intent to take any corrective action or address any issue that might be surfaced. As a result the peer review is simply window dressing and serves no purpose other than marketing. The reasons for the evolution to this dysfunctional state are many and clear. The key to the problem is the lack of ability to politically confront problems. A problem is often taken as a death sentence rather than a call to action. Since no issues or actual challenges will be confronted, much less solved, the only course of action is to ignore and bury them.
We then get to the level above who is being reviewed and closer to the source of the problem, the political system. Our political systems are poisonous to everything and everyone. We do not have a political system perhaps anywhere in the World that is functioning to govern. The result is a collective inability to deal with issues, problem and challenges at a massive scale. We see nothing, but stagnation and blockage. We have a complete lack of will to deal with anything that is imperfect. Politics is always present and important because science and engineering are still intrinsically human activities, and humans need politics. The problem is that truth, and reality must play some normative role in decisions. The rejection of effective peer review is a rejection of reality as being germane and important in decisions. This rejection is ultimately unstable and unsustainable. The only question is when and how reality will impose itself, but it will happen, and in all likelihood through some sort calamity.
To get to a better state visa-vis peer review trust and honesty needs to become a priority. This is a piece of a broader rubric for progress toward a system that values work that is high in quality. We are not talking about excellence as superficially declared by the current branding exercise peer review has become, but the actual achievement of unambiguous excellence and achievement. The combination of honesty, trust and the search for excellence and achievement are needed to begin to fix our system. Much of the basic structure of our modern society is arrayed against this sort of change. We need to recognize the stakes in this struggle and prepare our selves for difficult times. Producing a system that supports something that looks like peer review will be a monumental struggle. We have become accustomed to a system that feeds on false excellence and achievement and celebrates scandal as an opiate for the masses.
One only needs to look to the nature of the current public discourse and political climate. We are rapidly moving into a state where the discourse is utterly absent of any substance and the poisonous climate is teetering over into destructive. Reality is beginning to fight back against the flaws in the system. Socially we are seeing increased fear, violence and outright conflict. The problems with peer review pale in comparison to the tide rolling in, but reflect many of the same issues. Peer review is an introverted view of our numerous ills where the violence and damaging environment evident in our mass media is the extroverted side of the same coin. In this analysis peer view is simply another side effect of the massive issues confronting our entire world projected into the environment of science and engineering. Fixing all these issues is in the best interests of humanity, but it’s going to be hard and unpleasant. Because of the difficulty of fixing any of this, we will avoid it until the problems become unbearable for a large enough segment of humanity. Right now it is easier and simpler to just accept an intrinsically uncritical perspective and simply lie to ourselves about how good everything is and how excellent all of are.
If one doesn’t have the stomach to examine things through such a social lens, one might consider the impact of money on the system. In many ways the critical review of research can now be measured almost entirely in monetary terms. This is especially true in the organizational or laboratory environment where most people managing view money and its continued flow being the only form of review they care about. In such a system a critical peer review system becomes a threat instead a source of renewal. Gradually, over time the drive for technical excellence is replaced by the drive for financial stability. We have allowed financial stability to become disconnected from technical achievement, and in doing so killed peer review. When technical excellence and achievement become immaterial to any measure of success, and money only matters peer review is something to be ignored, avoided and managed because no perceived good can come from it.
Self-consciousness kills communication.
― Rick Steves
Worse than having no perceived good associated with it, peer review if done properly becomes evidence of problems. The problems exposed in peer review represent calls to action that today’s systems cannot handle because they are an affront to planning, schedules, milestones and budgetary allocations. Problems also expose flaws in the fundamental assumptions of today’s world that the technical work is high quality and does not need active focused (appropriate) management to succeed. As a result any problems induce a “shoot the messenger” mentality that acts to destroy the critique and send a clear message that peer review should not be done honestly or seriously. The result has been a continual erosion of the technical quality, so often assumed to be present a priori. This is a viscous cycle where technical problems remain unexposed, or hidden by a lack of vigorous, effective peer review. The problems then fester and grow because problems like these do not cure themselves, and the resistance to peer review or any form of critique only becomes further re-enforced. This doesn’t end well, and the end results are perfectly predictable. The only thing that stems the decay is the encroachment of decay ultimately ends the ability to conduct a peer review at all. Moreover, the culture that is arising in science acts as a further inhibition to effective review by removing the attitudes necessary for success from the basic repertoire of behaviors.
Some men are born mediocre, some men achieve mediocrity, and some men have mediocrity trust upon them.
― Joseph Heller