I much prefer the sharpest criticism of a single intelligent man to the thoughtless approval of the masses.

― Johannes Kepler

phd100107sFor some reason I’m having more “WTF” moments at work lately. Perhaps something is up, or I’m just paying attention to things. Yesterday we had a discussion about reviews, and people’s intense desire to avoid them. The topic came up because there have been numerous efforts to encourage and amplify technical review recently. There are a couple of reasons for this, mostly positive, but a tinge of negativity lies just below the surface. It might be useful to peel back the layers a bit and look at the dark underbelly.

The pleasure of criticizing takes away from us the pleasure of being moved by some very fine things.

― Jean de La Bruyère

First, the reasons for an increased emphasis on reviews should be examined. It turns out that most of the problems are lying on the surface. The general assumption is that peer review is one of the cornerstones of quality in science. It is a powerful mechanism for communication in both directions; the reviewers are experts you’d like to promote your ideas with, and the reviewers usually have something useful to say to you, at least if they are doing it right. Nonetheless, as many of you know, peer review can be emotionally draining, and painful to go through. A second, less positive aspect is the organizational desire to escape embarrassment from either shoddy or fraudulent work, which should be smoked out via peer review. A third aspect that also comes from the “dark side” of peer review is a sort of smoke and mirrors of using it to craft a veneer of due diligence and implied quality (more on this later). These are reviews that are mandated by organizations, and of course, such a mandate shouldn’t be needed to have this happen, but as such the mandate is actually a sign that a problem exists. Technical organizations should “know” that technical review is essential to its fundamental health.

phd111214sI touched on this topic a couple of weeks ago (https://wjrider.wordpress.com/2014/11/02/why-does-vv-get-me-in-trouble/), but classic peer review has many problems. Some of these are due to abuse of the anonymous nature and inadequate policing of this abuse by editorial boards. The difficult part of the review for the reviewer isn’t the critique as most papers always have weaknesses, but rather balancing it with appropriate praise. In all honesty, I personally struggle with this. It is the balance between doing a fair and complete job of reviewing while not being unfairly harsh in criticism. Despite my conscious efforts to deal with the problem, I probably fail to hit the mark. As I note later, no one ever taught me how to do a review; I discovered it through osmosis.

A more difficult topic is the organizational imperative to avoid embarrassment. Mandated reviews are a terrible way to handle this problem, and a terrible reason for reviews. The need to have work reviewed should flow from the basic duty of scientists to communicate their work to peers and receive feedback. The mandated review for the purpose of ferreting out fraud or garbage is unnecessarily confrontational. It puts a negative spin on the entire topic of review. The real core of the issue is management, which should be the responsible agent in knowing what it going on in the first place. A review isn’t a police action, and using it as such undermines the purpose of review in subtle and pernicious ways. Given the sorry state of peer review, these are hits it can’t take.

 People ask you for criticism, but they only want praise.

― W. Somerset Maugham

Unknown-1The biggest problems with peer reviews are “bullshit reviews”. These are reviews that are mandated by organizations for the organization. These always get graded and the grades have consequences. The review teams know this thus the reviews are always on a curve, a very generous curve. Any and all criticism is completely muted and soft because of the repercussions. Any harsh critique even if warranted puts the reviewers (and their compensation for the review at risk). As a result of this dynamic, these reviews are quite close to a complete waste of time.

UnknownBecause of the risk associated with the entire process, the organizations approach the review in an overly risk-averse manner, and control the whole thing. It ends up being all spin, and little content. Together with the dynamic created with the reviewers, the whole thing spirals into a wasteful mess that does no one any good. Even worse, the whole process has a corrosive impact on the perception of reviews. They end up having no up side; it is all down side and nothing useful comes out of them. All of this even though the risk from the reviews has been removed through a thoroughly incestuous process.

Don’t criticize what you can’t understand.

― Bob Dylan

US_House_CommitteeAn element in the overall dynamic is the societal image of external review as a sideshow meant to embarrass. The congressional hearing is emblematic of the worst sort of review. The whole point is grandstanding and/or destroying those being reviewed. Given this societal model, it is no wonder that reviews have a bad name. No one likes to be invited to their own execution.

When virtues are pointed out first, flaws seem less insurmountable.

― Judith Martin

What can be done about this? The answers are simple, but complex within the Peer review cartoonenvironment we find ourselves. First of all, people should be trained or educated in conducting, accepting and responding to reviews. Despite its importance to the scientific process, we are never trained how to conduct, accept or responds to a review (response happens a bit in a typical graduate education). Today, it is a purely experiential process. Next, we should stop including the scoring of reviews in any organizational “score”. Instead the quality of the review including the production or hard-hitting critique should be expected as a normal part of organizational functioning.

It’s easy to attack and destroy an act of creation. It’s a lot more difficult to perform one.

― Chuck Palahniuk

People, projects and organizations willing and capable of undergoing honest, critical review are usually much better than those who aren’t. The unwilling or softball review is itself is a better indicator of problems than a negative review itself. Peer review is essential for science and we must fix it. It is an essential element in our quality process that we cannot afford to remain completely broken.