, , , ,

The process of peer review is viewed as one of the cornerstones of our research system.  The approach to peer review is often cast as an adversarial interaction where the peer’s job is to critique the research, and in the process vet it for suitability for publication.  The most common form of this peer interaction is the refereeing of research papers for journals (and conferences). 

Very few scientists can’t share a few war stories about their experiences with peer review, and most of us have been on both the receiving and the giving end of a contentious exchange.  It is almost a form of professional hazing (or it IS a form of this).  Almost everyone can relate to the moment of dread upon opening the report from the referee.  What outlandish things will they demand?  Will they engage in ad hominem attacks?  What if it is an outright rejection?  I often left thinking, is this really the best system?  It is pretty clear that it’s not.

On the other hand, research without peer review is viewed dimly as it should be. Numerous times the referees report and my response to it made for a better paper.  A few times the referee has succeeded in making me publish a worse paper.  Seriously.  I’ve had referees defend backwards and ignorant practices and demand they continue. A better perspective on the topic is whether the current form is actually the most beneficial form for this practice?  I think we might do well to rethink our model.

I’ll make a modest suggestion for a huge problem; we should strive to engage more as “coaches” instead of as “referees”.

Sports analogies are often useful to may people, or at the very least they are useful to me.   Sports have four categories of people involved: players, coaches, referees and fans.  The players are the source of the action; coaches in training, planning and orchestrating the action and referees make sure the rules are followed.  Fans are primarily interested in the outcomes and only influence the outcome indirectly (or as much as they are allowed to).  Fans are basically like the (interested) public, and while this isn’t my focus today, although it might be an analogy worth developing.  I’m going to apply this analogy to work in science and how different people can or should adopt roles that parallel the actors in sports.

Referees are given authority; their authority is sanctioned.  This is true for a game and this is true for a peer review.  Despite this, they are generally reviled.  What happens when you try to referee without sanctioned authority?  Nothing good, that’s what. Even with the authority granted the act of refereeing is met with some degree of distain. 

At work, I spend a lot of time worrying about the quality of other people’s work. In fact I’m paid to do this, and sometimes have to do this professionally in reviewing papers or projects. This seems to be the lot of someone who does V&V.   What is the right way to do this?  It is easy for this sort of worrying-based activity to become horribly adversarial. 

After all, I know the right way to do things.  Of course I don’t, but the referee in me acts this way all the time.  This sort of interaction is bound to go south.

In this vein I’ve probably tended to act more like a referee in a game.  Everyone knows that players, coaches and fans generally despise referees.  Referees are not only hated, they are conned all the time.  Players and coaches love to get away with cheating the referee, and expect to get away with it.  This gets really ugly fast and I’ve come to realize this isn’t the best way to approach improving quality, which ought to be my main objective.  Even in the situation the sports referee is encouraged to engage in “social engineering” instead of hard-nosed policing.  Often you are negotiating and urging the players into behaving themselves.  This is much more effective than calling foul after foul because the game rapidly turns into a farce.  On the other hand there is fuzzy line that can’t be crossed where you must assert authority.  This line once crossed often results in a red card being shown.  Whenever I’ve had to do this it’s filled me with regret.  “What could I have done differently to prevent this?” 

Of course, referees also appear prominently in science as those anonymous people who review manuscripts.  Referees are viewed as being utterly essential to the entire system of the peer-reviewed literature.  They are also those who worry about quality, and are often despised.  Almost everyone who has published a paper has had to deal with an asshole referee.  I certainty have.  I might also honestly assess my own behavior as a referee; I’ve probably been an asshole too.  Hiding behind the cloak of anonymity has much to do with this; although the editors of journals know who the referees are and they have to tolerate the asshole behavior to some degree for it to persist.   It could be that some editors have a different threshold for what constitutes unreasonable or unprofessional conducts.  Of course some people are just assholes and enjoy being ones.

Of course, the game refereeing is a public, open activity and here the analogy gets a bit thin.  As poorly as refereeing goes in under the public eye, an anonymous referee would be worse.  The outright rejection of a paper is the proverbial red card, and its handed out anonymously.  I think this aspect of the system is ultimately harmful.  It isn’t that some work shouldn’t be rejected, but rather it shouldn’t be rejected behind a veil.  More than this, do we really give the rejected work a chance? Or the feedback it needs to succeed?  More often than not, the answer is no, and the system itself aids in the process.

Given this and lots of scandals appearing in the public eye one might reasonably ask whether the current system of peer review is working.  Most of the embarrassing cases are not in fields remotely related to computational science and physics, but it is probably only a matter of time till something bubbles up.  The flip side to the horrible, asshole referee is the crony referee.  The sports analogy is the corrupt referee who fixes the game.  In the academic literature this ought to be more clearly reviled. Again, the specter of anonymity raises its ugly head to assist the problem’s prominence.  Again, I might ask myself the question of whether I’m even slightly guilty of this sin?  The key point is that the nasty or the crony referees do not serve science well and their influence is hurting research.  This isn’t to say that most of the reviews aren’t well intentioned and professional.  Most are, but enough aren’t that it might be worth considering a different path.  This could be coupled to other aspects of the open-source, open-publishing movement, which is also coupled to reproducibility another big trend in publishing and peer review.

A more positive thing to think about is whether the current system is beneficial, or as beneficial as it could be.  The referees and editors are the gatekeepers, who can allow certain things to be published, and other things not to be published (in a given journal).  For the most part they do not work with the authors to improve the work.  A referee’s report usually only contains a small number of the items that might be improved.  The system limits the amount of feedback the referee might give.  Too much feedback is a sign that the paper should be rejected. 

I’ve seen cases where an earnest review on my part that was too detailed resulted in the editor rejecting the paper.  It was the exact opposite of what I wanted!  I ended regretting not writing a more effusive statement of the value and worth of the paper despite its flaws or necessary (in my opinion!) improvements.  The fact is that if I want to reject a paper I will not write a long review.  I will write why it isn’t “up to scratch” and what is valuable and worth focusing on when resubmitting.  A long review means, “I’m interested”.

The system is geared toward an adversarial relationship rather than a collaborative one.  The general adversarial approach permeates science and greatly undermines the collective ability of the community to solve deep problems. We should develop a peer review process that confers positive properties onto this essential interaction.  Negative behaviors are deeply empowered by the anonymity of the interaction.  If the referee was known to the authors and a collaborative model were established, the behavior of the referees would be better, and the spirit of collaboration could be established and used to improve the work.  Maybe an email exchange or phone call could be used to work out the details of how to improve a paper instead of the horribly inefficient system of review & revision.  Communication could take place in hours or days that now takes weeks or months. 

This is where the analogy of “coach” comes in (teacher or tutor works too).  The referee could be recast as a coach.  A coach for the different subfields the paper covers.  For example, I could be assigned as a “verification” or “numerical methods” coach for a paper.  As a coach I would suggest things to do in order for the work to improve.  Perhaps I might be given a minor, lesser stake in the paper.  In other words, the paper would still have authors, but the “coaches” would also be identified in a manner more significant than an acknowledgement, but less significant than authors.  This role would have to have value professionally for the idea to work. Maybe this could lead to a better, more collaborative community for publishing. A formal role would assist the reviewers in being more energetic and focused in their reviews.  They would become a stakeholder in the paper, not simply an anonymous servant.  Ultimately a positive interaction with a “coach” might lead to a more substantial interaction later.

Coaches operate in many ways, but their objectives are similar to win (i.e., succeed) and often winning is related to teaching.  Great coaches are very often great teachers.  The best coaches not only work toward the success of their team (or players), but also to develop their teams of players to their potential.  Coaches work in numerous ways to achieve these ends through drills, feedback, and challenges.  Each of these approaches develops something greater than what the team or players can do with themselves. As such, the coach is a model for developing an improved capability.  This is basically another way of stating the objectives of the model for replacing referees with something more positive and constructive.  I am saying that we should think more like a coach and less like a referee.

There is a need for some authority to be exercised.  There is a reason we have referees, and they are necessary.  Referees are adopted when the stakes are high and the outcome is meaningful.  Should we be viewing all of research as a high stakes “game”?  Or would approaching most research as a practice where we want to improve and saving the game approach to when things really matter better serve us.  This certainty should be applied to the educational setting where the current system does not maximize the training that young professionals receive.  Perhaps something more balanced can be used, where we have both coaches and referees being assigned to work with a given research.  The key for effective refereeing in fairness and impartiality, along with competence something we clearly don’t have enough of today. 

I think the same model should be adopted in looking at simulation quality in V&V and UQ work.  Those of us with such expertise should act more like coaches and less like referees.   This way we can work constructively to make computations better, we become part of the team instead of part of the “other” team, or the enemy in the middle.