Peer Review in Scientific Journals: double-blind versus single-blind. Discussion and resources

por

The Shape of Science. SCImago Journal & Country Rank

1. Introduction: what is Peer Review?

As is well-known, one of the key aspects of scientific journals is the process known as peer-review (or review by your peers, considered as experts). This process implies that journals only publish articles which have passed an evaluation conducted at least by two experts (but often three intervene). That is to say, unlike journalistic or cultural magazines where publications are those an editor-in-chief or section editor might see fit to publish, an academic journal is decisively based on opinions from experts who do not belong to its editorial board.

This process is one of the reasons to attribute quality to “modern science” (between quotation marks, because, according to experts on this matter, this system has been used since the 18th century; see the reference at the end of this post). The fact that only those articles which have passed an evaluation process conducted by independent experts can be published, without an (excessive) intervention from the editor-in-chief, is considered crucial for the advance of sciences

2. Double-blind vs single-blind review

However, like happens with every other interesting thing in the world, this process is being heatedly debated and has even been met with bitter criticism. The following might seem exaggerated, but I want to compare peer review with the historical criticism democracy as a form of government has been subjected to. I believe it was Winston Churchill who once settled the debate in a masterly way, by claiming that “democracy is the worst form of government, except for all the others” Considering the amount of grievances it generates, I believe the scientific system based on independent evaluations could be regarded similarly: it is the worst way to make science, except for all the others.

But there is a debate which I believe it is not very well-known in the Social Sciences. This debate does not refer to the usefulness of the peer review system (this post is based on the idea that it actually is useful), but to whether it is better to apply it following a double-blind or a single-blind model of review. These are the differences between them:

  • Double-blind: the evaluators are unaware of the identity of the authors, and the authors are unaware of the identity of the evaluators.
  • Single-blind: the evaluators are aware of the identity of the authors, but authors are unaware of the identity of the evaluators.

I do not consider the second option to be very well-known, since, at least regarding Social Sciences journals, most of them apply a double-blind model of review. Which might seem alright at first, but I am afraid they do not use it as a consequence of a much-needed reflection on the advantages and disadvantages of each model, but because most of the editors and scholars in this area do believe it is the only possible model of review to use in peer review evaluation.

Simply put, this means that they use the double-blind system because they “have to”. Certainly, authors prefer the double-blind system when asked about it: 56% are in favor of the double-blind system, whereas 25% are in favor of the single-blind system (https://helpmepublish.org/for_researchers.html). But one thing is the theoretical preference of the author, and the other one is the preference of the editor in a tangible reality, since most (but not all) Social Sciences journals use a double blind system, but most journals related to scientific disciplines such as Physics and Biomedicine, among others, use the single-blind system.

3. The problem of no choice

At least in my opinion, I believe there is a problem here. In the first place, it is not the same to do something in a certain way because we believe it is the only way to do it, as to do it because we believe it is the best after considering all the existing choices.

I do not see a problem if a journal claims “we use a double-blind system because, after examining its advantages and disadvantages, we consider it to be the best option”. However, I think there might be a problem when it seems to have been chosen because they believe there is not another way of doing things.

I would not want to seem to be advocating for a twisted approach, but I believe that this is what we might call the problem of NO choice. As I try to argue, this problem involves doing something because there is a misconception that there is no other choice besides that.

4. Advantages and disadvantages of each evaluation system

If we start to analyze the advantages and disadvantages of each system, to put it briefly, the double blind system is believed to guarantee a better objectivity, because it avoids possible biases that might come up if evaluators are aware of the identity of the authors. However, its detractors point to the fact that the anonymization of the articles adds to the workload and at the end does not prove to be very useful, because they often “guess” or detect who the authors are.

Specifically (see below), the American Economic Association claims that, thanks to databases and Google Scholar, on the one hand nowadays is practically impossible to efficiently anonymize the articles and, on the other hand, not knowing the authors prevents evaluators from detecting cases of conflict of interests between authors profiles and the topics they research on, a problem for which Economy journals have been harshly criticized in the last years.

The famous documentary on the economic crisis Inside Job was exceptionally apt at showing the undeclared and undetected conflicts of interests for journals, since famous economists published what seemingly were scientific reports concealing propaganda actions to benefit the companies that had subsidized them.

A similar line of argument was argued by Nature in an editorial from 2008, when they remarked the advisability of knowing the identity of authors so that evaluators might be able to determine whether true progress on the author’s research was being reported (or it was a case of “more of the same”). And, as a significant nod to the current state of things, they added that “knowing rather than guessing the identities of authors…” would encourage evaluators to deal with conflicts of interests as those already mentioned with economists.

Moreover, the single-blind system has the ability to make the whole process much more agile. It might not seem an important issue, but it is. Not only much energy is focused on anonymizing an article, but sometimes this process might turn into quite an awful experience, as when a prestigious journal is forced to reject a good article if the anonymization process has not been properly implemented, which sometimes does not depend on obvious factors such as deleting the metadata of the document or not citing its funding sources. The problem is that, if the journal is using a double-blind system, no exceptions should be allowed. In other cases, it is practically impossible to anonymize an article if it has been developed through singular projects; or even some articles become incomprehensible once any trace of previous research has been erased. Thus, it is not a simple question at all.

Regarding the single-blind system, the most significant risk is the bias (for and against) authors; not only for reasons of personal identification, but also regarding genre, background, nationality, etc. Some studies typically mention the possibility of genre discrimination, if the research comes from low-prestige universities, or from authors outside academia (see the JASIST article referenced at the end of this post).

But those biases could be minimized, precisely by being aware of them and developing a good editorial policy and a proper selection of evaluators (conflicts of interests should be avoided, and evaluators who have proved to be independent and present a mature judgement should be chosen). Anyhow, none of the two systems is perfect. This discussion is about choosing the group of advantages and disadvantages which is more balanced and not about choosing a perfect system, because there is none.

An interesting report analyzing both systems, written by R. T. Snodgrass, is referenced at the end of this post. In a 2007 review, Snodgrass remarked 6 positive aspects of the double-blind system (the main one being a protection against biases, since the other aspects are variations on this one), against 21 possible risks of the double-blind system, such as:

  • The quality of the reviews diminishes (n. 1)
  • It encourages the possibility of accepting articles that are not very original (n. 4)
  • It endangers well-known projects (n. 6)
  • It makes it difficult to detect conflicts of interests (n. 11)
  • It increases the possibility of plagiarism (n. 14)
  • It difficults the reviews (15)

Some of the 21 risks Snowdgrass presents are unlikely, but others are very real (like the difficulty of keeping the identities of the authors a secret). Interestingly, at the end he advocates for the double-blind system, but also adopting a series of measures to reduce the abovementioned problems.

As an editor (if anyone should ask him/herself) I am personally for a single-blind system, partly because I follow an already implemented editorial policy, but also because it actually justifies the whole process, both to us and to authors. Undoubtedly, the key is to choose evaluators well; but this is the same in the double-blind process. Without good evaluators, it is impossible to publish a scientific journal, no matter which system is used (this is why scholars like Irene Hames –see the reference at the end of the post- claim that good evaluators are the main asset for a scientific journal).

But there is also another, more subtle reason, which makes me prefer the single-blind system. Certainly, our mind changes when we know the author, but in this case I believe it is for a good reason. A certain bias might still be there, but I honestly believe it might help provide a better evaluation of who has worked on the article. This last argument is a perception solely based on my experiences, and I recognize I am not able to elaborate further on the matter, so you will have to take my word for it.

Finally, there is another reason many might find strange for not preferring a double-blind system, but there is solid background to support it, and it is the following: if an evaluator has seen a good idea in an article which has finally been rejected, how can the evaluator make the proper attribution? (thank you, Mark Wilson).

In any case, an indication of a possible advantage for the single-blind system is that other crucial science evaluation processes (like the evaluation of competitively funded projects) always adopt the form of a single-blind review. I am almost sure that, in those processes, the evaluation would be worse if the evaluators were unaware of the identity of the researchers. But, as I said before, I cannot provide a more thorough explanation on this argument, thus I am leaving it at that for now.

 

Even more important than choosing a double-blind or a single-blind system, is choosing consciously and properly for each instance. The huge publication scandal explained by Inside Job that has already plagued the articles in Economy journals might justify the preference for a single-blind system. The journal I am editor-in-chief of is a yearbook where we focus on articles presenting state of the art and research results coming from our competitive projects. Thus, a good percentage of our articles is asked directly to the authors after our editorial board decides on the topic to be covered each year. This is why, in our case, the choice of a single-blind system was due to our editorial policy (although we invite authors to publish, we might reject an article if it does not pass the evaluation). If another editorial policy was applied, we would probably choose a double-blind system.

5. Six resources to reflect on peer review processes

To finish this post, I provide a modest content curation regarding both double-blind and single-blind systems, for those who might interested in delving into the topic:

  • Journals weigh up double-blind peer review
    An article published by the prestigious journal Nature in 2014, which argues for a double-blind system. One should read between the lines, and realize that, according to the author’s background (and supposedly to most of his readers), the single-blind would be the most common option. Thus, the fact that he argues in favor of a double-blind system might surprise Social Sciences scholars, since it is an obvious choice.
  • Nature journals offer double-blind review
    More from Nature, this time from 2015. This is an advertisement by the publishing house, announcing its readers that Nature journals will apply the double-blind system if their authors ask for it. Which means that they will keep applying the single-blind system by default (remember that Nature is one of the two or three scientific publishing houses with the strongest worldwide impact). This position tends to surprise Social Sciences scholars when they look for the evaluation model of the highest impact journals: often, these journals do not make explicit whether they use a double or single-blind system because the single-blind system is expected (could this be understood as a no choice election problem, only defined the other way around?). Interestingly, Nature indicates that although most authors theoretically prefer a double-blind system, only a few asked for it since this option was enabled.
  • Making Peer Review Work For You
    Help page for authors, by the Help Me Publish (Universidad de Otago) service, with a succinct but very effective summary of advantages and disadvantages of the double-blind, single-blind and open peer review systems (in this last case, both authors and evaluators know each other). It also references the study on academic preferences for one or another abovementioned system.
  •  Single- versus double blind reviewing
    Currently available in the university repository of the author, Ricard Snodgrass, this interesting article (pdf) was originally published in a journal from the ACM (2007). It analyses the risk of biases in the single-blind system to later review problems and advantages of the double-blind system. It mentions 6 advantages of the double blind system before listing 21 costs it might involve.  Interestingly, at the end it recommends to apply a double-blind system.
  • Rejecting double blind
    Published in a 2011 issue of Inside Higher Ed, it is a discussion on a press release by the American Economic Association, where they announce they are no longer using a double-blind system. They are changing into a single-blind system, because, among other reasons, anonymity cannot be guaranteed and conflicts of interests should not be avoided. The author echoes other publications which have discussed this topic.
  • The case against double-blind peer review
    Also from 2011, this post was published in the blog of a Computer Science researcher from Quebec University, Daniel Lemire (and it has already gathered 1,500 citations on Google Scholar). He argues in favor of a single-blind system providing some interesting extra argument. This post also quotes the previously mentioned argument by Mark Wilson.

Other useful resources related to academic publications

6. Conclusions

The double-blind system is the first theoretical reference for scholars, but not for editors. It is not the most common system in some branches of science, and it has been the object of some criticism: it is not considered very useful to conceal the identities of authors, but it does demand a significant effort, particularly for authors but also for editors. In brief, it complicates the whole process and involves a lot of extra work while never fully guaranteeing anonymity.

The single-blind system is the most common system in some sciences (but not in Social Sciences). In theory it involves a risk of bias from evaluators, but, on the one hand, recent studies do not present dramatic figures on that matter, and on the other hand, this kind of studies might produce contradictory results. In any case, this bias might be prevented by choosing good evaluators. The main advantage of the single-blind system is the simplicity of the process, but it also presents positive aspects as a better identification of conflict of interests, better detection of self-plagiarism or the ability to identify the lack of true progress in authors’ researches. Some us claim that knowing the identity of the authors makes for a better appreciation of their work, which is also suggested by the fact that it is mandatory in other evaluations (such as those for scientific projects).

Of course there are other options besides these two we have presented: from open review (where everybody knows everybody and the observations of the evaluators are published next to the final article) up to systems like the one suggested by Nature, where evaluation is submitted to a single-blind system by default, but authors might ask for a double-blind review if they wish (which, interestingly, is not a very much demanded option), and also publications allowing to veto evaluators, or, on the contrary, to suggest experts to evaluate.

But surely, what is really important is that, when an editor adopts an evaluation system, they make it consciously, circumnavigating the no choice problem and providing, in the most transparent way, an explanation for their choice in the information sections regarding the editorial policy of their journal.

Two further references

Translation from Spanish by Raquel Herrera