Different peer review models

Different peer review models (2 hours)

There are many different ways peer review is carried out. A few relevant dimentions are the following:

Openness vs. anonymity (single-blind review, double-blind review, and open review)

Focus on impact vs. focus on methods

Pre- vs. post-publication

The anonymity dimension

There many things that people don't like about peer review. There is a potential for reviewers to be assholes (leaving mean and unconstructive comments, submitting lazy and unhelpful reviews, forcing authors to cite the reviewers' own papers a lot, missing deadlines and delaying the process, or even stealing the authors' ideas and writing their own paper on it instead). There is also a potential for bias (e.g., reviewers may discriminate against authors from non-Western backgrounds, or reviewers may be less critical towards papers from famous authors). Various types of review have been proposed to combat this.

Single-blind review. In traditional single-blind review, the reviewers know who the authors are and the authors do not know who the reviewers are. This is probably the most hated kind of review, as it is subject to both of the possible problems I mentioned above. It's also probably the most common kind of review in linguistics and psychology journals. Some people like it because they believe it's important to know the authors' biases (e.g., if a paper claims to show evidence that Theory X is the best theory ever, you might judge it differently depending on whether the author of the paper is the person who created Theory X, versus an independent person).

Note that "single-blind" could also mean the authors know who the reviewers are and the reviewers do not know who the author is. This model is rare, but some people have argued that it might be ideal—it would make reviewers less likely to be assholes, and it would also protect against reviewers making biased recommendations based on the authors' identities. On the other hand, it shares one of the potential problems that has been raised about open review (below), which is that less-prestigious reviewers might not feel safe to criticize papers by famous authors.

Double-blind review. In double-blind review, neither the reviewers nor the authors know who each other is. There is some weak evidence that double-blind reviewing reduces biases (see, for example, this paper examining double-blind reviewing and gender bias at a linguistics conference). Some people argue that double-blind reviewing is ineffective because, in a small field like ours, people can figure out who the author is anyway; however, some research has suggested that reviewers are actually not good at guessing the identity of authors. Of course, double-blind review cannot stop the problem of some reviewers acting like assholes behind a veil of anonymity. Overall, whether anonymity is good or bad is a subject of much debate; being able to criticize popular ideas without being fired is a good thing about anonymity, but being able to be an asshole is a bad thing about anonymity. (See this article for a discussion of the role of anonymity in science.)

Open review. Authors and reviewers both know who each other are. This is the opposite extreme from double-blind review. If knowing the authors' identity introduces bias, then open review will still be subject to that bias. However, it may protect against anonymous reviewers behaving like assholes, since they can be accountable for what they say in their review.

Hybrid models. There are other combinations of these things. For example, as mentioned above, a model in which authors know reviewers' identities but reviewers don't know authors' identities is logically possible, but rarely practiced at journals. But it can happen de facto. Some reviewers, including me, always sign our reviews—whether this is a good or a bad thing is another subject of substantial debate. If I review for a double-blind journal, where I don't know who reviewers are, and I sign my review, then I have created a single-blind situation where the authors know who I am and I don't know who they are.) Another interesting hybrid model is the one practiced by Frontiers journals: while review is going on it's single-blind (reviewers know who the authors are but authors don't know who the reviewers are), but if the paper gets accepted, the reviewers' names are published next to the article, so people can know who accepted it. If the paper gets rejected, the reviewers' names will remain hidden. Thus, it's single-blind if a paper is rejected, but it becomes "open" if a paper gets accepted.

The impact vs. methods dimension

Traditionally, journals would decide what research they consider "impactful" (i.e., what research they think will make a big difference in the field, what research they think is important for people to read) and publish that. Thus, they might reject papers not because of the scientific quality, but because of the impact (e.g., if a paper is not very different from previous work, or doesn't make a big important conclusion, many journals might not want to publish it).

In recent years, many new journals have emerged that follow a different model. These journals believe that it should not be up to 3-4 people to judge whether a paper has the potential impact. Instead, they believe impact should be judged by the field. Thus, they don't judge a paper's potential impact; instead, they publish any paper as long as the scientific methodology is good, and then they let the field decide what's impactful. If, a few years after publication, a paper has been cited and discussed a lot, then it clearly has made an impact on the field. On the other hand, if many years have passed since a paper was published and it has never been cited, then clearly nobody cares about it. The idea of these journals is to let impact be determined in that way, rather than letting a few reviewers say "I don't think this research is very important." Some journals that follow this model include PLoS ONE, Frontiers (all the Frontiers journals), Collabra, and Scientific Reports. At these journals, reviewers are meant to just focus on methods; this means that you can submit a paper there that only makes a minor advancement relative to existing literature, but you'd better make sure your methodology is excellent. (It's also good to know what kind of journal you are submitting to; peer reviewers are busy people and often don't have time to check all the details of the journal they're reviewing for, so they might still make comments like "I don't think this research is impactful". In such a situation, when you write your response-to-reviewers letter you should be prepared to say "actually impact is not one of the criteria at this journal.)

The pre- vs. post-publication dimension

The abovementioned issue about judging impact vs. judging methodology has also led to a similar debate and an even more radical proposal, which involves completely rejecting the traditional concept of journal publishing. In the traditional model of publishing, reviewers and journals are seen as "gatekeepers": they decide what is good enough to be published, and thus they basically decide what scientific knowledge deserves to be spread to the world.

In recent years, however, some people are starting to argue for a more democratic process. Their argument is that it should not be the responsibility of 3-4 people to decide what research is worthy or not; instead, every paper should get "published", and then the whole world can decide if it's good or not. Instead of reviewing papers before they're published and accepting or blocking them, we should instead review papers after they're published. In such a model, "journals" as we know it might not even exist at all; instead, everyone would just post their papers on open platforms (like ResearchGate, arXiv, and similar platforms), and anyone in the world could comment on those papers, and then the role of "journals" would be to find the best papers and compile them. For a detailed vision of how this might look, see the blog post "Let's just get rid of peer review."

Do the activities below to reflect on what you've learned about peer review.

Choose any one of the dimensions discussed previously (anonymity, pre vs. post, or impact vs. methods). What do you support, and why?

Note that all of these issues are widely discussed in blogs and social media; if you aren't sure what you support, you are welcome to search online and read some discussions about these issues to help give you some ideas.

There are a lot of social media platforms where people are discussing and arguing about peer review all the time. A few examples:

Facebook groups such as "Reviewer 2 Must Be Stopped" (for example, here is a recent discussion which touched on many of the issues brought up in this activity, with many different and conflicting opinions expressed)
The ResearchGate discussion forum

Check these or other groups (I'm sure similar groups exist in Chinese social media as well) to find at least one discussion about some question or controversy regarding peer review. (These are public pages, so you don't need a Facebook or ResearchGate account to see them.) Below, copy the link to the discussion, and let me know what you think about it (e.g., if it's a controversy about something that happened in a peer review, say which "side" you think is "right", and why).

When you finish this activity, you are done with the module (assuming all your work on this and the previous tasks has been satisfactory). However, you may still continue on to the advanced-level task for this module if you wish to complete this module at the advanced level (if you're aiming for a higher grade or if you are just particularly interested in this topic). Otherwise, you can return to the module homepage to review this module, or return to the class homepage to select a different module or assignment to do now.

by Stephen Politzer-Ahles. Last modified on 2021-05-04. CC-BY-4.0.