Different peer review models (2 hours)

↵ Back to module homepage

There are many different ways peer review is carried out. A few relevant dimentions are the following:

The anonymity dimension

There many things that people don't like about peer review. There is a potential for reviewers to be assholes (leaving mean and unconstructive comments, submitting lazy and unhelpful reviews, forcing authors to cite the reviewers' own papers a lot, missing deadlines and delaying the process, or even stealing the authors' ideas and writing their own paper on it instead). There is also a potential for bias (e.g., reviewers may discriminate against authors from non-Western backgrounds, or reviewers may be less critical towards papers from famous authors). Various types of review have been proposed to combat this.

The impact vs. methods dimension

Traditionally, journals would decide what research they consider "impactful" (i.e., what research they think will make a big difference in the field, what research they think is important for people to read) and publish that. Thus, they might reject papers not because of the scientific quality, but because of the impact (e.g., if a paper is not very different from previous work, or doesn't make a big important conclusion, many journals might not want to publish it).

In recent years, many new journals have emerged that follow a different model. These journals believe that it should not be up to 3-4 people to judge whether a paper has the potential impact. Instead, they believe impact should be judged by the field. Thus, they don't judge a paper's potential impact; instead, they publish any paper as long as the scientific methodology is good, and then they let the field decide what's impactful. If, a few years after publication, a paper has been cited and discussed a lot, then it clearly has made an impact on the field. On the other hand, if many years have passed since a paper was published and it has never been cited, then clearly nobody cares about it. The idea of these journals is to let impact be determined in that way, rather than letting a few reviewers say "I don't think this research is very important." Some journals that follow this model include PLoS ONE, Frontiers (all the Frontiers journals), Collabra, and Scientific Reports. At these journals, reviewers are meant to just focus on methods; this means that you can submit a paper there that only makes a minor advancement relative to existing literature, but you'd better make sure your methodology is excellent. (It's also good to know what kind of journal you are submitting to; peer reviewers are busy people and often don't have time to check all the details of the journal they're reviewing for, so they might still make comments like "I don't think this research is impactful". In such a situation, when you write your response-to-reviewers letter you should be prepared to say "actually impact is not one of the criteria at this journal.)

The pre- vs. post-publication dimension

The abovementioned issue about judging impact vs. judging methodology has also led to a similar debate and an even more radical proposal, which involves completely rejecting the traditional concept of journal publishing. In the traditional model of publishing, reviewers and journals are seen as "gatekeepers": they decide what is good enough to be published, and thus they basically decide what scientific knowledge deserves to be spread to the world.

In recent years, however, some people are starting to argue for a more democratic process. Their argument is that it should not be the responsibility of 3-4 people to decide what research is worthy or not; instead, every paper should get "published", and then the whole world can decide if it's good or not. Instead of reviewing papers before they're published and accepting or blocking them, we should instead review papers after they're published. In such a model, "journals" as we know it might not even exist at all; instead, everyone would just post their papers on open platforms (like ResearchGate, arXiv, and similar platforms), and anyone in the world could comment on those papers, and then the role of "journals" would be to find the best papers and compile them. For a detailed vision of how this might look, see the blog post "Let's just get rid of peer review."

Do the activities below to reflect on what you've learned about peer review.

Choose any one of the dimensions discussed previously (anonymity, pre vs. post, or impact vs. methods). What do you support, and why?

Note that all of these issues are widely discussed in blogs and social media; if you aren't sure what you support, you are welcome to search online and read some discussions about these issues to help give you some ideas.

There are a lot of social media platforms where people are discussing and arguing about peer review all the time. A few examples:

Check these or other groups (I'm sure similar groups exist in Chinese social media as well) to find at least one discussion about some question or controversy regarding peer review. (These are public pages, so you don't need a Facebook or ResearchGate account to see them.) Below, copy the link to the discussion, and let me know what you think about it (e.g., if it's a controversy about something that happened in a peer review, say which "side" you think is "right", and why).

When you finish this activity, you are done with the module (assuming all your work on this and the previous tasks has been satisfactory). However, you may still continue on to the advanced-level task for this module if you wish to complete this module at the advanced level (if you're aiming for a higher grade or if you are just particularly interested in this topic). Otherwise, you can return to the module homepage to review this module, or return to the class homepage to select a different module or assignment to do now.


by Stephen Politzer-Ahles. Last modified on 2021-05-04. CC-BY-4.0.