There are many different ways peer review is carried out. A few relevant dimentions are the following:
Openness vs. anonymity (single-blind review, double-blind review, and open review)
Focus on impact vs. focus on methods
Pre- vs. post-publication
The anonymity dimension
There many things that people don't like about peer review. There is a potential for reviewers to be assholes
(leaving mean and unconstructive comments, submitting lazy and unhelpful reviews, forcing authors to cite the reviewers' own
papers a lot, missing deadlines and delaying the process, or even stealing the authors' ideas and writing their own paper on it
instead). There is also a potential for bias (e.g., reviewers may discriminate against authors from non-Western backgrounds, or
reviewers may be less critical towards papers from famous authors). Various types of review have been proposed to combat this.
Single-blind review. In traditional single-blind review, the reviewers know who the authors are and the authors
do not know who the reviewers are. This is probably the most hated kind of review, as it is subject to both of the
possible problems I mentioned above. It's also probably the most common kind of review in linguistics and psychology journals.
Some people like it because they believe it's important to know the authors' biases (e.g., if a paper claims to show evidence
that Theory X is the best theory ever, you might judge it differently depending on whether the author of the paper is the
person who created Theory X, versus an independent person).
Note that "single-blind" could also mean the authors know who the reviewers are and the reviewers do not know who
the author is. This model is rare, but some people have argued that it might be ideal—it would make reviewers
less likely to be assholes, and it would also protect against reviewers making biased recommendations based on the
authors' identities. On the other hand, it shares one of the potential problems that has been raised about open
review (below), which is that less-prestigious reviewers might not feel safe to criticize papers by famous authors.
Double-blind review. In double-blind review, neither the reviewers nor the authors know who each other is.
There is some weak evidence that double-blind reviewing reduces biases (see, for example, this
paper examining double-blind reviewing and gender bias at a linguistics conference). Some people argue that double-blind
reviewing is ineffective because, in a small field like ours, people can figure out who the author is anyway; however, some
research has suggested that reviewers are actually not good at guessing the identity of authors. Of course, double-blind
review cannot stop the problem of some reviewers acting like assholes behind a veil of anonymity. Overall, whether anonymity
is good or bad is a subject of much debate; being able to criticize popular ideas without being fired is a good thing about
anonymity, but being able to be an asshole is a bad thing about anonymity. (See this
article for a discussion of the role of anonymity in science.)
Open review. Authors and reviewers both know who each other are. This is the opposite extreme from double-blind
review. If knowing the authors' identity introduces bias, then open review will still be subject to that bias. However, it may
protect against anonymous reviewers behaving like assholes, since they can be accountable for what they say in their review.
Hybrid models. There are other combinations of these things. For example, as mentioned above, a model in which
authors know reviewers' identities but reviewers don't know authors' identities is logically possible, but rarely practiced at
journals. But it can happen de facto. Some reviewers, including me, always sign our reviews—whether this is a
good or a bad thing is another subject of substantial debate. If I review for a double-blind journal, where I don't know who
reviewers are, and I sign my review, then I have created a single-blind situation where the authors know who I am and I don't
know who they are.) Another interesting hybrid model is the one practiced by Frontiers journals: while review is going
on it's single-blind (reviewers know who the authors are but authors don't know who the reviewers are), but if the paper gets
accepted, the reviewers' names are published next to the article, so people can know who accepted it. If the paper gets
rejected, the reviewers' names will remain hidden. Thus, it's single-blind if a paper is rejected, but it becomes "open"
if a paper gets accepted.
The impact vs. methods dimension
Traditionally, journals would decide what research they consider
"impactful" (i.e., what research they think will make a big difference in the field, what research they think is important for
people to read) and publish that. Thus, they might reject papers not because of the scientific quality, but because of the impact
(e.g., if a paper is not very different from previous work, or doesn't make a big important conclusion, many journals might not
want to publish it).
In recent years, many new journals have emerged that follow a different model. These journals believe that it
should not be up to 3-4 people to judge whether a paper has the potential impact. Instead, they believe impact should be judged by
the field. Thus, they don't judge a paper's potential impact; instead, they publish any paper as long as the scientific methodology
is good, and then they let the field decide what's impactful. If, a few years after publication, a paper has been cited and
discussed a lot, then it clearly has made an impact on the field. On the other hand, if many years have passed since a paper was
published and it has never been cited, then clearly nobody cares about it. The idea of these journals is to let impact be determined
in that way, rather than letting a few reviewers say "I don't think this research is very important." Some journals that follow this
model include PLoS ONE, Frontiers (all the Frontiers journals), Collabra, and Scientific Reports.
At these journals, reviewers are meant to just focus on methods; this means that you can submit a paper there that only makes a
minor advancement relative to existing literature, but you'd better make sure your methodology is excellent. (It's also good to know
what kind of journal you are submitting to; peer reviewers are busy people and often don't have time to check all the details of the
journal they're reviewing for, so they might still make comments like "I don't think this research is impactful". In such a
situation, when you write your response-to-reviewers letter you should be prepared to say "actually impact is not one of the
criteria at this journal.)
The pre- vs. post-publication dimension
The abovementioned issue about judging impact vs. judging methodology has also led to a similar debate and an even
more radical proposal, which involves completely rejecting the traditional concept of journal publishing. In the traditional model
of publishing, reviewers and journals are seen as "gatekeepers": they decide what is good
enough to be published, and thus they basically decide what scientific knowledge deserves to be spread to the world.
In recent years, however, some people are starting to argue for a more democratic process. Their argument is that it should
not be the responsibility of 3-4 people to decide what research is worthy or not; instead, every paper should get "published", and
then the whole world can decide if it's good or not. Instead of reviewing papers before they're published and accepting or
blocking them, we should instead review papers after they're published. In such a model, "journals" as we know it might not
even exist at all; instead, everyone would just post their papers on open platforms (like ResearchGate, arXiv, and similar
platforms), and anyone in the world could comment on those papers, and then the role of "journals" would be to find the best papers
and compile them. For a detailed vision of how this might look, see the blog post
"Let's just get rid of peer review."
Do the activities below to reflect on what you've learned about peer review.
Choose any one of the dimensions discussed previously (anonymity, pre vs. post, or impact vs. methods). What do you
support, and why?
Note that all of these issues are widely discussed in blogs and social media; if you aren't sure what you support, you
are welcome to search online and read some discussions about these issues to help give you some ideas.
There are a lot of social media platforms where people are discussing and arguing about peer review all the time. A few examples:
Facebook groups such as "Reviewer 2 Must Be Stopped" (for example,
here is a recent discussion which touched
on many of the issues brought up in this activity, with many different and conflicting opinions expressed)
Check these or other groups (I'm sure similar groups exist in Chinese social media as well) to find at least one discussion about some
question or controversy regarding peer review. (These are public pages, so you don't need a Facebook or ResearchGate account to see
them.) Below, copy the link to the discussion, and let me know what you think about it (e.g., if it's a controversy about something
that happened in a peer review, say which "side" you think is "right", and why).
When you finish this activity, you are done with the module (assuming all your work on this and the
previous tasks has been satisfactory). However, you may still continue on to the advanced-level
task for this module if you wish to complete this module at the advanced level (if you're aiming for a higher
grade or if you are just particularly interested in this topic). Otherwise, you can return to the
module homepage to review this module, or return to the class homepage to select a different module
or assignment to do now.