What you think of as peer review
When you have finished writing your scientific paper, you send it to a journal or conference and receive reports full of criticism from a few reviewers who are usually anonymous. They are supposed to be your peers, hence the name peer review. The practice of asking reviewers for advice on whether or not to publish a paper started in the 17th century but only became popular after the invention of the photocopier in 1959. Albert Einstein was one of the many scientists who thought peer review was a bad idea. I got that from this recent paper on the peer review crisis.
I’m with Albert on this one. I wrote my first paper thirty years ago, my first review a year later, and have since (co-)authored over a thousand peer-reviewed papers, reviewed thousands of papers myself, and read many more reviews as an associate editor or conference organizer. I’ve come to the conclusion that peer review is completely broken. I hope to elaborate on that in future posts. Here, I only want to stress that traditional peer review is just one of at least five ways in which the value of a paper is judged by other researchers.
1. Authorship
This is not typically seen as peer review, but your co-authors should critically read your paper and help revise it. By agreeing to be a co-author, they give permission to put their name on it, effectively giving it a stamp of approval. I think this is the most important form of peer review. Therefore, you should post your paper immediately on a preprint server such as arXiv and the commonly heard statement that these papers are not peer reviewed is simply incorrect. Unfortunately, PubMed does not yet include all arXiv papers, though they are running a pilot on it. I think publishing just on arXiv is the best way forward, and I believe overlay journals are the future.
2. Peer review
After your co-authors approve the paper, the next step is usually peer review, as explained above. So off you go in a cycle of major and minor revisions and often resubmission to multiple journals where the whole damn process starts again. An extremely inefficient process… Some solutions have been proposaed but academia is extrmely conservative, so so far these alternatives have not gained traction. Also a topic for another post.
Occasionally, you get a good review with advice that actually makes your paper better, but more often, you have to comply with the whims of so-called peers who typically rushed their report without properly reading your work because they had no time.
I have come to believe that peer review more often makes papers worse rather than better. I’m thinking about a way to study that scientifically. Feel free to contact me if you have ideas about this or would like to study it together.
3. Acceptance by a journal or a conference
If all goes well, your paper ends up in some prestigious conference proceedings or a journal. This is also a form of peer review. That venue has judged your paper good enough to print. The impact factor of a journal or conference is a measure of its prestige. Google Scholar uses its own metric to rank publications, the h5-index. You may note, surprisingly for almost all academics, that deep learning is currently the most important field of research on the planet.
4. Post-publication peer review
After publication, other researchers can praise your work or at least discuss it—in editorials, blog posts, podcasts, social media, or other platforms. Attempts to formalize open post-publication peer review in dedicated platforms have not been successful. As long as researchers are too busy to do regular peer review, this is not surprising. However, I think this form of peer review will become more important in the future.
5. Citations
You have managed to get your paper into a high-impact journal—congratulations! But many papers, even in top journals, receive almost no citations. The impact factor is just an average number of citations for recently published papers in that journal, and it is dominated by a few extremely highly cited papers, often review papers with little scientific value. Therefore, citations to your paper are a much better metric of positive peer review than the number of publications a researcher has managed to get into highly ranked journals or conferences.
If I want to find out more about a researcher—for example, when reviewing a grant they wrote—I always look at their Google Scholar profile, even though some funding agencies now forbid this. It provides a nice, quick overview. I think of it as a standardized CV. The h-index itself is not that meaningful, because every paper counts toward it, even if you are in the middle of a list of 100 authors. But on the first page of the Google Scholar profile, you see their most cited papers, which gives an immediate impression of what they work on, what career stage they are in, whether they have a proficient lab or tend to work alone, and so on. All peer-reviewed via citations!