Peer Review

From SkepticWiki

Peer Review is the process used by academic journals to check for errors in fact or methodology in papers submitted for publication.

1 Why Peer Review?
2 The Peer Review Process
3 Benefits of Peer Review
4 Problems with the Peer Review System
5 References
6 Related Articles

[edit] Why Peer Review?

Scientific papers are written by humans, and humans—even the most respected humans in the most prestigious of positions—make mistakes. They have biases. They have blind spots. They have things about their psyche that cause them, even subconsciously or accidentally, to come to the wrong conclusion.

Worse, some people commit actual fraud. They lie, cheat, obfuscate, cherry-pick data, and engage in other behavior designed to mislead others, for the purpose of pushing their agenda or acquiring a little unearned recognition.

People just cannot be relied on to find all of their own mistakes, nor can they be relied on to be honest 100% of the time. The problem is not greatly improved with a team of authors instead of just one. Peer Review is a means of quality control that scientific journals use to find the mistakes in a submission before the paper is actually published.

[edit] The Peer Review Process

An author's submission is given to a number of peers (also called referees) for review. These peers are exactly that; they should have the expertise needed to find the flaws in fact or methodology of whatever the paper covers. For example, a paper that relies on statistics should be reviewed by statisticians. Since papers generally cover several such areas, peers in each relevant subject area must be found.

The journal puts out a call for volunteers in these areas. Once a suitable panel is found, they are given the submission to evaluate. They are charged with two tasks. One is to identify the problems in the paper. The hope is, if enough peers are found in enough relevant areas, then almost all of the errors in the paper will be detected. The second is to describe the error and help the author revise his paper in order to correct the mistake.

The peers then send their findings to an editor, who then decides to accept or reject the paper on the basis of the review. If the paper is rejected, the author is sent the results of the peer review along with the rejection notice so that he can make revisions to his paper and resubmit it. Once resubmitted, the paper must be run through the peer review process again.

Peers generally are not paid for their duties. It is thought that this helps to preserve the integrity of the system as well as keep costs down.

[edit] Benefits of Peer Review

The better the Peer Review process, the more respected the journal. Two of the most respected journals are Science and Nature, and both are very rigorous in their peer reviews. In contrast, many feel that the lack of a peer review system is why National Geographic ended up being embarrassed when they published a cover story on the Archaeoraptor, which very soon after turned out to be a hoax.

Peer Review provides a benefit to both authors and researchers by issuing a thorough critique of the paper. The results of the peer review may be published along with the paper as commentary, as well as the author's response to the review. This gives all parties full access to all aspects of the information and debate surrounding the paper.

Peer Review encourages authors to be more thorough and self-policing, and gives both the scientific community and the public at large a method of obtaining quality information about scientific discoveries.

[edit] Problems with the Peer Review System

Just as authors are people who make mistakes, so are peers and editors. A peer review is only as good as the peers who do the reviewing. A bad selection of referees could reject a worthy paper or, even worse, cause a questionable paper to be approved for publication, resulting in embarrassment for the journal. Most papers are eventually published, but the peer review process is expensive. It can actually cost more for a journal to reject a paper, since the paper must be reviewed again upon resubmission.

Less reputable authors can also "shop around" until their paper is published. If their paper is rejected by a highly respected journal such as Science or Nature, they can work their way down to journals that are less rigorous in their peer review. They may eventually get published, but along the way they will use up considerable resources of all of the journals they submit to in the process.

Selecting suitable referees can be difficult. The editor or his staff must select appropriate experts from many different fields every time a call for volunteers is put out. They will have their hands full going through all of the responses, and this difficulty is multiplied because of the number of papers that are received. So they generally do not have time to thoroughly check credentials. Since the referees are generally unpaid volunteers, often those with an agenda are more prone to volunteering.

Another problem is that the peer review process is not suitable for everything. If an author submits the results of a study, the referees can check his math and his overall methodology, but they have no way of verifying the actual results. They do not repeat the study, nor do they conduct research and interviews of the study and those involved in it. The more theoretical and experimental a subject is, the less effective the peer review process. This also means that the peers have no way of detecting fraudulent data.

All of this amounts to the fact that Peer Review is not 100% effective at finding all of the problems and exposing all of the frauds. However, it is still an effective method, depending on how diligently the process is adhered to, and is certainly preferable to not peer-reviewing at all.