Difference between revisions of "Peer Review"

From P2P Foundation
Jump to navigation Jump to search
Line 135: Line 135:
 
#Grazia Ietto-Gillies: Replacing [[Peer Review]] by a [http://www.paecon.net/PAEReview/issue45/IettoGillies45.pdf ex-post bottom up peer comments system]
 
#Grazia Ietto-Gillies: Replacing [[Peer Review]] by a [http://www.paecon.net/PAEReview/issue45/IettoGillies45.pdf ex-post bottom up peer comments system]
 
#Lessons from the History and Philosophy of Science regarding the Research Assessment Exercise, at http://www.ucl.ac.uk/sts/gillies/
 
#Lessons from the History and Philosophy of Science regarding the Research Assessment Exercise, at http://www.ucl.ac.uk/sts/gillies/
#History and current practice of peer review, overview article at http://www.textjournal.com.au/april08/johnston_krauth.htm
+
#History and current practice of peer review, overview article at http://www.textjournal.com.au/april08/johnston_krauth.htm This article by Prof. Donald Gillies shows examples of why an excessive reliance on peer review can impede scientific progress, as major advances were in their time rejected by their peers.
This article by Prof. Donald Gillies shows examples of why an excessive reliance on peer review can impede scientific progress, as major advances were in their time rejected by their peers.
+
#[http://terpconnect.umd.edu/~cpikas/PikasPeerReview012908.pdf Peer Review]: The View from Social Studies of Science, by Christina K Pikas
  
  

Revision as of 10:05, 5 March 2009

In Peer Review, scientific articles are vetted by scientific colleagues.


It should be noted that the process of vetting in peer production, i.e. Communal Validation, based on Anti-Credentialism, is different.

See also our entry on the new trend of Open Peer Review


Discussion

Michael Nielsen: Three Myths about Peer Review

Excerpted from http://michaelnielsen.org/blog/?p=531:

Myth number 1: Scientists have always used peer review

The myth that scientists adopted peer review broadly and early in the history of science is surprisingly widely believed, despite being false. It’s true that peer review has been used for a long time - a process recognizably similar to the modern system was in use as early as 1731, in the Royal Society of Edinburgh’s Medical Essays and Observations (ref). But in most scientific journals, peer review wasn’t routine until the middle of the twentieth century, a fact documented in historical papers by Burnham, Kronick, and Spier.

This was a common practice in the days before peer review became widespread: decisions about what to publish and what to reject were usually made by journal editors, often acting largely on their own. These decisions were often made rapidly, with papers appearing days or weeks after submission, after a cursory review by the editor. Rejection rates at most journals were low, with only obviously inappropriate or unsound material being rejected; indeed, for some Society journals, Society members even asserted a “right” to publication, which occasionally caused friction with unhappy editors (ref).

What caused the change to the modern system of near-ubiquitous peer review? There were three main factors. The first was the increasing specialization of science (ref). As science became more specialized in the early 20th century, editors gradually found it harder to make informed decisions about what was worth publishing, even by the relatively relaxed standards common in many journals at the time.

The second factor in the move to peer review was the enormous increase in the number of scientific papers being published (ref). In the 1800s and early 1900s, journals often had too few submissions. Journal editors would actively round up submissions to make sure their journals remained active. The role of many editorial boards was to make sure enough papers were being submitted; if the journal came up short, members of the editorial board would be asked to submit papers themselves. As late as 1938, the editor-in-chief of the prestigious journal Science relied on personal solicitations for most articles (ref).

The twentieth century saw a massive increase in the number of scientists, a much easier process for writing papers, due to technologies such as typewriters, photocopiers, and computers, and a gradually increasing emphasis on publication in decisions about jobs, tenure, grants and prizes. These factors greatly increased the number of papers being written, and added pressure for filtering mechanisms, such as peer review.

The third factor in the move to peer review (ref) was the introduction of technologies for copying papers. It’s just plain editorially difficult to implement peer review if you can’t easily make copies of papers. The first step along this road was the introduction of typewriters and carbon paper in the 1890s, followed by the commercial introduction of photocopiers in 1959. Both technologies made peer review much easier to implement.

Nowadays, of course, the single biggest factor preserving the peer review system is probably social inertia: in most fields of science, a journal that’s not peer-reviewed isn’t regarded as serious, and so new journals invariably promote the fact that they are peer reviewed. But it wasn’t always that way.


Myth number 2: peer review is reliable

Every scientist has a story (or ten) about how they were poorly treated by peer review - the important paper that was unfairly rejected, or the silly editor who ignored their sage advice as a referee. Despite this, many strongly presume that the system works “pretty well”, overall.

There’s not much systematic evidence for that presumption. In 2002 Jefferson et al (ref) surveyed published studies of biomedical peer review. After an extensive search, they found just 19 studies which made some attempt to eliminate obvious confounding factors. Of those, just two addressed the impact of peer review on quality, and just one addressed the impact of peer review on validity; most of the rest of the studies were concerned with questions like the effect of double-blind reviewing. Furthermore, for the three studies that addressed quality and validity, Jefferson et al concluded that there were other problems with the studies which meant the results were of limited general interest; as they put it, “Editorial peer review, although widely used, is largely untested and its effects are uncertain”.

In short, at least in biomedicine, there’s not much we know for sure about the reliability of peer review. My searches of the literature suggest that we know don’t much more in other areas of science. If anything, biomedicine seems to be unusually well served, in large part because several biomedical journals (perhaps most notably the Journal of the American Medical Association) have over the last 20 years put a lot of effort into building a community of people studying the effects of peer review; Jefferson et al’s study is one of the outcomes from that effort.

In the absence of compelling systematic studies, is there anything we can say about the reliability of peer review?

The question of reliability should, in my opinion, really be broken up into three questions. First, does peer review help verify the validity of scientific studies; second, does peer review help us filter scientific studies, making the higher quality ones easier to find, because they get into the “best” journals, i.e., the ones with the most stringent peer review; third, to what extent does peer review suppress innovation?

As regards validity and quality, you don’t have to look far to find striking examples suggesting that peer review is at best partially reliable as a check of validity and a filter of quality.

What about the suppression of innovation? Every scientist knows of major discoveries that ran into trouble with peer review. David Horrobin has a remarkable paper (ref) where he documents some of the discoveries almost suppressed by peer review; as he points out, he can’t list the discoveries that were in fact suppressed by peer review, because we don’t know what those were. His list makes horrifying reading.


Here’s just a few instances that I find striking, drawn in part from his list. Note that I’m restricting myself to suppression of papers by peer review; I believe peer review of grants and job applications probably has a much greater effect in suppressing innovation.

  • George Zweig’s paper announcing the discovery of quarks, one of the fundamental building blocks of matter, was rejected by Physical Review Letters. It was eventually issued as a CERN report.
  • Berson and Yalow’s work on radioimmunoassay, which led to a Nobel Prize, was rejected by both Science and the Journal of Clinical Investigation. It was eventually published in the Journal of Clinical Investigation.
  • Krebs’ work on the citric acid cycle, which led to a Nobel Prize, was rejected by Nature. It was published in Experientia.
  • Wiesner’s paper introducing quantum cryptography was initially rejected, finally appearing well over a decade after it was written.

To sum up: there is very little reliable evidence about the effect of peer review available from systematic studies; peer review is at best an imperfect filter for validity and quality; and peer review sometimes has a chilling effect, suppressing important scientific discoveries.

At this point I expect most readers will have concluded that I don’t much like the current peer review system. Actually, that’s not true, a point that will become evident in my post about the future of peer review. There’s a great deal that’s good about the current peer review system, and that’s worth preserving. However, I do believe that many people, both scientists and non-scientists, have a falsely exalted view of how well the current peer review system functions. What I’m trying to do in this post is to establish a more realistic view, and that means understanding some of the faults of the current system.


Myth: Peer review is the way we determine what’s right and wrong in science

By now, it should be clear that the peer review system must play only a partial role in determing what scientists think of as established science. There’s no sign, for example, that the lack of peer review in the 19th and early 20th century meant that scientists then were more confused than now about what results should be regarded as well established, and what should not. Nor does it appear that the unreliability of the peer review process leaves us in any great danger of collectively coming to believe, over the long run, things that are false.

In practice, of course, nearly all scientists understand that peer review is only part of a much more complex process by which we evaluate and refine scientific knowledge, gradually coming to (provisionally) accept some findings as well established, and discarding the rest. So, in that sense, this third myth isn’t one that’s widely believed within the scientific community. But in many scientists’ shorthand accounts of how science progresses, peer review is given a falsely exaggerated role, and this is reflected in the understanding many people in the general public have of how science works. Many times I’ve had non-scientists mention to me that a paper has been “peer-reviewed!”, as though that somehow establishes that it is correct, or high quality. I’ve encountered this, for example, in some very good journalists, and it’s a concern, for peer review is only a small part of a much more complex and much more reliable system by which we determine what scientific discoveries are worth taking further, and what should be discarded." (http://michaelnielsen.org/blog/?p=531)


Difference between Communal Validation and peer review

Peer production is based on equipotential participation (see Equipotentiality, i.e. the a priori self-selection of participants, and the communal vetting of the quality of their work in the process of production itself. Peer review is based on credentialism, peer production vetting is based on Anti-Credentialism. Peer review is part of an elaborate process of institutional and prior validation of what constitutes valid knowledge; peer production vetting is a posteriory vetting by the community of participants.


A quote on the difference between peer to peer processes and academic peer review:

“One of the early precedents of open source intelligence is the process of academic peer review. As academia established a long time ago, in the absence of fixed and absolute authorities, knowledge has to be established through the tentative process of consensus building. At the core of this process is peer review, the practice of peers evaluating each other's work, rather than relying on external judges. The specifics of the reviewing process are variable, depending on the discipline, but the basic principle is universal. Consensus cannot be imposed, it has to be reached. Dissenting voices cannot be silenced, except through the arduous process of social stigmatization. Of course, not all peers are really equal, not all voices carry the same weight. The opinions of those people to whom high reputation has been assigned by their peers carry more weight. Since reputation must be accumulated over time, these authoritative voices tend to come from established members of the group. This gives the practice of peer review an inherently conservative tendency, particularly when access to the peer group is strictly policed, as it is the case in academia, where diplomas and appointments are necessary to enter the elite circle. The point is that the authority held by some members of the group- which can, at times, distort the consensus-building process - is attributed to them by the group, therefore it cannot be maintained against the will of the other group members." (Felix Stalder in: http://news.openflows.org/article.pl?sid=02/04/23/1518208 )


Peer Review is not Obsolete

Ward Cunningham at http://www.re-public.gr/en/?p=141

"Does the proliferation of wikis mark the eventual end of peer review? How is this development changing the nature of scientific communities?


W.C.: Wiki does not threaten peer review. Science needs peer review and it will get it. I do not see knowledge produced through wikis as being on the same ground with scientific knowledge. Wiki is best seen as a way of reporting, sharing, coordinating, problem framing and agenda setting. A wiki works best where you’re trying to answer a question that you can’t easily pose, where there’s not a natural structure that’s known in advance to what you need to know.


Science is based on repeatable experiment. The peer review is a means of assessing the quality of the experiments, not voting on the preference for a particular result. But we should not forget that what you get as a wiki reader is access to people who had no voice before. The people to whom we are giving voice are aware of what it’s like to write, and ship, a computer program.


If you want to contribute to a scientific journal you should be peer reviewed. Part of peer review is that you’re familiar with all the other literature. And the other literature somehow that has spiralled off into irrelevance. What was being written about programming didn’t match what practicing programmers felt. With wiki, practicing programmers who don’t have time to master the literature and get a column in a journal that’s going to be read have a place where they could say things that are important to them. The wiki provides a different view. In fact you can tell when someone is writing on wiki from their personal experience versus when they are quoting what they last read. " (http://www.re-public.gr/en/?p=141)


Research reveals weaknesses of peer review processes

Summary of the research from http://jp.senescence.info/thoughts/peer_review.html

"Although peer review is plagued by elitism, bias, and abuse, it is deemed by many as essential to the scientific process. Ironically, peer review has no valid scientific base. The few studies done on peer review suggest that reviewers -- also known as referees -- vary markedly in their opinions. Moreover, peer review does not prevent scientific fraud, hardly detects errors, and only modestly improves scientific quality (Smith, 1999; van Rooyen et al., 1999; Rothwell and Martyn, 2000). Importantly, peer review continues to be a conservative process that smothers innovative and unconventional ideas." (http://jp.senescence.info/thoughts/peer_review.html)


Limitations of the present regime of peer review

By M. Guedon at http://scholarlypublishing.blogspot.com/2007/07/scholarly-communication-open-access-and.html

"the present system is too rigid, too unwieldy to permit such small-scale, yet potentially crucial interventions. To make the proper corrections, one would have to republish and perhaps even go through the publisher if it is in print. The communication process is therefore limited or blocked.

There is a second type of difficulty: the present system of scholarly publishing relies more on a credential system and a co-operative system rather than on the intrinsic quality of individual intelligence and the excellence of the submitted text. One does not enter scientific or scholarly territories without showing the right kinds of references - diplomas, titles, names of institutions, etc. As a result, the scientific and scholarly enterprises work as a two-tier system where the authorized write and read and the others do not write and often cannot read because of economic barriers, such as high subscription prices and lack of affiliation to the right library).

To address these obstacles, M. Guédon touches on the granularity issue. The article is not the only possible model to contribute to scholarly or scientific research. This is even truer of the monograph in the humanities and, in fact, the article has superseded the monograph in most disciplines. He suggests that knowledge should be regarded as a conversation. People should freely be able to contribute to it. In the scientific community for example, moving closer to a wikipedia model could be the way of the future as knowledge would be made available to everyone; it can be created together, modified on a global scale, improved upon, and so forth. However, the argument of quality comes to mind. He counters that the present criteria for quality inherently rest on a hierarchical vision of society. When excellence is sought, the greater the number of minds involved, the greater the quality of the work done: the case of free software and some recent analyses of Wikipedia confirm this general rule. The greater the numbers of people involved in an issue, the better the answers are crafted. Consequently, the lines that separate the experts from the rest of society should be erased. We will always have experts in various fields, but to limit contributions to knowledge as a whole to experts only is to deprive all of humanity of its enormous potential for distributed intelligence." (http://scholarlypublishing.blogspot.com/2007/07/scholarly-communication-open-access-and.html)


More Discussion

Three part critique by Samir Chopra and Scott Dexter:

"we go on to talk about open, non-anonymous peer review as a particular solution, and about free software's methods of peer review and its value as an ideal for the practice of computer science at large. In the second post, I want to talk a bit about how badly, it seems to me, peer review is busted in the sciences. This will be anecdotal, insofar as I will be reliant upon my own experiences and observations. Still, considered as a report from the trenches, it might have some value for the reader. I should also qualify my comments by saying that while peer review seems to work reasonably well in journal article review, it is undeniably broke in conference article and grant proposal review, two fairly large and important parts of the practice of science today. We can then return to the solutions mentioned above." (http://decodingliberation.blogspot.com/2008/02/problems-with-peer-review-part-one.html)

Read: Part One ; Part Two; Part Three

More Information

  1. Stevan Harnad on why we need peer review, at http://cogprints.org/1646/
  2. Grazia Ietto-Gillies: Replacing Peer Review by a ex-post bottom up peer comments system
  3. Lessons from the History and Philosophy of Science regarding the Research Assessment Exercise, at http://www.ucl.ac.uk/sts/gillies/
  4. History and current practice of peer review, overview article at http://www.textjournal.com.au/april08/johnston_krauth.htm This article by Prof. Donald Gillies shows examples of why an excessive reliance on peer review can impede scientific progress, as major advances were in their time rejected by their peers.
  5. Peer Review: The View from Social Studies of Science, by Christina K Pikas