Peer Review

From P2P Foundation
Jump to navigation Jump to search

In Peer Review, scientific articles are vetted by scientific colleagues.


It should be noted that the process of vetting in peer production, i.e. Communal Validation, based on Anti-Credentialism, is different.

See also our entry on the new trend of Open Peer Review


Discussion

Difference between Communal Validation and peer review

Peer production is based on equipotential participation (see Equipotentiality, i.e. the a priori self-selection of participants, and the communal vetting of the quality of their work in the process of production itself. Peer review is based on credentialism, peer production vetting is based on Anti-Credentialism. Peer review is part of an elaborate process of institutional and prior validation of what constitutes valid knowledge; peer production vetting is a posteriory vetting by the community of participants.


A quote on the difference between peer to peer processes and academic peer review:

“One of the early precedents of open source intelligence is the process of academic peer review. As academia established a long time ago, in the absence of fixed and absolute authorities, knowledge has to be established through the tentative process of consensus building. At the core of this process is peer review, the practice of peers evaluating each other's work, rather than relying on external judges. The specifics of the reviewing process are variable, depending on the discipline, but the basic principle is universal. Consensus cannot be imposed, it has to be reached. Dissenting voices cannot be silenced, except through the arduous process of social stigmatization. Of course, not all peers are really equal, not all voices carry the same weight. The opinions of those people to whom high reputation has been assigned by their peers carry more weight. Since reputation must be accumulated over time, these authoritative voices tend to come from established members of the group. This gives the practice of peer review an inherently conservative tendency, particularly when access to the peer group is strictly policed, as it is the case in academia, where diplomas and appointments are necessary to enter the elite circle. The point is that the authority held by some members of the group- which can, at times, distort the consensus-building process - is attributed to them by the group, therefore it cannot be maintained against the will of the other group members." (Felix Stalder in: http://news.openflows.org/article.pl?sid=02/04/23/1518208 )


Peer Review is not Obsolete

Ward Cunningham at http://www.re-public.gr/en/?p=141

"Does the proliferation of wikis mark the eventual end of peer review? How is this development changing the nature of scientific communities?


W.C.: Wiki does not threaten peer review. Science needs peer review and it will get it. I do not see knowledge produced through wikis as being on the same ground with scientific knowledge. Wiki is best seen as a way of reporting, sharing, coordinating, problem framing and agenda setting. A wiki works best where you’re trying to answer a question that you can’t easily pose, where there’s not a natural structure that’s known in advance to what you need to know.


Science is based on repeatable experiment. The peer review is a means of assessing the quality of the experiments, not voting on the preference for a particular result. But we should not forget that what you get as a wiki reader is access to people who had no voice before. The people to whom we are giving voice are aware of what it’s like to write, and ship, a computer program.


If you want to contribute to a scientific journal you should be peer reviewed. Part of peer review is that you’re familiar with all the other literature. And the other literature somehow that has spiralled off into irrelevance. What was being written about programming didn’t match what practicing programmers felt. With wiki, practicing programmers who don’t have time to master the literature and get a column in a journal that’s going to be read have a place where they could say things that are important to them. The wiki provides a different view. In fact you can tell when someone is writing on wiki from their personal experience versus when they are quoting what they last read. " (http://www.re-public.gr/en/?p=141)


Research reveals weaknesses of peer review processes

Summary of the research from http://jp.senescence.info/thoughts/peer_review.html

"Although peer review is plagued by elitism, bias, and abuse, it is deemed by many as essential to the scientific process. Ironically, peer review has no valid scientific base. The few studies done on peer review suggest that reviewers -- also known as referees -- vary markedly in their opinions. Moreover, peer review does not prevent scientific fraud, hardly detects errors, and only modestly improves scientific quality (Smith, 1999; van Rooyen et al., 1999; Rothwell and Martyn, 2000). Importantly, peer review continues to be a conservative process that smothers innovative and unconventional ideas." (http://jp.senescence.info/thoughts/peer_review.html)


Limitations of the present regime of peer review

By M. Guedon at http://scholarlypublishing.blogspot.com/2007/07/scholarly-communication-open-access-and.html

"the present system is too rigid, too unwieldy to permit such small-scale, yet potentially crucial interventions. To make the proper corrections, one would have to republish and perhaps even go through the publisher if it is in print. The communication process is therefore limited or blocked.

There is a second type of difficulty: the present system of scholarly publishing relies more on a credential system and a co-operative system rather than on the intrinsic quality of individual intelligence and the excellence of the submitted text. One does not enter scientific or scholarly territories without showing the right kinds of references - diplomas, titles, names of institutions, etc. As a result, the scientific and scholarly enterprises work as a two-tier system where the authorized write and read and the others do not write and often cannot read because of economic barriers, such as high subscription prices and lack of affiliation to the right library).

To address these obstacles, M. Guédon touches on the granularity issue. The article is not the only possible model to contribute to scholarly or scientific research. This is even truer of the monograph in the humanities and, in fact, the article has superseded the monograph in most disciplines. He suggests that knowledge should be regarded as a conversation. People should freely be able to contribute to it. In the scientific community for example, moving closer to a wikipedia model could be the way of the future as knowledge would be made available to everyone; it can be created together, modified on a global scale, improved upon, and so forth. However, the argument of quality comes to mind. He counters that the present criteria for quality inherently rest on a hierarchical vision of society. When excellence is sought, the greater the number of minds involved, the greater the quality of the work done: the case of free software and some recent analyses of Wikipedia confirm this general rule. The greater the numbers of people involved in an issue, the better the answers are crafted. Consequently, the lines that separate the experts from the rest of society should be erased. We will always have experts in various fields, but to limit contributions to knowledge as a whole to experts only is to deprive all of humanity of its enormous potential for distributed intelligence." (http://scholarlypublishing.blogspot.com/2007/07/scholarly-communication-open-access-and.html)


More Discussion

Three part critique by Samir Chopra and Scott Dexter:

"we go on to talk about open, non-anonymous peer review as a particular solution, and about free software's methods of peer review and its value as an ideal for the practice of computer science at large. In the second post, I want to talk a bit about how badly, it seems to me, peer review is busted in the sciences. This will be anecdotal, insofar as I will be reliant upon my own experiences and observations. Still, considered as a report from the trenches, it might have some value for the reader. I should also qualify my comments by saying that while peer review seems to work reasonably well in journal article review, it is undeniably broke in conference article and grant proposal review, two fairly large and important parts of the practice of science today. We can then return to the solutions mentioned above." (http://decodingliberation.blogspot.com/2008/02/problems-with-peer-review-part-one.html)

Read: Part One ; Part Two; Part Three

More Information

  1. Stevan Harnad on why we need peer review, at http://cogprints.org/1646/
  2. Grazia Ietto-Gillies: Replacing Peer Review by a ex-post bottom up peer comments system
  3. Lessons from the History and Philosophy of Science regarding the Research Assessment Exercise, at http://www.ucl.ac.uk/sts/gillies/
  4. History and current practice of peer review, overview article at http://www.textjournal.com.au/april08/johnston_krauth.htm

This article by Prof. Donald Gillies shows examples of why an excessive reliance on peer review can impede scientific progress, as major advances were in their time rejected by their peers.