Is This Article Worth Reading? A Critical Analysis of Peer Review

by John Hardie, BDS, MSc, PhD, FRCDC

Introduction

Dr. John O’Keefe, Editor- in- Chief of the Journal of the Canadian Dental Association is of the opinion that, “The peer review process is the cornerstone of the JCDA. It ensures that the material presented in the publication meets certain criteria of quality, accuracy, and relevance to practice.”1 Dr. Richard Smith, a former editor of the British Medical Journal and for 13 years the Chief Executive of the British Medical Journal Publishing Group, is adamant that peer review, “is a flawed process, full of easily identified defects with little evidence that it works.”2

Both of these diametrically opposed views cannot be correct. The purpose of this article is to determine which is closer to reality.

Brief History

The development of peer review is shrouded in history. In 1731, the editor of Medical Essays and Observations (published by the Royal Society of Edinburgh) forwarded submitted articles for review to those whom he considered “most versed in these matters.”3 While the Royal College of Edinburgh might have been the first authoritative body to recognize the potential value of peer review, it did so with the significant disclaimer that peer review did not guarantee accuracy or truthfulness or even that accepted papers were better than non-peer reviewed ones.3

Editorial peer review on a casual unorganized manner became more common in the mid-19th century with the publication of an ever-expanding number of biomedical journals. It was not until the increasingly specialized world of the 20th century, with its demands for expert authority, that peer review became more frequently accepted and expected as a “stamp of approval.”3 To-day, peer review has become institutionalized and is used by most biomedical journals.3

Dr. Robbie Fox, editor of the Lancet from 1944 to 1964 was no admirer of peer review. He considered it to be an unimportant process.2 Indeed, the Lancet did not commence peer review until 1976, long after it had gained a justified worldwide reputation as a pre-eminent medical journal.2 This questions if there is any direct relationship between a journal’s reputation and its practice of peer review. In fact, Dr. Fox often wondered if any untoward consequences would occur if rejected papers replaced those that had been accepted.2

Thus, from its earliest days peer review generated questions as to its usefulness.

Definition

What is editorial peer review? It is a process whereby editors send submitted articles for assessment to persons deemed knowledgeable in the subject matter in order that only the best papers are published. A more comprehensive description is that peer review selects articles for publication and rejects those that are considered to be irrelevant, trivial, weak, misleading or potentially harmful while simultaneously improving the clarity, transparency, accuracy and utility of the submissions.4 The common perception of journal peer review is that it improves or enhances the quality of a paper and assists the editor in accepting it for publication.5 If peer review satisfied any of these descriptions it would act as a quality control filter by identifying only the best papers that were worthy of publication. Indeed, many knowledgeable individuals readily accept peer review as a guarantor of quality,6 as providing added value for a publication2 and even as a form of “Good Housekeeping Seal of Approval.”7

Dr. Richard Smith categorically rejects these beliefs as he is of the opinion that it is impossible to define peer review in operational terms.2 His reasons are that the words, “expert”, “peer”, “review”, and “quality” are mired in vagueness. As the former editor of the British Medical Journal, he questions if a peer is someone doing the same research and therefore a potential competitor, or merely someone in the same discipline who might be less informed than the author of the article. He has concerns that a review might involve only a casual reading of a paper as opposed to a thorough analysis that checks all data and references and offers constructive improvements. He worries that there is simply no agreed definition on what constitutes a good or quality paper.2

Dr. Smith’s frustration is understandable since there is no governing body that defines what is good peer review or which demands that certain standards be followed.7 This implies that journal editors have considerable latitude as to what they believe constitutes peer review. Without accepted and approved standards, readers and authors are subjected to peer review that reflects the very human foibles, biases and prejudices of editors and their reviewers.

Significant Concerns

By 1985, Lock and Bailar had conducted extensive investigations into the editorial peer review process and found that much of its methodology was flawed and was in desperate need of improving.8,9

These finding were supported in 1986 by Dr. Drummond Rennie, then Senior Contributing Editor of the Journal of the American Medical Association, who wondered why, if peer review was a quality control filter, most papers no matter how trivial, obscure, contradictory or fragmented eventually passed the review scrutiny and were published.5 Dr. Rennie composed his concerns into the following series of questions.

Are there generally accepted criteria as to what constitutes a paper? Is there a specific list of criteria applied to papers by all editors-or even some- in deciding whether to accept or reject or send out for review? If the latter, how are reviewers selected and how many should there be? Since expertise is required, what degree of conflict of interest is expected or allowed? What instructions are reviewers given? How are their reviews assessed? Should reviews be double-blinded? How do editors reconcile disagreement between reviewers, and between reviewers and authors? How much does this all cost in editorial, secretarial, and review hours as well as in mailing costs? How much does the process prevent fraudulent research? Does it encourage plagiarism and breaks in confidentiality? Would we be not all be better off with no review?5 Dr. Rennie went as far as to suggest that the only true review process was to have extensive correspondence sections where authors and their critics could debate in public.5

Dr. Rennie challenged researchers to find answers to his concerns and to present their findings in the spring of 1989 at an International Congress on “Peer Review in Biomedical Publications” sponsored by the American Medical Association.5 He accompanied the invitation by the insightful comment that, “research may find we would be better off to scrap peer review entirely.”5 The first International Congress in 1989 has been followed by five more with the last one being held in Vancouver in 2009.

Researchers accepted Dr. Rennie’s initial challenge. However, approximately ten years later, few of his concerns had been addressed. For example, a 1997 article in the British Medical Journal concluded that, “The problem with peer review is that we have good evidence on its deficiencies and poor evidence on its benefits. We know that it is expensive, slow, prone to bias, open to abuse, possible anti-innovatory, and unable to detect fraud. We also know that the published papers that emerge from the process are often grossly deficient.”10

In 2001 at the Fourth International Congress, Jefferson and colleagues presented their findings of an extensive systematic analysis of peer review methodology. The results convinced them that editorial peer re
view was an untested practice whose benefits were uncertain.11 Dr. Rennie left the Fourth Congress with his original concerns intact as evidenced by his opinion that, “Indeed, if the entire peer review system did not exist but were now to be proposed as a new invention, it would be hard to convince editors looking at the evidence to go through the trouble and expense.”12

Evidence

There is supporting evidence for the concerns expressed by Lock, Bailar, Rennie and Jefferson. Recent papers by Wager, Smith and Benos provide numerous examples of studies that demonstrate methodological flaws in peer review that, in turn, cast suspicion on the value of articles approved by the process.13,2,3 A few of the evidential studies will be described.

Errors

In a 1998 investigation, 200 reviewers failed to detect 75% of the mistakes that were deliberately inserted into a research article.14 In the same year, reviewers failed to identify 66% of the major errors introduced into a fake manuscript.15 A paper that eventually resulted in its author being awarded a Nobel Prize was rejected because the reviewer believed that the particles on the microscopic slide were deposits of dirt rather than evidence of the hepatitis B virus.16

Consistency and Objectivity

There is a belief that peer review is an objective, reliable and consistent process. A study by Peters and Ceci questions that myth. They resubmitted 12 published articles from prestigious institutions to the same journals that had accepted them 18-32 months previously. The only changes were in the original authors names and affiliations. One was accepted (again) for publication. Eight were rejected not because they were unoriginal but because of methodological weaknesses, and only three were identified as being duplicates.17 Smith illustrates the inconsistency among reviewers by this example of their comments on the same paper.

Reviewer A “I found this paper an extremely muddled paper with a large number of defects.”

Reviewer B “It is written in a clear style and would be understood by any reader.”2

Without standards that are uniformly accepted and implemented peer review is a subjective and inconsistent process.

Fraud

Peer review failed to identify that the cell biologist Wook Suk Hwang had made false claims regarding his creation of 11 human embryonic stem cell lines.3 Reviewers at such high profile journals as Science and Nature did not identify the many gross anomalies and fraudulent results that Jan Hendrick Schon made in numerous papers while acting as a researcher at Bell Laboratories.3 The US Office of Research Integrity has produced information on data fabrication and falsification that appeared in over 30 peer reviewed papers published by such respected journals as Blood, Nature, and the Proceedings of the National Academy of Science.18 In fact, a reviewer for the Proceedings of the National Academy of Science was found to have abused his position by falsely claiming to be working on a study that he was asked to review.19

Editorial peer review may deem a paper worthy of publication according to self-imposed criteria. The process, however, cannot ensure that the paper is truthful and devoid of fraud.3

Quality

Supporters of peer review promote its quality enhancing powers. Defining and identifying quality are not simple tasks. Jefferson and colleagues analysed a number of investigations that attempted to assess the quality of peer reviewed articles.4 They found no consistencies in the criteria that were used, and a multiplicity of rating systems most of which were not validated and were of low reliability. They suggested that quality criteria include, the importance, relevance, usefulness, and methodological and ethical soundness of the submission along with the clarity, accuracy and completeness of the text.4 They included indicators that could be used to determine to what degree each criterion had been obtained. The ideas promoted by Jefferson et al have not been encoded into standards against which any peer review may be assessed. Until this occurs, editors and reviewers have complete freedom to define quality according to their individual or collective whims. This supports Smith’s contention that there is no agreed upon definition of a good or quality paper.2

In consideration of the above, peer review is not the hallmark of quality except, perhaps, in the beliefs of its practitioners.

Accuracy

It might be assumed that peer reviewed articles were error free and statistically sound. In 1999, a study by Pitkin of major medical journals found a 18-68% rate of inconsistencies between details in abstracts compared with what appeared in the main text.20 An investigation of 64 peer review journals demonstrated a median proportion of inaccurate references of 36% (range 4-67%).21 The median proportion of errors so serious that reference retrieval was impossible was 8% (range 0-38%).21 The same study showed that the median percentage of inaccurate quotations was 20%. Randomized controlled trials are considered the “gold standard” of evidence-based care. A significant study of the quality of such trials appearing in peer review journals was completed in 1998. The results showed that 60-89% of the publications did not include details on sample size, confidence intervals, and lacked sufficient details on randomization and treatment allocation.22

It should not be assumed that peer review guarantees accuracy.

Reviewers

The role of the reviewer is complex. They are expected to assess the scientific or technical merits of a paper, determine its novelty, consider its clarity of presentation, verify the pertinence of illustrations, tables and references, identify flaws, suggest improvements and offer a decision regarding acceptance or rejection, usually in a timely manner and without any monetary compensation.4 Since reviewers are central to the success of the peer review process they should be chosen with care and deliberation. In 2007 a study of 306 experienced reviewers confirmed that, “there are no easily identifiable types of formal training or experience that predict reviewer performance. Skill in scientific peer review may be as ill defined and hard to impart as is common sense.”23 Currently there is no good method of assessing reviewer performance and no uniformly acceptable system for choosing them.24 It appears that many reviewers are selected because an editor deems them to be “experts’ and that they are available.3

Some reviewers will be knowledgeable and thorough, some will be biased and have conflicts of interest, while others will refuse to consider unconventional or controversial topics. Unfortunately, such characteristics are not made available to authors and readers.

Competition

Commercial journals compete for the same readers as peer reviewed journals. They have high advertisement-to-text ratios, are usually free, are not affiliated with professional associations, seldom have peer review articles and are rarely if ever cited in the medical/dental literature.25 Nevertheless, a study by Rochon and colleagues showed that compared with peer review journals, articles in for-profit commercial publications were considered by physicians to be easier to read and understand, to address more important issues and to provide more information relevant to clinical practice.25

Purpose

In his controversial paper, “Peer review a flawed process at the heart of science and journals” Dr. Smith asks the critical question, “What is peer review for?”2

Reviewers might consider it as an ego booster and as an enhancement to academic and professional careers. Editors might see peer review as a decision making aid regarding the rejection or acceptance of articles, while adding a certain cache to their journals. Authors find the delays in decisions and editors comments frustrating, and even those whose papers are accepted are disappointed with the quality of reviews.26 However, Dr. Smith’s question is answered by stating that there is little purpose in peer review unless it has a direct bearing on patient care.

To-day, systematic reviews, meta-analyses, and expert opinion are the foundation of evidence-based care. These are usually subjected to peer review. As has been demonstrated above, peer review is a far from perfect process. It has no standards regarding accuracy, quality and consistency, and no objective means of assessing the competency of its reviewers. Therefore, relying on peer review to direct patient care might be less than prudent. Not surprisingly, very little is known about the beneficial effects, if any, that peer review has on improving health status.4 Indeed, it is possible that it is less successful at this task than are the articles and clinical pointers published in the commercial journals.25

The ambiguity associated with peer review and its effect on patient care is demonstrated in recent editions of the Journal of the Canadian Dental Association.27,28 The Journal published six “Point of Care” articles in these editions. Two were subjected to peer review but four were not. Are readers to assume that these four are inferior even if they find the contents to be pertinent and of practical value? Should any confusion be avoided by simply ignoring the peer review designation? In which case, why have it?

Conclusion

Peer review is an imperfect process that has attracted considerable but justifiable adverse criticism. A Cochrane review of the subject found little evidence to support the efficacy of editorial peer review.29 Consequently, it would appear that Dr. Smith’s opinions are more realistic than those of Dr. O’Keefe’s.

Nevertheless, it may be premature to abandon peer review as a lost cause. The challenge facing all editors who support peer review is to prove, in a transparent and objective manner, that it always rejects irrelevant, misleading, trivial and weak papers, improves the accuracy, clarity, and usefulness of accepted papers, and has a positive effect on patient care.

If that challenge is met, peer review will have attributes that will justify its existence.

In the meantime, readers would be well advised to judge all articles on their merits irrespective of any peer review designation.

Dr. Hardie was intimately involved in the development of the RCDSO 1996 evidence-based guidelines. Since then he has maintained an interest in this topic and how it and related diseases have influenced dental infection control recommendations.

Oral Health welcomes this original article.

References

1. O’Keefe J. A Salute to Our Reviewers. J Can Dent Assoc 2010; 76(1):21.

2. Smith R. Peer review: a flawed process at the heart of science and journals. J R Soc Med 2006; 99:178-182.

3. Benos DJ, Bashiri E, Chaves JM et al. The ups and downs of peer review. Adv.Physiol Educ 2007; 31:145-152.

4. Jefferson T, Wager E, Davidoff F. Measuring the Quality of Editorial Peer Review. JAMA 2002; 287(21):2786-2789.

5. Rennie D. Guarding the Guardians: A Conference on Editorial Peer Review. JAMA 1986; 256(17):2391-2392.

6. McCullough BD, McKintrick R. Check the Numbers: The Case for Due Diligence in Policy Formation. Studies in Risk and Regulation. Fraser Institute, February 2009.

7. Kranish M. Flaws are found in validating medical studies. Many see need to overhaul standards for peer review. The Boston Globe, August 15th, 2005.

8. Bailar JC, Patterson K. Journal peer review: The need for a research agenda. N Engl J Med 1985; 312:654-657.

9. Lock S. A Difficult Balance: Editorial Peer Review in Medicine. London, The Nuffield Provincial Hospitals Trust, 1985.

10. Smith R. Peer Review: Reform or Revolution. BMJ 1997; 315:759-760.

11. Jefferson T, Alderson P, Wager E, Davidoff F. Effects of Editorial Peer Review. A Systematic Review. JAMA 2002; 287:2784-2786.

12. Rennie D. Fourth International Congress on Peer Review in Biomedical Publication. JAMA 2002; 287:2759- 2760.

13. Wager E, Jefferson T. Shortcomings of peer review in biomedical journals. Learned Publishing 2001; 14:257-263.

14. Godlee F, Gale CR, Martyn CN. Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: a randomized controlled trial. JAMA 1998; 280:237-240.

15. Callaham ML, Baxt WG, Waeckerle JF, Wears RL. Reliability of editors’ subjective quality ratings of peer reviews of manuscripts. JAMA 1989; 280:229-231.

16. Lock S. The grossest failures of peer review. BMJ 1993; 307:382.

17. Peters DP, Ceci SJ. Peer-review practices of psychological journals: the fate of published articles, submitted again. Behavioral and Brain Sciences 1982; 5:187-195.

18. Office of Research Integrity Annual report 1999. Washington, DC: Department of Health and Human Services. 1999.

19. Rennie D. Misconduct and peer review. In Godlee F, Jefferson T (eds), Peer Review in Health Sciences. London: BMJ Books, 1999, 90-99.

20. Pitkin R, Branagan MA, Burmeister LF. Accuracy of data in abstracts of published research articles. JAMA 1999; 281:1110-1111.

21. Wager E, Middleton P. Reference accuracy in peer reviewed journals: a systematic review. Accepted for presentation at 4th International Congress on Peer Review in Biomedical Publication. Barcelona, September, 2001.

22. Moher D. CONSORT: an evolving tool to help improve the quality of reports of randomized controlled studies. JAMA 1998; 279:1489-1491.

23. Callaham M, Trecier J. The Relationship of Previous Training and Experience of Journal Peer Reviewers to Subsequent Review Quality. PloS Med 4(1):e40.doi:10.1371/journal.pmed.0040040.

24. Polak JF. The Role of the Manuscript Reviewer in the Peer Review Process. AJR 1995; 165:686-688.

25. Rochon PA, Bero LA, Bay A et al. Comparison of Review Articles Published in Peer-Reviewed and Throwaway Journals. JAMA 2002; 287:2853-2856.

26. Weber EJ, Katz PP, Waeckerle JF, Callaham M. Author Perception of Peer Review. Impact of Review Quality and Acceptance on Satisfaction. JAMA 2002; 287:2790-2792.

27. J Can Dent Assoc 2010; 76(2):111-116.

28. J Can Dent Assoc 2010; 76(3):177-181.

29. Jefferson T, Rudin M, Brodney FS et al. Editorial peer review for improving the quality of reports of biomedical studies. Cochrane Database of Systematic Reviews 2007, Issue 2. Art. No.:MR000016.DOI:10.1002/14651858.MR000016.pub3.

RELATED NEWS

RESOURCES