How Trustworthy is the Peer Review System in Medicine?

Share this Article

Prestigious medical journals carry much prestige, commanding devout respect from the medical community, government, media and public. This prestige hinges on blind trust. Modern-day physicians religiously trust clinical trials that are peer-reviewed, doubly blinded and placebo-controlled. With a little bit of media spin, the studies’ findings become the buzz in town. 

However, many know the reality that money talks, that “gold standard” randomised controlled trials (RCTs) are ginormous studies requiring significant outlays that increasingly only private multinational pharmaceutical corporations can afford. The results of such ‘well-conducted’ studies are often predictable, considering the rules of engagement are now in the public domain. This is thanks to discovery via subpoenaed industry documents in criminal trials, resulting in over USD 114 billion in fines since 2000 [1]. 

While there are many authentic and genuine articles and studies, it can be argued that a substantial number of medical and scientific publications are fraught with errors, biases, and conflicts of interest [2]. The worst cases are products of outright manipulation & fraud. 

Who would prevent such situations from happening in science, and who would hold the fraudsters accountable? Sadly, it would be naïve to assume it is the peer review process. 

The 1960s and 1970s saw the birth of Evidence-based Medicine (EBM), in which treatment approaches were required to be science-based and supported by high-quality evidence. EBM incorporated the modern practice of peer review, where manuscripts submitted to medical publications would be scrutinised by peers, other researchers and clinicians with expertise on the topic. This undoubtedly was and remains preferable to not having a peer review, where there is no critique of the author’s methodology, results, conclusions and writing style.

This evidence base is considered a cauldron of research, with the highest respect given to randomised double-blind and placebo-controlled trials and the least attention given to observational studies regardless of how well they are conducted. RCTs have immense persuasive power, even if there is clear bedside evidence to the contrary. The medical community has unshakeable faith in RCTs, and anyone who questions their validity stands to be ridiculed. 

Yet just one example from the COVID-19 pandemic indicates this trust can be misplaced. A large multicentre (691 hospitals) worldwide observational study of ~96,000 COVID-19 patients, of whom ~15,000 were treated with hydroxychloroquine, was published in The Lancet. It produced supposedly negative results for this old drug. However, after a letter from scientists and clinicians pointed out that the data appeared to be made up, i.e. fraudulent, the ‘peer-reviewed’ study was retracted [3]. 

In medical research, data fraud is not always apparent. When an independent group of researchers obtained access to the pharmaceutical company GSK’s hidden raw data of ‘Study 329’ on an RCT for paroxetine for depressed adolescents, it was able to publish in the BMJ in 2015 findings of no efficacy and significant harms with increased suicidality. This contradicted the peer-reviewed study, ostensibly of the same data, published 14 years earlier in the Journal of the American Academy of Child & Adolescent Psychiatry that had concluded the drug was “generally well tolerated and effective” [4]. 

The birth of EBM seemed to be a ‘eureka moment’ for profit-centric industries and businesses that perceived it as a sure-shot long-term cash cow. This was a historical turning point where industry started to ‘indulge’ in science. 

Richard Smith and Stephen Lock, both former chief editors of the BMJ, have been battling these issues since the 1980s, but unfortunately, with little success. Smith believes that unless there is evidence to the contrary, we must start by assuming that all research is conducted and reported dishonestly and expect it to be unreliable. 

Large-scale RCTs require significant investments of time, resources, and particularly money. Most are funded by big pharmaceutical corporations seeking investment returns for shareholders. These are the ethics of business. The issue arises when business principles clash with scientific and medical ethics, and the profit motive wolf is dressed in the sheep’s clothing of a peer-reviewed paper in even the most prestigious academic medical journal. 

How is peer review undertaken today, and has the process been derailed since it began (with good intent) in its modern form in the 1970s? 

Peer review is a process undertaken by scientific and medical journals to assess the quality of papers submitted by researchers or scientists for publication in scientific journals and sometimes books. It endeavours to ensure the quality and integrity of the material provided. 

Much medical research is conducted in universities and teaching hospitals and funded by either government bodies such as the National Institutes of Health (NIH) in the USA or The Wellcome Trust in the UK, big multinational pharmaceutical companies or various NGOs. In Australia, the National Health and Medical Research Council (NHMRC) coordinates a large proportion of science funding grants and stresses the importance of peer review [5]. Funds are allocated as grants to researchers to undertake a study or project in a particular area of interest. The funder directs how the money is to be spent.

A researcher’s income often relies solely on these grants, thereby creating the potential for biases and conflicts of interest. Although inimical to the scientific method, there can be pressure for predetermined outcomes of studies, with the pressing question being, ‘How do we get the science there?’ Once a manuscript is completed, it may be submitted to the top-tier medical journals with persuasive power (The New England Journal of Medicine, The Lancet, etc). Papers published in “prestigious” journals have more influence on government health departments, which then create policies based on the evidence put forward. 

To peer review the manuscript, journal editors enlist the voluntary help of researchers and scientists with expertise in the field or topic. The appointed reviewer may be someone unknown, someone recommended by the authors, or colleagues of editors with expertise in the topic. The identity of the reviewers is usually not disclosed to the authors to avoid influencing the outcome, and a time frame is set for evaluating the paper as quickly as possible to expedite publication. This is usually a few weeks, but it can take much longer in practice. 

Invariably, reviewers are not reimbursed for their efforts, although they may be offered book discount vouchers. Many reviewers have full-time work commitments, which place time pressures on them. Combined with editors who may have many manuscripts to oversee if they are paid or manage their own clinical/academic work commitments if their role is voluntary, all this limits the capacity of reviewers and editors to thoroughly review the presented data and screen for fraud and bias. 

With the advent of online publishing, it is now universal to publish an electronic copy well before a hard copy in a paper journal. This allows information to be presented to the medical community well ahead of hard copies that take several months to print. 

Generally, the more prestigious established journals rely on income from the industry sponsors of clinical trials through advertising and to purchase copies of published papers for distribution by sales representatives to clinicians at conferences and continuing medical education seminars. This introduces apparent conflicts of interest for the journals to publish. These journals increasingly accept money from authors, sponsors, or institutions for ‘open access’ publishing. Less prestigious ‘open access journals’ rely primarily on income from authors submitting manuscripts. This also introduces a conflict of interest not to reject the manuscript. A range of journals that allegedly stray into rapid low-quality peer review to make money are termed ‘predatory journals’ and indicated on lists of such journals. 

Raw data is considered gold and supposedly a non-negotiable entity in peer-reviewed studies. But in practice, this is something of a myth. Usually, any data manipulation or hidden adverse findings would’ve been buried when drafting a manuscript. Further, only small-scale studies can present all the raw data to a journal for review. This is because the amount of raw data contained in individual subject ‘case report forms’ (CRFs) and even in the clinical trial results summaries termed ‘clinical study reports’ (CSRs) is too voluminous. A submitted manuscript nearly always only has an abbreviated version of the full raw data. This is outlined in a landmark paper in the BMJ titled “Restoring invisible and abandoned trials: a call for people to publish the findings” [6]. 

Even with the abbreviated/condensed results in a manuscript, reviewing a study’s data involves tremendous input of time, essentially replicating an outline of the whole study, which is cumbersome and beyond most reviewers’ capacity. The process of review involves critically reading the submitted work, which usually does not include the calculations or raw data used to generate the results. This critical data is generally withheld for reasons of medical confidentiality, volume, or deemed as proprietary and private property. 

Authors of sponsored studies typically sign non-disclosure agreements with the pharmaceutical company that owns the data. Often, reviewers only get to view summaries of the raw data and are unable to follow the calculations from raw data to results and conclusions. Thus, there is an element of blind trust in the system because full raw data is seldom in the manuscript. 

At its best, the peer review system works on trust that is honoured. At its worst, the system devolves into a cursory rubber stamp process. Often, the system is reflective of the biases, theories, paradigms and politics of the researchers, reviewers and editors involved – journals can fall into camps on controversial issues in Medicine and wage ideological combat via published papers in a system described as ‘pal review’ whereby you know a particular journal will reject your manuscript on ideological grounds, but another one might easily accept it. 

As chief editor of The Lancet, Sir Richard Horton wrote in the Medical Journal of Australia in 2000: ‘The mistake, of course, is to have thought that peer review was any more than a crude means to discover the acceptability—not the validity—of a new finding’ [7].Peer review struggles to detect wilful fraud and other less conscious biases. Without full data transparency, knowing which papers encompass fraud and bias and which are honest science is impossible. 

In his 2006 article, “Peer Review; a Flawed Process at the Heart of Science and Journals”, former BMJ chief editor Smith noted that “Peer review of scientific publications sometimes picks up fraud by chance, but generally it is not a reliable method of picking up fraud as the process works on trust”. Trust that the reviewers put in the authors and trust that the editors and reviewers have in each other’s critical scientific reasoning. It’s human nature not to mistrust one’s colleagues. Smith describes a small experiment done at the BMJ to assess the quality of their peer review process, where the team undertook studies, purposely inserting errors in them, hoping that their chosen reviewers would pick them up. He summarised- ‘While most reviewers picked up a quarter of the fraud, some picked up none, but interestingly, none of them spotted all of the errors’ [8]. 

Researching a topic begins with a hypothesis, a starting point based on existing knowledge. The hypothesis is open to being challenged, to either have its validity refuted or acknowledged. This has been the case since the inception of the scientific method. On the other hand, belief is a conviction that one has about a matter, and it holds little reason for a challenge as it is based on anecdotes, assumptions, guesses, personal opinions, biases and experiences. You either believe it or not. Critical scientific reasoning has to contend with beliefs and assumptions, often implicit, even among the best of scientists. 

In his 2011 BMJ article, “A Woeful Tale of the Uselessness of peer review”, Smith further affirms the issue: ‘The evidence, as opposed to the opinion, on pre-publication peer review shows that its effectiveness has not been demonstrated and that it is slow, expensive, largely a lottery, poor at spotting error, biased, anti-innovatory, prone to abuse, and unable to detect fraud. The global cost of peer review is $1.9 billion, and it’s a faith-based rather than evidence-based process, which is hugely ironic when it’s at the heart of science.’ He concludes by calling for the criminalisation of research fraud [9]. 

From his long experience as chief editor of the BMJ, Smith suggested some solutions in his paper “Medical Journals are an Extension of the Marketing Arm of Pharmaceutical Companies” [10]. They include greater public funding of RCTs, particularly of large head-to-head trials, publishing raw data on policed official websites with journals recruiting independent experts and biostatisticians to analyse the results, and the journals then publish these analyses. In essence, he proposes that the journals should shift from publishing research papers to publishing critiques of research methodology, data and results. 

Could this be the litmus test a paper goes through when the world at large can see a real-world analysis of RCT methodology and data? As optimistic as it sounds, it would require a complete revolution in scientific and medical publishing of the order of the Copernican Revolution!

Varghese Zachariah is an Integrated GP in Melbourne, Australia & Peter Parry is a Psychiatrist with academic status as an Associate Professor at the University of Queensland, Australia. 

References: 

1. Violation tracker industry summary page: Pharmaceuticals: https://violationtracker.goodjobsfirst.org/summary?major_industry_sum=pharmaceuticals

2. Ioannidis JPA. Why most published research findings are false. PLoS Medicine, 30 August 2005.  https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124&xid=17259,15700019,15700186,15700190,15700248

3. Heathers J. The Lancet has made one of the biggest retractions in modern history. How could this happen? The Guardian, 5 June 2020.  https://www.theguardian.com/commentisfree/2020/jun/05/lancet-had-to-do-one-of-the-biggest-retractions-in-modern-history-how-could-this-happen

4. Godlee F. Editor’s Choice: Study 329. BMJ, 17 September 2015.  https://www.bmj.com/content/351/bmj.h4973

5. NHMRC. Peer review. https://www.nhmrc.gov.au/funding/peer-review

6. Doshi P, Dickersin K, Healy D, Vedula SS, Jefferson T. Restoring invisible and abandoned trials: a call for people to publish the findings. BMJ, 13 June 2013.  https://www.bmj.com/content/346/bmj.f2865

7. Horton R. Genetically modified food: consternation, confusion, and crack-up,  The Medical Journal of Australia, 21 February 2000. https://www.mja.com.au/journal/2000/172/4/genetically-modified-food-consternation-confusion-and-crack

8. Smith R. Peer review: a flawed process at the heart of science and journals. Journal of the Royal Society of Medicine, April 2006.  https://journals.sagepub.com/doi/10.1177/014107680609900414

9. Smith R. A woeful tale of the uselessness of peer review. BMJ, 11 November 2011.  https://blogs.bmj.com/bmj/2011/11/11/richard-smith-a-woeful-tale-of-the-uselessness-of-peer-review/

10. Smith R. Medical journals are an extension of the marketing arm of pharmaceutical companies. PLoS Medicine, 17 May 2005.  https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020138

Share this article

Got something to add? Join the discussion and comment below.

Authors

  • Dr Varghese Zachariah

    Dr.Varghese Zachariah is an Australian-trained GP from Victoria. With a total of 16 years in General Practice, he now practices as a Functional & Integrative Medicine practitioner.

    View all posts
  • Dr Peter Parry

    Dr Peter Parry is a psychiatrist with academic status as an Associate Professor at the University of Queensland, Australia.

    View all posts
Follow Us

Join our Newsletter