• en

The Lancet paper by Shang et al.

In August 2005, the Lancet published a study1 which was reported as having compared 110 similar trials on homeopathy and conventional medicine, and reached the conclusion that homeopathy is no better than placebo.

An accompanying editorial by The Lancet editor entitled ‘The end of homoeopathy’2 lead to widespread media attention.

However, the conclusions of this study were in fact based on only 8 of the 110 trials, none of which involved usual homeopathic care. Furthermore, if you switch just one of the 8 trials they picked from the 110 for a different one, the results are reversed, showing that homeopathy works beyond placebo.3

This demonstrates that the findings of this paper are completely unreliable.

The sequence of events surrounding this issue of The Lancet, the political controversy which followed and key flaws of the study itself, are summarised in ‘Homeopathy and the Lancet‘.3

A paper with continued global impact

The Shang et al. paper has become notorious because, despite such fundamental flaws, it continues to be mis-quoted to this day as definitive ‘proof’ that homeopathy doesn’t work e.g.:

“Over 150 trials have failed to show it works”
Dr Ellie Cannon – resident GP for the Mail on Sunday. ‘Health Comment’, Mail on Sunday, 14 September 2014

“We’ve had over 100 trials of homeopathy now. For any medical treatment, after 100 trials have failed to give good overall evidence of benefit, this is the point at which a sensible person, with no vested interests in that one particular treatment, would loudly and clearly state: “No more money should be spent researching this blind alley”.
Ben Goldacre – British physician, academic and science writer.Pharma Chameleon’, New Europe, 17 April 2011 – Issue 93.

Results of the Shang paper

Shang et al.’s findings as reported in the abstract of the paper are:

110 homoeopathy trials and 110 matched conventional-medicine trials were analysed. The median study size was 65 participants (range ten to 1573). 21 homoeopathy trials (19%) and nine (8%) conventional-medicine trials were of higher quality. In both groups, smaller trials and those of lower quality showed more beneficial treatment effects than larger and higher-quality trials. When the analysis was restricted to large trials of higher quality, the odds ratio was 0·88 (95% CI 0·65–1·19) for homoeopathy (eight trials) and 0·58 (0·39–0·85) for conventional medicine (six trials).

Note: These ‘odds ratios’ mean that, for these particular 8 trials of homeopathy, if you analyse them together, the results suggest that the effects seen are due to placebo rather than a real clinical effect.

How reliable are these results?

The role of the Shang paper in the homeopathy debate cannot be overstated, so the issue of quality and reliability of the paper is paramount.

Unfortunately, the poor scientific quality of the paper is obvious, having been described by experts in the field of homeopathy research3,4,5,6,7 as well as independent researcher Prof Hahn, who highlights the main problem:

To conclude that homeopathy lacks clinical effect, more than 90% of the available clinical trials had to be disregarded.8

 

MoreLess

Understandably it has often been scientists, researchers and clinicians who specialise in homeopathy research and/or homeopathy who have been motivated to understand the paper in detail and write articles on this subject. Whilst this does not mean that their arguments are wrong, critics suggest that bias influences these opinions.

Prof R Hahn from Sweden has ‘never practiced, received, or studied homeopathy, but has worked in clinical medicine and performed traditional medical research in anesthesiology and surgery for the past 30 years’. His paper details why the results of the Shang paper, which includes trials on many different diseases, would not usually be accepted in scientific circles:

The ultimate argument against homeopathy is the ‘funnel plot’ published by Aijing Shang’s research group in 2005. However, the funnel plot is flawed when applied to a mixture of diseases.8 

Relevance of Shang’s 2005 paper in 2015

Reliability of the analysis is not the only problem with the Shang paper. As we take a fresh look at the evidence in 2015, we also need to consider how well this study reflects the entirety of today’s evidence base.

A rigorous study by Mathie et al. published in 2014, found that homeopathic medicines, when prescribed during individualised treatment, are 1.5- to 2.0-times more likely to have a beneficial effect than placebo.13

This study includes 151 placebo-controlled randomised trials – 41 more than Shang’s team identified in 2005, but which would have met their inclusion criteria if they had been available at the time.

This demonstrates the extent to which the 10 year-old Shang et al. paper, which now covers only 73% of the eligible trials, has been superseded.

Read HRI’s brief summary of Mathie et al.’s study or listen to Robert Mathie presenting his findings at the HRI Rome 2015 conference.

Shang et al. key facts

  • Although 110 homoeopathy trials and 110 matched conventional medicine trials were included in the study, the results were based on only 8 homeopathy trials and 6 conventional medicine trials described as ‘larger higher quality trials’.
  • If all 21 homeopathy trials they identified as higher quality had been analysed, they would have got the opposite result – that homeopathy has an effect beyond placebo.10 No justification has been given for discounting the other 13 high quality trials just because they were smaller.
  • The study fails a ‘sensitivity analysis’ i.e. if you switch just one of the 8 homeopathy trials they chose from the 110, you get the opposite result i.e. that homeopathy works better than placebo.4
  • None of the 8 homeopathy trials they used involved individualised homeopathic treatment – the form of homeopathy considered to be ‘usual care’ – instead they were testing non-individualised homeopathic medicines given without a consultation. The conclusions of this study therefore have no relevance to the evidence base ‘for or against’ usual homeopathic treatment.
  • The 8 trials used were testing a range of different homeopathic medicines being used for different conditions. One would assume that, as with conventional medicine, some homeopathic treatments work and others don’t, so it is no surprise that some of these studies were negative whilst others were positive. Mixing the results together to decide whether they show ‘homeopathy’ works makes no scientific sense.
  • In the original paper Shang et al. do not state which 8 homeopathy trials and which 6 conventional trials they used. Such lack of reporting transparency and precision is typically not tolerated in studies of conventional medicine.6 Following an outcry, the authors eventually provided this information.
  • The decade-old Shang paper is out of date and has been superseded. Since 2005, 41 more placebo-controlled randomised trials have been published which would have been suitable for inclusion in Shang’s review. These have been included in the most recent meta-analysis by Mathie et al. published in 2014 which reached a positive conclusion.13

ReferencesLess

  1. Shang A, Huwiler-Muntener K, Nartey L, et al. Are the clinical effects of homeopathy placebo effects? Comparative study of placebo-controlled trials of homeopathy and allopathy. Lancet 2005; 366: 726–732 | PubMed
  2. Editorial. The end of homeopathy. Lancet., 2005;366:690 | Full text
  3. Fisher P. Homeopathy and the Lancet. Evid Based Complement Alternat Med., 2006; 3(1): 145–147 | Full text
  4. Lüdtke R & Rutten ALB. The conclusions on the effectiveness of homeopathy highly depend on the set of analyzed trials. J. Clin. Epidemiol., 2008; 61: 1197–1204 | PubMed
  5. Helmut K, et al. Failure to Exclude False Negative Bias: A Fundamental Flaw in the Trial of Shang et al. JACM, 2005; 11(5): 783 | PubMed
  6. Bell IR. All Evidence Is Equal, but Some Evidence Is More Equal than Others: Can Logic Prevail over Emotion in the Homeopathy Debate? JACM, 2005; 11(5): 763-769 | PubMed
  7. Frass M, et al. Bias in the Trial and Reporting of Trials of Homeopathy: A Fundamental Breakdown in Peer Review and Standards? JACM, 2005; 11(5): 780-782 | PubMed
  8. Hahn RG. Homeopathy: Meta-Analyses of Pooled Clinical Data. Forsch Komplementmed., 2013;20:000–000 Published online: October 17, 2013 DOI: 10.1159/000355916 | PubMed
  9. Mathie RT, et al. Randomised controlled trials of homeopathy in humans: characterising the research journal literature for systematic review. Homeopat. J. Fac. Homeopat., 2013; 102:3–24 | PubMed
  10. Lewith G, Professor of Health Research at Southampton University, Letter to the Editor, Positive Health, December 2008 | Full text
  11. Goldacre, Ben.‘Pharma Chameleon’, New Europe, 17 April 2011 – Issue 931.
  12. Faculty of Homeopathy website: http://www.facultyofhomeopathy.org/research/ accessed 1 August 2014
  13. Mathie RT, et al. Randomised placebo-controlled trials of individualised homeopathic treatment: systematic review and meta-analysis. Systematic Reviews, 2014; 3: 142
 | Full text  

S0140673605X61475_cov150h

S0140673605X61475_cov150h