The Lancet paper by Shang et al.
In August 2005, the Lancet published a study1 which was reported as having compared 110 similar trials on homeopathy and conventional medicine, and reached the conclusion that homeopathy is no better than placebo.
An accompanying editorial by The Lancet editor entitled ‘The end of homoeopathy’2 lead to widespread media attention.
However, the conclusions of this study were in fact based on only 8 of the 110 trials, none of which involved usual homeopathic care. Furthermore, if you switch just one of the 8 trials they picked from the 110 for a different one, the results are reversed, showing that homeopathy works beyond placebo.3
This demonstrates that the findings of this paper are completely unreliable.
A paper with continued global impact
The Shang et al. paper has become notorious because, despite such fundamental flaws, it continues to be mis-quoted to this day as definitive ‘proof’ that homeopathy doesn’t work e.g.:
“Over 150 trials have failed to show it works”
Dr Ellie Cannon – resident GP for the Mail on Sunday. ‘Health Comment’, Mail on Sunday, 14 September 2014
“We’ve had over 100 trials of homeopathy now. For any medical treatment, after 100 trials have failed to give good overall evidence of benefit, this is the point at which a sensible person, with no vested interests in that one particular treatment, would loudly and clearly state: “No more money should be spent researching this blind alley”.
Ben Goldacre – British physician, academic and science writer.‘Pharma Chameleon’, New Europe, 17 April 2011 – Issue 93.
Results of the Shang paper
Shang et al.’s findings as reported in the abstract of the paper are:
110 homoeopathy trials and 110 matched conventional-medicine trials were analysed. The median study size was 65 participants (range ten to 1573). 21 homoeopathy trials (19%) and nine (8%) conventional-medicine trials were of higher quality. In both groups, smaller trials and those of lower quality showed more beneficial treatment effects than larger and higher-quality trials. When the analysis was restricted to large trials of higher quality, the odds ratio was 0·88 (95% CI 0·65–1·19) for homoeopathy (eight trials) and 0·58 (0·39–0·85) for conventional medicine (six trials).
Note: These ‘odds ratios’ mean that, for these particular 8 trials of homeopathy, if you analyse them together, the results suggest that the effects seen are due to placebo rather than a real clinical effect.
How reliable are these results?
The role of the Shang paper in the homeopathy debate cannot be overstated, so the issue of quality and reliability of the paper is paramount.
Unfortunately, the poor scientific quality of the paper is obvious, having been described by experts in the field of homeopathy research3,4,5,6,7 as well as independent researcher Prof Hahn, who highlights the main problem:
To conclude that homeopathy lacks clinical effect, more than 90% of the available clinical trials had to be disregarded.8
Prof R Hahn from Sweden has ‘never practiced, received, or studied homeopathy, but has worked in clinical medicine and performed traditional medical research in anesthesiology and surgery for the past 30 years’. His paper details why the results of the Shang paper, which includes trials on many different diseases, would not usually be accepted in scientific circles:
The ultimate argument against homeopathy is the ‘funnel plot’ published by Aijing Shang’s research group in 2005. However, the funnel plot is flawed when applied to a mixture of diseases.8
Relevance of Shang’s 2005 paper in 2015
Reliability of the analysis is not the only problem with the Shang paper. As we take a fresh look at the evidence in 2015, we also need to consider how well this study reflects the entirety of today’s evidence base.
A rigorous study by Mathie et al. published in 2014, found that homeopathic medicines, when prescribed during individualised treatment, are 1.5- to 2.0-times more likely to have a beneficial effect than placebo.13
This study includes 151 placebo-controlled randomised trials – 41 more than Shang’s team identified in 2005, but which would have met their inclusion criteria if they had been available at the time.
This demonstrates the extent to which the 10 year-old Shang et al. paper, which now covers only 73% of the eligible trials, has been superseded.
Shang et al. key facts
- Although 110 homoeopathy trials and 110 matched conventional medicine trials were included in the study, the results were based on only 8 homeopathy trials and 6 conventional medicine trials described as ‘larger higher quality trials’.
- If all 21 homeopathy trials they identified as higher quality had been analysed, they would have got the opposite result – that homeopathy has an effect beyond placebo.10 No justification has been given for discounting the other 13 high quality trials just because they were smaller.
- The study fails a ‘sensitivity analysis’ i.e. if you switch just one of the 8 homeopathy trials they chose from the 110, you get the opposite result i.e. that homeopathy works better than placebo.4
- None of the 8 homeopathy trials they used involved individualised homeopathic treatment – the form of homeopathy considered to be ‘usual care’ – instead they were testing non-individualised homeopathic medicines given without a consultation. The conclusions of this study therefore have no relevance to the evidence base ‘for or against’ usual homeopathic treatment.
- The 8 trials used were testing a range of different homeopathic medicines being used for different conditions. One would assume that, as with conventional medicine, some homeopathic treatments work and others don’t, so it is no surprise that some of these studies were negative whilst others were positive. Mixing the results together to decide whether they show ‘homeopathy’ works makes no scientific sense.
- In the original paper Shang et al. do not state which 8 homeopathy trials and which 6 conventional trials they used. Such lack of reporting transparency and precision is typically not tolerated in studies of conventional medicine.6 Following an outcry, the authors eventually provided this information.
- The decade-old Shang paper is out of date and has been superseded. Since 2005, 41 more placebo-controlled randomised trials have been published which would have been suitable for inclusion in Shang’s review. These have been included in the most recent meta-analysis by Mathie et al. published in 2014 which reached a positive conclusion.13