Australian NHMRC review in detail
Who assessed the evidence on homeopathy?
The evidence review on which the Information Paper was based was commissioned by the NHMRC, but conducted by Health Technology Analysts Pty Ltd (trading as Optum).
The NHMRC’s Homeopathy Working Committee, described as ‘a committee of experts’ played a crucial role, overseeing the process and giving external input on Optum’s findings. Usually such a committee would include several experts in the subject being reviewed, yet in this case the NHMRC chose not include a single person with any expertise in either homeopathy or homeopathy research. The NHMRC need to justify this highly unusual decision, particularly as the lack of relevant expertise in the team may explain why serious errors were made in how the scientific evidence was analysed.
What evidence was considered?
The evidence review produced two main documents whose findings were summarised in the Information Paper:
- Effectiveness of Homeopathy for Clinical Conditions: Evaluation of the Evidence – Overview Report. Described as an “independent systematic review of the available systematic reviews” on the effectiveness of homeopathy in treating a variety of clinical conditions in humans (300-page report plus 282 pages of appendices).
- Effectiveness of Homeopathy for Clinical Conditions: Evaluation of the Evidence – Review of Submitted Literature. This assessed additional evidence submitted from homeopathy stakeholder groups and members of the public (87 page report, plus 36-page list of submitted evidence).
Evidence‑based guidelines and government reports on the use of homeopathy were also taken into consideration.
What were the NHMRC’s overall findings?
“NHMRC concludes that the assessment of the evidence from research in humans does not show that homeopathy is effective for treating the range of health conditions considered.”1
- For some health conditions, homeopathy was found to be not more effective than placebo.
- For other health conditions, some studies reported that homeopathy was more effective than placebo, or as effective as another treatment, but those studies were not reliable.
- For the remaining health conditions it was not possible to make any conclusion about whether homeopathy was effective or not, because there was not enough evidence.
To be confident that the health benefits of homeopathy that were reported in some studies were not just due to chance or the placebo effect, they would need to be confirmed by other large, well-designed studies.”
This summary of the results is very different from the public perception that the NHMRC found homeopathy was no better than placebo for all 61 conditions.
HRI has therefore asked the NHMRC to state which conditions fall into each of these important categories so that the public’s understanding of their work can be corrected.
We have also asked them to identify the conditions for which, “studies reported that homeopathy was more effective than placebo, or as effective as another treatment”, and where the only reason they classified the evidence as unreliable was because the studies need to be repeated.
This list of conditions which the NHMRC has identified as needing further research would be most useful to the research community.
How did the NHMRC fail to identify the conditions for which the evidence does show that certain homeopathic treatments are effective?
Having examined the NHMRC’s review process, several reasons for their inaccurate findings emerge:
- For each condition, the results of all trials were analysed together as a whole, with negative trials considered to ‘cancel out’ positive ones, even though they were testing completely different treatments. This fundamental error means that the NHMRC’s conclusions about the evidence on many conditions covered in this review are completely invalid. The NHMRC need to explain why they analysed the data in this bizarre and unprecedented way, knowing that it could not possibly produce meaningful results.The reviewers asked the question,“Is homeopathy effective for condition Y?’. They appear to have worked from the premise that a positive trial showing that one homeopathic treatment is effective is somehow negated by a negative trial which shows that a completely different homeopathic treatment for that same condition is ineffective.This is a bizarre way of assessing scientific evidence. In conventional research the question asked would be,“Is treatment X effective for condition Y?”, not
“Is conventional medicine effective for condition Y?” based on combining the results of all drug trials together.
The primary clinical research question (Overview Report, 3.1, p.13) should therefore be changed from,
‘For patients with a specific clinical condition, is homeopathy an effective treatment, compared with no homeopathy/other treatments?’ to,
‘For patients with a specific clinical condition, is any homeopathic treatment effective, compared with no homeopathy/other treatments?’
- Only systematic reviews of controlled prospective studies on specific medical conditions were included. Therefore studies demonstrating effectiveness of homeopathy under ‘real world’ conditions for groups of patients with a variety of conditions were excluded e.g. service evaluations and observational studies in hospitals and GP practices.2, 3, 4
- Only trials published in English were included, risking exclusion of good quality positive studies e.g. a study on allergic rhinitis (hay fever) was dismissed on this basis (although it is actually in English).5
This approach is actively discouraged by the British Medical Journal, as it means that not all the available evidence is being taken into consideration.Bearing in mind Optum’s vast resources (according to Wikipedia their parent company UnitedHealth Group Inc posted net earnings in 2011 of $5 billion), it is very hard to see how this decision can be justified, especially when reviewing a relatively small field such as homeopathy research. Excluding relevant, good quality scientific evidence purely because it is in a language other than English is unacceptable.
- Trials with less than 150 participants were dismissed as unreliable, even if the results were statistically significant e.g. trials showing homeopathy was effective for allergic rhinitis6 and ear infections in children.7
The NHMRC give no scientific justification for setting the bar of ‘reliability’ at 150 participants.
A common problem with homeopathy studies is that, due to lack of funding, the study may be ‘underpowered’ i.e. it involves too few patients to generate ‘statistically significant’ results, meaning that you can’t rule out the possibility that the results are simply due to chance. This is why a trial with too few patients might show a positive ‘trend’ in the results, but would not be considered ‘reliable’ until it has been repeated with larger numbers.
- Trials which have been repeated by the same research team, but not yet repeated by another independent team were dismissed as unreliable e.g. trials showing effectiveness of a homeopathic medicine for allergic rhinitis.8
The NHMRC review does refer to the fact that there were some good quality studies showing that homeopathy works better than placebo or as well as an existing treatment, but they state that these need to be repeated before their findings can be considered ‘reliable’. However they do not say which conditions these were, nor which treatments were found to work in these studies. The NHMRC need to rectify this, to correct the misunderstanding that their review has found no evidence that homeopathy works for any of the 61 conditions, and to highlight the areas where more research needs to be carried out. It is ideal to have independent verification of positive trials before basing healthcare decisions on the results, but the fact that this replication stage has yet to be carried out does not mean that the trial results are in themselves unreliable. The NHMRC Information Paper should inform the public about good quality positive trials and encourage replication of these promising studies.
- Certain good quality positive trials were excluded from the review due to being the wrong type of studies, when in fact they fit the reviewers inclusion criteria e.g. a meta-analysis on childhood diarrhoea,9 and a trial showing a homeopathic treatment to be effective for vertigo.10
A study on childhood diarrhoea (Jacobs et al. 2003) was excluded on the following basis,
‘Unable to assign a level of evidence – non-systematic review. Wrong research type or wrong publication type.’ However this is not accurate. Jacobs reviewed all available Level II studies on individualised homeopathy for childhood diarrhea – RCTs which she had carried out – and conducted a meta-analysis. It is therefore a legitimate systematic review and also contains a meta-analysis. A relevant positive review on vertigo (Schneider et al. 2005) was also excluded, this time on the basis of being a Level III-3 study (comparative study without concurrent control). This is incorrect because the review contains two Level II studies (in addition to the two Level III-3 studies). The study therefore fits the NHMRC’s inclusion criteria.
- Certain positive trials were discounted as ‘unreliable’ for no justifiable reason e.g. a pilot study showing homeopathic treatment was as effective as conventional medicine for ear infections in children.7 Sinha and colleagues concluded that,“There were no significant differences between groups in the main outcome. Symptomatic improvement was quicker in the homeopathy group and there was a large difference in antibiotic requirements favouring homeopathy” yet the NHMRC reviewers concluded the exact opposite, saying“The addition of Sinha et al (2012) to the body of evidence for otitis media is consistent with the conclusion from the Overview Report that there is no reliable evidence that homeopathy is as effective as other therapies for the treatment of children with acute otitis media.” For reviewers to over-rule the original author’s conclusions is a serious issue which needs to be extremely well justified; the 3 reasons given by NHMRCs reviewers for dismissing this study, do not stand up to scientific scrutiny:
- Small sample size (N=81) This is a perfectly reasonable number of participants for a pilot study and does not negate the results of this comparative study as the lack of difference between arms was statistically significant (no significant difference between conventional treatment and homeopathic treatment (p=0.247).
- “Method of allocation concealment was not described, which may have been a source of selection bias.” This should be mentioned in a future paper, but it is a very minor point to raise as there is nothing in table 3 and 4 to suggest that participants in both groups were sufficiently dissimilar to suggest that bias did occur.
- “There was also a risk of bias in measuring the severity of disease, as the parents/guardians were asked to subjectively rate the severity of symptoms of their child.” This is an invalid criticism as the paper specifically states that, “The parents/guardian […] remained unaware of the patient’s group assigned throughout the study.”
Furthermore the reviewer makes no mention of the fact that symptom response was also measured by examination of the tympanic membrane by an ENT specialist. Colour, transparency, mobility and bulging of the eardrum were rated by the specialist at the beginning and end of the trial, and the difference calculated. This outcome measure also showed that conventional and homeopathic treatment achieved similar results.
This study is therefore suitable for inclusion in the NHMRC review as a good quality pilot study suggesting that homeopathy is as effective as conventional medicine for this condition and it should be repeated on a larger-scale trial
When one considers the combined impact of the series of choices made by the NHMRC above, it is not perhaps surprising that they reached the conclusion that there was no reliable evidence for the effectiveness of homeopathy for any of the conditions investigated.
HRI recommended that the NHMRC’s Overall Finding be amended to be clear about how they reached their conclusions i.e,
For the 61 health conditions considered, if we consider only prospective, controlled trials published in English, and discount all trials with less than 150 participants (even if they had positive statistically significant results), and if we discount positive trials that have not yet been repeated by other teams of researchers, and if we then combine all trial results for each condition, we can say that there was no reliable evidence demonstrating that homeopathy was effective.
Specific examples of how the NHMRCs approach distorted the results
The larger and more heterogeneous the set of primary studies for a given medical condition, the more the final conclusions were distorted by the NHMRC’s method of combining all trials; the smaller and more homogeneous the evidence base, the less effect this will have had on the outcome.
Two examples where the combined analysis approach lead to highly inaccurate conclusions being drawn are:
Childhood diarrhoeaNHMRC’s evidence statement(Overview report, Childhood diarrhea, p.38):
‘The one medium-sized, good-quality trial (292 participants) did not detect a difference between combined homeopathy and placebo in the treatment of children with diarrhoea.
The studies of individualised homeopathy are of insufficient quality and/or size to warrant further consideration of their findings. LOC: Low – moderate.
Based on the body of evidence evaluated in this review combined homeopathy is not more effective than placebo for the treatment of children with diarrhoea and there is no reliable evidence that individualised homeopathy is more effective than placebo for the treatment of children with diarrhoea.”
The NHMRC’s approach has lead to two conclusions:
- The entire approach known as ‘combined homeopathy’ does not work for children with diarrhoea (i.e. giving the same medicine, containing multiple ingredients, to all patients)
- There is no reliable evidence that individualised homeopathy works for children with diarrhoea (i.e. treatment by a homeopath, involving an individualised prescription tailored to the symptoms of each patient)
However, these conclusions are not supported by the evidence.
‘The one medium-sized, good-quality trial (292 participants) did not detect a difference between combined homeopathy and placebo in the treatment of children with diarrhoea.’
This refers to a trial testing a combination or ‘complex’ homeopathic medicine containing Arsenicum, Calcarea carbonica, Chamomilla, Podophyllum and Sulphur in 30c potency (Jacobs et al. 2006). The negative result tells us that this particular homeopathic medicine is no better than placebo. However it does not tell us whether other medicines with multiple ingredients (‘combined homeopathy’) work or not for this condition, so cannot justify the NHMRC’s conclusion that the entire approach is ineffective.
‘The studies of individualised homeopathy are of insufficient quality and/or size to warrant further consideration of their findings.’
This is only a valid statement because, rather than considering the existing meta-analysis of trials on individualised homeopathy for childhood diarrhoea (Jacobs et al. 2003) the reviewers have instead only assessed the three individual trials within that meta-analysis. By doing this, the reviewers can conclude that the trials are ‘unreliable’ as they have less than the N=150 participants required by the NHMRC to be classified as large enough to be reliable: N=126 (Jacobs et al, 2000), N=92 (Jacobs et al, 1994) and N=34 (Jacobs et al, 1993).
The Jacobs et al. 2003 meta-analysis was excluded on the following basis: ‘Unable to assign a level of evidence – non-systematic review. Wrong research type or publication type.’ However, Jacobs reviewed all available Level II studies on individualised homeopathy for childhood diarrhea – RCTs which she had previously carried out – and conducted a meta-analysis. This legitimate systematic review and meta-analysis meets the inclusion criteria and should have been considered by the authors.
Based on the pooled results within this meta-analysis (n = 242) the study concludes that individualised homeopathy is more effective than placebo (p = 0.008), providing a 0.66 day reduction in the duration of the condition.
The evidence therefore actually supports the following conclusions:
1. The combination medicine Arsenicum, Calcarea carbonica, Chamomilla, Podophyllum and Sulphur in 30c potency is ineffective for childhood diarrhea
2. Individualised homeopathic treatment is an effective treatment for childhood diarrhea (p = 0.008).
Allergic rhinitisNHMRC reviewers’ conclusion:
‘Based on the body of evidence evaluated in this review there is no reliable evidence that homeopathy is as effective as the other therapies for the treatment of people with allergic rhinitis.’
By contrast, if the evidence is analysed appropriately by specific treatments, we see that it supports the following very different conclusions:
- The non-individualised homeopathic medicine Galphimia glauca is effective for allergic rhinitis (Ernst 2011; Wiesenauer & Lüdtke1996)
- The non-individualised isopathic medicine Betula (made from Birch pollen) is not effective for allergic rhinitis (Aabel 2000)
- The non-individualised isopathic medicine Pollen 30c is effective for allergic rhinitis (Reilly et al.1986).
These examples highlight the need for amendment of the Information Paper prior to final publication.
Scale of the problem
As the work of the NHMRC amounted to over 700 detailed pages, we chose to focus on conditions where we expected the NHMRC to find good quality evidence that homeopathy is effective (i.e. allergic rhinitis, otitis media, diarrhoea in children and vertigo). However, this is not an exhaustive list of the conditions for which the level of evidence may have been inaccurately represented by the NHMRC’s findings.
How could an experienced review body such as the NHMRC make such errors?
As the NHMRC and Optum are organisations which specialise in reviewing scientific evidence it was extremely surprising to discover that they took the unprecedented approach of mixing the results of trials on different treatments in their analysis which have nullified their findings. HRI raised this issue in our submission to the public consultation and have yet to learn more about the reasoning behind the NHMRC’s strategy.
- NHMRC Draft Information Paper: Evidence on the effectiveness of homeopathy for treating health conditions [APRIL 2014], Summary of Evidence – Overall finding, p.10.| Full text
- Spence D, Thompson E A, Barron S J. Homeopathic treatment for chronic disease: a 6-year university-hospital outpatient observational study. J Altern Complement Med, 2005; 5: 793-798 | PubMed
- Witt C, Keil T, Selim D, et al. Outcome and costs of homeopathic and conventional treatment strategies: a comparative cohort study in patients with chronic disorders. Complement Ther Med, 2005; 13: 79-86 | PubMed
- Witt CM, Lüdtke R, Baur R, Willich SN. Homeopathic medical practice: long-term results of a cohort study with 3981 patients. BMC Public Health., 2005;5:115 | PubMed
- Wiesenauer M and Lüdtke R. A meta analysis of the homeopathic treatment of pollinosis with Galphimia glauca. Forsch. Komplementärmed. 1996; 3: 230-234. | Link
- Reilly, D. T., Taylor, M. A., et al. “Is homoeopathy a placebo response? Controlled trial of homoeopathic potency, with pollen in hayfever as model.” Lancet, 1986;2(8512): 881-886. | PubMed
- Sinha, M. N. et al. Randomized controlled pilot study to compare Homeopathy and Conventional therapy in Acute Otitis Media. Homeopat. J. Fac. Homeopat., 2012; 101: 5–12 | Pubmed
- Ernst E. Homeopathic Galphimia glauca for hay fever: a systematic review of randomised clinical trials and a critique of a published meta-analysis. FACT, 2011;16(3):200–3 | Link
- Jacobs, J., Jonas, W. B., Jiménez-Pérez, M. & Crothers, D. Homeopathy for childhood diarrhea: combined results and metaanalysis from three randomized, controlled clinical trials. Pediatr. Infect. Dis. J., 2003; 22: 229–234 | PubMed
- Schneider et al. Treatment of vertigo with a homeopathic complex remedy compared with usual treatments – a meta-analysis of clinical trials, Arzneim.-Forschung 2005; 55(1) 23-29. | PubMed