Rubbish in, rubbish out

Professor Edzard Ernst looks at the value of systematic reviews in traditional Chinese medicine.

Most GPs, I think, have learnt to appreciate systematic reviews. It is quite simply not possible to evaluate the multitude of trial data on any given question, assess contradictions within the data-set and determine what the totality of the most reliable evidence tells us.

Systematic reviews do all this for us and provide us with quick and usable answers – at least this is what we hope.

For traditional Chinese medicine (TCM), only very few systematic reviews existed until recently. During the last two or three years, this situation has been changing: a plethora of such publications (including numerous Cochrane reviews) are now rapidly emerging. A reason to celebrate? Sadly not!

After reading virtually dozens of such articles, I am more concerned than delighted. Let me explain why. Almost invariably, these reviews have the following characteristics:

• As one might expect, they rely (nearly) exclusively on primary studies published in Chinese

• They are authored by Chinese experts who sometimes recruit non-Chinese co-authors

• They arrive at (cautiously) positive conclusions.

And what is wrong with that, I hear you asking? I fear, the answer is A LOT! In most cases, neither the non-Chinese co-authors nor the peer-reviewers of the article (who usually do not read Chinese), would have had a chance to critically evaluate the primary studies. Otherwise, they would have noticed that all or the vast majority of them are of extremely low quality.

We should remember that several research groups have found that 100% of all Chinese TCM studies are positive. A review of such data is therefore likely to be a foregone conclusion.

Let's be clear, systematic reviews are definitely a useful tool. But, crucially, they have to include a strong element of critical evaluation of the primary studies. If, almost by definition, the primary studies can never be negative, and if they are not even accessible for independent evaluation, something is badly amiss.

In such a situation the old adage applies: 'rubbish in, rubbish out'. The danger is that, once 'rubbish' has been published in the form of a systematic review in a trusted journal by an international team of authors from respected institutions, it can easily be mistaken for reliable evidence. This, I believe, would be a mistake, and a dangerous one at that!

What does all this mean for busy GPs who want to inform themselves quickly and reliably? I fear it suggests that we cannot draw any meaningful conclusions from systematic reviews of TCM. Yet more often than not, we do find conclusions such as…x 'may improve quality of life' [1] or y "may be effective…" [2], to quote just two recent examples. To me, this seems misleading, to say the least.

Professor Edzard Ernst is professor of complementary medicine at the Peninsula Medical School, University of Exeter

Click here for more from Edzard Ernst Professor Edzard Ernst