Anecdotal evidence

When it comes to patient safety, anecdotes are often all we have to go on, says Professor Edzard Ernst

I recently came across a blog where a 2009 article of mine was severely and, I think, not very fairly criticized.

In it, a chiropractor wrote: 'There is "not a jot" of epidemiological evidence showing a causal relationship between chiropractic and stroke, in fact I would go as far to say this claim is "bogus". Ernst is relying on anecdotal evidence to support his claims. The exact same type of evidence he criticises the chiropractic profession for using to support the efficacy of chiropractic for treating children. When Ernst uses anecdotes he attaches great weight to them. When chiropractors use them they are worthless as far as Ernst and the sceptics are concerned.'

As I hear this argument regularly, and as it is of some importance, it seems worth addressing in more detail. It would be entirely wrong to assume that anecdotal evidence is important only when I use it, while, from others, I demand evidence from RCTs. The point is quite a different one.

When we talk about effectiveness or efficacy, anecdotes can be interesting but prove very little. We all know how many factors other than the therapy in question can affect the clinical outcome. Therefore we need more than anecdotes to convince us that a treatment works. However, when it comes to patients' safety, anecdotes can be of great value. The precautionary principle demands that we ‘err on the safe side'. This means taking anecdotes seriously in relation to safety issues while taking them with a pinch of salt when evaluating efficacy or effectiveness.

In fact, it would be irresponsible to discard anecdotes about adverse effects or other safety issues, particularly when they are reported repeatedly. Moreover, anecdotes are often all we have to go on.

In post-marketing surveillance, several anecdotes eventually generate a 'signal' which can be sufficient, for instance, to take a drug off the market. As anecdotes usually originate from entire populations, the sample size which they are based on is normally huge (albeit undefined). RCTs, on the other hand, usually have far too small sample sizes to reliably inform us about adverse effects. If this or that adverse event did not show up in a tightly controlled study with 100 patients, it may still happen quite frequently in the population at large.

So, relying on anecdotes for information about safety is not a question of double standards, it is a question of context and of common sense.