[Este artículo está disponible en español. La traducción al español apareció por primera vez en la revista Pensar.]
We are programmed to respond to testimonials. Blame evolution. For most of human history, we had only two ways to learn about the world: our own observations, and what other people told us about their experiences. Gathering information from others gave us a survival advantage. Unfortunately, those testimonials can be unreliable. We now have something far more reliable: science. But science doesn’t come naturally. We have to learn to trust studies over stories, and not everyone learns.
Complementary and alternative medicine (CAM) relies heavily on testimonials. They think that’s all they need. If people say a treatment helped them, that’s all the “evidence” they need or want. But testimonials are not evidence. Ten testimonials are no better than one; a thousand testimonials are no better than ten. Testimonials are useful, but only because they can indicate that an untested treatment is promising and worthy of formal scientific study. CAM does not value science and often doesn’t understand it. But CAM is increasingly aware that others do value science, so more and more they are trying to provide scientific evidence. Their efforts are often misguided and laughable, and often end in failure.
When I visit a CAM website, I look for a “Scientific evidence” tab. Sometimes they don’t even pretend to have any supporting science. When they do, the studies they cite are all too often poor science, junk science, or irrelevant. Sometimes they even disprove the claims they are alleged to support. What I find most disturbing is when a tab for “evidence” leads only to a series of testimonials. They know how convincing testimonials can be, and they don’t understand why testimonials don’t count as evidence.
How do we know a testimonial isn’t bogus? Does the person even exist? If it is a real testimonial, how do we know the person wasn’t misled by one of the many reasons people can erroneously believe that a treatment has helped them when it hasn’t? The late Barry Beyerstein listed seven reasons in his classic article “Why Bogus Therapies Often Seem to Work.”
- The disease may have run its natural course.
- Many diseases are cyclical.
- The placebo effect may be responsible.
- People who hedge their bets credit the wrong thing.
- The original diagnosis or prognosis may have been incorrect.
- Temporary mood improvement can be confused with cure.
- Psychological needs can distort what people perceive and do.
There are more than just these seven. Another source of error is our notoriously fallible memory. Fortunately, these pitfalls can be avoided by validating the treatment with scientific studies that control for confounding factors such as placebo responses, compliance effects, and judgmental errors.
Kailo is a pain relief patch based on laughable energy medicine quackery. Its advertising brags about the results of “An IRB-approved study.” As if “IRB-approved” meant something special. No, all studies on humans require approval by an Institutional Review Board to protect the rights of research subjects. Not only that, but the results they describe are only preliminary results and anyway the study is meaningless because it lacked a control group. Participants simply used the Kailo patch and a large percentage of them reported that their pain improved. No surprise there. Pain is subjective and suggestion is powerful; pretty much anything you offer will elicit reports of improvement.
That wasn’t a legitimate study; it was more like a group of testimonials. CAM loves testimonials because they convince customers and are easy to obtain. Every snake oil peddler and charlatan can provide testimonials for their wares.
CAM is all about testimonials. I don’t think there’s any way we could convince them that testimonials are not evidence. If they really “got” science and the need for real evidence, they could no longer believe in the effectiveness of CAM. Thank goodness some of us know better.
This article was originally published as a SkepDoc’s Corner column on the CSI website.
