Pitfalls in Research: Why Studies Are More Often Wrong than Right

Here is a course guide to episode 9, “Pitfalls of Research”, of my YouTube lecture series on science-based medicine.


Alternative medicine relies heavily on anecdotal evidence and personal experience, but it knows it won’t convince mainstream science unless it can provide scientific validation. It is quick to crow about positive results from scientific studies, often from studies in laboratory mice or Petri dishes, or from human studies that are poorly designed and may not even use a control group. Science-based medicine relies on evidence from clinical trials, especially those that are randomized, double-blind, and placebo-controlled. But that evidence can be misleading. The results of even the best studies are more likely to be wrong than right. Positive results from preliminary trials are all too often overturned by negative results from subsequent larger, better trials. Individual studies are often wrong, but that doesn’t mean we should ignore them. Science is a collaborative, self-correcting enterprise that can be relied on to come ever closer to reality over the course of time. A good rule of thumb is never to believe a single study, but to wait for replication, confirmation, and a consensus of all the available data. And to consider prior probability.

In my free YouTube video lecture course on science-based medicine, I talk about “Pitfalls in Research” in lecture 9. I have been asked to re-cycle the information in the course guide [PDF] for that lecture in the form of an SBMpost because it provides valuable guidance for evaluating scientific studies.

Ioannidis showed that most research findings are false, especially in the case of:

  • Small studies
  • Small effect size
  • Multiple endpoints
  • Financial interests and bias
  • Hot topic with more teams in competition
  • Research on improbable areas of CAM
Things to Keep in Mind:Because:
Media reports can’t be trustedReporters often don’t understand the science and merely regurgitate press releases
Learn what can go wrongThe same flaws keep repeating
Never believe one studyAll data pro and con must be considered
Prior probability mattersBasic science can’t be ignored

Bausell’s quick test

In Snake Oil Science, Bausell provided this quick 4-point checklist:

  1. Is the study randomized with a credible control group?
  2. Are there at least 50 subjects per group?
  3. Is the dropout rate 25% or less?
  4. Was it published in a high-quality, prestigious, peer-reviewed journal?
    (You can look up the journal’s impact factor)

If the answers to these questions are all Yes, that doesn’t guarantee that the study’s conclusions are credible, but if some of the answers are No, it makes the results more questionable.

Other questions to ask

Was the study done on people or fruit flies?Studies in animals and in vitro may not apply to humans.
Who are the subjects?Are they representative? Could they be biased?
Who funded the study?Could be a source of bias.
Could the authors have been biased?Look for conflicts of interest.
Was randomization adequate?Were the groups really comparable?
Was blinding adequate?Was there an exit poll to see if they could guess whether they were in the test group or the control group?
Were there multiple endpoints? (This, and the next point, are known as “p-hacking“)If so, were appropriate statistical corrections done?
Was there inappropriate data mining?Sometimes the study is negative but the data are tortured until they seem to confess something positive.
Where was the study done?Studies done in China are suspect because they never report negative results.
Were the results clinically meaningful?Statistical significance ≠ clinical significance.
Did the results merit the conclusions?Sometimes they don’t.
Were they studying something real?Tooth Fairy science tries to study imaginary things like Tooth Fairies, acupoints, and human energy fields.
Was it a “pragmatic” real-world study?Pragmatic studies are not meant to establish efficacy. An unproven treatment with strong placebo effects may appear to outperform a proven treatment.

Other things that might go wrong

  • Errors of math
  • Using the wrong statistical test for the kind of data collected
  • Contaminants
  • Poor compliance
  • Conscious or unconscious manipulation of data by technician
  • Procedures not carried out properly
  • Deliberate fraud

Important definitions to understand

Statistically significantSignificant at p 0.5 ≠ Truth. It just measures the chance a repeat study will get positive results if the null hypothesis is true (no difference between groups).
Clinically significantChanges disease outcome
Relative riskReports percentages
Absolute riskReports actual numbers of people
Number Needed to Treat (NNT)Number Needed to Treat for one person to benefit
Number Needed to Harm (NNH)Number Needed to Treat for one person to be harmed.
Confidence intervalsWe can be confident the true value is somewhere in this range.
Correlation ≠ causationCorrelation = a statistical relationship between 2 variables. Doesn’t mean one causes the other.
Sensitivity of a testPercentage of people with the disease who test positive.
Specificity of a testPercentage of people who don’t have the disease who test negative.
Base rateThe prevalence in the population. The rarer the condition, the more false positive test results.
Positive Predictive Value (PPV)The likelihood that a positive test is true: that you actually have the disease.
Negative Predictive Value (NPV)The likelihood that a negative test is true: that you don’t have the disease.
Problems that need correctingSolutions
· Poor quality studies
· Publish or perish
· Publication bias
· Lack of replication
· Mistakes missed by peer review
· Pay-to-publish journals
· Big Pharma malfeasance  
· Better education of researchers
· Quality control at journals
· Publish replications and negative studies
· Register all studies
· Full disclosure
· Improve media reporting  

Science is far from perfect

John Rennie said, “There seems to be no study too fragmented, no hypothesis too trivial, no literature citation too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, too obscure, and too contradictory, no analysis too self-serving, no argument too circular, no conclusions too trifling or too unjustified, and no grammar and syntax too offensive for a paper to end up in print.”

A warning

This warning statement should be attached to all research studies:
“Warning! Taking any action on the basis of this research could result in injury or death. The results described in this study have not been replicated and the long-term effects of this treatment are unknown. Past performance is no guarantee of future results. When subjected to further investigation, most published research findings turn out to be false.”

Two more things to remember

  1. My SkepDoc’s Rule: Never accept a claim until you understand who disagrees with it and why.
  2. Don’t ever forget to ask if you yourself might be biased.

This article was originally published in the Science-Based Medicine Blog.

Dr. Hall is a contributing editor to both Skeptic magazine and the Skeptical Inquirer. She is a weekly contributor to the Science-Based Medicine Blog and is one of its editors. She has also contributed to Quackwatch and to a number of other respected journals and publications. She is the author of Women Aren’t Supposed to Fly: The Memoirs of a Female Flight Surgeon and co-author of the textbook, Consumer Health: A Guide to Intelligent Decisions.

Scroll to top