How To Read A Scientific Study You Saw In The News

Most reporting on research compresses a careful paper into a claim. The compression is where the errors enter. A few questions, answerable in minutes from the paper's own abstract, filter out most of the noise.

Who or what was studied

Find the sample: how many, and of whom. A result in mice is a result in mice. A result in thirty undergraduates may not extend to anyone else.

Small samples produce large, unstable effects. That is a statistical property, not a criticism of the researchers — but it means an eye-catching effect from a small study is the finding most likely to shrink on replication.

Whether anything was assigned

Distinguish an experiment from an observation. If researchers assigned people to conditions at random, causal language may be warranted. If they observed people who already differed, then something else about those people may explain the result.

This single distinction accounts for a large share of misleading health coverage. Observational findings routinely become causal claims in the retelling.

What the comparison was

A treatment is always compared to something. Compared to nothing, compared to existing practice, and compared to a placebo are three different questions with three different answers.

How large the effect was

Statistical significance means an effect is unlikely to be chance. It says nothing about size. Look for the absolute change, not only the relative one: a large percentage change in a very rare outcome can be a tiny difference in practice.

Whether it stands alone

A single study is a data point. Where a body of evidence exists, a systematic review is far more informative than the newest paper, which is necessarily the least replicated.

And read the limitations

Papers state their own limitations, in a section headed limitations. It is the most useful paragraph in the document and the one least likely to survive into a news story.