Reading medical evidence: effect sizes, confidence, and the hierarchy

Not all medical evidence carries the same weight. The conventional hierarchy, from highest to lowest quality:

• Systematic reviews and meta-analyses of multiple randomized controlled trials.
• Randomized controlled trials (RCTs) — patients are randomized to treatment or control, ideally double-blinded. Randomization controls for confounding; blinding controls for measurement bias.
• Cohort studies — groups of patients are followed over time, comparing those who received an exposure to those who did not. Vulnerable to confounding because exposure is not randomly assigned.
• Case-control studies — start with patients who have the outcome and look backward at exposures. Useful for rare outcomes; vulnerable to recall and selection bias.
• Case series and case reports — descriptions of one or a few patients. Useful for generating hypotheses, not for establishing causation.
• Expert opinion — clinical judgment from experienced practitioners. Useful when better evidence is unavailable.

The hierarchy is not absolute. A small underpowered RCT may be weaker evidence than a large well-conducted cohort study. A single perfectly-designed mega-trial may be more conclusive than a meta-analysis of many small biased trials. The hierarchy is a default — start at the top, drop down only when the higher levels are unavailable or inconclusive.

The rest of this lesson examines how to read a trial result at the level the hierarchy points to: usually phase III RCTs reported in journals, summarized in regulatory documents, or pooled in systematic reviews.

A clinical trial reports an outcome — say, the rate of heart attacks. Two ways to express the treatment effect:

Absolute risk reduction (ARR). The difference in event rates between treatment and control.

Example: 10 of 1000 treatment patients have a heart attack (1.0%); 20 of 1000 control patients have a heart attack (2.0%). ARR = $2.0\% - 1.0\% = 1.0$ percentage point.

Relative risk reduction (RRR). The fraction of the control event rate that is averted by treatment.

From the same numbers: RRR = $(2.0\% - 1.0\%) / 2.0\% = 50\%$.

Both describe the same effect; both are mathematically correct. But they communicate differently:

• 'The drug cuts heart attack risk by 50%' sounds enormous.
• 'The drug cuts heart attack risk by 1 percentage point' sounds modest.
• 'The drug averts 10 heart attacks per 1000 patients treated' sounds concrete.

The three statements are the same number. Marketing materials, popular press, and even some journal abstracts preferentially quote RRR because it is the larger-sounding number; regulatory and HTA documents typically require ARR.

A general reading rule: when a percentage reduction is reported, ask for the baseline rate. A 50% RRR off a baseline of 0.1% is an ARR of 0.05%; the absolute clinical impact is small. A 50% RRR off a baseline of 30% is an ARR of 15%; the impact is large. The same RRR can be clinically transformative or clinically irrelevant depending on the baseline.

Number needed to treat (NNT) is the inverse of absolute risk reduction:

$\text{NNT} = \frac{1}{\text{ARR}}.$

It is the number of patients you must treat for one of them to benefit. From the previous example with ARR = 1.0%, NNT = $1 / 0.01 = 100$. You must treat 100 patients for 1 heart attack to be prevented.

NNT is the single most useful number for understanding clinical impact. A drug with NNT = 5 prevents one event per 5 patients treated — a large effect. A drug with NNT = 100 prevents one event per 100 patients treated — modest. A drug with NNT = 1000 prevents one event per 1000 patients treated — small at the individual level, but potentially significant at the population level if the cost and side-effect burden are also small.

Number needed to harm (NNH) is the parallel for adverse outcomes: the number of patients you must treat for one to experience a particular side effect. NNH together with NNT gives a benefit-harm ratio.

• NNT = 50 and NNH = 200 for a serious adverse event suggests the benefits clearly outweigh the harms.
• NNT = 50 and NNH = 10 suggests the harms outweigh the benefits.

The trade-off depends on the severity of the prevented event versus the severity of the side effect. Preventing a death is worth tolerating moderate side effects in many; preventing mild discomfort is not worth a serious side effect at any NNH. The numbers don't make the value judgment; they make the value judgment explicit.

A treatment effect estimated from a sample comes with uncertainty. A 95% confidence interval (CI) is the range of values consistent with the data; if you ran the same trial many times under the same conditions, 95% of the resulting intervals would contain the true effect.

What a CI tells you:

• Width. A narrow CI means the estimate is precise; a wide CI means the data don't pin the value down. Wide intervals usually mean the trial was underpowered.
• Position relative to the null. A CI that crosses the null value (e.g., includes 1.0 for a relative risk, 0 for an absolute difference) is consistent with no effect; a CI that excludes the null is evidence of an effect.
• Lower bound. For a treatment effect, the lower bound tells you the worst-case effect consistent with the data. If even the lower bound represents a clinically meaningful effect, the result is more compelling than if the lower bound is clinically negligible.

What a CI does not tell you:

• It is not a probability statement about the parameter. '95% probability the true effect is in this interval' is a Bayesian statement; the frequentist CI does not directly support it.
• A CI that excludes the null is necessary for statistical significance but does not by itself establish clinical importance.

A practical reading rule: when an effect estimate is reported, read the CI before the point estimate. A 'significant' effect with a CI from $1.01$ to $1.99$ on a hazard ratio is consistent with effects ranging from nearly nothing to substantial. Treating the point estimate as if it were precise overstates what the data establish.

A p-value is the probability of observing a result at least as extreme as the data, under the assumption that the null hypothesis is true. By convention, $p < 0.05$ is called 'statistically significant.'

What a p-value is not:

• Not the probability that the null hypothesis is true.
• Not the probability that the result was due to chance.
• Not the size of the effect.
• Not a measure of how strongly the data favor the alternative hypothesis.

Common misinterpretations follow from confusing the conditional ($P(\text{data} \mid \text{null})$) with the inverse ($P(\text{null} \mid \text{data})$). The two are related by Bayes' theorem and require knowledge of the prior probability of the null — which p-values do not provide.

The p < 0.05 convention is a historical artifact (Fisher's suggestion in 1925, never intended as a hard threshold) that has become a binary decision rule with unintended consequences:

• Publication bias. Studies with $p < 0.05$ get published more than studies with $p > 0.05$, even if the effect sizes are similar. The published literature systematically overstates effects.
• P-hacking. Subgroup analyses, optional stopping, and post-hoc selection of analyses can produce $p < 0.05$ from data with no real effect. Pre-registration and analysis-plan discipline mitigate this.
• The dichotomy problem. A result with $p = 0.04$ is treated as conclusive; a result with $p = 0.06$ is treated as nothing. The underlying evidence is nearly identical.

Most modern statistical writing recommends reporting effect estimates with confidence intervals rather than p-value-based conclusions. The American Statistical Association issued a formal statement (2016) cautioning against treating p < 0.05 as definitive.

A trial reports an average effect across all patients in the trial. The heterogeneity of treatment effects (HTE) asks: does the effect vary across patient subgroups, and by how much?

Real medical interventions almost never have uniform effects. Some patients benefit substantially; some not at all; some are harmed. The trial's average effect collapses this heterogeneity into one number.

Approaches to characterize heterogeneity:

• Pre-specified subgroup analyses. Analyze the effect separately in subgroups defined before the trial (age, sex, disease severity, biomarker status). Pre-specification controls for multiple-testing inflation. Effects can differ across subgroups in clinically important ways.
• Interaction tests. Statistical tests for whether the treatment effect differs across subgroups. Often underpowered because they require detecting an effect in the difference between subgroups, which has more variance than the effect within a single subgroup.
• Individual-level prediction. Risk models that predict, for each patient, the likely benefit and likely harm. The discipline of 'precision medicine' aims at this level of granularity.

The practical reading: a trial's headline effect estimate applies to a population that looks like the trial's population. A patient who differs substantially from that population (much older, sicker, on different concomitant medications, of a different ethnic group than the trial enrolled) may have an effect that differs from the average — possibly larger, possibly smaller, possibly opposite-sign.

The FDA labels and clinical guidelines try to communicate the trial population's characteristics, but the gap between trial-enrolled and real-world-treated patients is structural and large in many indications.

A systematic review is a structured synthesis of all available studies addressing a specific question. A meta-analysis is a quantitative synthesis that pools effect estimates across studies into a single combined estimate.

Why pool studies:

• A single trial may be underpowered to detect a modest effect; multiple trials together have more power.
• Different trials may apply to different populations; the pooled estimate represents a broader inference.
• Discordant trial results can be resolved by examining heterogeneity across trials.

Quality varies:

• A meta-analysis is only as good as the trials it includes. Pooling biased trials produces a confidently-wrong estimate.
• Heterogeneity statistics ($I^2$, Cochran's Q) quantify whether the included trials' effects are consistent. High heterogeneity means the trials disagree more than chance would predict and the pooled estimate may not represent a meaningful single effect.
• Publication bias affects meta-analyses: if positive trials are published and negative trials are not, the pool is skewed. Funnel-plot analysis and 'gray literature' searches partially address this.

Well-conducted meta-analyses are at the top of the evidence hierarchy. Poorly-conducted ones — pooling clinically dissimilar trials, ignoring quality differences, post-hoc subgrouping — produce misleading certainty. The reading skill is to assess the meta-analysis itself with the same discipline applied to individual trials.

A structural framework for reading any medical evidence claim.

• What is the study design? RCT, cohort, case-control, observational, expert opinion. The design determines the strength of inference.
• Who is in the study? What population, what inclusion and exclusion criteria? Does my patient resemble that population?
• What is the comparator? Placebo, standard of care, active control, no control. Effects vs placebo differ from effects vs standard of care.
• What is the endpoint? Hard or surrogate? Pre-specified or post-hoc? Mortality, morbidity, biomarker?
• What is the effect size? Absolute and relative reduction; NNT; effect size in clinical units. Is the effect clinically meaningful, not just statistically significant?
• What is the uncertainty? Confidence interval. How precisely is the effect known? Does the lower bound represent a clinically meaningful effect?
• What is the heterogeneity? Does the effect vary across pre-specified subgroups? Does it apply to populations not represented in the trial?
• What are the harms? Side effects, adverse events, NNH. How do they compare to the benefits?
• What is the conflict of interest? Industry-funded, government-funded, academic? Funding does not invalidate results but should sharpen scrutiny.
• Has the result been replicated? A single trial, however large, may not generalize. Independent replication is the strongest evidence.

Applying this framework consistently — to drug trials, to nutrition studies, to public health claims — produces a reading that matches the actual strength of the evidence. Skipping any element produces overconfidence in the direction the source biases toward.

The biology cursus closes here. Six lessons of structural framework: information flow from DNA to protein; editing the genome via CRISPR; editing without breaks via base and prime editors; hormone-receptor agonist pharmacology; the clinical-trial pipeline; and reading the resulting evidence. The frameworks outlast specific therapies; the therapies they cover today will be different a decade from now, but the structure of how to think about them will be similar.