meta-analysis-heterogeneity.../README.md

58 lines
7.6 KiB
Markdown
Raw Normal View History

2026-10-20 12:26:24 +00:00
# Understanding Heterogeneity in Meta-Analysis: A Reader's and Reviewer's Guide
2026-10-20 12:26:06 +00:00
2026-10-20 12:26:24 +00:00
When a meta-analysis combines results from several studies, the first question a careful reader asks is not "what is the pooled effect?" but "do these studies actually agree?" Heterogeneity is the term for variation in results beyond what chance alone would produce. It determines whether a single summary number is meaningful, how wide the uncertainty around it should be, and what the review can honestly conclude. This guide explains the main kinds of heterogeneity, the statistics used to describe it, and practical ways to investigate and report it. Along the way it notes how an AI research assistant like [Yila](https://yila.ai) can help with reading and organizing studies, while statistical judgments stay with the review team.
## Three kinds of heterogeneity
**Clinical or conceptual heterogeneity** refers to differences in participants, interventions, comparators, outcomes and settings. A review of exercise for back pain may include supervised gym programmes and home walking advice, adults in their twenties and people over seventy, and outcomes measured at two weeks or two years.
**Methodological heterogeneity** refers to differences in study design and risk of bias. Randomized trials with concealed allocation and blinded outcome assessment may give different answers from trials without those protections.
**Statistical heterogeneity** is what the numbers show: effect estimates that vary more than their confidence intervals would predict. Statistical heterogeneity is usually a consequence of the other two. It is a signal that something about the studies differs, not an explanation in itself.
## Looking before calculating
The most useful heterogeneity tool is the forest plot. Before reading any statistic, look at the plot. Do the confidence intervals overlap? Are the point estimates scattered on both sides of the line of no effect? Is one large study pulling the pooled estimate? Visual inspection often reveals patterns, such as one outlying study or a cluster of small studies with large effects, that a single number hides.
## The common statistics
**Cochran's Q** tests the null hypothesis that all studies share a common true effect. It has low power when there are few studies, so a non-significant Q does not prove homogeneity, and high power when there are many studies, so a significant Q may reflect trivial variation. Many reviewers use a more lenient threshold, such as p below 0.10, to compensate for low power.
**I-squared** describes the percentage of total variation across studies that is due to heterogeneity rather than chance. It is easy to report and widely used, but it is often misread. I-squared is a relative measure: it depends on the precision of the studies. With very large, precise studies, even small absolute differences in effect produce a high I-squared. With small, imprecise studies, large real differences may produce a modest one. Rough interpretation bands exist, such as low, moderate, substantial and considerable, but they overlap deliberately and should not be applied mechanically.
**Tau-squared** estimates the variance of true effects across studies in a random-effects model. Its square root, tau, is on the same scale as the effect measure, which makes it more interpretable than I-squared for judging whether the spread of effects matters in practice.
**Prediction intervals** use tau to estimate the range within which the true effect of a new, similar study would likely fall. A pooled estimate might show a clear benefit while the prediction interval crosses into harm, which tells readers that the intervention may not work in every setting. Prediction intervals are underused and very informative.
## Fixed-effect and random-effects models
A fixed-effect, or common-effect, model assumes that all studies estimate the same underlying effect and that differences are due only to sampling error. A random-effects model assumes that true effects vary between studies and estimates their average.
The choice should be based on what the review is trying to estimate, not on the result of a heterogeneity test. Where studies differ in populations and interventions, a random-effects model is usually the more realistic assumption. Reviewers should be aware that random-effects models give relatively more weight to small studies, which matters when small studies are systematically different, for instance because of publication bias. Reporting both models as a sensitivity analysis can be helpful.
## Investigating heterogeneity
When heterogeneity is present, the review should try to explain it, using methods specified in advance in the protocol.
**Subgroup analysis** compares pooled effects between groups of studies, such as different doses, populations or risk of bias levels. The key is to test for a difference between subgroups, not to note that one subgroup is statistically significant and another is not. Subgroup findings are observational even when the included studies are randomized, and they are prone to false positives when many subgroups are examined.
**Meta-regression** extends this to continuous characteristics, such as mean age or follow-up duration. It needs a reasonable number of studies, often cited as at least ten per characteristic, and suffers from the same observational limitations as subgroup analysis. Characteristics measured at the study level, like average age, can also mislead about individual-level relationships.
**Sensitivity analysis** checks whether conclusions change when particular decisions are varied, for example excluding studies at high risk of bias, changing the model, or removing an outlier. Sensitivity analyses are about robustness rather than explanation, but they often clarify which studies drive heterogeneity.
## When not to pool
Sometimes the right decision is not to combine the results at all. If studies measure fundamentally different interventions or outcomes, or if heterogeneity is extreme and unexplained, a single pooled number can be more misleading than informative. In such cases, a structured narrative synthesis, guided by reporting standards such as SWiM, may be more honest. Presenting a forest plot without a summary diamond is another option that shows the data while avoiding a misleading average.
## Reporting
Good reporting includes the forest plot, the chosen model and reason, tau-squared or tau, I-squared with its confidence interval if available, the prediction interval, any pre-specified subgroup or meta-regression analyses with tests for interaction, and sensitivity analyses. When rating the certainty of evidence with GRADE, unexplained heterogeneity is a reason to downgrade for inconsistency, and the review should state how that judgment was made.
## Where tools help
Statistical software handles the calculations. Where assistants help most is earlier: extracting study characteristics consistently so that clinical and methodological differences can be examined, summarizing how interventions differ, and keeping notes about decisions. A research assistant such as Yila can help organize this material and draft plain-language explanations of results for a wider audience, but the choice of model, the interpretation of statistics and the decision whether to pool belong to the review team.
## Summary
Heterogeneity is not a nuisance to be minimized but information about how and why study results differ. Look at the forest plot first, report absolute as well as relative measures, use prediction intervals, investigate differences with pre-specified analyses, and be willing not to pool when combining would mislead. Readers will trust a meta-analysis more when it explains its variation openly.