Anthony Ozerov

Meta-analysis is the combination of results from different studies. By using historical particle physics studies, we find that meta-analysis can lead to overconfident results, especially when many studies are being combined.

“In science, sometimes many teams of researchers separately try to estimate the same quantity —whether it be the effect of a drug, the mass of a fundamental particle, or the rate at which the Antarctic ice sheet is losing ice. It is then helpful to perform a meta-analysis to quantitatively synthesize the results from the studies into one estimate with a confidence interval or error bars. Meta-analysis is no trivial task, and requires making assumptions about how well the researchers estimated their values and how realistic their own error bars are. So, does meta- analysis work? Using historical particle physics studies to simulate meta-analyses and compare results to much more modern “”ground truth”” values, we find that meta-analysis can lead to overconfident results, especially when many studies are being combined. This can be due to them all sharing the same biases, or otherwise somehow being correlated, and thus violating the independence assumption built into meta-analysis methods.