Reliable inference: benefits of open raw data may be universal in meta-analysis

Danijela Žanko, Sunčana Geček, Azra Tafro, Livia Puljak, Antica Culina

While the benefits of open data are often discussed, they are rarely quantified. Here, we provide the first evidence of the potential gains from using raw data for evidence synthesis and introduce a tool that helps researchers determine when this approach is most beneficial. Classical meta-analysis (CMA) relies on published results, making it vulnerable to publication bias and p-hacking. We developed a simulation framework comparing CMA with raw data meta-analysis (RDMA) across varying true effect sizes, heterogeneity, and bias levels. Using ecology as a case study, we demonstrate that RDMA outperforms CMA in most scenarios. When true effects are small and bias is severe, RDMA reduces relative mean absolute error by 56–76%. Under moderate bias, reductions reach 31–62%. For medium true effects, reductions were 50–71% and 3–38%, respectively. RDMA maintained reliable confidence interval coverage across all scenarios, whereas CMA failed to do so. Crucially, RDMA's errors reflect natural sampling variation, while CMA's reflect systematic bias that persists regardless of the sample size. We provide a decision tool for meta-analysts across disciplines to calculate RDMA's benefits. Our results offer the first quantitative evidence that open data improves meta-analytic accuracy, strengthening the case for open science.