Yefeng Yang, Daniel W.A. Noble, Rebecca Spake, Alistair M Senior, Malgorzata Lagisz, Shinichi Nakagawa

1. Measuring heterogeneity, or inconsistency, among effect sizes is a crucial step for interpreting the meta-analytic evidence across diverse taxonomic groups and spatiotemporal contexts. However, ecologists and evolutionary biologists often interpret mean population effects (i.e., meta-analytic mean effect size) as consistent, either explicitly or implicitly, without proper heterogeneity quantification, thus assuming consistency in effects across contexts.

2. Here, we present a pluralistic approach aimed at quantifying heterogeneity by introducing complementary measures, each of which decomposes (stratifies) heterogeneity into within- and between-study variances. These measures include the traditional I², stratified I², the newly derived coefficient of variation (CV), and its transformation (M).

3. To demonstrate the benefits of the combined use of these measures, we synthesize 512 ecological and evolutionary meta-analyses. We show that total heterogeneity (variance in true effects) is, on average, ten times larger than statistical noise (sampling variance), contributing to 91% of the observed variance (median I² = 91%). This amount of heterogeneity is nearly twice the size of the mean population effect (median CV = 1.8 and transformation M = 0.6), indicating substantial variation among studies within a meta-analysis.

4. Surprisingly, despite a high amount of total heterogeneity is present in most meta-analyses, half of the meta-analyses had low among-study variance (and high within-study variance), indicating the meta-analytic mean effect could be generalizable across studies.

5. Our meta-synthesis can serve as new benchmarks for the interpretation of heterogeneity. Our proposed pluralistic approach provides our recommendations on how to quantify and report heterogeneity. Collectively, we could accelerate the future quest for generalizability of ecological and evolutionary phenomena via a better understanding of meta-analytic heterogeneity.