Ozan Cinar, Shinichi Nakagawa, Wolfgang Viechtbauer

1. Meta-analyses in ecology and evolution require special attention due to certain study characteristics in these fields. First, the primary articles in these fields usually report results that are observed from studies conducted with different species, and the phylogeny among the species violates the independence assumption. Second, articles frequently allow the computation of multiple effect sizes which cannot be accounted for by conventional meta-analytic models. While both issues can be dealt with by utilizing a multilevel model that accounts for phylogeny, the performance of such a model has not been examined extensively. In this article, we investigate the performance of this model in comparison with some simpler models.

2. We conducted an extensive simulation study where data with different hierarchical structures (in terms of study and species levels) were generated and then various models were fitted to examine their performance. The models we used include the conventional random-effects and multilevel random-effects models along with more complex multilevel models that account for species-level variance with different variance components. Furthermore, we present an illustrative application of these models based on the data from a meta-analysis on size-assortative mating and comment on the results in light of the findings from the simulation study.

3. Our simulation results show that, when the phylogenetic relationships among the species are at least moderately strong, only the most complex model that decomposes the species-level variance into non-phylogenetic and phylogenetic components provides approximately unbiased estimates of the overall mean and variance components and yields confidence intervals with an approximately nominal coverage rate. Contrarily, removing the phylogenetic or non-phylogenetic component leads to biased variance component estimates and an increased risk for incorrect inferences about the overall mean. These findings are supported by the results derived from the illustrative application.

4. Based on our results, we suggest that meta-analyses in ecology and evolution should use the model that accounts for both the non-phylogenetic and phylogenetic species-level variance in addition to the multilevel structure of the data. Any attempts to simplify this model, such as using only the phylogenetic variance component, may lead to erroneous inferences from the data.