Assessing veracity of web-based citizen science projects: An audit study of eBird

Henry Michael Streby 

1. Uncertainty in data accuracy and lack of diversity among participants are two common concerns in citizen science projects. But what if the methods used to simultaneously maximize data quality and user participation lead to unbalanced user experiences and biased data? While many studies focus on advancing techniques for analyzing large and noisy datasets, methods are needed to test the veracity of the systems that produce those data.
2. Here we present a web-based audit study, a standard and increasingly common method in computer and social sciences but novel to ecology, to test the veracity of methods used in web-based citizen science data collection and curation. We tested for several types of bias in eBird’s data quality control process, with the intent to provide recommendations to improve representation and scientific veracity in the world’s largest citizen science project. Following a basic design for auditing web-based platforms, we systematically submitted 19,491 fictitious bird observations from 196 simulated eBirders who varied by region, sex, and race, and we assessed how those variables were associated with eBird’s real acceptance or rejection of the fictitious observations.
3. We found near-total confirmation bias, in that eBird accepts 100% of false observations that fit eBird’s a priori expectations for species’ spatio-temporal ranges, while accepting 0 – 29% of true observations that do not fit eBird’s expectations. Hence, the eBird data quality control process does not validate data, but rather it produces illusory quality through forced conformity. We show how this problem contributes to a larger, self-perpetuating cycle of bias inherent to the design and function of eBird. We cannot report results from geographic or demographic analyses (see SI).
4. Regarding species’ spatio-temporal ranges, and changes thereto, we conclude that the eBird database almost entirely reflects eBird’s predetermined expectations, and changes that eBird makes to those expectations over time. This irreparably detaches the eBird dataset from ecological reality because no statistical technique can reconstitute patterns that are perpetually and permanently removed and replaced during data collection and curation. eBird is an undeniably magnificent, user-friendly birdwatching application, but as citizen science, its design obviates the citizen and feigns science. The ecological and evolutionary sciences could benefit from similar independent investigations of other web-based research projects.