In 2023, Google awarded a total of $10 million to researchers who found vulnerabilities in its products. The rationale? Allowing errors to go undetected could be far costlier, potentially leading to data breaches, refund claims, reduced customer trust, or legal liability.
Bug Bounty Programs: A Model for Science
It's not just private technology companies that invest in such 'bug bounty' programs. Between 2016 and 2021, the US Department of Defense awarded more than $650,000 to individuals who found weaknesses in its networks.
Launching the ERROR Program
Inspired by these examples, the scientific community should similarly reward the detection and correction of errors in scientific literature. To address this need, I have teamed up with meta-scientist Ian Hussey at the University of Bern and psychologist Ruben Arslan at Leipzig University to pilot a bug-bounty program for science, funded by the University of Bern.
Our project, Estimating the Reliability and Robustness of Research (ERROR), compensates specialists for reviewing highly cited published papers, initially focusing on the social and behavioral sciences. Reviewers receive a base rate of up to 1,000 Swiss francs (approximately $1,100) per paper, plus a bonus for any errors they find, with rewards scaling up to 2,500 francs for significant errors.
Benefits for Authors and Initial Results
Authors who allow their papers to be scrutinized are also compensated: 250 francs for the preparation work and an additional 250 francs if only minor or no errors are found. Launched in February, the program will run for at least four years. So far, nearly 60 invitations have been sent, with 13 sets of authors agreeing to have their papers reviewed. One review has been completed, revealing minor errors.
Addressing Systematic Issues in Peer Review
The goal of ERROR is to demonstrate the value of systematic error detection in published research. Current peer review processes are often insufficient, as unpaid reviewers are overburdened and lack incentives to meticulously examine data or code. This leads to mistakes slipping through and a lack of personal gain for researchers who identify errors.
Undetected errors come with high costs, including wasted resources, flawed research influencing critical areas such as healthcare and policymaking, and opportunity costs for other projects.
Advocating for Investment in Error Detection
Stakeholders in science must recognize the importance of error detection and correction. Funders have a vested interest in ensuring their grants are well-spent, publishers can enhance their reputations by prioritizing quality management, and scientific associations can help foster a culture where acknowledging errors is seen as commendable rather than shameful.
Addressing Concerns and Moving Forward
While ERROR is an ambitious experiment and has faced questions about potential exaggeration of errors and bias in paper selection, measures are in place to mitigate these risks. Reviewer reports are published openly, and a 'recommender' from ERROR’s staff evaluates findings before payouts are decided.
The ultimate goal is not to prove that bug bounties are the best solution for every scientific field but to start a conversation about the need for dedicated investment in error detection. Alternatives such as making error detection a viable career path and employing full-time staff to review papers could also be explored.
Science can be self-correcting, but only if we invest in mechanisms to ensure it. The ERROR program is a step toward making error detection an integral part of the scientific process.