When Azzurra Ruggeri of the Technical University of Munich saw the grant solicitation for bold new ideas in cultural studies, she was eager to submit her proposal to use her expertise as a developmental psychologist to study whether artificial intelligence (AI) could be designed to behave ethically. Only after Ruggeri applied did she notice an unusual condition in the grant competition’s rules: All applicants were required to review others who responded to the same solicitation, letting the funder take advantage of a ready-made pool of relevant experts.
The unconventional approach, called distributed peer review (DPR), has been touted for its potential to alleviate long-standing ailments with traditional reviews of grant applications and journal-article manuscripts, in which a panel of invited outside experts score and debate submissions. The problems include delays in decisions, the growing workload from an ever-rising number of submissions, the risk of bias or groupthink among the panels, and a lack of consistency across reviews. DPR, however, has its own potential shortcomings, including the risk that reviewers might score other applications poorly in hopes of making theirs stand out—a form of gaming that grant funders using the model have sought to deter by checking reviews and other mechanisms.
Now, results from the solicitation Ruggeri participated in—run by the Volkswagen Foundation—and another this year by the funder UK Research and Innovation (UKRI), to choose fellows who will study how AI is shaping science, offer some perspective on what works about DPR, as well as areas for improvement. The UKRI trial, for example, found the strategy cut the time from submission to funding decision by as much as two-thirds, to 2 months, compared with normal panel reviews. And for both grant solicitations, substantial numbers of applicants surveyed were open to participating again in another competition using DPR. Still, many worried whether they and their fellow reviewers possessed the right expertise to do the job well.
