A high-profile paper about ways to improve the rigor of research papers has been retracted after critics attacked its own rigor. The study, published on 9 November 2023 in Nature Human Behaviour, purported to show the benefits of rigor-boosting measures including so-called preregistration—announcing the goals, methods, and other planned features of a study ahead of time—large sample sizes, and methodological transparency. It reported that these measures boosted the “replicability” of 16 findings in social-behavioral science to 86%, far more than the 30% to 70% reported in some analyses.
“Editors no longer have confidence in the reliability of the findings and conclusions reported in this article,” the journal said in a retraction note published yesterday.
“The concerns relate to lack of transparency and misstatement of the hypotheses and predictions the reported metastudy was designed to test; lack of preregistration for measures and analyses supporting the titular claim (against statements asserting preregistration in the published article); selection of outcome measures and analyses with knowledge of the data; and incomplete reporting of data and analyses,” the note says.
When Science covered the study last year, some scientists were already questioning its setup and methods, even though its authors included leading advocates for rigor in science. On 20 November 2023, Joseph Bak-Coleman of the University of Konstanz and Berna Devezer of the University of Idaho published a critique of the study, entitled “Causal claims about scientific rigor require rigorous causal evidence,” on the preprint server PsyArXiv. After carrying out a postpublication peer review and examining materials made available by the authors of the 2023 paper, Nature Human Behaviour agreed with the duo’s concerns. It published their criticism along with its retraction note yesterday.
The study’s authors acknowledged significant problems. “We are embarrassed by our elementary errors with reporting on the preregistration, and agree that they need to be addressed,” they wrote in a statement posted online on 23 September. Based on those errors, they decided to accept the retraction, the statement says, which called some other complaints about the paper “inaccurate.” The journal has invited the authors to submit a new manuscript for peer review, and they say they will. “We will revise the paper to address the inaccuracies and consider other critiques about the substance of our claims during revision,” the authors write.
In a lengthy account of the case in the Chronicle of Higher Education, Bak-Coleman pointed out the irony of the situation: “Here, in the best-case scenario—in a paper about the importance of embracing these reforms, by the experts who developed these reforms—the reforms themselves haven’t been well-embraced.”
