Reversing the reproducibility crisis
The editors of Scientific American Worldview asked five experts one question: In your opinion, how bad is the problem of reproducibility in scientific research and what should be done about it? Here’s what they said.
The currency of science is publishing. Producing novel, positive and clean results maximizes the likelihood of publishing success because those are the best kind of results. There are multiple ways to produce such results: one, be a genius; two, be lucky; three, be patient; or four, employ flexible analytic and selective reporting practices to manufacture beauty. In a competitive marketplace with minimal accountability, it is hard to resist number four. There is accumulating evidence that the published literature is more beautiful than reality, perhaps much more. The solution is transparency. If you know how I arrived at my claims, then the credibility of those claims will stand or fall on their own merits—not on my insistence that you should believe me.
professor of psychology
University of Virginia
While it depends on which field you look at, and which analysis, at least half of findings in fields from preclinical cancer research to economics to psychology can’t be reproduced successfully. To be clear, 100% reproducibility would suggest that researchers aren’t thinking in very innovative ways. But under 50% is too low. We need to create incentives for replication, whether those are carrots—designated funding—or sticks—metrics that compare rates of reproducibility. One reason for some of these failures to replicate is that researchers don’t provide enough information about their methods, so we need to enforce those mandates, too.
Self-correction of science has been compromised due to external pressures to publish papers in high-profile journals and the difficulty in obtaining funding to support research efforts. Further, the increasing dependence of interdisciplinary approaches to address scientific questions makes it almost impossible for any one person to be sufficiently knowledgeable to identify potential flaws in published work. All stakeholders must engage in implementing solutions to redress this—better training is needed, incentives must be aligned with fostering rigor and reproducibility in science reporting, and publications must be evaluated on their content and not on the citation index of the journal.
Francis S. Collins
Lawrence A. Tabak
principal deputy director
U.S. National Institutes of Health
In a PLOS Biology paper published last June, we calculated a 50% irreproducibility rate, or nearly $28 billion per year spent on irreproducible research in the United States alone. Even if the dollar amount and rate are not exact, these are still big—and troubling—numbers. GBSI is developing actionable solutions where best practices and standards play a central role. For example, too many misidentified cell lines are still used in the lab and authenticating cells is a cost-effective solution. We applaud the Cancer Moonshot, but we can’t cure cancer if researchers aren’t using authenticated cancer cells in their labs.
Leonard P. Freedman
Global Biological Standards
It is a serious problem, with multiple causes. Incompetence, carelessness and fraud all contribute, but are minor—and fixable—in comparison to the main barrier to reproducibility: sheer complexity. Understanding how things “really are,” which is the ultimate goal of science, is beyond us for most questions in the life sciences, at least for now. Instead, we must pursue what in another context Vladimir Nabokov describes as “the rapture of endless approximation.” Biomarkers, genome-wide association studies, in vitro and in silico research provide useful approximations of health and ill-health but full understanding is a long way off. Maximum collaboration, respectful skepticism and a positive attitude will get us there.
president and editor-in-chief
Opinions expressed in worldVIEWpoint do not necessarily reflect those of the editors.
Illustration by © DAN PAGE; © FREEDMAN PHOTO: JUDY LICKT/GBSI
Enhanced with a new guidebook and region-specific ratings, the 2016 Scorecard ventures deeper than ever to track down the latest in biotech innovation