From the Deputy Director for Intramural Research

Enhancing Reproducibility of Research Findings

BY MICHAEL GOTTESMAN, DDIR

One of the signal characteristics of experimental research is that it is self-correcting: Other scientists may confirm the findings, apply more sophisticated approaches to enhance and extend the results, or demonstrate that the results or conclusions are flawed. There has been a growing concern, however, that more than a small minority of published preclinical studies using animal models cannot be easily replicated. Even studies of basic cell and structural biology, which depend on complex data sets that rely on technology-intense interpretation, may not be yielding valid conclusions.

The NIH, as the primary funder of biomedical research, has an obligation to ensure that as high a percentage as possible of published research results stands the test of time. In general, irreproducibility is not due to scientific misconduct or deliberate falsification, fabrication, or fraud. What we are talking is a complex array of other factors that may result in an inability of scientists to reproduce the work of other scientists.

The primary tool that NIH has is its ability to educate researchers about what can go wrong and how to avoid common pitfalls. In preclinical animal research, the list of possible problems is long, from poor statistical design to uncontrolled environmental influences. The NIH intramural program has been asked to help pilot a computer-based training module that outlines the kinds of problems that can beset animal studies. We will start by asking our fellows to beta test the current training module that was developed by Shai Silberberg (extramural program director in the National Institute of Neurological Disorders and Stroke) and then participate in focus groups to help us determine how to make this module more effective and user-friendly. Eventually we also expect to have, as part of the overall training in research integrity, a sophisticated course in experimental design. Our scientific directors have given strong support to training that improves the integrity and reproducibility of our science, and we trust that all of our staff will benefit from exposure to this training experience.

Another area that concerns me is the possibility for errors in interpreting data obtained from advanced technologies such as high-resolution structural models from nuclear magnetic resonance, crystallographic, mass spectroscopic, and cryoelectron microscopy data; high-resolution cell imaging and cell-based fluorescence resonance energy transfer and fluorescence-activated cell-sorting analyses; and studies using specialized cell lines and antibodies.

Intramural NIH is very fortunate to have world-class experts in these technologies; I have assembled a committee that will help educate us about them. We are planning a one-day workshop at NIH—“New Advances in Structural and Cell-based Analyses: Potential and Pitfalls”—that should attract our trainees and staff to hear about the latest advances and the kinds of reproducibility problems that can arise. The workshop will also include our extramural academic and industrial colleagues and representatives from journals and professional societies, all of whom share concerns about data reproducibility. The goals are to educate researchers about what these techniques can accomplish; provide a cautionary note to scientists who are inexperienced in using these techniques but who plan to use them; and educate others who are reading results in the literature. A white paper and videos of the presentations will be made available for our staff and our extramural colleagues.

NIH Principal Deputy Director Larry Tabak has taken the lead in assembling a toolkit of approaches to enhance reproducibility and transparency of research findings. NIH institutes and centers are undertaking multiple pilots, including efforts to study and try to reduce factors that may motivate careless publication including “perverse incentives” (the promise of promotion, tenure, and, in rare instances, cash rewards to researchers who publish in certain journals) related to publication and funding.

Another way that NIH is addressing the problem is by altering the format of the “biographical sketch” that grant applicants must complete. The proposed format will emphasize the significance of advances resulting from the work and reflect individual contributions to successful research projects.

We are hoping that intramural NIH can help pilot important educational materials as they become available and lead the way in promoting reproducibility in research results. As always, your suggestions about how best to achieve these important goals would be welcome.

You might also be interested in reading a recent Nature article—“NIH Plans to Enhance Reproducibility”—that was co-authored by NIH Director Francis Collins and Lawrence Tabak (Nature 505:612–613, 2014).