Commentary

Enhancing Reproducibility of Neuroscience Studies

A Report from the Society for Neuroscience Meeting

BY EMILY PETRUS, NINDS

Scientific disagreements are essential for progress. Hypotheses are rigorously tested, and results are peer reviewed at every step of discovery. But there is a growing concern that many published preclinical studies, especially ones using animal models, cannot be replicated.

At its annual meeting, held in Washington, D.C. last November, the Society for Neuroscience addressed that concern in a symposium—“Enhancing the Reproducibility of Neuroscience Studies”—chaired by Story Landis (recently retired director of the National Institute of Neurological Disorders and Stroke) and Thomas Insel (director of the National Institute of Mental Health). Panelists included NIH Director Francis Collins, an editor from Nature, and faculty members from two medical schools.

The panelists agreed that, for the most part, irreproducibility is not due to scientific misconduct. The main contributors, Collins explained, include cutting-edge science that moves quickly and uses new techniques that may not have been rigorously tested; confounding variables that may influence data, such as the exclusion of females from most studies; poor training of researchers in experimental design; and the constant pressure on investigators to find funding and to publish in high-profile journals.

NIH is addressing the challenges by developing training modules on enhancing reproducibility, with an emphasis on good experimental design; conducting workshops with scientists, policy makers, and others to design routes of funding that reward investigators who consistently generate reproducible data; developing a more systematic way to evaluate grant applications; and exploring ways to provide greater transparency of data.

Nature Executive Editor Véronique Kiermer focused on the role of journals in tackling irreproducibility. She described the recent increase in articles requiring corrections and retractions, which is either due to increased scrutiny or lower-quality data. To combat this trend, Nature and other journals have implemented such measures as eliminating word limits in the methods section of articles (so more details can be reported), hiring statisticians to ensure statistics are performed appropriately, placing a stronger emphasis on randomized datasets with blinded investigators, and enhancing transparency by providing raw data, spreadsheets, and analysis codes as supplemental information for articles. Although Kiermer acknowledged that journals have a responsibility for publishing quality data, she emphasized that training the next generation of scientists is just as important for reversing the trend of irreproducible data.

John Morrison (Icahn School of Medicine at Mount Sinai in New York) described his institution’s increasing emphasis on training excellent scientists. Currently the focus is on the student-faculty relationship, through which best practices should be taught and encouraged at the lab level. Professors are under enormous pressure to publish or perish, which can trickle down to students and postdocs. Mount Sinai aims to reduce the pressure on faculty by placing more emphasis on the rigor of research and not just whether it’s published in high-profile journals. In the lab, investigators are encouraged to enhance reproducibility by training students and fellows to become better scientists, leading by example, and placing high priority on these items.

Changing lab culture is important if we hope to improve data reproducibility and apply it to the clinical setting, said Huda Zoghbi (Baylor College of Medicine in Houston). She described how her lab spent two years attempting to reproduce a simple fear-extinction memory task in a mutant mouse. She was unsuccessful and unable to publish these negative results because they conflicted with published positive results. “We need to enhance reproducibility as a responsibility to each other!” was the moral of the story.

The symposium concluded with hopeful remarks from the entire panel and scientists in the audience. “We have seen the enemy, and it is us,” said Collins. “But the solution is also us.” Science should aim to seek the truth above all else.