Key Highlights Discussed by Individual Participants
- Steps to improve reproducibility of experimental results can be implemented by graduate programs (additional curriculum courses on experimental rigor and design), publishers (publish both positive and negative experimental results as well as detailed experimental methods), and funding agencies (issue longer awards and support replication studies) (Landis and Mason).
- Experimental design can be enhanced by incorporating discussions of best practices into every course taken by trainees, regardless of topic, thus ensuring ongoing learning that can be applied in a variety of contexts (Chesselet).
- Trainees need to have an understanding of what outcome measures are needed to appropriately test their hypotheses when designing their experiments (Marder).
- Scientists often do not have the statistical training required to determine the most appropriate analyses to perform for their experiments (Brown).
- Statistics modules tailored to the key subdisciplines of neuroscience can augment general courses on statistics, providing a more specific set of analytical skills that might benefit trainees according to their concentration (Brown).
- In addition to enhancing training in statistics, the field of neuroscience could benefit from more collaboration with statisticians (Brown and Weber).
NOTE: The items in this list were addressed by individual participants and were identified and summarized for this report by the rapporteurs. This is not intended to reflect a consensus among workshop participants.
Over the course of the workshop, participants discussed several opportunities for trainers to think differently about how to train the next generation of neuroscientists, in areas such as cross-discipline collaborations, handling of large datasets, and the development of new tools and technology, as described in the previous chapter. Despite the inherent challenges, including significant changes to the culture of neuroscience, training in these areas appears to have generated an overall sense of positivity and excitement within the neuroscience community. In contrast, many workshop participants identified noticeable gaps in three areas of neuroscience training: protocol design, experimental rigor, and quantitative skills.
Many participants discussed several challenges associated with trainees not learning the fundamentals of conducting rigorous experiments, including the risk of irreproducible findings. Marder led the discussion on protocol design, emphasizing the importance of using common sense and deep intuition about data to design and execute the right experiments as the landscape of neuroscience continues to evolve. Emery Brown, professor of computational neuroscience at the Massachusetts Institute of Technology, outlined gaps in the training on quantitative skills and the need for recruitment of experts in statistics to the field of neuroscience.
ENHANCING EXPERIMENTAL RIGOR AND REPRODUCIBILITY
The enterprise of science is based on making discoveries that can capture new knowledge about the world. That knowledge may be exploited to make predictions about the occurrence of natural phenomena, or it can be the inspiration to potentially manipulate the natural world. However, if discoveries are not repeatable, those predictions are insignificant and manipulations will be ineffective. Several workshop participants addressed science’s irreproducibility problem, which has been well documented in the past few years, notably in an article in The Economist, as well as in several journal articles (Begley and Ellis, 2012; Chatterjee, 2007; The Economist, 2013; Perrin, 2014; Prinz et al., 2011; Scott et al., 2008; Steward et al., 2012).
Participants discussed some causes of irreproducibility. Brown said the inability to reproduce results boils down to scientists’ inability to reason under uncertainty and understand how to analyze data. Martone also mentioned the role of data, specifically how the data are handled and tracked, as being critical for reproducing findings. Carol Mason, professor of pathology and cell biology, neuroscience, and ophthalmic science at Columbia University, informed participants about the Enhancing Reproducibility of Neuroscience Studies1 symposium that occurred at the 2014 Society for Neuroscience meeting, during which invited speakers listed several likely causes of poor reproducibility to include (Landis et al. 2012; Steward and Balice-Gordon, 2014):
- Difficulty of generating cutting-edge science
- Confounding variables
- Unreliable resources (cell lines, chemicals, antibodies)
- Deficient experimental procedures
- Lack of transparency in reporting findings
- Randomization, blinding, sample size estimations
- Publication bias
One workshop participant placed the responsibility for changing the culture around experimental rigor primarily on faculty members within graduate programs. He said that as journal reviewers and editors, grant reviewers, mentors, and hiring and promotion advisory board members faculty are in the best position to demand change and to model it to trainees. Landis suggested the development of curriculum for courses about experimental rigor that can be shared across universities to ensure that all students and faculty alike receive the same training in best practices (see program example in Box 3-1). In addition, Mason suggested that enhanced
Program Example: Harvard University Data Boot Camp
The methods used to teach the data analysis skills that Brian Litt mentioned are critical, according to Michael Springer, assistant professor of systems biology at Harvard Medical School. Along with colleague Rick Born, professor of neurobiology at Harvard Medical School, Springer runs a Matlab-based boot campa in programming and quantitative skills that employs innovative training methods. Boot camp students not only learn how to use programming tools, but they also learn the best tools to use for specific problems and how to evaluate if their tools are working properly. The boot camp focuses on image development, statistics, and bioinformatics and modeling. Through lectures, long examples, and hands-on experiences, students learn how to visualize data and how to approach them from different directions. Based on responses that students give during in-lecture quizzes using an interactive tool, teaching assistants can identify who would benefit from one-on-one interactions. Springer also finds that peer-to-peer mentoring can be more effective than classroom lectures in some situations.
training in ethics might help to address the increasing manipulation of data and plagiarism. She added that webinars on statistical reasoning and proper experimental design might help raise awareness of issues related to randomization, blinding, and calculating sample size.
DESIGNING EXPERIMENTS WITH COMMON SENSE AND INTUITION
The biggest challenge facing neuroscience is training students to comprehend the data they are collecting, said Marder. As next-generation technologies proliferate and experiments become more complex and multifaceted, she opined that common sense and intuition will be increasingly critical. Without the basic understanding of what their data should look like and what they mean, scientists will be unable to determine what experimental design details matter for what problem. Part of
this understanding includes knowing the appropriate statistical tests to use for a study ahead of time, and what outcome measures to test. According to Marder, simple prestudy power analyses should inform how much data should be collected rather than collecting data until a p-value reaches significance. She noted that intuition and communication are also important when working collaboratively; a statistician might not know how the data were collected or their meaning after conducting the statistical analyses.
Marder contends that it is difficult to comprehend one’s data without working with raw, unprocessed data. Neuroscientists too often become separated from their raw data by models embedded in the hardware they use to collect data and in off-the-shelf programs they use to analyze those data. Students working with functional magnetic resonance imaging (fMRI), for example, may not fully understand the algorithms that preprocess their data before they examine them. As a result, some students have difficulty with experimental design by not understanding their data analysis tools better, said Marder. She added that a lack of intuition about data also makes it challenging to troubleshoot problematic data, and can lead to faulty interpretations of the experimental results. Marder suggested that the development of new methodologies to help visualize large datasets and reduce the dimensionality might facilitate the comprehension of one’s data.
According to several workshop participants, another challenge to designing the right experiment is knowing what tool to use to collect the appropriate data. Akil warned that “falling in love with a tool” can get in the way of asking the right questions. Another participant quoted a colleague who told participants at a recent conference that “just because you have optogenetics does not mean you can turn your own brain off.” It is not always the case, that participant continued, that the best way to probe the function of a circuit is through an inducible knock out. Sometimes an “old-fashioned” pharmacological agent or antidromic activation is a better way. The goal, said the participant, is to get students to really think about what the question is and to have a broad enough perspective to say what the right technique is for that question. Marder added that trainees should ask themselves, how do I design experiments to capture the data needed to inform my understanding? What are the outcome measures needed? One way to encourage thinking of the right experimental design questions was offered by Indira Raman, professor of neurobiology and physiology at Northwestern University. She described a class she teaches in which students discuss classic neuroscience papers to get a sense of
the history of experimental design and how people express ideas. Marie-Francoise Chesselet suggested that rather than offering a single course on experimental design graduate programs should incorporate discussions of best practices in design, as well as statistics and ethics, into every neuroscience course, regardless of the topic, thereby ensuring ongoing learning that can be applied in a variety of contexts.
Marder further emphasized the importance of theory in experimental design. She pointed to a directive of the BRAIN Initiative, which states that experiments of the future need to be an interaction between theory, modeling, computation, and statistics. The BRAIN 2025 report also lists the following three important outputs of using theory to support experimental design (NIH, 2014):
- Predictions: “Theoretical studies will allow experimenters to check the rigor and robustness of new conceptualizations and to identify distinctive predictions of competing ideas to help direct further experiments.” (p. 90)
- Integration: “Theory and modeling should be woven into successive stages of ongoing experiments, enabling bridges to be built from single cells to connectivity, population dynamics, and behavior.” (p. 7)
- Multiscale models: “New analytic and computational methods are required to understand how behavior emerges from signaling events at the molecular, cellular, and circuit levels.” (p. 90)
Marder concluded by noting that no matter how well students are trained, graduate programs have difficulty today in training students to design the experiments they are going to be doing 20 or 30 years from now. The only way for scientists to stay relevant is to build on a base of common sense and intuition and continually develop new skills and knowledge throughout their careers.
DEFINING THE GAPS IN THE TRAINING OF QUANTITATIVE SKILLS
At the heart of scientists’ ability to determine whether models accurately and reliably describe data and the inferences that can be made from data, Brown said, is statistical reasoning, which itself is derived from a deep understanding of probability. However, he noted that most
people lack intuition about statistics and probability. Such intuition takes longer to develop than the single graduate-level statistics course that most students take. According to Brown, developing intuition about probability should begin in elementary school and develop throughout the student’s education. The current Common Core State Standards used for developing U.S. math curricula2 do not expressly address training in probability understanding. However, Brown suggested that teachers could incorporate training in probability into existing lessons.
Due to inadequate training in statistics, said Brown, students often consider what analyses they will employ on their data after the data have been collected. The result can be a study published with insufficient statistical power to properly test a hypothesis, unfortunately an all-too-common problem in neuroscience, especially human fMRI studies (Button et al., 2013). Another common pitfall, occurring in a significant number of neuroscience journal articles, is failing to account for the clustering, or dependency, of data from nearby or otherwise similar neurons, an error that produces false-positive results (Aarts et al., 2014). In addition to enhancing overall training in statistics, Brown suggested that graduate departments develop unique statistics modules tailored to five or so key subdisciplines of neuroscience. For example, electrophysiologists could learn techniques for decoding spike trains while students working with fMRI could focus on techniques for calculating spatial correlations in images. Bringing like-minded students together to focus on a specific set of analytical skills might enhance their training and sense of community, said Brown.
The NIH BRAIN Initiative Working Group, of which Brown was also a member, formalized objectives and goals focused on improving quantitative expertise at all levels—faculty, postdoctoral, and graduate student. As laid out in the BRAIN 2025 report, the goals are to (1) ensure that all neuroscience postdoctoral fellows and graduate students become proficient with basic statistical reasoning and methods; (2) ensure that trainees are able to analyze data at an appropriate level of sophistication, for example, by writing code; and (3) encourage trainees to construct models as a way to generate ideas and hypotheses or to explore the logic of their thinking.
Finally, Brown said that in addition to taking steps to improve the average trainee’s skills in statistical reasoning, the field of neuroscience should examine how to increase collaborations with expert statisticians.
Engineers, physicists, and computer scientists have been increasingly working in neuroscience laboratories, but not statisticians. Bringing this important expertise to neuroscience is critical, said Brown. A recent white paper by an American Statistical Association working group offered a handful of suggestions to encourage the successful integration of statisticians into neuroscience training programs as it relates to the BRAIN Initiative (American Statistical Association, 2014). The authors note that statistician trainees should be
- taught to design data collection and analysis strategies,
- required to “take neuroscience classes and embed themselves in neuroscience labs,” (p. 6)
- held to the same writing standards as neuroscience graduate students, and
- “educated in principles of ethical and effective collaborative behavior” (p. 6).