and the development of new questions and new studies. Scientists assemble and weigh evidence in its totality. This weighing requires an understanding of how to assess the strength of experimental design and execution, the strength of statistical methods and results, and the importance of multiple sources of related evidence.
What are data? One way to imagine the scientific method, said Dr. Goodman, is to visualize a single scientist toiling with a few students at the laboratory bench. Once a pertinent question has been studied, the activities, observations, calculations, and conclusions of such a scientist would be assembled, distilled, and published. Other researchers interested in the topic then would use the publication and perhaps ask for some of the scientist’s original materials to further their own studies of the topic. Bench scientists understand that if they do not report accurately and honestly their methods, results, and conclusions, their reputation within the scientific community could be jeopardized. This reality has always been a powerful force for integrity.
That information, as data, may then move through many levels during preparation of a study report: raw data, abstracted data, coded data, computerized data, cleaned or edited data, analyzable data, and, finally, analyzed data. It is important to realize that as data flows from one level to the next, researchers often have to evaluate or “clean” the particular items of data. For example, cleaned data often must be purged of “outlier” data that are interpreted as unlikely to be accurate or likely to distort the results. Only when the data are cleaned or edited are they brought into the analysis module of a statistical program and organized. This module typically is thus a very small subset of the total data, and it is organized to focus on one particular question. In complicated data sets, many questions are typically asked of the data over some years by multiple investigators. It is these analyzed data that appear in published form, highly compressed and processed, and often presented in graphs or tables. Although many choices are made in deciding which data enter the final analysis, these decisions are determined by trained scientists who have spent years understanding the limits of their methods and deciphering the particular research question.
What is peer review? One way to answer this question is by discerning what peer review is not. Peer review does not detect fraud, validate factual findings, dictate publication decisions, or substitute for the judgments of the scientific community as a whole. What it does do is provide a mechanism of independent outside advice to a journal editor about the importance of a paper’s findings, its strengths and weaknesses, and any modifications necessary to make the author’s claims match the strength of the reported evidence. Since it is those claims that are often what the