the trial, so the burden of reporting to would be due mainly to data entry. Instead, the experience at has shown that protocols are often vague, are not always followed, or in some cases may not even exist. In addition, summary data are not always readily available even for trials that have already been published. For many trials, no one can explain the structure of the trial or the analysis of the data, said Zarin. “What we learned is there is not an objective, easy-to-describe route from the initial participant-level data to the summary data. Many people and many judgments are involved.”

Structural changes to trials are also common. A trial can start as a two-arm study and then become a four-arm study. Participants come and go, so that the number of participants changes over time. Participant flow and baseline characteristic tables describe different populations than the outcomes table. Data providers often cannot explain the “denominators” for their results, the groups from which outcomes or adverse events are collected. Zarin described a study in which a year of close work was required with statisticians to figure out who the people in the study were and where they went as a result of structural changes to the study. “These are brilliant statisticians. They were in charge of the data. [But] this trial was basically too complicated for them to figure out. They were giving outcome measures without actually knowing what the denominators were. That is one kind of problem we have seen.”

In other cases, outcome measures were changed: a quality-of-life scale was replaced with a depression scale; 1-month data were replaced with 3-month data; the number of people with an event was replaced with time to an event; and all-cause mortality was replaced with time to relapse. Sometimes discrepancies are obvious. In one study, the mean for hours of sleep per day was listed as 823.32 hours. Another study of 14 people included data on 36 eyeballs. “As a consumer of the medical literature, these are not reassuring things,” Zarin observed.

In a study of 100 matched pairs of results and publication results, 82 percent had at least one important discrepancy. The inevitable conclusion is that summary data may not always be an accurate reflection of participant-level data. Although the deposition of clinical trial protocols and summary data into registries is a huge step forward in the direction of transparency, the validity and reproducibility of summary data are called into question by such inconsistencies. “This is a big problem,” Zarin asserted.

Providing more transparency about the process of converting one type of data into another type would help inspire trust, she said. Docu-

The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement