Moving Toward Broader Sex-Specific Reporting

During his tenure at the New England Journal of Medicine, Kassirer said that many policy decisions were made and implemented, for example, the Ingelfinger Rule regarding duplicate publication and conflict-of-interest policies. Researchers are eager to have their papers published in high-profile journals, such as the New England Journal of Medicine, JAMA, and Annals of Internal Medicine. As a result, editorial policies implemented by those journals can be effective in modifying behavior. If the major clinical journals banded together to promote policies that foster sex-specific reporting, it seems reasonable to assume that investigators would take notice. Kassirer said that there is no bar to journals’ introducing policies that require sex-specific reporting, as was done at JNCI.

Davidoff pointed out that until about 10–15 years ago, nearly all clinical-journal editors were men. In recent years, however, four of the five major general clinical journals have had female editors (today, two of the five have female editors). That is an important change, he said.

Legato suggested that editors put out a request for papers that directly address sex differences, perhaps for publication in a supplemental issue. Robert Golub, deputy editor of JAMA, said that his journal publishes theme issues on topics that are of immediate relevance, and sex differences could be considered as one of those topics. The goal of a theme issue is to highlight research or current thinking in a field. Floyd Bloom, former editor-in-chief of Science, supported the idea of editors’ commissioning special issues on themes as a way to stimulate sex-specific science that could then be appropriately reported in journals; he noted that this has been successful in the neurosciences.

IMPLICATIONS FOR JOURNALS OF SEX-SPECIFIC REPORTING POLICIES OF JOURNALS

Nonclinical Science Journals

In basic-science research, it is very rare that male and female animals are studied side by side, said Levine. Even more problematic is the fact that male animals are often the default choice to avoid confounding effects of the estrous cycle or other female-specific physiologic circumstances. Levine suggested that basic-science journals consider a policy whereby authors are asked to justify the use of males vs females and



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 26
26 SEX-SPECIFIC REPORTING OF SCIENTIFIC RESEARCH Moving Toward Broader Sex-Specific Reporting During his tenure at the New England Journal of Medicine, Kassirer said that many policy decisions were made and implemented, for example, the Ingelfinger Rule regarding duplicate publication and conflict-of-interest policies. Researchers are eager to have their papers published in high-profile journals, such as the New England Journal of Medicine, JAMA, and Annals of Internal Medicine. As a result, editorial policies implemented by those journals can be effective in modifying behavior. If the major clinical journals banded together to promote poli- cies that foster sex-specific reporting, it seems reasonable to assume that investigators would take notice. Kassirer said that there is no bar to jour- nals’ introducing policies that require sex-specific reporting, as was done at JNCI. Davidoff pointed out that until about 10–15 years ago, nearly all clinical-journal editors were men. In recent years, however, four of the five major general clinical journals have had female editors (today, two of the five have female editors). That is an important change, he said. Legato suggested that editors put out a request for papers that di- rectly address sex differences, perhaps for publication in a supplemental issue. Robert Golub, deputy editor of JAMA, said that his journal pub- lishes theme issues on topics that are of immediate relevance, and sex differences could be considered as one of those topics. The goal of a theme issue is to highlight research or current thinking in a field. Floyd Bloom, former editor-in-chief of Science, supported the idea of editors’ commissioning special issues on themes as a way to stimulate sex- specific science that could then be appropriately reported in journals; he noted that this has been successful in the neurosciences. IMPLICATIONS FOR JOURNALS OF SEX-SPECIFIC REPORTING POLICIES OF JOURNALS Nonclinical Science Journals In basic-science research, it is very rare that male and female an- imals are studied side by side, said Levine. Even more problematic is the fact that male animals are often the default choice to avoid confounding effects of the estrous cycle or other female-specific physiologic circum- stances. Levine suggested that basic-science journals consider a policy whereby authors are asked to justify the use of males vs females and

OCR for page 26
WORKSHOP SUMMARY 27 have reviewers check to see whether authors have considered the impli- cations of their choice of males vs females in the design, conduct, and interpretation of their experiments. Blaustein supported the idea of including the justification of the sex of animals as one of the items that reviewers and associate editors should look for, noting that it would help to increase awareness. He added that in many cases, the default choice is actually immature animals because researchers think that this avoids the effects of the estrous cycle and mix immature males and females. Blaustein cited an article on sex bias in biomedicine by Zucker and Beery (2010, p. 690), who suggested that to correct the sex bias in animal research, we need stringent, strictly enforced measures, not voluntary appeals. Journal edi- tors and reviewers should require authors of research studies that use only male or only female animals to state this in the ti- tle of their papers. This would highlight sex biases and spur researchers to balance the numbers of males and females that they use. Funding agencies should refuse to consider grant proposals that do not properly acknowledge the sex of the an- imals to used, and favor those that include males and females, and analyze data by sex. Guidance on conducting sex-differences research is available, Blaustein noted. For example, an article that was the product of a series of discussions sponsored by the Society for Women’s Health Research outlined strategies and methods for research on sex differences in behav- ior and the brain (Becker et al., 2005). The article was followed by a book, Sex Differences in the Brain, that discusses the subject in more depth (Becker et al., 2008). Guidance on designing and analyzing exper- iments to consider sex differences can be provided, Blaustein said, but that will not force researchers to look at both sexes. Interdisciplinary Science Journals Katrina Kelner, editor of the new journal Science Translational Medicine and former deputy editor for biology at Science, said that the issue of sex-specific reporting “is not on the radar screen” of the interdis- ciplinary journals (such as Proceedings of the National Academy of Sci- ences, Science, and Nature). Major issues that editors at Science have been working to address over the last 10 years include image manipula- tion, conflict of interest, and availability of the data and materials for the

OCR for page 26
28 SEX-SPECIFIC REPORTING OF SCIENTIFIC RESEARCH analysis being reported, whether climate change, physics, biomedicine, or another discipline. A challenge for any journal is to decide how stringently to en- force an editorial policy, Kelner said. High-profile journals must balance the large volume of manuscripts submitted against the limited time of reviewers and staff. A particular challenge for editors at an interdiscipli- nary journal is that one’s expertise in any particular topic is limited. Sci- ence has around 20 editors, about half of whom are physical scientists and half of whom are molecular biologists. If Science adopted the sex- specific reporting recommendation of the 2010 IOM report Women’s Health Research: Progress, Pitfalls, and Promise, the journal would have to rely, to some extent, on outside reviewers, Kelner said. There are limitations in relying on outside reviewers to ensure that appropriate analysis has been done. It can be difficult to find suitable reviewers— most are quite busy—and, because papers can be heterogeneous, editors do not always get the needed advice from a review. Science Translational Medicine has adopted some of the ICMJE recommendations—including the institutional review board, authorship, and clinical-trial registration requirements—but has not adopted the rec- ommendation about sex-specific reporting (although Kelner noted that she intends to revisit the issue after the workshop). To aid editors of interdisciplinary journals, Kelner suggested that a role for the IOM and the editorial organizations could be to ensure that educational resources are available to editors, including information that they can refer to when assessing sex-specific reporting in papers. Medical Journals Study Design, Analysis, and Reporting Golub shared the results of his informal audit of the 50 most recent randomized controlled trials published in JAMA. Of the 50, 21 reported single-sex results (on sex-specific topics) or presented results analyzed and stratified by sex. Of the 29 studies that did not report re- sults by sex, 19 included at least 40% women, 22 at least 30% women, and 27 at least 20% women. Golub opined that only one of the 29 studies was possibly adequately powered to do subgroup analyses; he reiter- ated that statistical power is a recurring problem. Adler suggested that having data similar to what Golub presented published annually would be helpful.

OCR for page 26
WORKSHOP SUMMARY 29 The issue of reporting results by sex is related not simply to the percentage of female participants in studies but to fundamental questions of adequate power and of whether subgroup analysis is appropriate. The harm in separating males and females in the absence of sufficient statisti- cal power is the risk of errors, such as a type II error in publishing com- parisons by sex that show no significant difference. Negative results do not necessarily mean that there is no difference. Women’s Health Research states that “in the absence of a com- pelling reason not to, it should be assumed that there are sex differences in conditions” (IOM, 2010, p. 233). If the intent is to report raw data by sex for future research, that assumption is not likely to be an issue. For analysis of data by sex, however, Golub said that unless there is a priori evidence to support the assumption, the analysis may be flawed, and a Bayesian statistical approach may lead to false associations. In other words, a starting assumption that there is a sex difference for any associ- ation being studied could lead to the publication of false conclusions. Gregory Curfman, executive editor the New England Journal of Medicine, concurred that the real challenge in reporting clinical-trial re- sults separately for males and females is whether there is adequate statis- tical power for subgroup analysis. The rationale for reporting data by sex is indisputable, he said, but if a clinical trial has not been adequately powered to look at males and females separately, the conclusions are not going to be statistically sound. In many cases, achieving statistical signif- icance for subgroup analyses would require unattainable or unjustifiable numbers of participants. Curfman therefore cautioned against editorial policies that require trials to be designed to reach valid statistical conclu- sions for males and females separately. Such editorial policies would create a “steep mountain to climb for investigators and for funding agen- cies,” he said. In addition, as has been discussed, it may not be enough to report results only by sex. Other important demographics, such as race and age, add to the complexity of reporting results and of sample-size calculations for large clinical trials. Curfman also raised a concern that reporting results by sex in studies underpowered for valid subgroup anal- ysis may be misleading and may be subject to misinterpretation by the health care community. Science and Science Translational Medicine have a strong em- phasis on data access, and Kelner endorsed the idea that even if a study has not been properly powered for male and female subgroup analysis, the resulting data should still be made available for others, perhaps in an appendix (with the appropriate warning that, because of statistical limita- tions, sex-specific data do not necessarily indicate a sex-specific differ-

OCR for page 26
30 SEX-SPECIFIC REPORTING OF SCIENTIFIC RESEARCH ence). Ensuring that all data are available to everyone once a study report is published is an important role for journals, Kelner said. Berlin added that appropriate study design and use of common definitions would im- prove the ability to conduct meta-analyses of such archived data. Current NIH policy on the participation of women in clinical tri- als has not achieved the desired effect of having enough women enrolled in all studies for sufficient statistical power, Golub said. And statistical power is an ever-increasing problem. For example, improvements in standards of care mean that control groups have better outcomes and that more participants must be enrolled to find a significant difference be- tween control and test treatments. In some studies, composite outcomes (such as major adverse cardiac events) are already necessary for achiev- ing adequate power, and this limits the possibility of valid subgroup analysis on individual outcomes. Composite outcomes can hide im- portant effects, Golub noted, and often the events that are driving a dif- ference are the less important events, such as rehospitalization, whereas the number of deaths may not be statistically different or may be under- powered for analysis. Given those challenges, is it feasible to have enough women enrolled to facilitate statistical analyses of the sexes sepa- rately? Will a Bayesian statistical trial design enable studies with a lower aggregate enrollment? Once the primary hypothesis and primary out- comes are established, the necessary enrollment is determined on the ba- sis of power analysis and expected attrition. Adding more patients is expensive, Golub said, and researchers cannot simply over-enroll in an- ticipation of unplanned subgroup analyses. Blaustein questioned the extent to which the number of partici- pants would need to be increased to see an effect of sex. In animal stud- ies, he noted, sex differences are often apparent even in small studies. Kassirer responded that as the difference in effect between two therapies becomes smaller and smaller, larger and larger studies are needed to be able to identify it. There are differences between animal studies and hu- man studies and between different types of human studies, Golub added. For example, in clinical trials with major cardiac outcomes, severe events, including death, have become much more rare, and the effect size that the investigator is willing to accept may be fairly small compared with a more basic study in which one would be able to detect effect dif- ferences with 10 subjects in each group. Sufficient power for subgroup analysis may be easier to achieve in some types of studies and harder to achieve in large clinical trials with rare outcome events. A participant raised a concern about the reliance on a statistical p value of 0.05 for accepting or rejecting a hypothesis. In some cases, a

OCR for page 26
WORKSHOP SUMMARY 31 finer p value, such as 0.000001, might be appropriate, and in others, a p value of 0.20 might be acceptable. It may be necessary sometimes to al- low more leeway in interpretation (which, it was noted, is not the same as exploratory data analysis). Christine Laine, editor-in-chief of Annals of Internal Medicine, said that publishing a large trial without sex-specific results does not necessarily mean that there are no sex-specific results; when discussing the limitations of a study, authors should point out that there could be sex differences. Absence of evidence is not evidence of absence, Davidoff agreed. He added that statisticians are often involved in clinical trials at a late stage and probably should be engaged much sooner. One of the quandaries that an editor faces, Golub said, is the need to cater to two different readerships: researchers and clinicians. Re- searchers are interested in hypothesis findings to inform the design of their next study, but the mission of medical journals is to publish studies that will affect clinical care, and it is increasingly difficult for clinical readers to understand the articles. New methods, including Bayesian analysis and adaptive clinical trials, will allow studies to be done without the need to anticipate everything from the start. The downside is that edi- tors will need to re-educate themselves so that they can understand the methods. More statisticians will be required as it becomes more difficult to find reviewers who can review the statistical design of a study. And again, there is the issue of whether readers will be able to understand the reports. A dilemma in drug development, said Marietta Anthony, director of women’s health programs at the Critical Path Institute, is that clinical studies are powered to show that a therapy is safe and effective in gen- eral, not necessarily safe and effective specifically in females or males. However, FDA guidelines mandate that medical products be demonstrat- ed to be safe and effective in the populations that will use them, so there is some question of interpretation. Parekh explained that FDA guidance documents for preclinical animal studies recommend studying both male and female animals. Phase 1 clinical studies look for safety and pharma- cokinetic differences among subpopulations of healthy volunteers, in- cluding women and men. If there are significant differences, they are monitored carefully in early phase 2 safety and efficacy studies. Early phase 2 studies in patients almost always include women. Phase 3 clini- cal trials are hypothesis-driven studies with prospectively defined end points and involve large numbers of patients. Sometimes the overall data will show no effect of a product but subgroup analysis reveals a signifi- cant effect in a particular population. Such a finding is hypothesis-

OCR for page 26
32 SEX-SPECIFIC REPORTING OF SCIENTIFIC RESEARCH generating, Parekh said, and another prospectively designed trial is then conducted to confirm the finding. Exploratory Subgroup Analyses Kramer said that in his experience, the issue with subgroup analyses in observational studies is not getting authors to do sex-specific analyses but preventing overinterpretation of results of exploratory analyses. Exploratory analyses—whether they are sex-specific or specific for comorbidity, age, race, or socioeconomic sta- tus—are different from prospective primary end-point analyses that are protected by randomization. JNCI does not publish post hoc analyses or exploratory analyses in article abstracts, and such analyses are clearly labeled in the results. Golub concurred in the need to exercise caution on exploratory analyses. When the primary outcome of a study is negative, authors will often stress a positive secondary or post hoc finding. Identifying and ad- dressing it is part of the peer-review process. Like JNCI, JAMA does not include exploratory analyses in abstracts and requires that they be clearly described as exploratory in the text. Golub noted that there is a difference between a prespecified secondary analysis and a post hoc exploratory analysis. Sex Subgroups Sex hormones influence virtually all cells, and stage of reproductive life and development should be considered in designing and reporting studies, Blaustein said. For instance, children are not the same as adults, and estrous-cycling females are not the same as acyclic fe- males. He acknowledged that considering such factors requires some knowledge of reproductive endocrinology or the involvement of a repro- ductive endocrinologist, which is unlikely in most cases. However, he pointed out that it is important to begin thinking about variations within the sexes. Influence of Journal Editorial Policies ICMJE and journal policies can influence researcher behavior. For example, Golub said, it is now rare to receive a report of a clinical trial that was not registered in a public database in accordance with the ICMJE policy, and all submissions are accompanied by author disclosure forms. It is important to note, however, that although a trial must be reg- istered before the study begins enrollment, most policies (for example, requiring financial disclosure forms and race and ethnicity statements), unlike a sex-specific reporting policy, require little of the authors until

OCR for page 26
WORKSHOP SUMMARY 33 the study is complete. Even though the trial-registration policy changed researcher behavior, its implementation was straightforward and did not require any major changes in study design or any substantial investment of resources. Kramer added that journals can assert important influence on the quality of reporting. For example, journal editors have become more attuned to looking for discrepancies between a primary end point in the original study as designed and a primary end point that the authors discuss in their paper. Golub questioned the extent to which an editorial policy could affect study design relative to sex-specific reporting, in light of the study- design issues involved (such as adequate power). Changing editorial pol- icy might send a signal to researchers who are beginning to design a study, but would they be able to make the changes needed and design studies with adequate power? Larger studies would require additional funding, which may not be available. Changing standards and publishing analyses that are not likely to be valid is not a good solution either. It comes down to the design of the studies, and, inasmuch as study design has not changed substantially in the last 10 years, it is not clear what would make it change now. If editorial policies required sex-based anal- ysis, would the funding follow? As noted in Women’s Health Research (IOM, 2010), not consid- ering sex and gender differences in the design, analysis, and reporting of studies has limited understanding of important sex differences and slowed progress in women’s health. Laine stressed that problems in the design and analysis of a study cannot be fixed simply by changing re- porting requirements. Journal editors can ask authors to reanalyze their data but cannot ask them to redesign their studies and redo them. Jour- nals could reject papers that do not report sex-specific results, but that is unlikely to happen. Journals do not provide research funding, and Blaustein suggest- ed that changes in how experiments are done start with funding agencies. Kelner concurred that funding agencies need to be partners in encourag- ing good research practices. There needs to be a culture shift within sci- ence. Questions about what can be accomplished by editors and publishers through setting standards for authors, whether these be rec- ommendations or mandates, versus the role of federal agencies and other funders in shaping research culture to embrace consideration of sex dif- ferences as part of sound study design, were raised in a number of com- ments by participants. As discussed earlier, the ICMJE policy is specific to medical journals. Laine listed several other editorial associations and published

OCR for page 26
34 SEX-SPECIFIC REPORTING OF SCIENTIFIC RESEARCH guidelines that do not address sex-specific reporting at all, including the Council of Science Editors (which covers science broadly, not only bio- logic science but the physical sciences), the World Association of Medi- cal Editors, the Guideline on Good Publication Practice of the pharmaceutical industry, and the EQUATOR (Enhancing the QUAlity and Transparency Of health Research) network, an umbrella organization that catalogs numerous reporting guidelines, such as CONSORT (Con- solidated Standards of Reporting Trails) for randomized trials and STROBE (STrengthening the Reporting of OBservational studies in Epi- demiology) for observational studies. (The CONSORT and STROBE guidelines cover the majority of the clinical research published in the major journals.) Annals of Internal Medicine does not have a specific policy on sex-specific reporting but follows the ICMJE policy, Laine said, and en- courages authors to follow reporting guidelines, including CONSORT and STROBE. She added that for many years, Annals has indicated in the title and abstract when a study includes only men or only women and indicates in the limitations of the study if data are insufficient to examine potentially relevant sex differences or racial or ethnic differences. Annals does not ask authors to report sex-specific results when the study design is insufficient to enable useful reporting of such results. A journal can put a policy into place, but there has to be a way to implement it. Laine offered clinical-trial registration as a case example: the ICMJE put its registration policy into place before there was any- where for sponsors to register their trials. Similarly, journals could re- quire that studies be powered for subgroup analysis, but that would entail the availability of resources to fund those types of studies. It will not work if the funders, researchers, and journal editors are not aligned. Edi- tors can foster more accurate reporting, but must be careful about making requirements that are not feasible, Laine cautioned. As data-sharing ad- vances, journals may be different a decade from now, and researchers whose studies do not meet some set criteria may move away from tradi- tional journals and publish their results in an open-access setting. Berlin added that a motivation for trial registration was to eliminate publication bias—to make all results available regardless of whether they are posi- tive or negative. He cautioned that a situation in which only studies with sex-specific results are published is not desirable. Another issue is the long pipeline of current high-quality clinical trials, many with long-term followup. These will be coming to comple- tion over the next decade or later, and panelists discussed how any edito- rial policy that affects study design would need to be phased in over an

OCR for page 26
WORKSHOP SUMMARY 35 extended period. In the interim, Golub said, it is unlikely that journals would forgo publishing a well-done, informative study that could affect patient care solely because it lacked enough power to permit valid sex- specific reporting. Conversely, journals are not willing to publish poor- quality studies or invalid or meaningless data or analyses. As discussed earlier, the present workshop was designed to con- sider a recommendation in Women’s Health Research (IOM, 2010) that “the International Committee of Medical Journal Editors and other edi- tors of relevant journals should adopt a guideline that all papers reporting the outcomes of clinical trials report on men and women separately un- less a trial is of a sex-specific condition” (IOM, 2010, p. 13). Laine raised several concerns about editorial policies that might be developed on the basis of that recommendation. First, it appears that observational studies are not included. Second, as discussed above, many trials include insufficient numbers of women or men to allow valid comparisons or within-group conclusions. Third, if randomization was not stratified by sex, the results should not be interpreted as causal relationships. Finally, simply reporting sex-specific results does not address the question of whether any of the observed sex differences are due to sex or to con- founding factors. Suggestions from the Editors On the basis of their experiences in implementing editorial poli- cies, the panelists offered a variety of suggestions regarding the inclusion of sex-specific information in scientific publications (summarized in Box 2). It was also suggested that the ICMJE consider adopting a strong- er sex-specific analysis and reporting statement similar to that of JNCI. Laine predicted, however, that ICMJE members would question why only sex was being addressed and not other key factors, such as age, race, ethnicity, and insurance coverage.