Methodologic Issues in Women’s Health Research
In reviewing and evaluating research on women’s health, the committee considered not only conditions1 and health determinants but also the types of research conducted. This chapter addresses methodologic issues with respect to women’s health, looking at study design, subject sampling, outcome measures, and analysis. The Women’s Health Initiative (WHI) is then discussed as an example of what has been learned about methods of women’s health research from the studies already conducted. The information in this chapter helps the committee address question 4 from Box 1-4, whether the most appropriate research methods are being used to study women’s health.
Research can be conducted on molecules, cells, and animals (basic research); on individuals or populations (clinical or observational studies); and on health systems (health-services and health-policy research). Each type of study has strengths and weaknesses, and progress generally requires congruence of evidence from multiple studies of different designs. For example, progress in breast and cervical cancer came through basic and experimental clinical research and other epidemiologic studies that provided support for similar conclusions (see Chapters 2 and 3).
Two major types of clinical studies are observational studies and clinical trials. An observational study is a study in which investigators do not manipulate the use of or deliver an intervention (that is, they do not assign subjects to treatment
and control groups) but only observe and measure outcomes in subjects who are (or are not, in the case of a comparison group) exposed to an intervention (for example, a smoking ban that decreases secondhand smoke exposure) (Rosenbaum, 2002). Such studies have less control of potential confounders than do experimental studies, such as randomized controlled trials, and are more prone to selection bias and to bias in the choice of comparison populations. Observational studies provide information for identifying associations and are especially useful for generating hypotheses for further testing; they are less useful for determining causality. The Nurses’ Health Study (NHS) and the Study of Women’s Health Across the Nation (SWAN) are examples of large observational studies. The NHS was originally intended to investigate the potential long-term consequences of oral contraceptives and later adapted to investigate factors that influence women’s health, especially in preventing cancer (NHS, 2008). SWAN was designed to collect information on the natural history of menopause (SWAN, 2010).
A randomized clinical trial is a prospective experiment in which investigators assign an eligible sample of people randomly to one or more treatment groups and a control group and follow subjects’ outcomes. Randomized clinical trials are usually considered the best for testing the efficacy of a treatment or intervention (Rosenbaum, 2002). The Women’s Health Study (WHS) was a randomized clinical trial in which the interventions were aspirin and vitamin E (WHS, 2009). The WHI consisted of both an observational study and three blinded, randomized clinical trial components that had hormone therapy, calcium and vitamin D, or dietary and exercise modification as the interventions (WHI, 2010). The randomized clinical trial in the WHI identified a risk of heart disease to be associated with combination estrogen hormone therapy, which was previously thought to be cardioprotective, and it confirmed the risk of breast cancer, venous thromboembolism, and stroke. Confidence in those results facilitated a decision to halt the study and led to a rapid change in prescribing practices (WHI, 2010).
Randomized clinical trials, however, have limitations of their own. Because of the expense and number of subjects needed to assess a given drug or other treatment, it is not possible to change key variables. The ability to extrapolate the results to a larger population might also be limited (Rosenbaum, 2002). In addition, ethical and practical considerations of the studies, including the ethics of placebo controls, need to be taken into account.
If study results are to be extrapolated to the general population, the research sample needs to reflect the general population. Ensuring that research can be applied to the general population requires more than simply incorporating members of a subpopulation as part of the overall sample; it requires adequate numbers to ensure the statistical power to evaluate effects in that subpopulation. It is important to note that using gender or sex as a control variable is not the same as
examining the effects of gender or sex on a given outcome. Thus, the issue is not simply including women in trials but including sufficient numbers to test effects on both women and men. To be fully informative, findings need to be reported separately by sex or gender. If a subpopulation, such as women in this case, is excluded or underrepresented in the sample, it is difficult to know whether the results will apply to the subpopulation or whether it would have responded differently. For example, that lack of data can delay translation of research findings on effective treatments to the excluded or underrepresented subpopulation or can lead to adverse outcomes because of inappropriate application to one population of treatments developed on another.
It might seem obvious that poor clinical outcomes can occur if it is presumed that there are no sex or gender differences when they do exist, but false inferences and bad outcomes can also result from a presumption of sex or gender differences when such differences do not exist (Baumeister, 1988). For example, the first randomized clinical trial of estrogen therapy, the Coronary Drug Project, was done in men. That study was discontinued prematurely because of a lack of evidence of a positive effect and a trend toward increased cardiovascular mortality in the treated group (The Coronary Drug Project Research Group, 1973). The doses in the trial were much higher than those given to women, so the results were thought not to be relevant to women, and estrogen therapy continued to be prescribed to women to reduce cardiovascular risks. More than 20 years after the study, postmenopausal hormones were still among the top-selling drugs in the United States—an estimated 15 million women were taking them (Hersh et al., 2004). Conversely, statins (that is, 3-hydroxy-3-methylglutaryl coenzyme A reductase inhibitors) were first shown to be effective in lowering cholesterol in a Scottish trial in men (Shepherd et al., 1995). Of 5 randomized controlled trials published in 1994–1998, the Scottish trial was in men only, and the other 4 included 14–19% women. That small number of women in statin trials limited conclusions for women and led to questions about the extrapolation of the data to women. LaRosa and colleagues (1999) conducted a meta-analysis of data from those trials and concluded that the risk reduction from statins is similar in men and women.2 Meta-analyses, however, are not optimal, especially when evaluating the leading cause of mortality in women, and more recently the efficacy and safety of statins, especially for primary prevention, has been questioned and is still being evaluated (Abramson and Wright, 2007; Mascitelli and Pezzetta, 2007; McPherson and Kavaslar, 2007; Ridker, 2010).
Before 1987, women were underrepresented in key randomized controlled trials because of policies that limited or prevented their participation mainly owing to concern about potential exposures of fetuses. Changes in National In-
stitutes of Health (NIH) and Food and Drug Administration (FDA) regulations and policies, starting in 1987, addressed that underrepresentation and aimed to increase the enrollment of women and analysis of data on women in clinical trials (GAO, 1992, 2000; Merkatz and Junod, 1994). Progress has been made since then in increasing enrollment of women. Women made up 51.7% and 60.0% of participants in NIH extramural and intramural clinical research in 1995 and 2008, respectively (HHS, 2009). The highest percentage, 64.2%, was seen in 2002; that corresponds to when large women-only studies related to breast cancer, menopause, and cardiovascular diseases (the WHS, the WHI, and SWAN) were conducted (HHS, 2009). The sex distribution in all the minority-group participants in 2008 was similar to that in the nonminority population; women made up 59.15% of minority-group participants (HHS, 2009).3
Data from clinical trials that looked at specific end points have provided additional insight into the participation of women and minorities. A recent analysis of FDA clinical trials found that although the number of trials enrolling women and the proportion of participants who are female participating in phase I trials have increased since 2001, women are still underrepresented (Pinnow et al., 2009). Stewart and colleagues (2007) found higher enrollments of women than of men in their study of cooperative group surgical oncology trials, primarily because of the large number of breast-cancer trials. Members of racial and ethnic minorities and older persons were less likely to be enrolled in the trials than were whites and younger subjects.
Human immunodeficiency virus (HIV) research historically has had low representation of women. Of the women who were eligible to participate in the largest cohort study of HIV-infected women in the United States (the Women’s Interagency HIV Study), about half would have been ineligible, on the basis of exclusion criteria, to participate in 20 of the AIDS Clinical Trials Group studies, which are among the largest HIV clinical-trial groups in the United States (Gandhi et al., 2005). Those results are consistent with an earlier meta-analysis published in abstract form that found that in 49 randomized controlled trials of antiretrovirals in 1990–2000, women averaged only 12.25% of the participants (Pardo et al., 2002).
In studies of cardiovascular disease, clinical-trial subjects have not been representative of the general population (Lloyd-Jones et al., 2001; Pedone and Lapane, 2003); one study discussed the predominance of men in cardiovascular trials (Sharpe, 2002). In 19 randomized controlled trials open to both men and
women that examined myocardial infarction, stroke, or death, the mean percentage of female subjects was only 27%.4 Only 13 of those studies presented sex-based analyses (Kim and Menon, 2009). A review of the literature by the Agency for Healthcare Research and Quality indicated that studies of coronary heart disease rarely included women in adequate numbers for analysis of the data by sex (Grady et al., 2003).
Part of the reason for the low participation of women is that many cardiovascular-disease clinical trials had inclusion criteria that were more appropriate for men than for women (Grady et al., 2003). Such inclusion criteria as early age of onset of myocardial infarction and chest pain as a presenting symptom will favor enrollment of men because women are on the average older at disease onset, are less likely than men to report chest pain during a heart attack, and are more likely to report other symptoms (Bairey Merz et al., 2006; Canto et al., 2007).
Even when women are included in clinical trials, having too few of them can be a barrier to obtaining statistically significant results related to women. Freedman and colleagues (1995) suggested conducting large clinical trials for conditions in which there is a priori evidence of sex differences. The NIH guidelines require inclusion of women and minorities in phase III clinical trials unless there is substantial evidence that sex differences do not exist (Bennett and the Board on Health Sciences Policy of the Institute of Medicine, 1993; Freedman et al., 1995). That implies that earlier research—including cells, animals, and phase I and II clinical trials—must have addressed potential sex differences sufficiently to support a choice not to include women in phase III clinical trials in numbers adequate for assessing sex differences. Underrepresentation of women in earlier phases could lead to interventions or treatments that are less effective or more toxic in women. For example, dose regimens are determined in phase I clinical trials, and conducting such studies mostly in men would result in drug doses based on male anatomy (Chen et al., 2000). Data indicate that women continue to be underrepresented in trials. For example, Jagsi (2009) found that women comprised only 38.8% of participants in non–sex-specific prospective clinical studies.
Because women might not be included in studies in adequate numbers to obtain a valid statistical analysis, another potential method of obtaining useful data on women is to perform meta-analysis of aggregated published data. That, however, requires that multiple studies be sufficiently similar in design (for example, having similar inclusion criteria and dosing regimens) and in clinical outcomes to be aggregated and also that the studies provide data on women separately (Berlin and Ellenberg, 2009). However, most clinical trials do not publish results on key subgroups of interest. Even when results on subgroups are published, the data
are typically presented in different ways among studies and difficult to combine in a meta-analysis (Berlin and Ellenberg, 2009).
Combining data on individual subjects from randomized trials is another approach for enhancing statistical power that increases the number of subjects available for analysis in clinical subgroups. Pooling of data on individual subjects overcomes the limitations of meta-analysis and allows use of more sensitive statistical methods, including analysis of survival times, multivariable models, and tests for treatment-by-covariate interactions (Samsa et al., 2005). It also enables the assessment of the combined effect of treatment for multiple end points—combining benefits and risks to capture net “value” (Antithrombotic Trialists Collaboration, 2009). However, the technique poses logistical challenges and requires collaboration among trial groups and support from funding agencies (Bravata et al., 2007).
A Bayesian approach could also be used to determine whether sex or gender differences exist, and that information could form the basis of further research. The Bayesian approach is an iterative one that adds more subjects from a subgroup on the basis of probabilities estimated from previous or preliminary results (Berry, 2006). In Bayesian analysis, the effect in a small number of women could be compared with the effect in a larger sample of men (or vice versa). If the distribution of results for several outcomes is the same between the sexes or genders, the study can proceed and continue to include a small number of women and to conduct periodic analyses to determine whether sex or gender differences are evident. If the distributions are different, the next phase of the clinical trial would incorporate larger numbers of women to assess the differences. This method could be applied to individual and pooled trials.
An alternative to executing clinical trials with women and men and analyzing sex- or gender-specific data is to conduct women-only studies, particularly in cases in which there are gaps in knowledge about women. That has been done in a few men-only studies that demonstrated the benefit of a drug. For example, the original Physicians Health Study was a randomized controlled trial in men that found that daily aspirin led to a significant reduction in myocardial infarction but not in cardiovascular death (Hennekens and Buring, 1989; Hennekens and Eberlein, 1985). It was not known whether the results would be the same in women. Later the women-only WHS had slightly different results: daily aspirin lowered the risk of stroke but did not affect the risk of myocardial infarction or cardiovascular death (Ridker et al., 2005). Sex differences were then examined in a study that pooled individual-level data from 6 primary-prevention randomized trials and 16 secondary-prevention randomized trials (looking for prevention after a coronary event), including both the sex-specific trials discussed above. No sex differences in the effect of aspirin on overall serious cardiovascular events were seen, and the risk of cardiovascular events was reduced in both men and women. However, there were slight sex differences in aspirin’s value in primary prevention (depending on the statistical analysis): less primary prevention of
major coronary events and more primary prevention of strokes were seen in women than in men. No sex differences were seen in secondary prevention of either end point—aspirin was protective in both sexes (Antithrombotic Trialists Collaboration, 2009). Overall, the authors concluded that aspirin is beneficial for protecting against secondary events in both women and men but that protection against primary events needs to be weighed against the risks posed by daily aspirin for both sexes.
Female-Appropriate End Points
Sex and gender differences need to be considered not only for inclusion and exclusion criteria but also when determining the end points to be studied. If study end points are based on male pathophysiology, clinical outcomes relevant to women will be missed. For example, women are more likely to have unstable angina (DeCara, 2003), unrecognized myocardial infarction (Sheifer et al., 2001), and stroke as cardiovascular outcomes than men (Lloyd-Jones et al., 2009; Tow-fighi et al., 2007). If a clinical trial looking at a cardiovascular-disease treatment assesses fatal and nonfatal myocardial infarction as its outcome but does not assess such events as unstable angina that are more common in women, it will underestimate the prevalence of cardiovascular disease in women and be biased against finding a treatment effect in women.
Quality of Life as an End Point
Incidence and 5-year survival rates are often the end points evaluated in clinical trials, including studies of women’s health; fewer studies assess morbidity or health-related quality-of-life (HRQoL) end points. The focus of research on mortality is also reflected in the relative lack of attention paid to conditions associated more closely with morbidity than with mortality, such as autoimmune disease, thyroid disease, and nonmalignant gynecologic disorders. Many of those chronic disabling disorders and depressive and anxiety disorders affect women more than men (Rieker and Bird, 2005), and women, when surveyed, generally report worse health than men even though men have shorter life expectancy and lower age of onset of such diseases as cardiovascular disease (Rieker and Bird, 2005). In addition, women rank quality-of-life end points high when considering what aspects of health matter to them, and this points to the need to assess HRQoL end points in women (Fryback et al., 2007).
One challenge in including HRQoL end points in studies is the need for consistent and accurate metrics for them. In particular for women’s health, including metrics that measure what matters to women is important. Metrics for HRQoL end points have been developed as interest in assessing them in observational
studies, clinical trials, and health-services research has increased. Measures of HRQoL end points can reflect specific symptoms, constellations of symptoms associated with specific conditions, or the combined effects on overall well-being that reflect symptoms that affect HRQoL (that is, quantify a global measure of quality of life) (Gold et al., 2002). HRQoL metrics, such as the SF-36, which is a short-form health survey with 36 questions, quantify quality of life in terms of domains (for example, physical, psychologic, economic, spiritual, and social) and allow comparisons among conditions, but some may lack sensitivity to sex- and gender-specific issues (such as menopause and premenstrual-syndrome symptoms), and some are affected by sex and gender (Fleishman and Lawrence, 2003). Research is beginning to identify and improve understanding of those differences to capture quality-of-life end points for women better (Fryback et al., 2007). Improved measures of HRQoL in women will help not only in assessing women’s health but in communicating risks and benefits associated with treatment and intervention options to women and to facilitate informed decision making for female patients—an important aspect of translating research into practice, which is discussed further in Chapter 5.
Including adequate numbers of women in clinical trials is necessary but not sufficient to ensure that results are applicable to women. Despite improved inclusion of women in trials funded by NIH and reviewed by FDA, there has been a lag in the routine analysis and reporting of data by sex (GAO, 2000, 2001). Often there is no mention of separate male and female analysis in publications, and it is not possible to know whether an analysis by sex was not conducted or was conducted but not reported, especially inasmuch as negative findings are often not reported. Many trials are designed to test an intervention, not to test whether the intervention is safe and effective in both men and women.
Another consideration is that the volume of health-research data is expanding, and new initiatives are underway to capture those data. The initiatives include developing a health-information-technology infrastructure and large databases, including the i2b2, the Cancer Biomedical Informatics Grid, improvement in the Medicare and Medicaid claims databases, and use of distributed data networks for an FDA sentinel system to detect adverse drug events (Bach et al., 2002; Kakazu et al., 2004; Murphy et al., 2006; Platt et al., 2009). If those technologies are to achieve their full potential in improving women’s health, the ability to capture and analyze sex- and gender-specific data needs to be considered during the design of such systems (Brittle and Bird, 2007; McKinley et al., 2002; Weisman, 2000). Additionally, data relevant to women’s health needs to be captured better in health-services research (for example, by using metrics of care quality specific to women) so that there can be more accurate measures of the translation
of research findings into health-care services and delivery (Chou et al., 2007a,b; Correa-de-Araujo, 2004; NCQA, 2010).
METHODOLOGIC LESSONS FROM THE WOMEN’S HEALTH INITIATIVE
Much has been learned from the WHI about how to design women’s health research (see Appendix C for details of this study). The WHI, which is the largest clinical study done exclusively on women, was designed as a study of primary prevention of diseases of aging (coronary heart disease, breast and colorectal cancer, and hip and other fractures), but it also assessed other end points (stroke, venous and pulmonary emboli, ovarian and endometrial cancer, gall bladder disease, cognition, and death). A global index was developed as a summary measure of the effect of treatment for potentially life-threatening events (Resnick et al., 2006, 2009). The WHI consisted of an observational study that was designed to identify predictors of disease in women and a clinical trial that consisted of three randomized components (Anderson et al., 2003):
trials that evaluated the effects of the postmenopausal hormone therapy, conjugated equine estrogen (Premarin™) on heart disease, fractures, and breast and colorectal cancer in 10,739 postmenopausal women who did not have uteruses, or conjugated equine estrogen plus medroxyprogesterone (Prempro™) in 16,608 postmenopausal omen who had uteruses;
a trial that assessed whether a calcium and vitamin D supplement reduces the risk of colorectal cancer and the frequency of hip and other fractures in over 36,000 postmenopausal women; and
a trial that assessed the effects of a diet low in fat and high in fruits, vegetables, and grains on breast cancer, colorectal cancer, and heart disease in almost 49,000 postmenopausal women.
The hormone-therapy component of the WHI was initiated to assess the risks and benefits associated with menopausal hormone therapy (estrogen or estrogen plus progestin) and to help to settle controversies about the efficacy of those therapies in preventing cardiovascular disease (HHS, 2010). Basic research had suggested a cardioprotective effect of estrogen in animals (Gerhard-Herman et al., 2000); data from individual studies and pooled data from observational studies, such as the NHS, found a significantly lower risk of coronary heart disease in postmenopausal women who were on estrogen alone (Barrett-Connor and Grady, 1998; Grady et al., 1992; Grodstein et al., 1996, 2000; Stampfer and Colditz, 1991); and data from a randomized controlled trial (the Postmenopausal Estrogen/Progestin Interventions [PEPI]) that examined intermediate end points (lipid profiles) as proxies for coronary heart disease found a somewhat beneficial
effect of PEPI (reducing low-density–lipoprotein cholesterol and increasing high-density–lipoprotein cholesterol by 10–15%) (Espeland et al., 1998). The NHS specifically showed decreased rates of coronary heart disease and of death from cardiovascular disease, but no effect on the rate of stroke, with estrogen therapy (Stampfer et al., 1991). At the time, hormone therapy was routinely prescribed from menopause on, and many thought it likely that the WHI would be halted prematurely because the beneficial effects of such therapy on cardiovascular disease would be demonstrated early in the study (IOM, 1993).
At the same time, however, other randomized controlled trials—such as the Heart and Estrogen/Progestin Replacement Study (HERS), which examined heart attacks and death from coronary heart disease (Herrington et al., 2000), and the Estrogen Replacement and Atherosclerosis (ERA) study, which examined lipid concentrations, angiographic end points, and cardiovascular events—did not show that estrogen plus progestin prevented further heart attacks or death from coronary heart disease in postmenopausal women who had heart disease and, in the case of HERS, actually resulted in a higher rate of coronary heart disease soon after initiation of treatment (Grady et al., 2000; Herrington et al., 2000; Hulley et al., 1998). Earlier, the Framingham Study, the only prospective observational study designed specifically to measure coronary–heart-disease end points, did not show beneficial effects of estrogen on mortality from all causes or from cardiovascular disease (Wilson et al., 1985). The Framingham Study differed from other observational studies in its inclusion of angina and systematic ascertainment of silent myocardial infarction through routine electrocardiography. Those examples highlight the importance of including study end points that reflect both female and male physiology. It is noteworthy that the observational studies showed effects similar to those in the WHI of both hormone treatments for disease outcomes other than cardiovascular disease (that is, breast, colorectal, and endometrial cancer; stroke; and venous thromboembolism) (WHI, 2010).
Because the WHI was a large, randomized controlled trial that looked at multiple end points, it provided more definitive results than the previous conflicting findings. That design helped to detect an increased risk of stroke and a lack of effect (increase or decrease) on heart disease in the estrogen-only portion of the WHI, and it enabled scientific confidence in those results (Anderson et al., 2004). That led to the halting of the studies for safety reasons (HHS, 2004).
The conflicting results on coronary heart disease in the NHS and the WHI led to exploration of new statistical methods to adjust for potential confounders in the use of nonrandomized observational data. One method, the use of propensity scores, was able to replicate the WHI’s findings on coronary heart disease better when applied to the observational data (Hernán et al., 2008; Tannen et al., 2007).
The WHI illustrates the utility of developing a composite score to assess benefits and risks together and among diseases. The WHI’s “global index” gave equal weight to the effects of hormone therapy on specific monitored outcomes—
coronary heart disease, stroke, pulmonary embolism, breast and colorectal cancer, hip fractures, and death. It was helpful in understanding the net effect of hormone therapy on those clinical outcomes. One limitation of the index, however, is in making decisions about treatment for menopause symptoms. The index does not include the effect of hormone therapy on menopausal symptoms, because the hormone-therapy component of the WHI was designed to evaluate effects on diseases of aging, not on menopause symptoms. The index also did not include dementia, urinary incontinence, and other end points that might affect treatment decisions. To be most useful for treatment decisions, a global measure of net effect should include all clinically relevant end points, especially those of greatest importance to women. The end points need to be weighted (either equally or differently) in an index; this can be done by using analytic decision models in which end points can be weighted implicitly according to their likelihood of occurrence, effect on mortality, and effect on HRQoL. The WHI recruited a large number of older, postmenopausal women and was relatively successful in meeting not only its overall recruitment goals but also its recruitment goals for minority women (Hays et al., 2003), and lessons can be learned from its recruitment strategies (Limacher, 2003). In addition, the lack of information on the social roles of the women in the WHI (for example, marital history) limits the ability to assess barriers to behavioral change, especially for the dietary arm.
The WHI also pointed to the importance of considering the timing and duration of treatment in observational studies. Some researchers pointed to the treatment of older postmenopausal women and questioned the generalizability of the findings to women who take hormone therapy at earlier stages for menopausal symptoms (Harman et al., 2004). A reanalysis of the data from the NHS demonstrated the importance of accounting for age, timing and duration of treatment, and the onset of adverse events in observational studies, with the reanalysis yielding results for the NHS closer to WHI than originally reported (Hernán et al., 2008). That reanalysis generated some controversy (Stampfer, 2008), but does point to the importance of timing issues in studies and the potential for novel analytical techniques to shed light on study differences (Hernán and Robins, 2008; Hernán et al., 2008; Hoover, 2008; Prentice, 2008; Wilcox and Wacholder, 2008; Willet et al., 2008).
There are several lessons to be learned regarding methodologic and statistical approaches to women’s health research. Etiology and risk factors can be investigated with well-designed and well-executed observational studies that measure and adjust for known confounders, ensure appropriate ascertainment of end points (especially coronary heart disease, whose ascertainment is more challenging), and use appropriate statistical techniques to analyze data. Caution needs to be exercised in extrapolating from studies that use biomarkers rather
than clinical outcomes. In using biomarkers for clinical outcomes, it is crucial that the measures selected be relevant to women, especially in clinical studies that involve both women and men. For example, the results of observational studies that reported on the effect of hormone therapy on lipids were interpreted as proxy evidence of their effect on coronary–heart-disease end points, but this was not confirmed or supported in randomized controlled trials that examined these end points.
Single-sex studies like the WHI can provide valuable information and fill in research gaps—especially in, for example, coronary heart disease—in clinical trials where women have been underrepresented. Most clinical trials, however, now include both women and men, and it will be critical to develop new methods and approaches to analysis of data by subgroup. After many years, the analysis of data by sex is still inadequate, and it is unclear whether adequate numbers of women are being enrolled in clinical trials to allow adequate analysis (GAO, 2000, 2001).
The limitations of randomized controlled trials, such as expense and the number of subjects needed, are reflected in questions that arose regarding the randomized controlled trials of the WHI. One question was related to the choice of hormone therapy—conjugated equine estrogen with progestin (Prempro™) and without progestin (Premarin™). Conjugated equine estrogen is a complex compound, and it is not known whether these results would apply to estradiol alone. It is also not known whether other routes of administration—dermal or intranasal—would have produced different results or whether different types of progestin would have different effects on coronary heart disease.
Women need to be considered in the design, inclusion and exclusion criteria, recruitment, outcome measures, and analysis of research. Adequate numbers of women need to be enrolled in studies to allow statistically significant sex-based analyses, study outcomes need to include symptoms and effects seen in women, analyses need to be conducted to determine sex or gender differences, and the results of the analyses need to be published. A number of design and analytic techniques can be explored to improve sex-specific analyses while limiting the increase in sample size.
Much research focuses on improving disease survival; insufficient attention has been paid to improving disease-related morbidity (for example, autoimmune diseases that affect a large number of women and the health effects that follow breast-cancer treatment and recovery) and especially to wellness and quality of life as health outcomes.
In the absence of a compelling reason not to, it should be assumed that there are sex differences in conditions (that is, sex matters), so research studies should be designed to include women in sufficient numbers to allow the resulting data to be analyzed.
Basic research should include analysis of effects in females. Information from this basic research can guide the focus of sex and gender differences in clinical studies.
Sex and gender differences in the manifestation of disease should be considered in the design of research to incorporate the spectrum of outcomes that are relevant to women. Government and other funding agencies should ensure adequate participation of women and reporting of sex-stratified analyses in health research. One possible mechanism would be to expand the role of data-safety monitoring boards to monitor participation, efficacy, and adverse outcomes by sex.
Given the practical limitations in the size of research studies, research designs and statistical techniques that facilitate analysis of data on subgroups—without substantially increasing the overall size of a study population—should be explored. Conferences or meetings with a specific goal of developing consensus guidelines or recommendations on such study methods (for example, the use of Bayesian statistics and the pooling of data across study groups) should be convened by NIH and other federal agencies and relevant professional organizations.
To gain knowledge from existing studies that individually do not have sufficient numbers of female subjects for separate analysis, the director of the Department of Health and Human Services Office of the National Coordinator for Health Information Technology should support development and application of mechanisms for pooling patient and subject data to answer research questions that are not definitively answered by single studies.
For approval of medical products (drugs, devices, and biologics) coming to market, FDA should enforce compliance with the requirement for sex-stratified analyses of efficacy and safety and should take these analyses into account in regulatory decisions.
When it is possible, analysis of clinical research should be stratified by sex and should include power calculations to prevent type II errors in interpretation that might lead to withholding of therapy from women.
The International Committee of Medical Journal Editors and editors of relevant journals not represented on that committee should adopt as a guideline that all papers that report the outcomes of clinical trials report outcomes in men and women separately except for trials involving sex-specific conditions (such as endometrial cancer and prostatic cancer). NIH should sponsor a meeting to facilitate establishment of such guidelines.
The federal government should ensure that a data infrastructure is designed to capture data in forms that facilitate its analysis by sex and gender.
Research should be conducted on women’s quality of life, including the development of better measures to compare effects not only of health conditions but of interventions and treatments on quality of life. The end points or outcomes examined in studies should include quality-of-life outcomes (for example, functional status, mobility, and pain) in addition to mortality.
Abramson, J., and J. M. Wright. 2007. Are lipid-lowering guidelines evidence-based? Lancet 369(9557):168–169.
Anderson, G., J. Manson, R. Wallace, B. Lund, D. Hall, S. Davis, S. Shumaker, C. Y. Wang, E. Stein, and R. L. Prentice. 2003. Implementation of the women’s health initiative study design. Annals of Epidemiology 13(9 Suppl.):S5–S17.
Anderson, G., M. Limacher, A. Assaf, T. Bassford, S. Beresford, H. Black, D. Bonds, R. Brunner, R. Brzyski, B. Caan, R. Chlebowski, D. Curb, M. Gass, J. Hays, G. Heiss, S. Hendrix, B. Howard, J. Hsia, A. Hubbell, R. Jackson, K. Johnson, H. Judd, J. Kotchen, L. Kuller, A. LaCroix, D. Lane, R. Langer, N. Lasser, C. Lewis, J. Manson, K. Margolis, J. Ockene, M. O’Sullivan, L. Phillips, R. Prentice, C. Ritenbaugh, J. Robbins, J. Rossouw, G. Sarto, M. Stefanick, L. Van Horn, J. Wactawski-Wende, R. Wallace, and S. Wassertheil-Smoller. 2004. Effects of conjugated equine estrogen in postmenopausal women with hysterectomy: The women’s health initiative randomized controlled trial. Journal of the American Medical Association 291(14):1701–1712.
Antithrombotic Trialists Collaboration. 2009. Aspirin in the primary and secondary prevention of vascular disease: Collaborative meta-analysis of individual participant data from randomised trials. Lancet 373(9678):1849–1860.
Bach, P. B., E. Guadagnoli, D. Schrag, N. Schussler, and J. L. Warren. 2002. Patient demographic and socioeconomic characteristics in the SEER-Medicare database applications and limitations. Medical Care 40(8 Suppl.):IV19–IV25.
Bairey Merz, C. N., L. J. Shaw, S. E. Reis, V. Bittner, S. F. Kelsey, M. Olson, B. D. Johnson, C. J. Pepine, S. Mankad, B. L. Sharaf, W. J. Rogers, G. M. Pohost, A. Lerman, A. A. Quyyumi, and G. Sopko. 2006. Insights from the NHLBI-sponsored Women’s Ischemia Syndrome Evaluation (WISE) study: Part II: Gender differences in presentation, diagnosis, and outcome with regard to gender-based pathophysiology of atherosclerosis and macrovascular and microvascular coronary disease. Journal of the American College of Cardiology 47(3 Suppl.):S21–S29.
Barrett-Connor, E., and D. Grady. 1998. Hormone replacement therapy, heart disease, and other considerations. Annual Review of Public Health 19:55–72.
Baumeister, R. F. 1988. Should we stop studying sex differences altogether? [editorial]. American Psychologist 43:1092–1095.
Bennett, J. C., and the Board on Health Sciences Policy of the Institute of Medicine. 1993. Inclusion of women in clinical trials—policies for population subgroups. New England Journal of Medicine 329(4):288–292.
Berlin, J. A., and S. S. Ellenberg. 2009. Inclusion of women in clinical trials. BMC Medicine 7:56.
Berry, D. A. 2006. Bayesian clinical trials. Nature Reviews Drug Discovery 5(1):27–36.
Bravata, D. M., A. L. Gienger, K. M. McDonald, V. Sundaram, M. V. Perez, R. Varghese, J. R. Kapoor, R. Ardehali, D. K. Owens, and M. A. Hlatky. 2007. Systematic review: The comparative effectiveness of percutaneous coronary interventions and coronary artery bypass graft surgery. Annals of Internal Medicine 147(10):703–716.
Brittle, C., and C. E. Bird. 2007. Literature Review on Effective Sex- and Gender-Based Systems/Models of Care. Arlington, VA: Uncommon Insights.
Canto, J. G., R. J. Goldberg, M. M. Hand, R. O. Bonow, G. Sopko, C. J. Pepine, and T. Long. 2007. Symptom presentation of women with acute coronary syndromes: Myth vs reality. Archives of Internal Medicine 167(22):2405–2413.
Chen, M. L., S. C. Lee, M. J. Ng, D. J. Schuirmann, L. J. Lesko, and R. L. Williams. 2000. Pharmacokinetic analysis of bioequivalence trials: Implications for sex-related issues in clinical pharmacology and biopharmaceutics. Clinical Pharmacology and Therapeutics 68(5):510–521.
Chou, A. F., S. H. Scholle, C. S. Weisman, A. S. Bierman, R. Correa-de-Araujo, and L. Mosca. 2007a. Gender disparities in the quality of cardiovascular disease care in private managed care plans. Women’s Health Issues 17(3):120–130.
Chou, A. F., L. Wong, C. S. Weisman, S. Chan, A. S. Bierman, R. Correa-de-Araujo, and S. H. Scholle. 2007b. Gender disparities in cardiovascular disease care among commercial and medicare managed care plans. Women’s Health Issues 17(3):139–149.
The Coronary Drug Project Research Group. 1973. The Coronary Drug Project: Findings leading to discontinuation of the 2.5-mg/day estrogen group. Journal of the American Medical Association 226(6):652–657.
Correa-de-Araujo, R. 2004. A wake-up call to advance women’s health. Women’s Health Issues 14(2):31–34.
DeCara, J. M. 2003. Noninvasive cardiac testing in women. Journal of the American Medical Women’s Association 58(4):254–263.
Espeland, M. A., S. M. Marcovina, V. Miller, P. D. Wood, C. Wasilauskas, R. Sherwin, H. Schrott, and T. L. Bush. 1998. Effect of postmenopausal hormone therapy on lipoprotein(a) concentration. PEPI investigators. Postmenopausal estrogen/progestin interventions. Circulation 97(10): 979–986.
Fleishman, J. A., and W. F. Lawrence. 2003. Demographic variation in SF-12 scores: True differences or differential item functioning? Medical Care 41(7 Suppl.):III75–III86.
Freedman, L. S., R. Simon, M. A. Foulkes, L. Friedman, N. L. Geller, D. J. Gordon, and R. Mowery. 1995. Inclusion of women and minorities in clinical trials and the NIH revitalization act of 1993—the perspective of NIH clinical trialists. Controlled Clinical Trials 16(5):277–285; discussion 286–289, 293–309.
Fryback, D. G., N. C. Dunham, M. Palta, J. Hanmer, J. Buechner, D. Cherepanov, S. A. Herrington, R. D. Hays, R. M. Kaplan, T. G. Ganiats, D. Feeny, and P. Kind. 2007. US norms for six generic health-related quality-of-life indexes from the National Health Measurement Study. Medical Care 45(12):1162–1170.
Gandhi, M., N. Ameli, P. Bacchetti, G. B. Sharp, A. L. French, M. Young, S. J. Gange, K. Anastos, S. Holman, A. Levine, and R. M. Greenblatt. 2005. Eligibility criteria for HIV clinical trials and generalizability of results: The gap between published reports and study protocols. AIDS 19(16):1885–1896.
GAO (General Acounting Office). 1992. Women’s Health: FDA Needs to Ensure More Study of Gender Differences in Prescription Drug Testing: Report to Congressional Requesters. Washington, DC: GAO.
———. 2000. Women’s Health: NIH Has Increased Its Efforts to Include Women in Research: Report to Congressional Requesters. Washington, DC: GAO.
———. 2001. Women’s Health: Women Sufficiently Represented in New Drug Testing, but FDA Oversight Needs Improvement. Washington, DC: GAO.
Gerhard-Herman, M., N. Hamburg, and P. Ganz. 2000. Hormone replacement therapy and cardiovascular risk. Current Cardiology Reports 2(4):288–292.
Gold, M. R., D. Stevenson, and D. G. Fryback. 2002. HALYS and QALYS and DALYS, oh my: Similarities and differences in summary measures of population health. Annual Review of Public Health 23(1):115–134.
Grady, D., S. M. Rubin, D. B. Petitti, C. S. Fox, D. Black, B. Ettinger, V. L. Ernster, and S. R. Cummings. 1992. Hormone therapy to prevent disease and prolong life in postmenopausal women. Annals of Internal Medicine 117(12):1016–1037.
Grady, D., N. K. Wenger, D. Herrington, S. Khan, C. Furberg, D. Hunninghake, E. Vittinghoff, and S. Hulley. 2000. Postmenopausal hormone therapy increases risk for venous thromboembolic disease. The Heart and Estrogen/Progestin Replacement Study. Annals of Internal Medicine 132(9):689–696.
Grady, D., L. Chaput, and M. Kristof. 2003. Results of systematic review of research on diagnosis and treatment of coronary heart disease in women. In Evidence Report/Technology Assessment. No. 80. (Prepared by the University of California, San Francisco—Stanford Evidence-based Practice Center under Contract No 290-97-0013.) AHRQ Publication No. 03-0035. Rockville, MD: Agency for Healthcare Research and Quality.
Grodstein, F., M. J. Stampfer, J. E. Manson, G. A. Colditz, W. C. Willett, B. Rosner, F. E. Speizer, and C. H. Hennekens. 1996. Postmenopausal estrogen and progestin use and the risk of cardiovascular disease. New England Journal of Medicine 335(7):453-461.
Grodstein, F., J. E. Manson, G. A. Colditz, W. C. Willett, F. E. Speizer, and M. J. Stampfer. 2000. A prospective, observational study of postmenopausal hormone therapy and primary prevention of cardiovascular disease. Annals of Internal Medicine 133(12):933–941.
Harman, S. M., E. A. Brinton, T. Clarkson, C. B. Heward, H. S. Hecht, R. H. Karas, D. R. Judelson, and F. Naftolin. 2004. Is the WHI relevant to HRT started in the perimenopause? Endocrine 24(3):195–202.
Hays, J., J. R. Hunt, F. A. Hubbell, G. L. Anderson, M. Limacher, C. Allen, and J. E. Rossouw. 2003. The Women’s Health Initiative recruitment methods and results. Annals of Epidemiology 13(9 Suppl.):S18–S77.
Hennekens, C. H., and J. E. Buring. 1989. Methodologic considerations in the design and conduct of randomized trials: The US Physicians’ Health Study. Controlled Clinical Trials 10(4 Suppl.):142S–150S.
Hennekens, C. H., and K. Eberlein. 1985. A randomized trial of aspirin and beta-carotene among US Physicians. Preventive Medicine 14(2):165–168.
Hernán, M. A., and J. M. Robins. 2008. Authors’ response, part I: Observational studies analyzed like randomized experiments: Best of both worlds. Epidemiology 19(6):789–792.
Hernán, M. A., A. Alonso, R. Logan, F. Grodstein, K. B. Michels, W. C. Willett, J. E. Manson, and J. M. Robins. 2008. Observational studies analyzed like randomized experiments: An application to postmenopausal hormone therapy and coronary heart disease. Epidemiology 19(6):766–779.
Herrington, D. M., D. M. Reboussin, K. B. Brosnihan, P. C. Sharp, S. A. Shumaker, T. E. Snyder, C. D. Furberg, G. J. Kowalchuk, T. D. Stuckey, W. J. Rogers, D. H. Givens, and D. Waters. 2000. Effects of estrogen replacement on the progression of coronary-artery atherosclerosis. New England Journal of Medicine 343(8):522–529.
Hersh, A. L., M. L. Stefanick, and R. S. Stafford. 2004. National use of postmenopausal hormone therapy: Annual trends and response to recent evidence. Journal of the American Medical Association 291(1):47–53.
HHS (US Department of Health and Human Services). 2004. NHLBI Advisory for Physicians on the WHI Trial of Conjugated Equine Estrogens Versus Placebo. http://www.nhlbi.nih.gov/whi/e-a_advisory.htm (accessed April 12, 2010).
———. 2009. Monitoring Adherence to the NIH Policy on the Inclusion of Women and Minorities as Subjects in Clinical Research.
———. 2010. Women’s Health Initiative. http://orwh.od.nih.gov/inclusion/FinalAnnualReport2007.pdf (accessed May 6, 2010).
Hoover, R. N. 2008. The sound and the fury: Was it all worth it? Epidemiology 19(6):780–782; discussion 789–793.
Hulley, S., D. Grady, T. Bush, C. Furberg, D. Herrington, B. Riggs, and E. Vittinghoff. 1998. Randomized trial of estrogen plus progestin for secondary prevention of coronary heart disease in postmenopausal women. Heart and Estrogen/progestin Replacement Study (HERS) Research Group. Journal of the American Medical Association 280(7):605–613.
IOM (Institute of Medicine). 1993. An Assessment of the NIH Women’s Health Initiative. Washington, DC: National Academy Press.
Jagsi, R., A. R. Motomura, S. Amarnath, A. Jankovic, N. Sheets, and P. A. Ubel. 2009. Under-representation of women in high-impact published clinical cancer research. Cancer 115(14): 3293–3301.
Kakazu, K. K., L. W. Cheung, and W. Lynne. 2004. The cancer biomedical informatics grid (caBIG): Pioneering an expansive network of information and tools for collaborative cancer research. Hawaii Medical Journal 63(9):273–275.
Kim, E. S., and V. Menon. 2009. Status of women in cardiovascular clinical trials. Arteriosclerosis, Thrombosis, and Vascular Biology 29(3):279–283.
LaRosa, J. C., J. He, and S. Vupputuri. 1999. Effect of statins on risk of coronary disease: A meta-analysis of randomized controlled trials. Journal of the American Medical Association 282(24):2340–2346.
Limacher, M. C.2003. Recruitment and retention: Lessons learned from the Women’s Health Initiative. In Science Meets Reality: Recruitment and Retention of Women in Clinical Studies, and the Critical Role of Relevance. A Report of the Task Force Sponsored by the NIH Office of Women’s Health, edited by HHS. Washington, DC.
Lloyd-Jones, D., R. Adams, M. Carnethon, G. De Simone, T. B. Ferguson, K. Flegal, E. Ford, K. Furie, A. Go, K. Greenlund, N. Haase, S. Hailpern, M. Ho, V. Howard, B. Kissela, S. Kittner, D. Lackland, L. Lisabeth, A. Marelli, M. McDermott, J. Meigs, D. Mozaffarian, G. Nichol, C. O’Donnell, V. Roger, W. Rosamond, R. Sacco, P. Sorlie, R. Stafford, J. Steinberger, T. Thom, S. Wasserthiel-Smoller, N. Wong, J. Wylie-Rosett, and Y. Hong. 2009. Heart disease and stroke statistics—2009 update: A report from the American Heart Association Statistics Committee and Stroke Statistics Subcommittee. Circulation 119(3):480–486.
Lloyd-Jones, D. M., C. J. O’Donnell, R. B. D’Agostino, J. Massaro, H. Silbershatz, and P. W. F. Wilson. 2001. Applicability of cholesterol-lowering primary prevention trials to a general population: The Framingham Heart Study. Archives of Internal Medicine 161(7):949–954.
Mascitelli, L., and F. Pezzetta. 2007. Statins for primary prevention of coronary artery disease. Lancet 369(9567):1078–1079; author reply 1079.
McKinley, E. D., J. W. Thompson, J. Briefer-French, L. S. Wilcox, C. S. Weisman, and W. C. Andrews. 2002. Performance indicators in women’s health: Incorporating women’s health in the health plan employer data and information set (HEDIS). Women’s Health Issues 12(1):46–58.
McPherson, R., and N. Kavaslar. 2007. Statins for primary prevention of coronary artery disease. Lancet 369(9567):1078; author reply 1079.
Merkatz, R. B., and S. W. Junod. 1994. Historical background of changes in FDA policy on the study and evaluation of drugs in women. Academic Medicine 69(9):703–707.
Murphy, S. N., M. E. Mendis, D. A. Berkowitz, I. Kohane, and H. C. Chueh. 2006. Integration of clinical and genetic data in the i2b2 architecture. AMIA Annual Symposium Proceedings:1040.
NCQA (National Committee for Quality Assurance). 2010. HEDIS 2010 Measures. http://www.ncqa.org/tabid/59/Default.aspx (accessed March 23, 2010).
NHS (Nurse’s Health Study). 2008. History of the Nurse’s Health Study. http://www.channing.harvard.edu/nhs/index.php/history/ (accessed March 9, 2010).
Pardo, M., M. Ruiz, A. Gimeno, L. Navarro, A. Garcia, M. Tarazona, and M. Aznar. 2002. Gender Bias in Clinical Trials of AIDS Drugs. Paper read at International Conference on AIDS, July 7–12, 2002; Barcelona, Spain.
Pedone, C., and K. Lapane. 2003. Generalizability of guidelines and physicians’ adherence. Case study on the Sixth Joint National Committee’s Guidelines on Hypertension. BMC Public Health 3(1):24.
Pinnow, E., P. Sharma, A. Parekh, N. Gevorkian, and K. Uhl. 2009. Increasing participation of women in early phase clinical trials approved by the FDA. Women’s Health Issues 19(2):89–93.
Platt, R., M. Wilson, K. A. Chan, J. S. Benner, J. Marchibroda, and M. McClellan. 2009. The new sentinel network—improving the evidence of medical-product safety. New England Journal of Medicine 361(7):645–647.
Prentice, R. L. 2008. Data analysis methods and the reliability of analytic epidemiologic research. Epidemiology 19(6):785–788; discussion 789–793.
Resnick, S. M., P. M. Maki, S. R. Rapp, M. A. Espeland, R. Brunner, L. H. Coker, I. A. Granek, P. Hogan, J. K. Ockene, and S. A. Shumaker. 2006. Effects of combination estrogen plus progestin hormone treatment on cognition and affect. Journal of Clinical Endocrinology and Metabolism 91(5):1802–1810.
Resnick, S. M., M. A. Espeland, Y. An, P. M. Maki, L. H. Coker, R. Jackson, M. L. Stefanick, R. Wallace, and S. R. Rapp. 2009. Effects of conjugated equine estrogens on cognition and affect in postmenopausal women with prior hysterectomy. Journal of Clinical Endocrinology and Metabolism 94(11):4152–4161.
Ridker, P. M. 2010. Statin therapy for low-LDL, high-hsCRP patients: From JUPITER to CORONA. Clinical Chemistry 56(4):505–507.
Ridker, P. M., N. R. Cook, I. M. Lee, D. Gordon, J. M. Gaziano, J. E. Manson, C. H. Hennekens, and J. E. Buring. 2005. A randomized trial of low-dose aspirin in the primary prevention of cardiovascular disease in women. New England Journal of Medicine 352(13):1293–1304.
Rieker, P. P., and C. E. Bird. 2005. Rethinking gender differences in health: Why we need to integrate social and biological perspectives. Journals of Gerontology. Series B, Psychological Sciences and Series B, Psychological Sciences and Social Sciences 60(Special Issue 2):S40–S47.
Rosenbaum, P. R. 2002. Observational Studies. 2nd ed. New York: Springer-Verlag.
Samsa, G., G. Hu, and M. Root. 2005. Combining information from multiple data sources to create multivariable risk models: Illustration and preliminary assessment of a new method. Journal of Biomedicine and Biotechnology 2005(2):113–123.
Sharpe, N. 2002. Clinical trials and the real world: Selection bias and generalisability of trial results. Cardiovascular Drugs and Therapy 16(1):75–77.
Sheifer, S. E., T. A. Manolio, and B. J. Gersh. 2001. Unrecognized myocardial infarction. Annals of Internal Medicine 135(9):801–811.
Shepherd, J., S. M. Cobbe, I. Ford, C. G. Isles, A. R. Lorimer, P. W. MacFarlane, J. H. McKillop, and C. J. Packard. 1995. Prevention of coronary heart disease with pravastatin in men with hypercholesterolemia. West of Scotland Coronary Prevention Study Group. New England Journal of Medicine 333(20):1301–1307.
Stampfer, M. J. 2008. ITT for observational data: Worst of both worlds? Epidemiology 19(6):783–784; discussion 789–793.
Stampfer, M. J., and G. A. Colditz. 1991. Estrogen replacement therapy and coronary heart disease: A quantitative assessment of the epidemiologic evidence. Preventive Medicine 20(1):47–63.
Stampfer, M. J., G. A. Colditz, W. C. Willett, J. E. Manson, B. Rosner, F. E. Speizer, and C. H. Hennekens. 1991. Postmenopausal estrogen therapy and cardiovascular disease. Ten-year follow-up from the Nurses’ Health Study. New England Journal of Medicine 325(11):756–762.
Stewart, J. H., A. G. Bertoni, J. L. Staten, E. A. Levine, and C. P. Gross. 2007. Participation in surgical oncology clinical trials: Gender-, race/ethnicity-, and age-based disparities. Annals of Surgical Oncology 14(12):3328–3334.
SWAN (Study of Women’s Health Across the Nation). 2010. Swan History. http://www.swanstudy.org/history.asp (accessed April 9, 2010).
Tannen, R. L., M. G. Weiner, D. Xie, and K. Barnhart. 2007. Estrogen affects post-menopausal women differently than estrogen plus progestin replacement therapy. Human Reproduction 22(6):1769–1777.
Towfighi, A., J. L. Saver, R. Engelhardt, and B. Ovbiagele. 2007. A midlife stroke surge among women in the United States. Neurology 69(20):1898–1904.
Weisman, C. S. 2000. Advocating for gender-specific health care: A historical perspective. Journal of Gender-Specific Medicine 3(3):22–24.
WHI (Women’s Health Initiative). 2010. About Women’s Health Initiative. http://whi.org/about/ (accessed April 8, 2010).
WHS (Women’s Health Study). 2009. Women’s Health Study (WHS): A Randomized Trial of Low-dose Aspirin and Vitamin E in the Primary Prevention of Cardiovascular Disease and Cancer. http://clinicaltrials.gov/ct/show/NCT00000479 (accessed April 9, 2010).
Wilcox, A., and S. Wacholder. 2008. Observational data and clinical trials: Narrowing the gap? Epidemiology 19(6):765.
Willet, W. C., J. E. Manson, and F. Grodstein. 2008. Author’s response, part II. Epidemiology 19(6):793.
Wilson, P. W., R. J. Garrison, and W. P. Castelli. 1985. Postmenopausal estrogen use, cigarette smoking, and cardiovascular morbidity in women over 50. The Framingham Study. New England Journal of Medicine 313(17):1038–1043.