National Academies Press: OpenBook

Saving Women's Lives: Strategies for Improving Breast Cancer Detection and Diagnosis (2005)

Chapter: Appendix D: Common Weaknesses in Study Designs

« Previous: Appendix C: ROC Analysis: Key Statistical Tool for Evaluating Detection Technologies
Suggested Citation:"Appendix D: Common Weaknesses in Study Designs." Institute of Medicine and National Research Council. 2005. Saving Women's Lives: Strategies for Improving Breast Cancer Detection and Diagnosis. Washington, DC: The National Academies Press. doi: 10.17226/11016.
×

Appendix D
Common Weaknesses in Study Designs

The following section describes some of the most common weaknesses in study design seen by medical technology evaluators.

Poorly Described Patient Populations. Unless the criteria used to determine patients’ eligibility for a study are clearly outlined, and the characteristics of the patients who actually get enrolled are clearly described, it is impossible to know to which populations of patients the results of the study can be confidently applied, or whether the results from two different studies are truly comparable. If the experience of particular subgroups of patients is important and likely to vary, then enrollment should be stratified on the basis of those patient subgroup characteristics.

Too Narrow a Patient Population. The patients enrolled in a study of a new technology should be similar to the patients in whom the technology is most likely to be used. The enrolled population should include individuals without the target disease of interest, such as patients with risk factors for the disease but without the disease itself, patients with different, but commonly confused conditions, and patients with other types of pathology in the same organ systems.a Failure to enroll the appropriate spectrum of patients in a study of a new diagnostic technology can lead to overestimates

a  

For example, different breast cancer detection technologies vary in their ability to detect microcalcifications, which are not cancerous lesions but are significant breast cancer risk factors. For instance, ultrasound is highly sensitive to lesions but does a poor job of detecting microcalicifications.

Suggested Citation:"Appendix D: Common Weaknesses in Study Designs." Institute of Medicine and National Research Council. 2005. Saving Women's Lives: Strategies for Improving Breast Cancer Detection and Diagnosis. Washington, DC: The National Academies Press. doi: 10.17226/11016.
×

of both the sensitivity and specificity of the new technology. Similarly, failure to enroll an appropriate spectrum of patients in a study of a new therapy can lead to an overestimate of the effectiveness of that therapy. This particular failure lies at the root of countless headlines announcing new breakthrough procedures or therapies that kindle excitement, but deliver only false hopes—and leave the public wondering why there are so few breakthroughs in their own treatment.

Failure to Use Appropriate Controls or Comparison Groups. The purpose of a control group is to allow the observer to conclude that any change observed in the “active treatment group” is due to the treatment being studied, rather than to other factors. Control groups are particularly important when factors in addition to the intervention under study can affect the outcome of interest, when the new technology of interest and some established technology are both effective, and when the natural course of untreated disease is not clear or consistent, as is the case with breast cancer. Failure to use a control group, or use of an inappropriate control group, can make it impossible to draw meaningful conclusions from a study.

Failure to Demonstrate the Comparability of Patients in Treatment and Control Groups. Given the purpose of a control group, it is important that patients in the treatment and control groups be similar in terms of baseline characteristics that can influence the outcome of the intervention under study. For example, if one study group included more women at high risk for breast cancer than another group, then a detection technology tested in the high-risk group would likely detect more cancer cases than a technology tested in the low-risk group, leading to the perception that the detection system was more sensitive than a system tested in lower risk patients.

Unclear Definition of Study Endpoints. Medical technologies can be assessed a multiple levels, depending upon whether they are diagnostic or therapeutic. The most basic level at which a diagnostic technology can be assessed is definition of its performance characteristics—sensitivity and specificity. Even this basic level of assessment is not easy to perform. It requires comparison of the performance of the new technology with that of a gold standard. And true gold standards (such as tissue obtained during surgery) are not always available.

Bias. The confidence that you can have that the results of using a technology described in a study are the same results you would get if you used the technology in a similar fashion depends on the absence of bias. Bias is systematic sources of variation that distort the results of a study in one direction or another. There are many types of bias that have been

Suggested Citation:"Appendix D: Common Weaknesses in Study Designs." Institute of Medicine and National Research Council. 2005. Saving Women's Lives: Strategies for Improving Breast Cancer Detection and Diagnosis. Washington, DC: The National Academies Press. doi: 10.17226/11016.
×

described including those that are especially problematic in cancer screening. The most common general sources of bias in clinical trials are:

Confounding. A confounding variable is one that falsely obscures or accentuates the relationship between two factors, such as the effect of a treatment on patient outcome. Confounding occurs when a factor other than the interventions being compared is not distributed equally in the study groups being assessed and affects the outcome of interest.

Systematic Errors or Differences in Measurement. Selection bias can occur inadvertently if there are systematic errors or differences in the way particular patient characteristics (e.g., eligibility criteria) are measured or in the way a determination is made of the intervention to which a patient was exposed. (The latter could be a problem when exposure to the intervention is ascertained from insurance claims data, which may or may not be comprehensive or accurate.) The most common sources of bias due to measurement error, however, arise in evaluation of the outcomes of patients in two arms of a study. Ascertainment of patient outcomes by an “unblinded’” investigator who knows what intervention each patient received poses a serious risk of bias. An unblinded investigator, for example, may interpret particular findings differently, or look for particular findings with varying efforts, if she or he has preconceived notions about the comparative effects of the two technologies under study. Finally, although it may seem obvious that measurements of the outcomes of patients in two arms of a study should be performed in an identical manner and at the same point in time (relative to the interventions under study), this important aspect of study design is not always followed.

Loss of Patients to Follow-Up. Anyone who has conducted an observational study knows how difficult it is to follow patients over time. Loss of patients to follow-up becomes a threat to the internal validity of a study when it occurs in a substantial proportion of patients and at differential rates in the various arms of a study. Failure to account for all patients who were initially enrolled in a study is particularly problematic. In one study submitted to the Food and Drug Administration (FDA), for example, data on patients who had received a new device were reported only for those patients who were followed for at least one year. Many patients dropped out of the study prior to the one-year endpoint, however, due either to side effects of to ineffectiveness of the device. Consequently, the results reported to the FDA exaggerated the effectiveness and tolerability of the device. All enrolled patients thus must be accounted for. If some patients withdraw or are lost to follow-up, the number of withdrawals and losses in each arm should be reported with specification of the reasons for withdrawal.

Suggested Citation:"Appendix D: Common Weaknesses in Study Designs." Institute of Medicine and National Research Council. 2005. Saving Women's Lives: Strategies for Improving Breast Cancer Detection and Diagnosis. Washington, DC: The National Academies Press. doi: 10.17226/11016.
×

Inappropriate Statistical Analysis and Planning. On occasion, statistical analyses reported in published studies are not performed correctly, or the most appropriate statistical analyses are not performed. In other instances, statistical issues, such as statistical power to detect a difference between two arms of a study if one really existed, do not seem to have been adequately considered in planning the study, or had to be compromised for practical reasons (such as study cost or patient availability). As a result, the results reported in some studies are misleading and have a significant probability of being wrong. Investigators should report the statistical significance of their results, and provide 95 percent confidence intervals around group differences or main effects. In addition, if relative risks or odds ratios are reported (such as reporting that a particular outcome is twice as likely to occur with treatment A as with treatment B), the absolute rate with which the outcome occurs also should be reported.

Poorly Described Techniques. Diagnostic and therapeutic techniques are often employed using very specific protocols or techniques that affect the effectiveness or safety of the interventions. For example, different pulse sequences can be used in magnetic resonance imaging studies and different software might base comparisons of digitized mammography images on different calculations. Unless the technology under study, and the technologies to which it is being compared, are clearly described, it is not possible to meaningfully compare the results of one study to those of other studies of what appears to be the same technology. Without such descriptions it also may be difficult or even impossible to judge the relevance of the study results.

Suggested Citation:"Appendix D: Common Weaknesses in Study Designs." Institute of Medicine and National Research Council. 2005. Saving Women's Lives: Strategies for Improving Breast Cancer Detection and Diagnosis. Washington, DC: The National Academies Press. doi: 10.17226/11016.
×
Page 322
Suggested Citation:"Appendix D: Common Weaknesses in Study Designs." Institute of Medicine and National Research Council. 2005. Saving Women's Lives: Strategies for Improving Breast Cancer Detection and Diagnosis. Washington, DC: The National Academies Press. doi: 10.17226/11016.
×
Page 323
Suggested Citation:"Appendix D: Common Weaknesses in Study Designs." Institute of Medicine and National Research Council. 2005. Saving Women's Lives: Strategies for Improving Breast Cancer Detection and Diagnosis. Washington, DC: The National Academies Press. doi: 10.17226/11016.
×
Page 324
Suggested Citation:"Appendix D: Common Weaknesses in Study Designs." Institute of Medicine and National Research Council. 2005. Saving Women's Lives: Strategies for Improving Breast Cancer Detection and Diagnosis. Washington, DC: The National Academies Press. doi: 10.17226/11016.
×
Page 325
Next: Glossary »
Saving Women's Lives: Strategies for Improving Breast Cancer Detection and Diagnosis Get This Book
×
Buy Hardback | $65.00 Buy Ebook | $49.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The outlook for women with breast cancer has improved in recent years. Due to the combination of improved treatments and the benefits of mammography screening, breast cancer mortality has decreased steadily since 1989. Yet breast cancer remains a major problem, second only to lung cancer as a leading cause of death from cancer for women. To date, no means to prevent breast cancer has been discovered and experience has shown that treatments are most effective when a cancer is detected early, before it has spread to other tissues. These two facts suggest that the most effective way to continue reducing the death toll from breast cancer is improved early detection and diagnosis.

Building on the 2001 report Mammography and Beyond, this new book not only examines ways to improve implementation and use of new and current breast cancer detection technologies but also evaluates the need to develop tools that identify women who would benefit most from early detection screening. Saving Women’s Lives: Strategies for Improving Breast Cancer Detection and Diagnosis encourages more research that integrates the development, validation, and analysis of the types of technologies in clinical practice that promote improved risk identification techniques. In this way, methods and technologies that improve detection and diagnosis can be more effectively developed and implemented.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!