Improving Breast Imaging Quality Standards (2005) / Chapter Skim
Currently Skimming:
2 Improving Interpretive Performance in Mammography
2 Improving Interpretive Performance in Mammography
Pages 24-81
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 24... ...
The available evidence indicates that interpretive performance is quite variable, but the ambiguities of human decision making, the complexities of clinical practice settings, and the rare occurrence of cancer make measurement, evaluation, and improvement of mammography interpretation a much more difficult task. The components of current MQSA regulations pertinent to interpretive performance include: (1)
|
From page 25... ...
Before describing the measures, it is important to clearly define a positive and negative test. The Breast Imaging Reporting and Data System (BI-RADS)
|
From page 26... ...
. Some variability in calculated performance measures can, therefore, be attributed to variance among interpreting physicians on what constitutes an abnormal mammogram.
|
From page 27... ...
, the specification of the period of follow-up after a mammogram is needed to observe women for the occurrence of cancer and calculate performance indices that can be compared in a meaningful way. With the above definitions, it is possible to identify several measures of interpretive performance.
|
From page 28... ...
Because most interpreting physicians do not have that complete information (no linkage to regional tumor registry) or the volumes to create stable estimates, measurement of sensitivity will be of very limited use for individual interpreting physicians in practice.
|
From page 29... ...
Although studies indicate that some interpreting physicians inappropriately assign women who need additional imaging a Category 3 BI-RADS assessment (Poplack et al., 2000; Taplin et al., 2002) , this practice should be discouraged, and all women needing additional imaging should be tracked.
|
From page 30... ...
The ACR recommends that interpreting physicians who choose to perform one of the two types of audits described in the BI-RADS atlas should track all women referred for additional imaging for their subsequent cancer status (PPV1) (American College of Radiology, 2003)
|
From page 31... ...
. Desirable goals for recall rates for highly skilled interpreting physicians were set at less than or equal to 10 percent in the early 1990s (Bassett et al., 1994)
|
From page 32... ...
Node positivity, however, is not necessarily a useful surrogate measure of an interpreting physician's interpretive performance because inherently aggressive tumors may metastasize to the axillary lymph nodes early, when the tumor is still small, or even before the tumor becomes visible on a mammogram.
|
From page 33... ...
Focusing on sensitivity simply indicates how an interpreting physician operates when cancer is present. Focusing on specificity simply indicates how an interpreting physician operates when cancer is not present.
|
From page 34... ...
Estimating the AUC is possible if the status of all examined women is known and the appropriate computer software is employed. It has the advantage of reflecting the discriminatory ability of the interpreting physician and incorporates both sensitivity and specificity into a single measure, accounting for the trade-offs between the two measures.
|
From page 35... ...
Given the challenges and limitations, the Committee recommends a focus on PPV2. Calculating the cancer detection rate and the rate of abnormal interpretation (women whose mammogram interpretation leads to a recommendation for additional imaging or biopsy)
|
From page 36... ...
ctionteeD a 0 nosis b 00, Diag 0 000 000 000 6.1/1 1,00 Cancer Rate 4.8/1, (adjusted) 4.2/1, (unadjusted)
|
From page 37... ...
. Interpretive Volume and Interpreting Physicians' Levels of Experience Interpretive volume and interpreting physicians' levels of experience (length of time interpreting mammography)
|
From page 38... ...
Whether the findings from this small sample size from a program in Canada, where the qualifying standards for interpreting physicians are quite different, can be generalized to practice in the United States is not clear. Another recent study from a population-based breast cancer screening program in Quebec showed that the rate of breast cancer detection was unrelated to the radiologist's interpretive volume, but increased with the facility's screening volume (Theberge et al., 2005)
|
From page 39... ...
e of rather y effect ntse =21) n( effect practice represen- cers)
|
From page 40... ...
and for fo-a acticerp wase recall lower higher umelov rience oni o neither acticerp ing tricemlev but volum to expe criter but mamm overall included no actual both of Sens ed screen ed cy actual and on accuracy Analytic Considerations Used· data Higher· related and Spec; earsy affect calling positive, affect accura Used· data cus Included· of for of AUC for and 0+ per AUC mam for2 77%; inter of 0.92 0.9 000; ber 12, 002,rof + focus se 000 high fewer sma Sens: fal ex 10%; 04, with 50% + Sens/Spec/ Adjusted num ogramsm preted: <1,000; 1,00 0.92 (p=0.94) Mean mean rate interpretation 2,50 eary screening had false t Biopsy Info -- - men Recall Rate 10.4% - Improve of -eD Rate Measures Cancer tection -- - 6 sample 12 (n=124 sample ,04 (n=209 n)
|
From page 41... ...
the of ngi tsn cted event standardizeda interpretationla viddiyb cou expe the by of Derived abnorm ratio observed event counts Analytic Considerations · was AUC 99 those la readers 9993, 2,9 2,000 for 00 than e.u 00 3,0 2,0 eary than val Sens/Spec/ Standardized abnorm interpretation better of and per less predictive t positive Biopsy Info - men PPV= Recall Rate - Improve of -eD Rate Spec=specificity, Measures Cancer tection - s =sensitivity, ns ogist Se, radiol rveuc Population 35 atingr per- ope ' nce.a receiver relationship Canada) rform screening the /Evaluation/ the radiologists BC, pegn Level annual (in eenir under ande Continued scdnuor area Intervention Volume Determined between volum formance AUC= 2-4 dent Inci a NOTE: al.
|
From page 42... ...
Being the owner of the practice, increased use of diagnostic imaging and interventional procedures, and double reading were associated with increased accuracy, while presence of a computerized system to monitor and track screening, facility classification as hospitalbased radiology department or multispecialty medical clinic (compared to breast diagnostic center, mammography screening center) , and presence of a formal pathology correlation conference were negatively associated with accuracy.
|
From page 43... ...
Based on discussions with these investigators and a neutral biostatistician, Anna Tosteson, Sc.D., who is an expert in this field, the Committee concludes that there were reasonable arguments for each analytic technique, but that regardless of which method was chosen, neither showed a significant influence of volume on overall accuracy. More study is needed to establish the implications, advantages, and disadvantages of statistical approaches to evaluating the influence of volume on interpretive performance.
|
From page 44... ...
. The Committee discussed the potential impact of a modest increase in interpretive volume to 1,000 per year, and concludes that this increase alone was unlikely to change interpretive performance or to facilitate the ability of interpreting physicians to selfassess true-positive or false-negative interpretations.
|
From page 45... ...
DOUBLE-READING METHODS AND TECHNICAL TOOLS DESIGNED TO IMPROVE PERFORMANCE Double Reading One approach to improving interpretive performance is double reading. This approach may take several forms, but the two extremes include: (1)
|
From page 46... ...
found no overall difference in cancer detection rates among breast imaging specialists in academic practice (cancer detection rate of 3.49/1,000 without CAD versus 3.55/1,000 with CAD, p=0.68)
|
From page 47... ...
THE IMPACT OF RESIDENCY/FELLOWSHIP TRAINING AND CME ON INTERPRETIVE SKILLS The effectiveness of screening mammography greatly depends on the skills of the personnel interpreting the images. A portion of MQSA, consequently, addresses ways to ensure that physicians interpreting mammograms are adequately trained.
|
From page 48... ...
Insofar as all practicing interpreting physicians have been required by MQSA regulations to obtain substantial amounts of CME since 1994, these results do not address the ability of
|
From page 49... ...
Funding should be provided for comprehensive research studies on the impact of various existing and innovative teaching interventions on mammography interpretive skills. THE INFLUENCE OF SKILLS ASSESSMENT AND FEEDBACK ON PERFORMANCE Overview Theoretically, assessment via medical audits is designed to link practice patterns to patient outcomes in a way that can influence provider behavior and performance.
|
From page 50... ...
. For mammography, performance data on individual interpreting physicians may be misleading without adequate consideration of patient and physician characteristics (Elmore et al., 2002)
|
From page 51... ...
. Interpreting physicians, who undergo specific training in order to participate in the NHSBSP, are reported to be satisfied with these monitoring and review processes.
|
From page 52... ...
This system collects data on truepositive rate, false-positive rate, positive predictive value, and cancer detection rate, and evaluates the data in aggregated form for every central screening facility that interprets mammograms. In addition, a small contingent of experts from the National Training and Expert Center for Breast Cancer Screening conducts onsite audits of every interpretation facility once every 2 to 3 years.
|
From page 53... ...
, 15 of 16 anonymous respondents agreed that if they had the opportunity to revisit their choice to join the group, they would do so again. CHALLENGES TO USING MEDICAL AUDIT DATA TO IMPROVE INTERPRETIVE PERFORMANCE IN THE UNITED STATES A Lack of Data and Information to Guide Audit and Feedback U.S.
|
From page 54... ...
. The Committee does not recommend mandatory proficiency testing via selfassessment exams for all interpreting physicians at present because the available testing procedures have not been rigorously evaluated and proven to have a direct positive impact on interpretive performance in clinical practice.
|
From page 55... ...
The radiographic images can be magnified and panned on the computer screen, and the location of abnormal findings on each displayed image can be identified with a mouse click. The images are accompanied by multiple-choice questions that test important aspects of breast imaging practice.
|
From page 56... ...
, its mechanisms for encrypting data preclude the ability to identify individual performance of interpreting physicians or facilities, and its focus on the facilities precludes its use as a national repository in its current format. Although the individual registries that contribute data to the BCSC do collect radiologist-specific and facility-specific data that could be used for quality improvement, the BCSC was not intended to be used for quality assurance purposes and facilities may be less likely to participate in the BCSC if data are used for purposes other than research.
|
From page 57... ...
. But a pilot study that used the same methods followed by NSQIP to provide quality improvement to the surgeries performed by three academic hospitals found it to be a feasible and valid system that is applicable to non-VA medical centers (Waynant et al., 1999; Khuri et al., 2002)
|
From page 58... ...
Privacy Rule offers extensive safeguards to ensure the confidentiality of medical information. But it exempts from its stringent requirements the disclosure of protected health information that is used for "conducting quality assurance and quality improvement, including outcomes evaluation and development of clinical guidelines," provided that the work is mainly intended to improve the operations of a specific organization rather than for research (FDA, 2004; Gunn et al., 2004)
|
From page 59... ...
. Batch mode interpretation of screening mammograms allows interpreting physicians to focus their attention and allows interpretation of screening films in a quieter environment with fewer distractions and less ambient light, factors that are important for the conspicuity of subtle lesions.
|
From page 60... ...
Current Radiology Information Systems Used by Mammography Facilities Following implementation of MQSA regulations, several computerized mammography management systems were developed to assist mammography facilities in collecting, organizing, and linking mammography information to assist with reporting requirements. These include but are not limited to: · Insight Radiology Information System · PenRad · Amber Diagnostics Radiology Management Systems · iCAD Radiology Information Management · OmniCare Mammography Management Systems
|
From page 61... ...
Sickles, M.D., Chief of Breast Imaging, University of California, San Francisco, School of Medicine, October 15, 2004. 5 Personal communication, B
|
From page 62... ...
) , in its current form it does little more than burden facilities with processes that generate data that are inadequate for ensuring or improving interpretive performance.
|
From page 63... ...
Thus, an inherent component of this advanced audit program is the creation of a centralized data and statistical coordinating center, where standardized pooled data are electronically compiled, analyzed, and reported back to participating facilities to provide the type of meaningful feedback that is given in organized screening programs in some other countries. Statistical coordinating center staff should be qualified to collect and maintain data from disparate sources, standardize data collection procedures, conduct accurate advanced-level audits, provide feedback, and help develop, implement, and evaluate selfimprovement plans for interpreting physicians or facilities that do not achieve performance benchmarks.
|
From page 64... ...
BREAST IMAGING CENTERS OF EXCELLENCE As noted previously, centralized breast cancer screening programs with extensive interpretive quality assurance activities currently operate in several other countries, and some organized screening programs have been established in the United States as well. Although evidence is lacking to assess the impact of individual elements of these programs, when implemented together, they appear to be effective for improving interpretive performance and quality assurance.
|
From page 65... ...
Centers would thus have the expertise to develop and host on site training programs in breast imaging -- for mammography as well as other imaging modalities. Programs could be tailored for initial training and CME, as well as personalized training for interpreting physicians whose medical audit results indicate that they need to improve their skills.
|
From page 66... ...
An interdisciplinary approach to breast cancer detection and treatment may be especially important in the United States, with its fragmented and specialized delivery of medical care. Initially, this led to the development of freestanding, comprehensive breast care programs such as the Van Nuys (California)
|
From page 67... ...
THE NEED FOR A SUPPORTIVE ENVIRONMENT TO PROMOTE QUALITY IMPROVEMENT Increased regulations alone may cause facilities to focus on meeting mandatory minimum requirements rather than motivating them to strive for maximal quality assurance and improvement that could additionally benefit the public's health. Supportive elements to help improve interpretive performance may not be as easily implemented as regulations, but in their absence, new requirements may not manifest meaningful improvements and would be viewed primarily as an added burden by mammography facilities and personnel.
|
From page 68... ...
. Thus, the Committee believes that it is essential to provide positive incentives for facilities and individual interpreting physicians to aim for the highest level of quality assurance.
|
From page 69... ...
The program will investigate shared decision making for patient-centered care, redesign of care networks to focus on outcome improvement, methods to reduce practice variation, and quality incentives. The Physician Group Practice Demonstration combines fee-for-service payments with incentive programs to reward group practices for financial performance and quality improvement.
|
From page 70... ...
Medical audits should be designed to link practice patterns to patient outcomes in a way that can influence provider performance. Interpreting physicians need to understand their current level of performance before they can take action to improve their interpretive accuracy.
|
From page 71... ...
While there is some evidence to suggest that these approaches could also improve the quality of mammography interpretation, the data available to date are insufficient to justify changes to MQSA legislation or regulations. However, the Committee recommends that additional studies be rapidly undertaken to develop a stronger evidence base for the effects of CME, reader volume, double reading, and CAD on interpretive performance.
|
From page 72... ...
2002. Does training in the Breast Imaging Reporting and Data System (BI-RADS)
|
From page 73... ...
2004. Can computer-aided detection with double reading of screening mammograms help decrease the false-negative rate?
|
From page 74... ...
2004. Independent double reading of screening mammograms in the Netherlands: Effect of arbitration following reader disagree ments.
|
From page 75... ...
2003. Increase in cancer detection and recall rates with independent double interpretation of screening mammography.
|
From page 76... ...
2005. Saving Women's Lives: Strategies for Improving Breast Cancer Detection and Diag nosis.
|
From page 77... ...
1997. The effect of hormone replacement therapy on recall rate in the National Health Service Breast Screening Programme.
|
From page 78... ...
2003. Interpretive skills in the National Health Service Breast Screening Programme: Performance indicators and remedial measures.
|
From page 79... ...
In: Kopans DB, ed. Categorical Course in Breast Imaging.
|
From page 80... ...
Submitted. Testing the effect of computer assisted detection upon interpretive performance in screening mammography.
|
From page 81... ...
2001. Association of recall rates with sensitiv ity and positive predictive values of screening mammography.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.