Graduate Medical Education Outcomes and Metrics Proceedings of a Workshop (2018) / Chapter Skim
Currently Skimming:
Proceedings of a Workshop
Proceedings of a Workshop
Pages 1-60
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 1... ...
The current lack of established GME outcome measures limits our ability to assess the impact of individual graduates, the performance of residency programs and teaching institutions, and the collective contribution of GME graduates to the physician workforce. Much of GME is based on tradition, regulatory requirements, and hospitals' care delivery needs rather than on evidence of effective education (IOM, 2014)
|
From page 2... ...
To examine these opportunities and challenges in measuring and assessing GME outcomes, the Board on Health Care Services of the National Academies of Sciences, Engineering, and Medicine held a workshop, Graduate Medical Education Outcomes and Metrics, on October 10–11, 2017, in Washington, DC. Workshop participants discussed • meaningful and measurable outcomes of GME; • possible metrics that could be used to track these GME outcomes; • possible mechanisms for collecting, collating, analyzing, and re porting these data; and • further work to accomplish this ambitious goal.
|
From page 3... ...
Ten thousand GME residency programs and hundreds of institutions that sponsor them function within a common set of rules, are governed by the same group of oversight organizations, are funded largely by the same mechanism, and share a fundamental purpose. However, residency programs and teaching institutions are largely independent actors without coordination.
|
From page 4... ...
(Rajkomar) • Use direct feedback from trainees to assess residency programs.
|
From page 5... ...
Achieving Buy-In for a National GME Database • Define the purpose of a national GME database and effectively com municate that to trainees, residency programs, teaching institutions, and professional organizations. (Conklin)
|
From page 6... ...
Rather, Weinstein suggested that national goals should be met collectively, without each residency program and teaching institution being expected to contribute to every goal of GME. Privacy issues relating to data regarding individual physicians, residency programs, and teaching institutions would need careful consideration; these issues are the focus of a workshop session.
|
From page 7... ...
Why Now? Despite the obstacles and challenges, the time to develop metrics to measure GME outcomes is now for several reasons, Weinstein asserted, including the following: • GME could capitalize on rapid advances in health information technology, big data, and data science.
|
From page 8... ...
Several participants acknowledged that a fundamental challenge of focusing on clinical outcomes is identifying the most meaningful outcome metrics. For example, Donald Brady from the Vanderbilt University Medical Center noted that patient outcomes can be measured at different time points, but the longer the time frame, the less likely the outcome is to be directly associated with a particular clinician or program.
|
From page 9... ...
. GME residency programs influence the specialty mix, but they are more focused on current hospital staffing needs than the national workforce needs.
|
From page 10... ...
At present, though, a focus on producing physicians who practice high-value health care across institutions is not commonly on the agenda for GME meetings. Residency programs vary significantly, from academically oriented programs that include a goal of training practitioners across a broad spectrum of specialties, to physician scientists and others with unique expertise, to community-based programs that focus on training primary care physicians to serve the community.
|
From page 11... ...
asked the panel whether requirements at the residency review committee level, including service obligations and location of training, which she said is often still limited to inpatient care, stifles innovation in GME. Mullan said that medical centers place a high priority on meeting accreditation standards and performing well in reviews by the Accreditation Council for Graduate Medical Education (ACGME)
|
From page 12... ...
Changing who receives the money was a recommendation of the 2014 report, responded Wilensky, although little progress has been made in that regard. Many other participants, such as teaching health centers, are important and should also be able to receive the fund ing, Wilensky said.
|
From page 13... ...
Finally, Step 3 also includes clinical case simulations of managing a patient. To develop these examinations during any given year, more than 300 faculty members in various disciplines from medical schools across the country create items for the examinations, review the performance of items that have been created previously, and participate in the management and governance of the USMLE program.6 Although the USMLE's primary purpose is as a licensing examination, most students consider it a residency placement examination because of the critical role USMLE scores play in the selection of candidates by residency programs.
|
From page 14... ...
He said the cautionary lesson from the Data Commons example is the need to consider regulatory and market forces as both carrots and sticks to promote use of future systems. The Pediatric Milestone Assessment Collaborative An encouraging example of successfully measuring a resident's progress, said Katsufrakis, is the Pediatric Milestone Assessment Collaborative 10 See https://www.nbme.org/PDF/Publications/Examiner-2013-SpringSummer.pdf (accessed July 31, 2019)
|
From page 15... ...
GME training model, Holmboe described how the five Dreyfus levels (Dreyfus and Dreyfus, 1980) of the ACGME milestones translate for residency programs, including (1)
|
From page 16... ...
Lawrence Smith from the Donald and Barbara Zucker School of Medicine at Hofstra/ Northwell raised a question about the reliability of measurements, asking if an assessment from one program is comparable to another program, and recognizing that the evaluator's frame of reference may also differ. olmboe H responded that the assessment framework between residency programs is a challenge and that researchers are trying to identify ways to improve Milestone assessments.
|
From page 17... ...
The Milestones are co-branded, owned by both ACGME and the American Board of Medical Specialties (ABMS) , Nasca responded, noting that they are working with LCME to provide Milestone data to medical schools for their graduates.
|
From page 18... ...
The Bureau supports stipends and other payment costs associated with GME through two types of programs: GME grants for preventive medicine residency programs, and GME payment programs, including the Children's Hospital GME (CHGME) and Teaching Health Center GME (THCGME)
|
From page 19... ...
Proposed legislation to continue THCGME funding would add more reporting requirements, including the number of patients served by THCGME programs and retention in primary care and underserved primary care settings. aSee https://www.congress.gov/113/plaws/publ98/PLAW-113publ98.pdf (ac cessed February 15, 2018)
|
From page 20... ...
Ultimately, connecting NPI numbers with Medicare and Medicaid claims would allow longitudinal tracking and insight into whether individuals trained in these GME programs continue to provide care for these critical populations, Chen said. Finally, there is currently no mechanism to link funding to GME metrics, she said.
|
From page 21... ...
examination administered during residency. Providing evidence that board certification makes a difference, Nora cited research showing that patient outcomes across a wide variety of clinical conditions are better when care is provided by an ABMS board-certified physician (Hawkins et al., 2013; Sharp et al., 2002)
|
From page 22... ...
This survey provides feedback to residency programs about how prepared their graduates thought they were for practice, and what they are doing in practice. The Family Medicine Review Committee requires surveys of graduates, but Phillips said that this has not been done rigorously in the past; the ABFM–AFMRD partnership provides greater standardization through their survey, which had a 67 percent response rate in its first year.
|
From page 23... ...
Too often, process measures are not well correlated, at least at a provider level, with patient outcomes, said Werner. For example, when Medicare began rating hospitals on the quality of care they delivered, they did so using a series of process measures (Werner and Bradlow, 2006)
|
From page 24... ...
One risk to patients could result from "cream-skimming," in which risk adjustment of patient outcomes is imperfect because providers treat patients who appear healthier, intending to maintain a high-quality rating on these outcome metrics, Werner continued. Because the patients receiving treatment are healthier to begin with, there is an appearance of improved
|
From page 25... ...
. The opposite was true, however, in a similar unpublished study of surgeons, implying that a surgeon's years of experience correlated with improved patient outcomes.26 To control for 24 See https://www.optumlabs.com (accessed December 20, 2017)
|
From page 26... ...
Assessing residency programs and medical schools on the basis of patient outcomes rather than processes is a useful avenue to consider, but there are challenges, he said. One challenge is determining whether the patient outcomes are due to residency training or to the types of doctors who choose particular residency programs.
|
From page 27... ...
. Other studies of "click 27 See https://developer.amazon.com/alexa (accessed December 20, 2017)
|
From page 28... ...
Caverzagie said the second metric focuses on value of care (patient outcomes per cost of care) from the perspectives of both the public and the profession.
|
From page 29... ...
• Clinical learning environments • Communities of commitment • Expectations of accreditation • Graduate outcomes • Value Federal-, State-, and Regional-Level Metrics for Self-Regulators, Policy Makers, Teaching Institutions, Patients, and Trainees (summarized by Norcini and Warchol) • Board certification • Care for underserved populations • Clinical outcomes • Geographic distribution • Patient access to care (specialty distribution)
|
From page 30... ...
Many group participants suggested specialty board certification rates as a third metric, which are already used in practice as a metric across most residency programs, said McDonald. Strengths of this metric include public accountability, easy measurability, availability, and connection to many other metrics of good care.
|
From page 31... ...
Additional important metrics include board certification rates and MOC. The strength of reporting these metrics, said Glover, includes implications for the public, especially because the residency programs are publicly funded.
|
From page 32... ...
National-, Regional-, or State-Level Metrics John Norcini from the Foundation for Advancement of International Medical Education and Research and Jordan Warchol from The George Washington University addressed the topic of national-, regional-, and state-level metrics by presenting metrics that their breakout group suggested for four primary stakeholders, including self-regulators, such as boards and specialty societies, state and federal policy makers, patients, and trainees. For self-regulators, Norcini and Warchol described five metrics, including board certification, patient outcomes, physician joy in practice, physician sociobehavioral skills, and professionalism and disciplinary action.
|
From page 33... ...
CAN OUTCOME MEASUREMENT PROVIDE A PATH TOWARD EVIDENCE-BASED EDUCATION? In discussing how outcome measurements could guide evidence-based education going forward, panelists provided examples of how they are using or could use data to modify their curricula, create innovative educational programs, and guide and assess how residency programs are meeting society's medical workforce needs.
|
From page 34... ...
For example, the system can link a first-year medical student's multiple-choice answer with the clinical orders the student later Institution A Institution B Institution C Medical School Training Clinical Practice Exams and Evaluations CME Performance Competency Portfolios MCAT / Admissions NRMP ACGME USMLE Boards and licensure Curriculum Clinical patient outcomes Warehouse of longitudinally integrated learner data FIGURE 2 The medical education data continuum. NOTE: ACGME = Accreditation Council for Graduate Medical Education; CME = continuing medical education; MCAT = Medical College Admission Test; NRMP = National Resident Matching Program; USMLE = U.S.
|
From page 35... ...
However, these analyses are limited to the 20–30 percent of students remaining at NYU after residency, Triola added. To track graduates after they leave NYU, Triola and his team integrated several databases, including the AMA Masterfile32 for NYU, a database containing identifiable data for 8,500 current and former NYU medical students and 12,000 NYU GME graduates; CMS Physician Compare,33 an open database of doctors participating in Medicare that includes their NPI, medical school, and graduation year; the NPI database34; CMS Part D prescription database35; CMS usage and payment database36; and New York State SPARCS (Statewide Planning and Research Cooperative System)
|
From page 36... ...
In summary, Triola emphasized the opportunities for using currently available data to support evidence-based education. These data can inform curricular planning, transition to residency programs, mentoring and
|
From page 37... ...
Kelz emphasized the importance of targeting residency programs to implement improved surgical practice because targeting the 268 surgical residency programs is more efficient than attempting to reach more than 135,000 surgeons who practice at more than 5,000 hospitals40 (see Figure 4)
|
From page 38... ...
. Another study focused on general surgery found a 10 percent difference between low- and high-ranking surgical residency programs but also found significantly greater variation among surgeons with 5 years in practice compared to those with 10 to 20 years in practice (Bansal et al., 2016)
|
From page 39... ...
PROCEEDINGS OF A WORKSHOP 39 FIGURE 5 A framework for the assessment of graduate medical education. NOTES: Tier 1 includes standard outcome measures of residency training programs, such as certification and credentialing.
|
From page 40... ...
As a result, a self-sustaining community program run through the surgical interest group at the Perelman School of Medicine now brings health care–related information focused on surgical problems into community centers, houses of worship, and high schools to improve the health of the population that they serve. There are real opportunities like this to get physicians into the community and into safety net hospitals through undergraduate medical education and GME programs to improve care through patient and caregiver education before health services are needed, Kelz said.
|
From page 41... ...
The GME Mapper allows every institution to see its primary care and rural physician outputs over time, and provides a ranking of residency programs. For example, data from the GME Mapper shows major differences in two hospitals in Manhattan; at Mount Sinai, 26 percent and 7.6 percent of graduates went into primary care and rural settings, respectively, compared with only 8.6 percent and 1.4 percent at New York Presbyterian.
|
From page 42... ...
With future studies, Phillips and his team hope to identify training program components with these effects, and create interventions to make positive changes to residency programs. Phillips emphasized the opportunities to make evidence-based changes to GME through purposeful faculty development and team-based training.
|
From page 43... ...
A substantial amount of data is available on the physician workforce, but the key questions pertain to future planning. She said one potential goal is to address predicted physician shortfalls, noting that AAMC uses data on the effects of managed care, millennial recruitment, and physician retirement to project physician shortfalls.
|
From page 44... ...
The state holds a very critical role, in addition to the federal government, in the discussion of how to deal with the physician workforce in that particular state. Orlowski also provided an example of a state with effective statewide collaboration efforts to meet workforce needs.
|
From page 45... ...
. Orlowski concluded by noting that in the early 2000s, AAMC called for an increase of 30 percent in the number of matriculates in medical schools, and undergraduate medical education will exceed that goal (AAMC, 2017)
|
From page 46... ...
States were selected based on having undertaken efforts to reform Medicaid GME. One finding was that the impetus for GME reform stems from multiple sources, said Fraher, and included maldistribution of physicians by specialty, geography, and setting; ensuring that there are enough GME slots to match medical school expansions; potential loss of Teaching Health Center funds; and disparities in GME funding received by different training institutions in the state.
|
From page 47... ...
Lisk Health Policy Analysis asked about the need to differentiate how residency programs are funded. Funds should be targeted, said Orlowski.
|
From page 48... ...
Hoyt responded that the American College of Surgeons currently has deidentified databases assoiated c with each of its 2,700 surgical training programs.47 Research and focused feedback of information valuable to individual residency programs is a reward for providing those data to a consortium, said Smith, as he expressed support for making data available for research purposes. Several participants discussed the benefits of data sharing and transparency from the perspective of stimulating quality improvement and reducing cost.
|
From page 49... ...
Pilot projects are relatively flexible, allowing learning and modification as the project progresses, which can help inform the design of larger subsequent projects, she said. For example, the ACGME Milestone Project started out as a pilot, beginning with only three residency programs and then adding another four to enable learning over time before including
|
From page 50... ...
The data should be protected to prevent misuse of data, and pilot projects could help mitigate unintended consequences, said Conklin. He added that some data sharing would likely need to be mandatory and global, while other data could be selected as relevant to the mission of the teaching institution or residency program.
|
From page 51... ...
For example, rankings about which residency programs generate the most physicians who go into rural practice or which programs have the most contact with lowincome urban patient populations should be available because such data could influence residency selection and funding decisions, said Anderson. How Should a National Database Be Maintained and Funded?
|
From page 52... ...
One option for governance is through HHS, including CMS and the Center for Medicare & Medicaid Innovation, because they are paying for GME through Medicare and Medicaid. CMS also already manages extensive data on GME programs, but it is a different role to develop a database that would be shared with the community, Salsberg noted.
|
From page 53... ...
and to evaluate resources. One critical challenge, noted William Pinsky from the Educational Commission for Foreign Medical Graduates and the Foundation for Advancement of International Medical Education and Research, is to identify a neutral body to convene the organizations.
|
From page 54... ...
Nonetheless, assessing patient outcomes is challenging because they are confounded by complexities such as patient risk and the clinical learning environment, he said. Steven Shannon from the American Association of Schools of Osteopathic Medicine emphasized the importance of workforce outcomes in relation to the societal support provided to GME and the need to look at both team-based and individual learning outcomes.
|
From page 55... ...
Third, Thibault reviewed the need for meaningful metrics at the level of trainees, residency programs, and teaching institutions, as well as at the national level. This includes information about what graduates are doing and how they are performing across clinical and academic settings.
|
From page 56... ...
Thibault also described his work with the Macy Faculty Scholars Program,50 which promotes the careers of innovators in nursing and medical education. Finally, the seventh theme of the discussions focused on the concept of a data repository for assessing GME outcomes, said Thibault, noting that two questions came up repeatedly: Where and how should a repository be established?
|
From page 57... ...
2009. Evaluating obstetrical residency programs using patient outcomes.
|
From page 58... ...
Accreditation Council for Graduate Medical Education. http://www.acgme.org/Portals/0/ PDFs/Milestones/MilestonesAnnualReport2017.pdf?
|
From page 59... ...
Journal of Graduate Medical Education 9(1)
|
From page 60... ...
Recom mendations for reforming graduate medical education to meet the needs of the public. The Second of Two Conferences -- the Content and Format of GME.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.