Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 143
Part II
INTRODUCTION
In Part II of this report, the Committee on Redesigning Health Insur-
ance Performance Measures, Payment, and Performance Improvement Pro-
grams presents a description of the Quality Improvement Organization
(QIO) program in the 7th and 8th scopes of work (SOWs). Part I stressed
the importance of looking to what needs to be done in the future to improve
the quality of health care. Part II presents the basis for those recommenda-
tions and documents the current status of the program. It will serve as a
resource for future examinations of the QIO program. Part II includes eight
chapters that provide the information outlined below.
Chapter 6 describes the study approach. In particular, it explains the
various methodologies used and the limitations encountered in accumulat-
ing information about the QIO program from stakeholders.
Chapter 7 examines the overall structure and financing of the QIO pro-
gram. It evaluates the structural requirements, governance, and staffing of
the individual QIOs, organizations holding QIO contracts, and the QIO
Support Centers (QIOSCs). The funding for the entire program is also pre-
sented, including funding for the core contracts, special studies, and sup-
port contracts. The external activities and funding of the QIOs are also
described.
Chapter 8 provides a detailed description of the technical assistance
activities required of the QIOs in the 7th and 8th SOWs. First, the methods
used to provide technical assistance are discussed in general, as well as how
143
OCR for page 144
144 MEDICARE'S QUALITY IMPROVEMENT ORGANIZATION PROGRAM
QIOs recruit providers for these activities. Next, the chapter presents a
detailed description of those activities in each setting.
Chapter 9 describes the impact of technical assistance on quality im-
provement and knowledge transfer. The chapter includes a literature review
and considers quality improvement interventions in general and those of
QIOs specifically, discusses different approaches to quality improvement,
and provides an overview of knowledge transfer and its impact in the QIO
program.
Chapter 10 evaluates the impact of quality improvement activities in
the QIO program specifically. The detailed evaluation formulas as well as
the means used to identify high performing QIOs are discussed.
Chapter 11 describes the communications and beneficiary education
activities of the QIOs in the 7th and 8th SOWs. This includes the promo-
tion of public information on websites, assistance to providers for the col-
lection and reporting of data for public reporting, and the maintenance of
toll-free help lines.
Chapter 12 describes the protective activities of the QIOs. All activities
related to the review of quality concerns (including the new mediation pro-
cess) as well as reviews designed to protect the integrity of the Medicare
Trust Funds, such as the Hospital Payment Monitoring Program, are exam-
ined. The different categories and types of reviews and the review process
are described in detail, and data that can be used to help assess their impact
are presented.
Chapter 13 describes the Centers for Medicare and Medicaid Services'
(CMS's) oversight of the QIO program. The structure of the QIO program
as it fits into CMS's framework is first described, including the offices and
the personnel involved in the management of this complex program. Next,
the committee describes the various communications methods that these
personnel used, including their written, oral, and electronic communica-
tions systems. The data-related issues evaluated include the various tools
that CMS uses to report data and the flow of data in each of the provider
settings. The manner in which the QIO and QIOSC contracts themselves
are structured and awarded is then examined. Finally, the chapter discusses
the overall guidance of the QIO program, including strategic planning,
policy direction, program coordination, and overall program evaluation.
Part II provides a comprehensive, retrospective view of the QIO pro-
gram in the 7th and 8th SOWs. The assessment presented here serves as a
primary source of information about this complex program, with a descrip-
tion of its recent past and current status, as well as a base on which recom-
mendations for the role of QIOs in the future of quality health care have
been made.
OCR for page 145
6
Study Approach
CHAPTER SUMMARY
This chapter presents the committee's methodological approach for
this study. To address the items in the legislative mandate, it was
necessary to design a multifaceted study that used a mix of data
collection and analytic methods, including both qualitative and
quantitative analyses, rather than an experimental design. The
chapter discusses each of the main data sources, including: a fo-
cused literature review, an internet-based data collection, site visits,
interviews, and quantitative data requests. Finally, the research
strengths and challenges are examined.
In response to the legislative mandate in Section 109 of the Medicare
Prescription Drug, Improvement, and Modernization Act of 2003 (P.L. 108-
173), the Institute of Medicine (IOM) formed an authoring committee, the
Committee on Redesigning Health Insurance Performance Measures, Pay-
ment, and Performance Improvement Programs, to prepare three reports in
a single project and a separate subcommittee to focus on the Quality Im-
provement Organization (QIO) study and guide the data collection and
analysis. These committees included members with the necessary expertise
and experience in such areas as clinical quality improvement; health eco-
nomics; Centers for Medicare and Medicaid Services (CMS) management;
research on organizational change, consumer behavior, and program evalu-
ation; and the delivery of health care in physicians' offices, long-term care
facilities, hospitals, academic medical centers, and integrated health care
delivery systems. During the course of the study contract (April 2004 to
June 2005), the members of the QIO subcommittee met three times in per-
son and once by conference call to guide the research and data collection
efforts. The main committee met seven times and gave policy direction to
the research, discussed drafts, and reached consensus on the final wording
of the report and its recommendations.
145
OCR for page 146
146 MEDICARE'S QUALITY IMPROVEMENT ORGANIZATION PROGRAM
To discuss the items mandated by the legislation, such as an overview
of the QIO program, the duties of the contracting organizations in the pro-
gram, the sources and amounts of their funding, and a review of the over-
sight of the program, the committee needed to present descriptive data. To
address the other items in the mandate, such as the extent to which QIOs
improve the quality of care, the extent to which other organizations could
do their duties, and the effectiveness of case review and other QIO activi-
ties, the committee needed evaluative analyses and expert judgment.
This study uses a retrospective examination of the data to present much
of the descriptive information as well as to evaluate the impact of the
program's efforts. Although the chapters with a retrospective view examine
the data from the beginning of the Medicare quality assurance program to
provide an overview, the main focus is on more recent program activities;
and most data and analyses relate to the time period of the contract for the
7th scope of work (SOW), which was from August 2002 to August 2005.
This study also looks at the program concurrently, because the period of
the IOM study contract overlapped with the end of the 7th SOW and the
beginning of the 8th SOW. The information presented in this report reflects
the data collected through the midsummer of 2005 and is descriptive of the
8th SOW but does not include data reflecting the CMS and QIO work
produced during the 8th SOW, which had barely begun when the report
was drafted.
The unit of analysis varied, depending on the questions addressed. For
example, the committee wanted to know what, if any, impact the whole
program has had on Medicare beneficiaries' care and considered informa-
tion reflecting the activities of all QIOs, other contractors, and CMS. Thus,
the committee used aggregated data from all QIOs to examine possible
national effects. When the committee examined the effects of specific qual-
ity improvement interventions on the provision of care and care outcomes,
a more common focus of the published literature, the unit of analysis was
often an individual QIO or a particular intervention project of one or more
QIOs. The committee tried to identify the organizational and structural
characteristics of the QIOs that might contribute to greater quality improve-
ment. That is, were there characteristics of high-performing QIOs or as-
pects of how CMS managed them that could be identified and spread to
other QIOs to help them improve the national program's overall perfor-
mance? Comparisons of QIOs' structural characteristics and performance
measures were necessary to attempt to answer those questions.
DATA SOURCES
The committee chose to gather data from a wide variety of sources and
to compare and integrate the findings, as no one source fully covered the
OCR for page 147
STUDY APPROACH 147
many facets of the QIO program and the questions being asked in the study.
Fortunately, the committee found both the QIO community and CMS wel-
coming, willing to provide reasonable requests for data, and open and eager
to talk with the committee about the program. By the end of the study, the
IOM project had both personal and substantive e-mail contacts with the
chief executive officer (CEO) from each organization holding a QIO con-
tract. The main data collection methods and sources for this study included
the following:
· a focused review of the literature on the impact of quality improve-
ment and the impact of QIO efforts at quality improvement;
· data collected in 2005 from all 53 QIOs through the SurveyMonkey
internet-based data collection tool;
· quantitative analyses of QIOs' relative performance on various
subtasks of the 7th SOW;
· site visits by 16 committee members and IOM staff to 11 QIOs and
one CMS Regional Office;
· telephone interviews with 20 CEOs from randomly selected QIOs;
· a 3-day briefing by CMS staff;
· a half-day workshop for the committee and subcommittee, including
formal presentations from academic researchers as well as experts working
in quality improvement in the field and comments from various stake-
holders;
· access to QIONet, a CMS internal website for the QIO program that
includes performance data by state on the Dashboard section;
· face-to-face interviews with CEOs and staff from certain QIOs and
randomly selected QIO Support Centers (QIOSCs);
· a focus group discussion with 11 QIO CEOs;
· specific requests for data from CMS staff;
· many formal and informal discussions with the staff of the American
Health Quality Association (AHQA; the national organization representing
all QIOs) and the attendees at their functions (mainly QIO staff);
· informal discussions with representatives of consumer and benefi-
ciary organizations and various providers;
· data collected from presentations and question and discussion ses-
sions; the observation of interactions among QIO and CMS staff; and in-
formation from the informal questioning of attendees at AHQA's annual
technical conference and other national meetings, CMS's annual QualityNet
Conference, and miscellaneous smaller meetings; and
· suggestions from committee and subcommittee members and review-
ers of businesses and other organizations performing QIO-like functions
("other entities").
OCR for page 148
148 MEDICARE'S QUALITY IMPROVEMENT ORGANIZATION PROGRAM
It is noteworthy that the committee gained knowledge of and insight into
the QIO program through multiple sources, all of which could not be fully
referenced in the text. The committee attempted to indicate the main source
of information throughout the report, as appropriate. Each of the formal
research approaches is briefly discussed below.
FOCUSED LITERATURE REVIEW
The committee searched for an understanding of how quality improve-
ment interventions with health care providers influence the quality of health
care, and it examined literature chosen from two sources. (The committee's
previous report, Performance Measurement: Accelerating Improvement,
addressed quality improvements resulting from the availability of health
care quality data for providers and the public reporting of such data [IOM,
2006].) First, the committee extracted the majority of studies from a bibli-
ography compiled by AHQA in August 2004. This bibliography lists more
than 600 studies that either were conducted in partnership with a QIO,
used QIO data, or were written by an author affiliated with a QIO. Poten-
tial biases may have existed in the selection of the studies for the bibliogra-
phy and the methods by which they were conducted. Although most articles
in AHQA's bibliography refer to clinical treatment options for Medicare
patients and descriptions of the QIO program, the committee focused on a
smaller subset of studies that examined QIO approaches to improving the
quality of health care. Articles were selected from AHQA's bibliography on
the basis of their titles by use of an internal reliability check by two mem-
bers of the IOM staff.
The committee found most of the remaining studies by the use of
Medline searches, performed in November 2004 for articles published in
the previous 5 years, using the following medical search headings: "quality
improvement," "effectiveness," and "intervention." In addition, the com-
mittee evaluated a selection of studies to supplement the evidence base for
outcome measures, collaborative studies, audits and feedback, and evalua-
tions of the QIO program published from November 2004 through July
2005. The committee chose studies from references in previously selected
studies and by searching the Cochrane Database of Systematic Reviews and
the Agency for Healthcare Research and Quality systematic reviews. These
studies did not focus specifically on QIO activities and are part of a broader
literature on quality intervention. It would be impossible to know whether
the health care providers in those studies had prior knowledge of or had
participated in various QIO efforts. The literature review is discussed in
more detail in Chapter 9, and a summary of selected studies on quality
improvement is provided in Table A.1 in Appendix A.
OCR for page 149
STUDY APPROACH 149
The committee categorized the studies obtained from the literature re-
view on the basis of the strength of the research methodology (see
Table A.2). In general, the results and the lessons learned from randomized
controlled trials outweighed those from studies with nonexperimental de-
signs. However, the committee recognized the value of observational and
qualitative studies for identifying major themes from past interventions and
considered them useful starting points for understanding the impacts of the
previous interventions.
WEB-BASED DATA COLLECTION
In 2005, the committee, along with a consultant, Allyson Ross Davies,
used software on the SurveyMonkey.com platform to design a two-part
data collection instrument, collect responses from all organizations holding
one or more QIO contracts for the 7th SOW, and export raw data for
analysis using SAS software. The main committee and the QIO subcommit-
tee developed draft content. Key staff at CMS and AHQA and a CEO of an
organization holding multiple QIO contracts for the 7th SOW reviewed a
draft of the data collection instrument for clarity, terminology, user friend-
liness, and duplication of ongoing data collection efforts. The committee
selected the SurveyMonkey platform because of both the ease of online
questionnaire design and the familiarity of the QIO community with this
survey platform. Both CMS and AHQA actively encouraged the QIOs to
fully participate in this data collection effort.
The committee assured the respondents that they would remain anony-
mous, and SurveyMonkey guaranteed the secure transmission of responses
through the use of Secure Sockets Layer encryption. The consultant ex-
ported raw data from the SurveyMonkey website for analysis, at which
point she closed the data collection instruments and deleted them from the
site. She did not share raw data with IOM staff, the committee, or CMS.
The consultant presented only deidentified, aggregated information about
the organizations to the project's main committee and the QIO subcommit-
tee. This report includes only deidentified and aggregated data.
Content of Data Collection Instrument
Part 1 of the data collection instrument obtained information from each
of the 41 organizations holding one or more QIO contracts for the 7th
SOW. The questions in Part 1 focused on organizational structure; boards
of directors or trustees; and financial information, which was related to
both the QIO contract(s) and other business lines, when relevant.
Part 2 requested separate data on the 7th SOW from each of the 53
QIOs. Background questions focused on factual information about the or-
OCR for page 150
150 MEDICARE'S QUALITY IMPROVEMENT ORGANIZATION PROGRAM
ganization that held the QIO contract(s), its business in the state(s) in which
it held the QIO contract(s), and related topics. Other questions were related
to staffing; staff qualifications, skills, and capabilities; the composition and
experience of the leadership team; the organization of work; ratings on
selected aspects of the working relationship between the QIO in the state,
district, or territory and CMS (both the Central and the Regional Offices);
the ratings of other services (e.g., the Standard Data Processing System and
QIONet) provided by or for CMS to the QIO community; and perspectives
on the "best" QIOs (overall and by task area).
With a few exceptions for text-based answers, the questions offered
fixed-choice response categories, and an answer was required before the
next question was presented on the screen. Respondents were asked to se-
lect a substantive answer whenever possible; a "don't know/prefer not to
answer" response was offered as a backup.
Approach to Data Collection
The committee e-mailed a memorandum to the CEOs of all organiza-
tions holding QIO contracts for the 7th SOW. In that memorandum the
committee requested the cooperation of the CEOs in overseeing completion
of both portions of the online data collection instrument; provided instruc-
tions and links to the online questionnaires; and offered to provide assis-
tance, if needed, by e-mail or telephone. A 4-week response period was
offered for each part.
Completion Rates and Missing Data
All 41 organizations holding one or more QIO contracts for the 7th
SOW completed Part 1 within the response period, representing a 100 per-
cent response rate. Missing data rates were very low; only 2 of 41 organiza-
tions consistently selected the "don't know/prefer not to answer" response
category, and those responses were chiefly for questions in the section on
financial data.
For Part 2, 52 of 53 QIOs provided data, representing a response rate
of slightly greater than 98 percent. Again, missing data rates were very low;
only 2 of the 52 respondents selected the "don't know/prefer not to an-
swer" category, and this was generally done for questions asking about
current or planned lines of non-QIO business and some of the rating items.
QUANTITATIVE ANALYSES OF QIO PERFORMANCE
The committee conducted analyses of the QIOs' performance, based on
the scores that CMS constructs to assess contract performance, to deter-
OCR for page 151
STUDY APPROACH 151
mine why some states and provider settings show greater improvement on
given measures. The committee attempted to correlate the QIO scores with
variables such as performance on other quality improvement subtasks,
spending per beneficiary, contract round, QIO region, and provider satis-
faction. The analyses and findings are discussed in Chapter 10.
VISITS TO QIOS
The budget limited this study, in that it did not allow the committee or
IOM staff travel to all the QIOs. To gain a firsthand understanding of a
QIO and to reach as many different QIOs as possible with a minimum of
travel, committee members and staff visited either their local QIO or one
that was in an area to which travel for other purposes was planned. In total,
16 committee members and staff visited 10 different QIOs in person and an
11th one by telephone. A semistructured interview protocol was used dur-
ing most of these visits. The questions included in the protocol concerned
the strengths, weaknesses, opportunities, and threats to the QIO perceived
by the QIO respondents. Note that although the visiting committee mem-
bers and IOM staff used similar protocols, not all QIOs were asked the
exact same questions, and often, topics arose as a result of the discussions.
The text attempts to reflect the number of QIOs that mentioned specific
concerns, but a less than 100 percent response rate does not indicate dis-
agreement among those QIOs.
In addition, the QIOs generally presented key staff, such as the presi-
dent, medical director, director of quality improvement, and chief financial
officer, who discussed programs as well as responded to additional ques-
tions that were not part of the protocol. The site visits lasted from a couple
of hours to all day or more. The visiting committee members and staff
submitted written notes to the IOM after each visit, but the notes for each
site visit varied substantially, depending on multiple factors, including who
the visitor happened to be. In addition to the written reports, the committee
discussed the highlights of most visits at committee meetings. IOM staff
reviewed both the notes and the presentations to identify themes, issues of
concern to the QIOs, and useful ideas for further research.
TELEPHONE INTERVIEWS
To gain more insight into the views of the executive directors of the
QIOs on their programs, Cheryl Ulmer, a health services research consult-
ant working with the committee, interviewed CEOs from 20 of the 53 QIOs
by telephone during March 2005.
OCR for page 152
152 MEDICARE'S QUALITY IMPROVEMENT ORGANIZATION PROGRAM
Selection of Participating QIOs
The committee randomly selected 20 QIOs after stratification by re-
gion and the sizes of their beneficiary populations in the following manner:
(1) the committee divided the states into the four CMS geographic regions;
(2) the ultimate number of states selected from each region was propor-
tional to the number in each region (ultimately, one region had four states
in the sample, two regions had five states, and one region had six states);
and (3) after each state was labeled large or small on the basis of whether
the beneficiary population size was above or below the median size of state
beneficiary populations, the committee randomly selected two or three states
from the subgroups of states with large and small beneficiary populations
in each region. When a region had four or six states in the sample, equal
numbers of states with large and small beneficiary populations were se-
lected. When a region had five states, three states were selected from the
states with large or small beneficiary populations, depending on which sub-
group consisted of more states.
The goal was to ensure that states in each region had a chance of being
selected that was proportional to the number of states in that region. Within
each region, states with large or small size beneficiary populations would
have a chance of being selected proportionally to the number of states with
large and small populations. The committee e-mailed letters to each se-
lected CEO in February 2005. The committee first requested a telephone
interview and then outlined the general topics for discussion. These topics
were primarily related to knowledge transfer. All interviews occurred be-
tween March 7 and 16, 2005.
Nature of Interviews and Questions
The committee developed an interview script with open-ended ques-
tions and then, while attending the AHQA national meeting in February
2005, conducted interviews with four CEOs not previously selected to pre-
test the interview protocol. Upon review of the interview notes and a dis-
cussion with the consultant, the committee developed a wide-ranging
interview protocol that went beyond knowledge transfer and covered topics
such as QIOSCs; the selection of identified participants; diffusion of knowl-
edge about quality improvement activities beyond the identified partici-
pants; the pros and cons of targeting the poorest-performing providers;
evaluation of the QIOs; the importance of beneficiary education, case re-
view, and payment error functions; the impact of competition for QIO con-
tracts; the timeliness and the quality of data; and barriers to knowledge
transfer. During the interview, the CEOs raised additional topics: electronic
OCR for page 153
STUDY APPROACH 153
health records, Regional Health Information Organizations, and pay for
performance.
The interview protocol design primarily included qualitative questions,
with priorities set among the questions, in recognition of the proposed
45-minute limit of the interview with each QIO. The detailed questions
were not provided to the CEOs in advance. The primary questions were
open-ended; some questions contained formal probes, but even these
were open-ended and did not require a certain set of closed-ended re-
sponses. In the preinterview correspondence and on the telephone, the com-
mittee and the consultant assured the CEOs that their responses would be
held in confidence, would not be taped, and that handwritten or typed
notes would not go to the committee or public access file. In her analysis,
the consultant deidentified information attributable to specific QIOs.
The CEOs were very responsive to the consultant. Although they were
advocates for their programs and quality improvement in general, they were
reflective; were willing to be adaptable to changes in the quality improve-
ment process in the future; and had particularly strong opinions on their
relationships with CMS, on which types of providers moved the field for-
ward, on the need for further incentives to prod providers to change, and
on the need for more timely and comprehensive data.
The consultant conducted each interview in one continuous telephone
session, with all but two of the interviews lasting 90 minutes rather than the
scheduled 45 minutes. She used the interview protocol as a guide and took
thorough handwritten notes that were typewritten after the telephone inter-
view. Because the interviews were not taped, the notes were not verbatim;
but they still captured the content and the context of the directors' com-
ments in detail. The consultant did not filter the responses during the note-
taking process.
Analysis of Interview Content
The committee staff and the consultant discussed the results of the first
interviews to see if there were other areas that needed to be probed. For
example, in the first interviews, no one mentioned collaborative activities,
but upon questioning, every QIO questioned about this in later interviews
indicated their use. Similarly, although there had been no specific questions
on pay for performance or electronic health records, when these topics came
up repeatedly, the consultant started asking the CEOs for their views.
After the completion of all 20 interviews, the consultant analyzed the
contents of her notes to identify the themes that emerged in response to
each topic. She then examined each theme to determine how many QIOs
addressed it to identify a denominator for calculation of the percentage of
QIOs that supported a particular idea or that mentioned an issue. Deliber-
OCR for page 154
154 MEDICARE'S QUALITY IMPROVEMENT ORGANIZATION PROGRAM
ate quantitative analysis of the responses helps confirm the general sense
that one gathers during the interviews but also helps refute strongly held
opinions of a few CEOs that might not reflect those of the whole respon-
dent group. During the interviews, however, some CEOs offered un-
prompted comments. Therefore, a less than 100 percent response rate does
not necessarily indicate disagreement among the interviewees. (In this re-
port the committee has indicated, where appropriate, when all CEOs were
asked the same question.) The consultant illustrated themes by the use of
quotes from the CEOs. If there was a minority view or an issue that might
be of potential value for further research, even if it was voiced by only one
QIO, it was mentioned in the analysis for consideration in case the issue
came up from other data sources.
THREE-DAY BRIEFING BY CMS
The study's Project Officer at CMS, Joyce Kelly, arranged an intensive
series of presentations by CMS staff, both those in the Central Office and
those in the various QIO Regional Offices, that covered a wide range of
elements of the QIO program. She worked with IOM staff in advance to
develop the agenda, soliciting questions that the committee wanted ad-
dressed and proposing topics that CMS thought would be useful to the
committee. The IOM requested discussions of issues such as Central Office
oversight and direction, the roles of the Clinical Data Abstraction Centers,
the role of QIOs in public reporting, the volume of complaints, and benefi-
ciary satisfaction. The briefing provided an excellent overview and intro-
duction to the program for the whole IOM project team. Although most of
the time was consumed by the presentations, IOM staff had the opportu-
nity to ask each speaker questions. All the materials and the PowerPoint
presentations from this briefing are available on the World Wide Web
(CMS, 2004).
ACCESS TO QIONET
CMS uses an internal website to facilitate confidential program com-
munications with the QIOs and the reporting of certain types of data. This
site includes the Dashboard section, which presents state-specific data, up-
dated quarterly, on many of the measures used to evaluate the performance
of each QIO under its contract with CMS. CMS arranged for the IOM to
have access to the website during the course of the study, with the under-
standing that the data would not be used with state-specific identifiers.
Many of the data used in this report to assess the impacts of the QIOs on
particular quality measures came from the Dashboard section of QIONet
(see Table A.3 in Appendix A for a detailed list of the measures.)
OCR for page 155
STUDY APPROACH 155
FACE-TO-FACE INTERVIEWS
The committee conducted two sets of formal face-to-face interviews for
the study. In September 2004, during CMS's annual conference for QIOs,
the committee interviewed staff and CEOs from five randomly selected or-
ganizations (which held a total of seven QIOSC contracts); one interview
served as a pretest of the interview protocol. All organizations selected
agreed to participate. The interviewers used the same script for each of the
four remaining interviews, although the topic and assignment of each
QIOSC were different. Questions focused on topics such as the purpose of
the QIOSC, its main activities, why it was selected for a QIOSC contract,
the QIOSC's key partners, how the QIOSC interacts with the QIOs and
CMS, and how the QIOSC is evaluated by CMS. Detailed written notes
were taken during each interview. The information gleaned from the inter-
views guided further research on QIOSC issues.
The committee conducted the second set of interviews on February 25,
2005, at the AHQA Annual Technical Conference in San Francisco, Cali-
fornia. The committee selected the CEOs of the QIOs of four states from
the list of those that were not randomly chosen for the telephone interviews
or that were not visited by representatives of the project. The CEOs of all
four QIOs agreed to participate in a 45-minute interview, which primarily
concerned various mechanisms for knowledge transfer. The interviews
served as a pretest of the script to be used for the telephone inter-
views. Committee staff took detailed written notes during the interviews.
FOCUS GROUP
The committee selected the CEOs of the QIOs of 11 states from those
not already chosen for the telephone interviews or visited by representatives
of the study. All 11 CEOs agreed to participate, although one CEO recom-
mended that the medical director be interviewed as a substitute. The par-
ticipants represented at least two states from each Regional Office involved
in the QIO program, including both predominantly rural and predominantly
urban states. Also, the committee selected states to include both those with
small numbers and those with large numbers of beneficiaries and providers.
The QIO subcommittee chair, a professional researcher experienced with
focus groups, moderated the 90-minute discussion while committee staff
took notes. The moderator used a guide that included general, open-ended
questions about the direction of the program in the upcoming SOW, chal-
lenges concerning information technologyrelated tasks, criteria important
for assessment of the value of individual QIOs, and how QIO functions
might change when all beneficiaries had electronic health records.
OCR for page 156
156 MEDICARE'S QUALITY IMPROVEMENT ORGANIZATION PROGRAM
SPECIFIC DATA REQUESTS TO CMS STAFF
It would have been extremely difficult to conduct this study without the
excellent cooperation of individuals responsible for the QIO program's
management. Throughout the course of the contract they have willingly
responded to numerous requests from IOM, spent hours of their own time
and their staff's time explaining the nuances and complexities of the pro-
gram, collected data to fulfill requests, and shared their knowledge of CMS
and the QIO program. For example, the committee requested a major
amount of data on the budget for the 7th SOW, both by each state or QIO
and as aggregated data at the national level. The budgets were broken down
by core tasks and included costs for special studies and support contracts.
The committee also requested information about the content of major sup-
port and special study contracts. Other requests included requests for infor-
mation about organizational structures, the results of QIO evaluations,
CMS staffing needs, and the timing for the transition to the 8th SOW.
FORMAL AND INFORMAL DISCUSSIONS WITH AHQA
From the beginning of the study, AHQA's staff and QIO members pro-
vided background information, technical advice, documents, and data as
well as access to their meetings. AHQA's active support of the IOM project
enhanced the receptiveness of its members' participation in the IOM study.
Formal presentations and written documents are cited by author name, but
information provided informally or confidentially is not.
INFORMAL DISCUSSIONS WITH
CONSUMER ORGANIZATIONS AND PROVIDERS
The committee conducted informal discussions with people from con-
sumer and beneficiary organizations as well as with practitioners and indi-
viduals from provider organizations at meetings and by telephone. Time,
budget constraints, and federal rules prevented the committee from per-
forming more formal interview surveys with these stakeholders.
However, the committee did recognize that CMS contracted with
Westat to conduct surveys of providers' satisfaction with QIOs. These
surveys, further discussed in Chapter 10, were conducted among nurs-
ing homes, home health agencies, physicians offices, hospitals, and
Medicare+Choice plans for all 53 QIOs in the 7th SOW. (For nursing
homes and home health agencies, survey respondents included both identi-
fied participants and nonidentified participants.) Over 21,000 providers
responded to questions about their level of satisfaction, including their
impressions of the effectiveness of their respective QIOs.
OCR for page 157
STUDY APPROACH 157
The federal Paperwork Reduction Act of 1995 (P.L. 104-13) prohibits
the use of a survey, interview guide or systematic data collection from 10 or
more persons by a federal contractor without clearance from the Office of
Management and Budget. That process can sometimes take many months.
Despite the committee's desire to survey providers and consumers, it deter-
mined that would not be feasible within the time limits of the study's con-
tract with CMS.
NATIONAL CONFERENCES AND MEETINGS
The committee gathered much information and understanding about
the QIO program by attending national technical conferences held by CMS
and AHQA, as well as smaller meetings sponsored by various organizations
that focused more narrowly on particular aspects of the QIO program and
related issues, such as public reporting of quality measures, information
technology use in health care, and implementation of the Medicare Pre-
scription Drug, Improvement, and Modernization Act of 2003 Part D
prescription drug benefit. A side benefit of the meetings was the opportu-
nity for informal conversations with the employees of QIOs from across the
country. These discussions had some influence on the directions of the re-
search and the richness of the descriptions of the QIOs, but they are not
cited specifically. In addition, CMS allowed project staff to attend a strate-
gic planning meeting and a confidential evaluation panel meeting.
SUGGESTIONS OF "OTHER ENTITIES"
Suggestions were solicited from the committee and subcommittee mem-
bers and additional names were offered by the report reviewers for private
businesses performing functions similar to those of QIOs. All relevant sug-
gestions were included, even if the company currently offers only one of the
related functions. Data on each company were pulled from its website for
the table in Appendix B. Each business was offered the opportunity to up-
date and verify its data. The committee tabulated the data but was unable
to assess the specific capabilities or potential strengths and weaknesses of
each business.
RESEARCH CHALLENGES
Given the timing of the congressional mandate, this study could not be
built into the beginning of the 7th or the 8th SOW and produce the report
on schedule. The 7th SOW was already well under way at the time that the
legislation mandating this study was signed into law. Although the 8th SOW
began before the end of the IOM's contract with CMS, it was too late to be
OCR for page 158
158 MEDICARE'S QUALITY IMPROVEMENT ORGANIZATION PROGRAM
able to collect any new data reflecting the activities of the 8th SOW and
meet the study's deadline. Thus, a study design and data collection method-
ology for the project could not be included directly in the core QIO con-
tracts. Although CMS instructed the QIOs to consider data requests from
IOM as a separate part of their activities for the 7th SOW, it did not give
them additional funds to cover their time or effort spent responding.
Given those constraints, the committee could not use an experimental
design. The QIO program operates in all states, participation by providers
is voluntary, and all the QIOs have essentially the same basic tasks, so
randomization and the use of control groups would not have been feasible
without a specially designed experiment. The program was already fully
operational and would continue to function and evolve during the course of
the study. A Hawthorne effect might have contributed to confounding, but
it was not considered a limitation of the study. For example, the committee's
assessment of CMS's oversight responsibilities was based on both retro-
spective and concurrent information. The fact that the U.S. Congress and
the committee asked particular questions about the program could have
influenced CMS management to change course or expand or reduce certain
activities during the course of the study.
Another constraint that affected much of the data collection effort was
the need to maintain confidentiality. To gain access to many of the data
from CMS and individual QIOs, the committee agreed to report only aggre-
gated or deidentified data so that individual QIOs could not be identified.
Because state-identified data could jeopardize the competitive business situ-
ation of individual QIOs and limit their willingness to communicate on
sensitive issues, the IOM agreed to maintain confidentiality at the minimum
level necessary.
STRENGTH OF RESEARCH CONCLUSIONS
Although limitations to each of the various research methods used in
this study existed, altogether they provided a substantial amount of data
and information. The committee triangulated methods by checking the con-
sistency of the findings obtained by the different data collection methods,
such as the data collected through the web-based data collection tool, the
focus group, the telephone interviews, and the site visits. Triangulation
among data sources was also possible. For example, the committee com-
pared the information about the program provided by CMS with the infor-
mation provided by the QIOs.
Because more than one committee member or IOM staff member often
made site visits and more than one staff person generally attended most
meetings, the committee triangulated among multiple analysts. The com-
OCR for page 159
STUDY APPROACH 159
mittee members routinely compared notes and impressions after a visit,
interview, or meeting and discussed their perceptions and findings.
The study puts more weight on information gathered through the
SurveyMonkey web-based data collection tool, which covered all QIOs,
and on data collected from the telephone interviews of a random sample of
QIO CEOs than on individual interviews and informal comments, which
served to strengthen consistent findings. An opinion voiced by one QIO
staff member might include an important idea, but it carried more weight
with the committee if multiple sources expressed similar ideas or if several
different research approaches led to the same conclusions.
Each research method had limitations, and it was necessary to accept
CMS program data as they were given, without auditing of the data. None-
theless, by using so many different methods to gather data, the committee
strengthened its ability to paint a more complete and accurate picture of the
QIO program.
REFERENCES
CMS (Centers for Medicare and Medicaid Services). 2004. The Quality Improvement Organi-
zation Program: CMS Briefing for IOM Staff. [Online]. Available: http://www.
medqic.org/dcs/ContentServer?cid=1105558772835&pagename=Medqic%2FMQ
GeneralPage%2FGeneralPageTemplate&c=MQGeneralPage [accessed December 26,
2005].
IOM (Institute of Medicine). 2006. Performance Measurement: Accelerating Improvement.
Washington, DC: The National Academies Press.
Representative terms from entire chapter:
core contract
