Click for next page ( 144


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 143
Part II INTRODUCTION In Part II of this report, the Committee on Redesigning Health Insur- ance Performance Measures, Payment, and Performance Improvement Pro- grams presents a description of the Quality Improvement Organization (QIO) program in the 7th and 8th scopes of work (SOWs). Part I stressed the importance of looking to what needs to be done in the future to improve the quality of health care. Part II presents the basis for those recommenda- tions and documents the current status of the program. It will serve as a resource for future examinations of the QIO program. Part II includes eight chapters that provide the information outlined below. Chapter 6 describes the study approach. In particular, it explains the various methodologies used and the limitations encountered in accumulat- ing information about the QIO program from stakeholders. Chapter 7 examines the overall structure and financing of the QIO pro- gram. It evaluates the structural requirements, governance, and staffing of the individual QIOs, organizations holding QIO contracts, and the QIO Support Centers (QIOSCs). The funding for the entire program is also pre- sented, including funding for the core contracts, special studies, and sup- port contracts. The external activities and funding of the QIOs are also described. Chapter 8 provides a detailed description of the technical assistance activities required of the QIOs in the 7th and 8th SOWs. First, the methods used to provide technical assistance are discussed in general, as well as how 143

OCR for page 143
144 MEDICARE'S QUALITY IMPROVEMENT ORGANIZATION PROGRAM QIOs recruit providers for these activities. Next, the chapter presents a detailed description of those activities in each setting. Chapter 9 describes the impact of technical assistance on quality im- provement and knowledge transfer. The chapter includes a literature review and considers quality improvement interventions in general and those of QIOs specifically, discusses different approaches to quality improvement, and provides an overview of knowledge transfer and its impact in the QIO program. Chapter 10 evaluates the impact of quality improvement activities in the QIO program specifically. The detailed evaluation formulas as well as the means used to identify high performing QIOs are discussed. Chapter 11 describes the communications and beneficiary education activities of the QIOs in the 7th and 8th SOWs. This includes the promo- tion of public information on websites, assistance to providers for the col- lection and reporting of data for public reporting, and the maintenance of toll-free help lines. Chapter 12 describes the protective activities of the QIOs. All activities related to the review of quality concerns (including the new mediation pro- cess) as well as reviews designed to protect the integrity of the Medicare Trust Funds, such as the Hospital Payment Monitoring Program, are exam- ined. The different categories and types of reviews and the review process are described in detail, and data that can be used to help assess their impact are presented. Chapter 13 describes the Centers for Medicare and Medicaid Services' (CMS's) oversight of the QIO program. The structure of the QIO program as it fits into CMS's framework is first described, including the offices and the personnel involved in the management of this complex program. Next, the committee describes the various communications methods that these personnel used, including their written, oral, and electronic communica- tions systems. The data-related issues evaluated include the various tools that CMS uses to report data and the flow of data in each of the provider settings. The manner in which the QIO and QIOSC contracts themselves are structured and awarded is then examined. Finally, the chapter discusses the overall guidance of the QIO program, including strategic planning, policy direction, program coordination, and overall program evaluation. Part II provides a comprehensive, retrospective view of the QIO pro- gram in the 7th and 8th SOWs. The assessment presented here serves as a primary source of information about this complex program, with a descrip- tion of its recent past and current status, as well as a base on which recom- mendations for the role of QIOs in the future of quality health care have been made.

OCR for page 143
6 Study Approach CHAPTER SUMMARY This chapter presents the committee's methodological approach for this study. To address the items in the legislative mandate, it was necessary to design a multifaceted study that used a mix of data collection and analytic methods, including both qualitative and quantitative analyses, rather than an experimental design. The chapter discusses each of the main data sources, including: a fo- cused literature review, an internet-based data collection, site visits, interviews, and quantitative data requests. Finally, the research strengths and challenges are examined. In response to the legislative mandate in Section 109 of the Medicare Prescription Drug, Improvement, and Modernization Act of 2003 (P.L. 108- 173), the Institute of Medicine (IOM) formed an authoring committee, the Committee on Redesigning Health Insurance Performance Measures, Pay- ment, and Performance Improvement Programs, to prepare three reports in a single project and a separate subcommittee to focus on the Quality Im- provement Organization (QIO) study and guide the data collection and analysis. These committees included members with the necessary expertise and experience in such areas as clinical quality improvement; health eco- nomics; Centers for Medicare and Medicaid Services (CMS) management; research on organizational change, consumer behavior, and program evalu- ation; and the delivery of health care in physicians' offices, long-term care facilities, hospitals, academic medical centers, and integrated health care delivery systems. During the course of the study contract (April 2004 to June 2005), the members of the QIO subcommittee met three times in per- son and once by conference call to guide the research and data collection efforts. The main committee met seven times and gave policy direction to the research, discussed drafts, and reached consensus on the final wording of the report and its recommendations. 145

OCR for page 143
146 MEDICARE'S QUALITY IMPROVEMENT ORGANIZATION PROGRAM To discuss the items mandated by the legislation, such as an overview of the QIO program, the duties of the contracting organizations in the pro- gram, the sources and amounts of their funding, and a review of the over- sight of the program, the committee needed to present descriptive data. To address the other items in the mandate, such as the extent to which QIOs improve the quality of care, the extent to which other organizations could do their duties, and the effectiveness of case review and other QIO activi- ties, the committee needed evaluative analyses and expert judgment. This study uses a retrospective examination of the data to present much of the descriptive information as well as to evaluate the impact of the program's efforts. Although the chapters with a retrospective view examine the data from the beginning of the Medicare quality assurance program to provide an overview, the main focus is on more recent program activities; and most data and analyses relate to the time period of the contract for the 7th scope of work (SOW), which was from August 2002 to August 2005. This study also looks at the program concurrently, because the period of the IOM study contract overlapped with the end of the 7th SOW and the beginning of the 8th SOW. The information presented in this report reflects the data collected through the midsummer of 2005 and is descriptive of the 8th SOW but does not include data reflecting the CMS and QIO work produced during the 8th SOW, which had barely begun when the report was drafted. The unit of analysis varied, depending on the questions addressed. For example, the committee wanted to know what, if any, impact the whole program has had on Medicare beneficiaries' care and considered informa- tion reflecting the activities of all QIOs, other contractors, and CMS. Thus, the committee used aggregated data from all QIOs to examine possible national effects. When the committee examined the effects of specific qual- ity improvement interventions on the provision of care and care outcomes, a more common focus of the published literature, the unit of analysis was often an individual QIO or a particular intervention project of one or more QIOs. The committee tried to identify the organizational and structural characteristics of the QIOs that might contribute to greater quality improve- ment. That is, were there characteristics of high-performing QIOs or as- pects of how CMS managed them that could be identified and spread to other QIOs to help them improve the national program's overall perfor- mance? Comparisons of QIOs' structural characteristics and performance measures were necessary to attempt to answer those questions. DATA SOURCES The committee chose to gather data from a wide variety of sources and to compare and integrate the findings, as no one source fully covered the

OCR for page 143
STUDY APPROACH 147 many facets of the QIO program and the questions being asked in the study. Fortunately, the committee found both the QIO community and CMS wel- coming, willing to provide reasonable requests for data, and open and eager to talk with the committee about the program. By the end of the study, the IOM project had both personal and substantive e-mail contacts with the chief executive officer (CEO) from each organization holding a QIO con- tract. The main data collection methods and sources for this study included the following: a focused review of the literature on the impact of quality improve- ment and the impact of QIO efforts at quality improvement; data collected in 2005 from all 53 QIOs through the SurveyMonkey internet-based data collection tool; quantitative analyses of QIOs' relative performance on various subtasks of the 7th SOW; site visits by 16 committee members and IOM staff to 11 QIOs and one CMS Regional Office; telephone interviews with 20 CEOs from randomly selected QIOs; a 3-day briefing by CMS staff; a half-day workshop for the committee and subcommittee, including formal presentations from academic researchers as well as experts working in quality improvement in the field and comments from various stake- holders; access to QIONet, a CMS internal website for the QIO program that includes performance data by state on the Dashboard section; face-to-face interviews with CEOs and staff from certain QIOs and randomly selected QIO Support Centers (QIOSCs); a focus group discussion with 11 QIO CEOs; specific requests for data from CMS staff; many formal and informal discussions with the staff of the American Health Quality Association (AHQA; the national organization representing all QIOs) and the attendees at their functions (mainly QIO staff); informal discussions with representatives of consumer and benefi- ciary organizations and various providers; data collected from presentations and question and discussion ses- sions; the observation of interactions among QIO and CMS staff; and in- formation from the informal questioning of attendees at AHQA's annual technical conference and other national meetings, CMS's annual QualityNet Conference, and miscellaneous smaller meetings; and suggestions from committee and subcommittee members and review- ers of businesses and other organizations performing QIO-like functions ("other entities").

OCR for page 143
148 MEDICARE'S QUALITY IMPROVEMENT ORGANIZATION PROGRAM It is noteworthy that the committee gained knowledge of and insight into the QIO program through multiple sources, all of which could not be fully referenced in the text. The committee attempted to indicate the main source of information throughout the report, as appropriate. Each of the formal research approaches is briefly discussed below. FOCUSED LITERATURE REVIEW The committee searched for an understanding of how quality improve- ment interventions with health care providers influence the quality of health care, and it examined literature chosen from two sources. (The committee's previous report, Performance Measurement: Accelerating Improvement, addressed quality improvements resulting from the availability of health care quality data for providers and the public reporting of such data [IOM, 2006].) First, the committee extracted the majority of studies from a bibli- ography compiled by AHQA in August 2004. This bibliography lists more than 600 studies that either were conducted in partnership with a QIO, used QIO data, or were written by an author affiliated with a QIO. Poten- tial biases may have existed in the selection of the studies for the bibliogra- phy and the methods by which they were conducted. Although most articles in AHQA's bibliography refer to clinical treatment options for Medicare patients and descriptions of the QIO program, the committee focused on a smaller subset of studies that examined QIO approaches to improving the quality of health care. Articles were selected from AHQA's bibliography on the basis of their titles by use of an internal reliability check by two mem- bers of the IOM staff. The committee found most of the remaining studies by the use of Medline searches, performed in November 2004 for articles published in the previous 5 years, using the following medical search headings: "quality improvement," "effectiveness," and "intervention." In addition, the com- mittee evaluated a selection of studies to supplement the evidence base for outcome measures, collaborative studies, audits and feedback, and evalua- tions of the QIO program published from November 2004 through July 2005. The committee chose studies from references in previously selected studies and by searching the Cochrane Database of Systematic Reviews and the Agency for Healthcare Research and Quality systematic reviews. These studies did not focus specifically on QIO activities and are part of a broader literature on quality intervention. It would be impossible to know whether the health care providers in those studies had prior knowledge of or had participated in various QIO efforts. The literature review is discussed in more detail in Chapter 9, and a summary of selected studies on quality improvement is provided in Table A.1 in Appendix A.

OCR for page 143
STUDY APPROACH 149 The committee categorized the studies obtained from the literature re- view on the basis of the strength of the research methodology (see Table A.2). In general, the results and the lessons learned from randomized controlled trials outweighed those from studies with nonexperimental de- signs. However, the committee recognized the value of observational and qualitative studies for identifying major themes from past interventions and considered them useful starting points for understanding the impacts of the previous interventions. WEB-BASED DATA COLLECTION In 2005, the committee, along with a consultant, Allyson Ross Davies, used software on the SurveyMonkey.com platform to design a two-part data collection instrument, collect responses from all organizations holding one or more QIO contracts for the 7th SOW, and export raw data for analysis using SAS software. The main committee and the QIO subcommit- tee developed draft content. Key staff at CMS and AHQA and a CEO of an organization holding multiple QIO contracts for the 7th SOW reviewed a draft of the data collection instrument for clarity, terminology, user friend- liness, and duplication of ongoing data collection efforts. The committee selected the SurveyMonkey platform because of both the ease of online questionnaire design and the familiarity of the QIO community with this survey platform. Both CMS and AHQA actively encouraged the QIOs to fully participate in this data collection effort. The committee assured the respondents that they would remain anony- mous, and SurveyMonkey guaranteed the secure transmission of responses through the use of Secure Sockets Layer encryption. The consultant ex- ported raw data from the SurveyMonkey website for analysis, at which point she closed the data collection instruments and deleted them from the site. She did not share raw data with IOM staff, the committee, or CMS. The consultant presented only deidentified, aggregated information about the organizations to the project's main committee and the QIO subcommit- tee. This report includes only deidentified and aggregated data. Content of Data Collection Instrument Part 1 of the data collection instrument obtained information from each of the 41 organizations holding one or more QIO contracts for the 7th SOW. The questions in Part 1 focused on organizational structure; boards of directors or trustees; and financial information, which was related to both the QIO contract(s) and other business lines, when relevant. Part 2 requested separate data on the 7th SOW from each of the 53 QIOs. Background questions focused on factual information about the or-

OCR for page 143
150 MEDICARE'S QUALITY IMPROVEMENT ORGANIZATION PROGRAM ganization that held the QIO contract(s), its business in the state(s) in which it held the QIO contract(s), and related topics. Other questions were related to staffing; staff qualifications, skills, and capabilities; the composition and experience of the leadership team; the organization of work; ratings on selected aspects of the working relationship between the QIO in the state, district, or territory and CMS (both the Central and the Regional Offices); the ratings of other services (e.g., the Standard Data Processing System and QIONet) provided by or for CMS to the QIO community; and perspectives on the "best" QIOs (overall and by task area). With a few exceptions for text-based answers, the questions offered fixed-choice response categories, and an answer was required before the next question was presented on the screen. Respondents were asked to se- lect a substantive answer whenever possible; a "don't know/prefer not to answer" response was offered as a backup. Approach to Data Collection The committee e-mailed a memorandum to the CEOs of all organiza- tions holding QIO contracts for the 7th SOW. In that memorandum the committee requested the cooperation of the CEOs in overseeing completion of both portions of the online data collection instrument; provided instruc- tions and links to the online questionnaires; and offered to provide assis- tance, if needed, by e-mail or telephone. A 4-week response period was offered for each part. Completion Rates and Missing Data All 41 organizations holding one or more QIO contracts for the 7th SOW completed Part 1 within the response period, representing a 100 per- cent response rate. Missing data rates were very low; only 2 of 41 organiza- tions consistently selected the "don't know/prefer not to answer" response category, and those responses were chiefly for questions in the section on financial data. For Part 2, 52 of 53 QIOs provided data, representing a response rate of slightly greater than 98 percent. Again, missing data rates were very low; only 2 of the 52 respondents selected the "don't know/prefer not to an- swer" category, and this was generally done for questions asking about current or planned lines of non-QIO business and some of the rating items. QUANTITATIVE ANALYSES OF QIO PERFORMANCE The committee conducted analyses of the QIOs' performance, based on the scores that CMS constructs to assess contract performance, to deter-

OCR for page 143
STUDY APPROACH 151 mine why some states and provider settings show greater improvement on given measures. The committee attempted to correlate the QIO scores with variables such as performance on other quality improvement subtasks, spending per beneficiary, contract round, QIO region, and provider satis- faction. The analyses and findings are discussed in Chapter 10. VISITS TO QIOS The budget limited this study, in that it did not allow the committee or IOM staff travel to all the QIOs. To gain a firsthand understanding of a QIO and to reach as many different QIOs as possible with a minimum of travel, committee members and staff visited either their local QIO or one that was in an area to which travel for other purposes was planned. In total, 16 committee members and staff visited 10 different QIOs in person and an 11th one by telephone. A semistructured interview protocol was used dur- ing most of these visits. The questions included in the protocol concerned the strengths, weaknesses, opportunities, and threats to the QIO perceived by the QIO respondents. Note that although the visiting committee mem- bers and IOM staff used similar protocols, not all QIOs were asked the exact same questions, and often, topics arose as a result of the discussions. The text attempts to reflect the number of QIOs that mentioned specific concerns, but a less than 100 percent response rate does not indicate dis- agreement among those QIOs. In addition, the QIOs generally presented key staff, such as the presi- dent, medical director, director of quality improvement, and chief financial officer, who discussed programs as well as responded to additional ques- tions that were not part of the protocol. The site visits lasted from a couple of hours to all day or more. The visiting committee members and staff submitted written notes to the IOM after each visit, but the notes for each site visit varied substantially, depending on multiple factors, including who the visitor happened to be. In addition to the written reports, the committee discussed the highlights of most visits at committee meetings. IOM staff reviewed both the notes and the presentations to identify themes, issues of concern to the QIOs, and useful ideas for further research. TELEPHONE INTERVIEWS To gain more insight into the views of the executive directors of the QIOs on their programs, Cheryl Ulmer, a health services research consult- ant working with the committee, interviewed CEOs from 20 of the 53 QIOs by telephone during March 2005.

OCR for page 143
152 MEDICARE'S QUALITY IMPROVEMENT ORGANIZATION PROGRAM Selection of Participating QIOs The committee randomly selected 20 QIOs after stratification by re- gion and the sizes of their beneficiary populations in the following manner: (1) the committee divided the states into the four CMS geographic regions; (2) the ultimate number of states selected from each region was propor- tional to the number in each region (ultimately, one region had four states in the sample, two regions had five states, and one region had six states); and (3) after each state was labeled large or small on the basis of whether the beneficiary population size was above or below the median size of state beneficiary populations, the committee randomly selected two or three states from the subgroups of states with large and small beneficiary populations in each region. When a region had four or six states in the sample, equal numbers of states with large and small beneficiary populations were se- lected. When a region had five states, three states were selected from the states with large or small beneficiary populations, depending on which sub- group consisted of more states. The goal was to ensure that states in each region had a chance of being selected that was proportional to the number of states in that region. Within each region, states with large or small size beneficiary populations would have a chance of being selected proportionally to the number of states with large and small populations. The committee e-mailed letters to each se- lected CEO in February 2005. The committee first requested a telephone interview and then outlined the general topics for discussion. These topics were primarily related to knowledge transfer. All interviews occurred be- tween March 7 and 16, 2005. Nature of Interviews and Questions The committee developed an interview script with open-ended ques- tions and then, while attending the AHQA national meeting in February 2005, conducted interviews with four CEOs not previously selected to pre- test the interview protocol. Upon review of the interview notes and a dis- cussion with the consultant, the committee developed a wide-ranging interview protocol that went beyond knowledge transfer and covered topics such as QIOSCs; the selection of identified participants; diffusion of knowl- edge about quality improvement activities beyond the identified partici- pants; the pros and cons of targeting the poorest-performing providers; evaluation of the QIOs; the importance of beneficiary education, case re- view, and payment error functions; the impact of competition for QIO con- tracts; the timeliness and the quality of data; and barriers to knowledge transfer. During the interview, the CEOs raised additional topics: electronic

OCR for page 143
STUDY APPROACH 153 health records, Regional Health Information Organizations, and pay for performance. The interview protocol design primarily included qualitative questions, with priorities set among the questions, in recognition of the proposed 45-minute limit of the interview with each QIO. The detailed questions were not provided to the CEOs in advance. The primary questions were open-ended; some questions contained formal probes, but even these were open-ended and did not require a certain set of closed-ended re- sponses. In the preinterview correspondence and on the telephone, the com- mittee and the consultant assured the CEOs that their responses would be held in confidence, would not be taped, and that handwritten or typed notes would not go to the committee or public access file. In her analysis, the consultant deidentified information attributable to specific QIOs. The CEOs were very responsive to the consultant. Although they were advocates for their programs and quality improvement in general, they were reflective; were willing to be adaptable to changes in the quality improve- ment process in the future; and had particularly strong opinions on their relationships with CMS, on which types of providers moved the field for- ward, on the need for further incentives to prod providers to change, and on the need for more timely and comprehensive data. The consultant conducted each interview in one continuous telephone session, with all but two of the interviews lasting 90 minutes rather than the scheduled 45 minutes. She used the interview protocol as a guide and took thorough handwritten notes that were typewritten after the telephone inter- view. Because the interviews were not taped, the notes were not verbatim; but they still captured the content and the context of the directors' com- ments in detail. The consultant did not filter the responses during the note- taking process. Analysis of Interview Content The committee staff and the consultant discussed the results of the first interviews to see if there were other areas that needed to be probed. For example, in the first interviews, no one mentioned collaborative activities, but upon questioning, every QIO questioned about this in later interviews indicated their use. Similarly, although there had been no specific questions on pay for performance or electronic health records, when these topics came up repeatedly, the consultant started asking the CEOs for their views. After the completion of all 20 interviews, the consultant analyzed the contents of her notes to identify the themes that emerged in response to each topic. She then examined each theme to determine how many QIOs addressed it to identify a denominator for calculation of the percentage of QIOs that supported a particular idea or that mentioned an issue. Deliber-

OCR for page 143
154 MEDICARE'S QUALITY IMPROVEMENT ORGANIZATION PROGRAM ate quantitative analysis of the responses helps confirm the general sense that one gathers during the interviews but also helps refute strongly held opinions of a few CEOs that might not reflect those of the whole respon- dent group. During the interviews, however, some CEOs offered un- prompted comments. Therefore, a less than 100 percent response rate does not necessarily indicate disagreement among the interviewees. (In this re- port the committee has indicated, where appropriate, when all CEOs were asked the same question.) The consultant illustrated themes by the use of quotes from the CEOs. If there was a minority view or an issue that might be of potential value for further research, even if it was voiced by only one QIO, it was mentioned in the analysis for consideration in case the issue came up from other data sources. THREE-DAY BRIEFING BY CMS The study's Project Officer at CMS, Joyce Kelly, arranged an intensive series of presentations by CMS staff, both those in the Central Office and those in the various QIO Regional Offices, that covered a wide range of elements of the QIO program. She worked with IOM staff in advance to develop the agenda, soliciting questions that the committee wanted ad- dressed and proposing topics that CMS thought would be useful to the committee. The IOM requested discussions of issues such as Central Office oversight and direction, the roles of the Clinical Data Abstraction Centers, the role of QIOs in public reporting, the volume of complaints, and benefi- ciary satisfaction. The briefing provided an excellent overview and intro- duction to the program for the whole IOM project team. Although most of the time was consumed by the presentations, IOM staff had the opportu- nity to ask each speaker questions. All the materials and the PowerPoint presentations from this briefing are available on the World Wide Web (CMS, 2004). ACCESS TO QIONET CMS uses an internal website to facilitate confidential program com- munications with the QIOs and the reporting of certain types of data. This site includes the Dashboard section, which presents state-specific data, up- dated quarterly, on many of the measures used to evaluate the performance of each QIO under its contract with CMS. CMS arranged for the IOM to have access to the website during the course of the study, with the under- standing that the data would not be used with state-specific identifiers. Many of the data used in this report to assess the impacts of the QIOs on particular quality measures came from the Dashboard section of QIONet (see Table A.3 in Appendix A for a detailed list of the measures.)

OCR for page 143
STUDY APPROACH 155 FACE-TO-FACE INTERVIEWS The committee conducted two sets of formal face-to-face interviews for the study. In September 2004, during CMS's annual conference for QIOs, the committee interviewed staff and CEOs from five randomly selected or- ganizations (which held a total of seven QIOSC contracts); one interview served as a pretest of the interview protocol. All organizations selected agreed to participate. The interviewers used the same script for each of the four remaining interviews, although the topic and assignment of each QIOSC were different. Questions focused on topics such as the purpose of the QIOSC, its main activities, why it was selected for a QIOSC contract, the QIOSC's key partners, how the QIOSC interacts with the QIOs and CMS, and how the QIOSC is evaluated by CMS. Detailed written notes were taken during each interview. The information gleaned from the inter- views guided further research on QIOSC issues. The committee conducted the second set of interviews on February 25, 2005, at the AHQA Annual Technical Conference in San Francisco, Cali- fornia. The committee selected the CEOs of the QIOs of four states from the list of those that were not randomly chosen for the telephone interviews or that were not visited by representatives of the project. The CEOs of all four QIOs agreed to participate in a 45-minute interview, which primarily concerned various mechanisms for knowledge transfer. The interviews served as a pretest of the script to be used for the telephone inter- views. Committee staff took detailed written notes during the interviews. FOCUS GROUP The committee selected the CEOs of the QIOs of 11 states from those not already chosen for the telephone interviews or visited by representatives of the study. All 11 CEOs agreed to participate, although one CEO recom- mended that the medical director be interviewed as a substitute. The par- ticipants represented at least two states from each Regional Office involved in the QIO program, including both predominantly rural and predominantly urban states. Also, the committee selected states to include both those with small numbers and those with large numbers of beneficiaries and providers. The QIO subcommittee chair, a professional researcher experienced with focus groups, moderated the 90-minute discussion while committee staff took notes. The moderator used a guide that included general, open-ended questions about the direction of the program in the upcoming SOW, chal- lenges concerning information technologyrelated tasks, criteria important for assessment of the value of individual QIOs, and how QIO functions might change when all beneficiaries had electronic health records.

OCR for page 143
156 MEDICARE'S QUALITY IMPROVEMENT ORGANIZATION PROGRAM SPECIFIC DATA REQUESTS TO CMS STAFF It would have been extremely difficult to conduct this study without the excellent cooperation of individuals responsible for the QIO program's management. Throughout the course of the contract they have willingly responded to numerous requests from IOM, spent hours of their own time and their staff's time explaining the nuances and complexities of the pro- gram, collected data to fulfill requests, and shared their knowledge of CMS and the QIO program. For example, the committee requested a major amount of data on the budget for the 7th SOW, both by each state or QIO and as aggregated data at the national level. The budgets were broken down by core tasks and included costs for special studies and support contracts. The committee also requested information about the content of major sup- port and special study contracts. Other requests included requests for infor- mation about organizational structures, the results of QIO evaluations, CMS staffing needs, and the timing for the transition to the 8th SOW. FORMAL AND INFORMAL DISCUSSIONS WITH AHQA From the beginning of the study, AHQA's staff and QIO members pro- vided background information, technical advice, documents, and data as well as access to their meetings. AHQA's active support of the IOM project enhanced the receptiveness of its members' participation in the IOM study. Formal presentations and written documents are cited by author name, but information provided informally or confidentially is not. INFORMAL DISCUSSIONS WITH CONSUMER ORGANIZATIONS AND PROVIDERS The committee conducted informal discussions with people from con- sumer and beneficiary organizations as well as with practitioners and indi- viduals from provider organizations at meetings and by telephone. Time, budget constraints, and federal rules prevented the committee from per- forming more formal interview surveys with these stakeholders. However, the committee did recognize that CMS contracted with Westat to conduct surveys of providers' satisfaction with QIOs. These surveys, further discussed in Chapter 10, were conducted among nurs- ing homes, home health agencies, physicians offices, hospitals, and Medicare+Choice plans for all 53 QIOs in the 7th SOW. (For nursing homes and home health agencies, survey respondents included both identi- fied participants and nonidentified participants.) Over 21,000 providers responded to questions about their level of satisfaction, including their impressions of the effectiveness of their respective QIOs.

OCR for page 143
STUDY APPROACH 157 The federal Paperwork Reduction Act of 1995 (P.L. 104-13) prohibits the use of a survey, interview guide or systematic data collection from 10 or more persons by a federal contractor without clearance from the Office of Management and Budget. That process can sometimes take many months. Despite the committee's desire to survey providers and consumers, it deter- mined that would not be feasible within the time limits of the study's con- tract with CMS. NATIONAL CONFERENCES AND MEETINGS The committee gathered much information and understanding about the QIO program by attending national technical conferences held by CMS and AHQA, as well as smaller meetings sponsored by various organizations that focused more narrowly on particular aspects of the QIO program and related issues, such as public reporting of quality measures, information technology use in health care, and implementation of the Medicare Pre- scription Drug, Improvement, and Modernization Act of 2003 Part D prescription drug benefit. A side benefit of the meetings was the opportu- nity for informal conversations with the employees of QIOs from across the country. These discussions had some influence on the directions of the re- search and the richness of the descriptions of the QIOs, but they are not cited specifically. In addition, CMS allowed project staff to attend a strate- gic planning meeting and a confidential evaluation panel meeting. SUGGESTIONS OF "OTHER ENTITIES" Suggestions were solicited from the committee and subcommittee mem- bers and additional names were offered by the report reviewers for private businesses performing functions similar to those of QIOs. All relevant sug- gestions were included, even if the company currently offers only one of the related functions. Data on each company were pulled from its website for the table in Appendix B. Each business was offered the opportunity to up- date and verify its data. The committee tabulated the data but was unable to assess the specific capabilities or potential strengths and weaknesses of each business. RESEARCH CHALLENGES Given the timing of the congressional mandate, this study could not be built into the beginning of the 7th or the 8th SOW and produce the report on schedule. The 7th SOW was already well under way at the time that the legislation mandating this study was signed into law. Although the 8th SOW began before the end of the IOM's contract with CMS, it was too late to be

OCR for page 143
158 MEDICARE'S QUALITY IMPROVEMENT ORGANIZATION PROGRAM able to collect any new data reflecting the activities of the 8th SOW and meet the study's deadline. Thus, a study design and data collection method- ology for the project could not be included directly in the core QIO con- tracts. Although CMS instructed the QIOs to consider data requests from IOM as a separate part of their activities for the 7th SOW, it did not give them additional funds to cover their time or effort spent responding. Given those constraints, the committee could not use an experimental design. The QIO program operates in all states, participation by providers is voluntary, and all the QIOs have essentially the same basic tasks, so randomization and the use of control groups would not have been feasible without a specially designed experiment. The program was already fully operational and would continue to function and evolve during the course of the study. A Hawthorne effect might have contributed to confounding, but it was not considered a limitation of the study. For example, the committee's assessment of CMS's oversight responsibilities was based on both retro- spective and concurrent information. The fact that the U.S. Congress and the committee asked particular questions about the program could have influenced CMS management to change course or expand or reduce certain activities during the course of the study. Another constraint that affected much of the data collection effort was the need to maintain confidentiality. To gain access to many of the data from CMS and individual QIOs, the committee agreed to report only aggre- gated or deidentified data so that individual QIOs could not be identified. Because state-identified data could jeopardize the competitive business situ- ation of individual QIOs and limit their willingness to communicate on sensitive issues, the IOM agreed to maintain confidentiality at the minimum level necessary. STRENGTH OF RESEARCH CONCLUSIONS Although limitations to each of the various research methods used in this study existed, altogether they provided a substantial amount of data and information. The committee triangulated methods by checking the con- sistency of the findings obtained by the different data collection methods, such as the data collected through the web-based data collection tool, the focus group, the telephone interviews, and the site visits. Triangulation among data sources was also possible. For example, the committee com- pared the information about the program provided by CMS with the infor- mation provided by the QIOs. Because more than one committee member or IOM staff member often made site visits and more than one staff person generally attended most meetings, the committee triangulated among multiple analysts. The com-

OCR for page 143
STUDY APPROACH 159 mittee members routinely compared notes and impressions after a visit, interview, or meeting and discussed their perceptions and findings. The study puts more weight on information gathered through the SurveyMonkey web-based data collection tool, which covered all QIOs, and on data collected from the telephone interviews of a random sample of QIO CEOs than on individual interviews and informal comments, which served to strengthen consistent findings. An opinion voiced by one QIO staff member might include an important idea, but it carried more weight with the committee if multiple sources expressed similar ideas or if several different research approaches led to the same conclusions. Each research method had limitations, and it was necessary to accept CMS program data as they were given, without auditing of the data. None- theless, by using so many different methods to gather data, the committee strengthened its ability to paint a more complete and accurate picture of the QIO program. REFERENCES CMS (Centers for Medicare and Medicaid Services). 2004. The Quality Improvement Organi- zation Program: CMS Briefing for IOM Staff. [Online]. Available: http://www. medqic.org/dcs/ContentServer?cid=1105558772835&pagename=Medqic%2FMQ GeneralPage%2FGeneralPageTemplate&c=MQGeneralPage [accessed December 26, 2005]. IOM (Institute of Medicine). 2006. Performance Measurement: Accelerating Improvement. Washington, DC: The National Academies Press.