Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 257
10
Evaluation of Quality Improvement
Achieved by the QIO Program
CHAPTER SUMMARY
This chapter addresses how the Quality Improvement Organiza-
tions (QIOs) performed on each subtask in the 7th scope of work
(SOW) and how the Centers for Medicare and Medicaid Services
(CMS) evaluated them, how CMS plans to evaluate the QIOs in
the 8th SOW, the impact on quality improvement of a QIO work-
ing more intensely with an identified group of participants in a
state as compared to the improvement among all providers state-
wide, and an assessment of provider satisfaction with QIOs.
At the end of the 3-year Quality Improvement Organization (QIO) pro-
gram contracts, the Centers for Medicare and Medicaid Services (CMS)
evaluates the performance of the QIOs on the basis of demonstrated im-
provements in the quality of care provided in their respective states or juris-
dictions. The assessment of a QIO's provision of technical assistance, Task
1 of the contract for the 7th scope of work (SOW), is based on, among
other things, its ability to improve clinical quality performance measures
for activities in four provider settings: the nursing home, home health, hos-
pital, and physician's office settings. CMS calculates the scores for each
care setting. These scores lead to several questions: Is a QIO that success-
fully shows improvement in one setting likely to have high improvement
scores in other settings? Does improvement in one setting lead to improve-
ment in other settings? What patterns of improvement exist?
CMS EVALUATION OF QIO PERFORMANCE
ON TECHNICAL ASSISTANCE TASKS
Evaluation of QIOs in 7th SOW
A section of the contract for the 7th SOW (Section J-7) defines CMS's
plan for the evaluation of QIO performance and involves intricate formu-
257
OCR for page 258
258 MEDICARE'S QUALITY IMPROVEMENT ORGANIZATION PROGRAM
TABLE 10.1 Measures of Clinical Quality
Care Setting Clinical Quality Score Measure Calculation
Nursing home Relative change 1 (follow-up/baseline)
Home health Significant improvement (SI) (Number of identified participants
rate with SI)/(total number of providers
in state eligible for SI)
Hospital Reduction in failure rate ( Follow-up baseline)/(1 baseline)
Physician's office Reduction in failure rate ( Follow-up baseline)/(1 baseline)
SOURCE: CMS (2005b).
las. The evaluation formulas separately consider each subtask of Task 1,
technical assistance, which correlates with activities related to each care
setting (see Table A.5 in Appendix A). Improvement scores are a function
of clinical quality measures and provider satisfaction. CMS weights im-
provements in clinical quality more heavily than it does those in provider
satisfaction in the calculation of overall quality improvement scores for
each subtask, as depicted in the following formula1:
overall subtask improvement score =
0.8 (clinical quality score) + 0.2 (satisfaction score)
Section J-7 delineates different equations for the calculation of im-
provement in clinical quality measures for each care setting (Table 10.1).
These clinical quality measures are then individually divided by the target
levels of improvement to create a clinical quality score for each setting.
Provider satisfaction scores reflect the actual satisfaction improvement rates
divided by the target satisfaction improvement rates. Overall subtask im-
provement scores produce numerical values, which represent the quality
improvement achieved by the provider. (See Box 10.1 for an example of
scoring.) The scores are a function of change, so minimum and maximum
scores did not exist in the 7th SOW (see Chapter 2 for further discussion).
Adding to the complexity of scoring, the clinical quality score for the
nursing home, home health, and physician's office settings includes addi-
tional components: identified participant scores and statewide scores. QIOs
must work more intensely with a group of providers within the state, called
"identified participants." In the 7th SOW, the hospital subtask (Task 1c)
1Evaluation of the hospital subtask of Task 1 (Task 1c) differs by weighting clinical quality
0.75 and satisfaction 0.25.
OCR for page 259
EVALUATION OF THE QIO PROGRAM 259
did not require identified participants. Because QIOs work more closely
with identified participants, improvements made by identified participants
are weighted more heavily than statewide improvements in the calculation
of clinical quality scores. However, QIOs spend only approximately one-
third of the appropriated funds on Task 1 subtasks with identified partici-
pants; the remaining two-thirds of the funds are devoted to making im-
provements statewide (CMS, 2004b). CMS evaluated QIO work with
underserved populations (Task 1f) on the basis of the judgment of the
Project Officer; QIOs either pass or fail this subtask without an official
numerical score. A hypothetical example of the scoring system is provided
in Box 10.1.
Contract Renewal in the 7th SOW
To merit noncompetitive renewal of contracts for the 8th SOW, CMS
required the QIOs to meet the performance criteria on 10 of the 12 subtasks
in Tasks 1 to 3 of the contract for the 7th SOW. For Tasks 1a to 1e and
Task 2b, the quantifiable subtasks, the QIOs had to score 1.0 or higher to
meet the performance criteria. For the subtasks for which a QIO did not
meet the performance criteria, the QIO had to meet the following mini-
mums: (1) a score of 0.6 or higher on quantifiable subtasks (Tasks 1a
through 1e and Task 2b) and (2) approval of the Project Officer on all
nonquantitative tasks (Tasks 1f, Task 2a and 2c, and Tasks 3a through 3c).
If these standards were not met, a QIO had to compete for the contract for
the 8th SOW (CMS, 2002).
Evaluations in 8th SOW
For the 8th SOW, CMS developed evaluation formulas similar to those
used in the 7th SOW. However, as discussed in Chapter 8, the focus of the
8th SOW broadened to include the areas of (1) clinical performance mea-
sure results, (2) clinical performance measurement and reporting, (3) sys-
tems improvement, (4) process improvement, and (5) organization culture
change. CMS weights each of these areas differently, depending on the
subtask (see Table A.6 in Appendix A). As in the 7th SOW, the improve-
ments made by identified participants tend to carry greater weight than the
improvements made statewide, although this does not hold true for
all subtasks. The passing scores vary by subtask. Evaluation of provider
satisfaction differs in the 8th SOW, with the addition of a stakeholder
and a knowledge and perception survey (see the discussion below). All
Task 1 subtasks require an 80 percent overall satisfaction and knowledge-
perception score to pass. For all subtasks (for Tasks 1 and 3), CMS rates
QIO performance as either excellent pass, full pass, conditional pass, or
OCR for page 260
260 MEDICARE'S QUALITY IMPROVEMENT ORGANIZATION PROGRAM
BOX 10.1 Hypothetical Scoring Example
Charles Hill Nursing Home has volunteered to work with Quality-
Quest QIO as part of QualityQuest's identified participant group. Qual-
ityQuest would like to work on reducing the incidences of the following
selected four quality measures: (1) chronic care residents with pain; (2)
chronic care residents with pressure sores; (3) acute care residents with
pain; and (4) acute care residents with delirium.
In 2002, the baseline, Charles Hill reported that 22 percent of its
chronic care residents reported having pain. In 2005, the follow-up, the
nursing home reported that only 11 percent of its chronic care residents
reported having pain. For the chronic pain measure, the relative change
for Charles Hill is as follows:
Relative change = 1 (performance at follow-up/performance
at the baseline)
= 1 (11 percent/22 percent) = 0.50
(50 percent relative change)
This calculation is repeated for all four measures. The worst score
on a performance measure is dropped, and the scores for the remaining
three measures are summed and averaged. This average score is then
averaged with (1) those of the other identified participants in the state to
create the QIO's identified participant score and (2) all the nursing homes
in the state for the QIO's statewide score.
QualityQuest Identified Participant Clinical Quality Score
Calculation of Identified Participant Weight
Identified participant nursing homes as a percentage of all nursing
homes in the state = 13.8 percent
Target percent participating = 10 percent
Identified participant weight = 0.44 × (percent participating identified
participants/target percent participating)
Identified participant weight = 0.44 × (13.8 percent/10 percent) = 0.6
Calculation of Identified Participant Score
Baseline average of four measures: 9 percent
Follow-up average of four measures: 6.9 percent
not pass. CMS deems scores (see the discussions below for the derivation
of the scores) greater than 0.95 as excellent pass, scores between 0.75 and
0.94 as full pass, scores between 0.65 and 0.74 as conditional pass, and
any score below 0.65 as not pass. All QIOs must complete the core activi-
ties of each subtask to be considered for noncompetitive contract renewal.
OCR for page 261
EVALUATION OF THE QIO PROGRAM 261
Identified participant relative change = 23 percent improvement on
the four measures
Target level of improvement for identified participants (set by CMS):
8 percent
Identified participant score = weight × (relative change in improve-
ment/target improvement)
Identified participant score = 0.6 × (23 percent/8 percent) = 1.72
QualityQuest Statewide Clinical Quality Score
Calculation of Statewide Weight
Statewide weight = 0.8 identified participant weight
Statewide weight = 0.8 0.6 = 0.2
Calculation of Statewide Score
Baseline average of four measures: 12 percent
Follow-up average of four measures: 11.1 percent
Relative change statewide: 7.5 percent average improvement on the
four measures
Target level of improvement statewide (set by CMS): 8 percent
Statewide score = weight × (relative change in improvement/target
improvement)
Statewide score = 0.2 × (7.5 percent/8 percent) = 0.19
QualityQuest Satisfaction Score (surveys completed only by
identified participants)
Identified participants: 89 percent
More than 80 percent of QualityQuest's identified participants were
satisfied with the assistance that they received. Therefore, QualityQuest
has passed the satisfaction component and will receive a score of 0.2.
Overall Nursing Home Improvement Score
Overall score = 0.8 (clinical quality score) + 0.2 (satisfaction score)
= 1.72 + 0.19 + 0.2 = 2.11
QualityQuest has scored above 1 and therefore has passed the nurs-
ing home subtask.
In addition, eligibility for noncompetitive renewal is generally contingent
on the QIO achieving at least one conditional pass; noncompetitive re-
newal is rewarded if the QIO receives a full pass or an excellent pass on
seven of the nine subtasks. Upon reception of a not pass on any subtask,
CMS may invite the QIO to the evaluation panel (CMS, 2005c).
OCR for page 262
262 MEDICARE'S QUALITY IMPROVEMENT ORGANIZATION PROGRAM
TABLE 10.2 QIO Results for Task 1 of the 7th SOW
Percentage of
Task 1 Subtask Range of Scoresa QIOs Passing Other Subtasks
Nursing home (Task 1a) 1.717.37 100 No
Home health care (Task 1b) 0.222.2 35 No
Hospital (Task 1c) 0.773.2 94 No
Physician's office (Task 1d) 0.132.4 56 No
Underserved/rural (Task 1e) 0.61.6 94 No
Managed care (Task 1f) NA 100 No
NOTE: NA = not applicable.
aThe range of scores indicates the lowest and the highest scores achieved among all QIOs in
the 7th SOW, based on the above overall subtask improvement scoring formula.
SOURCE: Derived from data collected from CMS's Dashboard section on its internal
website.
IOM ANALYSIS OF TASK 1 PERFORMANCE BY QIOS
The Institute of Medicine (IOM) committee performed the analyses
described in this section on the basis of CMS's evaluation scores for the
QIOs in Task 1 of the contract for the 7th SOW. The IOM committee
collected data from CMS's Dashboard section, located on its internal
website, QIONet (see the discussion below and in Chapter 13), as well as
other data from CMS, as requested. In the interest of understanding why
some states and provider settings showed more improvement on QIO qual-
ity measures than others, the committee conducted correlations between
how the QIOs scored per clinical quality improvement task and the poten-
tial presence of confounding variables, such as performance on other Task
1 subtasks, the spending per beneficiary on that subtask, the QIO contract
round, the QIO region, and provider satisfaction.2 The data used to deter-
mine correlations for all tasks were current through December 2004 and
were obtained from the Dashboard section of CMS's internal website, un-
less noted otherwise. Table 10.2 summarizes the results.
2These correlations used QIO scores determined by CMS evaluations described in Section J-
7 of the 7th SOW and discussed earlier in this chapter. Correlations between improvement
scores, spending per beneficiary, provider satisfaction, QIO contract round, and region were
determined by calculation of the correlation coefficient, r. On a scale from 1 to 1 (with 1
indicating a direct negative correlation, 0 indicating no correlation, and 1 indicating a direct
OCR for page 263
EVALUATION OF THE QIO PROGRAM 263
Correlation of Overall Subtask Score with:
Spending per Beneficiary QIO Round QIO Region Provider Satisfaction
No No No No
No Some No No
No No Some No
No No Some No
No No No NA
No No No NA
For all clinical quality improvement tasks, the overall improvement
scores for each Task 1 subtask were not found to correlate with improve-
ment scores in any of the other Task 1 subtasks. For example, a QIO's
success in improving the quality of care provided by nursing homes did not
indicate that the QIO would necessarily have success in improving the qual-
ity of care provided by hospitals. Also, the IOM committee did not detect
any correlations between subtask improvement scores and QIO spending
per beneficiary or provider satisfaction. Some association between the over-
all subtask score and the QIO contract round was shown only for the home
health setting, suggesting that QIOs beginning in later contract rounds
achieved greater improvements. The scores for hospitals and physicians'
offices showed some association with QIO region.
As noted above, CMS calculated the scores of QIO performance in the
7th SOW for each subtask. However, no overall score for all technical assis-
tance subtasks in Task 1 combined was calculated. During the IOM com-
positive correlation), all r values were less than 0.3. QIO round refers to the CMS contract
cycle, which is implemented in a three-stage approach separated by 3 months for each stage.
QIOs are split into four geographic regions: Boston, Dallas, Kansas City, and Seattle. Differ-
ences between QIO improvement scores per task and QIO region were calculated by using a
one-way analysis of variance (ANOVA) test, followed by the Scheffe test; significance was
considered if the P value was <0.05.
OCR for page 264
264 MEDICARE'S QUALITY IMPROVEMENT ORGANIZATION PROGRAM
mittee's study, the question of being able to identify the overall high- and
low-performing QIOs thus arose. Such an identification would, however,
necessitate the derivation of a single composite score that could be used to
rate the progress in quality improvement that each QIO had made in all
four health care settings. Although the identification of high and low per-
formers in a single care setting was possible (this was done by the Best
Practices Special Study for the hospital setting) (CMS, 2005a), this was not
the case when an attempt was made to include the scores for all care settings.
The IOM committee used multiple methods in its attempt to identify
the overall high and low performers, but no clear method for the categori-
zation of the QIOs on the basis of CMS scores could be devised. For in-
stance, the scores for all four settings could be added together; however, if a
QIO scored a 5.4 in the nursing home setting but scored 0.1 in the
physician's office setting, there would be no way to tell that the QIO's
technical assistance might have worsened in physicians' offices during the
7th SOW. The IOM committee encountered other challenges, such as how
to ensure that all subtasks received equal weight and how to set the cutoff
between high and low performance. The lack of a correlation among tasks
and wide variations in performance, complemented by the complexity of
the QIO program, thus made the committee unable to identify the high-
and low-performing QIOs for Task 1 as a whole in a valid manner.
Although QIOs in general achieved improvements in measures that were
calculated at the baseline (usually at the beginning of the 7th SOW) and
that were then remeasured at the end of the 7th SOW, many of these im-
provements were not statistically significant. Many other factors may have
affected provider performance, such as quality interventions from other or-
ganizations, but CMS did not document these factors. The evaluation in the
7th SOW focused on specific measures, subtasks, and individual QIO per-
formance and does not demonstrate the actual impact of the QIO program
or attribute improvements to QIO interventions; however, they do show
changes in specific quality measures in each state.
Nursing Homes
7th SOW
Under Task 1a of the 7th SOW, each QIO worked to improve quality-
of-care measures for identified participants as well as for all nursing homes
statewide and developed a plan for this work as one of its four deliverables
(see Table A.7 in Appendix A). For evaluation purposes, CMS used the
reduction in failure rate to define "improvement." CMS based the evalua-
tion of success on three components. First, the QIO had to demonstrate at
least an 8 percent improvement statewide on three to five QIO-selected clini-
OCR for page 265
EVALUATION OF THE QIO PROGRAM 265
cal measures (see Table A.3a in Appendix A). Second, the QIO had to dem-
onstrate an 8 percent improvement on these measures for the identified par-
ticipants. Finally, according to provider surveys conducted by CMS, at least
80 percent of nursing homes, whether they were identified participants or
not, had to report an adequate level of satisfaction with the work of the
QIOs. CMS rolled these three components into an overall score for Task
1a, weighting statewide improvement less heavily than improvement among
the identified participants. Clinical improvement constituted 80 percent of
the score, with provider satisfaction levels counting for 20 percent. QIOs
that scored a value of 1 or greater passed the task; a score of less than 1 was
considered failing (CMS, 2002).
For Task 1a, the IOM committee found no correlations between over-
all improvement scores and QIO spending per beneficiary, nursing home
satisfaction rates, the QIO contract round, or the QIO region. In the 7th
SOW, all QIOs achieved a passing score on this subtask (for the scoring
formula, see Table A.5 in Appendix A), with improvement scores ranging
from a low of 1.71 to a high of 7.37. Data from CMS's Dashboard shows
that the identified participants had greater levels of improvement across all
nursing home measures, with an average relative change of 46.5 percent, in
comparison with the statewide average relative change of 16.7 percent. As
of December 2004, all QIOs recruited more than the required 10 percent of
identified participants, with one QIO involving up to 100 percent of all
nursing home providers in the state (CMS, 2004a). Nationwide, during the
7th SOW half of all nursing homes participated with their QIOs with vari-
ous levels of involvement in multiple interventions (personal communica-
tion, Y. Harris, December 28, 2004).
8th SOW
In the 8th SOW, the QIOs must provide 10 deliverables (see Table A.7
in Appendix A) for the nursing home task. A notable deliverable for evalu-
ation is that QIOs set their own statewide targets for clinical improvement
for all measures in this task. The identified participants must set their own
personal targets for clinical improvement (CMS, 2005c).
As described earlier in this chapter, evaluation of QIO performance in
the 8th SOW will build upon the components of the evaluation of perfor-
mance in the 7th SOW (clinical quality and provider satisfaction) to in-
clude the following, where applicable: clinical performance measure re-
sults, process improvement, and organization culture change (see Table A.6
of Appendix A). As in the 7th SOW, CMS will weight the identified par-
ticipant scores more heavily than the statewide scores, and the most em-
phasis will be placed on improvements in the clinical performance measure
results. The total score for this subtask is 1.1 points, but the total possible
OCR for page 266
266 MEDICARE'S QUALITY IMPROVEMENT ORGANIZATION PROGRAM
score is 1.3 points, as the process improvement activities are worth 0.2
point of extra credit. However, the QIOs must meet the following core
standards in the following areas: clinical performance measure results and
organization culture change, as well as the satisfaction and knowledge-
perception criteria. A score of 80 percent or higher on the satisfaction and
knowledge-perception activities adds 0.1 point to the overall subtask score
(this is discussed later in this chapter) (CMS, 2005c).
Home Health
7th SOW
The three QIO deliverables for Task 1b in the 7th SOW included the
QIO training of home health agencies on the Outcome-Based Quality Im-
provement System (OBQI) (see Table A.7 in Appendix A). The QIOs had to
demonstrate improvement in OBQI measures for the set of identified par-
ticipants. The evaluation did not include a measure of statewide improve-
ment. CMS based the evaluation of success on two components: (1) a statis-
tically significant improvement in at least one indicator by 30 percent of the
identified participants and (2) satisfaction with QIO performance by at least
80 percent of the participating home health agencies. Completion of the
home health task required a score of 1 point or higher to pass (see Table A.5
in Appendix A) (CMS, 2002).
As in Task 1a, the IOM committee found no correlation between home
health improvement scores and spending on that task per beneficiary, pro-
vider satisfaction, or QIO region. The QIOs in the first contract round,
however, appeared to have higher evaluation scores than the QIOs in the
second and third contract rounds, suggesting that the length of time in-
vested in change may influence improvement in this subtask. Work in this
provider setting was new in the 7th SOW, and 35 percent of the QIOs
achieved a passing score. However, the contracts for the 65 percent of the
QIOs that failed (scoring below 1.0) were not put up for bid on the basis of
this failure, as most scored between 0.6 and 1.0. As of December 2004, the
scores for the home health subtask ranged from 0.22 to 2.2 points (CMS,
2004a). These are not final scores for the 7th SOW, as the results for the
second and the third rounds were not available as of this writing. An aver-
age of 22 percent of the identified participants in all QIOs attained signifi-
cant improvement, which was less than the target of 30 percent. Only 17
QIOs reported that more than the required number of identified partici-
pants had significant improvement; one state reported that 73 percent of all
home health providers in the state had significant improvements.
OCR for page 267
EVALUATION OF THE QIO PROGRAM 267
8th SOW
In the 8th SOW, the QIOs are responsible for providing nine deliver-
ables (see Table A.7 in Appendix A), including lists of identified partici-
pants, the selection of one Outcome and Assessment Information Set
(OASIS) measure (see Chapter 8 for a description of OASIS measures) for
statewide improvement, and the home health agencies' plans to reduce
acute care hospitalizations. Unlike under the 7th SOW, the QIOs will be
evaluated on the basis of the improvements both statewide and among the
identified participants. The target reduction in failure rates varies by mea-
sure and is prescribed by CMS for this task, unlike in the nursing home
setting, in which each QIO is allowed to define its own targets. The targets
for statewide reductions in failure rates for home health settings are lower
than the targets for identified participant reductions (Table 10.3). The tar-
gets for identified participants were derived in order to match the 75th
percentile of the 7th SOW. Selection of statewide targets included consid-
eration of the rates for identified participants as well as the complexity of
the measures.
TABLE 10.3 Task 1b Target Reduction Rates in the 8th SOW
Target Target Identified
Statewide RFRa Participant RFR
OASIS Publicly Reported Measures (percent) (percent)
Improvement in bathing 14 34
Improvement in transferring 8 31
Improvement in ambulation and locomotion 9 20
Improvement in management of oral medications 8 18
Improvement in pain interfering with activity 11 41
Improvement in status of surgical wounds 6 38
Improvement in dyspnea 17 41
Improvement in urinary incontinence 9 34
Provision of any emergent careb
Acute care hospitalization 30 50
Discharge to community 10 35
aRFR = reduction in failure rate. Identified participants are excluded from calculation of
the statewide reduction in failure rate.
bThis measure will not be used in the 8th SOW.
SOURCE: CMS (2005c).
OCR for page 268
268 MEDICARE'S QUALITY IMPROVEMENT ORGANIZATION PROGRAM
The evaluation plan for the 8th SOW weights clinical performance mea-
sures more heavily than the other components of evaluation for this task,
with particular emphasis on acute care hospitalization. In addition to im-
proving clinical performance measures statewide, the QIOs must improve
immunization assessment processes statewide. As in Task 1a, the QIOs must
achieve the targets set for the following core activities: improvements by
identified participants in one selected OASIS measure, acute care hospital-
ization, and telehealth. Statewide improvements must be met for clinical
performance for acute care hospitalization, immunization surveys, and the
satisfaction and knowledge-perception standards. The total score for this
subtask is 1.0 point; 0.27 point of partial credit and extra credit is avail-
able, making the total possible score 1.27 points. QIOs that achieve the
satisfaction and knowledge-perception standards receive 0.1 point (CMS,
2005c).
Hospitals
7th SOW
The evaluation plan for Task 1c of the 7th SOW based success on state-
wide improvements in quality-of-care measures and hospital satisfaction
with QIO performance. The only deliverable for Task 1c was a list of con-
tact information for every hospital in the state (see Table A.7 in Appendix
A). CMS calculated a combined topic average based on improvements on
the quality indicators in each of the four topic areas. The QIO had to dem-
onstrate at least an 8 percent reduction in failure rate for a combined topic
average statewide. Additionally, a provider satisfaction survey had to indi-
cate that at least 80 percent of the hospitals in the state or jurisdiction were
satisfied with the activities of the QIO. CMS determined scores above 1.0
for this task to be passing (CMS, 2002).
The IOM committee found that the hospital measure improvement rates
did not correlate with spending on this task per beneficiary, the QIO con-
tract round, or provider satisfaction. An evaluation of hospital scores by
QIO region did detect differences among regions (P < 0.05). This difference
was driven by higher scores in the Kansas City region compared with those
in both the Seattle and the Boston regions; the score for the Dallas region
was not significantly different from that for any other region. On the basis
of data from Dashboard, 94 percent of the QIOs appear to have achieved a
passing score on the hospital subtask in the 7th SOW as of the end of 2004
(CMS, 2004a). At that time, the scores on this subtask ranged from 0.77 to
3.2 points.
OCR for page 269
EVALUATION OF THE QIO PROGRAM 269
8th SOW
For the 8th SOW, CMS subdivided hospital work into two subtasks:
Task 1c1 focuses on hospitals in general, whereas Task 1c2 focuses on
critical access hospitals. Successful performance on Task 1c1 requires the
provision of six deliverables (see Table A.7 in Appendix A), including the
implementation of systems improvement interventions, such as computer-
ized provider order entry, bar coding, or telehealth. The eight deliverables
for Task 1c2 include the submission of quality improvement measures, in-
terventions, and change models as well as a safety culture survey (CMS,
2005c).
CMS bases its evaluations of these subtasks on both statewide and iden-
tified participant improvements. Statewide improvement carries more
weight than identified participant improvement in the evaluation of Task
1c1; in Task 1c2, identified participant improvement weights more heavily
than statewide improvement. Statewide, the core activities focus on the re-
porting of Hospital Quality Alliance measures (see Table A.3a in Appendix
A) for both Tasks 1c1 and 1c2. For Task 1c1, identified participant work
focuses on improving clinical performance measure results, processes for
surgical care, and the use of electronic clinical information systems. For
Task 1c2, identified participants must address the culture of safety in the
critical access hospital, as well as the implementation of electronic systems.
Task 1c1 is evaluated out of a total score of 1.1 points, with 0.2 point of
extra credit and partial credit available, for a possible score of 1.3 points.
The total possible score for Task 1c2 is 1.35 points. The meeting of satisfac-
tion and knowledge-perception standards adds 0.1 point to the overall
subtask score (CMS, 2005c).
Physician Office and Physician Practice
7th SOW
Task 1d had two deliverables, including a listing of all identified par-
ticipants along with the participant's Unique Physician Identification Num-
ber3 via PARTner (see Chapter 13 for a description of PARTner). CMS
used three criteria to judge a QIO's success in Task 1d. First, the QIO had
to demonstrate an 8 percent overall improvement statewide on quality-of-
care measures for diabetes, cancer, and immunizations using a combined
topic average. The score included weighting of the Health Plan Employer
3Medicare assigns a Unique Physician Identification Number to each provider or practition-
er who participates in the Medicare program.
OCR for page 270
270 MEDICARE'S QUALITY IMPROVEMENT ORGANIZATION PROGRAM
Data and Information Set (HEDIS) data so that Medicare+Choice benefi-
ciaries and fee-for-service beneficiaries could be considered equally. The
second criterion for the successful completion of Task 1d was achieving at
least an 8 percent improvement on measures related to diabetes and cancer
screening for the identified participant group. Finally, provider surveys had
to yield at least an 80 percent satisfaction rate among the identified partici-
pants. QIOs scoring 1.0 point and above passed the physician's office task
(CMS, 2002)
For this task, the IOM committee found no correlations between evalu-
ation scores and QIO spending per beneficiary, the QIO contract round, or
physician's office satisfaction. In testing for variations among QIO regions,
the committee detected significant differences (P < 0.05) in physician's of-
fice scores, driven by the higher average scores in the Boston region com-
pared with those in the Seattle region. Differences in performance among
other regions were not significant. As of this writing, 56 percent of the
QIOs appear to have achieved a passing score on this subtask. As with the
home health care analysis, this does not insinuate that 44 percent of the
QIO contracts were not automatically renewed, as many could have scored
above 0.6 point. The scores ranged from 0.13 to 2.4 points. Physicians'
offices volunteering to be an identified participant tended to have higher
reductions in failure rates than physicians' offices statewide (with average
reductions of 12.1 and 6.1 percent, respectively). As of December 2004,
only two QIOs did not include the required 5 percent of active primary care
physicians, and one QIO included up to 16 percent of all physicians' offices
in the state (CMS, 2004a).
8th SOW
In the 8th SOW, CMS separated work with physicians' practices into
three groups: physician practice (Task 1d1), physician practice for under-
served populations (Task 1d2), and physician practice and pharmacy: Part
D prescription drug benefit (Task 1d3). Task 1d1 calls for 12 deliverables,
including documentation that assistance was provided to Medicare Advan-
tage plans, documentation of support for the Physician Voluntary Report-
ing Program, implementation of electronic health records, and lists of iden-
tified participant groups. The core activities for this subtask are systems
improvement by the identified participant groups and satisfaction and
knowledge-perception surveys. Clinical performance measures are evalu-
ated by the Project Officer and have been broadened to include the provi-
sion of statewide support for the Physician Voluntary Reporting Program,
prevention and disease-based care processes, Medicare Advantage plans,
End-Stage Renal Disease Networks, and offices participating in the Medi-
care Management Demonstration Project. The areas of focus for the identi-
OCR for page 271
EVALUATION OF THE QIO PROGRAM 271
fied participant groups are clinical performance measurement and report-
ing, process improvement, and systems improvement. The total score is 1.2
points; partial credit is offered for the systems improvement dimension of
the subtask. The QIOs must achieve an 80 percent rate on the satisfaction
and knowledge-participation surveys for 0.1 point (CMS, 2005c).
The four deliverables for Task 1d2 mainly consist of cultural and lin-
guistic competencies, addressed primarily by the identified participant
group. The identified participant activities for systems improvement and
process improvement are the core activities for this subtask. Clinical mea-
sures focus on statewide improvements in immunization rates, mammog-
raphy rates, and diabetes measures for underserved populations. CMS
weights statewide improvements less heavily than improvements among
the identified participants. The total score is 1.0 point; the satisfaction and
knowledge-perception surveys are core activities that can add 0.1 point to
the overall subtask score (CMS, 2005c).
Task 1d3--physician practice and pharmacy: Part D prescription drug
benefit--has nine deliverables, including assessments of electronic prescrib-
ing feasibility, the baseline performance of Task 1d3 measures, and com-
prehensive responses to beneficiary complaints about prescription medica-
tions. Evaluation of performance on this subtask will be determined solely
on the basis of the identified participants' activities and will be performed
by the Project Officer and the Task 1d3 Government Task Leader. Achieve-
ment of the satisfaction and knowledge-perception surveys requirement is
required to pass this subtask (discussed further in this chapter) (CMS,
2005c).
Underserved and Rural Beneficiaries
7th SOW
For Task 1e of the 7th SOW, CMS required the QIOs to demonstrate a
reduction in a chosen disparity in a defined population from the baseline to
remeasurement by the completion of three deliverables. CMS compared the
improvements made by an identified group with those made by a reference
group of beneficiaries; the difference in these improvements between the
two groups had to be smaller at the remeasurement than at the baseline.
(This is a significant distinction, because a concomitant improvement in the
reference group can affect the difference in improvement between the two
groups.) QIOs could earn extra points by fully documenting the description
of the intervention group, the relationship between the disparity and the
chosen intervention, and the effectiveness of the intervention (CMS, 2002).
CMS considered a score of 1.0 point to be passing, whereas the maximum
obtainable score was 1.6 points.
OCR for page 272
272 MEDICARE'S QUALITY IMPROVEMENT ORGANIZATION PROGRAM
The IOM committee found no correlations between overall Task 1e
scores and spending on this subtask per beneficiary per QIO or QIO round.
In total, 94 percent of the QIOs passed this subtask; the scores ranged from
0.6 to 1.6 points (personal communication, J. Kelly, July 5, 2005).
8th SOW
In the 8th SOW, CMS has incorporated the underserved and rural ben-
eficiary focus of the 7th SOW into selected subtasks of Task 1 (CMS,
2005c).
Managed Care Organizations
7th SOW
Successful performance of Task 1f required two deliverables, including
submission of a plan of action to incorporate Medicare+Choice organiza-
tions into all other quality improvement tasks. Under Task 1f, the Project
Officer considered whether the QIO demonstrated adequate initiative to
include Medicare+Choice organizations in Tasks 1a to 1e. CMS also con-
sidered the Medicare+Choice organizations' satisfaction with their in-
teractions with QIOs, with an expected minimum satisfaction level of
80 percent. Additionally, CMS looked to Medicare+Choice Quality Review
organizations or other accreditation organizations to determine whether
the Medicare+Choice organizations achieved demonstrable success in their
Quality Assessment Performance Improvement projects. For overall evalua-
tion of a QIO's success on Task 1f, the Project Officer gave equal weight to
Quality Assessment and Performance Improvement projects (see Chapter 8)
in conjunction with QIO technical assistance and achievement of an 80 per-
cent satisfaction level (CMS, 2002). All QIOs with Medicare+Choice orga-
nizations in their states passed Task 1f.
8th SOW
QIO work with Medicare Advantage plans (formerly Medicare+Choice
organizations) is incorporated into the appropriate settings during the 8th
SOW. CMS did not separate out this work as a distinct task (CMS, 2005c).
IMPACT OF INTENSE QIO ASSISTANCE
In the 7th SOW, subsets of providers in the nursing home, home health
care, and physician's office settings volunteered to work more closely with
the QIO than other providers in the state; the QIO program recognizes
OCR for page 273
EVALUATION OF THE QIO PROGRAM 273
these providers as "identified participants." CMS's analyses of QIO inter-
ventions in the 7th SOW attempted to determine whether the identified
participants achieved greater improvements than the nonidentified partici-
pants; one goal of such determinations is to show whether more intense
QIO interactions yield a better quality of care. CMS shared preliminary
results in public forums, such as the American Health Quality Association
Technical Meeting in February 2005. The results suggested that the identi-
fied participants scored higher than the nonidentified participants (Rollow,
2004).
Upon initial analysis of these data, the IOM committee noted that the
results are limited by many confounders, including the following: potential
biases between identified participants and other providers, ignorance of
other quality improvement interventions in which the providers participate,
limited knowledge of the QIOs' statewide quality improvement efforts, and
the impacts of these other interventions. In addition, the methods and time
frames for determining the impacts of the intensity of work with the QIOs
are not consistent across care settings (i.e., inconsistent remeasurement pe-
riods) (personal communication, W. C. Rollow, July 8, 2005).
CMS plans to publish a study with more complete data that support
these findings; an advance summary of that study was provided to the com-
mittee. The study assesses the level of intensity of provider work with the
QIOs by separating the providers into groups (Table 10.4). Providers work-
ing more intensely with QIOs appear to have achieved greater improve-
ments on measures than those that did not. However, the statistical signifi-
cance of these improvements could not be determined from this summary.
CMS concluded that QIO assistance is, in general, valuable for improving
quality (CMS, 2005b).
PROVIDER SATISFACTION
7th SOW
In the 7th SOW, the scores on which QIO performance was based in-
cluded dimensions of provider satisfaction with their interactions with QIOs
as well as the relative improvement on various performance measures.
QIOs needed to attain an 80 percent rate of provider satisfaction for Tasks
1a, 1b, 1c, 1d, and 1f, which contributed to 20 percent of the QIO's overall
score on each subtask (see Table A.5 in Appendix A). CMS contracted with
Westat to survey all the identified participants; CMS also included a selec-
tion of nonidentified participants from the nursing home and home health
settings for these surveys. The surveys yielded a response rate of 90 percent
from 21,710 providers nationwide. Westat sent the surveys by mail, the
Internet, and telephone; and the provider representative who served as the
OCR for page 274
274 MEDICARE'S QUALITY IMPROVEMENT ORGANIZATION PROGRAM
TABLE 10.4 Summary of CMS Evaluation of 7th SOW
Care Setting
Physician's
Evaluation Parameter Nursing Home Home Health Hospital Office
Levels of intensity Nonidentified Nonidentified NAb Nonidentified
participants, participants, participants
identified identified and identified
participants, participants, participants
and select and select
identified identified
participantsa participants
Number of measures 4/6 measures 10/11 measures 17/18 measures 2/4 measures
improved improved improved improved improved
Number of providers 13,000 providers 6,000 providers 3,700 providers 1.7 million
or beneficiaries beneficiaries
(approximate)
Measurement periodc Q2c 2002 April 2002 2000 20012004
Q2 2004 January 2005 Q4c 2004
aSelect identified participants are identified participants focusing on improving a specific
quality measure.
bNA = hospital data were unavailable for this summary.
cQ2 = second quarter; Q4 = fourth quarter.
SOURCE: CMS (2005b).
main point of contact with the QIO completed the survey. Questions ranged
from asking about QIO-to-provider communications processes (i.e., "How
satisfied or dissatisfied were you with the one-to-one e-mail communica-
tion?" and "How satisfied or dissatisfied were you with the timeliness of
the QIO's response to your question or request for assistance?") to the
content and outcome of QIO technical assistance (i.e., "When implement-
ing our quality improvement projects, we used the information received
from this QIO" and "Medicare beneficiaries are now better served by our
organization thanks to the assistance we received from this QIO") (Westat,
2005:5859).
The surveys covered the following six topic areas: access to health care,
timeliness of response, information dissemination, technical support, pro-
fessionalism and courtesy, and overall responsiveness to needs. The scores
incorporated the aggregated survey responses, with each topic area weighted
equally.
Westat determined satisfaction on the basis of a range of choices: very
satisfied, somewhat satisfied, neither satisfied nor dissatisfied, somewhat
dissatisfied, and very dissatisfied. When applicable, Westat used compa-
OCR for page 275
EVALUATION OF THE QIO PROGRAM 275
TABLE 10.5 Provider Satisfaction with QIOs
Task and Survey Respondent Provider Satisfaction (percent)
Task 1a Nursing home identified participants 91
Nursing home nonidentified participants 75
Task 1b Home health identified participants 94
Home health nonidentified participants 81
Task 1c Hospital 90
Task 1d Physician's office 85
Task 1e Underserved and rural NAa
Task 1f Medicare+Choice 93
aNA = not applicable.
SOURCE: Westat (2005).
rable choices of strongly agree, somewhat agree, neither agree nor disagree,
somewhat disagree, and strongly disagree. Westat included responses of
very satisfied and somewhat satisfied in the calculation of the satisfaction
rate. Table 10.5 shows the providers' overall satisfaction with QIOs, listed
by subtask.
Westat also performed an analysis of "key drivers" for satisfaction.
That analysis determined that 93 percent of all providers responded that
they were satisfied with both the ease of access of contacting QIOs and the
timeliness of the response. Satisfaction with the provision of technical sup-
port varied depending on the type of assistance and ranged from a low of
84 percent (telephone conference calls) to 93 percent (training workshops,
one-on-one telephone calls, and one-to-one e-mails). The overall value of
QIO assistance was not equal across all types of providers and differed by
participant status, as depicted in Table 10.6. The report suggests that the
perceived usefulness may predict higher overall satisfaction with the QIOs
(Westat, 2005).
8th SOW
In the 8th SOW, the QIO program will continue to evaluate the satis-
faction of providers as a component of Task 1 subtasks. However, a new
topic will be added to the survey and will ask providers about their knowl-
edge and perception of both CMS and the QIO with which they work.
Under the umbrella of satisfaction surveys, an independent contractor will
also survey the beneficiaries about their interactions with the QIO pro-
gram. Previously, the only beneficiary satisfaction surveys administered
were by the QIOs themselves as part of their case review activities (see
Chapter 12). In the 8th SOW, QIOs achieving at least an 80 percent score
OCR for page 276
276 MEDICARE'S QUALITY IMPROVEMENT ORGANIZATION PROGRAM
TABLE 10.6 Value of QIO Assistance
Task and Survey Respondent Overall Value (percent)
Task 1a Nursing home identified participants 81
Nursing home nonidentified participants 60
Task 1b Home health identified participants 88
Home health nonidentified participants 70
Task 1c Hospital 77
Task 1d Physician's office 72
Task 1e Underserved and rural NAa
Task 1f Medicare+Choice 74
aNA = not applicable.
SOURCE: Westat (2005).
on the satisfaction and knowledge-perception surveys--core activities for
all parts of Task 1--will add 0.1 point to their subtask scores. Task 1d3 is
the only exception, for which 80 percent satisfaction is required to pass
(CMS, 2005c).
A stakeholder survey and a QIO survey will also be introduced in the
8th SOW. The stakeholder survey will be conducted twice during the 3-year
contract period and will measure how key health care system stakeholders
view the QIOs and CMS. The QIO survey will allow the QIOs to voice
their satisfaction or dissatisfaction about CMS's operation and manage-
ment of the overall QIO program (personal communication, M. G. Wang,
July 6, 2005). The QIOs will also be surveyed on their views toward the
QIO Support Centers (QIOSCs), which will contribute to the formal per-
formance evaluation of each QIOSC (personal communication, J. Taylor,
April 29, 2005).
Although the IOM committee believes that surveys can be valuable
sources of information for determination of the impact of the QIO pro-
gram, as discussed in Chapter 5, the surveys must be designed and adminis-
tered in a fair and clear manner with specific, actionable questions. In addi-
tion, analyses should be conducted to discern the characteristics of QIOs
receiving high and low satisfaction ratings. Also, in keeping with the trans-
parency of public reporting expected of providers, it would be appropriate
to make public the various satisfaction scores of each QIO.
PROGRAMWIDE EVALUATION OF IMPACT
As shown throughout this chapter, evaluations of quality impacts have
focused on specific measures, subtasks, and individual QIOs. Some data are
OCR for page 277
EVALUATION OF THE QIO PROGRAM 277
presented for the nation as a whole but are based on summaries of changes
in individual states. The program has not assessed the impact of the entire
7th SOW (including the combination of technical assistance activities with
other requirements related to the protection of beneficiaries and program
integrity). Although some QIOs have mentioned that the performance of
case reviews sometimes brings quality improvement issues to light, these
have not been documented in a method available for evaluation, nor has
CMS assessed the impacts of program spending outside the core contracts,
which is almost one-third of the funds apportioned.
SUMMARY
This chapter has discussed issues related to evaluation of the impact of
quality improvement in the QIO program. The following are some of the
main themes of this chapter, which are reflected in the findings and conclu-
sions presented in Chapter 2:
· CMS evaluates QIO performance on the basis of a number of pro-
vider quality measures and deliverables provided for each task. Objective
data that measure quality improvements are limited to what is in the evalu-
ation scores.
· The method by which QIOs are evaluated is detailed and compli-
cated, does not reflect any program priorities, and thus, neither reflects
patterns of effective technical assistance nor helps the QIOs prioritize how
best to approach the provision of technical assistance. This complicated
method of evaluation is made even more complex in the 8th SOW. The
levels of expectation for the various measures have implications for answer-
ing the question of whether QIOs improve quality; but because of the intri-
cacies of CMS's evaluation formulas, the IOM committee was not able to
determine whether the levels of expected improvement for each setting were
adequate.
· The data used for evaluation may have selection bias. For example,
QIOs may not recruit identified participants randomly; instead, identified
participants may be selected on the basis of their ability to achieve the great-
est improvement, thus yielding higher scores.
· Evaluation formulas do not reflect the differences between the mon-
ies spent on a particular subtask and the weight given to calculation of
identified participant and statewide scores.
· Because identified participants volunteer to work with the QIO,
they may be biased toward improving quality. This is important, as CMS
weights identified participant improvements more heavily than statewide
improvements.
OCR for page 278
278 MEDICARE'S QUALITY IMPROVEMENT ORGANIZATION PROGRAM
On the basis of the QIO contract performance evaluations:
· In general, providers in all settings seem to be improving quality on
most of the performance measures on which they are evaluated.
· The IOM committee was not able to identify correlations with sub-
stantial implications between subtasks, QIO spending per beneficiary, con-
tract round, region, or level of provider satisfaction.
· However, a small but growing amount of evidence indicates that
providers who work intensely with their QIOs achieve higher levels of im-
provement, are more satisfied with QIO assistance, and value QIO assis-
tance more than those who do not.
· The lack of an overall program evaluation limits the IOM com-
mittee's ability to draw conclusions about the overall impact that the QIOs
have had on quality.
REFERENCES
CMS (Centers for Medicare and Medicaid Services). 2002. 7th Statement of Work (SOW).
[Online]. Available: http://www.cms.hhs.gov/qio [accessed April 9, 2005].
CMS. 2004a. QualityNet Dashboard. [Online]. [accessed January 2005].
CMS. 2004b. QualityNet Dashboard. [Online]. [accessed August 26, 2004].
CMS. 2005a. Best Practice Methods Special Study: Report of First Year Scan of QIO Inpatient
Practice. Baltimore, MD: Centers for Medicare and Medicaid Services.
CMS. 2005b. Evaluation of the 7th Scope of Work: Special Advance Summary Report Submit-
ted to the Institute of Medicine. September. Unpublished. Baltimore, MD: Centers for
Medicare and Medicaid Services.
CMS. 2005c. 8th Statement of Work (SOW), Version #080105-1. [Online]. Available: http://
www.cms.hhs.gov/qio [accessed November 4, 2005].
Rollow WC. 2004. Presentation to IOM Committee: Evaluating the HCQIP Program. Un-
published. October 4. Washington, DC: Institute of Medicine.
Westat. 2005. Survey for Provider Satisfaction with Quality Improvement Organizations.
Unpublished. July 1. Rockville, MD: Westat.
Representative terms from entire chapter:
qio program
