Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 183
Appendix C
Quality Assurance for RTI Payment Simulations
BACKGROUND
The committee took on a variety of oversight responsibilities for the RTI analysis. A data
assurance plan was developed by Institute of Medicine (IOM) staff and the committee and
submitted to the Report Review Committee (final version August 24, 2011), and a quality
assurance subcommittee was designated to take the lead on technical aspects of the direction
and oversight. The subcommittee was chaired by Dr. Alan Zaslavsky of Harvard, with members
Dr. Frank Sloan of Duke (committee chair), Dr. Jack Kalbfleisch of University of Michigan, and
Dr. Jane Sisk of the National Center for Health Statistics at the Centers for Disease Control and
Prevention. The subcommittee had several conference calls and also reported back to the full
committee at committee meetings.
An independent contractor, IHS Global Insight, was designated by the subcommittee to
review the final RTI analyses, review all documentation of analyses, and prepare a report on
its findings. The subcommittee and staff held two conference calls to direct the independent
contractor in his role.
The RTI contract was modified to include a task to provide sufficient documentation of the
data sources and methods used in the simulations so they could be replicated by other interested
parties later on, if they so chose. Prior to preparing the report presented below, IHS interviewed
RTI about the documentation and asked several clarifying questions.
The RTI team members presented data findings at every committee meeting, where they
took questions and received direction from the full committee, and where some errors were
caught by RTI or committee members as they discussed the findings. RTI was in regular con-
tact with IOM staff and designated committee members about technical issues as they arose,
and issues were usually resolved by e-mail or conference call. Weekly conference calls were
held between RTI and IOM staff, and call notes were provided by RTI every week to document
progress, including how the documentation was being written for public release, which analy-
ses needed to be rerun because of the need for different categorical breakdowns, and so on.
183
OCR for page 184
184 GEOGRAPHIC ADJUSTMENT IN MEDICARE PAYMENT
RTI instituted an internal triple check process for all data output. In some cases, they have
rerun analyses to make sure the analysis was done correctly. Replication of the computer runs
was not possible due to the number of iterations and refinements requested by committee mem-
bers after the preliminary analyses started. The time and expense would have been prohibitive.
When the report is released to the public, all of the RTI data simulations in the form of Excel
tables (Appendix A-2) will be released on the study website (http://www.iom.edu/GeoAdjust
PaymentSimulations) in a downloadable format so they are available to anyone who chooses
to review the data in detail.
In sum, the payment simulations were highly technical and complex, and multiple cross-
checks were followed by RTI in preparing and executing programming code and reviewing
output. To the extent feasible, qualified committee members provided direction and oversight
for the simulations, but they were reluctant to review detailed documentation themselves and
chose an independent contractor to perform that function on their behalf.
REPORT ON QUALITY ASSURANCE OF RTI PAYMENT SIMULATIONS
Prepared by Tim Dall, IHS Global Insight
Since its establishment in 1970, the IOM has played a prominent role in helping to inform
and shape the national debate on health care policy and delivery. IOM's current commission
from the Department of Health and Human Services is to conduct analysis of data sources and
methods available to calculate the geographic adjustments used by the Centers for Medicare
& Medicaid Services (CMS) for fee-for-service Medicare payments to hospitals and providers.
IOM's Phase I report provided recommendations for improving the accuracy of the geo-
graphic adjustment factors. IOM's Phase II report investigates the extent to which recommen-
dations in the Phase I report, if implemented, would affect access to health care services. With
Medicare spending approaching $525 billion in 2010, ensuring the accuracy of the geographic
adjustments has substantial implications for hospitals and providers of medical services and the
people they serve.
With such prominence in shaping policy, and given the importance of Medicare reimburse-
ment in care delivery, comes a responsibility to provide the highest quality of work possible
given data and resource constraints. IOM engaged RTI International in this 2-year study to
conduct analyses and simulations related to the accuracy of the data and methods used to
develop CMS's geographic adjustments. IOM engaged IHS Global Insight to provide an inde-
pendent quality assurance review of the work completed by RTI and presented in the second
year report titled Geographic Adjustment in Medicare Payment: Phase II: Implications for Access,
Quality, and Efficiency.
The goal of the independent analysis reported in this appendix is to review documentation
provided by RTI detailing the data and methods used to develop their payment simulations
and results, to verify the integrity of the underlying data, and to assess the transparency and
completeness of work presented. This independent review is in addition to review activities
conducted by members of IOM's Committee on Geographic Adjustment Factors in Medicare
Payment, a group of prominent subject matter experts who have volunteered their time to
participate in this study.
The committee determined early in its deliberations that replication of the payment simula-
tions would not be practical, given the limited resources available. However, the committee did
OCR for page 185
APPENDIX C 185
develop a data assurance plan that involved several steps, including forming a subcommittee
that directed and reviewed RTI's intermediate and final analyses, discussing interim analyses
at full committee meetings, and engaging IHS Global Insight as an independent contractor to
review all data sources and analyses when the simulations were completed.
Because of the prodigious amount of work completed by RTI over the past 2 years leading
up to the final results presented, it is not feasible to verify 100 percent accuracy in study find-
ings. However, IOM requires that contractors performing quantitative analysis have a quality
assurance plan and implement quality control procedures. RTI appears to have rigorous internal
processes to help ensure quality, and all indications are that RTI has complied with its internal
quality assurance protocols and implemented its work in accordance with industry best prac-
tices. Dobson et al., in their 2011 assessment of RTI International's Year 1 report to IOM. provide
an overview of RTI International's programming quality assurance (QA) protocols.1 RTI's internal
QA process is documented in the firm's Quality Assurance Management Handbook. Exhibit C-1
summarizes key quality assurance protocols relevant to the IOM study and discusses some of
the quality control activities conducted by RTI and IHS.
IHS reviewed copies of the MS Excel files that RTI prepared for committee members to
share the payment simulation results. During calls with committee members, RTI discussed the
results of the simulations and described the key factors contributing to differences in the results
between the various simulation scenarios. A review of the Excel files identified quality control
mechanisms built into the spreadsheets to help ensure accuracy.
In conclusion, to the best of IHS's knowledge based on a review of work products and com-
puter programming code produced by RTI, and through discussions with the RTI Project Director
and team members, the payment simulations were completed in accordance with industry best
practices for quality control. Where possible, information from the internal documentation was
compared to findings in the report to verify accuracy.
Key conclusions are
1.The technical write-up in the Phase I and II reports, including Appendix A of the Phase II
report, provide sufficient detail to understand the data and methods used to construct the
payment simulations. Combined with the project team's internal documentation, there
is sufficient information to replicate the results. However, key data used in the analysis
are not publically available. Specifically, special data runs by the Census Bureau (with
the American Community Survey) and by the Bureau of Labor Statistics were prepared
by bureau staff because they used data in small markets that were suppressed from the
public-use files due to privacy rules.
2.The prodigious amount of information produced by RTI did not allow for checking of the
data and replicating the analysis to ensure that it is free of error. However, we verified
that RTI has a rigorous quality assurance plan, and all indications are that RTI complied
with its quality assurance protocols. Discussions with the RTI project director and team
suggest that the quantitative work was completed in accordance with industry best
practices. Simulation results were presented to committee members, and study findings
appear consistent with expectations.
3.The analytical approach helps address the research question of how changes to the
1 Dobson, A., J. E. DaVanzo, A. El-Gamil, G. Berger, and J. Freeman. An examination of the data, materials, and assump-
tions used in the Institute of Medicine report Geographic Adjustment in Medicare Payment: Phase 1: Improving Accuracy.
Dobson/DaVanzo final report to the Institute of Medicine. June 2011.
OCR for page 186
186 GEOGRAPHIC ADJUSTMENT IN MEDICARE PAYMENT
EXHIBIT C-1 Quality Assurance Protocols and Quality Control Activities
RTI Quality Assurance Protocol Quality Control Activities
Analysts and programmers IHS reviewed the programming code and documentation prepared
provide sufficient documentation by RTI. The contents of Appendix A combined with the internal
so that their work can be audited documentation appears sufficient that one can understand the data and
if needed. methods used to complete the analyses and, if needed, replicate RTI's
payment simulations.
Analysts and programmers check While no benchmarks exist for which to compare the payment
and compare outputs to available simulations, substantial efforts were made by RTI, the IOM committee
benchmarks. members, and IHS to compare the payment simulation results to CMS's
current geographic adjustments factors and to understand the drivers of
differences between the various geographic indices.
Report tables identify the source The IOM report and Appendix A identify those tables where the
computer run so that table table content was generated from an RTI simulation. RTI's internal
contents can be verified against documentation identifies which computer program produced each
the program(s) that generated table. Computer logs of the program code were reviewed by IHS.
them.
Key study findings are sufficiently This QA protocol is similar to a requirement of peer-reviewed journal
documented to allow for articles--that sufficient information is provided to allow for replication
replication by other researchers. of study findings. While RTI's documentation does allow for researchers
to understand the data and methods, some key data sources are
not publically available. Consequently, replication of study findings
would require access to special data tabulations prepared by various
government agencies and purchase of nonpublic data.
All data tables shared with the Discussions with the RTI project team indicates that the RTI project co-
IOM committee or included in directors were intimately involved with all aspects of this analysis--from
the IOM should be closely reviewing programmer code to reviewing all data tables provided to
reviewed by the project director. the IOM committee and/or included in the IOM report.
Verify the correctness of the In line with industry best practices, RTI produced frequency
original and constructed variables distributions and summary statistics (e.g., mean, minimum, maximum
before beginning the analysis. values) for both the original data and the variables created. This helped
identify outliers or data anomalies that were then checked for accuracy.
NOTES: CMS = Centers for Medicare & Medicaid Services; IOM = Institute of Medicine; QA = quality assurance.
geographic reimbursement might impact access to services. However, the approach
does not fully address the question. The payment simulations show the impact on
redistribution of Medicare dollars by geographic area and shortage designation, and by
different characteristics of facilities providing care to Medicare patients. The report notes
the paucity of extant research to describe how changes in Medicare payments might
affect the propensity of individual providers and facilities to serve Medicare patients.
While one would expect that lower Medicare payments to a particular geographic area
will have a detrimental effect on provider supply in that area (thus reducing patient
access to care), additional research is required to fully understand the implications on
patient access and quality of care. The data and methods used to quantify the financial
implications of changes to Medicare's geographic adjustment appear appropriate.
