Appendix C
Quality Assurance for RTI Payment Simulations
BACKGROUND
The committee took on a variety of oversight responsibilities for the RTI analysis. A data assurance plan was developed by Institute of Medicine (IOM) staff and the committee and submitted to the Report Review Committee (final version August 24, 2011), and a quality assurance subcommittee was designated to take the lead on technical aspects of the direction and oversight. The subcommittee was chaired by Dr. Alan Zaslavsky of Harvard, with members Dr. Frank Sloan of Duke (committee chair), Dr. Jack Kalbfleisch of University of Michigan, and Dr. Jane Sisk of the National Center for Health Statistics at the Centers for Disease Control and Prevention. The subcommittee had several conference calls and also reported back to the full committee at committee meetings.
An independent contractor, IHS Global Insight, was designated by the subcommittee to review the final RTI analyses, review all documentation of analyses, and prepare a report on its findings. The subcommittee and staff held two conference calls to direct the independent contractor in his role.
The RTI contract was modified to include a task to provide sufficient documentation of the data sources and methods used in the simulations so they could be replicated by other interested parties later on, if they so chose. Prior to preparing the report presented below, IHS interviewed RTI about the documentation and asked several clarifying questions.
The RTI team members presented data findings at every committee meeting, where they took questions and received direction from the full committee, and where some errors were caught by RTI or committee members as they discussed the findings. RTI was in regular contact with IOM staff and designated committee members about technical issues as they arose, and issues were usually resolved by e-mail or conference call. Weekly conference calls were held between RTI and IOM staff, and call notes were provided by RTI every week to document progress, including how the documentation was being written for public release, which analyses needed to be rerun because of the need for different categorical breakdowns, and so on.
RTI instituted an internal triple check process for all data output. In some cases, they have rerun analyses to make sure the analysis was done correctly. Replication of the computer runs was not possible due to the number of iterations and refinements requested by committee members after the preliminary analyses started. The time and expense would have been prohibitive.
When the report is released to the public, all of the RTI data simulations in the form of Excel tables (Appendix A-2) will be released on the study website (http://www.iom.edu/GeoAdjustPaymentSimulations) in a downloadable format so they are available to anyone who chooses to review the data in detail.
In sum, the payment simulations were highly technical and complex, and multiple crosschecks were followed by RTI in preparing and executing programming code and reviewing output. To the extent feasible, qualified committee members provided direction and oversight for the simulations, but they were reluctant to review detailed documentation themselves and chose an independent contractor to perform that function on their behalf.
REPORT ON QUALITY ASSURANCE OF RTI PAYMENT SIMULATIONS
Prepared by Tim Dali, IHS Global Insight
Since its establishment in 1970, the IOM has played a prominent role in helping to inform and shape the national debate on health care policy and delivery. IOM’s current commission from the Department of Health and Human Services is to conduct analysis of data sources and methods available to calculate the geographic adjustments used by the Centers for Medicare & Medicaid Services (CMS) for fee-for-service Medicare payments to hospitals and providers.
IOM’s Phase I report provided recommendations for improving the accuracy of the geographic adjustment factors. IOM’s Phase II report investigates the extent to which recommendations in the Phase I report, if implemented, would affect access to health care services. With Medicare spending approaching $525 billion in 2010, ensuring the accuracy of the geographic adjustments has substantial implications for hospitals and providers of medical services and the people they serve.
With such prominence in shaping policy, and given the importance of Medicare reimbursement in care delivery, comes a responsibility to provide the highest quality of work possible given data and resource constraints. IOM engaged RTI International in this 2-year study to conduct analyses and simulations related to the accuracy of the data and methods used to develop CMS’s geographic adjustments. IOM engaged IHS Global Insight to provide an independent quality assurance review of the work completed by RTI and presented in the second year report titled Geographic Adjustment in Medicare Payment: Phase II: Implications for Access, Quality, and Efficiency.
The goal of the independent analysis reported in this appendix is to review documentation provided by RTI detailing the data and methods used to develop their payment simulations and results, to verify the integrity of the underlying data, and to assess the transparency and completeness of work presented. This independent review is in addition to review activities conducted by members of IOM’s Committee on Geographic Adjustment Factors in Medicare Payment, a group of prominent subject matter experts who have volunteered their time to participate in this study.
The committee determined early in its deliberations that replication of the payment simulations would not be practical, given the limited resources available. However, the committee did
develop a data assurance plan that involved several steps, including forming a subcommittee that directed and reviewed RTI’s intermediate and final analyses, discussing interim analyses at full committee meetings, and engaging IHS Global Insight as an independent contractor to review all data sources and analyses when the simulations were completed.
Because of the prodigious amount of work completed by RTI over the past 2 years leading up to the final results presented, it is not feasible to verify 100 percent accuracy in study findings. However, IOM requires that contractors performing quantitative analysis have a quality assurance plan and implement quality control procedures. RTI appears to have rigorous internal processes to help ensure quality, and all indications are that RTI has complied with its internal quality assurance protocols and implemented its work in accordance with industry best practices. Dobson et al., in their 2011 assessment of RTI International’s Year 1 report to IOM. provide an overview of RTI International’s programming quality assurance (QA) protocols.1 RTI’s internal QA process is documented in the firm’s Quality Assurance Management Handbook. Exhibit C-1 summarizes key quality assurance protocols relevant to the IOM study and discusses some of the quality control activities conducted by RTI and IHS.
IHS reviewed copies of the MS Excel files that RTI prepared for committee members to share the payment simulation results. During calls with committee members, RTI discussed the results of the simulations and described the key factors contributing to differences in the results between the various simulation scenarios. A review of the Excel files identified quality control mechanisms built into the spreadsheets to help ensure accuracy.
In conclusion, to the best of IHS’s knowledge based on a review of work products and computer programming code produced by RTI, and through discussions with the RTI Project Director and team members, the payment simulations were completed in accordance with industry best practices for quality control. Where possible, information from the internal documentation was compared to findings in the report to verify accuracy.
Key conclusions are
1. The technical write-up in the Phase I and II reports, including Appendix A of the Phase II report, provide sufficient detail to understand the data and methods used to construct the payment simulations. Combined with the project team’s internal documentation, there is sufficient information to replicate the results. However, key data used in the analysis are not publically available. Specifically, special data runs by the Census Bureau (with the American Community Survey) and by the Bureau of Labor Statistics were prepared by bureau staff because they used data in small markets that were suppressed from the public-use files due to privacy rules.
2. The prodigious amount of information produced by RTI did not allow for checking of the data and replicating the analysis to ensure that it is free of error. However, we verified that RTI has a rigorous quality assurance plan, and all indications are that RTI complied with its quality assurance protocols. Discussions with the RTI project director and team suggest that the quantitative work was completed in accordance with industry best practices. Simulation results were presented to committee members, and study findings appear consistent with expectations.
3. The analytical approach helps address the research question of how changes to the
______________
1 Dobson, A., J. E. DaVanzo, A. El-Gamil, G. Berger, and J. Freeman. An examination of the data, materials, and assumptions used in the Institute of Medicine report Geographic Adjustment in Medicare Payment: Phase 1: Improving Accuracy. Dobson/DaVanzo final report to the Institute of Medicine. June 2011.
EXHIBIT C-1 Quality Assurance Protocols and Quality Control Activities
|
RTI Quality Assurance Protocol |
Quality Control Activities |
|---|---|
|
Analysts and programmers provide sufficient documentation so that their work can be audited if needed. |
IHS reviewed the programming code and documentation prepared by RTI. The contents of Appendix A combined with the internal documentation appears sufficient that one can understand the data and methods used to complete the analyses and, if needed, replicate RTI’s payment simulations. |
|
Analysts and programmers check and compare outputs to available benchmarks. |
While no benchmarks exist for which to compare the payment simulations, substantial efforts were made by RTI, the IOM committee members, and IHS to compare the payment simulation results to CMS’s current geographic adjustments factors and to understand the drivers of differences between the various geographic indices. |
|
Report tables identify the source computer run so that table contents can be verified against the program(s) that generated them. |
The IOM report and Appendix A identify those tables where the table content was generated from an RTI simulation. RTI’s internal documentation identifies which computer program produced each table. Computer logs of the program code were reviewed by IHS. |
|
Key study findings are sufficiently documented to allow for replication by other researchers. |
This QA protocol is similar to a requirement of peer-reviewed journal articles—that sufficient information is provided to allow for replication of study findings. While RTI’s documentation does allow for researchers to understand the data and methods, some key data sources are not publically available. Consequently, replication of study findings would require access to special data tabulations prepared by various government agencies and purchase of nonpublic data. |
|
All data tables shared with the IOM committee or included in the IOM should be closely reviewed by the project director. |
Discussions with the RTI project team indicates that the RTI project codirectors were intimately involved with all aspects of this analysis—from reviewing programmer code to reviewing all data tables provided to the IOM committee and/or included in the IOM report. |
|
Verify the correctness of the original and constructed variables before beginning the analysis. |
In line with industry best practices, RTI produced frequency distributions and summary statistics (e.g., mean, minimum, maximum values) for both the original data and the variables created. This helped identify outliers or data anomalies that were then checked for accuracy. |
NOTES: CMS = Centers for Medicare & Medicaid Services; IOM = Institute of Medicine; QA = quality assurance.
geographic reimbursement might impact access to services. However, the approach does not fully address the question. The payment simulations show the impact on redistribution of Medicare dollars by geographic area and shortage designation, and by different characteristics of facilities providing care to Medicare patients. The report notes the paucity of extant research to describe how changes in Medicare payments might affect the propensity of individual providers and facilities to serve Medicare patients. While one would expect that lower Medicare payments to a particular geographic area will have a detrimental effect on provider supply in that area (thus reducing patient access to care), additional research is required to fully understand the implications on patient access and quality of care. The data and methods used to quantify the financial implications of changes to Medicare’s geographic adjustment appear appropriate.
