The previous two sessions focused on the research and issues related to the concepts of thresholds and resources in a measure of medical care economic risk (MCER). This session addresses issues related to data needs associated with the implementation of such a measure: what data are available now, what relevant data will become available under health care reform, data quality and timeliness concerns, and whether the measure of MCER could be released at the same time as the Supplemental Poverty Measure (SPM).
MEASURING MCER: AN ASSESSMENT OF DATA SOURCES
John Czajka (Mathematica Policy Research) presented an overview of the background paper he prepared for the workshop (see Part III of this volume), explaining that he restricted himself to using federal data sets, because it is the federal agencies that are going to produce the estimates. Although the design of a measure of MCER need not be constrained by currently available data, any such measure produced in the next few years will need to be based almost exclusively on data collected currently. It may be possible to add a modest number of new items to an existing survey, but funding to support significant revisions or additions is not available.
The sponsor of the study panel, the Office of the Assistant Secretary for Planning and Evaluation (ASPE), has asked for the development and implementation of a new measure of medical risk as a companion measure to the new supplemental income poverty measure. With regard to data, the Medical Expenditure Panel Survey (MEPS), which became continuous in
1996, will presumably be the bedrock source for estimating the parameters of a medical care risk index. However, that estimation should use variables on risk factors and insurance coverage that are available in the Current Population Survey Annual Social and Economic Supplement (CPS ASEC), so that a family’s poverty status can be compared with its medical care risk. Czajka said that he thought that any new measure, if it is intended to be a companion to the SPM, should be estimated from the same data as the SPM for at least a few years, so that users can see how the two measures work together. After that one can think about whether there is some good reason to separate them.
Factors Affecting Data Requirements
Alternative design choices have a big impact on the data requirements for a medical care economic risk measure. There are two main choices. The first is between a retrospective and a prospective measurement of risk. The first workshop session focused on prospective measures. Retrospective measures got attention in the second session. The second choice is whether resources should include assets or be limited to only income.
It is also important to distinguish between data used to develop a measure and data used to produce the measure that is disseminated to the public. Development of a measure requires data that, ideally, will support alternative measures and enable evaluation. Longitudinal data would be valuable for evaluation and validation—especially with a prospective measure. Production of a measure requires data to support only one, not multiple measures; however, timeliness, representativeness, and statistical precision become more important.
Measures of Resources
Regarding measures of resources, the CPS ASEC is the official source of estimates of income and poverty for the United States. It is also being used to construct the SPM, as it has been used to construct a number of experimental measures over the years. It includes the official measure of money income, which is what is used to estimate poverty. And it also does or will provide the measure of disposable income that goes into the SPM.
It is notable that some of the components of the SPM, and even in the CPS, have to be imputed. Not all items are collected, and taxes are the big item that has to be modeled and applied through the model or through imputation. Information on taxes is not really collected in any of these surveys.
The CPS added the measure of medical out-of-pocket expenditures and a few other components in 2010, to help support the new SPM. The
assessment of the out-of-pocket medical expenditures data was surprisingly good, considering that this survey has had trouble measuring health insurance coverage.
With regard to resources, the CPS does not include everything that one may think of as income. It does not include capital gains, which are picked up in taxes and which MEPS picks up. It does not include a lot of what people withdraw from their retirement accounts, which again is something that is reported on the tax return and taxed, and MEPS picks up some of those items. These are differences that can be important, especially for the elderly, who have a lot of these types of resources.
A critical consideration with regard to the CPS ASEC is that it collects no asset data of any kind and never has. To deal with that deficiency, Czajka thought that, rather than trying to impute assets, it would be better to add a small number of items to the survey. But quality of data cannot be ensured without careful testing. The difficulty with imputation for financial assets is that, in including assets, one is really interested in what people have that is not reflected in their income. But the likelihood is that the imputations will be driven heavily by income. And that does not get the orthogonal piece that is needed. At the same time, it is very challenging to collect asset data, and the idea that one could write a question and get the answer wanted is optimistic. That is a particular problem with the CPS.
MEPS collects sources of income that correspond reasonably closely to the CPS concept of money income. MEPS income questions follow the federal tax form and include capital gains and state tax refunds, which are not counted in CPS money income. Respondents who refer to their tax returns would omit portions of earnings and possibly Social Security benefits excluded from taxation. MEPS collects fewer of the expenses that differentiate money income from disposable income. Like the CPS, however, MEPS does not capture taxes paid (or earned income tax credit received). Unlike the CPS, MEPS collects data on assets. Assets are divided into six broad categories, and amounts are collected for all six types.
One potential concern about MEPS is that, in following the tax return, if people really do pull out their returns, they would be reporting items that do not correspond conceptually to what is picked up in the CPS, which looks at gross income.
Measures of MCER
The CPS ASEC contains a fairly small set of items relating to measuring medical care risk. It added medical out-of-pocket expenditures in 2010. Data compare favorably to MEPS and the Survey of Income and Program Participation (SIPP), despite the more detailed measurement in these other surveys. The CPS ASEC collects sources of health insurance coverage in the
“past year” but no additional information on what expenditures are covered. It also collects basic work and activity limitations and general health status—potentially useful in defining risk groups and matching to MEPS.
MEPS, in contrast, collects very extensive data on health conditions, health status, the use of medical services, charges and payments, access to care, and health insurance coverage over time. What it lacks is information on what is actually covered by each sample member’s health insurance. Nevertheless, MEPS can support both prospective and retrospective measures of medical care risk.
Limited information on data quality for these various components suggests some areas in which improvements would be desirable.
With respect to income, CPS is the official measure and comparisons show that it does collect more total income than the other major surveys. Despite its overall strength, CPS ASEC income data have notable weaknesses. Reporting of all components of retirement income other than Social Security is well below SIPP, which falls short of the CPS ASEC on most other income sources. This suggests that a data area on which one would rely heavily in evaluating the MCER of the elderly is one in which the CPS would need to be improved.
Supplemental Nutrition Assistance Program (SNAP, formerly the Food Stamp Program) benefits are currently received by 15 percent of the population. The latest estimates comparing what CPS captures with what is actually paid out suggest that the CPS is missing nearly half of total SNAP benefits. Because SNAP benefits are part of what gets added to disposable income to create the SPM, the new measure will not reflect the status of low-income persons as accurately as it would if SNAP benefits were measured more effectively.
Nonresponse to income questions is high; 30 percent of total income is imputed to respondents. A note of caution with the imputation methods is that the medical care risk index is a new measure, and the CPS imputation procedures may not take account of a lot of the components that may be part of this new measure and that may introduce certain kinds of error into the resulting index. If imputation procedures do not account for covariates of medical care risk, the index is weakened. Finally, limitations of CPS ASEC health insurance measures are well known.
Data collected in MEPS on measures of medical service use and medical out-of-pocket expenditures are unique in their detail. MEPS data set the standard, but there is little out there to compare to them. Because of the MEPS panel design, attrition may be the principal concern. The first MEPS interview is actually the second interview with the MEPS sample, because
the sample is drawn from National Health Interview Survey (NHIS) respondents. Are persons with high MCER overrepresented among attriters?
One of the strengths of MEPS is that it does have overlapping panels, so that there is consistent representativeness over time. This is a very important feature for a measure that will be used to track change over time.
Czajka noted that, after tracking the estimates from the NHIS and the CPS ASEC for most of the past decade, MEPS uninsured rates for adults and children rose sharply in 2007 and 2008, when the other major surveys showed stable or declining rates.
Czajka then discussed briefly three other surveys.
Survey of Income and Program Participation
The Panel on Poverty and Family Assistance, authors of the 1995 National Research Council (NRC) report Measuring Poverty: A New Approach, viewed SIPP as the survey of choice for a new poverty measure. It had the advantage that it was designed expressly to support policy analysis; it collected more detailed income data than any other federal survey, and the quality of these data was almost uniformly high. SIPP’s design, with collection of substantial core data in every wave and supplemental topical modules with varying content, was ideally suited to a new poverty measure that would require new data but not in every wave.
A decade later, the view was different. Some of the reasons were that the 1996 redesign replaced overlapping panels, critical to consistent cross-sectional representativeness; evidence of deterioration in income and asset data emerged; and timeliness issues and repeated budget/sample cuts detracted from the stability needed to support a key national indicator.
SIPP was terminated in 2006 but then restored in response to objections from users. A new panel started in 2008 will continue until replaced by a reengineered SIPP to be fielded in early 2014. Under the reengineered SIPP, annual interviews will replace the 4-month interviews; event history calendar methods will be used to collect monthly data with 12-month recall; most of SIPP core content will be retained; and key items from annual topical modules—such as assets and medical and work-related expenditures—will be added to annual interviews.
There are other issues, however, in using SIPP for development or initial production of the medical care risk index. To monitor implementation of health care reform, the index must be in place before the first new SIPP data will be available. Initial, small sample tests of the new design are encouraging, but it is not possible to fully assess the survey yet. SIPP’s
funding history and current budget climate raise concerns about sustained funding.
However, the current SIPP, with panels longer than the MEPS panel, could play a role in evaluating a prospective medical care risk index. Does the subsequent experience of subpopulations match their estimated risk? Where and why do deviations occur?
American Community Survey
The American Community Survey (ACS) has also been mentioned as a potential source of data for an MCER measure. The ACS is attractive because of its large sample size; 2 million households interviewed each year would offer unmatched geographic detail. It captures similar content to the CPS ASEC but is more limited in depth. The areas in which ACS data are richer than the CPS ASEC tend not to be relevant to MCER. Another problem is that the ACS questionnaire tends to be fixed for long periods of time because the survey is designed to allow estimates to be cumulated over periods of 3 to 5 years, to produce very small geographic detail. Also, much of the content of the ACS is mandated by law. The bottom line: ACS does not provide a viable option for developing or producing a medical care risk index.
The NHIS provides the sampling frame for MEPS. It is larger, and most of the content is released on a more timely basis. It collects detailed information on health status, which could enrich a prospective measure of risk. But on most of the other potential components of the MCER measure, the NHIS data are more limited than MEPS or nonexistent. In addition, because NHIS provides the sampling frame for MEPS, NHIS data can be linked to MEPS sample records; thus NHIS would add no new content. NHIS by itself is therefore not an option for developing or producing a medical care risk index.
In conclusion, Czajka reiterated that questions about data source are reduced to what is collected in two surveys: the CPS and MEPS. MEPS collects essentially all data elements needed to construct alternative versions of the medical care risk index, whereas the CPS is missing critical variables for certain variants on these measures. Yet the CPS ASEC will be used to produce the new SPM to which MCER is intended to be a companion measure. Having both measures in the same survey would allow researchers
to compare and contrast how families and individuals are classified by the two measures. Such comparisons may be helpful in establishing the value added by a measure of MCER.
Czajka noted other advantages of the CPS ASEC. A CPS-based index could be released concurrently or shortly after the SPM or 10-11 months after the end of the survey reference period, which is the prior calendar year; MEPS would require an additional year. The CPS ASEC sample size is five times the largest recent MEPS samples. In addition to its size, the CPS ASEC sample combines independent, representative samples of the 50 states and DC. State estimates, although lacking in precision for individual years, could be important for state comparisons in monitoring implementation of the Affordable Care Act.
However, a prospective index would depend on data collected in MEPS; these data would have to lag a year, or release of the index would have to be delayed a year.
Finally, whatever survey is chosen, it is important to reassess both data and methodology within a few years of implementation.
Participants expressed their views on the various issues flowing from the presentation. Kenneth Finegold suggested that the study panel should consider using the Transfer Income Model, version 3 (TRIM3) as part of the discussion about developing the medical care risk index. TRIM3 is ASPE’s model based on the CPS, but it has modules that do a lot of different things, including aligning Medicaid participation to match administrative totals in response to underreporting.
Given the limitations of the data sources that Czajka suggested, he said, one way to go would be to match TRIM3 to MEPS. Finegold was not sure that has been done, but TRIM3 was matched to NHIS for a number of years. Matching with MEPS would help to get at assets and medical conditions, information that is not in the CPS. The TRIM3 model has been used, he thought, under contract to the NRC, in development of the SPM over the years.
In response to a question about the currency of TRIM3, Laura Wheaton (Urban Institute), who is on the TRIM3 microsimulation project, responded. The work on this project is conducted for ASPE. When the analysts get each year’s CPS data, over the course probably of the following year, they do the corrections for underreporting that Finegold was talking about. Depending on different project priorities, that schedule sometimes slips. So it is certainly not as timely as the CPS, she said. Various things often delay the baselines. For example, the project incorporates imputations of immigrant status to identify undocumented aliens, an important issue
for modeling eligibility for these various programs that adds some delay. If TRIM3 were to be used, it would certainly take more time than the release of the CPS. It would be at least 4 months after the release of the CPS before anything would be available.
Barbara Wolfe had two questions: first, is there any way of knowing more about the underreporting of income and assets by quintile? The relevance of the underreporting of SNAP clearly applies to the low-income population, but her impression had been that some of the assets are really underreported at the highest levels. She asked whether, for this particular population group, if it is possible to think more narrowly about the underreporting, particularly in the CPS, but in both data sets.
Second, is there any way that MEPS could in fact be available earlier, so that there would not be a year’s delay for outside people to work with it? Maybe there is some way that in fact that could be speeded up so it would not be that long a delay.
Steven Cohen explained the method by which MEPS produces its expenditure estimates. After the household interview, the team conducts a Medical Provider Survey to get detail on all sources of payment. For this type of analysis, it actually produces an annual file on utilization and insurance coverage about 5 to 6 months after the period from the CPS turnaround. If the Agency for Healthcare Research and Quality (AHRQ) was just getting the out-of-pocket information to add to the file rather than depending on the Medical Provider Survey, which includes all the different aspects of expenditures, and it is also getting income data, there could be a way of accelerating that sort of a component of MEPS that would then be updated with purer estimates from the medical provider survey.
If one is depending just on the out-of-pocket information, perhaps the premium information that is paid out-of-pocket and the income information that is available before the Medical Provider Survey is conducted would be sufficient. The out-of-pocket information does get corrected, but it is not much of a departure. So there are some options in moving forward with these potential data resource needs for AHRQ to consider.
Czajka responded to the question about asset and income data. High-end assets are a big problem for all surveys. But the Survey of Consumer Finances (SCF) has a sample of tax returns, high income, and the ability to poststratify those people because it has data on who had what income. Comparisons between SIPP and the SCF show, however, that below the really high end there are big differences by type of asset. People do a very good job consistently across surveys in reporting the value of their home and the debt on their home, so there is a good estimate of home equity. That may be because people get a statement every month that tells them what they owe on their house. But that is not true of most of the other types of assets. Information on business assets is really bad in SIPP. It is not some-
thing that is evident from looking at the question. There are differences across different components, so that is why he hesitates to say one can just pop in some questions and get the results that one wants.
With respect to income, it is not really possible to look at comparisons with administrative records by quintile in quite the same way because the focus is family income, and one cannot put together those pieces. He reported working a few years ago on comparing several federal surveys and comparing by quintile across the surveys. For example, although the CPS did much better on earnings than the other surveys for the most part, if one looked at the low end, SIPP was getting more earnings. Also, the ACS was getting more earnings. With SIPP one can expect that a lot of the problem at the low end is that people are changing jobs. They may have a number of jobs, and if asked about this at the end of a year, they are likely to forget. With SIPP they are asked every 4 months about this employment. Although a survey may be the best income survey, it can still have some serious deficiencies.
Peter Cunningham commented that he was pleasantly surprised about the CPS out-of-pocket estimates. But not having seen a comparison, he was still somewhat skeptical. He wanted to know if the CPS asked about both out-of-pocket expenditures on premiums as well as on services, and how much detail there is.
Kyle Caswell responded that on the 2010 survey there are just two summary questions, and basically the same questions that they took from SIPP. One question asks people retrospectively how much did they spend in the last 12 months on health insurance premiums, at the individual level. The second question asks about nonpremium medical out-of-pocket expenses. It is very simple. But the comparison across the most available data at the time—MEPS, SIPP, and the CPS—did look good. The main issue with the CPS data is that it had a lot of people who reported a zero amount for either premium or nonpremium out-of-pocket expenditures compared with the other sources. Other components of the data, or different percentiles, looked surprisingly good, considering how much less effort was made than with MEPS.
Cunningham asked if they had looked separately at people with high expenditures, the tails of the distribution, as well, because he thought that when people are reporting from recall without any referring back to records, the underreporting tends to be severe. Caswell responded that different points of the distribution were not wildly different. That is the main story, he said, especially compared with the differences in data collection across the surveys.
Caswell commented that, given the conceptual model that was presented and thinking about risk in a prospective way, he thought that modeling health status and particular health conditions would be very important. She asked what the study panel would think about using a data source that
does not collect that information. For example, to use MEPS and then glue that health information onto the CPS, what is involved?
Czajka responded that this general approach is what is used for the SPM and the experimental measures, and recently only the CPS has started collecting these other components. The approach has relied on these kinds of methods, imputing for the expenditures largely from MEPS and also maybe SIPP. It is done in microsimulation models and clearly has risks. One is trying to match, and making assumptions that the variables that one is not matching on are lining up with the variables that one is matching. It is not perfect. He said he thought that, down the road, they would probably have to think about expanding the CPS, if that is the source, to add other measures, and hope for the same success that the Bureau has had with the medical out-of-pocket expenditures.
Pamela Short asked if the CPS has a scale for excellent, very good, good, fair, and poor health. Bringing that into the statistical matching would generate a lot more confidence than without that variable.
Sarah Meier commented that, with respect to the conceptual model, the risk adjustment models explain only a relatively small amount of the variation in health expenditures. So working with a very complex model that includes ICD-9 diagnosis codes and all sorts of other information, is actually not going to be a big improvement from working with just a scale of poor, good, and very good health status. Stepping back from the idealized version of what one might want to do if there were no data limitations, the types of variables in terms of health status that are available in the CPS would be a reasonable base for the health characteristic cells of interest here.
The bigger issue would be the expenditure information, how strong and solid that is in modeling of the actual expenditure distributions. So one could think of modeling those in MEPS and then attaching that to the CPS. The big issue with implementing the full type of model is the lack of insurance characteristics in the CPS.
Banthin cautioned that she would be concerned about matching MEPS to the CPS. The out-of-pocket spending measure is just out-of-pocket, it is not total, she said. And there are those extra zeros. It involves the entire distribution.
And just like the orthogonality of assets to income, it is important to preserve the orthogonality of expenditures, even the distribution of medical expenditures to income or to out-of-pocket. It is a different dimension.
Also, a scale for excellent, very good, fair, and poor health is a great predictor, but additional data are needed. Although she has built many a simulation model in which expenditures are matched to others, that was for simulating changes in policy. This is for the construction of a medical care risk index. So it does concern her that one of the key variables would be imputed with only a limited set of matching covariates.
Jennifer Madans (National Center for Health Statistics) stated that a scale of excellent, good, fair, and poor health is good at the extremes, but it is not very good in the middle. To know if someone is going to go into long-term care or die or will have a lot of expenditures, it is probably good at one end, and excellent is really good. But it is really bad about differentiating that big group of people in the middle. She has never been a big proponent of counting conditions either, because of the big variability in what that means. So there needs to be something else added. She said that some of the disability work is going in that direction, to really get a better composite measure of health that will be a better predictor of the use of resources. She did not think any of the surveys have that yet, but they are moving toward it, and some composite of that is going to be needed.