The Massachusetts Experience: Using Survey Data to Evaluate State Health Care Reform
Sharon K. Long
The Urban Institute
In April 2006, Massachusetts passed landmark legislation that sought to move the state to near-universal health insurance coverage and to improve access to high-quality health care. Following the passage of the legislation, there was substantial concern among key stakeholders in the state about the ability to evaluate the impacts of health care reform with existing state and national data sources. This led the Blue Cross/Blue Shield of Massachusetts Foundation, along with the Commonwealth Fund and the Robert Wood Johnson Foundation, to support a new survey effort dedicated to the evaluation of health care reform—the Massachusetts Health Reform Survey (MHRS). In addition, concerns about the reliability of estimates from an existing state survey of health insurance coverage—the Massachusetts Health Insurance Survey (MHIS)—led to a redesign of that survey to better support the tracking of insurance coverage over time. The MHIS is sponsored by the Massachusetts Division of Health Care Finance and Policy. The Urban Institute, along with Social Science Research Solutions, conducts the MHRS and the MHIS. This paper addresses the lessons learned from the two Massachusetts surveys that may apply to efforts to monitor health insurance coverage of children in the states.
WHY A SEPARATE STATE SURVEY?
Prior to 2006, Massachusetts, like a number of other states, sponsored its own household survey in order to obtain (1) a larger sample for state
estimates than is available in national surveys, including larger samples for important subpopulations in the state and for substate geographic areas; (2) information on insurance coverage that addressed the full scope of coverage options available in the state; (3) information on other key issues of relevance to the states (e.g., access to health care and the affordability of health care); and (4) access to the data in a more timely manner to support analyses, policy development, and program design. The MHRS, the new survey effort sponsored by the foundations, had those goals as well, with an additional focus on expanding the information available to track the impacts of the state’s 2006 health care reform legislation beyond insurance coverage, such as access to and use of care, health care costs and affordability, the quality of insurance coverage, and support for reform (among other outcomes), and providing timely updates on the impacts of reform.
In 2008, a new source of estimates for uninsurance in Massachusetts became available based on the American Community Survey (ACS), which had added a question on health insurance coverage in the 2008 survey. That survey, which provides a much larger sample size for Massachusetts than is available from other surveys (including the two state-specific surveys), addresses one of the factors that has led states to invest in separate state surveys—sample size. However, the ACS does not address the other needs: state-specific insurance coverage options in the survey questions, information on health care outcomes beyond insurance coverage, and more timely data.1
DESIGN OF THE MASSACHUSETTS SURVEYS
Massachusetts Health Insurance Survey
The Massachusetts Division of Health Care Finance and Policy began fielding its health insurance survey in 1998 to provide estimates of the uninsurance rate in the state. The survey was redesigned in 2008 to better position the state to track insurance coverage over time following the passage of the state’s 2006 health care reform legislation.2 Key changes included expanding the survey sample frame to include all residential households (not just those with a land line telephone) and modifying the survey instrument to capture more of the health insurance and health care
For more information on the ACS, see: http://www.census.gov/acs/www/ [July 2010].
See the Massachusetts Division of Health Care Finance and Policy (2007) for information on the early years of the MHRS, and Long et al. (2009) for information on the survey after the 2008 redesign.
options in the state in response to the changes introduced under health care reform.
In order to ensure that the MHIS covered nearly all residents of Massachusetts, a dual sample frame was employed, combining a random digit dial (RDD) landline telephone sample with an address-based sample. The survey, which is available in English, Spanish, and Portuguese, is conducted via telephone, web, and mail, with almost half of the survey respondents completing the survey on the web.
Conducted in the spring of each year, the MHIS collects information on health insurance coverage and basic demographic characteristics for all members of the household. More detailed socioeconomic characteristics and health care information are collected for one randomly selected household member (referred to as the target person in the household) and the other members of the target person’s family residing in the household.
The sample size of the MHIS is 4,000 to 5,000 households in each year. The margin of error due to sampling in the 2009 MHIS for estimates based on the full sample of Massachusetts residents is +/−1.5 percentage points at the 95 percent confidence interval.
Massachusetts Health Reform Survey
The MHRS gathers information on nonelderly adults in the state, the primary target population for many of the components of the 2006 health care reform initiative.3 The MHRS relies on an RDD sample, collecting information on a single, randomly selected adult aged 18 to 64 in each sample household. Like the MHIS, the MHRS is available in English, Spanish, and Portuguese. The key advantages of the MHRS relative to the MHIS are consistent data for the period prior to and following health care reform, a much broader set of questions beyond health insurance coverage, and oversamples of the populations most likely to be affected by the 2006 health care reform legislation—low- and moderate-income adults and uninsured adults.4 The oversamples of low- and moderate-income adults are obtained through geographic oversamples of households in lower income areas in the state. The oversample of uninsured adults is obtained by screening on insurance status as part of the introduction to the survey.
The MHRS is fielded in the fall of each year. The sample size of the MHRS is 3,000 to 4,000 nonelderly adults in each year, of which 300 to 400 are uninsured. The margin of error due to sampling in the 2009 MHRS for
estimates based on the full sample of nonelderly adults is +/–2.7 percentage points at the 95 percent confidence interval.
COMPARISON OF ESTIMATES OF THE UNINSURANCE RATE FOR MASSACHUSETTS
Uninsurance Prior to Health Care Reform
Estimates of uninsurance in Massachusetts in 2006, just prior to the implementation of key elements of health care reform, are available from both of the state-specific surveys—the MHIS (prior to the redesign) and the MHRS, as well as from national survey efforts, including the Current Population Survey (CPS),5 the National Health Interview Survey (NHIS),6 and the Behavioral Risk Factor Surveillance System (BRFSS).7 The MHIS, the CPS, and the NHIS report coverage for all residents of the state, whereas the BRFSS and the MHRS are limited to nonelderly adults.8
The five surveys generated very different estimates of the uninsurance rate in Massachusetts in 2006 (see Figure 12-1). For children, for whom three surveys provided data in Massachusetts in 2006, the uninsurance estimates prior to health care reform ranged from 2.5 percent (MHIS) to 7.1 percent (CPS). For nonelderly adults, for whom five surveys provided data in 2006, the MHIS and the CPS anchored the estimates as well, with the lowest estimate from the MHIS, at 9.2 percent, and the highest estimates from the CPS, at 13.8 percent.
Differences in estimates of the uninsurance rate across surveys are not unique to Massachusetts: the federal government produces multiple estimates of uninsurance based on its own surveys, and estimates from state-sponsored surveys often differ from estimates obtained from the national surveys (Call, Davern, and Blewett, 2007). The differences for Massachusetts from the different surveys reflect many factors, including differences in the sample populations included, differences in the wording of the insurance questions asked, differences in question placement
For more information on the health insurance measures in the CPS, see DeNavas-Walt, Proctor, and Smith (2007).
For more information on the NHIS, see Cohen and Martinez (2009).
For more information about the BRFSS, see: http://www.cdc.gov/brfss/about.htm [July 2010].
Estimates of uninsurance rates reported in this section are based on Urban Institute tabulations for the ACS, BRFSS, CPS, and MHRS. Estimates based on the MHIS prior to 2008 are taken from Massachusetts Division of Health Care Finance and Policy (2007), while MHIS estimates for 2008 and 2009 are based on Urban Institute tabulations. Uninsurance estimates from the NHIS are taken from Cohen and Martinez (2006, 2007, and 2009) and Cohen, Martinez, and Free (2008).
and context, differences in survey design and fielding strategies, and the survey time frame, among other things (Long et al., 2008).9 In addition, the surveys are based on samples of the Massachusetts population, which, by definition, are subject to error. Consequently, one would not expect different surveys to yield identical estimates of the uninsurance rate in the state.
Uninsurance Under Health Care Reform
Despite the variation in the point estimates of the uninsurance rate in particular years, the general trend in uninsurance in the state is similar
across the surveys. This holds true for the total Massachusetts population (see Figure 12-2), nonelderly adults (see Figure 12-3), and children (see Figure 12-4). Note that the NHIS does not report on the uninsurance rate for children in 2007 and 2008 because the relative standard error of that estimate was greater than 50 percent (Cohen and Martinez, 2009).
All of the available surveys show evidence of a substantial drop in uninsurance in Massachusetts since health care reform began in 2006. For the overall population, both the NHIS and the CPS show a drop of about 5 percentage points in the uninsurance rate between 2006 and 2008. For nonelderly adults, the estimate of the drop in the uninsurance rate ranges from 5 percentage points (BRFSS) to 9 percentage points (MHRS), with both the NHIS and the CPS showing a drop of about 6 percentage points. These findings are in contrast to trends in the nation as a whole, in which there was little change between 2006 and 2008 in the high levels of uninsurance (data not shown). Based on tabulations from the NHIS and the CPS, uninsurance for the total U.S. population remained steady at
about 15 percent between 2006 and 2008, with uninsurance at 20 percent for nonelderly adults in both surveys. The uninsurance rate for children remained steady at 9 percent based on the NHIS, however, it dropped from 12 to 10 percent in the CPS over the same period.
Looking in more detail at the estimates of the uninsurance rate for children in 2008, one sees that all three of the available surveys report very low levels of uninsurance for children in Massachusetts (see Figure 12-5). For children overall and for children in each of the income groups, the highest estimate of the uninsurance rate is reported by the CPS and the lowest estimate by the MHIS, with the ACS between the two. For example, the CPS estimates that 3.4 percent of Massachusetts children were uninsured in 2008, compared with an estimate of 2.1 percent in the ACS and 1.2 percent in the MHIS.
Over time, estimates of the uninsurance rate based on the CPS have tended to be higher than those of other surveys in Massachusetts. This pattern is consistent with studies for other states, which generally have found uninsurance estimates higher in the CPS than in state-specific sur-
veys (State Health Access Data Assistance Center, 2007). A key difference between the CPS and the other surveys reported here is the insurance question used. The CPS, which is fielded in March of each year, asks about insurance coverage over the prior calendar year, whereas the other surveys ask about insurance coverage at the time of the survey. However, the available evidence suggests that respondents are not reporting coverage for the prior calendar year accurately in the CPS (Klerman et al., 2009), raising questions about the appropriate interpretation of the insurance measure in the CPS.
The two Massachusetts surveys have provided valuable information as the state has implemented its landmark health care reform initiative, including timely feedback on the impacts of reform on insurance coverage, health care access and use, health care affordability, support for health care reform and other measures, and information on key population subgroups in the state. The range of analyses supported by those
surveys could not have been accomplished with any other available data sources. However, the surveys have faced a number of challenges:
Maintaining funding over time. The MHRS began with support from three foundations; however, with the economic downturn and changes in foundation priorities, two of the foundations decided not to continue supporting the survey effort in 2009. The MHIS, which is sponsored by the state, is facing cutbacks in 2010 as Massachusetts, like every other state, is facing serious financial difficulties.
Limited ability to support research on survey methods. Given the relatively limited funds available for the surveys, few resources have been available to support research on survey methods. Both surveys have undertaken new strategies to try to address specific survey goals (e.g., including cell phone–only households in the MHRS, increasing survey response rates), which researchers have not been able to evaluate. Of particular relevance, the MHIS introduced a new, dual frame survey design that relies heavily on survey respondents completing the survey on the web. It would be very useful to conduct a controlled experiment to see if responses
obtained via the web differ from those obtained using a telephone interviewer. This is an area in which federal funding to improve survey research methods would be valuable.
Limited ability to share the data. Limits on the available survey funds have also prevented the creation, dissemination, and support of public-use data files based on the two surveys. This, in turn, limits the use of the survey data for additional research projects. This is another area in which federal funding would be valuable to help support the creation, maintenance, and dissemination of public-use data files.
Comparisons to other states. Although both surveys provide valuable data on Massachusetts, the lack of comparable national data makes it difficult to compare Massachusetts with the rest of the country on a number of important outcomes. Such information would be valuable to place the findings for Massachusetts in the context of other states, as well as to support stronger evaluation designs than are possible with data for a single state.
Although there is likely to be a continuing role for state-sponsored surveys to address issues of particular policy relevance in each state, a number of strategies would increase the value of existing national surveys for state-specific studies. These include
providing much larger state and local samples, overall and for key population groups (including children);
making state identifiers available outside research data center settings;
providing more geocoding of local areas;
adding state-specific program names to health insurance coverage questions;
expanding survey content to include questions on access, use, and costs of care, along with other issues of relevance to national health care reform; and
making data files available more quickly and in user-friendly formats to facilitate their use by state analysts.
This paper reflects the views of the author and does not necessarily represent the views of the Urban Institute, its sponsors, or its trustees.
Call, K.T., Davern, M., and Blewett, L.A. (2007). Estimates of health insurance coverage: Comparing state surveys with the Current Population Survey. Health Affairs, 26(1), 269-278.
Cohen, R.A., and Martinez, M.E. (2006, June). Health Insurance Coverage: Early Release of Estimates from the National Health Interview Survey, 2005. Division of Health Interview Statistics, National Center for Health Statistics. Available: http://www.cdc.gov/nchs/data/nhis/earlyrelease/insur200606.pdf [May 2010].
Cohen, R.A., and Martinez, M.E. (2007, June). Health Insurance Coverage: Early Release of Estimates from the National Health Interview Survey, 2006. Division of Health Interview Statistics, National Center for Health Statistics. Available: http://www.cdc.gov/nchs/data/nhis/earlyrelease/insur200706.pdf [May 2010].
Cohen, R.A., and Martinez, M.E. (2009, June). Health Insurance Coverage: Early Release of Estimates from the National Health Interview Survey, 2008. Division of Health Interview Statistics, National Center for Health Statistics. Available: http://www.cdc.gov/nchs/data/nhis/earlyrelease/insur200906.pdf [May 2010].
Cohen, R.A., Martinez, M.E., and Free, H.L. (2008, June). Health Insurance Coverage: Early Release of Estimates from the National Health Interview Survey, 2007. Division of Health Interview Statistics, National Center for Health Statistics. Available: http://www.cdc.gov/nchs/data/nhis/earlyrelease/insur200806.pdf [May 2010].
DeNavas-Walt, C., Proctor, B.D., and Smith, J. (2007). Income, Poverty, and Health Insurance Coverage in the United States: 2006. U.S. Census Bureau, August. Available: http://www.census.gov/prod/2007pubs/p60-233.pdf [May 2010].
Klerman, J.A., Davern, M., Call, K.T., Lynch, V., and Ringel, J.D. (2009). Understanding the Current Population Survey’s insurance estimates and the Medicaid “undercount”. Health Affairs, 28(6), w991-w1001.
Long, S.K. (2010). The Massachusetts Health Reform Survey. Washington, DC: The Urban Institute. Available: http://www.urban.org/UploadedPDF/411649_mass_reform_survey.pdf [May 2010].
Long, S.K., Zuckerman, S., Triplett, T., Cook, A., Nordahl, K., Siegrist, T., and Wacks, C. (2008). Estimates of the Uninsurance Rate in Massachusetts from Survey Data: Why Are They So Different? Boston: Massachusetts Division of Health Care Finance and Policy, August. Available: http://www.mass.gov/Eeohhs2/docs/dhcfp/r/pubs/08/est_of_uninsur_rate.pdf [May 2010].
Long, S.K., Triplett, T., Dutwin, D., and Sherr, S. (2009). 2008 Massachusetts Health Insurance Survey, Methodology Report. Boston: Executive Office of Health and Human Services. Available: http://www.mass.gov/Eeohhs2/docs/dhcfp/r/survey/08his_methodology_rev.pdf [May 2010].
Massachusetts Division of Health Care Finance and Policy. (2007). Massachusetts Household Survey on Health, Insurance Status, 2007. Boston: Executive Office of Health and Human Services. Available: http://www.mass.gov/Eeohhs2/docs/dhcfp/r/pubs/08/hh_survey_07.ppt [May 2010].
State Health Access Data Assistance Center (SHADAC). (2007). State Health Insurance Coverage Estimates: A Fresh Look at Why State Survey Estimates Differ from the CPS. Issue Brief #12, November. Minneapolis: University of Minnesota. Available: http://www.shadac.org/files/IssueBrief12.pdf [May 2010].