State of the USA Health Indicators: Letter Report (2009)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Appendix C Domain Estimates, Reliability, and Small-Area Estimation Many indicators can be presented at the most aggregated level (national means or rates) and for smaller subgroups. Subgroups for which measures are estimated are called “domains” in the survey world, and in this section we use this term to refer generically to any such subgroup. Domains of interest for the State of the USA, Inc. (SUSA) might include geographic areas (regions, states, counties, metropolitan areas, rural/ urban comparisons) and demographic subgroups (racial/ethnic groups, age groups, socioeconomic strata). (This use of the term should not be confused with use of the term “domain” elsewhere in the report when it then refers to a conceptual domain, i.e., a set of related concepts or measures.) Although the primary presentation of indicators on the SUSA website will be at the national level, indicators may also be presented for more refined domains for at least two reasons. First, visitors to the SUSA site will often be interested in and engaged by indicators that are specific to their local areas, or to other domains of particular interest to them such as their age group or national origin group. Viewing such domain-specific information may make the indicators more meaningful and relevant to the personal experience of the user. While SUSA cannot substitute for or incorporate the vast array of information resources available for planning by state and local government as well as commercial actors, presentation of domain indicators can enhance the value of the site to the general user. Second, measures of disparities are essentially comparisons across particular domains. Presentation of disparities was highlighted as a key 73 

74 STATE OF THE USA HEALTH INDICATORS objective for the health indicator set and is relevant to many of the other SUSA topic areas. The disparities section of this report details the types of comparisons that might be presented as disparities and how these and other domain comparisons might be presented as “drill-downs” linked to the main indicator displays. Development of domain-level indicators for SUSA presents a variety of challenges based on the varying relevance of the indicators and avail- ability of data to calculate them for various types of domains. Not every indicator can be defined meaningfully for every domain; for example, a measure of air quality might be defined for states but might not be mean- ingful for age groups. The following discussion focuses on population-based measures for which domain estimates are conceptually meaningful, although not always practically obtainable with the data currently available. Two gen- eral issues affect the potential for developing domain estimates. First, the necessary variables might not be available from the same data source to both produce the indicator and define the domains of interest. For exam- ple, mortality reports used to estimate life expectancy typically include race but not income or education. A variant of this problem is that even if the data are collected in a suitable form, detailed domain-level linkages might not be publicly available due to concerns about confidentiality. For example, microdata (individual-level data) from some of the key federal health surveys, including the National Health Interview Survey (NHIS), are not generally released to the public with geographical detail below four regions, although more detailed geographical identifiers can be ana- lyzed by researchers subject to restrictions on access. Second, as data are broken down to smaller domains, the amount of data available for each domain also becomes smaller, reducing the sta- tistical precision of estimates for those domains. With a simple random sampling survey design, the standard error (SE) of estimates, a measure of the typical size of errors due to random variation, is inversely propor- tional to the square root of the sample (or population size). Thus for a domain constituting a fourth of the population, the SE would be twice as large as for the entire population. With the more complicated designs typical of national surveys, the relationship of domain size to SE might be more complex; for example, the NHIS design collects no data at all from many counties. The more or less gradual decline in precision for smaller domains suggests that some standards are required for precision and that domain measures should not be reported if they do not meet those standards. Such a procedure is followed in some standard tabular reports, where low reliability (or confidentiality concerns) might lead to suppression of measures for some small domains; criteria for suppression are set by 

APPENDIX C 75 the various agencies producing the reports. The criterion for suppression might be a large SE relative to some absolute standard of an important difference. Alternatively, the criterion might be based on a statistical mea- sure of reliability, comparing the SE of domain measures to the underlying true variation among domains. Zaslavsky (2001) argues, for example, for interunit reliability (IUR) as a measure of reliability of indicators used for comparisons among health care providers, with IUR > .7 as a minimal standard for comparison among domains and IUR > .9 desirable to make most comparisons statistically significant when there is a real difference. It might be argued that sampling variation is not an issue for indica- tors based on essentially complete data, such as vital statistics or the cen- sus. This argument would be valid if the purpose of SUSA were simply to report what has happened in the past. However, even indicators based on population information (complete, rather than sampled) about the relevant events might lack reliability for making broader inferences about the domains of interest. Thus, for example, we might be able to state with a high level of certainty that a small town had three deaths in the past year, but this would not be particularly useful for deciding whether some meaningful and persistent pattern of excess or low mortality held in that town (Elliott et al., 2006). Schematically, indicator estimation for domains could fall into one of the several scenarios. 1.  dequate data might be available for domain estimates from the A preferred data source, that is, the same source used for the national indicator. In this case, the domain indicators would be obtained in the same manner as the national indicator. (Such estimators based only on data from within the domain are called “direct” estima- tors.) This would be the case for most domains, for example, for indicators based on the census or vital statistics, but as noted above even these might not be reliable for very small domains. 2.  dequate data are available for domain estimates from a usable A data source other than the preferred data source. An example might be “percentage with a usual source of care.” The recommended data source for that national indicator is the Medical Expenditure Panel Survey-Household Component (MEPS-HC), a high-quality survey conducted using nationally uniform methods by a single agency. The MEPS-HC sample consists of 15,000 households per year, or an average of about 300 per state. Furthermore the sample is based on the NHIS, which uses a clustered sample based on counties; thus a given state might be represented by only one or a few counties. Hence the representation of most states does not sup- port an adequately precise measure at the state level, and indeed 

76 STATE OF THE USA HEALTH INDICATORS MEPS data are not released at the state level. Another survey, the Behavioral Risk Factor Surveillance System (BRFSS), asks a similar question and uses a much larger sample size (more than 300,000 respondents per year) with adequate representation of every state and some large counties. However the BRFSS data collection is lim- ited to households with land-line telephones, has limited follow-up of non-respondents, is conducted separately by each state, and uses item wording that is different from the MEPS, all of which make it a less valid population measure and not entirely comparable to the MEPS estimates. Therefore, one would want to use the MEPS for the national measure and for demographic (e.g., racial/ethnic subgroup) domains, and to use the BRFSS for geographic domains such as states or substate areas. Another approach that might work for some other measures with small sample sizes would be to combine data across years, obtaining a measure that is a less valid estimate in any given year but more precise for domains than a single-year measure.   A general disadvantage of these approaches is that the domain indicator data would not aggregate up to the national indicator because they are based on a different methodology. Hence it would be essential to include information on the page with domain esti- mates explaining to the user that the information is presented only for comparative purposes and is not comparable to the main national indicator. To allow valid comparisons of domain indica- tors to national indicators, the national mean for the alternative data source should also be presented with the domain indica- tors. (Another approach that might avoid such inconsistencies is described below.) 3.  here are no data of acceptable precision available for the domains T in question, due to small sample sizes and/or a sample design that does not cover all areas. In this case, indirect estimators (using data from outside the domain) might be used, as described below. Statistical “small-area estimation” approaches can sometimes be used to develop usable domain estimates in scenarios 2 and 3 above. “Small- area estimation” refers to methods for obtaining usable estimates for domains for which sample sizes are inadequate to produce adequately precise estimates using only data from within the domain. The essence of these methods is to use data from outside the domain (often, national data) to estimate relationships of the measure of interest to other vari- ables, and then use those relationships to improve estimation of the mea- sure. Several statistical approaches to small-area estimation are described briefly; a comprehensive review appears in Ghosh and Rao (1994). 

APPENDIX C 77 •  ynthetic estimation: First estimate rates or means for demographic S groups, such as white males aged 30–34, and then combine these rates weighted by the proportions the groups constitute of the population in the domain to obtain a domain estimate. This method captures variations due to the differing composition of the domains. •  egression estimation: Regress area measures on other variables R (covariates) measured for each area with greater precision; then calculate and report regression predictions for each domain. Note that regression estimation can be used to calibrate a measure with large sample size but lesser validity (like BRFSS, in the preceding example) to match estimates from another system with less sample but better validity (like the MEPS); see Xie et al. (2007). •  omposite estimates: Calculate model-based predictions for each C domain, using synthetic or regression estimation or some other variant. Then combine the model and direct estimate to obtain a composite estimate that is more accurate than either of its compo- nents alone. •  mpirical or hierarchical Bayes estimators use a multilevel model E to derive the best weighting to give the model and direct esti- mates when they are combined in a composite. In essence, each is weighted proportionally to its precision. If the direct estimator is more precise relative to the predictive accuracy of the model, the direct estimate receives more weight; conversely if the direct esti- mator is less precise (due to small sample size in the domain) then the model-based estimator receives more weight. Similar models can also be used to improve estimates by combining information over time or by jointly estimating several related variables, such as income levels in several age groups. When domain estimates are drawn from a reliable but inconsistent source (as in the example of the MEPS and the BRFSS described above), the domain estimates can be made consistent with the national estimates from a different source (“calibrated” to national estimates) by either sim- ple or more sophisticated statistical methods. Simple calibration methods include ratio adjustments or weighting to make a total from one survey consistent with the other. As mentioned above, regression estimation can also be used for this purpose. For more sophisticated adjustments, measures from the detailed but less valid survey (e.g., the BRFSS) can be regarded as domain-level covariates for small-area estimation for the national survey (e.g., the MEPS). The Small Area Income and Poverty Estimates (SAIPE) program of the Census Bureau releases small-area estimates of income and poverty 

78 STATE OF THE USA HEALTH INDICATORS by age group for states, counties and school districts (http://www.census. gov/hhes/www/saipe/, REFS to CNSTAT reports). These are calculated using a multivariate hierarchical Bayes model. A relatively new Small Area Health Insurance Estimates (SAHIE) program releases similar estimates for states and counties (http://www.census.gov/hhes/www/sahie/). Estimates from these programs are likely to become increasingly accurate as data from the American Community Survey (ACS) become available, providing additional small-area detail. The Substance Abuse and Mental Health Services Administration (SAMHSA) releases small-area estimates for states and sub-state areas of variables related to substance use, treat- ment and mental health, based on data from the National Survey on Drug Use and Health (NSDUH) (SAMHSA, 2008). Numerous research studies have been performed to develop small-area estimates of health-related indicators, of which we cite only a few examples (Nandram and Choi, 2005; Schenker and Raghunathan, 2007; Xie et al., 2007). However, it is not evident that any of these have been adopted by any agency to be produced as an ongoing series. Thus although small-area estimation has the potential to fill important gaps in availability of domain estimates, the actual availability of such estimates is limited. The SUSA project should monitor the future availability of such estimates and encourage their development on an ongoing basis by agencies. REFERENCES Elliott, M., A. Zaslavsky, and P. Cleary. 2006. Are finite population corrections appropriate when profiling institutions? Health Services and Outcomes Research Methodology 6(3):153– 156. Ghosh, M., and J. N. K. Rao. 1994. Small area estimation: An appraisal. Statistical Science 9(1):55–76. Nandram, B., and J. W. Choi. 2005. Hierarchical bayesian nonignorable nonresponse regres- sion models for small areas: An application to the NHANES data. Survey Methodology 31(1):73–84. SAMHSA (Substance Abuse and Mental Health Services Administration). 2008. Substate estimates from the 2004–2006 national surveys on drug use and health. Rockville, MD: SAMHSA. Schenker, N., and T. E. Raghunathan. 2007. Combining information from multiple surveys to enhance estimation of measures of health. Statistics in Medicine 26(8):1802–1811. Xie, D., T. E. Raghunathan, and J. M. Lepkowski. 2007. Estimation of the proportion of overweight individuals in small areas—a robust extension of the fay-herriot model. Statistics in Medicine 26(13):2699–2715. Zaslavsky, A. M. 2001. Statistical issues in reporting quality data: Small samples and casemix variation. International Journal for Quality in Health Care 13(6):481–488.