The changes in data required from national household surveys to accurately monitor health and social welfare programs in the United States pose substantial challenges for data collectors. These challenges include improved coordination between the statistical agencies that collect data and the policy and research agencies that use the data; research to inform questionnaire design and administration; approaches to validating survey responses; means to build flexibility into surveys with respect to content changes; and modification of sampling schemes to provide reliable subnational estimates.
Workshop participants stressed that constraints on survey budgets will make it particularly difficult to meet these challenges, placing yet greater importance on coordination of effort and identification of priorities for data. Coordination has the potential to reduce duplicative work, fill gaps in knowledge, standardize concepts and definitions, and facilitate consistent approaches to questionnaire design, sampling strategies, and survey administration. Coordination is also essential for data collectors to properly evaluate tradeoffs and make the best possible choices among competing uses of limited resources for new and modified data collection, expanded samples, and research on data collection methods and data quality.
The need for partnerships among agencies was a major theme of the workshop. In particular, better coordination is needed between the federal statistical agencies (e.g., the Census Bureau, the National Center for Health Statistics, and the Bureau of Labor Statistics) and the federal program planning and evaluation agencies (e.g., the Office of the Assistant Secretary for Planning and Evaluation, the Administration for Children and Families, the Agency for Health Care Policy and Research, and the Health Care Financing Administration in the U.S. Department of Health and Human Services; as well as the Social Security Administration). The statistical agencies played little role in the development of PRWORA; they must now work closely with the agencies that are involved in assessing the effects of changes in health and welfare programs if national household surveys are to be able to provide relevant information for analysis purposes.
Federal agencies that sponsor related research, such as the National Institute on Aging, the National Institute on Child Health and Human Development, and the National Science Foundation,
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 15
--> Implications For Data Collectors The changes in data required from national household surveys to accurately monitor health and social welfare programs in the United States pose substantial challenges for data collectors. These challenges include improved coordination between the statistical agencies that collect data and the policy and research agencies that use the data; research to inform questionnaire design and administration; approaches to validating survey responses; means to build flexibility into surveys with respect to content changes; and modification of sampling schemes to provide reliable subnational estimates. Workshop participants stressed that constraints on survey budgets will make it particularly difficult to meet these challenges, placing yet greater importance on coordination of effort and identification of priorities for data. Coordination has the potential to reduce duplicative work, fill gaps in knowledge, standardize concepts and definitions, and facilitate consistent approaches to questionnaire design, sampling strategies, and survey administration. Coordination is also essential for data collectors to properly evaluate tradeoffs and make the best possible choices among competing uses of limited resources for new and modified data collection, expanded samples, and research on data collection methods and data quality. Coordination The need for partnerships among agencies was a major theme of the workshop. In particular, better coordination is needed between the federal statistical agencies (e.g., the Census Bureau, the National Center for Health Statistics, and the Bureau of Labor Statistics) and the federal program planning and evaluation agencies (e.g., the Office of the Assistant Secretary for Planning and Evaluation, the Administration for Children and Families, the Agency for Health Care Policy and Research, and the Health Care Financing Administration in the U.S. Department of Health and Human Services; as well as the Social Security Administration). The statistical agencies played little role in the development of PRWORA; they must now work closely with the agencies that are involved in assessing the effects of changes in health and welfare programs if national household surveys are to be able to provide relevant information for analysis purposes. Federal agencies that sponsor related research, such as the National Institute on Aging, the National Institute on Child Health and Human Development, and the National Science Foundation,
OCR for page 15
--> could also participate in cross-agency cooperative efforts to improve the available data for monitoring and evaluating health and social welfare programs. These agencies sponsor surveys (e.g., the Panel Study of Income Dynamics) that have provided useful information in the past and that could both contribute to and build on improvements in the major national household surveys. Given that national surveys cannot carry the entire burden of providing needed data for program analysis, workshop participants stressed the importance of forming partnerships that could provide useful data for linking with national surveys, supplementing them, and evaluating the quality of their responses. Such partnerships would involve the collaboration of federal statistical and policy agencies with state agencies. Workshop participants also noted that several private foundations and other organizations are involved in data collection to assess changes in health and welfare programs (see Table 1). While private organizations often have their own interests in terms of question content and may obtain only limited data, it could be mutually beneficial for them and government agencies to exchange information about questionnaire design and administration and survey results. The utility of coordinating efforts between the states and the federal government was underscored at the workshop. Linking national survey data with state and local administrative data through exact matches of records for individuals and households is a way not only to validate the survey data, but also to add information from the administrative records that is not readily collected in a survey. States can also help to ensure the relevance and quality of national household surveys by reviewing questionnaire wording and other aspects of survey design from their knowledge of local program operations and policy concerns. Workshop participants noted further that, under PRWORA, states have strong financial incentives to produce timely reports but few incentives or resources to design comparable, reliable and accessible data, while the federal government has a strong interest in ensuring the development of a consistent and reliable tracking system for the new state programs. By forming partnerships, the states and federal government could better assure consistency and reliability of measures, while the states could reduce the costs of implementing new administrative data systems.10 However, an unintended result of devolution may be that it places new barriers in the way of intergovernmental partnerships that are designed to improve the accuracy and availability of data on health and social welfare programs. PRWORA, along with the granting of administrative waivers, makes state and local governments increasingly responsible for welfare programs and their outcomes. To the extent that data provided by federal-state linkages can be used to critically examine welfare programs, states may be reluctant to reveal information and to enter into partnerships with the federal government. The National Assessment of Educational Progress (NAEP) provides an example of a successful federal-state data collaboration effort on a highly politically charged issue. Originally, states did not want measures of educational achievement that could be compared across jurisdictions, and NAEP was designed so that only national-level estimates could be produced from the data. Subsequently, increasing public concern about educational issues led to a willingness on the part of the states to compare their performance, and NAEP was recently redesigned to provide state-level estimates. Other types of data linkages that were discussed at the workshop concerned integration of survey responses from households with data from other sources, such as reports of employers and 10 Brady and Snow (1996) suggest that the development of data systems to track caseloads over time and across counties, as mandated by PRWORA, will pose substantial challenges for many states.
OCR for page 15
--> providers, or data on contextual characteristics of areas in which respondents reside (e.g., unemployment rates). Linking household survey responses to employer and provider-level data has always been important for analyses of health care programs, which require accurate measures of medical care expenditures and features of health insurance plans that households typically cannot provide. In order to have regularly updated measures for assessing health care programs, MEPS was initiated as a continuing survey of medical care expenditures in which household survey reports are augmented by reports from the households' employers, insurance carriers, and medical care providers. (Previously, such medical expenditure surveys were only conducted at long intervals.) In addition, MEPS is linked to other health-related surveys: the sample for MEPS is a subsample of NHIS, which provides a rich set of data on health status and conditions, and the MEPS employer questionnaire is the same as that used for the National Employer Health Insurance Survey (NEHIS), which is a large annual survey of employers about their health insurance plans. Historically, there has not been a parallel need in the welfare context to link household survey responses with employer or provider data, which can be difficult and costly to obtain. However, this may change with the new emphasis on such third-party benefits as employer wage subsidies and day care services. Also, it may be possible through collaborative efforts of statistical and policy agencies to identify opportunities to link welfare-related surveys in ways that are similar to those for health-related surveys.11 The Survey of Program Dynamics (SPD), which PRWORA mandated to help track the effects of welfare reform, particularly for children, represents a type of linkage. The SPD sample cases, who will be followed annually for 6 years beginning in 1997, are drawn from the 1992 and 1993 SIPP panels, making it possible to link the SIPP data already collected with information collected subsequently in the SPD. Several participants noted that the Health Insurance Portability and Accountability Act provides important new opportunities for data standardization and integration in the health area. In particular, the administrative simplification provisions of HIPAA direct the secretary of the Department of Health and Human Services to develop a single set of national standards and identifiers for certain electronic transactions, including claims, enrollment, eligibility, payments, and coordination of benefits. Health care providers and health plans will be required to use these national standards for all electronic administrative and financial transactions. If properly implemented, HIPAA opens the door for the development of a core, standardized health benefits database that potentially could be readily linked to other data. Workshop participants stressed the need to protect the confidentiality of individual information in considering data integration and linkage. It may not be easy to provide access to data sets that link survey responses with administrative records or with reports of employers or providers in ways that maintain the assurances of confidentiality protection that are embodied in law and statistical agency practice. 12 More generally, throughout the workshop participants stressed the need to carefully consider both the legal and ethical issues involved in collecting detailed survey data on program participants. 11 Such linkages among federal surveys may be facilitated by new legislation. The administration recently proposed a bill that provides for sharing of microdata among federal statistical agencies to permit the development of more efficient sample designs and similar tasks. The bill was not acted on in the 104th Congress (1995-1996), but its provisions were incorporated into a bill that is being considered in the 105th Congress (1997-1998). 12 U.S. Code Title 13, for example, prohibits the release of individually identifiable data for samples drawn directly or indirectly from the decennial census. See Duncan, Jabinc, and de Wolf (1993) for a review of confidentiality provisions and issues for government statistics.
OCR for page 15
--> Questionnaire Design And Administration Workshop participants underscored the need for federal statistical agencies to conduct methodological research on such issues as the appropriate design of questionnaires and wording of question content to elicit accurate survey reports of participation and benefits in an era of welfare program devolution and experimentation. Cognitive research and such techniques as focus group analysis will likely be needed to develop questionnaires that respondents can understand and respond to appropriately (e.g., indicate their participation in a welfare program when benefits take such forms as wage or day care subsidies and not cash). Another topic for research is that of optimum questionnaire length, balancing data needs against survey costs and burden on respondents. The need for more information about children, both to accurately characterize program eligibility (especially for child-only cases) and to provide adequate outcome measures for children, raises issues of questionnaire length because most household surveys do not now obtain as much information about children as about adults. This need also raises issues of how to obtain accurate information for children who do not report for themselves. Finally, research will be needed to develop appropriate training methods and materials for interviewers so that they are able to elicit accurate responses to the extent possible. Coordination among federal statistical agencies for all of these kinds of methodological research would be beneficial so that research can be conducted in a cost-effective manner and research findings can inform improvements in all of the major national household surveys. Data Validation Monitoring and assessment of health and social welfare programs in the new environment will require accurate measures of program participation and benefits, of characteristics that are needed to estimate program eligibility, and of outcome variables. Maintaining current levels of accuracy, let alone improving on them, will be difficult given the likelihood of increased variation in program provisions across states and localities and over time.13 Survey response rates may also fall if questionnaires are lengthened to accommodate new data needs. Validation of survey responses will be essential, not only to establish the quality of the data collected, but also to suggest changes to questionnaires and survey procedures that could improve data quality. Workshop participants suggested a number of specific methods for validating survey questionnaires, including matches of survey and administrative data for cross-validation, cognitive research designed to ensure that respondents understand survey inquiries, and experiments embedded in surveys to test the sensitivity of results to different ways of asking questions about new program features. Workshop participants also suggested collecting information about programs from respondents, as a validation technique. Thus, inquiries about program names, eligibility criteria, and benefit levels may not only measure awareness, but also help to establish the accuracy of responses. Participants noted that proper training and informing of interviewers about the details of state and local programs may reduce respondent confusion and reporting errors. 13 Not only are program provisions likely to vary across jurisdictions, but caseworkers may have greater flexibility to tailor program provisions to the circumstances of individual participants (e.g., in determining what activities meet the work requirements).
OCR for page 15
--> Flexibility Flexibility in designing and implementing survey questions emerged as an important issue in the workshop discussions. Since states may now experiment with different programs with different names, participants suggested that the need to make last-minute modifications to survey questionnaires may be common. Surveys that ask respondents whether they participated in program X will be useless if the program is now called Y. Although computer-aided surveying techniques, such as computer-assisted personal interviewing (CAPI) and computer-assisted telephone interviewing (CATI), have provided such benefits as reducing inconsistent responses (inconsistencies can be resolved at the time of the interview instead of in a subsequent editing process), statistical agencies have not found it easy to modify CAPI/CATI questionnaires in a timely manner. One suggested short-term solution is to supplement the computer-aided survey with a small sample that is interviewed with a paper questionnaire that can be modified as needed. Sample Design The need for statistical agencies to carefully evaluate appropriate sampling schemes for the major national household surveys in light of increased data needs for program analysis was a recurring theme of the workshop. Participants discussed the possible benefits of expanding survey sample sizes to allow for subnational analyses and of developing new stratification schemes to overrepresent certain vulnerable populations. Taking account of attrition in longitudinal surveys and, if necessary, developing refreshment samples were also stressed as important sampling design considerations. The existing federal household surveys arc designed for national and sometimes regional level analyses, but they are generally unable to provide reliable statistics on specific subnational areas. Given program devolution, the ability to track and monitor outcomes by state—and even by county—may become increasingly important. Thus, one suggested modification to sampling schemes is to supplement the existing samples so that the surveys are representative of individual states. None of the major surveys can now be used to provide reliable estimates for each state, although the CPS sample design is representative of all states and is designed to provide reliable estimates for the largest states. (The NHIS sample design is also state representative.)14 Several workshop participants also suggested the value of modifying the existing sampling schemes to overrepresent vulnerable populations. PRWORA imposes strict new eligibility requirements that affect specific groups of people, including disabled children, legal immigrants, unemployed people, and teenage parents. Often, these groups are small segments of both the total population and most national samples of the population, so that the current national surveys are often unable to provide reliable estimates for them,15 yet such groups may be the ones that are most affected by the new legislation. Redesigning national household survey samples to provide more reliable data for analysis of states or small population groups could be very costly. Workshop participants stressed the need to search for sampling schemes that balance costs and benefits. For example, one possibility is to supplement national samples each year with added sample for selected states, on a rotating basis. 14 See Appendix C for additional details on the sampling schemes used in CPS, SIPP, NHIS, and MEPS. 15 The existing surveys are not only unable to provide precise estimates for these groups, but they are also often unable to even identify some of these subgroups, such as immigrants and disabled children (see Appendix C).
OCR for page 15
--> Also, it may be possible to identify groups of states with sufficiently similar programs to permit pooling their samples, thereby reducing the added sample size that might otherwise be required. Workshop participants expressed a wide variety of views in the extensive discussion about modifying the sampling schemes for the major national household surveys. Some participants argued that the sample designs should be completely revised, others argued to make no changes, and still others suggested some middle ground. Perhaps the most consistent theme was the need for a more careful examination of these issues in light of long-term data requirements and existing budget constraints. In addition, participants stressed the need to carefully consider the statistical issues involved in supplementing existing samples while maintaining a nationally representative design.