8

Discussion and Next Steps

THE NEED FOR CHANGE

The issues and challenges facing federal data collections and the sustainability of the current system were revisited by several participants at the end of the workshop. Robert Groves said that the increasing costs of data collections combined with the possibility of declining budgets are bringing the federal statistical system to the “edge of chaos,” where a small decline in a statistical agency’s budget could threaten the existence of entire surveys. He argued that agencies should work together to develop contingency plans for situations in which a survey may have to be dropped, thinking about whether the statistical system collectively would still be able to produce some of the necessary data after a cut of this type. Robert Kominski voiced a similar concern, saying that federal statistical agencies tend to make decisions in a methodical and organized way, based on information available about the past. However, changes in the environment can happen, and sometimes these changes are quite large.

Graham Kalton went further, suggesting that the system is characterized by a tendency to maintain the status quo and fear of the possible adverse consequences of change. He was not sure that questioning the sustainability of the federal statistical system was warranted, but he agreed that the current surveys are not in line with many of the current needs described, especially increasing demand for data at smaller geographic areas and disaggregated for smaller subgroups to inform more focused policy-making decisions. The growth in this area has been a trend for many years, and it is time to discuss ways of addressing these needs.

Katharine Abraham agreed that the increased need for richer information



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 89
8 Discussion and Next Steps THE NEED FOR CHANGE The issues and challenges facing federal data collections and the sustain - ability of the current system were revisited by several participants at the end of the workshop. Robert Groves said that the increasing costs of data collections combined with the possibility of declining budgets are bringing the federal statistical system to the “edge of chaos,” where a small decline in a statistical agency’s budget could threaten the existence of entire surveys. He argued that agencies should work together to develop contingency plans for situations in which a survey may have to be dropped, thinking about whether the statistical system collectively would still be able to produce some of the necessary data after a cut of this type. Robert Kominski voiced a similar concern, saying that federal statistical agencies tend to make decisions in a methodical and organized way, based on information available about the past. However, changes in the environment can happen, and sometimes these changes are quite large. Graham Kalton went further, suggesting that the system is characterized by a tendency to maintain the status quo and fear of the possible adverse con - sequences of change. He was not sure that questioning the sustainability of the federal statistical system was warranted, but he agreed that the current surveys are not in line with many of the current needs described, especially increasing demand for data at smaller geographic areas and disaggregated for smaller subgroups to inform more focused policy-making decisions. The growth in this area has been a trend for many years, and it is time to discuss ways of address - ing these needs. Katharine Abraham agreed that the increased need for richer information 89

OCR for page 89
90 THE FUTURE OF FEDERAL HOUSEHOLD SURVEYS is evident from the discussions at the workshop. She emphasized that a global evaluation of the current state and the future of federal household surveys will involve making some difficult choices and setting priorities. Kalton argued that approaching the task incrementally is quite appropriate. Groves said that, although he agrees, he would like to see a vision crystallize in the near future. Parts of a vision have seemed to emerge during the workshop and nailing that down soon would make incremental steps toward a specific vision possible. Andrew White also urged participants to spell out the intended goals and line up initiatives with their expected outcomes, especially in light of the magnitude of the projects discussed. INTEGRATION OF SURVEY CONTENT Abraham summarized one of the main themes of the workshop as the importance of survey content integration. One aspect of this is the use of com - mon definitions for the concepts measured—to the extent that this is appro - priate—because comparability enables researchers to make better use of the information available. Kalton said that the discussion of the development of standardized disability measures was a good example of the benefits, especially when the questions are set up so that additional measures can be added to expand the definition of a concept. The main set of questions provides a valu- able benchmark for comparison across surveys. Abraham argued that making headway in the area of integration of con - tent would require agencies working together from the planning stages of a survey and collaborating during redesign efforts to determine crucial content. The burden cannot be placed entirely on the Office of Management and Bud - get (OMB). Cynthia Clark recalled her experience working on the United Nations Global Strategy to Improve Agricultural and Rural Statistics, which brought together organizations to identify the core data items that needed to be produced. Trivellore Raghunathan compared federal statistical agencies to academic departments, in which researchers are focused on their particular disciplines. His own work illustrates that bringing together interdisciplinary teams to address these types of issues works well. This was echoed in Groves’s comments that people have to stop talking to just themselves and begin a dialogue with others whom they do not usually think about when they design data collections. Hal Stern raised the question of whether, given the costs of data collec - tions, there is information currently collected by federal statistical agencies that goes beyond what is mandated or widely used. As an “outsider” (an academic), he said he can afford to raise difficult questions, but his question tied in with Abraham’s point about addressing priorities and determining collectively which measures are crucial. Edward Sondik (National Center for Health Statistics) also sees as valu-

OCR for page 89
91 DISCUSSION AND NEXT STEPS able setting core standards and benchmarks for what represents critical data in a field. In the area of health, there is an explosion of information, including data collections funded by the National Institutes of Health, and many of these data collections do not go through OMB. Private companies are also producing more and more data. Sondik said that this is not necessarily good or bad, but the increase in the volume of information from an increasing variety of sources will require federal statistical agencies to step up and provide an assessment—a “consumer’s report”—on the quality of these data. This is perhaps an important future role for the federal system, he said. Kominski reminded participants that the decentralized nature of the sta - tistical system is one of its virtues. For example, the high school dropout rate published by the Census Bureau differs from the one published by the Depart - ment of Education. This reflects differences in terms of what to measure and how to measure it, and it is not necessarily a problem, but something to con - sider when assessing the challenges involved in getting different agencies to coordinate their measures. He added that it is nevertheless important to ensure that coordination happens in a systematic way. Making a similar argument as Sondik, he observed that this is particularly true in light of increasing volumes of data produced outside the federal statisti - cal system that are receiving substantial attention, in part because they can be made available much faster than federal data. An example of this is the Google consumer price index, which is based on the tracking of online price data. Although the value and potential of these types of data are not clear, there is little doubt that researchers should at least be paying attention to these alterna - tive approaches and that the role and usefulness of “official statistics” should be evaluated in this context as well. Groves warned that the timeliness of data releases is a particularly big con - cern, because federal statistical agencies are out of sync with competing sources of information. For example, the quality of an alternative price index may be really poor, but if it is available in real time, then that may be a compelling argument for some uses. Abraham responded that a lot of the economic data are released very quickly: for example, the unemployment rate is published on the first Friday after the month to which the estimates apply, and that is quite good. Groves agreed that timeliness is relatively good in terms of the economic data released, even though he questioned why the unemployment data cannot be published weekly. However, he emphasized that in other areas the lack of timeliness is a significant problem—for example, in many cases the data released are two years old. The question becomes whether defensible estimates could be produced at a higher frequency, even if this requires more resources. Reflecting on the topic of official statistics, Kominski argued that there are relatively few statistics that are declared official. Some are used as if they were official only because there are no alternatives available. However, having more

OCR for page 89
92 THE FUTURE OF FEDERAL HOUSEHOLD SURVEYS data on similar concepts typically leads to having to confront the question of which measures are official. SMALL-AREA ESTIMATION Some of the discussion revolved around the need for small-area data and modeling techniques used to produce estimates when direct estimation is not possible. Kalton clarified that the challenges in this area are usually a combina - tion of a small-area and a small-domain problem. If the population of interest itself is small, as in the case of 5-17-year-olds in the Small Area Income and Poverty Estimates (SAIPE) Program, then the sample size of this population in a small area will also be very small. In addition, the estimate itself is often a very small proportion. These factors have consequences for modeling. He added that it is important to not lose sight of the quality of the auxiliary data used, because that is more important than the model. For example, there are distortions introduced if the data are not collected the same way in all areas, as is the case with the information about free and reduced price lunches. Concerns were raised related to data users’ willingness to embrace model- based estimates in the same way they embrace direct estimates. Kominski said that the procedures involved in SAIPE seem a little bit like “voodoo econom - ics” to many, but focusing on educating users would go a long way toward ensuring that these types of estimates are better received. Labeling the estimates as experimental or research series would also be useful, according to Groves, who said that people need some relief from the thinking that everything pub - lished by the federal statistical system is official, because that stifles innovation. Abraham agreed, saying that when statistical agencies have gone out on a limb in the past and produced what amounts to experimental series, yet explained what they were doing clearly, the user base followed. Another concern was the lack of statisticians with the skills required to implement advanced modeling techniques. Groves said that there is a commu - nity of people around the country who have these skills, as long as agencies are willing to look outside their existing staff and form alliances. INTEGRATION OF SAMPLING FRAMES Another possible direction for integration discussed at the workshop is coordination among the statistical agencies in the area of sampling frames. Clark argued that the time is right to consider the idea of a common sampling frame, and the Census Bureau’s Master Address File (MAF) represents a start - ing point to consider. Although sharing information from the MAF outside the Census Bureau is subject to confidentiality restrictions, it is important to consider whether some parts of it are not subject to these restrictions and could be made available to other agencies under some kind of agreement. One source

OCR for page 89
93 DISCUSSION AND NEXT STEPS of input to the MAF is the U.S. Postal Service’s Delivery Sequence File (DSF); perhaps the Census Bureau could add information to it and make that product available to others. Kalton recalled the Canadian example of the address register that is con- tinuously updated, in part through their labor force survey. What if the United States were to bring together all of its surveys to improve an overall address frame that everyone in the statistical community could benefit from, possibly even beyond the federal statistical agencies? Groves said that thinking about the continuous updating of the address frame does not have to be limited to the updating of addresses. Instead, it could be conceptualized as a collection of observable auxiliary data about the addresses, and various organizations could contribute information to it. Kalton added that if some of the data come from sources other than government agen - cies, the limitations could be different. For example, faster delivery times could be possible, and the confidentiality restrictions may also be less stringent. THE ROLE OF THE AMERICAN COMMUNITY SURVEY The discussions of both integration of content and sampling frames circled back to the American Community Survey (ACS) on a number of occasions. Clark said that the most important function of the ACS is to provide estimates for small areas and that it is in fact the only good source of direct estimates for small geographies. Nevertheless, other promising uses mentioned at the work - shop could certainly be discussed further. Abraham summarized the discussion about one possible use of the ACS as a more integrated household survey, with a set of rotated modules. This could increase efficiencies and lead to data that serve a broader array of analytic pur- poses. Clark talked about the possibility of using the ACS to help other agencies test and develop new modules. However, there are some obvious challenges emphasized by Abraham, including the burden placed on ACS respondents, the survey’s inability to collect information that is comparable in depth to topic- specific surveys, and practical barriers that were brought up by the ACS team. The possibility of using the ACS as a sampling frame for other surveys was also discussed. Clark said that this model works well in the National Agri- cultural Statistics Service; the Census of Agriculture accommodates screening questions for other surveys. This approach has enabled them to meet emerging needs, such as measuring bioenergy and organic production. However, she mentioned that the ACS itself in its current form has some weaknesses when it comes to rural populations, and it would not be a suitable screener for a study focused on rural America. Kalton would have especially liked further discussion about the idea of the ACS providing sample on a rolling basis. Currently, one year’s worth of ACS data has to be processed before the National Science Foundation can receive

OCR for page 89
94 THE FUTURE OF FEDERAL HOUSEHOLD SURVEYS sample for the National Survey of College Graduates (NSCG), for example. He acknowledged that providing sample on a rolling basis would involve additional data management tasks, but he thought that it was an idea worth discussing. He would have also liked more discussion of the issue of misclassification in the sample provided and how it affects a sampling design that involves rare populations. Stern brought up the point that the ACS collects a lot of data that are not released for small areas, except after five years of aggregation. He wondered whether some of the data available could at least be used for modeling pur- poses, even if they are not released. Kominski said that the ACS appeared to emerge as the silver bullet from many of the discussions, and this is perhaps not surprising given that it is a mas - sive data system and most people have not even fully considered the power of the five-year estimates, released on a yearly basis. Even with the overlap across the data contained in those releases, 10 or 15 years of these estimates will have huge potential. However, he cautioned against limiting the thinking about the future of the federal statistical system to the ACS, especially in terms of pursu - ing the idea of adding modules to the survey. He used the example of the CPS, which does have supplements, but the space is booked for every month for the next three years. The CPS has been routinely used for the past 40 years by researchers both inside and outside the government as the staging ground for many new ideas and problems to be measured, and the process has been fairly efficient, but it is not an elegant method and not necessarily something that should be transferred to the ACS. Scott Boggess (Census Bureau) reminded the workshop participants of every- thing the ACS is already doing. He pointed out that the ACS does in about four months what the 2000 census took approximately two years to do, and it does it with fewer resources. In addition to the long-awaited five-year estimates, they have been producing one- and three-year estimates, redesigned their weighting approach to improve variances at the tract level, redesigned their data products, developed a Spanish-language questionnaire, and added Puerto Rico and group quarters to the sample. The ACS is fast and responsive, he said, but he also made the point that it takes a long time to change an entire system. Kalton said that the many ideas that emerged during the workshop made him question whether another survey is needed to accomplish the goals dis - cussed. After all, the ACS has to fulfill its mandated roles before doing anything else. ADMINISTRATIVE RECORDS Participants were encouraged by the progress reported by Rochelle Martinez from OMB in the area of administrative records use. Clark mentioned that, while she was at the Census Bureau, she and her colleagues started the

OCR for page 89
95 DISCUSSION AND NEXT STEPS Statistical Administrative Records System (StARS) database, and it would be of great value if that could be made available to other agencies. Some obvious uses for administrative records are direct use, imputation, verification of data, and covariates in models, but there may be others and it is important to think broadly, she said. Kalton added that administrative records can represent a source of longi - tudinal data, sometimes with information available before and after the time of the survey data collection. The Panel Study of Income Dynamics (PSID) and the Health and Retirement Study (HRS), for example, use Social Security data to chart income patterns over respondents’ lifetimes. Jean-Louis Tambay encouraged the participants to imagine meeting five years from now and to identify current opportunities related to administrative records that will look like a real shame to have missed looking back from the future. Regarding the use of administrative records abroad, Kalton commented that Julie Trépanier’s presentation about the use of tax records in Canada was an example of a use that reduces respondent burden and is communicated to the respondent as such. The presentation by Jelke Bethlehem about the popu - lation register in the Netherlands led to a lot of debate during the workshop, and Kalton encouraged the participants to continue that dialogue, even if a register is unlikely to be implemented in the United States in a similar form. Stern made a similar argument, saying that it is difficult to imagine that there would be political will in the United States for implementing something similar to what other countries are doing with administrative records, but that does not preclude it from discussion, because registers have the potential to offer enormous cost savings. BROADER INTEGRATION OF DATA COLLECTIONS A lot of the discussion centered around the more ambitious notion of inte - gration advanced by Raghunathan, who used his own work to illustrate a way of thinking about a research problem in terms of a matrix of the information necessary to address it. Missing pieces in the matrix can be filled in with data from a variety of sources and combined using the latest modeling techniques. The analogy he drew to the statistical system as a whole generated a lot of discussion. Abraham said that the concept of the statistical system as a giant matrix with interlocking pieces was intriguing, because it perhaps presents a solu - tion to the dilemma of not being able to obtain all the data needed from one survey, as well as to the difficulties related to combining information from surveys that have evolved independently of each other. She emphasized that implementing something similar would require a more global way of thinking about the household surveys in the federal system. Roderick Little added that what is necessary is a new way of thinking about survey design and the associ-

OCR for page 89
96 THE FUTURE OF FEDERAL HOUSEHOLD SURVEYS ated analysis that goes beyond concentrating efforts on the specific survey one happens to be working on. In Abraham’s view, an overarching model, such as the matrix idea, would provide additional incentive for a discussion about what types of estimates are appropriate for federal statistical agencies to be generating. Sondik added that the lack of resources and capacity to produce needed small-area estimates should focus attention on defining core measures and indicators. Kalton observed that Don Dillman’s discussion of mixed-mode surveys becomes especially relevant in the context of integration among surveys. Although research has explored the effects of mixed-mode data collection within a survey, less is known about the consequences of combining data from two surveys that are conducted through different modes. The discussion of the disability measures illustrated that estimates are not necessarily the same, even when the questions are the same, and this could in part be due to a mode effect. Kalton made the point that surveys that use other surveys as a source of sampling for rare populations could make better use of the information avail - able from the source if there was more attention paid to coordinating content as well. In other words, if the new survey was thought of as an extension of the existing survey, then the data could be combined and used for purposes beyond what is possible with the individual surveys. Thinking about the possibilities of linking surveys can extend beyond research domains, according to Stern. He made the point that currently surveys that rely on other surveys as a source of sample tend to do so within the same domain. An example of this is the relationship between the Medical Expendi- ture Panel Survey (MEPS) and the National Health Interview Survey (NHIS). Other major benefits are possible in looking beyond the institutional boundar- ies and to other disciplines. According to Sondik, a report on developing key national indicators for children—which recognized that to accomplish this goal required going beyond established domains—is an example that could apply in a variety of areas, including health, education, and the economic situation. This recognition could inform more of what is done and lead to a focus on the critical information needed to serve as benchmarks. For example, the NHIS could also pick up basic information related to education and housing, in addition to its current content. Abraham said that the initiatives in the area of administrative records also fit well with this model if one thinks beyond survey integration to envision data integration, in which administrative records are contributing an important piece. She encouraged the participants to be bold in moving forward.