Most federal surveys come through the Office of Management and Budget’s (OMB) Office of Information Regulatory Affairs (OIRA) as part of the review process required by the Paperwork Reduction Act. Margo Schwab, of OIRA’s Statistical and Science Policy Branch, described the work of her office in promoting consistency among federal surveys in the area of disability measures.
One reason for promoting consistency is to reduce the confusion among the public and Congress caused by the availability of different statistics on what appear to be similar or identical concepts. The questions generated by measures of the number of people without health insurance in the context of the health care debate are a case in point. The Patient Protection and Affordable Care Act calls for consistent measures on disability to assess disparities, and OMB has been helping agencies realize this goal. Although consistency is not always possible (e.g., several federal agencies have their own definitions of disability that determine participation in various benefit programs), the availability of a substantial body of research on the disability measures provides a strong foundation for developing consistency in areas in which there are no statutory constraints.
Another motive for pursuing consistency is the hope that coordinated measures will allow researchers to gain deeper insight into various dimensions of disability and related policies—for example, such topics as health, housing, and transportation, which are often measured separately by different agencies. Finally, disability measures are well suited for a modular data collection plat-
form, allowing for the flexible gathering of additional information of interest to researchers, along with the disability data.
Although the definition of disability is context dependent, in recent years the concept has shifted from a focus on physical condition, disease, and impairment to more emphasis on functional limitations caused by these factors. This involves measuring limitations and outcomes separately to understand how disparities in outcomes may be eliminated. For example, the Bureau of Labor Statistics assesses disability independently and then reports employment outcomes.
The work underlying the development of the conceptual framework for measuring disability that OMB now supports for most federal surveys was initiated in the context of the American Community Survey (ACS). The Census Bureau assembled an interagency group, which included, among others, researchers from the Veterans Administration, the Department of Housing and Urban Development, the National Science Foundation, the Department of Education, the Bureau of Labor Statistics, and the Bureau of Justice Statistics, as well as various agencies in the Department of Health and Human Services. The group reviewed the legislative mandates and needs for disability data in the context of various programs and evaluated the restrictions imposed by the format of the existing questions on the ACS. The primary measurement objective identified by the group was what Schwab called “equalization of opportunity;” in other words, a measure that could identify persons who are at risk of discrimination or who lack adequate opportunities for participation in social life as a result of their limitations in functioning. Another goal was to measure severe disability in order to identify persons who need assistance to maintain independence.
Box 7-1 shows the new disability measures used in the ACS. The questions cover limitations in vision, hearing, mobility, cognitive functioning, and self-care. Those over 15 years of age are also asked about their ability to interact with their environment, including their ability to do errands alone.
The measures developed for the ACS are now used on a variety of government surveys, including the Current Population Survey (CPS), the National Crime Victimization Survey (NCVS), the National Health Interview Survey (NHIS), the Survey of Income and Program Participation (SIPP), and the American Housing Survey (AHS).
A key characteristic of the measures is the modular platform that allows various agencies to combine the items with additional questions of particular interest to their work. For example, transportation researchers can add questions about mobility, and surveys focused on employment can add questions about accommodations in the workplace. Using the same set of key measures across a variety of studies will allow researchers to examine different dimensions of disability, and they are just beginning to reap the benefits.
Schwab said that OMB endorses wider use of the measures because they are the result of a thorough review of the existing literature and extensive test-
American Community Survey Disability Measures
For sample persons 1 year of age and older:
1. Is this person deaf or does he/she have serious difficulty hearing?
2. Is this person blind or does he/she have serious difficulty seeing even when wearing glasses?
For sample persons 5 years of age and older:
3. Because of a physical, mental, or emotional condition, does this person have serious difficulty concentrating, remembering, or making decisions?
4. Does this person have serious difficulty walking or climbing stairs?
5. Does this person have difficulty dressing or bathing?
For sample persons 15 years of age and older:
6. Because of a physical, mental, or emotional condition, does this person have difficulty doing errands alone, such as visiting a doctor’s office or shopping?
SOURCE: Workshop presentation by Margo Schwab based on U.S. Census Bureau (2010).
ing, including cognitive testing and focus groups conducted by the National Center for Health Statistics. A reliability study with a split sample was also performed as part of the 2006 fielding of the ACS. This does not mean, however, that the measures could not be further improved. There is indication that using a severity scale for each of the questions about limitations might be more useful than the current dichotomous (yes/no) answer options. A measure of upper body mobility could also be added. Research is also ongoing to understand how much of the difference in prevalence estimates is due to differences in the way the surveys using these measures are administered, particularly differences in survey mode, the purpose of the survey, and whether the respondent is answering a question about himself or herself rather than about others in the household.
Charles Nelson (Census Bureau) discussed income and poverty estimates produced by the Census Bureau to illustrate situations in which a variety of complementary measures may be most appropriate. His talk focused on the estimates from the Annual Social and Economic Supplement (ASEC) of the Current Population Survey and from the American Community Survey, two
surveys that conceptualize income in the same way but are methodologically different. Other sources of income and poverty data include the Survey of Income and Program Participation and the Small Area Income and Poverty Estimates (SAIPE) Program.
The CPS ASEC, considered the source of official poverty estimates in the United States, is a computer-assisted telephone interview conducted in the spring of every year. It collects information about the previous year’s income based on a detailed set of questions that cover approximately 50 possible sources of income. It also collects information about benefits, including non-cash benefits, for a broader picture of economic well-being. The CPS has also been the source of key statistics on related topics, such as employment, Nelson said. One of its strengths is the flexibility of its content relative to that of the ACS, while maintaining the continuity of the measures in the core topic areas.
The ACS is conducted as a mailout/mailback survey throughout the year, and the reference period for the income questions is the previous 12 months. The “annual estimate” from the ACS thus becomes an estimate that spans two years, depending on when the survey is conducted. The number of questions on the topic of income is smaller than in the CPS, with approximately eight broad questions covering all sources of income. The strength of the ACS is that it produces data for all levels of geography on a wide range of topics.
The Census Bureau is planning on releasing new poverty estimates, called the supplemental poverty measure (SPM), which is based on recommendations of a panel of the Committee on National Statistics (National Research Council, 1995) and broadens the concept of income beyond money income. The Census Bureau has been researching alternative poverty measures for the past few years, and content flexibility in existing surveys has become an increasingly relevant issue as part of this work. The CPS is able to capture more benefits (e.g., school lunch, rental subsidies) and additional expenses (e.g., out-of-pocket medical expenses, child support) than the ACS.
According to Nelson, the Census Bureau’s data release schedule reflects the challenges related to publishing different estimates on income and poverty from the two sources. During the early years of the ACS, its numbers were always released before the CPS, and the strategy was to highlight topic areas other than income and poverty as part of the ACS release. But in 2003 the income and poverty numbers from the ACS nevertheless received substantial attention in the media, and this created a lot of confusion when the CPS data were released a week or two later. In subsequent years, the ACS and the CPS income and poverty estimates were released together in a single press conference. During the past two years, the Census Bureau has started releasing the CPS numbers first.
An exact match analysis revealed no systematic differences at the national level between the two data sources. Figure 7-1 shows the poverty estimates, and Figure 7-2 shows the median household income from the two surveys for
the period 2000-2009. The estimates are remarkably similar, considering the differing methodologies.
At the state level, the differences are more noticeable. Comparing two-year (2008-2009) CPS averages to 2009 ACS estimates shows overall strong correlations between the two sets of state estimates. However, 24 states and the District of Columbia had statistically different median household incomes, and 16 states had statistically different poverty rates. In most of these states, the ACS poverty rate was higher than the CPS one.
Although over the long run survey sponsors and data users may be helped by complementary measures such as these, in the short run it is often difficult to explain the differences. Nelson cited the example of the headline “Census Data Give Contradictory Views on State of Child Poverty in Maryland,” published in the Baltimore Sun, referring to a poverty rate of 7 percent based on the CPS and 13 percent based on the ACS.
In addition to being prepared to answer questions about discrepant estimates, the Census Bureau’s current strategy is to minimize the overlap in the release schedules, and highlight the strengths of each of the data sources, such as the national-level time-series data for the CPS and the subnational estimates for the ACS. The Census Bureau has also been placing emphasis on releasing documentation about the surveys’ methodology concurrently with the data in order to assist data users in interpreting the numbers. This includes a fact sheet summarizing the differences between the CPS and the ACS income and poverty estimates, a guide for when to use the ACS and when to use the CPS income data, and information about additional sources of income and poverty estimates produced by the Census Bureau.
In terms of the timing of the releases, Nelson said, the Census Bureau’s experience shows that there is no easy answer, because data users, especially the media, are more likely to use the data that are released first. During the years when the ACS was released before the CPS, there was a tendency for people to use the ACS as the “official” poverty and income estimates, rather than wait for the CPS. When the ACS and the CPS were released simultaneously, media coverage often mixed the two sources of data, and there were also logistical issues to overcome in coordinating the schedule of the two surveys. During recent years, when the CPS was released first, users often turned to the CPS for subnational data, even though the Census Bureau stopped including the state estimates in the CPS annual report, in an effort to encourage users to wait for the ACS for information at the subnational level.
Chester Bowie (National Opinion Research Center) presented the work he has done with Jennifer Madans (National Center for Health Statistics).
Although Madans is the primary author of the presentation, she was unable to attend the workshop.
Bowie, a former director of the Census Bureau’s Demographic Surveys Division, described the ACS as a national treasure, which has the potential to serve additional uses beyond the current ones. The goal of the presentation was to envision possibilities for the ACS, setting aside currently existing limitations, such as Title 13 restrictions and consideration of the existing procedures for determining the content of the questionnaire.
Currently the Office of Management and Budget and the Census Bureau cochair an interagency committee that reviews and updates the justification for each of the questions on the ACS. The committee includes over 30 federal agencies, with OMB having the authority to make the final decision about the questions.
The interagency committee also evaluates the need for new questions, which are then tested as part of a very thorough content test scheduled to occur every five years. The content test typically involves cognitive interviews, a large-scale split-ballot field test, and follow-up interviews in some cases. The agencies also have input into the evaluation criteria during the content test.
Due to the design of the ACS, which relies on estimates over a five-year period to produce small-area data, once a question is added to the ACS, it has to remain on the survey for at least five years in order to be useful for small-area estimates. Naturally, agencies and other data users would like all of their favorite questions to stay on the ACS indefinitely. Continuity is also important from the perspective of trend data. Bowie referred back to the disability questions discussed by Schwab as an example of revisions that resulted in a break in the trend data available on this topic. The changes introduced in 2008 have affected the estimates about the populations with and without a disability.
In other words, the vision of the ACS as a resource of substantive data useful to a variety of agencies across the federal statistical system represents some practical challenges. An alternative goal for the survey would be to serve as a sampling frame for other surveys in the system. This would involve limiting the small-area data produced on the basis of the ACS to possibly only a core set of demographic variables, focusing instead on collecting data that would primarily be useful for building sampling frames. This idea builds on the examples presented by Keith Rust (see Chapter 3).
The more widespread use of the ACS as a sampling frame would still involve a difficult process of prioritizing the different agencies’ needs. Even the large ACS sample may not be large enough to accommodate multiple frames for use for follow-up studies for rare populations. These populations would need to be included not only when they are the focus of a primary study but also when they are of interest as a source of sample. This would also reduce the usefulness of the ACS as a direct source of data that meet analytic needs, which could
create the appearance that the survey is less useful overall. Possible funding implications of this type of change would therefore also need to be considered.
Another alternative for the future of the ACS would be to serve as a platform for collecting data on a set of key national indicators in several areas. This would involve revising the current questions to be more in line with the core needs on various topics. For example, although there are some health-related questions currently on the ACS, these are not the most crucial set of health questions. Again, prioritizing the different agencies’ needs and defining the core or key indicators would involve significant challenges and possibly the need to include a larger number of questions in some areas to meet these needs. However, Bowie argued that having each of a handful of agencies adding a small number of questions would be a realistic option that would lead to key data for small areas on a number of important topics. A variation on this approach would be to develop a core set of questions and include the expanded topic areas as modules that are on the survey for five-year periods at a time.
Bowie emphasized that the strengths of the ACS as a large-scale survey capable of providing data for the smallest geographic areas also means that prioritizing the agencies’ needs and making decisions about its use will require difficult choices. He also acknowledged the possibility that the current design and scope are the most effective approach for the survey, but he encouraged participants to consider the alternatives and envision possibilities for how the different pieces in the system could fit together more efficiently.
Drawing on his experience, Hermann Habermann (Committee on National Statistics) discussed the role of OMB in the federal statistical system and shared his thoughts on the concept of official statistics. He remarked that OMB is a powerful institution, with authority over budgets and surveys, yet it has not assumed an active role in many years in some of the areas discussed, including greater integration in the federal statistical system and shaping official statistics. Indeed, he observed, OMB’s primary role is to protect the Office of the President—in other words, to prevent bad things from happening. Activities related to this take significant time, leaving few resources for building coalitions among agencies. There is also inertia to consider and the narrowly construed “stovepipe” nature of many organizations, including OMB. Statistical policy initiatives have to cut across many agencies, each with its own separate budget.
Although it is important for OMB to provide leadership, Habermann argued that the individual agencies also have an important role to play, without which change would be very limited. He reminded participants of some remarkable initiatives that have progressed despite the challenges, including the
Confidential Information Protection and Statistical Efficiency Act (CIPSEA), the Statistical Community of Practice and Engagement (SCOPE), and work on administrative records. Nevertheless, OMB and the agencies need to be looking at the bigger picture as well as further into the future.
Habermann referred to Jelke Bethlehem’s presentation about the use of population registers in the Netherlands. Although the American public tends to be less open to the concept of a national register than many other countries, the truth is that similar databases already exist, particularly at the Census Bureau. Habermann said that making better use of the information that already exists could perhaps be considered under a label such as “improving the use of existing products for statistical purposes,” rather than “registers.” He also said that a panel convened by the National Research Council would be in an ideal position to examine how this would work, what it would take to implement it, and how it would change the way surveys are designed. This would be one way of approaching the task of developing a new model for federal surveys systematically.
As an introduction to the topic of official statistics, Habermann recalled the 2000 census and the Census Bureau’s concerns about the quality of some of the data. There was a debate whether the data were of adequate quality to be released, and in the end they were all released because the Census Bureau’s mandate and obligation are to publish the data collected using taxpayer funds, provided that pledges of confidentiality are not violated. The debate made clear how difficult it is for an agency to be in a position in which it has to consider what is good enough, let alone official.
Habermann also mentioned some advantages to having the definition of a concept developed through a process outside the agency that collects the data. For example, even though the Census Bureau publishes poverty data, it does not define what poverty is—that is defined by society. This underscores the importance of considering the roles of OMB, statistical agencies, and others not only in the development of what should constitute official data, but also in defining the concepts that are measured.
In terms of competing measures of the same concept, Habermann made a case for transparency about the methodologies employed. Although he used to assume that data users want discrepant measures to be reconciled, he has learned that they often just want to understand the reasons for the discrepancies and are comfortable using competing estimates as long as the methodologies are clearly explained. This is part of the reason why competing estimates still exist on so many key topics, such as poverty estimates.
Habermann also talked about alternative data sources that often compete with official federal statistics. In several topic areas, many—if not most—of the data do not come from federal statistical agencies. For example, some of the data on environmental topics come from state, local, and nongovernmental offices. Gender statistics are often based on the work of nongovernmental orga-
nizations. Private companies are also producing their own data on a variety of topics; Google’s consumer price index is a good example of this. The inclination may be to dismiss these data as less accurate than federal statistics, but it may be more productive to acknowledge these trends. Federal statistical agencies could help data users, the media, and the public better understand the data available from these additional sources. Some agencies have in fact already gone beyond this type of role, engaging in discussions with Google about making the best use of available data. Exploring possibilities for combining federal statistics with data from other sources is another topic that would benefit from the insights of a National Research Council panel.
Habermann ended by reminding participants that the goal is to move the discussion on the future of federal surveys forward. Many of the speakers before him described the challenges related to the current system, and budget pressures in the future will possibly increase these challenges further. This means that the future is likely to be different; the question is what role OMB and the statistical agencies want to play in shaping that future.
Alan Zaslavsky tied the sessions on small-area estimation and survey content together by saying that a possible criterion for deciding what should be on the ACS is to include questions that are predictive of other things for which small-area estimates are needed. What is available from administrative records and other sources could also be considered, making the ability to fill in gaps another criterion. There is no need to spend a lot of resources on collecting data that are already available.
He added that there are good reasons for Schwab’s argument for measuring concepts in a uniform way if the goal is to merge the data, but for modeling purposes what is needed is the ability to link the data, knowing what the correlations are. It is not even necessary to do this every year, just once in a while to refresh information in the small-area framework.
Barbara O’Hare followed up by proposing that a question about telephone access (landline, cell, both, or neither) could be added to the ACS. Information about telephone status could then become a link to other surveys that are conducted by phone. This is especially important given budget pressures and the uncertain future of random digit dialing surveys.
Following up on a comment by Fay, Deborah Griffin (Census Bureau) clarified that the reason the Census Bureau moved away from referring to the ACS as a replacement to the census long form is because the ACS is, in fact, different, and the Census Bureau would like to be able to convey that better to data users. This does not mean that the ACS cannot be used for the same types of analysis that the long form was used for. The ACS can do much more
than what the long form could, and that is also becoming evident from this workshop.
Today the primary purpose of the ACS is to produce small-area data for the content that was inherited from the long form and that the Census Bureau promised to produce, Griffin said. The release of the first five-year estimates will represent the point at which the Census Bureau has met that goal. However, along the way, the Census Bureau has had to publish other data to keep people interested, so there were one-year and three-year estimates, and then people started talking about other things as well, such as using the data as input for SAIPE or using the survey as a test vehicle for the 2020 census or as a sampling frame. In Griffin’s view, the Census Bureau staff who are currently working on the small-area estimates, who see that as the survey’s mandate, are not going to want to turn the ACS into simply a sampling frame, just because it would make a good sampling frame, even if the five-year data are not going to be perfect. These ideas will have to be revisited to realign priorities, she said, and also to make sure that the ACS does not try to do so many things that it can no longer do anything well.
Daniel Kasprzyk (National Opinion Research Center) agreed with Bowie that the ACS is a national treasure, observing that the owner of that treasure has the duty to plan ahead for the ACS in ways that benefit the entire federal statistical community. Beginning the planning process cannot wait, in his view, because it will take 10 or 15 years to implement changes that are planned today.