National Academies Press: OpenBook

Principles and Practices for a Federal Statistical Agency: Sixth Edition (2017)

Chapter: Practice 3: Use of Multiple Data Sources for Statistics That Meet User Needs

« Previous: Practice 2: Necessary Authority to Protect Independence
Suggested Citation:"Practice 3: Use of Multiple Data Sources for Statistics That Meet User Needs." National Academies of Sciences, Engineering, and Medicine. 2017. Principles and Practices for a Federal Statistical Agency: Sixth Edition. Washington, DC: The National Academies Press. doi: 10.17226/24810.
×

Practice 3: Use of Multiple Data Sources for Statistics That Meet User Needs

FEDERAL STATISTICAL AGENCIES AND PROGRAMS cannot be static. To provide information of continued relevance for public and policy use, they must regularly engage with a broad range of users to identify emerging needs and look for ways to develop new information that can serve broad purposes. To improve the quality and timeliness of their products, they must keep abreast of methodological and technological advances and be prepared to implement new procedures in a timely manner (see Practice 9). They must also continually seek ways to make their operations more efficient and less burdensome (see Practice 10).

Preparing for the future requires that agencies periodically assess the justification, scope, and frequency of existing data series, plan new or modified data series as required, and be innovative and open to new ways to improve their programs. Because of the decentralized nature of the federal statistical system, innovation often requires cross-agency collaboration (see Practice 13) and a willingness to implement different kinds of data collection efforts to answer different needs.

Two changes in policy and outlook can help statistical agencies foster the needed spirit of innovation. The first is to focus on the desired outputs of their programs by defining their primary business as that of providing relevant, accurate, and timely statistics obtained in a cost-effective manner. An output-oriented focus should help agencies justify and implement difficult decisions to modify or replace data collection and estimation programs that have lost their relevance, timeliness, or accuracy.

Suggested Citation:"Practice 3: Use of Multiple Data Sources for Statistics That Meet User Needs." National Academies of Sciences, Engineering, and Medicine. 2017. Principles and Practices for a Federal Statistical Agency: Sixth Edition. Washington, DC: The National Academies Press. doi: 10.17226/24810.
×

The second is to adopt as a matter of stated policy a paradigm of using multiple data sources to generate needed information as an expansion of the long-dominant paradigm of using probability sample surveys. This new paradigm, which federal statistical agencies are already embracing, recognizes the continued importance of surveys, both cross-sectional and longitudinal. At the same time, it explicitly recognizes the roles of administrative records and other third-party sources, along with the use of new methods for combining data from multiple sources, as key elements of a cost-effective strategy to serve users’ needs.42

In considering new data collection, estimation, and dissemination strategies for the future, statistical agencies must be mindful of tradeoffs among relevance, accuracy, timeliness, comparability over time and with other data sources, transparency, costs, and respondent burden. It will not usually be possible to maximize all seven criteria at the same time, given constrained budgets, but using multiple data sources will enable statistical agencies to better address this challenge.

ROLES FOR SURVEYS

Many current statistical programs rely on well-established probability sampling methods that draw representative samples of a population, such as household members or business establishments, interview the sample units, and produce estimates that account for known errors in population coverage and missing data and have a quantifiable level of uncertainty from sampling variability. Box III.1 provides a brief history of probability sampling for federal statistics and lists examples of long-running federal surveys.

Declining rates of response over the past 30 years in the United States (and in other countries), however, are making it increasingly difficult to contain the costs of data collection with traditional surveys in ways that do not risk compromising the quality of the data (see, e.g., Brick and Williams, 2013; de Leeuw and de Heer, 2002).43 User demands for timeliness and granularity of estimates also strain the ability of statistical agencies to respond using established survey techniques.

Survey researchers are actively seeking ways to maintain and improve both the quality and the cost-effectiveness of surveys. For example, more surveys are using multiple modes to facilitate response (Internet,

__________________

42 See Lohr and Raghunathan (2017) on methods for combining survey and nonsurvey data with examples of applications.

43 Lower response rates reduce the effective sample size and increase the sampling error of survey estimates; lower rates may also increase response bias in survey estimates.

Suggested Citation:"Practice 3: Use of Multiple Data Sources for Statistics That Meet User Needs." National Academies of Sciences, Engineering, and Medicine. 2017. Principles and Practices for a Federal Statistical Agency: Sixth Edition. Washington, DC: The National Academies Press. doi: 10.17226/24810.
×
Suggested Citation:"Practice 3: Use of Multiple Data Sources for Statistics That Meet User Needs." National Academies of Sciences, Engineering, and Medicine. 2017. Principles and Practices for a Federal Statistical Agency: Sixth Edition. Washington, DC: The National Academies Press. doi: 10.17226/24810.
×

smartphone, telephone, mail, in person), as well as using paradata44 to improve survey operations and facilitate “responsive” or “adaptive” survey designs (see National Research Council, 2013a).

Surveys should remain an important component of federal statistical agencies’ portfolios for two major reasons: (1) some information is not readily ascertained except by asking questions; and (2) surveys can collect information on many characteristics at the same time, thereby permitting rich multivariate analysis. Yet the challenges to the survey paradigm make it essential to consider how use of other data sources can bolster the completeness, quality, and utility of statistical estimates while containing costs and reducing respondent burden (see National Academies of Sciences, Engineering, and Medicine, 2016, 2017b).

ROLES FOR ADMINISTRATIVE RECORDS

Administrative records include records of federal, state, and local government agencies that are used to administer a government program. Examples include U.S. Social Security Administration records of payroll taxes collected from workers and benefits paid out to retirees and other beneficiaries; state agency records of information provided by applicants for assistance programs and payments to applicants deemed eligible; and property tax records of local governments.

Administrative records are not generated probabilistically, as are surveys, but they are not unlike household or business censuses and can be evaluated

__________________

44 Paradata are data about the source that are gathered in real time, such as the length of time to complete a survey.

Suggested Citation:"Practice 3: Use of Multiple Data Sources for Statistics That Meet User Needs." National Academies of Sciences, Engineering, and Medicine. 2017. Principles and Practices for a Federal Statistical Agency: Sixth Edition. Washington, DC: The National Academies Press. doi: 10.17226/24810.
×

in similar ways. Administrative records are designed to capture information for all instances of a specified population (e.g., program beneficiaries) according to a set of rules typically based in statute or regulation, and, like censuses, they may have omissions or duplications, and the variables in the records may differ in accuracy.45 The records may also be stored in difficult-to-use formats, not well documented, or not provided on a timely basis. Acquiring the records requires negotiations with the custodial agency, and their contents may change when program rules change. Yet efforts to develop error profiles for administrative records (see Practice 9) and productive relationships with the custodial agency (see Practice 7) can have sizable payoffs, as is evident in several well-established statistical uses of records.

For example, administrative records are used to generate up-to-date population estimates by age, race and ethnicity, and gender. In turn, these estimates are used to adjust population survey weights for coverage errors and for many other purposes.46 Tax records are used instead of questionnaires for the Census Bureau’s economic censuses and surveys for nonemployer businesses. Administrative records are more and more used with survey data to produce model-based estimates with improved accuracy for small geographic areas or population groups.47

There are many other statistical uses that agencies should consider for administrative records, for which the investment in data collection has already been made. In some instances, records could improve the cost-effectiveness and data quality of current statistical programs (e.g., by substituting administrative records for survey questions). In other instances, they could add richness to the combined dataset (e.g., by appending administrative records variables to matched survey records).48Box III.2 presents examples of innovative uses of administrative records for federal statistics. National Research Council (2009e) provides a comprehensive strategy for using administrative records to improve income information in the Census Bureau’s Survey of Income and Program Participation.

__________________

45 For example, payments to beneficiaries may be more accurate than information provided at the time of application regarding a beneficiary’s characteristics.

46 See, e.g., National Research Council (2004a, 2007b).

47 See, e.g., National Research Council (2000c,d), on the Census Bureau’s Small-Area Income and Poverty Estimates (SAIPE) Program and recommended improvements.

48 Extant matches include: (1) matches of Social Security earnings histories and Medicare benefits with the Health and Retirement Study and other surveys to analyze retirement decisions and the effect of medical care use on income security (see National Research Council, 1997a; National Research Council and Institute of Medicine, 2012); and (2) matches of employer and employee survey data with state employment security agency records in the Census Bureau’s Longitudinal Employer-Household Dynamics Program to analyze business and employment dynamics (see National Research Council, 2007a). Access to matched datasets must be restricted to protect confidentiality (see Practice 8).

Suggested Citation:"Practice 3: Use of Multiple Data Sources for Statistics That Meet User Needs." National Academies of Sciences, Engineering, and Medicine. 2017. Principles and Practices for a Federal Statistical Agency: Sixth Edition. Washington, DC: The National Academies Press. doi: 10.17226/24810.
×

Some uses may require not only state-of-the-art confidentiality protection techniques (see Practice 8), but also explicit legal authority (see “Toward the Paradigm of Multiple Data Sources,” below).49

ROLES FOR NONTRADITIONAL DATA SOURCES

Statistical agencies are currently exploring the use of data sources, other than surveys and administrative records, that hold promise to improve the

__________________

49 Statistics Canada, the national Canadian statistical agency, which has full access to administrative datasets, asks respondents to the Canadian Income Survey (and its predecessor) for permission to use income tax records in place of questions: see http://www23.statcan.gc.ca/imdb/p2SV.pl?Function=getSurvey&SDDS=5200 [April 2017].

Suggested Citation:"Practice 3: Use of Multiple Data Sources for Statistics That Meet User Needs." National Academies of Sciences, Engineering, and Medicine. 2017. Principles and Practices for a Federal Statistical Agency: Sixth Edition. Washington, DC: The National Academies Press. doi: 10.17226/24810.
×

relevance, accuracy, and timeliness of federal statistics. These nontraditional data sources include, among others, data gleaned from relevant Internet websites (e.g., price quotes or social media postings), extracted from sensors (e.g., traffic cameras), and obtained from the private sector (e.g., scanner data on consumer purchases). Often, these sources generate large volumes of data that require data mining and other computationally intensive techniques for extracting information (see National Research Council, 2008a, esp. App. H). 50

Some agencies are already using nontraditional data sources. For example, the Economic Research Service (in the U.S. Department of Agriculture) obtains expenditure data scanned by households from store receipts from a private vendor and has evaluated the quality of the data (Muth et al., 2016). The National Center for Health Statistics (NCHS), in its surveys of hospitals and other health care providers, obtains data from questionnaires, abstracts of samples of patient records, and providers’ electronic medical care claim records.51 See Box III.3 for other examples of current uses of nontraditional data sources.

Most nontraditional data sources present significant challenges to statistical agencies to evaluate the accuracy and error properties of the information. For example, harvesting website data to develop up-to-the-minute consumer price indexes52 may offer significant timeliness and cost savings compared with traditional methods, but it is not clear how to adjust these data for consumer expenditures that occur off-line so that they accurately represent the universe of purchases. More generally, information that is taken from the Internet cannot usually be described or evaluated according to either a probability survey paradigm or a rules-based administrative records paradigm. For example, people who post items to sell on an auction website do not comprise any specified population. Another challenge of nontraditional data is that statistical agencies lack control over the consistency over time or among vendors or sites, so that deciding to rely heavily on such data sources carries high risks of compromising key time series if a vendor or site ceases operation or there are marked changes in data content or population coverage.

Yet in an era when data users expect timeliness and when budgets are constrained, statistical agencies should actively explore means by

__________________

50 Such data are often referred to as “big data,” which are characterized by high volume, velocity, and variety and require new tools and methods to capture, curate, manage, and process in an efficient way. See https://unstats.un.org/bigdata/ and https://unstats.un.org/unsd/statcom/doc15/BG-BigData.pdf [April 2017].

51 See, for example, “What Does Participation in the NHCS [National Hospital Care Survey] Entail,” and related frequently asked questions at http://www.cdc.gov/nchs/nhcs/faq.htm [April 2017].

52 This is currently being done by the Billion Prices Project at the Massachusetts Institute of Technology; see http://bpp.mit.edu [April 2017].

Suggested Citation:"Practice 3: Use of Multiple Data Sources for Statistics That Meet User Needs." National Academies of Sciences, Engineering, and Medicine. 2017. Principles and Practices for a Federal Statistical Agency: Sixth Edition. Washington, DC: The National Academies Press. doi: 10.17226/24810.
×

which nontraditional data sources can contribute to their programs. Such means could include: (1) augmenting information obtained from traditional sources; (2) replacing information elements previously obtained from traditional sources; (3) providing earlier estimates that are later benchmarked with traditional sources; and (4) analyzing information streams to identify needed changes (e.g., in types of jobs, education majors) in statistical classifications and survey questions.

Just as more and more surveys use multiple data collection modes, so a growing number of statistical programs will likely benefit from using multiple data sources, including nontraditional sources. To garner acceptance by policy makers and the public, statistical agencies should invest resources in documentation and user training and education. Agencies may need to “wall off” data series that are derived from nontraditional sources by labeling them as experimental or for research use until their statistical characteristics can be fully understood. If it is not possible to evaluate a nontraditional source sufficiently to establish its quality and suitability for inclusion in a statistical program, then a statistical agency should not use the data, although it may assist users by informing them of the problems with the source.

Suggested Citation:"Practice 3: Use of Multiple Data Sources for Statistics That Meet User Needs." National Academies of Sciences, Engineering, and Medicine. 2017. Principles and Practices for a Federal Statistical Agency: Sixth Edition. Washington, DC: The National Academies Press. doi: 10.17226/24810.
×

INTEGRATION AND SYNCHRONIZATION OF DATA ACROSS AGENCIES

Statistical agencies that collect similar information should pursue integration of their microdata records for specified statistical uses as another way to improve data quality, develop new kinds of information, and increase cost-effectiveness. One cost-effective approach is for a large survey to provide the sampling frame and additional content for a smaller, more specialized survey. Currently, the National Health Interview Survey of NCHS serves this function for the Medical Expenditure Panel Survey of the Agency for Healthcare Research and Quality. Similarly, the American Community Survey serves this function for the National Survey of College Graduates that the Census Bureau conducts for the National Center for Science and Engineering Statistics (see National Research Council, 2008b).

Another cost-effective approach is to synchronize or harmonize similar data held by different agencies. For example, both the Bureau of Labor Statistics (BLS) and the Census Bureau maintain business establishment lists. The lists derive from different sources (state employment security records for BLS and a variety of sources, including federal income tax records, for the Census Bureau). Research has demonstrated that synchronization of the lists would improve the accuracy of the information and the coverage of business establishments in the United States (National Research Council, 2006b, 2007a).

A major step toward synchronization was taken in the Confidential Information Protection and Statistical Efficiency Act (CIPSEA) of 2002. The act authorized the synchronization of business data among the three principal statistical agencies that produce the nation’s key economic statistics—the Bureau of Economic Analysis (BEA), BLS, and the Census Bureau (see Appendix B). However, synchronization of business establishment lists between BLS and the Census Bureau cannot currently be done because the Census Bureau is prohibited by law (Title 26 of the U.S. Code) from sharing with BLS (or BEA) any tax information of businesses or individuals that it may acquire from the Internal Revenue Service (IRS), even for statistical purposes.53

TOWARD THE PARADIGM OF MULTIPLE DATA SOURCES

There have been several initiatives in the past decade to further the use of multiple data sources by statistical agencies. The Federal Committee on Statistical Methodology established a Subcommittee on

__________________

53 Efforts have been under way since CIPSEA was enacted to permit business data synchronization involving IRS records, but, to date, it has not occurred.

Suggested Citation:"Practice 3: Use of Multiple Data Sources for Statistics That Meet User Needs." National Academies of Sciences, Engineering, and Medicine. 2017. Principles and Practices for a Federal Statistical Agency: Sixth Edition. Washington, DC: The National Academies Press. doi: 10.17226/24810.
×

Administrative Records in 2008, subsequently renamed the Subcommittee on Administrative, Alternative, and Blended Data, which to date has developed examples and protocols for accessing, using, and evaluating administrative records (Federal Committee on Statistical Methodology, 2009, 2013). OMB guidance issued in 2014 states that federal administrative records as a matter of practice should be considered for federal statistics (U.S. Office of Management and Budget, 2014a; see also Appendix A). The United States participates in the United Nations Global Working Group on Big Data for Official Statistics, established in 2014.54 The bipartisan 15-member commission created by the Evidence-Based Policymaking Commission Act of 2016 will report in September 2017 on such topics as how to integrate administrative and survey data and make them available for research and evaluation while protecting privacy and confidentiality; how data infrastructure, database security, and statistical protocols should be modified toward these ends; and whether a federal clearinghouse should be created for government survey and administrative data.55

In moving to expand their use of administrative records or other nontraditional data sources together with surveys, statistical agencies should consider at least six factors in assessing the benefits and costs:

  1. the need for upfront investment to facilitate the most effective approach to acquisition and use of administrative records or another nontraditional data source, accompanied by estimates of the likely longer term cost savings or reductions in respondent burden (or both) and timeliness or data quality improvements (or both);
  2. the protocols and criteria to follow to ensure full understanding by the statistical agency of the properties of a specific nonsurvey data source (e.g., its population coverage, frequency of updating): see the quality frameworks discussed in Practice 9;
  3. changes to established processing and estimation methods required to incorporate a nonsurvey source to maximize quality and timeliness of the resulting estimates: see Practice 10;
  4. the means by which the confidentiality of linked or augmented datasets can be protected while allowing access for research purposes: see National Research Council (2005b) and Practice 8;
  5. the requirements for expanded documentation, user outreach, and user education to assure acceptance of the resulting estimates and understanding of their strengths and limitations: see Practice 4; and

__________________

54 See https://unstats.un.org/bigdata/ [April 2017].

55 See www.cep.gov [April 2017].

Suggested Citation:"Practice 3: Use of Multiple Data Sources for Statistics That Meet User Needs." National Academies of Sciences, Engineering, and Medicine. 2017. Principles and Practices for a Federal Statistical Agency: Sixth Edition. Washington, DC: The National Academies Press. doi: 10.17226/24810.
×
  1. the risks to the availability, consistency, and quality of statistical series if a provider discontinues or changes a nontraditional data source in significant ways and how those risks can be mitigated.

In the United States, a challenge that must first be addressed is the legality and feasibility of acquisition of data sources not already “owned” by a statistical agency. National Academies of Sciences, Engineering, and Medicine (2017b:1) concluded that “legal and administrative barriers limit the statistical use of administrative datasets by federal statistical agencies.” The problem is particularly acute for records for which states have custody; incentives (as some federal-state cooperative statistics programs provide) are likely needed for states to share their records for federal statistical purposes. For nontraditional data sources, such as scanner data and the like, the problems are more that such data vary greatly in their fitness for use in official statistics.

National Academies of Sciences, Engineering, and Medicine (2017b) recommended stepped-up efforts led by the Interagency Council on Statistical Policy to coordinate research to evaluate different nontraditional data sources, develop appropriate estimation methods based on combining data, develop quality metrics for combined estimates, implement state-of-the-art data privacy protection techniques, and similar matters (see Practice 13). The report also recommended (p. 2) that “a new entity or an existing entity should be designated to facilitate secure access to data for statistical purposes to enhance the quality of federal statistics.” The report argues that the current system—whereby individual statistical agencies must negotiate separately with other federal agencies, state agencies, and the private sector—is burdensome on all parties and precludes the necessary economies of scale for resolving the common challenges to realizing the potential of multiple data sources for statistics that meet user needs.

Suggested Citation:"Practice 3: Use of Multiple Data Sources for Statistics That Meet User Needs." National Academies of Sciences, Engineering, and Medicine. 2017. Principles and Practices for a Federal Statistical Agency: Sixth Edition. Washington, DC: The National Academies Press. doi: 10.17226/24810.
×

This page intentionally left blank.

Suggested Citation:"Practice 3: Use of Multiple Data Sources for Statistics That Meet User Needs." National Academies of Sciences, Engineering, and Medicine. 2017. Principles and Practices for a Federal Statistical Agency: Sixth Edition. Washington, DC: The National Academies Press. doi: 10.17226/24810.
×
Page 69
Suggested Citation:"Practice 3: Use of Multiple Data Sources for Statistics That Meet User Needs." National Academies of Sciences, Engineering, and Medicine. 2017. Principles and Practices for a Federal Statistical Agency: Sixth Edition. Washington, DC: The National Academies Press. doi: 10.17226/24810.
×
Page 70
Suggested Citation:"Practice 3: Use of Multiple Data Sources for Statistics That Meet User Needs." National Academies of Sciences, Engineering, and Medicine. 2017. Principles and Practices for a Federal Statistical Agency: Sixth Edition. Washington, DC: The National Academies Press. doi: 10.17226/24810.
×
Page 71
Suggested Citation:"Practice 3: Use of Multiple Data Sources for Statistics That Meet User Needs." National Academies of Sciences, Engineering, and Medicine. 2017. Principles and Practices for a Federal Statistical Agency: Sixth Edition. Washington, DC: The National Academies Press. doi: 10.17226/24810.
×
Page 72
Suggested Citation:"Practice 3: Use of Multiple Data Sources for Statistics That Meet User Needs." National Academies of Sciences, Engineering, and Medicine. 2017. Principles and Practices for a Federal Statistical Agency: Sixth Edition. Washington, DC: The National Academies Press. doi: 10.17226/24810.
×
Page 73
Suggested Citation:"Practice 3: Use of Multiple Data Sources for Statistics That Meet User Needs." National Academies of Sciences, Engineering, and Medicine. 2017. Principles and Practices for a Federal Statistical Agency: Sixth Edition. Washington, DC: The National Academies Press. doi: 10.17226/24810.
×
Page 74
Suggested Citation:"Practice 3: Use of Multiple Data Sources for Statistics That Meet User Needs." National Academies of Sciences, Engineering, and Medicine. 2017. Principles and Practices for a Federal Statistical Agency: Sixth Edition. Washington, DC: The National Academies Press. doi: 10.17226/24810.
×
Page 75
Suggested Citation:"Practice 3: Use of Multiple Data Sources for Statistics That Meet User Needs." National Academies of Sciences, Engineering, and Medicine. 2017. Principles and Practices for a Federal Statistical Agency: Sixth Edition. Washington, DC: The National Academies Press. doi: 10.17226/24810.
×
Page 76
Suggested Citation:"Practice 3: Use of Multiple Data Sources for Statistics That Meet User Needs." National Academies of Sciences, Engineering, and Medicine. 2017. Principles and Practices for a Federal Statistical Agency: Sixth Edition. Washington, DC: The National Academies Press. doi: 10.17226/24810.
×
Page 77
Suggested Citation:"Practice 3: Use of Multiple Data Sources for Statistics That Meet User Needs." National Academies of Sciences, Engineering, and Medicine. 2017. Principles and Practices for a Federal Statistical Agency: Sixth Edition. Washington, DC: The National Academies Press. doi: 10.17226/24810.
×
Page 78
Suggested Citation:"Practice 3: Use of Multiple Data Sources for Statistics That Meet User Needs." National Academies of Sciences, Engineering, and Medicine. 2017. Principles and Practices for a Federal Statistical Agency: Sixth Edition. Washington, DC: The National Academies Press. doi: 10.17226/24810.
×
Page 79
Suggested Citation:"Practice 3: Use of Multiple Data Sources for Statistics That Meet User Needs." National Academies of Sciences, Engineering, and Medicine. 2017. Principles and Practices for a Federal Statistical Agency: Sixth Edition. Washington, DC: The National Academies Press. doi: 10.17226/24810.
×
Page 80
Next: Practice 4: Openness about Sources and Limitations of the Data Provided »
Principles and Practices for a Federal Statistical Agency: Sixth Edition Get This Book
×
Buy Paperback | $54.00 Buy Ebook | $43.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Publicly available statistics from government agencies that are credible, relevant, accurate, and timely are essential for policy makers, individuals, households, businesses, academic institutions, and other organizations to make informed decisions. Even more, the effective operation of a democratic system of government depends on the unhindered flow of statistical information to its citizens.

In the United States, federal statistical agencies in cabinet departments and independent agencies are the governmental units whose principal function is to compile, analyze, and disseminate information for such statistical purposes as describing population characteristics and trends, planning and monitoring programs, and conducting research and evaluation. The work of these agencies is coordinated by the U.S. Office of Management and Budget. Statistical agencies may acquire information not only from surveys or censuses of people and organizations, but also from such sources as government administrative records, private-sector datasets, and Internet sources that are judged of suitable quality and relevance for statistical use. They may conduct analyses, but they do not advocate policies or take partisan positions. Statistical purposes for which they provide information relate to descriptions of groups and exclude any interest in or identification of an individual person, institution, or economic unit.

Four principles are fundamental for a federal statistical agency: relevance to policy issues, credibility among data users, trust among data providers, and independence from political and other undue external influence.Β Principles and Practices for a Federal Statistical Agency: Sixth Edition presents and comments on these principles as they’ve been impacted by changes in laws, regulations, and other aspects of the environment of federal statistical agencies over the past 4 years.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!