National Academies Press: OpenBook

Principles and Practices for a Federal Statistical Agency: Sixth Edition (2017)

Chapter: Practice 4: Openness about Sources and Limitations of the Data Provided

« Previous: Practice 3: Use of Multiple Data Sources for Statistics That Meet User Needs
Suggested Citation:"Practice 4: Openness about Sources and Limitations of the Data Provided." National Academies of Sciences, Engineering, and Medicine. 2017. Principles and Practices for a Federal Statistical Agency: Sixth Edition. Washington, DC: The National Academies Press. doi: 10.17226/24810.
×

Practice 4: Openness about Sources and Limitations of the Data Provided

A STATISTICAL AGENCY SHOULD BE OPEN about the strengths and limitations of its data, taking as much care to understand and explain how its statistics may fall short of accuracy as it does to produce accurate data. Metadata, or “data about data,” are a critical element of data dissemination and curation (see Practice 5). All data contain some uncertainty and error, which does not mean the data are wrong, but that they need to be used with understanding of their limitations.56

Openness requires that data releases from a statistical program include a full description of the purpose of the program; the methods and assumptions used for data collection, processing, and estimation; what is known and not known about the quality and relevance of the data; sufficient information for estimating variability and other errors in the data, when possible; appropriate analysis methods that take account of variability and other sources of error; and the results of research on the methods and data. Openness also means that a statistical agency should describe how decisions on methods and procedures were made for a data collection program and provide ready access to research results that entered into such decisions. Such transparency is essential for credibility with data users and trust of data providers.

Openness about data limitations requires much more than providing estimates of sampling error for surveys or basic attributes for administrative

__________________

56Manski (2015) points to a need for fuller measurement and communication of uncertainty in official statistics.

Suggested Citation:"Practice 4: Openness about Sources and Limitations of the Data Provided." National Academies of Sciences, Engineering, and Medicine. 2017. Principles and Practices for a Federal Statistical Agency: Sixth Edition. Washington, DC: The National Academies Press. doi: 10.17226/24810.
×

records or other nonsurvey data sources. In addition to a discussion of aspects that statisticians characterize as nonsampling errors—such as coverage errors, nonresponse errors, measurement errors, and processing errors—it is valuable to have a description of the concepts used and how they relate to the major uses of the data. Descriptions of the shortcomings of and problems with the data should be provided in sufficient detail to permit a user to take them into account in analysis and interpretation. Descriptions of how the data relate to similar data collected by other agencies should also be provided, particularly when the estimates from two or more surveys or other data sources exhibit large differences that may have policy implications.

On occasion, the objective of presenting the most accurate data possible may conflict with user needs for timely information. When concerns for timeliness prompt the release of preliminary estimates (as is done for some economic indicators), consideration should be given to the frequency of revisions and the mode of presentation from the point of view of the users as well as the issuers of the data. Agencies that release preliminary estimates need to educate the public about differences among preliminary, revised, and final estimates.

To meet their responsibility to users for openness, some statistical agencies in the 1990s developed detailed “quality profiles” for major surveys, including the American Housing Survey (Chakrabarty and Torres, 1996); the Residential Energy Consumption Survey (Energy Information Administration, 1996); the Schools and Staffing Survey (Kalton et al., 2000); and the Survey of Income and Program Participation (U.S. Census Bureau, 1998). Previously the Federal Committee on Statistical Methodology (1978) developed a quality profile for employment as measured in the Current Population Survey. These profiles documented what was and was not known about errors in estimates as a help to experienced users and agency personnel (see Federal Committee on Statistical Methodology, 2001; National Research Council, 1993a, 2007b). As print publications, however, they quickly became outdated and were rarely updated given the burden on agency staff. Today, the Internet enables easier maintenance and updating of quality profile-type information (e.g., separate web pages for major types of error).57

__________________

57 The Census Bureau posts basic quality indicators for the American Community Survey for the nation and states; these include sample size, population coverage, household response rates, and item response rates. See https://www.census.gov/acs/www/methodology/sample_size_and_data_quality/ [April 2017]. The Bureau of Labor Statistics issued a prototype data quality report for the Consumer Expenditure Quarterly and Diary Surveys, which it characterized as the “first in a series of iterations towards a single reference source on a comprehensive set of CE data quality metrics that are timely and routinely updated.” The metrics provided in the prototype report refer to data for 2009–2013. Available: https://www.bls.gov/cex/ce_methodology.htm#dqreports0 [April 2017].

Suggested Citation:"Practice 4: Openness about Sources and Limitations of the Data Provided." National Academies of Sciences, Engineering, and Medicine. 2017. Principles and Practices for a Federal Statistical Agency: Sixth Edition. Washington, DC: The National Academies Press. doi: 10.17226/24810.
×

Error (conversely, “accuracy”) is not the only dimension of quality of concern to statistical agencies and their data users. Building on a seminal paper by Brackstone (1999), many statistical agencies around the world have adopted “quality frameworks,” which are typologies of key attributes or dimensions to use in systematically measuring, improving, and documenting data quality. For example, the Eurostat (2000) framework includes relevance, accuracy, timeliness and punctuality, accessibility and clarity, comparability (across time and geography), and consistency (with other series). Biemer et al. (2014) decompose “accuracy” into sampling error (where applicable) and seven types of nonsampling error. Daas et al. (2012) and Federal Committee on Statistical Methodology (2013) address quality attributes for administrative records-based data series.

In the United States, the Information Quality Act of 2000 required all federal agencies to develop written guidelines for how they ensure the quality of the information they disseminate to the public. Using a framework developed by the Interagency Council on Statistical Policy, individual statistical agencies developed quality guidelines (see Practice 9 and Appendix A). However, the guidelines, in response to the legislation, are process oriented (e.g., indicating how to request correction of a datum) and are not quality frameworks as described above. Federal statistical agencies should consider adopting and implementing quality frameworks for their programs.

An important aspect of openness not addressed in quality frameworks concerns the treatment of mistakes that are discovered subsequent to data release. Openness means that an agency has an obligation to issue corrections publicly and in a timely manner. The agency should use not only the same dissemination avenues to announce corrections that it used to release the original statistics, but also additional vehicles, as appropriate, to alert the widest possible audience of current and future users of the corrections in the information. Agencies should be proactive in seeking ways to alert known and likely users of the data about the nature of a problem and the corrective action that it is taking or that users should take.

Overall, agencies should treat the effort to provide information on the quality, limitations, and appropriate use of their data as an essential part of their mission. Such information should be made available in ways that are readily accessible to all known and potential users (see National Research Council, 1993a, 1997b, 2007b).

Suggested Citation:"Practice 4: Openness about Sources and Limitations of the Data Provided." National Academies of Sciences, Engineering, and Medicine. 2017. Principles and Practices for a Federal Statistical Agency: Sixth Edition. Washington, DC: The National Academies Press. doi: 10.17226/24810.
×

This page intentionally left blank.

Suggested Citation:"Practice 4: Openness about Sources and Limitations of the Data Provided." National Academies of Sciences, Engineering, and Medicine. 2017. Principles and Practices for a Federal Statistical Agency: Sixth Edition. Washington, DC: The National Academies Press. doi: 10.17226/24810.
×
Page 81
Suggested Citation:"Practice 4: Openness about Sources and Limitations of the Data Provided." National Academies of Sciences, Engineering, and Medicine. 2017. Principles and Practices for a Federal Statistical Agency: Sixth Edition. Washington, DC: The National Academies Press. doi: 10.17226/24810.
×
Page 82
Suggested Citation:"Practice 4: Openness about Sources and Limitations of the Data Provided." National Academies of Sciences, Engineering, and Medicine. 2017. Principles and Practices for a Federal Statistical Agency: Sixth Edition. Washington, DC: The National Academies Press. doi: 10.17226/24810.
×
Page 83
Suggested Citation:"Practice 4: Openness about Sources and Limitations of the Data Provided." National Academies of Sciences, Engineering, and Medicine. 2017. Principles and Practices for a Federal Statistical Agency: Sixth Edition. Washington, DC: The National Academies Press. doi: 10.17226/24810.
×
Page 84
Next: Practice 5: Wide Dissemination of Data »
Principles and Practices for a Federal Statistical Agency: Sixth Edition Get This Book
×
Buy Paperback | $54.00 Buy Ebook | $43.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Publicly available statistics from government agencies that are credible, relevant, accurate, and timely are essential for policy makers, individuals, households, businesses, academic institutions, and other organizations to make informed decisions. Even more, the effective operation of a democratic system of government depends on the unhindered flow of statistical information to its citizens.

In the United States, federal statistical agencies in cabinet departments and independent agencies are the governmental units whose principal function is to compile, analyze, and disseminate information for such statistical purposes as describing population characteristics and trends, planning and monitoring programs, and conducting research and evaluation. The work of these agencies is coordinated by the U.S. Office of Management and Budget. Statistical agencies may acquire information not only from surveys or censuses of people and organizations, but also from such sources as government administrative records, private-sector datasets, and Internet sources that are judged of suitable quality and relevance for statistical use. They may conduct analyses, but they do not advocate policies or take partisan positions. Statistical purposes for which they provide information relate to descriptions of groups and exclude any interest in or identification of an individual person, institution, or economic unit.

Four principles are fundamental for a federal statistical agency: relevance to policy issues, credibility among data users, trust among data providers, and independence from political and other undue external influence.Β Principles and Practices for a Federal Statistical Agency: Sixth Edition presents and comments on these principles as they’ve been impacted by changes in laws, regulations, and other aspects of the environment of federal statistical agencies over the past 4 years.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!