Strengthening the Nation’s Data Infrastructure
The National Academies offer expert advice for federal statistical agencies and others who create, support, and use our nation’s data infrastructure
Data Infrastructure and Why It Matters
Data infrastructure refers to systems that hold, link, and support access and analysis of data. Our nation’s data infrastructure underpins our national statistics, making it a vital resource for informing decision making and public policy.
13 Principal Statistical Agencies
- Bureau of Economic Analysis (U.S. Department of Commerce)
- Bureau of Justice Statistics (U.S. Department of Justice)
- Bureau of Labor Statistics (U.S. Department of Labor)
- Bureau of Transportation Statistics (U.S. Department of Transportation)
- Census Bureau (U.S. Department of Commerce)
- Economic Research Service (U.S. Department of Agriculture)
- Energy Information Administration (U.S. Department of Energy)
- National Agricultural Statistics Service (U.S. Department of Agriculture)
- National Center for Education Statistics (U.S. Department of Education)
- National Center for Health Statistics (U.S. Department of Health and Human Services)
- National Center for Science and Engineering Statistics (National Science Foundation)
- Office of Research, Evaluation, and Statistics (Social Security Administration)
- Statistics of Income Division (U.S. Department of the Treasury)
Test Your Knowledge: Concepts & Terms
Select a term to learn more about different data types.
- Statistics
Statistics is the science of collecting, analyzing, and interpreting data. The products of data analysis are also referred to as statistics.
- Structured data
Structured data is information that is in a standardized format following a pre-defined model. Examples include postal addresses, social security numbers, and survey data.
- Nonstructured data
Nonstructured (or unstructured) data is information that is not organized according to a pre-defined model. Examples include audio recordings and PDFs.
- Survey data
Survey data is information collected scientifically, typically using standardized questionnaires and from a sample that can be considered to represent a broader population. Surveys are a primary source of data for producing national statistics; the national census is one example.
- Administrative data
Administrative data is information collected by a government body as part of routine operations (but not explicitly for statistical purposes). Examples include tax records and vital records.
- Private-sector data
Private-sector data is information collected or generated by commercial and other nongovernmental entities. Examples include social media activity and store scanner data.
Credible statistical information is foundational to the functioning of democratic societies.
Toward a 21st Century National Data Infrastructure: Mobilizing Information for the Common Good
Data Sources
Survey Data
Surveys have traditionally been the cornerstone of national statistical data. Federal surveys collect information on the economy, health, crime, agriculture, education, and many other facets of life in America.
Bottom line:
Experts see a continuing role for surveys in national statistics, but the landscape has changed.
Administrative Data
Administrative data—collected by government entities as part of their routine operations—exist at local, state, and federal levels. Examples of administrative data include tax records, vital records, criminal justice records, and information from Medicaid, Medicare, and nutritional assistance programs.
Bottom line:
Although they are not collected specifically for statistical purposes, administrative data can be useful for enhancing national statistical data and processes when thoughtfully integrated and properly vetted for fitness for use.
Private-Sector Data
Private-sector data are collected or generated by commercial entities. Examples include data obtained from websites and social media platforms, transactional and store scanner data, and data collected by companies such as insurers and service providers.
Bottom line:
Statistical agencies can potentially partner with commercial entities to utilize data generated in the private sector, but special care is needed to mitigate the drawbacks.
Survey response rates, including in government surveys, have been falling, raising concerns about costs and the ability of the statistical system to rely solely on survey data in the future. In this context, nonsurvey data sources are being increasingly called on to fill the void.
Measuring Alternative Work Arrangements for Research and Policy
Using Data
Assessing Data Quality
As agencies work to modernize data collection processes, it will be important to study the validity and reliability of new data sources and methods. Frameworks and coordination are needed to assess data quality and determine when alternative data sources are acceptable for use in producing national statistics.
- Related Reports
- Toward a 21st Century National Data Infrastructure: Mobilizing Information for the Common Good outlines dimensions of data quality and posits that a new framework of quality standards is needed to evaluate the fitness for use of multiple data sources.
- A Consumer Food Data System for 2030 and Beyond provides recommendations for assessing the quality of commercial data.
- In Improving the American Community Survey: Proceedings of a Workshop, practitioners articulate the importance of data quality as a primary goal and suggest a need for criteria for acceptance of alternative data sources.
Combining Multiple Sources of Data
Integrating multiple sources of survey, administrative, and private-sector data can enhance the content, timeliness, and granularity of national statistical data. However, this represents a significant undertaking and requires a systematic approach.
- Related Reports
- Toward a 21st Century National Data Infrastructure: Mobilizing Information for the Common Good discusses how blended data fits within a new national data infrastructure and the new statistical methods needed to enable this.
- Toward a 21st Century Data Infrastructure: Enhancing Survey Programs by Using Multiple Data Sources identifies ways that combining data from different sources can improve data quality and provide new resources for social and economic research, and considers examples in income, health, crime, and agriculture.
- Innovations in Federal Statistics: Combining Data Sources While Protecting Privacy presents a strategy for advancing the paradigm of combining data sources in national statistics.
- Measuring Alternative Work Arrangements for Research and Policy discusses the longer-term promise of combining data sources in the context of labor market statistics.
- Improving Crop Estimates by Integrating Multiple Data Sources addresses challenges and opportunities in integrating multiple data sources for developing county-level crop estimates.
Strengthening Stakeholder Partnerships
Effective partnerships are foundational to facilitating access to various data sources and gaining the context necessary to assess their quality and applicability for national statistics. In addition, transparency and effective communication are vital to ensuring the responsible collection, sharing, and use of data.
- Related Reports
- Toward a 21st Century National Data Infrastructure: Mobilizing Information for the Common Good highlights the importance of convening stakeholders to articulate how combined data sources can be used responsibly and notes the need for new partnerships to facilitate the use of private-sector data.
- A Vision and Roadmap for Education Statistics outlines the need for statistical agencies to engage with each other to facilitate data linkage and access and to engage with the users of their data products to gather feedback and better meet their needs.
Advancing Data Equity
Equity is an essential consideration for any data system. Gaps in coverage, disparities in quality, and misrepresentation can lead to inaccurate data and information. Further, people should not be harmed through the collection and dissemination of their data. A variety of approaches can enhance equity with regard to what types of data are collected, from whom, and how data are accessed, interpreted, and used.
- Related Reports
- Toward a 21st Century Data Infrastructure: Enhancing Survey Programs by Using Multiple Data Sources examines how different data sources can be used to enhance data equity.
- A Vision and Roadmap for Education Statistics discusses how statistical agencies can reflect the lived experiences of diverse populations and provide more accurate data in order to help inform understanding of social issues.
Protecting Privacy
Privacy protection is a fundamental requirement for the nation’s data infrastructure. The usefulness of the information in the infrastructure depends upon the willingness of persons and entities to allow their data to be included. They must trust that their data will be used only for statistical purposes, and that they will not be harmed by anyone accessing their information. Data holders have a variety of technical tools and policy approaches that can be used in combination for effective management of risks of disclosure.
- Related Reports
- Toward a 21st Century National Data Infrastructure: Managing Privacy and Confidentiality Risks with Blended Data provides a model framework for making decisions about data protection strategies for blended data and promote careful consideration of key questions.
- Federal Statistics, Multiple Data Sources, and Privacy Protection: Next Steps describes approaches for preserving privacy through technology, statistical methods, and administrative procedures.
- Principles and Practices for a Federal Statistical Agency explicitly includes a principle focused on public trust and a practice devoted to respecting data providers (respondents and subjects) and protecting their data.
Charting a Path Forward
What does the data infrastructure of the future look like? What steps are needed to effectively transition our nation’s statistical systems to meet the challenges of a changing world? Our reports outline concrete steps toward closing gaps and supporting the country’s data needs.
- Related Reports
- Toward a 21st Century National Data Infrastructure: Mobilizing Information for the Common Good identifies seven attributes of a modern national data infrastructure and short- and medium-term activities to achieve this vision.
- Federal Statistics, Multiple Data Sources, and Privacy Protection: Next Steps outlines the need for an entity dedicated to facilitating secure access to data for statistical purposes and the features and functions of such an entity.
- A Consumer Food Data System for 2030 and Beyond identifies features of a successful system including comprehensiveness, representativeness, timeliness, openness, flexibility, accuracy, suitability, and fiscal responsibility.
The effectiveness of the federal statistical system to meet future data demands will largely depend on the extent to which data sources—survey and nonsurvey, national and local, public and private—can be combined in synergistic ways.
Navigating Change
Closing Gaps
Strategies for collecting data that have worked in the past are not working as well now. This creates gaps in the information collected and our ability to compare data over time. New types of data are needed to answer today’s pressing questions.
Modernizing Methods
We now live, work, learn, eat, and shop in very different ways than we did even a short time ago. How we measure those things needs to change, too. New data sources and analytical methods have great potential to modernize data collection, but it’s important to use them carefully.
Doing It Right
It is imperative to attend to privacy, ethics, and equity as we work to improve our nation’s data infrastructure. Transparency and communication are essential to fostering trust among data generators, users, and the public.