3
Changes in Archiving Practices to Improve Transparency
This chapter explores the relationship between archiving and transparency, and then compares current practices with recommended best practices. Archiving means preserving objects. That implies both selecting which objects will be preserved and which will not and taking actions to maintain the possibility of access to those objects that are preserved. Storage alone is not necessarily preservation, and this is especially true in an age in which many objects are “born digital.”
The National Archives and Records Administration (NARA) is focused on preservation and therefore is an archive in the classic meaning of the term. NARA takes physical custody of government records and then strives to preserve them, according to stated goals, strategies, and methods for physical1 and digital2 records. These records have already been subject to decisions about whether or not they should be archived, in the form of “records (control) schedules.” A government agency is legally obligated to “manage” records—that is, to make decisions about creating and preserving such records—in order to document the functioning and decision making of the agency (44 U.S.C. 3101–3107).3 Records may be “disposed” (destroyed) as well, but such decisions are captured in the records schedule, of which NARA keeps a copy in a searchable archive (https://www.archives.gov/records-mgmt/rcs). NARA is the agency that decides what to
___________________
1https://www.archives.gov/preservation/storage/specs-housing-exhibition-2015-current.html.
2https://www.archives.gov/preservation/electronic-records/digital-preservation-strategy.
3 Records Management by Federal Agencies (44 U.S.C. Chapter 31): https://www.archives.gov/about/laws/fed-agencies.html.
preserve and for how long, subject to guidance4 and review by the archivists of NARA. NARA provides access to its archives through the National Archives Catalog5 and its archival databases.6
TRANSPARENCY AND ARCHIVES
Archives increase the transparency of federal statistics by providing third parties (or the statistical agencies themselves at another point in time) with the records of what statistical agencies have collected and how they have used what was collected to produce statistical estimates. It is of course equally important that there be transparency prior to and during the data collection process, well before any long-term preservation in an archive (see relevant sections in Chapter 7, Tables 7-1 through 7-6). Archiving also facilitates improvements in statistics, particularly in a system as decentralized as the U.S. federal statistical system. This is because other agencies (and external users) can learn from successful innovations as well as the challenges faced by others, but this learning occurs only if the other agencies can find and analyze what has been done. A transparent policy about what is preserved—and what is not preserved—is important as well for confidence in the statistical system. As Office of Management and Budget (OMB) memo M-12-187 states:
Records are the foundation of open government, supporting the principles of transparency, participation, and collaboration. Well-managed records can be used to assess the impact of programs, to improve business processes, and to share knowledge across the Government. Records protect the rights and interests of people, and hold officials accountable for their actions. Permanent records document our nation’s history. (p. 1)
For the federal statistical system, transparency regarding the key elements of both the process and the results of data collection is critical to public confidence. Transparency is thus also critical to public participation in and provision of information to the federal statistical system, improving data quality and reducing the costs of data collection. To provide this transparency, it is necessary to archive—to preserve and make accessible—the full data life cycle, including questionnaires, metadata about the data collection process, metadata about the transformation of raw data into data products, and the data products themselves. Archiving allows us to learn from past practices and to put current data into context by understanding
___________________
4https://www.archives.gov/preservation/electronic-records/digital-preservation-guidance.
5https://www.archives.gov/research/catalog.
6https://aad.archives.gov/aad/.
7 OMB Memorandum M-12-18, https://www.archives.gov/files/records-mgmt/m-12-18.pdf.
those practices. But more importantly, archiving can serve to enhance trust and confidence in the federal statistical system.
More directly relevant to the question of the transparency of federal statistics is the 2019 OMB memorandum on the Transition to Electronic Records, which requires all federal agencies (not just statistical agencies) to “Ensure that all Federal records are created, retained, and managed in electronic formats, with appropriate metadata.”8
There are general principles, codified in legislation, regulations, and policies of the United States, as well as the United Nations and the Organisation for Economic Co-operation and Development (OECD), that guide and provide some specificity about how and what federal statistics should be archived.9 The Fundamental Principles of Official Statistics were adopted in 1992 by the United Nations Economic Commission for Europe and subsequently were endorsed as a global standard by the United Nations Statistical Commission. The first three of these principles lay out some of the criteria of transparency, access, and accountability (italic emphasis added):
Principle 1. Official statistics provide an indispensable element in the information system of a democratic society, serving the Government, the economy and the public with data about the economic, demographic, social and environmental situation. To this end, official statistics that meet the test of practical utility are to be compiled and made available on an impartial basis by official statistical agencies to honour citizens’ entitlement to public information.
Principle 2. To retain trust in official statistics, the statistical agencies need to decide according to strictly professional considerations, including scientific principles and professional ethics, on the methods and procedures for the collection, processing, storage and presentation of statistical data.
Principle 3. To facilitate a correct interpretation of the data, the statistical agencies are to present information according to scientific standards on the sources, methods and procedures of the statistics. (p. 2)10
The FAIR (Findable, Accessible, Interoperable, and Reusable) principles provide further guidance. The FAIR principles focus on making data products accessible and useful, but do not specifically address questions of preservation or providing the metadata necessary to evaluate data quality
___________________
8https://www.whitehouse.gov/wp-content/uploads/2019/06/M-19-21.pdf.
9 According to the OECD, “Official statistics are statistics disseminated by the national statistical system, excepting those that are explicitly stated not to be official.” See https://stats.oecd.org/glossary/search.asp.
(as opposed to, for example, the metadata necessary for data to be reusable by a third party or to be interoperable with other data resources). Data accessibility is fundamental to public confidence, including the public’s willingness to invest in federal statistics. For the transparency of federal statistics, the FAIR principles are a starting place.11
ARCHIVING HISTORY AND PRACTICES
A high-quality archive (or archives) is one that meets standards for best practices for preservation, discoverability, and accessibility—of all its data products, including microdata, aggregates, and indicators.12 Thousands of official depositories at libraries around the country previously provided such archiving for public-use federal statistics, that is, aggregates and indicators. These official depositories made it possible for people across the country to reconstruct what other people believed or knew at a particular time. They also meant that statistical agencies did not consider comprehensive and consistent archiving as their responsibility. Those government document libraries largely no longer exist or are not maintained.13
In today’s virtual world, it should be relatively easy and inexpensive to preserve data and make them accessible. Appropriate and sustainable institutions are required to do what the official repositories previously did. Because digital data can be altered in ways that are not transparent, it is now often not possible to find out precisely what was published at a particular point in time. More generally, there is no systematic preservation of the public-use digital data products of the federal statistical system. What is preserved is ad hoc and decentralized, and it is often not discoverable by the public. So, while FAIR may be a minimal standard for the transparency of federal statistics, it is not yet being met.
Records (control) schedules (see below) obligate federal agencies, including statistical agencies, to make decisions about the preservation of certain data products (44 U.S.C. 3101 and 3103). The schedules are reviewed in conjunction with archivists at NARA, a “disposition authority number” is assigned, and disposal (destruction) or preservation then follows that
___________________
11 See https://www.go-fair.org/fair-principles/ and http://www.nature.com/articles/sdata201618. See also the White House Office of Science and Technology Policy memorandum, Increasing Access to the Results of Federally Funded Scientific Research, http://www.whitehouse.gov/sites/default/files/microsites/ostp/ostp_public_access_memo_2013.pdf, which calls for preservation and access to federally funded research data products, to increase their transparency as well as the efficiency of the federal research enterprise.
12 Best practice standards for archiving are captured by the Core Trust Seal: https://www.coretrustseal.org/.
13 See for instance the work put into collecting data for the Bracero program, https://doi.org/10.7910/DVN/DJHVHB.
schedule. Historically, records schedules were poorly monitored, and many of the preservation decisions still appear to be focused on final output products, not necessarily on materials that are of value to research and future activities of statistical agencies, such as metadata and paradata. However, recently there has been an effort to better organize and review records schedules, including searchable catalogs of records schedules and the preserved assets and databases at NARA14 (see also the discussion below on findability), and the incorporation of records schedule principles into the regular training of federal employees at all levels. Because more data can be more easily captured today, a more expansive view of what constitutes a “record” (44 U.S.C. 3301) might allow for greater preservation of metadata and paradata, which have been traditionally either ignored or identified as “temporary” data, to be destroyed after only a brief delay.
A more complicated question is whether it is important to archive the underlying raw data and programs that are used to produce published statistics. Many recent examples of research have leveraged deep archives of historical data to obtain new insights. For example, the Census Bureau still has the responses to most of the Decennial Censuses, most but not all of which have been digitized. When made public after 72 years, these data have become the most used Census Bureau products ever, sustaining businesses large and small in the family history (genealogy) industry. Social, economic, and methodological research has also used the underlying raw data, even before they are made public. For example, the Decennial Census Digitization and Linkage Project is using the underlying microdata files from the Decennial Censuses between 1940 and 2010 to create a longitudinal data resource that will provide a statistical infrastructure for studying the U.S. population in heretofore unimagined ways (e.g., see Genadek and Alexander, 2019; Genadek et al., 2018).
Research can leverage such raw data, even when it is not digitized. Robert Fogel and Douglas North won the Nobel Memorial Prize in economics for their work analyzing data from federal statistical agencies that had been archived and that they digitized, transforming our understanding of American history and economic development. Various research teams have used data originally preserved by the National Archives, but subsequently (partially) digitized by nongovernmental parties (Ancestry.com, Church of Latter Day Saints, and the IPUMS [Integrated Public Use Microdata Series] project to look into intergenerational linkages (Abramitzky et al., 2019). Research into reclassification errors—see Box 3-1 for a recent example—and how they might affect official or other statistics using the microdata
___________________
14 Record control schedules: https://www.archives.gov/records-mgmt/rcs/schedules/index.html;databases: https://aad.archives.gov/aad/; search for preserved assets: https://aad.archives.gov/aad/.
have been conducted in the past, using both public microdata (Hirsch and Schumacher, 2004) and confidential (raw) survey data (Larrimore et al., 2008).
The National Archives has Manufacturing Censuses from 1929, 1931, 1933, 1935, and 1937 that have been the basis for valuable research, as were the manufacturing and agricultural schedules collected by the U.S. Census Bureau.15 Some have been digitized and are available through the
___________________
15 Examples include Raff (1988); Bertin et al. (1996); and Benguria, Vickers, and Ziebarth (2020).
Inter-university Consortium for Political and Social Research (ICPSR).16 More recent U.S. economic censuses and surveys are still held by the Census Bureau and made available to the research community through the FSRDC.17
The lack of systematic documentation and preservation of digital data continues to create challenges and increase costs—and result in the loss of valuable data. For example, when the U.S. Census Bureau was retiring VAX machines in the early 2000s, it discovered hundreds of data files from surveys undertaken in the 1960s and 1970s for which there was little or no documentation. With the assistance of IPUMS and ICPSR, and leadership from the Census Bureau, these files were preserved, and handwritten documentation was digitized, but much of these data still remain unusable.
Archiving might mean preserving and making accessible the administrative data, including tax data, which are used to produce the Economic Census. Transparency would require that there be public-use documentation of the provenance of the underlying data. Archiving would mean preserving the survey data, the administrative data, and the processes that were used to integrate the two sources of data to create the public estimates. Even in cases where the underlying microdata could not be made publicly available, the legitimacy of the resulting public estimates would be increased if the process by which they were produced and the way multiple data sources were integrated were publicly available.
Becker (2015) and Becker and Grim (2011) report on efforts (and difficulties) to recover usable microdata from the 1950s, 1960s, and 1970s that had not been preserved in the National Archives.18 They encountered incomplete nondigital metadata on the digital data, failing last-of-its-kind hardware, and unique file formats—all issues that a proper archive with preservation and curation strategies is designed to address. Note that in addition, identifying the precise strategies used to recover the data increased confidence that the analytical results in the research were relevant and meaningful, a consequence of transparency.
Basker et al. (2019) reports on how the historical findings led to new questions concerning contemporaneous surveys. Four new lines of inquiry were added to the 2017 Economic Census regarding (1) retail health clinics,
___________________
16 Examples of ICPSR projects using Census of Manufacturing data from the 1930s include Vickers and Zeibarth (2018), available at https://www.icpsr.umich.edu/web/ICPSR/studies/37114/versions/V1; and Bresnahan and Raff (2018), available at https://doi.org/10.3886/ICPSR37208.v1. An example of earlier digitized Census of Manufacturing (1850–1870) data was done by Atack, Bateman, and Weiss (2006); see https://doi.org/10.3886/ICPSR04048.v1.
17 U.S. Census Bureau microdata from its economic censuses and surveys that are available for research access are described at https://www.icpsr.umich.edu/web/pages/appfed/index.html.
18 Additional information is available in the Center for Economic Studies Annual Report for 2009, Ch. 4 (https://www2.census.gov/library/publications/2010/adrm/ces/2009-researchreport.pdf).
(2) management practices in health care services, (3) self-service in retail and service industries, and (4) water use in manufacturing and mining industries. These were proposed by economists from the Census Bureau’s Center for Economic Studies to fill data gaps in current Census Bureau products concerning the U.S. economy. The new content addresses such issues as the rise in importance of health care and its complexity, the adoption of automation technologies, and the importance of measuring water, a critical input to many manufacturing and mining industries.
Today the Census Bureau and other statistical agencies rely on administrative data that are not obtained through surveys and are covered by different laws (for example, the Family Educational Rights and Privacy Act and Title 26 of the IRS Act) that may exempt data from being archived at NARA and may sometimes even prohibit it. Modern surveys often use digital-only survey forms, which may be both adaptive in their structure and dynamic over time, but with no printed “static” equivalent form. These surveys create novel and sometimes unresolved challenges to an agency’s ability to archive and preserve its raw input data, processes, and methods.
CURRENT PRACTICES WITH RECORD SCHEDULES AND DATA MANAGEMENT PLANS
As mentioned previously in this report, and as is well known, currently the great majority of the input datasets for official statistics in the United States are either survey based or administrative records based (often from tax data, which are extremely sensitive). These input data sources contain personally identifiable information, and as such can only be made available to the public in an electronically secure environment, such as a federal statistical research data center. In those cases where the input data must remain confidential, what is archived through use of metadata is a description of the variables that the datasets contain and how those datasets are structured.
Identifying which files to preserve, in what formats, and for how long is critical for overall transparency. Ideally, these decisions are made when data are generated, not after the fact. Such documents—defining the how, why, and for how long—exist in a variety of formats. We describe here two dominant examples: Data Management Plans (DMPs), used primarily for researcher-generated data, and records schedules, used by U.S. government agencies. They are similar in scope, albeit not in the detail captured.
Data Management Plans
A DMP is a knowledge management document, prepared initially as a specific research or survey project is being planned, to lay out types of data
to be collected, the possible presence of sensitive data, the roles of project members in relation to the data, and the planned archiving and preservation of the data. A DMP is a living document that may change many times over the course of the research or survey project. All changes in a DMP need to be documented and retained.19 Federal research funders have required such DMPs from certain disciplines for some time, but are increasingly expanding the scope of such requirements.
The Proposal and Award Policies and Procedures Guide20 of the National Science Foundation (NSF) notes a requirement for (two-page) DMPs but only as a supplementary material, leaving the evaluation thereof in the realm of individual divisions and program managers. In 2019, NSF encouraged, but did not mandate, the use of DMPs.21 Discipline-specific criteria can be more tightly enforced. The Interdisciplinary Earth Data Alliance offers its tool to provide proof of compliance with NSF Data Policies. The NSF Directorate for Biological Sciences22 notes that it conducts post-award monitoring of compliance through annual progress reports.
In October 2020, the National Institutes of Health (NIH) issued its Final NIH Policy for Data Management and Sharing (effective January 2023), which “promote[s] the management and sharing of scientific data generated from NIH-funded or conducted research.”23 Guidance on creating DMPs is provided by numerous entities,24 and various online tools exist to assist researchers in crafting DMPs.25
Records Schedules
All U.S. government agencies are required to maintain “records schedules.” All federal records, including those created or maintained for the government by a contractor, must be covered by a NARA-approved agency disposition authority SF 115, Request for Records Disposition Authority, or the NARA General Records Schedules (36 CFR § 1225.10). General Records Schedules (GRS) are schedules issued by the Archivist of
___________________
19 NIH defines a Data Management and Sharing Plan as a “plan describing how scientific data will be managed, preserved, and shared with others (e.g., researchers, institutions, the broader public), as appropriate.” (84 FR 60398) https://grants.nih.gov/grants/guide/noticefiles/NOT-OD-21-014.html.
20https://www.nsf.gov/publications/pub_summ.jsp?ods_key=pappg&WT.z_pims_id=0.
21https://www.nsf.gov/pubs/2019/nsf19069/nsf19069.jsp.
22https://www.nsf.gov/bio/pubs/BIODMP061511.pdf.
23https://grants.nih.gov/grants/guide/notice-files/NOT-OD-21-013.html.
24 See latest guidance from the National Institutes of Health at https://grants.nih.gov/grants/guide/notice-files/NOT-OD-21-014.html. ICPSR also offers guidance at https://www.icpsr.umich.edu/web/pages/datamanagement/dmp/.
25 Examples of tools include DMPTool (California Digital Libraries), DMPOnline (Digital Curation Centre—UK), and the Interdisciplinary Earth Data Alliance DMP Tool.
the United States (NARA) that authorize, after specified periods of time, the destruction of temporary records or the transfer to the National Archives of the United States of permanent records that are common to several or all agencies (36 CFR § 1227.10). All agencies must follow the disposition instructions of the GRS, regardless of whether or not they have existing schedules.26
Government agencies are subject to various rules about retention, in particular the 2011 Presidential Memorandum, “Managing Government Records,” and the 2012 Presidential Memorandum of the same title.”27 Agencies are mostly autonomous in deciding what constitutes “records,” although the legal code uses the term to refer to information that is “appropriate for preservation […] as evidence of the organization, functions, policies, decisions, procedures, operations, or other activities of the United States Government or because of the informational value of data in them” (44 U.S.C. 3301). Records schedules can persist for a long time, and are not frequently updated, though recent executive branch memos and efforts at the National Archives may lead to updates and modernizations across U.S. government agencies.
As an example, the surveys conducted by the National Center for Science and Engineering Statistics (NCSES) are captured in the “Request for Records Disposition Authority” form,28 which specifies that survey forms and questionnaires are destroyed once the survey is finalized; edited microdata are transferred to NARA 2 years after completion of the survey, with no disposition noted. The records schedule stems from 1994; there seem to be no other records for the NSF that cover surveys.29 Raw microdata or paradata are not mentioned, and are thus not preserved by NARA, although NCSES may keep copies for access by researchers.
As pointed out earlier, modern approaches and technical capabilities suggest that even some basic paradata should be preserved (as discussed later in this chapter) as digital traces of the methods and procedures used to create the final statistics. For instance, survey paradata can capture how long respondents linger over questions in digital-only survey instruments, and clear documentation of the edits made to the raw respondent
___________________
26 For additional information on record management, see https://www.archives.gov/about/laws (accessed January 13, 2021) and for records schedules, see https://www.archives.gov/records-mgmt/faqs/rcs.html#one.
27https://obamawhitehouse.archives.gov/the-press-office/2011/11/28/presidential-memorandum-managing-government-records.
28https://www.archives.gov/files/records-mgmt/rcs/schedules/independent-agencies/rg-0307/n1-307-93-001_sf115.pdf.
29https://www.archives.gov/records-mgmt/rcs/schedules/index.html?dir=/independent-agencies/rg-0307; and https://www.archives.gov/records-mgmt/rcs/schedules/independent-agencies/rg0307/n1-307-93-001_sf115.pdf.
data enhances credibility, countering suggestions of improper or erroneous data manipulation (see also Box 3-2, NCSES and Paradata). NARA keeps only those federal records that are judged to have continuing value; these amount to about 2 to 5 percent of the records generated in any given year.30
___________________
30https://www.archives.gov/publications/general-info-leaflets/1-about-archives.html.
THE ROLE OF CATALOGS AND SEARCHABLE METADATA
Preserving data and related records does not in and of itself provide transparency if these materials are not themselves discoverable. There are multiple ways national statistics become discoverable, including through the National Archives and data.gov.31 Some statistical agencies have created their own libraries or archives, such as the National Transportation Library. Specialized archives can be preferable if they are able to make more transparent the embedded knowledge about the data held by particular agencies. On the other hand, given the decentralized nature and limited resources of the U.S. statistical agencies, creation of multiple archives can also decrease discoverability, increase costs, and discourage adoption of best practices. In each agency, it will be important to balance the benefits from leveraging their local knowledge and their ability to reach their user communities with the benefits from coordination and integration and the leveraging of existing archival resources. One way to maintain the benefits of decentralization while promoting discoverability is to encourage the use of common metadata standards that facilitate federated searching (e.g., the use of the Data Catalog Vocabulary [DCAT]32 and Schema.org standards makes data discoverable through Google Dataset Search). For details, see Chapter 5.
An inventory of all data resources for each cabinet or other federal agency has been required by policy for nearly a decade, starting with the 2013 OMB memo M-13-13 and now codified by the Foundations for Evidence Based Policy Act of 2019 (now 44 U.S.C. § 3511; hereafter referred to as the Evidence Act of 2019). The M-13-13 policy established the federated architecture whereby each agency compiles an enterprise-wide inventory of all its data assets, using a common metadata schema published as a single JSON file on its website, so that Data.gov and other aggregators can assemble and update combined data catalogs in an automated way. The metadata schema required by M-13-13 was originally referred to as the Project Open Data Metadata Schema, but was revised in 2014 to align with the W3C (World Wide Web Consortium) DCAT metadata standard and is now known as the DCAT-US schema to denote that it is compatible with the international DCAT standard used by many other catalogs.
The DCAT metadata standard is used by a number of other national governments, including those countries following the EU-managed DCAT-AP standard, and it serves as the basis for the Schema.org Dataset schema used by Google Dataset Search and others. Since 2013, most major federal agencies have implemented comprehensive dataset inventories following the metadata standard using metadata management platforms provided by their
___________________
31Data.Gov, while not an archive, does contribute to the harmonization and preservation of metadata and improves the discovery of federal data resources.
32 For a description of DCAT, see https://www.w3.org/TR/vocab-dcat-3/.
agencies or by Data.gov. To support the federal policy and other DCAT-based requirements, many commercial and open-source metadata management tools have been updated to support the DCAT-based metadata standard. The Evidence Act of 2019, as shown in Box 3-3, codified the federated model from the 2013 policy in statute and expanded the scope beyond the 24 CFO-Act agencies to also include other federal agencies. This data inventory policy and process is one of many in the federal government. For example, the 2018 National Geospatial Data Act also codified in statute another long-established OMB policy, Circular A-16, regarding geospatial data and metadata. Nevertheless, the Evidence Act of 2019 is noteworthy for how comprehensive and widely implemented it is.
The basic DCAT-based metadata captured in the data inventories following M-13-13 and the current Evidence Act requirements are unlikely to capture the rich detail needed for adequate transparency for statistical products. However, since these metadata records are required for all data products, and since most agencies have implemented the metadata management systems and processes to support them, they present a useful starting point. They will enable metadata managers to add additional metadata by reference and for researchers to find the core metadata records in the first place. Additionally, metadata standards like DCAT are widely used and meant to be extensible, with domain-specific variants like GeoDCAT and StatDCAT being explored by other national governments. The widespread use of DCAT and its Schema.org variant across the Internet also helps compatible federal metadata records become discoverable through larger aggregators like Google Dataset Search.
An inventory of all data resources for each cabinet or other federal agency is required by the Evidence Act of 2019, as shown in Box 3-3. (Chapter 5 discusses the use of metadata which would make inventories of such resources easier to use.)
Recommendation 3.1: The agencies that produce federal statistics, through the leadership of the Interagency Council on Statistical Policy and the Chief Statistician of the United States, should fully comply with federal record schedules, ensuring that the input datasets that can legally be retained, and official estimates that are produced, are archived in the National Archives and Records Administration. The metadata that accompany such data should also be preserved using broadly accepted metadata standards appropriate to the data at hand. The records schedules, which describe the plans for retaining, preserving, and making accessible microdata and associated metadata, should be easily accessible on each statistical agency Website so that users know when and where microdata and associated metadata will be made available, and when they are scheduled to be destroyed.
ISSUES ARISING WITH PARADATA
While our focus has been on the need for transparency of metadata in the survey collection process, we note that data about the data collection process, so-called “paradata,” can be important for understanding the quality of the data collected as a function of elements of that process. Although the paradata concept was branded almost 25 years ago (see, e.g., Couper, 1998), the analysis of such data and the appropriate inferences one can draw from doing so continue to evolve. Kreuter (2013) describes paradata as “additional data that can be captured during the process of producing a statistic.” Such data are obtained throughout the survey process—as part of the initial interaction, the field staff’s observations, and the respondent’s actions. The data can also be used to help ascertain and improve the quality of the collected data. In addition, it must be stressed that these data may be sensitive, because they are tied directly to an individual respondent and therefore their release can lead to a breach in confidentiality.
Currently, there is no clear dividing line that separates paradata and metadata. But it is fair to assert that some metadata arise from or are aggregate paradata (e.g., response rates are a summary of individual data on participation or refusal decisions at each call).
The cases below illustrate the nature and level of detail of paradata. We include a short discussion of them here, because these data are collected as part of the production process and can inform survey data improvements and assessments of survey data quality, most often internal to a statistical agency. As a result, maintaining and understanding these data can be viewed as a component of an agency’s transparency about the information it makes available.
As described above, in addition to knowing which mode of data collection was used, what measurement instruments generated the data, and (or) detailed information about the population (i.e., metadata), having information about the actual data-generating process—the paradata—will increase transparency. In situations where the data-generating process is in the hands of statistical agencies or other well-defined entities and is designed for a specific purpose (i.e., a survey), systems can be put into place to capture sufficient information about that process itself. Even if the design is not under the control of the agency, effort should be made to capture and retain relevant information to evaluate the data generating process (while the data collection is happening or post hoc). (We note that the methods of Chapter 5 address the means for organizing paradata for archival purposes by conforming to another metadata specification.)
Consider the example of a survey-data production process. It is a process that involves many actors making impromptu decisions, informed by observations from the ongoing data collection process and affecting the
final product. For example, when addresses are canvassed, listers walk or drive around neighborhoods and decide about the inclusion or exclusion of certain housing units. Those managing field work use their experience and judgment to decide which interviewers to send to which locations, where to intensify efforts, and where to reduce them. In face-to-face interviews, ample discretion is given to interviewers in most surveys as to how often to approach a household and how and when to engage with a specific housing unit. During the interview, respondents take varying amounts of time to answer a particular question, additional people might be in the room observing and influencing the interview, or the interviewer might probe in specific ways that affect respondents’ answers.
Getting feedback from all of these actors about the data-generating process would be a tall order. However, given that most survey-data production processes are aided by digital devices, over the last decade paradata have been collected and used not only to guide process decisions but also to evaluate the quality of the data afterwards.
Case Studies
In this last section we review three types of paradata whose capture and preservation may prove highly useful: data concerning interviewers, call record data, and data derived from measurements of respondent survey behavior such as keystrokes.
Interviewer IDs
Many products produced by statistical agencies still rest on interviewer-administered surveys. In these surveys, interviewers influence greatly the collection and processing of the collected information, so capturing paradata about the interviewers and their activities is necessary to ensure transparency. A recent publication on the effects interviewers have on the statistics product (Olson et al., 2020) demonstrated the variety of influences interviewers can have not only in interviewer-administered surveys but also in a variety of mixed-mode surveys. These surveys may rely on interviewers, for example, to list housing units, to create or augment a sampling frame, to contact and persuade sampled cases, to record additional information about the sampled units (including, for example, biomarkers, specimens, and purchase receipts), or to code answers following the survey. Consequently, even if interviewers are not involved in the actual administration of a survey, there is ample opportunity for correlated response variance due to specific interviewer behaviors (Hansen, Hurwitz, and Bershad, 1961; Kish, 1962), or due to biased responses (Schaeffer et al., 2018).
Measurement error corrections, and investigations into quality, are only possible if an anonymized interviewer ID becomes a standard survey release item. The survey methodology community refers to such information as paradata, distinguishing these types of information from the actual survey content and metadata, which typically capture more aggregated descriptions of the data collection process (i.e., response rates), rather than information at each row of a dataset.
Call Record Data
Many survey data collection operations within the federal government use contact history instruments or case management systems designed to capture information about what happened during the attempts to contact a sampled case, including the length of an interview. Interviewers can record whether a successful contact was made, who responded, and what reasons were given for nonresponse, or make additional observations about the household (especially during face-to-face contact attempts). Methodological research has demonstrated such data to be of sufficient quality to guide the data collection process or to investigate failures afterwards (Bates et al., 2010; Lavrakas, 2008; West et al., 2020).
Examining contact attempts jointly with interview length and interviewer ID information has allowed several survey products to uncover fraud or other problems with specific cases or interviewers (Schwanhäuser et al., 2020). Some agencies have started to make use of call records to convert to more responsive data collection designs that are more cost efficient (Chun, Heeringa, and Schouten, 2018). However, many agencies lack the internal resources to do so, and access to call record data (as well as other paradata) would allow external researchers33 not only to evaluate the quality of the final survey product, but also to make suggestions on how the data collection process could be improved. Currently it is an open question how large the user base for paradata from call record data is, and this needs to be weighed against the effort to make such data available. A good example of disseminating nonresponse-related paradata is the National Health Interview Survey (Dahlhamer and Simile, 2009; Maitland, Casas-Cordero, and Kreuter, 2009).
___________________
33 Adam Eck, then a graduate student, had an internship at Census with the goal of obtaining those data for external analyses. As a result, he wrote a program to deidentify those data internally at Census that passed the disclosure review board, and access to a large number of audit trails was made available. This program, presumably, could be used to release audit trails more generally from the ATUS, but we are unaware of its application to date.
Keystrokes and Other Measurement-Related Paradata
Keystrokes and other measurement-related paradata, generated as a byproduct of computer-assisted data collection, were the first processes referred to as paradata by Mick Couper in a presentation at the Joint Statistical Meeting in Dallas (Couper, 1998). Respondents or interviewers leave electronic traces as they answer survey questions, captured through keystrokes and mouse clicks. Today, when computer-aided data collection is even more common, information from respondent’s actions, such as keystrokes (outside of time stamps), mouse movements, or voice recordings, can easily be captured. Nevertheless, the systematic analytic use of such information is still less common (Callegaro, 2013; Kunz and Hadler, 2020; Olson, 2013). This could be because their use is not as well established, or because their storage and dissemination are even more difficult. In the American Community Survey, for example, while one unit of observation can represent upwards of 20 contact attempts, a full interview likely has hundreds of keystrokes submitted whether in a computer-aided interview setting or by the respondents in self-administered Web settings. Few examples exist where log data or audit trails are available as public-use data. A notable exception is the American Time Use Survey (Belli et al., 2019), where a dataset has been created containing the interactions by interviewers with a computer-assisted telephone interview (CATI) instrument while entering responses provided by respondents. Its metadata show the expansive size of such files: in a study where 13,200 people were interviewed, paradata describing the interactions with each question amounted to 2,061,889 records. Also, for the National Assessment of Educational Progress’s Mathematics Assessment, keystroke and timing paradata from approximately 29,000 students resulted in nearly 16 million records.
To the research community, it is currently unclear for which of the surveys such paradata are available and what the possibilities are if researchers are given access. Since there is a finite set of operating systems and data collection products for CATIs, CAPIs (computer-assisted telephone and personal interviews, respectively), and Web surveys, it is conceivable to create standardized code to extract relevant indicators from the broad set of paradata to store alongside the responses. Retaining all relevant individual raw data indefinitely may be feasible.
One possibility is to develop guidelines for the retention of some paradata. As an example, a small, common denominator of paradata availability guidelines that seems feasible is outlined in Box 3-4.
The case studies above illustrate uses of paradata and suggest that their availability, with appropriate documentation, can help survey researchers understand the survey-data collection process. This can be especially valuable for continuing ongoing data programs where research on a data
collection cycle can help improve the data collected in a future data cycle. The use of paradata, mostly motivated by survey methodologists, has grown over paradata’s brief history. Nevertheless, paradata’s use is not widespread, and it is hampered by a lack of documentation, privacy concerns, the need for special processing, cost, and a relatively small research community.
Paradata stemming from agency actions (e.g., data on the type and extent of effort expended to collect responses, such as contact history) might be retained with response data for as long as the response data themselves are retained. The same is true for paradata from case management details, such as cases that were moved from one mode to a more expensive alternative mode. Paradata that involve respondent reactions to attempts at data collection, such as keystrokes or mouse movements that suggest backtracking, or response latency, could suggest a lack of clarity in what is being asked of the respondent, which in turn could result in lower-quality responses. Therefore, there might be a good reason to retain these for a fixed period of time for examination. In Chapter 5, we will argue that metadata standards provide the means for organizing data for archival purposes through conforming to a metadata specification, and this could be useful in archiving paradata as well.
Continued research on the use of paradata is important to improve the quality of survey data in ongoing data programs. NCSES should investigate how their data programs use paradata, identify programs that would
benefit from the use of paradata, identify what data are valuable to maintain, determine the length of time such data are available to researchers, and ensure that record schedules include the disposition status of such data. While individual programs have different requirements and uses, for the purpose of transparency NCSES management should develop a policy concerning the availability and use of paradata consistent with its mission.
The federal statistical agencies should retain, preserve, and make accessible machine- and human-readable metadata—including survey instruments and the provenance of any administrative data—used in the production of official statistics. In addition, because paradata help to provide a better understanding of the quality of survey data, the federal statistical agencies should retain, preserve, and make accessible both machine- and human-readable paradata necessary for evaluating data quality. The comprehensiveness of the retained, preserved, and accessible paradata should be necessarily greater for official statistics used in high-profile decision making, such as the allocation of federal dollars. This includes, but is not limited to, the decennial census, the American Community Survey, and the principal federal economic indicators as defined in Statistical Policy Directive No. 3.
Recommendation 3.2: Federal statistical programs, whose inputs include survey data, should make available, for as long as the data are believed to be of interest to researchers, associated paradata to help users assess the quality of the survey inputs.
Further, when such paradata are associated with a statistical program that is used to distribute political power or substantial federal funds (such as the Decennial Census) and the paradata are a key measure of the quality of inputs to such a program, statistical agencies should make public such assessments for relatively disaggregated demographic-geographic domains.
This page intentionally left blank.
