Building an Electronic Records Archive at the National Archives and Records Administration: Recommendations for a Long-Term Strategy (2005)

Chapter 1 describes factors driving the shift to electronic records and the growing volume and variety of such records. Electronic records offer considerable new flexibility with respect to duplication (perfect digital copies can be made readily and cheaply), transfer (records can be transferred electronically using networks or physical media), and physical custody (because storage is cheap, it is feasible to maintain multiple copies). Many electronic records exist within information systems, which makes it possible to capture records and associated metadata systematically, and to automate their transfer and processing. The properties of electronic records support the establishment of new processes and relationships with agencies for scheduling and transferring custody of records. The anticipated volume of records that will be submitted for archiving compels the use of automated transfer and ingest.

This chapter first reviews three National Archives and Records Administration (NARA) initiatives that bear on process reengineering. It then points out a number of additional opportunities for NARA.

Through three major initiatives—the Electronic Records Archives (ERA) program, the Records Management Redesign initiative, and an initiative on records management under the administration’s e-government initiative—NARA has taken a number of steps toward building needed technical capabilities for the archiving and preserving of the federal government’s electronic records and toward making associated changes in process.

NARA’s Electronic Records Archives program was begun in 1998. Its purpose was to address the growing volume and diversity of electronic records being used and relied on by

government agencies, as well as to address issues of format obsolescence and preservation. A fundamental objective of the ERA program is to confront the formidable challenge of creating a “system that will authentically preserve and provide access to any kind of electronic record, free from dependency on any specific hardware or software.”¹ Early steps in the ERA program included NARA’s exploration of possible solutions and undertaking of development projects with partners. The partners included the San Diego Supercomputer Center, the University of Maryland, and the Georgia Tech Research Institute.

In August 2003, NARA released a draft request for proposals² (RFP) for the ERA system, “seeking input from industry to ensure that the final RFP clearly communicates the purpose of the acquisition.”³ Before preparing this report, the National Research Council’s Committee on Digital Archiving and the National Archives and Records Administration prepared two reports providing advice related to the ERA acquisition: its first report recommended a strategy for engineering and acquiring the ERA using an iterative design approach, and a letter report provided specific comments to NARA regarding the draft RFP.⁴ Also in August 2003, the General Accounting Office released a report⁵ that “found several deficiencies in NARA’s plan for the [system’s] acquisition,” including the lack of a “concept of operations for the system from the users’ perspective” and an “incomplete schedule and process to track the costs of the program.”⁶

In 2004, NARA released the final version of its RFP and subsequently selected two contractors to develop designs for the ERA.⁷ As NARA observes, much “remains to be done before the vision of ERA becomes a reality. The necessary projects and tasks fall in two large categories: [1] designing the structure and [2] building it.”⁸ NARA hopes to “have a functional subset of the [ERA] system operational in 2007.”⁹

Responding to “technological advances, budget cuts, and other events [that] have significantly changed the environment surrounding Federal records management,”¹⁰ the National

Archives and Records Administration began a redesign of federal records management in 2001. This was done through an initiative with the ultimate goals of (1) documenting the current record-keeping and records use environments in the federal government, (2) using subsequent findings to analyze NARA’s records-management policies, and (3) redesigning the scheduling, appraisal, and accessioning processes where necessary.

With respect to the first of those goals, NARA issued in December 2001 a report prepared for NARA by SRA International. The report examined the views and perspectives on record keeping and archiving of a sample of federal employees (Box 2.1). The results of NARA’s own records systems analyses—which pointed to the importance of “situational factors” within agency records-management operations—were included in the report, which offered “a number of approaches, options, and interventions that may assist NARA and the agencies to improve Federal [records management].”¹¹

Building on the work that produced the 2001 report, NARA issued a draft proposal in July 2002¹² for rethinking the regulations, policies, and processes for managing federal records and for developing a set of strategies to support federal records management. A year later, after “working with stakeholders and [testing] proposals,” NARA released Strategic Directions for Federal Records Management, its “roadmap for redesigning NARA’s records-management program and records management in the federal government.”¹³

The Strategic Directions report begins by asserting NARA’s goals with respect to its redesign of federal records management before moving on to more specific strategies and tactics for reaching those goals. The following are among the report’s enumerated strategies:

• [NARA] will demonstrate that effective records management adds value to agency business processes….

• [NARA’s] approach to records management will be based on the ISO [International Organization for Standardization] Records Management Standard 15489.

• [NARA] will focus on those records that are essential to the government as a whole for accountability, protection of rights, and documentation of the national experience….

• [NARA will] partner with Federal agencies and others to develop, adapt, or adopt products and practices that support good records management.

• [NARA] will provide leadership, in partnership with other key stakeholders, to focus agency attention on electronic records needs and to guide and support solutions to electronic records issues and problems.¹⁴

The Strategic Directions report also details some of the tactics that NARA will use to pro-

mote these strategies and goals. Those tactics include such things as promoting good communications between NARA and stakeholders, modifying guidance and training programs, providing assistance to agencies, providing agency oversight, supporting the development of records-management tools, and reengineering NARA’s own business processes.

Recently, NARA has also issued several white papers in support of the Strategic Directions report. The white papers focus on—among other things—advocacy, appraisal policy, certification of training, custody, expansion of the records-management training program, inspections and studies of records management, and resource allocation.¹⁵

NARA’s Electronic Records Management (ERM) initiative was launched as part of the Bush administration’s e-government agenda. This initiative is intended to provide “policy guidance to help [federal] agencies … better manage their electronic records, so that records information can be … used to support timely and effective decision making, enhance service delivery, and ensure accountability.” The ERM “will provide the tools that agencies will need to manage their records in electronic form, addressing specific areas of electronic records management where agencies are having major difficulties.”¹⁶

The ERM Initiative was started with the following three goals in mind:

1. To integrate e-records-management concepts and practices with comprehensive information-management policies, processes, and objectives in order to assure the integrity of e-records and information;

2. To employ ERM to support interoperability, timely and effective decision making, and improved services to customers; and

3. To provide the tools for agencies to access e-records for as long as required and to transfer permanent e-records to NARA for preservation and future use by government and citizens.

As the managing agency for the ERM initiative, NARA has organized the project into four issue areas:

1. Correspondence tracking,

2. Enterprise-wide ERM (to provide tools for agencies to use as they plan and implement ERM systems),

3. Electronic information-management standards (to promote government-wide use of the Department of Defense 5015.2 Standard for Records Management for federal agencies), and

4. Transfer of permanent electronic records to NARA (to address an expansion of both the number of formats that NARA can accept and the media and techniques that can be used by federal agencies when transferring their permanently valuable electronic records to NARA).

The overall ERM initiative includes a number of deliverables. For example, NARA has expanded the acceptable methods for transferring electronic records to NARA, and it has issued guidance for transferring permanent electronic records of the following types: e-mail records (including attachments), Web content records, scanned images of textual records, digital photography records, digital geospatial data records, and records in portable document format.

Outlined below are broad areas of process reengineering that will be important to the success of the ERA and, more generally, to NARA’s success in handling electronic records.

Existing processes for ingesting records rely heavily on manual processes. Paper records naturally require a good deal of manual processing to acquire and appraise. Current procedures for transferring electronic records also involve significant manual intervention, starting with the effort associated with creating and organizing metadata into finding aids and the transfer of physical media or the initiation of a network file transfer. Similarly, appraisal and description require significant manual effort, including such steps as manually examining record structures and capturing metadata.

As outlined in Chapter 1, the volume of electronic materials will grow substantially and rapidly. At the same time, a major increase in NARA’s budget for archiving appears unlikely. The only possible resolution in order for the ERA system to be able to handle the anticipated volume of records is to make a major shift to automation. Fortunately, it is possible to make this change by improving how records are delivered to NARA, by establishing procedures and systems to ingest new records automatically on a routine basis, and by capturing metadata at record creation and using automatic techniques to help fill in gaps.

To make the automation effort successful, it will be critical to establish how to handle errors that occur in the ingest process. This is important because the costs of exception handling are a common reason for the failure of automation. Careful consideration should be given to the amount of effort that should be expended on trying to fix problems, on the extent to which the responsibility for resolving problems can be pushed back to the originating agency, on ways that errors can be corrected in the future when they are discovered by archives users, and so forth.

Even as NARA develops capabilities and processes to handle new records through automation, a considerable backlog of electronic records will continue to grow before these new measures take full effect. To avoid being swamped by the backlog, triage measures may be needed. These could include avoiding significant manual processing, preserving the obvious artifacts that would support the extraction of automatic metadata in the future (e.g., when accepting e-mail, it is important also to receive the mapping from e-mail address to personal directory information), and using automatic metadata extraction (and accepting its shortcomings).

The ingest of electronic records decades after their creation can be problematical—obsolete data and metadata formats may be difficult to interpret, and it may be quite difficult to modify the systems in which the records reside so as to facilitate ingest. When the originating systems are still active, preservation problems can be tackled before obsolescence.

One way to mitigate problems at record ingest is to conduct dry-run testing (running through all of the steps that would be taken at ingest, conducting the dry run over full sets of records, or using a sample as a spot check). Such testing allows a number of potential problems to be uncovered and resolved, including the following: Are the data and metadata for-

mats known, and can they be processed by the ERA’s (automated) ingest process? Is appropriate information as to the record’s authenticity and integrity available? Are access and transmission procedures in place at the creating agency and at NARA to effect ingest? This type of system debugging serves as an early warning of problems.

Another option is the early capture of records in anticipation of future custody transfer to NARA. Unlike paper records, electronic records offer considerable new flexibility—copies are readily made and cheaply stored. Early capture is best done by designing systems that generate permanent records so that records are created archive-ready at birth—automatically recording metadata, providing users with very easy ways to create metadata manually as records are created, and designing systems that produce outputs that are easy to interpret, self-describing, and otherwise well suited for long-term preservation.

“Pull” technologies—such as automated download of records and Web harvesting—are one way to routinely obtain records earlier in their life cycle. This technique is not limited to publicly accessible records. For example, an agency could provide a restricted-access Web service for NARA to use to extract records as they are created or to search the agency’s internal record-keeping system. The resulting archival copies might also provide additional incentives for agencies to support the creation of archive-ready records and their delivery to NARA, since these records would then serve as off-site backups of agency operational systems.

Thus, there are two almost-identical techniques: a passive one in which NARA is sent records early, on a provisional basis, for purposes of a dry run; and an active one in which NARA extracts and ingests records. Both give early warning. The active mode is probably harder for agencies (and NARA) to implement, but it gives NARA the greatest visibility into the records that it will have to handle. The active approach also provides the potential incentive of NARA’s copies serving as a backup.

Archival practice was developed primarily to handle paper and other tangible-media records. However, several trends over the past three decades have made it more difficult to carry out traditional records scheduling and disposition activities:

• New ways of working that supplant the traditional organizational hierarchy with greater use of teams, task forces, and so on that are project- or goal-oriented, rather than procedure-oriented;

• The reduction in administrative and clerical staff responsible for organizing, filing, and retrieving records and developing retention and disposition schedules, and the resulting transfer of responsibility for record management to end-user agency staff;

• The preference for electronic communication and record keeping over paper-based systems; and

• The dearth of useful tools that can integrate records management into end-user tasks and activities.

At the same time, several technology trends compel a rethinking of selection, appraisal, and description in two major areas:

• First, the granularity of appraisal should be larger. The appraisal of physical records was driven in part by a need to reduce significant storage costs in an archive. But with ever-declining storage costs, the opposite is the case with digital information. Investing time in reducing a set of records by 10 percent or 40 percent or even 60 percent costs more than it saves, because these decisions require human intervention and professional judgment and research. Also, records are more interconnected when they are digital. The information or evidential value of a set of records considered in isolation may not be obvious, but one set of records may be crucial for understanding or using another set.

  Rather than examining sets of records carefully and marking only a select few for retention, it may become more expedient and more appropriate to retain larger “chunks” of records, The committee is not advocating a “save everything” approach, but a rethinking of selection criteria. For example, it may be better to preserve entire classes of electronic records (e.g., all of the records from certain offices, all of the records covering function X in both the federal and regional offices, and so on) that would not have been feasible for paper records, even though some records with little permanent value may be intermingled. NARA appears to be moving in this direction in the Records Management Redesign initiative, with its emphasis on functional appraisal, which implicitly is more broadly granular.

  Consider a specific example. For the material that will be included in the U.S. Department of State’s new SMART system, NARA currently has more than 2,000 retention and disposition schedules for series of records with various retention periods. It might be much more cost-effective simply to keep everything from this system, or to replace the current fine level of appraisal and disposition with one that has fewer than a dozen schedules.

• Second, archives should place less emphasis on manual record description of records and on creation of finding aids and more on automated tools for improving access. The characteristics of electronic records permit new access methods that can decrease the burden on archivists to describe records and create finding aids. Agencies can supply records in a structured format and attach metadata. Full-content search on the records and associated metadata can supplement—or even supplant—manually generated finding aids. Finally, agencies themselves create search and access tools for some of their records. Such tools might be maintained by the originating agency (perhaps as part of the sort of agreement to distribute archiving responsibility, as discussed below) or adapted for use by NARA.

  Another way to think about the trade-offs discussed here is that it may be advisable in many cases to make access for end users (when and if it occurs) potentially more difficult (by increasing the total volume of records to be examined and describing records in less detail) in exchange for making ingest cheap (by reducing the burden on the archivist to select and describe materials). Of course, as tools such as full-content search improve, the burden on the end user will be reduced.

Electronic records transfer to NARA and their ingest are greatly eased if records can be supplied to NARA “archive-ready.” Archive-ready records, at a minimum, are packaged with the metadata required for them to be transferred and otherwise processed using largely automated processes. Preferably, archive-ready records would also be provided in structured, standardized formats approved by NARA as amenable to long-term preservation.

The closer to record creation that archive readiness is established, the more likely it is that

the appropriate metadata will be captured. This means that less work (and cost) will be incurred in adding the information later. Ideally, agencies would design systems so that records are created archive-ready at birth. This condition would occur through automatic recording of metadata, agencies providing their own staff members with very easy ways to manually create metadata as records are created, and so forth. Alternatively, agencies can add necessary metadata at some point after creation, but then still deliver records to NARA archive-ready.

If a large fraction of the records delivered to NARA are to be archive-ready in the long run, archiving considerations will have to become part of the government software procurement and development process. This is especially true for those systems that produce or are deemed likely to produce permanent records. Successful implementation of the ERA system may also require NARA to become more actively involved in the establishment of standards used in constructing federal systems, such as those governing the formats used to represent records and associated metadata, and for NARA—or other bodies responsible for information policy, such as the Office of Management and Budget—to establish guidance to agencies and auditing and enforcement measures as necessary.

As a practical matter, these efforts should link archive-ready concerns with agency interests in supporting the agencies’ own record-keeping operations. Experience shows the difficulty of implementing systems that depend on user compliance with externally imposed requirements that necessitate effort and do not support the user’s internal business needs. As a result, requirements should support agency record-keeping needs, and implementing the records should be realistically possible. Experience also shows that schemes that depend on user-supplied metadata are unlikely to succeed, meaning that software and systems that minimize additional manual contributions of metadata are more likely to succeed.

Close coordination will also be needed between the Records Management Design initiative and the ERA program, because if changes along the lines of those envisioned in the RMI do not occur, the flow of records into the ERA will be seriously impeded. The amount of work that NARA would then need to do to make the records useful would be unsustainable.

As discussed above, metadata describing electronic records can and should be created with as little recourse to manual description as possible. It is useful to distinguish three ways in which metadata can be collected automatically:

1. Metadata can be collected as a matter of course when a record is created (or by virtue of the series or collection in which it lies). For example, a word processor may automatically attach metadata identifying the author and last modification date (assuming that the system is correctly configured with such information).

2. Metadata can be automatically extracted from a record without any recourse to information extraction or otherwise “understanding” the record. For example, header information in an e-mail message is readily extracted by parsing the message.

3. Metadata can be automatically extracted from a record using information-extraction or categorization techniques. As discussed in Chapter 1, these techniques are imperfect, but they can be expected to improve with time.

The first of these options is relatively easy and inexpensive. It is the preferred approach for authored records. Ideally, agency record-keeping systems would have facilities for automatic capture built in. The second option is also easy and inexpensive for records that have the necessary structure.

The third option should be viewed as a fallback. Even when it is necessary to use automatic metadata-extraction techniques, this is better done early on, by the creating agency, which may be able to use content-specific techniques.

In cases in which NARA will have to resort to automatic metadata extraction using natural language techniques, it will have to accept significant imperfection in some of its metadata—a shift from the traditional mind-set of archival institutions. It will also be necessary for systems and processes to distinguish among different degrees of quality in the metadata, which have varying reliability. Finally, as metadata extraction techniques improve, the metadata can be upgraded (possibly on-the-fly as records are accessed). The ERA should accommodate making such updates to record descriptors.

The increased flexibility afforded by electronic records opens up a number of options for custody and access in partnership between NARA and federal agencies. The model of having certain agencies retain responsibility for certain types of records is well established; for example, several agencies have been given the responsibility for preserving large scientific data sets. NARA policy has established criteria for determining whether to establish such an affiliated archive.¹⁷ In 2003, NARA entered into a new partnership with the U.S. Government Printing Office (GPO) which makes the GPO an “archival affiliate.”¹⁸ The agreement ensures the permanent availability of the documents available today (and in the future) through the GPO Access Web site, which provides online public access to more than 250,000 publications, including the Congressional Record, Federal Register, and Code of Federal Regulations. The agreement gives NARA legal custody of the documents while delegating to GPO the responsibility for public access and preservation. Users are directed from NARA’s catalogs to documents held for access and preservation by GPO.

This approach has several important advantages:

• By leveraging the skills and resources of other agencies for preservation and access, NARA can spend its scarce resources on problems that agencies cannot address on their own.

• Giving an agency responsibility for the preservation and access of records means that the specialized expertise required to understand or manage a complex system need not be transferred to NARA. For example, the U.S. Census Bureau currently provides public access to survey data through a Web interface and supporting analysis software. It would be impractical to expect NARA to develop domain expertise sufficient to operate online access to all forms of government data.

• It provides a clear path for NARA to identify permanently valuable essential evidence from a government-wide perspective, and it also enables agencies to make independent judgments as to whatever else they want to retain for their own purposes.

• Diversity of responsibility is created with multiple agencies taking on the responsibility of preservation, and this diversity provides some additional measure of safety compared to that afforded by a monolithic repository.

• By banding together, NARA and other organizations stand a much better chance of spreading widely the standards and tools for preparing records that are archive-ready, of developing affordable software to meet the needs of researchers and other clients of these repositories, and of otherwise making electronic records archives practical in the long run.

Despite these advantages to cooperative arrangements, there is a potential downside to a distributed set of archives. If systems proliferate and diverge, users of affiliated archives will be frustrated by different user interfaces, different access or authorization requirements, and different implementation of services (such as downloading electronic records). To increase the utility to users, the affiliates should federate search and authorization techniques and coordinate their services.

If such arrangements become more common, NARA’s role would shift to that of a standards-setter, federator, troubleshooter, and repository of last resort.