The U.S. Geological Survey (USGS) mission is to provide reliable and impartial scientific information “to describe and understand the Earth; minimize loss of life and property from natural disasters; manage water, biological, energy, and mineral resources; and enhance and protect our quality of life.”1 Data collection, analysis, interpretation, and dissemination are central to everything the USGS does.
The USGS operates some 250 laboratories to analyze physical or biological samples, including water, sediment, rock, plants, invertebrates, fish, and wildlife. The data generated in the laboratories help answer pressing scientific and societal questions associated with energy and mineral resources, natural hazards, surface and groundwater quality and quantity, biology and ecology, environmental health, and other topics. A growing number of these topics touch on controversial issues, such as those concerning environmental pollutants or climate change, and USGS scientific results are coming under increasing scrutiny. For these issues especially, the USGS reputation and the self-correction of science through peer-reviewed publications may no longer suffice to maintain public trust in USGS research results.
The importance of providing reliable information came to the fore in 2016, when an Inspector General report found scientific misconduct and data manipulation at a USGS laboratory in Lakewood, Colorado (IG, 2016). Two laboratory analysts had adjusted values outside of protocols over two extended periods. Reports of such incidents are rare, but they highlight the importance of data quality assurance and quality control procedures. To restore confidence in USGS data, the USGS has begun implementing a quality management system (QMS)—a structured and documented system that establishes the requirements for how work is to be managed, conducted, and monitored to assure data quality. A QMS is a paradigm shift for the USGS because instead of each laboratory developing its own quality assurance procedures, all USGS laboratories will be required to implement a centrally defined quality standard in a similar and consistent way, as applicable to the work performed (USGS, 2018). The laboratory elements typically managed in a QMS are illustrated in Figure 1.1.
In addition to developing a QMS, the USGS has commissioned external reviews of its QMS implementation, including this report.
At the request of William Werkheiser, then USGS Deputy Director, the National Academies of Sciences, Engineering, and Medicine established a committee to review a selection
of USGS analytical laboratories and identify best practices and procedures for assuring the integrity and reliability of laboratory results. The specific tasks to the committee are given in Box 1.1.
When the tasks in Box 1.1 were developed, USGS managers had only begun collecting information about its laboratories and their quality assurance practices. As their knowledge grew, it became clear that some of the information needed to address the tasks would not be
available for the study. Consequently, some tasks could not be addressed exactly as written. At the first committee meeting, the USGS deputy director suggested ways to address the intention, rather than the exact wording, of these tasks. With those guidelines, the committee addressed the tasks as described below.
Task 1 was to provide an overview of all USGS analytical laboratories, including their science and applications objectives, budget, staff and user profiles, and history of sample throughput. The USGS collected and analyzed this and other information (e.g., data quality procedures) in 2016 (see Phillips et al., 2018). Consequently, the committee used the 2016 USGS laboratory inventory as a baseline and supplemented it with information gathered at site visits to USGS laboratories. Site visits and associated committee meetings were held in four cities: Reston, Virginia; Denver, Colorado; Menlo Park, California; and Kearneysville, West Virginia (see Table 1.1). Only 17 laboratories (6 percent), 16 active and 1 inactive at the time, could be visited in the confines of the study, and they were chosen to sample a diversity of sizes and activities. Visits to a small number of diverse laboratories cannot yield a comprehensive picture of the USGS laboratory landscape. However, a number of common themes emerged that informed the committee’s response to Task 1.
TABLE 1.1 USGS Laboratories Visited by the Committee
|June 2018||Reston, Virginia||
|August 2018||Denver, Colorado||
|November 2018||Menlo Park, California||
|February 2019||Kearneysville, West Virginia||
Task 2 was to describe analytical procedures, standards, and data management procedures for laboratories at the USGS, other federal agencies, and other geological surveys. Describing analytical and data management procedures in hundreds of laboratories was not feasible in the confines of this study. Consequently, the USGS asked the committee to focus on laboratory QMS procedures in order to provide a benchmark for its QMS effort. The committee addressed the task by asking a selection of U.S. federal agencies and academic institutions and geological surveys in other countries to discuss a set of questions on QMS and alternative approaches for assuring data quality at committee meetings.
Task 3 was to assess the extent to which laboratory resources are sufficient to meet their objectives. The USGS-collected data were not sufficiently detailed to carry out this assessment. Laboratory budget information was available in ranges, staff (principal investigators, laboratory managers, analysts, and students) were grouped into total full-time equivalents, and research and applications objectives were general. Consequently, the committee used the opinions of laboratory staff and its own observations at the laboratory visits to draw high-level conclusions on the adequacy of resources.
Tasks 4, 5, and 6 concern the identification of best practices for assuring data quality in USGS laboratories. Best practices and data quality terms used in this report are defined in Box 1.2. The USGS distinguishes two types of laboratories:
- Research laboratories support innovation or scientific discovery and are typically led by a principal investigator (senior scientist). Some research laboratories also develop methods that are needed to answer a scientific question or that other laboratories can use for routine analyses.
- Production laboratories carry out routine or repeated analyses for USGS or external customers, are sometimes supported by user fees, and are generally led by laboratory managers.
The USGS data do not indicate which laboratories do routine analyses or are led by laboratory managers, but they do specify which have external users, such as regulators and resource managers, which is an indication of production activities.
Task 4 was to develop criteria for assessing laboratory protocols and procedures used by the organizations mentioned in Task 2 and to use them to identify best practices for USGS production laboratories. Numerous assessments of laboratory procedures and recommended best practices have been published, and the committee drew from these to address Task 4.
Task 5 was to recommend best practices and procedures for achieving objectives and assuring the integrity and reliability of results for USGS production laboratories, and Task 6 was to comment on best practices and procedures for research and method development laboratories. The committee used what it learned from the laboratory site visits, meeting presentations, and the published literature to address Tasks 5 and 6.
The USGS has a federally funded budget of roughly $1 billion and has some 8,000 employees. The USGS is organized in a matrix structure that involves seven mission areas chosen to reflect USGS science and applications objectives and seven regional offices chosen to reflect geographic variations in geological, biological, and hydrological characteristics
(e.g., northeast, southwest, and Alaska). The mission areas are core science systems, ecosystems, energy and minerals, environmental health, land resources, natural hazards, and water resources. The USGS is headed by a director, under whom are a deputy director and seven associate directors corresponding to the mission areas (Figure 1.2). The directors of the seven regional offices report to the USGS deputy director. Each regional office has several science centers, which manage the science activities of the region and also oversee most of the laboratories.
Line management, personnel management, and science center management are organized through the regional offices. However, the QMS is being implemented by mission area. The reporting structure is complex. Although QMS implementation plans are being developed by the mission areas, QMS reporting lines flow through the Deputy Director. The Bureau QMS Coordinator and staff—who oversee implementation of the QMS—report to the Deputy Director through the director of the Office of Science Quality and Integrity (Figure 1.3). The other QMS staff, including QMS champions, report to their Deputy/Associate Science Center Director, and not to the Bureau QMS Coordinator.2
This report examines approaches to managing laboratory data quality and recommends best practices and procedures for assuring the integrity of USGS laboratory results. The report is organized to emphasize the information that USGS most needs to develop its quality assurance program. Chapter 2 summarizes the literature on systematic approaches for assuring data quality, including assessments of laboratory procedures and best practices (Task 4), and describes example approaches that are relevant for USGS laboratories. Chapter 3 provides an overview of the approximately 250 USGS laboratories being examined, including the adequacy of resources to meet their science and applications objectives, and their approach to assuring data quality (Tasks 1, 2, and 3). Chapter 4 highlights experiences with quality management systems at the USGS, other federal agencies, other geological surveys, and research institutions (Task 2). Chapter 5 provides the committee’s recommendations for the USGS, including the development and implementation of its QMS, and best practices and procedures for USGS laboratories (Tasks 5 and 6). The report concludes with a list of references and biographical sketches of committee members (Appendix).
2 The language of this sentence was modified after release of the publication version to reflect current USGS staffing plans.
3 This figure was replaced with a current organizational chart after release of the publication version.