The U.S. Geological Survey (USGS) mission is to provide reliable and impartial scientific information to understand Earth, minimize loss of life and property from natural disasters, and manage water, biological, energy, and mineral resources. Data collection, analysis, interpretation, and dissemination are central to everything the USGS does. Among other activities, the USGS operates some 250 laboratories across the country to analyze physical and biological samples, including water, sediment, rock, plants, invertebrates, fish, and wildlife. The data generated in the laboratories help answer pressing scientific and societal questions or support regulation, resource management, or commercial applications. Consequently, it is important to maintain public trust in USGS data.
In 2016, an Inspector General report found scientific misconduct and data manipulation at a USGS laboratory in Colorado. Two laboratory analysts had adjusted values outside of protocols over two extended periods. To restore confidence in USGS data, the USGS began developing a quality management system (QMS) in 2016 and set an aggressive schedule for its implementation. A QMS is a structured system that establishes and documents the requirements for how work is to be managed, conducted, and monitored to assure data quality. A QMS is a paradigm shift for the USGS because all its laboratories will be required to implement a centrally defined quality standard in a similar and consistent way.
At the request of the USGS, the committee reviewed a representative sample of USGS laboratories, examined QMS and other approaches for assuring the quality of laboratory results, and recommended best practices and procedures for USGS laboratories. The specific tasks to the committee are given in Box S.1 and discussed below.
The tasks in Box S.1 specify two types of laboratories:
- Research laboratories, which support innovation or scientific discovery and are typically led by a principal investigator (senior scientist). Some research laboratories also develop methods that are needed to answer a scientific question or that other laboratories can use for routine analyses.
- Production laboratories, which carry out routine or repeated analyses for USGS or external customers, are sometimes supported by user fees, and are generally led by laboratory managers.
In practice, many USGS laboratories carry out a mix of these activities. Consequently, the committee sought to differentiate laboratories that primarily serve scientists, and thus likely support research activities, from laboratories that serve external customers (regulators, resource managers, and private companies), and thus likely support production activities.
Tasks 1 and 3 concern USGS laboratory characteristics. Before 2016, the USGS did not have a complete inventory of its laboratories and their capabilities. Consequently, the agency issued two data calls to its employees: one on basic laboratory information and one on data quality procedures. In responding to the questionnaires, laboratory managers and principal investigators defined their own laboratory boundaries, with some grouping similar activities into a single laboratory, and others splitting similar activities into more than one laboratory. This exercise yielded 257 analytical laboratories. The committee examined the USGS 2016 laboratory inventory in the context of its tasks.
Task 1 was to provide an overview of USGS laboratories, including scientific objectives, budget, staff, user profiles, and sample throughput rate. The committee found substantial diversity in all of these factors. Collectively, the laboratory science and applications objectives cover all USGS mission areas: core science systems, ecosystems, energy and minerals, environmental health, land resources, natural hazards, and water resources. Some laboratories focus on a mission area, a region (e.g., Grand Canyon), or a measurement technique (e.g., stable isotopes or molecular genetics). Sample throughput, which depends on the type of sample being analyzed and the analytical procedures being performed, ranged from 60 samples per year to 33,000 samples per year for the laboratories visited by the committee.
Most laboratories have multiple sources of funding, such as the USGS, grants from other federal agencies, or user fees. Analyses may also be traded for other services. Similarly, most laboratories serve multiple types of users, including scientists (USGS, other federal agencies, and academic), regulators and resource managers (federal, state, and local), and private companies. A few serve only USGS scientists.
All of the laboratories supply analytical data to USGS researchers, external users, or both. The laboratories focused primarily on supporting research are generally small (two or three full-time equivalents [FTEs] on average), have low annual budgets (typically $0.2 million or less), and serve USGS scientists as well as scientists in other federal agencies and academic institutions. In contrast, laboratories that serve regulators, resource managers, and commercial users in addition to scientists generally have more staff (seven FTEs on average) and larger annual budgets (typically two to four times higher) than laboratories primarily supporting research. The largest USGS laboratory—the National Water Quality Laboratory—primarily provides sample analyses and specialized services to customers, and it has 134 FTEs and an annual budget of $6 million or more. Finally, all of the USGS laboratories have some quality assurance and quality control procedures in place, but those procedures are generally more comprehensive and better documented in laboratories supporting production activities than in laboratories primarily supporting research.
Task 3 was to assess the extent to which operational and personnel resources are sufficient to meet the scientific and applications objectives of USGS laboratories. The 2016 USGS laboratory inventory was not sufficiently detailed to carry out this assessment. Consequently, the USGS asked the committee to draw high-level conclusions on the adequacy of resources based on the views of laboratory staff and committee’s observations at its laboratory visits. The committee found that staff in the 17 laboratories visited were skilled in what they do and took pride in their science, their reputation, and the societal impact of their work. Most of the laboratories had state-of-the art instruments as well as older equipment purchased over their long history. The laboratories appeared be to meeting their science and applications objectives. However, staffing shortfalls and turnover were a common resource problem. Adding responsibilities for implementing a complex QMS, especially without adding sufficient resources, may hinder their future ability to meet their science goals.
APPROACHES FOR IMPROVING LABORATORY DATA QUALITY
Tasks 2 and 4 concern data quality assurance procedures and best practices at laboratories in a variety of institutions. An extensive literature from biomedical science and other scientific fields evaluates research practices that affect data quality, assesses laboratory procedures, and recommends best practices for laboratories. These publications show that errors and poor habits occur frequently enough to affect data reliability, and that these problems could be mitigated by systematically managing laboratory processes to assure the quality and integrity of data and records. Quality assurance programs are designed to establish the criteria for assessing and improving laboratory performance, and to ensure that best practices are routinely identified and adopted across laboratory activities.
Task 2 was to describe the laboratory protocols, analytical procedures and standards, and data management processes for laboratories at the USGS, other federal agencies, and geological surveys in other countries. Evaluating analytical and data management procedures in hundreds of laboratories was not feasible in the confines of this study. Consequently, the USGS asked the committee to focus on laboratory QMS procedures in order to provide a benchmark for its QMS effort. The committee invited eight organizations that are using a QMS for at least some of their laboratories to share their approach and experiences at open meetings. The organizations chosen were the USGS, Navy, Centers for Disease Control and Prevention, Environmental Protection Agency, the French and Norwegian geological surveys, Texas A&M University, and Duke University Medical Center. Each of these organizations had different motivations for developing a QMS, different QMS challenges (e.g., number and
diversity of laboratories and laboratory activities), and different QMS implementation strategies (e.g., top down or bottom up, and fast or slow).
Despite these differences, some common themes emerged. The presenters told the committee that implementing a QMS provides benefits, such as improving documentation, reliability, or reproducibility of laboratory data; finding and correcting data quality problems; and enhancing the organization’s reputation for quality data. However, these benefits come with substantial monetary and personnel costs. The high costs and paperwork burden associated with implementing a QMS, as well as the need to learn a new way of doing things can create resistance among laboratory scientists and staff. Institutional commitment and strong leadership are required to gain buy-in and to change the organization’s culture. Consistent messaging is important for explaining why a QMS is needed, and good two-way communication between managers and laboratory staff is essential for developing a QMS that meets the needs of the laboratories. Finally, implementing the QMS slowly allows the system to evolve in response to lessons learned and thus ensure that the system fulfills its intended purpose.
Task 4 was to develop criteria for assessing protocols and procedures used by the organizations in Task 2, and use them to identify relevant best practices and procedures for USGS production laboratories. Criteria for assessing laboratory procedures are typically developed by managers, regulatory agencies, or professional organizations that offer laboratory inspection and accreditation programs. However, numerous assessments of laboratory procedures and recommended best practices have been published, and the committee drew from these to address Task 4.
A variety of approaches can be taken to assure data quality in laboratories. Approaches range from highly autonomous scientific oversight programs designed to meet individualized requirements to centrally controlled quality management systems designed to meet the requirements of an organization. Examples of approaches relevant to the USGS include the following:
- Scientist-defined procedures and protocols that are implemented at the individual laboratory level. With this approach, scientists have the autonomy and flexibility to be creative and innovative in developing their laboratory methods. However, the practices may be ad hoc, may be highly variable across laboratories, or may not cover quality planning, quality control, and quality improvements for all processes that contribute to data quality. This approach is common in academic laboratories and was used by most USGS laboratories prior to 2016.
- Institution-defined best practices that are implemented at the individual laboratory level. Best practices are procedures that have been shown by research and experience to produce optimal results and that are established or proposed as a standard for widespread adoption. With this approach, institutional management establishes its general
expectations for how data quality should be maintained and demonstrated, and the laboratory lead scientist develops a program to meet those expectations. Having standardized expectations for data quality should improve the consistency, reliability, and efficiency of processes across the laboratory system. However, implementation requires more time, effort, training, and oversight than the previous approach. A variety of generalized guidelines and best practices for implementing this approach are available.
- Institution-defined QMS requirements that are implemented throughout the institution. This centralized approach is used to achieve consistency, efficiency, and a shared quality culture across the laboratory network. However, it increases cost because quality assurance professionals are needed to coordinate and monitor activities (document control, change control, personnel, equipment, methods, materials, error management, and internal or external audits) across the organization, and staff require training and support to take on additional quality assurance activities. The USGS is implementing this approach.
- Externally-defined QMS requirements that are implemented at the institution or individual laboratory level to demonstrate compliance with an external quality standard. With this approach, an institution adopts an external standard to carry out a particular line of work or to meet the specific requirements of clients, collaborators, or regulatory agencies. It demonstrates a high level of research accountability, but expensive, periodic external reviews (audits) are required for laboratories to maintain accreditation. The USGS National Water Quality Laboratory is using this approach.
Approaches 2 through 4 are examples of organization-wide quality assurance programs, which describe the activities put in place to meet the requirements and expectations of a data quality standard. Implementation of a quality assurance program is a recommended best practice because it is systematic, process oriented, and addresses all aspects of the work being done. Approach 1 does not take this system-wide approach to data quality.
The committee’s last two tasks concern best practices and procedures for achieving scientific and applications objectives and assuring the integrity and reliability of results for USGS laboratories. Task 5 was to recommend best practices for production laboratories (those carrying out routine analyses for USGS or external users), and Task 6 was to comment on best practices for research laboratories (those supporting innovation or scientific discovery).
Tasks 5 and 6: Best Practices
Institution-defined expectations of data quality are important for generating data of known and consistent quality across large organizations such as the USGS, which has to manage some 250 diverse laboratories around the country. The USGS is already implementing one
type of institution-defined approach (QMS; step 3 in Figure S.1) for its laboratories. This is a good fit for laboratories that carry out well-characterized and routine analyses for internal or external users (production activities). A few of these laboratories may also need to meet additional externally-defined QMS requirements (step 4 in Figure S.1) for some procedures, based on client requirements.
Approximately half of USGS laboratories are used primarily by researchers. In these laboratories, methods are frequently adjusted as research hypotheses unfold, or as the process of optimization and validation proceeds. Creative experimentation is necessary before processes can be standardized. For these laboratories, institution-defined best practices (step 2 in Figure S.1) are appropriate to accommodate continually developing and improving analytical procedures while still meeting the best practice guidelines chosen by the USGS. Moving from scientist-defined procedures to institution-defined best practices would mean the research-oriented laboratories would fully participate in a centralized USGS laboratory culture committed to accountability and data quality and integrity. Adding periodic independent data quality checks (e.g., peer review, internal audits, and sample exchange with external laboratories) would confirm that institution-defined best practices have become routine in research and method development laboratories at the USGS.
Few USGS laboratories support only research activities or only production activities. Consequently, the USGS, in consultation with its laboratories and their users, will have to decide which laboratories need a QMS and which need institution-defined best practices.
Recommendation 1. The USGS should implement institution-defined best practices (step 2) or institution-defined QMS (step 3), as appropriate, for its laboratories.
A key responsibility of management is to support implementation and maintenance of the quality assurance program. However, current USGS resource commitments, quality assurance staffing, and training are insufficient to implement the central QMS for all USGS laboratories. The USGS expects its laboratories will devote an estimated 20 percent of their resources for about 2 years to implement the QMS and about 10 percent annually thereafter to maintain the system. This substantial effort should be recognized, supported, facilitated, and rewarded by USGS management.
Institution-defined best practices are less expensive to implement than a comprehensive QMS. Consequently, implementing institution-defined best practices for the laboratories focused primarily on research would free up central USGS resources to support QMS implementation and maintenance for the subset of laboratories engaged in production activities.
Recommendation 2. The USGS should optimize and prioritize centralized resources for the subset of laboratories doing production activities that would most benefit from a QMS.
The USGS is developing and implementing its QMS too quickly. QMS development began in 2016 and the system was implemented in 11 energy laboratories in mid-2017. Quality assurance programs such as QMS and institution-defined best practices are relatively new concepts in most academic and government research environments. Such systems are complex and take time to develop, implement, and evolve. The USGS will need to take the time to
- Communicate more extensively with staff, including explaining the quality goals of the organization and gaining staff input and feedback on system design and implementation;
- Provide staff training, including meetings with quality assurance experts;
- Establish mechanisms to recognize, support, and reward the substantial time and resources invested by laboratory scientists and quality assurance experts to meet USGS data quality goals;
- Develop QMS champions who would help lead the necessary culture change; and
- Learn from implementation experiences and continually improve the system.
Recommendation 3. The USGS should slow implementation of its QMS and allow ample time to develop institution-defined best practices, take advantage of lessons learned, provide training, and obtain input and buy-in from USGS laboratory staff.
The committee commends the USGS for pursuing recognized best practices to produce data of known and documented quality. A well-resourced and gradual implementation of a flexible approach that incorporates institution-defined best practices for research activities and QMS for production activities would meet the quality goals of the USGS and the diverse needs of its laboratories, foster staff buy-in, and cultivate an enduring quality culture across the agency.