The previous chapters described the U.S. Geological Survey’s (USGS’s) motivations for developing a quality management system (QMS) for its laboratories (Chapter 1), examples of approaches used to assure the quality of laboratory data (Chapter 2), current USGS laboratory quality assurance practices (Chapter 3), and the experiences of federal agencies, research institutions, and geological surveys in implementing a QMS (Chapters 3 and 4). The committee drew on this information to develop recommendations and conclusions on quality assurance programs for the USGS.
This chapter recommends best practices for USGS production and research laboratories (Tasks 5 and 6), the allocation of resources for implementing quality assurance programs, and the timeline for their implementation.
The committee’s last two tasks concern best practices and procedures for achieving scientific and applications objectives and assuring the integrity and reliability of results for USGS laboratories. Task 5 was to recommend best practices for production laboratories (those carrying out routine analyses for USGS or external users), and Task 6 was to comment on best practices for research laboratories (those supporting innovation, method development, or scientific discovery). Below, the committee uses the step diagram introduced in Chapter 2 and simplified in Figure 5.1 to illustrate best practices for both types of laboratories.
Before QMS development began in 2016, most USGS laboratory principal investigators defined their own quality assurance procedures and best practices (step 1 in Figure 5.1), similar to the practices of academic laboratories (see Chapter 3). The other laboratories had quality assurance procedures more consistent with a QMS (step 3), although some QMS elements might be missing, and a few met externally-defined QMS requirements (step 4). The current USGS plan is to require that all of its laboratories follow institution-defined QMS requirements (Step 3). This change would mean a substantial increase in management oversight, resources, and requirements for most USGS laboratories.
Institution-defined expectations of data quality are important for generating data of known consistent quality across large organizations such as the USGS, which has to manage some 250 diverse laboratories across the country. The committee found two approaches that would meet this goal for most USGS laboratory activities: institution-defined best practices (step 2) and institution-defined QMS (step 3).
For laboratories focused on supporting research and method development, which need the flexibility to innovate and experiment, institution-defined best practices (step 2) are appropriate. This approach would allow the USGS to set general expectations for how its research-oriented laboratories are managed, while providing flexibility to principal investigators to develop a program that meets those expectations. Examples of how this might work are given in Box 5.1. Moving from step 1 to step 2 would retain the ability of these laboratories to experiment and innovate, while fully participating in a centralized USGS laboratory culture committed to accountability and data quality and integrity. Adding periodic independent data quality checks (e.g., peer review and internal audits) would confirm that institution-defined best practices have become routine in research and method development laboratories at the USGS. This innovative approach would also provide a leadership opportunity for the USGS, since relatively few large government organizations or academic institutions have attempted such broad-scale implementation of a quality assurance program in basic research settings.
In contrast, laboratories that carry out well-characterized and routine analyses for internal or external users (production activities) would benefit from an institution-defined QMS (step 3), which the USGS is already implementing. Because of organizational objectives or user demands, these laboratories typically need to meet the most stringent strategies for maintaining and demonstrating data quality. A few of these laboratories (e.g., the National Water Quality Laboratory) may also need to meet externally-defined QMS requirements (step 4) for some procedures.
All USGS laboratories support research and perhaps half also carry out production activities. Available data do not distinguish which laboratories do production work or in
what proportion. The USGS, in consultation with its laboratories and their users, will need to decide which laboratories need a centrally controlled QMS and which need institution-defined best practices.
Recommendation 1. The USGS should implement institution-defined best practices (step 2) or institution-defined QMS (step 3), as appropriate, for its laboratories.
A key responsibility of management is to support implementation and maintenance of the quality assurance program. However, current USGS resource commitments, quality assurance staffing, and training are insufficient to implement a centrally controlled QMS for all USGS laboratories, especially on a fast timeframe. The USGS plans to cover the shortfall by splitting costs with its laboratories: the USGS will provide central training and quality assurance experts, and the laboratories will devote an estimated 20 percent of their resources for about 2 years to implement the QMS and about 10 percent annually thereafter to maintain the system. The USGS had considered having as many as one quality assurance expert for every 30 laboratory staff.1 However, it has been able to hire only a half dozen quality assurance experts, not nearly enough to support QMS implementation for all of its laboratories. These personnel shortages and financial costs were a common concern of staff in the USGS laboratories the committee visited (see Chapter 3).
An additional reason for adopting the two quality assurance programs recommended above is that doing so would reduce the immediate resources required because it is less expensive to implement institution-defined research best practices than it is to implement a centrally administered QMS. Implementing institution-defined best practices for the laboratories focused primarily on research would free up central USGS resources to support QMS implementation and maintenance for the subset of laboratories engaged in production activities.
Recommendation 2. The USGS should optimize and prioritize centralized resources for the subset of laboratories doing production activities that would most benefit from a QMS.
The USGS is embarking on one of the largest QMS efforts attempted by a national research organization. The organizations with smaller QMS programs that the committee
1 The language of this sentence was modified after release of the publication version to reflect current USGS staffing plans.
consulted took substantial time (years) to implement a QMS and create a quality culture. It will likely take longer for the USGS to achieve these goals, given the scale and complexity of the USGS QMS effort. However, the USGS is developing and implementing its QMS more quickly. QMS development began in 2016, and the system was implemented in 11 energy laboratories in mid-2017. The USGS’s goal is to complete QMS implementation in all USGS laboratories in 2024. Both QMS and institution-defined best practices are relatively new concepts in academic and government research environments and require the development of new systems to support production of data of known and documented quality. In addition, training programs are needed to increase knowledge and support the development of quality assurance skills. Such systems take time to develop, implement, and evolve.
Adopting QMS and institution-defined best practices requires a gradual shift in laboratory and agency culture and is more likely to succeed with buy-in from USGS staff at all levels of the organization. The USGS will need to take the time to gain substantive input and feedback from laboratory principal investigators, laboratory managers, quality assurance experts, and other organizations using these approaches to develop the systems, as well as to use lessons learned to make course corrections. Slowing implementation of the QMS would also make resources go further.
Recommendation 3. The USGS should slow implementation of its QMS to allow ample time to develop institution-defined best practices, take advantage of lessons learned, provide training, and obtain input and buy-in from USGS laboratory staff.
The USGS is to be commended for pursuing recognized best practices to produce data of known and documented quality. The agency has chosen to implement a centrally controlled QMS to achieve consistency and a shared quality culture across its approximately 250 analytical laboratories. The QMS was developed from existing QMSs in laboratories doing routine production work (e.g., National Water Quality Laboratory). However, approximately half of USGS laboratories support exploratory research and method development, which have fewer established procedures and require the flexibility to make frequent changes as the methods and analyses are refined. Concerns that a QMS would restrict their autonomy, flexibility, or creativity were raised by staff in a number of USGS laboratories. Moreover, the current QMS implementation plan is unrealistic, given resource and time constraints. Few large research organizations have attempted to establish a QMS under these conditions.
The USGS has an additional option (institution-defined best practices) that would achieve its quality goals, retain laboratory abilities to experiment and innovate, and reduce resource needs. For laboratories focused primarily on supporting research, the USGS
should establish expectations for maintaining and demonstrating data quality, and allow each laboratory principal investigator to develop data quality procedures that meet those expectations. Because institution-defined best practices are less expensive to implement than a QMS, more resources could be directed to laboratories that carry out substantial production activities requiring a QMS.
For both approaches, ample time is needed to
- Communicate with staff, including explaining the quality goals of the organization and gaining staff input and feedback on system design and implementation;
- Provide staff training, including meetings with quality assurance experts;
- Establish mechanisms to recognize, support, and reward the substantial time and resources invested by laboratory scientists and quality assurance experts to meet USGS data quality goals;
- Develop QMS champions who would help lead the necessary culture change; and
- Learn from implementation experiences and continually improve the system.
This flexible approach that incorporates institution-defined best practices for research activities and QMS for production activities would better meet the quality goals of the USGS and the diverse needs of its laboratories, foster staff buy-in, and cultivate an enduring quality culture across the agency.