National Academies Press: OpenBook

Assuring Data Quality at U.S. Geological Survey Laboratories (2019)

Chapter: 4 Experiences with Quality Management Systems

« Previous: 3 USGS Laboratories
Suggested Citation:"4 Experiences with Quality Management Systems." National Academies of Sciences, Engineering, and Medicine. 2019. Assuring Data Quality at U.S. Geological Survey Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/25524.
×

4

Experiences with Quality Management Systems

The committee’s second task was to describe the analytical procedures and data management practices in laboratories at the U.S. Geological Survey (USGS), other federal agencies, and geological surveys in other countries. Because the USGS is implementing a quality management system (QMS), it asked the committee to focus on laboratory QMS procedures. The data quality assurance practices of USGS laboratories are summarized in Chapter 3. This chapter focuses on the experiences of selected organizations in implementing and maintaining a QMS.

The committee invited eight organizations to share their approach and experiences with a QMS. The committee heard from four U.S. federal agencies with a substantial number of analytical laboratories using a QMS: the USGS, the Navy’s Environmental Laboratory Accreditation Program (ELAP) and Naval Sea Systems Command (NAVSEA) Shipyard Laboratory Program, the Centers for Disease Control and Prevention (CDC), and the Environmental Protection Agency (EPA). A QMS is less common in geological surveys and is rare in academic institutions. Two of five geological surveys from other nations that the committee consulted (France and Norway) use a QMS. Finally, the committee heard from the Geochemical and Environmental Research Group (GERG) Laboratory at Texas A&M University and a group of laboratories under quality assurance oversight of Duke University Medical Center that use a QMS. This group includes laboratories at Duke and other academic institutions. For simplicity, they are referred to in this chapter as the Duke University Medical Center laboratories.

This sample of organizations is small and cannot be considered representative of all organizations. However, the committee sought diversity in terms of scientific focus, number of laboratories under the QMS umbrella (see Table 4.1), and position of the presenters. Presentations by federal agencies were given by QMS-related managers, and presentations by geological surveys and academic institutions were given by scientists or laboratory managers. The diverse perspectives and experiences of these organizations are highly relevant to the USGS, which is incorporating lessons learned into its own QMS.

Suggested Citation:"4 Experiences with Quality Management Systems." National Academies of Sciences, Engineering, and Medicine. 2019. Assuring Data Quality at U.S. Geological Survey Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/25524.
×

TABLE 4.1 Scientific Focus and Number of Laboratories in the Presenting Organizations

Organization Number of Laboratories with QMS Number of Staff Scientific Focus
USGS ~250 planned ~1,300 Geology, hydrology, and ecosystems
CDC ~200 proposed ~1,700 Infectious disease
Navy 85 ELAP NA Environmental restoration and ship materials
5 NAVSEA ~130
EPA NA (100 percent) NA Public health and the environment
Duke University Medical Center 16 NA Human clinical trials
Texas A&M GERG Laboratory 1 15 Contaminants, petroleum markers, and metals
French Geological Survey 1 77 Geophysics and the environment
Norwegian Geological Survey 1 18 Geology

NOTE: ELAP = Environmental Laboratory Accreditation Program; NA = not available; NAVSEA = Shipyard Laboratory Program.

PERSPECTIVES ON QMS FROM SELECTED FEDERAL AGENCIES, GEOLOGICAL SURVEYS, AND RESEARCH INSTITUTIONS

Each organization was asked to address the following points in their presentation:

  • Why did you establish a QMS?
  • Describe your current QMS.
  • Is a QMS applied in all of your internal laboratories?
  • How is the QMS resourced, managed, and assessed?
  • Challenges in initially integrating QMS (what did you learn?)
  • Challenges in maintaining the QMS (what are you learning?)
  • Has the QMS evolved significantly and why?
  • Has the QMS added value (e.g., examples where it either thwarted significant problems, saved money, created efficiencies, or produced defensible results that held up to scrutiny)?
  • Opportunities or needs for interagency collaboration on developing consensus-based approaches for QMS in nonregulated laboratories.
Suggested Citation:"4 Experiences with Quality Management Systems." National Academies of Sciences, Engineering, and Medicine. 2019. Assuring Data Quality at U.S. Geological Survey Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/25524.
×

The diverse perspectives of these organizations and common themes that emerged from the presentations are summarized below.

  1. Organizations have different motivations for having a QMS, and they find different benefits in it.

    All four federal agencies presenting had developed a QMS to improve the quality, reliability, and reproducibility of laboratory data. In addition, the USGS, EPA, and CDC sought to preserve or enhance their reputation for providing high-quality data. USGS data support policy making and CDC data support public health, and so high-quality data are needed to avoid unaccounted-for risk and error and to maintain public trust. EPA data support regulation, and so the data must be scientifically and legally defensible. The French and Norwegian geological surveys also cited enhancing customer confidence in laboratory results as a key motivation for implementing a QMS. The Navy mentioned the benefit of a QMS for continually improving processes.

    Some laboratories adopted a QMS to pursue contract work and research projects in fields that require compliance with specific data standards. The Texas A&M GERG laboratory complies with EPA standards for work on contaminants (e.g., pharmaceuticals, pesticides, and dioxin), petroleum biomarkers, and metals. The Duke University Medical Center laboratories comply with National Institutes of Health requirements to support clinical trials.

    A key QMS benefit identified by many of the presenters was the value of good documentation. A documented, step-by-step quality assurance process allows consistent performance, data verification, and proof that accepted protocols were followed. The need for consistent performance was emphasized by the Norwegian Geological Survey, where multiple team members carry out analyses and need uniform standards for quality assurance and quality control. The value of documentation for retaining institutional memory through staff turnover was noted by staff in several laboratories visited by the committee (see Chapter 3).

    Several presenters noted that a QMS does not prevent all data quality problems, but it can leave clues that help laboratory staff identify and respond to problems relatively quickly. Some of the problems mentioned concerned sample or instrument contamination, and the loss of original data through machine overwriting or human editing. The built-in external audit feature of a QMS is intended to catch problems. For example, an external audit of USGS Energy Mission Area laboratories identified a data quality problem related to anion concentrations measured using ion chromatography (Meeks et al., 2018).

Suggested Citation:"4 Experiences with Quality Management Systems." National Academies of Sciences, Engineering, and Medicine. 2019. Assuring Data Quality at U.S. Geological Survey Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/25524.
×
  1. Some organizations are not implementing QMS for all their laboratories.

    Four of the presenting organizations have implemented or plan to implement QMS for all of their laboratories. EPA has one QMS that covers EPA organizations as well as other organizations that receive EPA funds, but allows flexibility in how these external organizations implement the system. The CDC is slowly implementing QMS through pilot programs that recruit successively more laboratories. Each CDC laboratory in the pilot program chooses from three QMS options: standard, customized, or both. The USGS has begun implementing a single QMS for all of its laboratories. The Norwegian Geological Survey has a QMS for its one laboratory.

    The other organizations are implementing a QMS for only part of their enterprise. The Navy laboratories that operate under the Environmental Laboratory Accreditation Program and NAVSEA Shipyard Laboratory program require a QMS, but other Navy laboratory programs do not. The French Geological Survey laboratories are accredited for some environmental chemistry and mineral characterization activities, but not for isotope geochemistry or multiscale experimentation activities. The Texas A&M GERG laboratory embraced a flexible approach to support its diversity of programs. It maintains a QMS for most contract work, but not for research and development projects. Half of the 32 laboratories under the quality assurance oversight of Duke University Medical Center use the QMS.

  2. A QMS requires substantial resources. The cost of implementation and maintenance may be covered by the organization or by its laboratories.

    The USGS has found that its laboratory staff spent about 20 percent of their time to implement the QMS. This figure is in line with the experiences of other presenters—who estimated that their laboratories spent 10 to 20 percent of their time implementing a QMS and 5 to 10 percent thereafter maintaining it—and with published time estimates (e.g., Vermaercke, 2000). A substantial part of the investment involves record-keeping requirements. Laboratory Information Management Systems or electronic QMS provide a higher level of traceability than traditional spreadsheets for data management, but they may require procurement and maintenance funding or dedicated staff. External audits required in some QMSs can also be costly.

    The organizations have different approaches for paying the costs of the QMS. The French and Norwegian geological surveys fund their QMS centrally. The USGS and the CDC pay for training and support all quality assurance staff. No additional funding is provided to their laboratories for QMS implementation and maintenance. The laboratories under QMS programs at the Navy, EPA, Texas A&M GERG, and Duke University Medical Center pay all QMS costs, including training and quality

Suggested Citation:"4 Experiences with Quality Management Systems." National Academies of Sciences, Engineering, and Medicine. 2019. Assuring Data Quality at U.S. Geological Survey Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/25524.
×
  1. assurance specialists. The Duke University Medical Center laboratories are also covering information technology support for their electronic QMS.

  2. Implementation and maintenance of an effective QMS requires institutional commitment, leadership, and buy-in from the staff.

    A point made by several presenters was the role of leadership and culture in implementing a QMS. Initial resistance from laboratory staff was not unusual, and a substantial amount of institutional change was often required. The Texas A&M GERG Laboratory presenter noted that implementing a QMS was a huge undertaking, and getting staff to buy into the QMS was a major challenge. The Navy encountered resistance from laboratory staff, particularly those in smaller laboratories, when a QMS was introduced. The Norwegian Geological Survey stressed the importance of ongoing leadership support and laboratory staff awareness for maintaining their QMS.

    EPA leadership set out to persuade scientists that a QMS supports science, rather than interferes with it. Some of the desired cultural shift toward a QMS occurred as management and scientists made decisions together. At the Duke University Medical Center, the participating laboratories were involved from the beginning to change attitudes and create buy-in. The CDC also involved agency managers and scientists in developing laboratory quality management policies. The well-respected principal investigators in the pilot laboratories served as “QMS champions,” who were able to demonstrate the feasibility and benefits of a QMS and thus help facilitate cultural shift. The Navy accelerated culture change by first showing its laboratories a functioning QMS that yielded benefits, then requiring the laboratories to adopt it.

  3. Effective two-way communication between laboratories and management is important for developing and evolving a QMS.

    Presenters pointed out that communication between laboratories and management is important for explaining why a QMS is needed, creating buy-in, developing a QMS that meets the needs of the laboratories, and evolving the QMS to accommodate new equipment or meet new user needs. The USGS found communication a challenge, given the complexity of the organization and the rapid development and implementation of its QMS. USGS leadership is working to provide training and communication plans that focus on why a QMS is necessary. The CDC noted the importance of consistent messaging about the QMS from both leadership and QMS staff. EPA found it helpful to keep quality assurance jargon to a minimum. The French Geological Survey noted the importance of reminders of why the QMS was put in place, especially in light of staff turnover.

Suggested Citation:"4 Experiences with Quality Management Systems." National Academies of Sciences, Engineering, and Medicine. 2019. Assuring Data Quality at U.S. Geological Survey Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/25524.
×
  1. Starting small and taking advantage of lessons learned helps the system evolve over time.

    The Navy’s NAVSEA Shipyard Laboratory Program experienced significant growing pains, including a struggle to develop documents and maintain records, as it implemented a top-down QMS for its laboratories. The USGS faced a steep learning curve by developing and implementing a QMS on an accelerated schedule. In contrast, the CDC found that implementing a QMS slowly and in stages increased the likelihood of success because it allowed the agency to test and modify its approach, measure the necessary level of effort and investment, and demonstrate the feasibility and benefits of the QMS. It also provided an opportunity to develop guidance documents, templates, and expertise to support implementation across the agency. The number of laboratories under the Duke University Medical Center QMS expanded from 3 to 16 as it was determined that the research would benefit from standard operating procedures and QMS.

SUMMARY AND ANSWER TO TASK 2

Task 2 was to summarize data management practices, specifically QMS, for laboratories at the USGS, other federal agencies, and geological surveys in other countries. The committee chose eight organizations that had different motivations for developing a QMS for some or all of its laboratories, different QMS challenges (e.g., number and diversity of laboratories and laboratory activities), and different QMS implementation strategies (e.g., top down or bottom up, and fast or slow). Despite these differences, some common themes emerged. First, implementing a QMS provides benefits, such as improving documentation, reliability, reproducibility of laboratory data; finding and correcting data quality problems; and enhancing the organization’s reputation for quality data. However, these benefits come with substantial monetary and personnel costs. The high costs and paperwork burden associated with implementing a QMS, as well as the need to learn a new way of doing things, can create resistance among laboratory staff. Institutional commitment and strong leadership are required to gain staff buy-in and to change the organization’s culture. Consistent messaging is important in explaining why a QMS is needed, and good two-way communication between managers, quality assurance staff, and laboratory staff is essential for developing a QMS that meets the needs of the laboratories. Finally, implementing the QMS slowly allows the system to evolve in response to lessons learned and thus fulfills its intended purpose.

Suggested Citation:"4 Experiences with Quality Management Systems." National Academies of Sciences, Engineering, and Medicine. 2019. Assuring Data Quality at U.S. Geological Survey Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/25524.
×
Page 55
Suggested Citation:"4 Experiences with Quality Management Systems." National Academies of Sciences, Engineering, and Medicine. 2019. Assuring Data Quality at U.S. Geological Survey Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/25524.
×
Page 56
Suggested Citation:"4 Experiences with Quality Management Systems." National Academies of Sciences, Engineering, and Medicine. 2019. Assuring Data Quality at U.S. Geological Survey Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/25524.
×
Page 57
Suggested Citation:"4 Experiences with Quality Management Systems." National Academies of Sciences, Engineering, and Medicine. 2019. Assuring Data Quality at U.S. Geological Survey Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/25524.
×
Page 58
Suggested Citation:"4 Experiences with Quality Management Systems." National Academies of Sciences, Engineering, and Medicine. 2019. Assuring Data Quality at U.S. Geological Survey Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/25524.
×
Page 59
Suggested Citation:"4 Experiences with Quality Management Systems." National Academies of Sciences, Engineering, and Medicine. 2019. Assuring Data Quality at U.S. Geological Survey Laboratories. Washington, DC: The National Academies Press. doi: 10.17226/25524.
×
Page 60
Next: 5 Recommendations and Conclusions »
Assuring Data Quality at U.S. Geological Survey Laboratories Get This Book
×
Buy Paperback | $60.00 Buy Ebook | $48.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The U.S. Geological Survey (USGS) mission is to provide reliable and impartial scientific information to understand Earth, minimize loss of life and property from natural disasters, and manage water, biological, energy, and mineral resources. Data collection, analysis, interpretation, and dissemination are central to everything the USGS does. Among other activities, the USGS operates some 250 laboratories across the country to analyze physical and biological samples, including water, sediment, rock, plants, invertebrates, fish, and wildlife. The data generated in the laboratories help answer pressing scientific and societal questions or support regulation, resource management, or commercial applications.

At the request of the USGS, this study reviews a representative sample of USGS laboratories to examine quality management systems and other approaches for assuring the quality of laboratory results and recommends best practices and procedures for USGS laboratories.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!