4
Data Management and Products

QUALITY ASSURANCE/QUALITY CONTROL

The scope and complexity of the monitoring requirements of the Restoration Plan warrant the direction of special attention and resources to the development, implementation and maintenance of a quality assurance/quality control (QA/QC) program. According to a report of the NRC (1996a), “Currently, a great deal of monitoring data is collected in the United States. However, the data are incomplete…of varied quality, and non-standardized in collection protocol.” In the case of the Restoration Plan, the potential for these undesirable outcomes is perhaps increased because of multiple or vague monitoring goals or requirements (e.g., testing research hypotheses versus meeting regulatory requirements); involvement of numerous parties in plan development and implementation (federal and state agencies, private sector firms and universities); massive data storage requirements; and the extended “life expectancy” of the project.

As Restoration Plan participants are aware, a successful QA/QC program should consist of various components including planning, implementation, assessment, and reporting. In fact, the Environmental Protection Agency requires that all projects within its regulatory purview develop an approved Quality Assurance Project Plan (QAPP) before project implementation (US EPA, 1998). (This presents a practical dilemma for the Restoration Plan since baseline data have been obtained from numerous sources over a number of years; acquisition of such data continues in the absence of a project-wide QAPP.) Data-quality objectives (DQOs) should be established during the planning stage of the program and used to develop measurable performance criteria. Successful planning will allow managers to identify financial, personnel, and information technology resources needed for implementation, assessment and reporting. Obviously, if this model of a QA/QC program is deemed appropriate for the CERP, considerable effort must be devoted to the development of a QA/QC program. Since monitoring performance indicators and QA/QC performance objectives are intimately related, adding DQOs incrementally to an existing monitoring plan could prove wasteful and inefficient.

As important as it is for the Restoration Plan to develop a comprehensive QAPP, it is just as important that the plan have the flexibility to accommodate the requirements of various aspects of the monitoring program. The actual data and information needs of the Restoration Plan should drive the development of the QAPP as opposed to vice versa. Whereas a rigorous



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 47
4 Data Management and Products QUALITY ASSURANCE/QUALITY CONTROL The scope and complexity of the monitoring requirements of the Restoration Plan warrant the direction of special attention and resources to the development, implementation and maintenance of a quality assurance/quality control (QA/QC) program. According to a report of the NRC (1996a), “Currently, a great deal of monitoring data is collected in the United States. However, the data are incomplete…of varied quality, and non-standardized in collection protocol.” In the case of the Restoration Plan, the potential for these undesirable outcomes is perhaps increased because of multiple or vague monitoring goals or requirements (e.g., testing research hypotheses versus meeting regulatory requirements); involvement of numerous parties in plan development and implementation (federal and state agencies, private sector firms and universities); massive data storage requirements; and the extended “life expectancy” of the project. As Restoration Plan participants are aware, a successful QA/QC program should consist of various components including planning, implementation, assessment, and reporting. In fact, the Environmental Protection Agency requires that all projects within its regulatory purview develop an approved Quality Assurance Project Plan (QAPP) before project implementation (US EPA, 1998). (This presents a practical dilemma for the Restoration Plan since baseline data have been obtained from numerous sources over a number of years; acquisition of such data continues in the absence of a project-wide QAPP.) Data-quality objectives (DQOs) should be established during the planning stage of the program and used to develop measurable performance criteria. Successful planning will allow managers to identify financial, personnel, and information technology resources needed for implementation, assessment and reporting. Obviously, if this model of a QA/QC program is deemed appropriate for the CERP, considerable effort must be devoted to the development of a QA/QC program. Since monitoring performance indicators and QA/QC performance objectives are intimately related, adding DQOs incrementally to an existing monitoring plan could prove wasteful and inefficient. As important as it is for the Restoration Plan to develop a comprehensive QAPP, it is just as important that the plan have the flexibility to accommodate the requirements of various aspects of the monitoring program. The actual data and information needs of the Restoration Plan should drive the development of the QAPP as opposed to vice versa. Whereas a rigorous

OCR for page 47
set of EPA or other guidelines might apply well to regulatory required monitoring, a different approach could be applied to monitoring designed for research or hypothesis testing. The QA/QC applied to experimental methods, documentation, etc. might be similar in both cases, but validation in the latter case could extend well beyond statistical accuracy and precision of experimental data. Ultimately, the inferences made from hypothesis-testing data will be evaluated by the scientific peer-review process. This suggests that the QAPP should include methods for both selecting entities to conduct research-based inquiries and evaluating the conclusions reached from these investigations. Even in cases in which routine monitoring data are used to impact management decisions, the quality of these decisions should be scrutinized by the appropriate component of the QAPP. For example, the Restoration Plan needs to include an objective process for selecting, designing, implementing, and evaluating field experiments and modeling exercises. Consideration should be given to the process of selecting (or certifying) individuals or teams to engage in research projects (competitive solicitations might be one mechanism). There should also be a process for evaluating experimental results, i.e., peer review. Most important, the plan should address how these results will be used in the decision-making process. DATA AND INFORMATION MANAGEMENT The Restoration Plan’s adaptive management strategy cannot succeed without a well-designed and adequately supported data and information system. Given the complexity and duration of the Restoration Plan, desirable features of such a system include the following: clear data and metadata policies and standards; policies and procedures for data validation; mechanisms to ensure Restoration Plan data integrity and security; mechanisms for inter-organizational data and information sharing; policies and procedures for public information access and outreach; database software and database models to facilitate storage and retrieval; tools to facilitate data analysis and learning through shared computing hardware and software resources; and human and technological capacity to maintain a growing and increasingly complex store of data and information. The Restoration Plan Project Management Plan for Data Management (USACE and SFWMD, 2002a) calls for a program-wide phased approach to management and acquisition of data, including activities to “identify, standardize, organize, document, serve and preserve program data.” This document is mainly concerned with identifying relevant standards for Restoration Plan data. Some federal data and metadata standards are identified for GIS, Computer Aided Design and Drafting, and survey data. The plan calls for an “enterprise Geographic Information System (GIS)” consisting of a central repository of spatial data gathered and used by multiple organizations based on agreed-upon standards. The plan also calls for establishment of a Data Clearinghouse, a Data Oversight Committee, and a program to bring existing data into the Restoration Plan’s common spatial framework. In summary, the plan partially addresses items 1-4 above. Technical and logistical details of data management (items 5-7) are to be addressed in the next phase of data management activities.

OCR for page 47
Section 10 of the Project Management Plan for Data Management specifically addresses RECOVER data, but only in very general terms that hint at but do not provide substantive solutions to the large technical and institutional challenges to implementing an effective data and information management system for adaptive assessment in the Restoration Plan. Data relevant to the Restoration Plan’s adaptive assessment will be gathered at thousands of locations by perhaps hundreds of organizations and individuals. These include physical, biological and socioeconomic data gathered over many scales using a wide array of methods. Adaptive assessment depends on integrating these data across space and time for exploration, visualization, statistical and simulation modeling, and performance monitoring. At this stage, Restoration Plan data and information activities are largely at the conceptual stage. However, based on the material produced to date, two general concerns arise: Substantive work on data and information management appears to be lagging well behind other aspects of the Restoration Plan in general and the MAP in particular. It seems that inadequate attention and resources are being committed to this component of the Restoration Plan. Many if not most data relevant to the Restoration Plan’s adaptive assessment will be collected for other purposes by non-Restoration Plan personnel. The Restoration Plan strategy for data and information management strategy should consider moving beyond centralized databases, rigid standards, and data clearinghouses. The Restoration Plan could exert leadership by creating mechanisms and providing funding for promoting confederation of databases that are controlled and maintained locally by participating organizations and individuals. Mechanisms include development and/or dissemination of tools that promote good data management practices and shared data and metadata syntax and semantics. For example, Jones et al. (2001) described a network-enabled database framework for research ecologists that allows individual scientists to customize metadata to meet their needs while also promoting the use of standards such as the U.S. Ecological Metadata Standard and National Biological Information Infrastructure’s Biological Data Profile. SYNTHESIS OF DATA Finally, synthesis of monitoring data and preparation of adaptive-assessment reports should be a prominent feature of the MAP. While development of these portions of the MAP is still in its infancy, as these sections of the plan begin to be formulated the following questions should be addressed: How often should formal reviews of the Restoration Plan performance be conducted? Are there ecosystem responses that will trigger a formal review in addition to scheduled reviews? Is sufficient time for data analysis and synthesis built into the assessment process? How will independent peer-review of the data collection and synthesis be conducted? Who will insure that monitoring results be incorporated into the implementation and operation of the Restoration Plan, and how will that be accomplished? These issues are critical to successfully incorporating adaptive assessment into the Restoration Plan to insure that the MAP does not become “data rich and information poor,” a problem common to many monitoring and assessment projects.

OCR for page 47