National Academies Press: OpenBook
« Previous: 1 The Committee's First Report and Its Impact
Suggested Citation:"2 Evaluating Research Implementation and Progress." National Research Council. 1999. Research Priorities for Airborne Particulate Matter: II. Evaluating Research Progress and Updating the Portfolio. Washington, DC: The National Academies Press. doi: 10.17226/9646.
×

2
EVALUATING RESEARCH IMPLEMENTATION AND PROGRESS

In this chapter, the committee discusses plans for monitoring and evaluating independently the progress of research on airborne particulate matter (PM),1 responding to that element of its charge from Congress and the Environmental Protection Agency (EPA). Monitoring of research progress is particularly necessary in view of the broad scope and multidisciplinary character of the research program presented in the committee's first report (NRC 1998). As discussed in chapters I and 3 of this report, the first report's recommendations already appear to have had a substantial effect on research efforts of EPA and other agencies and organizations. Evaluating research plans and results requires the assessment of various levels of the research enterprise—ranging from individual research projects to broader programs—against research goals. The committee intends to review research progress for each of its ten recommended research topic areas, assessing research supported not only by EPA but by other agencies and organizations. The portfolio of research recommendations will also be updated periodically as new information is obtained (Chapter 3).

The committee formulated an approach for evaluating PM research progress that begins with the research-planning process and extends

1  

 Some researchers use the term "atmospheric aerosol" or "ambient aerosol" to refer to airborne particulate matter. An aerosol is a suspension of a solid or liquid particles in a gas.

Suggested Citation:"2 Evaluating Research Implementation and Progress." National Research Council. 1999. Research Priorities for Airborne Particulate Matter: II. Evaluating Research Progress and Updating the Portfolio. Washington, DC: The National Academies Press. doi: 10.17226/9646.
×

to evaluating the scientific knowledge and policy-relevant information that is obtained. As noted in the first report (NRC 1998), there are numerous uncertainties in the evidence available for airborne particulate matter, ranging from the source-receptor relationships to the biological mechanisms of injury (see Table 1.2). Effective management of risks related to airborne particulate matter will require research findings that significantly reduce uncertainties for the variety of atmospheric, exposure assessment, and health-effects questions relevant to standard setting and implementation.

The committee's first report (NRC 1998) presented three principal criteria for identifying research priorities for particulate matter—scientific value, decisionmaking value, and feasibility and timing. Cost was also considered. Those criteria guided the development of the committee's research portfolio, including the recommended research topics, estimated budgets, and approximate time frames for conducting research.

To reiterate, the criteria included

  • Scientific value of the information, judged in terms of knowledge that fills important data gaps, provides information on cause-and-effect relationships, builds on previous findings, and contributes to the development of an integrated understanding of the health effects of particulate matter and gaseous copollutants.

  • Decisionmaking value, or the usefulness of the results of particulate matter research findings to decisionmakers with a particular focus on reducing key uncertainties associated with regulatory standard setting and other risk-management decisions for the next review of the PM2.5 NAAQS in 2002 and beyond.

  • Timing and feasibility, the consideration that research needs to be operationally, technically, and financially feasible and conducted in a time frame responsive to decisionmakers' needs.

The committee concludes that the above criteria will serve equally well as measures for evaluating research progress, and it plans to apply them for that purpose.

Suggested Citation:"2 Evaluating Research Implementation and Progress." National Research Council. 1999. Research Priorities for Airborne Particulate Matter: II. Evaluating Research Progress and Updating the Portfolio. Washington, DC: The National Academies Press. doi: 10.17226/9646.
×

The committee now introduces three additional criteria relevant to evaluating the planning, management, and implementation of the recommended research:

  • Interaction—How well do the scientists involved in the research engage in collaboration across scientific disciplines, and are the scientific expertise, capacity and resources appropriately used to enhance scientific creativity, quality, and productivity?

  • Integration—How effectively is research planning, management, and budgeting integrated in a joint process involving governmental and private organizations?

  • Accessibility—How well is information about research plans, budgets, progress and results openly conveyed and shared among research organizations and other interested parties?

Interactions among investigators should be encouraged, and integration among organizations is needed to ensure that the return on the scientific investment is optimized. For example, sharing of experimental approaches and methods used in toxicological studies may avoid apparent confusion from the use of unnecessarily disparate experimental models. In addition, coordination of data-collection approaches and data bases improves the scientific basis for analysis by epidemiologists and statisticians. Early sharing of research plans and results across investigative groups can reduce potential confusion and can also stimulate advances in research protocols. Accessibility and dear communication of research plans and other information help to inform and assure stakeholders about the research program.

To achieve the overall objectives of the PM research portfolio recommended in the committee's first report, PM research must be viewed as a national effort, involving Congress and many agencies of the Executive Branch, as well as states and the private and nonprofit sectors. Integrated management of PM research is key to successful implementation of the research portfolio. Research on particulate matter is funded by several federal agencies, including EPA, Department of Energy, Department of the Interior, National Institute of Environmental

Suggested Citation:"2 Evaluating Research Implementation and Progress." National Research Council. 1999. Research Priorities for Airborne Particulate Matter: II. Evaluating Research Progress and Updating the Portfolio. Washington, DC: The National Academies Press. doi: 10.17226/9646.
×

Health Sciences, National Oceanic and Atmospheric Administration, and others. The scope of funded research on particulate matter is expanding rapidly. The growing diversity of sponsors and the many different kinds of scientific investigations under way add to the complexity of the task of monitoring research progress and increase the need for federal interagency coordination and management of the effort.

Some aspects of airborne PM research are identified and coordinated across federal agencies at the level of the National Science and Technology Council and the White House Office of Science and Technology Policy. The Air Quality Research Subcommittee of the interagency Committee on Environment and Natural Resources (CENR) can provide a focal point for information dissemination among agencies and facilitates opportunities for research planning, primarily on atmospheric science research issues.2

In contrast to the coordination of atmospheric research, effective mechanisms have not been apparent for integrating the overall management of exposure and health-effects research with the physical-science activities in atmospheric research for airborne particulate matter across the federal government. Instead, individual researchers and agencies have organized ad hoc meetings or have exchanged information through informal networks or at professional meetings. No effective planning process has existed within the federal government to integrate the interrelated aspects of atmospheric, exposure, and health-effects research planning and management. In the committee's view, it will be very difficult for the PM research program to achieve its most significant objectives in the absence of an effective structure or process to operate across the agencies and disciplines to ensure common awareness of research activities, to engage in joint planning or funding

2  

 CENR member departments and agencies include the Department of Agriculture, Department of Energy, Department of Health and Human Services, Department of Housing and Urban Development, Department of the Interior, Department of State, Department of Transportation, Environmental Protection Agency, National Aeronautics and Space Administration, National Science Foundation, Office of Management and Budget, Office of Science and Technology Policy, and the Tennessee Valley Authority.

Suggested Citation:"2 Evaluating Research Implementation and Progress." National Research Council. 1999. Research Priorities for Airborne Particulate Matter: II. Evaluating Research Progress and Updating the Portfolio. Washington, DC: The National Academies Press. doi: 10.17226/9646.
×

of individual projects, or to provide effective and comprehensive coordination. This problem is by no means unique to airborne PM research. Multiorganizational integration and multidisciplinary interaction have historically been elusive goals in research. The various research organizations and scientific disciplines typically have different interests, funding sources, and professional cultures.

The committee was pleased to hear at its November 1998 meeting of a proposal to expand the charter of the Air Quality Subcommittee of the CENR to more actively encompass all federally sponsored PM research, including health research. That expansion should be encouraged to promote greater coordination of the resources of the federal government on PM research. CENR should move, as well, to establish a formal mechanism to enlist the collaboration of the nonfederal research community. One of the initial goals of this process should be to prepare a coordinated interagency strategy for implementing this committee's research portfolio. This strategy should be independently peer reviewed and should include

  • A process and budget to implement the PM research portfolio recommended by the committee.

  • The specific methods that will be used to coordinate research across agencies on a continuing basis.

  • Strategies and mechanisms for leveraging funding with the federal sector, state governments, and the private and nonprofit sectors.

EPA is the agency with the largest mandate and budget for PM research. This NRC committee has received several briefings from staff of EPA's Office of Research and Development (ORD) and Office of Air Quality Planning and Standards (OAQPS) on the process of PM research planning and the status of individual research activities. Information obtained in those sessions, as well as the individual experiences of committee members in planning and managing research, provided a basis for analysis and conclusions by the committee.

The process of planning and managing PM research within EPA will undoubtedly produce many individual high-quality and policy-relevant

Suggested Citation:"2 Evaluating Research Implementation and Progress." National Research Council. 1999. Research Priorities for Airborne Particulate Matter: II. Evaluating Research Progress and Updating the Portfolio. Washington, DC: The National Academies Press. doi: 10.17226/9646.
×

research projects and results. EPA research administrators and staff have already demonstrated their willingness to use the committee's research recommendations as a template for the agency's PM research agenda. In the time since the committee's first report (NRC 1998), responsiveness on the part of EPA has been excellent. However, efforts to implement a multidisciplinary research program across the agency have revealed needs for improvement in the agency's overall research coordination approach for airborne particulate matter. Like the federal government overall, EPA has not had an effectively integrated structure or process through which to plan and implement the range of key atmospheric and health-related research across the scientific disciplines. In many cases, research within EPA is integrated among the disciplines through informal exchanges among researchers and ad hoc meetings and workshops. Although informal interactions are essential to the conduct of multidisciplinary research, the scope of the needed research on particulate matter requires a more formal process. Top EPA research and policy officials should participate and provide management guidance during all major decision points in planning, managing, implementing, and evaluating the PM research program.

EPA has begun to organize a formal process for integrating health and air-quality considerations in research planning for a key part of this program: PM chemical-speciation and supersites monitoring. At the committee's November 1998 meeting, in response to the recommendations of a July 1998 EPA/NARSTO workshop and comments raised by individual members of the committee at its September 1998 meeting, EPA presented a plan for a management approach developed by officials of the ORD and Office of Air and Radiation. The committee views this as a promising step toward integrated planning and management for these monitoring programs. The committee is also aware that EPA is coordinating plans for monitoring particulate matter and hazardous air pollutants. In addition, the agency needs to enhance its efforts to integrate the expertise and experience of the broader health, exposure, and atmospheric research communities into the ongoing design and implementation of the monitoring program.

In addition to an effective management structure, EPA's integrated planning process for research on particulate matter should include the following elements:

Suggested Citation:"2 Evaluating Research Implementation and Progress." National Research Council. 1999. Research Priorities for Airborne Particulate Matter: II. Evaluating Research Progress and Updating the Portfolio. Washington, DC: The National Academies Press. doi: 10.17226/9646.
×
  • Developing formally stated research objectives and time frames for each of the major research activities addressing particulate matter.

  • Articulating the specific disciplines and skills needed to achieve the research objectives.

  • Differentiating between the kinds of research that can be conducted intramurally and extramurally, including research best sponsored jointly with the private sector, other federal agencies, and nonprofit research institutions.

  • Building specific planning mechanisms to design and coordinate research projects within EPA and between EPA and other institutions. This coordination should include relevant activities from other research programs, such as those planned for hazardous air pollutants.

  • Maintaining an inventory of ongoing EPA and externally funded PM research projects. As stated in the committee's first report, such an inventory should be a continually updated status report of research in progress.

  • Preparing periodic reports synthesizing the status of research activities and the progress being made to reduce key uncertainties, such as those identified in the committee's research portfolio. Such reports should not be mere compendia of ongoing research, but rather integrated summaries that enable investigators actively engaged in PM research, the broader scientific community, policymakers, Congress, and the public to understand major accomplishments and to identify key challenges still to be met.

Each of the above elements of the integrated research-planning process should contain information on resources invested or needed to carry out the plans. In addition, all aspects of the integrated plan should provide quick and easy access to relevant summary information for all interested parties.

Suggested Citation:"2 Evaluating Research Implementation and Progress." National Research Council. 1999. Research Priorities for Airborne Particulate Matter: II. Evaluating Research Progress and Updating the Portfolio. Washington, DC: The National Academies Press. doi: 10.17226/9646.
×

In May 1999, EPA issued a revised draft research-strategy document (EPA 1999) for airborne particulate matter that was reviewed by CASAC on June 10, 1999. The date of issue of the document precluded review by this NRC committee in time to consider it in this report. The committee urges EPA to continue to update its PM research strategy regularly and, to make certain that what the strategy documents say and what is actually being done are closely linked.

In addition to federal agencies, airborne PM research is actively pursued by some states, the private sector, nonprofit organizations, and international research organizations, often in collaboration with one or more U.S. federal agencies. However, as in the case of the federal agencies, no effective ongoing mechanism is available for coordinated planning and integration of research across health, exposure, and atmospheric sciences.

NARSTO is the foremost existing example of such a mechanism, a public/private partnership with membership including government, industry, and academia throughout Canada, Mexico, and the United States, which was organized originally through the Air Quality Subcommittee of CENR. NARSTO has recently been discussing possible ways to link atmospheric research related to particulate matter with research on health effects. But no comparable private/public umbrella organization exists for the health and exposure sciences alone, or for all disciplines together.

Enhanced coordination and integration across all research-sponsoring organizations could provide substantial benefits in terms of leveraging broad-based support for key elements of the program, ensuring that the full research community is actively engaged in planning all relevant aspects of the research program, promoting accessibility to all results of the program, and avoiding unnecessary duplication of effort.

The committee supports recent statements by NARSTO that it will seek to reach out to the broader health and exposure-research communities to gain their perspectives in planning for upcoming atmospheric measurement and research programs. The committee also encourages the expanded CENR Air Quality Subcommittee to enhance its efforts by

  • Formally including the full range of research-funding organizations in the development and implementation of research strategies.

Suggested Citation:"2 Evaluating Research Implementation and Progress." National Research Council. 1999. Research Priorities for Airborne Particulate Matter: II. Evaluating Research Progress and Updating the Portfolio. Washington, DC: The National Academies Press. doi: 10.17226/9646.
×
  • Ensuring that federal agency research leaders such as EPA, DOE, and NOAA convene targeted workshops regularly to bring together institutions and investigators involved in addressing specific research topics (e.g., the recent efforts to coordinate among all ongoing exposure assessment studies).

  • Taking advantage of upcoming and regularly scheduled cross-disciplinary research meetings, such as the biannual Colloquium on Particulate Matter and Human Health, to bring together and exchange information among all active funders and investigators.

  • To the extent appropriate, exploring the development of an ongoing health and exposure collaborative organization analogous to NARSTO that could continue collaboration across disciplines and facilitate collaboration between the atmospheric sciences and health and exposure communities.

Such mechanisms would help to ensure that to the maximum practical extent, the full national investment in PM research represents an integrated and collaborative program with the highest likelihood of providing timely, cost-effective, and useful results.

Much of the success of the PM research program will depend upon the ability of extramural scientists from a variety of institutions to collaborate across disciplines and organizations and to communicate the status and progress of their research in an effective and accessible manner. At present, neither EPA nor other major funding sources sufficiently facilitate such interdisciplinary interaction and communication. Consequently, researchers tend to remain within traditional disciplinary boundaries in developing new research proposals.

EPA's PM research program cannot be fully successful without stimulating agency and extramural scientists to cross disciplinary boundaries and work as teams to achieve concrete research objectives. Team-based research planning necessitates a new set of incentives that, where appropriate, motivates scientists to develop transdisciplinary proposals and teams, and in all cases, regularly provides opportunities for investigators to communicate and coordinate research across disciplines. The committee encourages EPA to create a formal plan to pro-

Suggested Citation:"2 Evaluating Research Implementation and Progress." National Research Council. 1999. Research Priorities for Airborne Particulate Matter: II. Evaluating Research Progress and Updating the Portfolio. Washington, DC: The National Academies Press. doi: 10.17226/9646.
×

mote greater collaboration in planning and managing the PM research program. One opportunity to implement incentives lies in providing funding for planning interdisciplinary research through the EPA PM research centers program. In addition, where appropriate, EPA should encourage interdisciplinary research through its Science and Technology to Achieve Results (STAR) research grants program.

In its evaluation of PM research over the next 4 years, the committee plans to conduct an iterative, ongoing process of evaluation and will organize meetings, workshops, and other mechanisms to monitor research progress. As part of its evaluations, the committee plans to implement the following three steps:

  • Array the results of particulate research sponsored by EPA and other institutions against the committee's recommended research portfolio to obtain an ongoing assessment of research gaps that need to be addressed.

  • Compare research results against the six criteria presented at the beginning of this chapter. Particular priority will be assigned to evaluating the value of research results against the major issues that policymakers must address in setting and implementing standards.

  • Conduct a series of public workshops to obtain information on the progress of EPA's PM research program and the programs of other funding agencies and organizations. Such workshops will include scientists, research managers, and policy experts from EPA, other federal and state agencies, universities, and the private sector, including industry and nonprofit organizations.

The committee believes there is an urgent need for the development of a comprehensive data base that identifies research that is under way, linking the research to the committee's ten portfolio recommendations and tracking progress. The committee has developed a prototype data base for this inventory but has concluded that further refinement and maintenance would exceed the committee's resources and charge. EPA should assume responsibility for further development and maintenance

Suggested Citation:"2 Evaluating Research Implementation and Progress." National Research Council. 1999. Research Priorities for Airborne Particulate Matter: II. Evaluating Research Progress and Updating the Portfolio. Washington, DC: The National Academies Press. doi: 10.17226/9646.
×

of the data base, perhaps in partnership with the Health Effects Institute (HEI), which, along with EPA, has already prepared summaries of research in progress. Without such an inventory, neither this committee, EPA, or other funding organizations will be able to monitor research progress adequately. In addition, the data base can facilitate the development of new research to address gaps and avoid inappropriate duplication of work already in progress. The committee has been informed by EPA and HEI that a planning effort for such a research inventory is in progress.

Suggested Citation:"2 Evaluating Research Implementation and Progress." National Research Council. 1999. Research Priorities for Airborne Particulate Matter: II. Evaluating Research Progress and Updating the Portfolio. Washington, DC: The National Academies Press. doi: 10.17226/9646.
×
This page in the original is blank.
Suggested Citation:"2 Evaluating Research Implementation and Progress." National Research Council. 1999. Research Priorities for Airborne Particulate Matter: II. Evaluating Research Progress and Updating the Portfolio. Washington, DC: The National Academies Press. doi: 10.17226/9646.
×
Page 27
Suggested Citation:"2 Evaluating Research Implementation and Progress." National Research Council. 1999. Research Priorities for Airborne Particulate Matter: II. Evaluating Research Progress and Updating the Portfolio. Washington, DC: The National Academies Press. doi: 10.17226/9646.
×
Page 28
Suggested Citation:"2 Evaluating Research Implementation and Progress." National Research Council. 1999. Research Priorities for Airborne Particulate Matter: II. Evaluating Research Progress and Updating the Portfolio. Washington, DC: The National Academies Press. doi: 10.17226/9646.
×
Page 29
Suggested Citation:"2 Evaluating Research Implementation and Progress." National Research Council. 1999. Research Priorities for Airborne Particulate Matter: II. Evaluating Research Progress and Updating the Portfolio. Washington, DC: The National Academies Press. doi: 10.17226/9646.
×
Page 30
Suggested Citation:"2 Evaluating Research Implementation and Progress." National Research Council. 1999. Research Priorities for Airborne Particulate Matter: II. Evaluating Research Progress and Updating the Portfolio. Washington, DC: The National Academies Press. doi: 10.17226/9646.
×
Page 31
Suggested Citation:"2 Evaluating Research Implementation and Progress." National Research Council. 1999. Research Priorities for Airborne Particulate Matter: II. Evaluating Research Progress and Updating the Portfolio. Washington, DC: The National Academies Press. doi: 10.17226/9646.
×
Page 32
Suggested Citation:"2 Evaluating Research Implementation and Progress." National Research Council. 1999. Research Priorities for Airborne Particulate Matter: II. Evaluating Research Progress and Updating the Portfolio. Washington, DC: The National Academies Press. doi: 10.17226/9646.
×
Page 33
Suggested Citation:"2 Evaluating Research Implementation and Progress." National Research Council. 1999. Research Priorities for Airborne Particulate Matter: II. Evaluating Research Progress and Updating the Portfolio. Washington, DC: The National Academies Press. doi: 10.17226/9646.
×
Page 34
Suggested Citation:"2 Evaluating Research Implementation and Progress." National Research Council. 1999. Research Priorities for Airborne Particulate Matter: II. Evaluating Research Progress and Updating the Portfolio. Washington, DC: The National Academies Press. doi: 10.17226/9646.
×
Page 35
Suggested Citation:"2 Evaluating Research Implementation and Progress." National Research Council. 1999. Research Priorities for Airborne Particulate Matter: II. Evaluating Research Progress and Updating the Portfolio. Washington, DC: The National Academies Press. doi: 10.17226/9646.
×
Page 36
Suggested Citation:"2 Evaluating Research Implementation and Progress." National Research Council. 1999. Research Priorities for Airborne Particulate Matter: II. Evaluating Research Progress and Updating the Portfolio. Washington, DC: The National Academies Press. doi: 10.17226/9646.
×
Page 37
Suggested Citation:"2 Evaluating Research Implementation and Progress." National Research Council. 1999. Research Priorities for Airborne Particulate Matter: II. Evaluating Research Progress and Updating the Portfolio. Washington, DC: The National Academies Press. doi: 10.17226/9646.
×
Page 38
Next: 3 Updating the Research Portfolio »
Research Priorities for Airborne Particulate Matter: II. Evaluating Research Progress and Updating the Portfolio Get This Book
×
Buy Paperback | $44.00 Buy Ebook | $35.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

In the effort to reduce the scientific and technical uncertainties over regulation of airborne particulate matter in the United States, Research Priorities for Airborne Particulate Matter: II. Evaluating Research Progress and Updating the Portfolio, the second book in a four-part series requested by Congress, describes the plans of the committee to monitor the progress of the research on particulate matter conducted by the U.S. Environmental Protection Agency (EPA), other federal and state government agencies, and nongovernmental organizations.

The book also reviews and updates the committee's portfolio of recommended research in its first volume, Research Priorities for Airborne Particulate Matter: I. Immediate Priorities and a Long-Range Research Portfolio (NRC, 1998). The committee substantially revised two of the ten high-priority research areas recommended in Part I. Part II notes that Congress, EPA, and the scientific community have given strong support to the committee's recommendations and have implemented substantial changes in research efforts in response to Part I of the series. One important research area-studies of the effects of long-term exposure to particulate matter and other major air pollutants-however, does not appear to be underway or planned.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!