National Academies Press: OpenBook
« Previous: 4. Measure for Measure
Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×

5
Taking the Measure of STAR

This chapter contains the committee’s evaluation of the Environmental Protection Agency (EPA) Science To Achieve Results (STAR) program. For its evaluation, the committee selected a set of metrics and recommends that EPA consider them as it adopts evaluative criteria for future evaluations of the STAR program. The evaluation is structured according to the guidelines that the president’s Office of Management and Budget (OMB) issued for government agencies to use in assessing their research programs for FY 2004 (OSTP/OMB 2002). As described in Chapter 4, the OMB guidelines set forth three major criteria for evaluating research programs: quality, relevance, and performance. The committee considered that following the OMB guidelines would be most valuable to EPA as it strives to comply with the requirements of the Government Performance and Results Act (GPRA) of 1993.

Under each of the three OMB criteria, the committee has developed metrics related to processes and products of the STAR grants program. Specifically, the committee reviewed numerous documents, including materials from EPA, OMB, and the National Research Council (NRC) and materials provided by people in academe, to derive these metrics. The process metrics are used to evaluate the adequacy of the operation or procedures of the STAR program, and the product metrics are used to evaluate the outputs of the program, such as the number of publications or the influence or effect the program has had or may have. The committee evaluated the STAR fellowship program independently from the grants program because it is a small part of the STAR program and operates somewhat independently.

Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×

The committee was not able to evaluate some of the metrics completely, particularly those related to products, primarily because STAR has not been in operation long enough to produce a sufficient number of products to allow a complete evaluation. Other reasons for incomplete evaluations are that EPA has not collected sufficient information relevant to a metric and that some of the metrics are not intended to be addressed solely by the STAR program, but cover issues that are broader and that must be addressed by ORD and EPA. The set of metrics used by the committee in evaluating the STAR grants program is presented in Table 5-1. The goals and objectives of the fellowship program differ from the grants program, and it is evaluated more briefly at the end of this chapter. For each metric, this chapter summarizes the pertinent information provided to (or otherwise obtained by) the committee and then presents the committee’s conclusions about how adequately the STAR program appears to be addressing it with respect to the available information—except that, as stated above, for some product metrics the STAR program is too young to have produced sufficient products to permit a complete evaluation.

RESEARCH PROGRAM

Quality

Evaluating research quality is extremely difficult. It cannot be measured with a simple metric, such as a thermometer or a yardstick, or by the number of reports or number of pages produced. As indicated in Chapter 4, even more sophisticated measures, such as the number of citations in the technical literature, need to be carefully interpreted.

The STAR program has tended to focus more on the quality of its process than on the quality of its products. That is understandable and appropriate. It is understandable because, being a relatively young program, STAR has had to focus on trying to get the process right and is only now beginning to accumulate a sufficient number of products to support a quality evaluation and because most of the external reviews of the program have tended to focus on process issues.

It is appropriate because, as indicated in Chapter 4, a high-quality pro-cess is generally a necessary condition for producing high-quality products. Frequent evaluations of the quality of the process will also provide an early warning of possible problems in the quality of the products. The causes of inadequate products usually lie in inadequacies of procedures that occurred many years previously.

Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×

TABLE 5-1 Metrics Used in the Committee’s Evaluation of the STAR Grants Program

QUALITY

Process

Does the program have an effective process to ensure receipt of high-quality proposals for its grant awards?

Does the program have an effective process to ensure the selection of high-quality proposals?

Does the program have a mechanism for encouraging high-quality research?

Does the program have a clearly defined plan for regular, external reviews of its research quality, and has this plan been effectively carried out?

Product

Is the STAR program sponsoring high-quality research?

Has the program made significant contributions to advancing the state of the science in particular topics?

Do bibliometric and citation analyses demonstrate excellence in the program’s research?

RELEVANCE

Process

Does the STAR portfolio support EPA’s mission, GPRA goals, and ORD’s strategic plans, research strategies, and multiyear plans?

Are the processes used to define the research initiatives that will be supported by the STAR program sufficient to target the topics of most important uncertainty, highest impact, or highest priority?

Does the program have a “clear plan for external reviews of the program’s relevance” (OSTP/OMB 2002), and has this plan been effectively carried out?

Does the program have an effective process for identifying and communicating with the potential audiences and users of the research results?

Product

Is the STAR portfolio appropriately mixed between core and problem-driven research and between human health and ecologic research?

Does the program have a good plan for integrating and synthesizing results, and has this plan been carried out effectively?

Have the program’s results been used in EPA, state, or international decision-support documents?

Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×

RELEVANCE

Product

Have the research results in one or more subjects significantly improved the scientific foundation for decision-making?

Can a link between STAR research and improved protection of human health and ecologic systems be identified?

PERFORMANCE

Process

Is the STAR budget appropriate to fulfill the program’s mission?

Is the program effectively complementing ORD’s other research efforts?

Is the program well balanced?

Does the program award grants expeditiously?

Does the program have a process to demonstrate the communication of individual grant results in the professional literature?

Is there a process in place for reviewing the performance of individual investigators and research centers?

Product

Is the program funding relevant research that otherwise would not be funded?

Does the program have a schedule for the products it intends to produce, and how well is it adhering to that schedule?

To what extent are site-specific studies designed to be replicated at other locations?

Although a high-quality process may be necessary for producing high-quality products, it is not sufficient. It is time for the STAR program to begin to implement product evaluations that will ensure that both its process and its products have the high quality that the nation needs to support an effective and efficient environmental-management program.

Process Metrics

Does the program have an effective process to ensure receipt of high- quality proposals for its grant awards?

EPA provided the committee, through presentations and interviews,

Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×

with a substantial amount of information (summarized in Chapter 2) concerning how the STAR program attempts to elicit good proposals and how it evaluates those it receives.

There are two basic steps in establishing a good process for satisfying this metric. The first is to identify the topic to be addressed by the research in a way that stimulates good responses. The second is to advertise the availability of research support broadly in the research communities most qualified to undertake high-quality research on the desired topic.

With respect to the first step, EPA puts substantial effort into defining its research agenda, and the STAR program submits its proposed requests for applications (RFAs) to extensive review within the agency. That effort is intended to ensure that the RFAs are focused on the most important issues and that they define the research requirements properly. However, although substantial effort is devoted to the process in the agency, neither the research plans nor the proposed RFAs are externally peer-reviewed by subject-matter experts, except when the research is being supported jointly by another organization, in which case, representatives of the other organization participate in drafting and reviewing the RFA.

With respect to notifying potential researchers of the funding opportunities, the National Center for Environmental Research (NCER) makes a substantial effort to reach out to a broad scientific community and to recruit the most capable scientists. NCER disseminates its RFAs widely through its Web site, the Federal Register, announcements at professional meetings, and e-mail distributions to individuals or institutions that sign up on the STAR Web site. When the desired research falls outside EPA’s traditional research fields and may therefore involve scientists that are not already tied into the agency’s research program, STAR solicits the help of other agencies that traditionally work with these scientists to ensure that they are aware of the funding opportunities.

The committee concludes that the processes established by the STAR program compare favorably with and in many cases substantially exceed those established by other research-supporting organizations. Subjecting the research plans and draft RFAs to independent peer review would strengthen the process even more, but the committee recognizes that doing so might reduce the agency’s current flexibility to respond quickly to new research needs and might unduly delay the process of issuing RFAs. However, EPA should consider using external peer reviewers for RFAs when they do not have the in-house expertise.

Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×

Does the program have an effective process to ensure the selection of high-quality proposals?

The process for selecting proposals was described to the committee by the person responsible for managing it and by several STAR project officers, and some committee members had participated in the process previously.

The STAR program has established a rigorous peer-review process to evaluate the quality of proposals. Such peer review is the foundation on which excellence is achieved in all research programs, such as those of the National Institutes of Health (NIH) and the National Science Foundation (NSF). The agency has taken strong steps to ensure that this process does not suffer from conflicts of interest and is independent. The program’s procedures provide for a firewall that shields the peer-review process from any influence or potential conflicts of the project officers and staff who oversee the individual investigator, fellowship, and center awards. For instance, project officers can provide the names of potential reviewers to the Science Review Administrators (SRAs), also known as peer-review officers, but it is the sole responsibility of the SRAs to select reviewers and to make reviewer assignments. Project officers may attend peer-review meetings as observers but may not provide any comments that would affect peer review (Bryan 2002).

The program selects peer reviewers from a large number of sources, including people who have served on previous panels, keyword searches of databases (such as, Community of Science, the National Library of Medicine’s PUBMED, and faculty listings), keyword searches of NCER’s peerreview panelist information system, input from project officers and program-office scientists, and lists of the attendees of pertinent technical conferences (Bryan 2002). The agency is unusual in that it pays the members of its peer-review panels a daily stipend, presumably to provide additional encouragement for experts to participate. Although lower than the consulting fees that such experts might earn in the private sector, the stipends do reduce the financial disincentives associated with serving on the panels.

The committee received some comments suggesting that in the program’s early days, some members of the peer-review panels might not always have had the necessary qualifications to be effective members, but this problem appears to have disappeared as the program has matured.

The committee heard no suggestion that the process of selecting peer reviewers was influenced by conflicts of interest.

Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×

The final selection of proposals to be funded is influenced by a “relevance review” that is carried out by agency staff. That review is limited to proposals that have been rated as “excellent” or “very good” by the quality peer-review panel.

The committee concludes that, given EPA’s desire to avoid appearances of conflicts of interest by completely separating the selection of peer-review panel members from the influence of project officers, the agency has established a rigorous, independent peer-review process for selecting the highestquality proposals.

Does the program have a mechanism for encouraging high-quality research?

To gain a better understanding of how the STAR program encourages high-quality research, the committee received briefings from NCER staff on its processes, reviewed material that was publicly available, attended several progress-review meetings, and discussed the program’s procedures with STAR grant recipients.

STAR has implemented several mechanisms for encouraging highquality research by its investigators. Investigators are required to submit annual progress reports that describe the research being undertaken and its progress. The progress reports are reviewed by STAR project officers, and summaries are posted on the NCER Web site.

In addition, the STAR program sponsors progress review workshops on research topics. Principal investigators of all STAR grants receiving support pertinent to the topic are expected to attend. The meetings are also attended by other EPA Office of Research and Development (ORD) staff conducting relevant research; all meetings are open to the public. At the meetings, principal investigators must present their research progress to their colleagues and EPA staff, opening it to peer review. The meetings also provide an opportunity for researchers to share ideas and coordinate research efforts. Some of the meetings have apparently been much more successful than others in accomplishing their objectives.

The committee concludes that EPA has established procedures for reviewing the quality of research in progress that in several ways exceed those adopted by most other research-supporting organizations. Those procedures could be enhanced by ensuring that progress-review workshops were held at the most expeditious times and were run most efficiently to stimulate peer review and collaboration. The program should review the success of its past progress-review meetings and organize its future meet

Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×

ings to emulate the ones that were most successful in accomplishing their objectives.

Does the program have a clearly defined plan for regular, external reviews of its research quality, and has this plan been effectively carried out?

As mentioned in Chapter 2, numerous reviews have been done on the processes and operation of the STAR grants program, and not on the products of the grants (EPA/BOSC 1998; EPA 2000; EPA/NSF 2000; EPA/SAB/BOSC 2000; GAO 2000; EPA/SAB 2001; EPA 2002a). The majority of the reviews have been conducted by EPA’s Science Advisory Board (SAB) or Board of Scientific Counselors (BOSC).

The committee is concerned that too many and too frequent reviews of the STAR program have the potential to be damaging in that they may divert necessary financial and personnel resources from the program. The committee believes strongly that because of the nature of research, which takes a considerable amount of time and many projects to advance the state of knowledge, too frequent reviews of STAR add little value to the understanding of the operation and results of the program.

The committee recommends that the STAR program establish a schedule of product reviews at the appropriate level (as discussed in Chapter 4). By establishing such a schedule, the program may protect itself from the apparent excess of external reviews that have been imposed on it in the past.

Product Metrics

Is the STAR program sponsoring high-quality research?

The committee was presented with some anecdotes concerning the quality of the research being sponsored by the STAR program, but not with any systematic reviews or evidence concerning the quality of the program’s products.

Evaluating the quality of research products is very difficult, involving substantial judgment on the part of scientists who have expertise in the research topic being reviewed. As indicated in Chapter 4, the committee considers that the most effective method for evaluating research is the use of independent expert review committees focusing on specific topics (a level 2 review). The committee recommended that the STAR program establish a schedule for such reviews (see Chapter 4).

Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×

Lacking product evaluations, the committee reviewed the backgrounds and accomplishments of a sampling of the STAR principal investigators. Although a rigorous sampling procedure was not conducted and thus the results of the review are only indicative and not definitive, this review indicated that the STAR program is funding many scientists who have outstanding research credentials. The scientists have impressive track records and are leaders in their fields. Many are editors of journals or officers in societies and have received awards of distinction. Some were attracted to the STAR program from fields outside EPA’s mission, so it can be said that the program has been successful in attracting the best and the brightest. Many of the investigators, however, have long been active in the relevant fields, and the STAR program has enabled them to continue to make contributions. The investigator mix also included young investigators who will be the leaders of the future.

The committee notes that EPA’s rate of funding of individual investigator and center awards tends to be lower than that of other federal grants programs, such as those sponsored by NSF and NIH; this reflects the competitiveness of the program. As indicated in Chapter 2, data from FY 19992001 indicate that EPA funds an average of 10-15% of the proposals it receives. In contrast, as indicated in Chapter 3, agencies like NIH and NSF strive to fund at least about 25-30% of the proposals received. STAR is able to fund only about 60% of proposals rated as “excellent” or “very good” by its independent quality peer-review panels. It funds no proposals that receive a lower ranking (“good” or lower).

On the basis of the STAR program’s process for awarding grants, the quality of the people and institutions being funded by the STAR program, and the highly competitive nature of its awards, the committee is confident that the products of these grants will be of the highest quality.

Has the program made significant contributions to advancing the state of the science in particular topics?

Although the STAR program does not systematically identify the significant contributions it makes to filling important gaps in the state of the science in particular topics, examples were presented to the committee by EPA staff and by investigators supported by STAR funding. Some of the committee members were also familiar with the advances that STAR-funded research was making to some particular topics.

Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×

The planning process that leads to the preparation of RFAs is to a large extent focused on ensuring that STAR grants will fill information gaps in topics of greatest interest to the agency. That focus is maintained throughout the research process.

The committee was presented with several examples of STAR-supported research efforts that had made significant contributions to scientific understanding in particular topics (see Boxes 5-1, 5-2, and 5-3). For example, STAR-sponsored research in endocrine disruptors, particulate matter, and ecologic assessment has resulted in peer-reviewed groups of publications of immediate interest in understanding causes of, exposures to, and effects of environmental pollution.

To determine whether STAR research has filled a critical knowledge gap or otherwise strengthened and improved the foundation for decision making, it would be useful to assess the state of the science in a particular issue before STAR-funded projects are completed and then synthesize the results of the research after the projects are completed. Such assessments would help EPA to target RFAs at the front end, as well as to analyze net results at the back end. A particular issue could be assessed by a panel convened by ORD, by STAR, or by others in the field (such as the Ecological Society of America, or the Society of Environmental Toxicology and Chemistry). There have been several successful examples of state-of-the-science documents, including the particulate-matter (PM) reports produced by the NRC’s Committee on Research Priorities for Airborne Particulate Matter (NRC 1998, 1999b, 2001). In general, better integration of research results by STAR and ORD and the state-of-the-science assessments mentioned above should provide most of the information necessary to report on this metric.

The committee recognizes of course that this metric involves substantial subjective judgment and that it is often difficult to identify the effect of any particular set of research results. The judgments implicit in this metric can probably be best rendered by the use of expert review, as suggested in Chapter 4.

The committee concludes that STAR-supported research is making significant contributions to advancing the state of the science in many of the topics that it is addressing. The committee suggests that the program undertake a more systematic effort to identify the contributions and the success in filling the knowledge gaps identified in the research planning process by preparing research synthesis reports when research is completed, as recommended elsewhere in this chapter.

Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×

BOX 5-1 Results of STAR Endocrine-Disruptors Program

  • Determined that exposure to high concentrations of polybrominated biphenyls prenatally and in breast milk may affect puberty in girls (Blanck et al. 2000).

  • Discovered a new (third) estrogen receptor in vertebrates and demon-strated that estrogens and xenoestrogens can act on cells at the membrane level (Hawkins et al. 2000).

  • Developed and refined an in vivo model using medaka to identify endocrine disrupting chemicals (Cooke and Hinton 1999).

  • Identified androgenic compounds (male-hormone mimics) in paper-mill effluent by using a screening assay in fish (Jenkins et al. 2001).

  • Developed integrated array of computational tools undergoing valida-tion by EPA’s Office of Pollution Prevention and Toxic Substances (OPPTS) for setting priorities for science and technology programs (Xing et al. 1999).

  • Determined concentrations of phytoestrogens in human amniotic fluid and effects of exposure to them in animal models (Hughes et al. 2001).

Source: Adapted from P. Preuss, EPA, presentation to National Research Council committee, March 18, 2002.

Do bibliometric and citation analyses demonstrate excellence in the program’s research?

Although EPA encourages its grantees to provide the STAR program with information on the articles and other publications that stem from the research it supports, the program has no mechanism for monitoring such publications after a grant is completed or for conducting bibliometric and citation analyses that would demonstrate the influence of the research on other work being conducted. The committee did sponsor an ad hoc bibliometric analysis to gain a better understanding of the value of such an approach (IISCO, Atlanta, GA, unpublished material, 2002) (see Chapter 4). As indicated in Table 4-1, the bibliometric analysis conducted for the committee indicated that the citation rate of publications that result from STARsupported research is similar to that of other research publications in the topics for which the analyses were undertaken.

As stated in Chapter 4, the committee considers that bibliometric and citation analyses are important quantitative metrics for gauging the quality of research but cautions that these types of analyses have many limitations.

Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×

BOX 5-2 Sampling of Results from the STAR Particulate-Matter Program

  • Are diabetics more susceptible to the health effects of airborne particles? This study examined whether diabetes modifies the effects of PM. Researchers studied the association of PM10 with hospital admissions for heart and lung disease in persons with and without diabetes as a comorbidity. Using Medicare data for Cook County, Illinois, the investigators found that a 10-ug/m3 increase in PM10 was associated with a 2.01% increase in admissions for heart disease with diabetes but only a 0.94% increase in persons without diabetes. Similar effect modification was not seen for lung diseases. When analyzing by age, researchers found twice the PM10-associated risk for heart disease in diabetics as in nondiabetics in both age groups examined. Investigators concluded that people with diabetes are a more susceptible population (Zanobetti and Schwartz 2001).

  • Long-term effects of PM exposure include lung cancer. A major epidemiology study determined that long-term exposure to combustion-related fine-particle air pollution is an important environmental risk factor for cardiopulmonary and lung-cancer mortality. The magnitude of lung-cancer mortality associated with fine-particle exposure has been equated to that from exposure to environmental tobacco smoke. The investigators linked data on 500,000 adults in the American Cancer Society’s prospective mortality study with airpollution data for metropolitan areas throughout the United States (Pope et al. 2002).

  • Early findings on biologic mechanisms associated with exposure to concentrated air particles. Inhalation of concentrated urban air particles by rats results in modest pulmonary inflammation in normal animals and model populations in most experiments. Increases in circulating neutrophils as evidence of systemic inflammation were variable in these experiments. However, when circulating neutrophils were increased, there was not a measurable increase in circulating cytokines. In studies of rats with induced pulmonary inflammation, exposure to concentrated urban air particles resulted in the deaths of some animals with electrocardiographic evidence of sympathetic nervous system enhancement and arrhythmia. This research contributed to the beginnings of our understanding of the biologic mechanisms by which inhaled ambient particles cause health effects (Clarke et al. 1999; Lovett et al. 1999).

  • Chemical composition of atmospheric ultrafine particles. Measure-ments of ultrafine particle mass concentration in seven southern California cities show that the chemical composition of these ultrafine particle samples averages 50% organic compounds, 14% trace-metal oxides, 8.7% elemental carbon, 8.2% sulfate, and 6.8% nitrate. A source emissions inventory constructed for the

Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×

South Coast Air Basin that surrounds Los Angeles shows a primary ultrafine particle emission rate of 13 tons/day. Those ultrafine emissions arise principally from mobile and stationary fuel-combustion sources and are estimated to consist of 65% organic compounds (Cass et al. 2000).

Source: S. Katz, EPA, personal commun., November 27, 2002.

For instance, average citation rates differ among topics and disciplines. It is also important to consider the prestige of the journals in which articles are published. Such analyses must be interpreted by experts familiar with the particular topics in question.

The committee also has concerns about the utility of bibliometric analysis to STAR, in as much as the program has not been in existence long enough to make bibliometric analysis an effective metric for research funded beyond the initial years of the program. Similar sentiments are echoed in the BOSC review of the STAR program (EPA/SAB/BOSC 2000) in which BOSC, in recommending that citations be used as a metric, cautioned that “4 to 6 years must pass between the completion of the STARfunded research and the use of citations to judge success.”

The committee recommends that EPA undertake bibliometric and citation analyses not as independent evaluations of research quality but in conjunction with expert review. The expert review will assist in placing the results of the bibliometric analysis in the context of the current state of scientific knowledge.

Relevance

In its discussion of relevance, OMB asks (OSTP/OMB 2002), “Does the agency’s research address subjects in which new understanding could be important in fulfilling the agency’s mission?” In the committee’s view, mission relevance requires that STAR research improve the knowledge base required to identify environmental issues and make sound environmental decisions. The current STAR portfolio includes both core research (research that provides an understanding of the structure and function of environmental systems, the effects of human perturbations on those systems, and the resulting effects on human health and quality of life) and problemdriven research (research that focuses more specifically on questions related

Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×

BOX 5-3 Sampling of Achievements of STAR Ecosystem-Protection Program

  • Developed estuarine index of biotic integrity that has shown broad applicability within the southern New England ecoregion. This index could be a valuable monitoring tool to assess the recovery of ecosystem function after eutrophication remediation (Deegan et al. 1997).

  • Developed a dynamic, ecologic, economically linked model to evaluate the driving forces and ecologic consequences of land-use change. This model was used to demonstrate how changes in zoning would affect water quality in Calvert County, Maryland (Voinov et al. 1999).

  • Developed and tested an integrated ecologic-assessment and decision-support tool for the Lake Erie ecosystem. The method was designed to assist managers and stakeholders involved in the Lake Erie Lakewide Management Plan (LaMP) and other Lake Erie management processes to define objectives and evaluate tradeoffs and risks associated with future uses (Locci and Koonce 1999).

  • Assessed the impact of SO2 and NO2 emission reductions on precipita-tion and air quality by comparing emissions before and after the Clean Air Act Amendments. Results show a significant reduction in SO2 emissions for most states except Texas, North Carolina, Illinois, Florida, and Alabama. However, only two states show a reduction in NOx (Butler et al. 2001).

  • Developed a reliable method for the simultaneous measurement of concentrations of viruses, bacteria, and protozoans in water using disposable hollow ultrafilters; this method allowed for Cryptosporidium oocyst recoveries of about 60-80%. The new method was published, making it available to the scientific community, EPA, and the general public for use in place of other, less accurate methods (Juliano and Sobsey 1997).

  • Developed and validated a new approach for using satellite radar imag-ery to monitor the spatial and temporal patterns of flooding and drying in the wetland ecosystems of south Florida. This technology has been used to monitor the effects of human activities (such as the construction of roads and the operation of water-control structures) on natural hydroperiods in south Florida wetlands (Kasischke 2000).

Source: B. Levinson, EPA, personal commun., October 16, 2002.

to upcoming regulatory decisions). Although the core research is farther removed from the agency’s regulatory efforts, it can provide new understanding important to the agency for fulfilling its mission.

Although the agency is the primary audience for the STAR program,

Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×

EPA’s mission to safeguard the environment includes not only the performance of its own regulatory tasks but also providing the scientific foundation for decisions by other entities. In addition to its own efforts, EPA research aids environmental decision making by state, local, tribal, and international government agencies, other federal agencies, and the public. Accordingly, it is relevant for ORD and STAR research portfolios to incorporate research that will promote sound environmental decision making by all those users.

The committee believes that, first, the STAR portfolio should be related to EPA’s mission and focus on EPA’s highest-priority environmental science and engineering needs. Second, the body of knowledge provided by STAR research should contribute to identifiable progress toward practical outcomes. Third, a smaller but significant portion of STAR research portfolio should explore potential environmental problems. The metrics addressing the relevance of the STAR program, discussed below, touch on each of those general intents.

Process Metrics

Does the STAR portfolio support EPA’s mission, GPRA goals, and ORD’s strategic plan, research strategies, and multiyear plans?

The committee heard presentations from several EPA representatives regarding the agency’s research planning process, discussed this process with other NCER staff, and reviewed information relevant to this metric that was available on EPA’s Web site.

ORD invests a substantial amount of effort in the research planning process to ensure that the STAR portfolio will be relevant to EPA’s mission and enhance ORD’s ability to meet its strategic goals. For instance, STAR has adopted a planning process in which the development of the STAR portfolio evolves from EPA’s strategic plan and GPRA goals, ORD’s strategic plan, research strategies, and multiyear plans.1 The RFAs, which are designed to carry out the research plans, are circulated throughout the rest of the agency to ensure that they are consistent with them.

1  

In general, the distinction between these is that the EPA and ORD strategic plans define the universe of mission-related activities that ORD should target; the ORD research strategies define which issues are important and frame the scientific questions associated with them; and the ORD multiyear plans apportion the research between intramural and extramural (STAR) programs.

Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×

However, research strategies and research plans do not necessarily exist for all the topics supported by STAR grants; and even when they do, they are often internal working documents and are not available for public review. The ecologic indicators (EPA 1998a) and airborne particulate matter (EPA 1999) research strategies were available, and comparison of the strategies with the relevant RFAs shows that the STAR program has been designed to answer the research questions posed by the strategies. The endocrine-disruptors research plan (EPA 1998b) tracks closely with the research questions addressed in the RFAs.

In addition, review of all the RFAs issued during FY 1999, 2000, and 2001 shows that the topics addressed in the RFAs are mission-relevant and related to the EPA and ORD strategic plans.

Therefore, on the basis of the information available, the committee concludes that the STAR portfolio does support EPA’s mission and GPRA goals and ORD’s strategic plans, research strategies, and multiyear plans.

Are the processes used to define the research initiatives that will be supported by the STAR program sufficient to target the topics of most important uncertainty, highest impact, or highest priority?

The processes used to define STAR research initiatives are entrenched in the agency’s extensive research planning. Most of the planning efforts are internal and were not available for review by the committee.

The process used to develop RFAs is designed to focus STAR research on gaps in knowledge related to EPA’s mission, its high-priority research needs, and subjects with the greatest uncertainty and potential impact. However, only EPA staff have a substantial input into this process. In some circumstances, the expertise of the agency personnel may not always be sufficient to select among the various options for deciding the highest-priority research for funding, particularly when the issue being addressed by the research falls outside EPA’s regulatory purview.

One activity that would assist in the identification of highest-priority research would be a survey of the state of the science in potential research topics before the designing of RFAs. In some cases, that would require additional resources, for example, to convene a panel charged with reviewing the state of the science and suggesting topics in which new research may have the greatest impact on EPA’s mission (see also the discussion of “relevance-review” panels in NRC 1999a). In others, particularly those for which detailed research plans have been developed, additional advice from selected outside experts may suffice.

Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×

The committee’s brief review of the RFAs for research in ecologic indicators, endocrine disruptors, and particulate matter showed increasingly useful focus on critical issues and questions as the programs matured. In fact, an EPA program officer commented that RFAs have become more specific as the agency has learned that a field is advanced further when research is more specifically targeted (E. Francis, EPA, Washington, D.C., personal commun., August 6, 2002).

Although the committee concludes, on the basis of available information, that the STAR program has been increasingly successful in identifying and defining research efforts that are of highest priority, it recommends that the agency explore methods for improvement, particularly when research is not directly related to the agency’s regulatory programs. A potential method of incorporating a relevance analysis into grant applications was offered by the EPA SAB (EPA/SAB 2001). In this review of the STAR Water and Watersheds program, the SAB suggested that grant applications incorporate a number of questions that would require the researchers to evaluate how the proposed research would inform future watershed management decisions, provide increased understanding of the structure and function of watersheds, and relate to the priorities listed in ORD’s strategic plan.

Does the program have a “clear plan for external reviews of the program’s relevance” (OSTP/OMB 2002), and has this plan been effectively carried out?

The committee examined copies of all the reviews of the STAR program it received from EPA. The report by the General Accounting Office (GAO 2000) commented explicitly on the relevance of the STAR program to the EPA program offices, but this was not a “planned review.”

The research planning processes and the agency’s budgeting process provide an element of such reviews in that the program’s primary audience, EPA’s program offices, has a substantial opportunity to comment on the relevance of the program’s research and to influence its funding. However, those activities may not fall into the category of “external reviews” as defined by OMB, and they do not include audiences outside the agency.

The committee concludes that the program has no established plan for external reviews of its relevance and recommends that, after the program has adequately identified its potential audiences as recommended below, it institute such a plan as discussed in Chapter 4.

Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×

Does the program have an effective process for identifying and communicating with the potential audiences and users of the research results?

From EPA presentations to the committee and interviews with NCER staff, the committee obtained substantial information about the STAR program efforts to be responsive to the needs of the scientific community, EPA program offices, and other potential audiences. Committee members also attended some of the outreach programs that STAR sponsored. However, the committee received little information about efforts to identify other potential users and to ensure that research results are relevant to them.

In response to questions raised in a report to ORD’s BOSC, NCER indicated that it “believes that its audiences include not only its own organization, ORD, and EPA’s regional and program offices, but also other agencies and professional societies, and the outside world of Capitol Hill, the scientific community, and the public” (EPA 2002a).

Although the STAR program does not have a process for identifying its potential audiences, EPA seems to recognize the need to disseminate research results to a wider audience. Indeed, as the STAR program has matured, it has devoted increasing effort to finding effective means of communicating its research results to potential audiences. It provides a considerable amount of information about its projects on the NCER Web site, and the quality of the reports it makes available there has improved. The program has also experimented with some unique mechanisms for communication. For instance, the STAR program and EPA Region III supported the Mid-Atlantic Integrated Assessment project, which was designed to acquaint state and local agencies and environmental and community organizations in the mid-Atlantic region with the results of STAR-supported research efforts and to assess the relevance of the results to their needs (Bradley 2002). The STAR program also sponsored a workshop with Region I (New England) that was intended to bring STAR research results to state, federal, and tribal environmental programs in New England.

However, the program has not yet effectively developed a strategy for communicating to its wider user community; most of the emphasis has been on the scientific community and the EPA program offices. Any user who lacks the sophistication or incentive to visit EPA’s Web site and wade through individual grant reports or research publications is unlikely to benefit directly from the program.

One difficulty that the program faces in identifying and communicating with its audiences is that much STAR-sponsored research is designed to

Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×

complement other research efforts undertaken by ORD. In many cases, the relevant question is whether the entire EPA research effort, not just the part supported by STAR, is satisfying the user community. The user community is likely to have little knowledge of and to be largely uninterested in who produced the information. The user community’s primary interest is that the information be available and dependable. Indeed, the EPA program office staff, especially those who do not routinely review the scientific literature, are generally not aware of STAR results. However, since this is the primary audience for the STAR program, it is clearly in the program’s best interest to find ways of making this audience more aware of its value.

The committee understands that serving and effectively communicating to a diverse audience is difficult for a research organization, and it commends the STAR program for the innovative approaches it has developed to improve its communication efforts. However, an effective process for communicating the results of the STAR grants with its wider user community remains among the most important improvements that can be made in the program. The first step in accomplishing that is to define the audiences better. The committee recommends that the program, in association with the rest of ORD, then develop aggressive plans to disseminate STAR results to the audiences.

Product Metrics

Is the STAR portfolio appropriately mixed between core and problem- driven research and between human health and ecologic research?

In presentations to the committee, STAR representatives often mentioned the desire to maintain a balance between core and problem-driven research as called for in ORD’s strategic plan and between research efforts focused on human health and ecologic assessment.

Assessing whether a research program has maintained such balances is difficult and subjective. It is often difficult to determine how a particular RFA should be classified as to whether it is core or problem-driven. If it addresses both human health and ecologic effects, how should it be classified, particularly if much of what is termed ecologic research actually elucidates the processes by which environmental stressors affect both humans and other ecosystem attributes?

The committee did not attempt to conduct a comprehensive review of all of STAR’s individual grants and center research plans to determine whether such balances existed. However, a brief review of the RFAs for

Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×

FY 1999, 2000, and 2001 by the committee indicates that there was a rough balance between core and problem-driven research. In addition, the STAR portfolio has maintained a rough balance between human health and ecologic research, althoughdata for FY 1998-2002 indicate a recent tendency to tip the budget more toward human health (J. Puzak, EPA, Washington, D.C., personal commun., April 2, 2002).

Determining whether there is a balance is difficult enough. Assessing whether a given balance is “appropriate” is largely subjective, particularly because an effort to maintain a balance may be in conflict with the program’s goal to fund the highest-priority research. Should the program’s limited funds be used for research projects that will help to maintain a balance even if the projects are not of the “highest priority”?

The answer to that question is probably a qualified yes. For instance, the tight deadlines often imposed on EPA by Congress can force the agency to focus all its attention on what needs to be done tomorrow, but an effective environmental-management program requires a view of the future and the problems that it may hold. Similarly, some types of problems may, because of congressional mandates or other pressures, be more immediate, but few problems disappear, and it would be inappropriate for a research program to vacillate from one problem to another.

Given all the difficulties of making such an assessment, the committee nevertheless concludes that the STAR program has generally maintained appropriate balances between core and problem-driven research and between human health and ecologic research, and it encourages the program to maintain such balances in the future.

Does the program have a good plan for integrating and synthesizing results, and has this plan been carried out effectively?

Several of the EPA presentations to the committee addressed plans for integrating and synthesizing research results. Committee members obtained additional information from interviews with NCER staff.

As indicated in Chapter 2, the STAR program produces two types of reports, STAR Research Capsules and STAR Integrated Topical Searches, that provide summaries of individual research efforts addressing a particular topic. The program has also attempted to prepare synthesis reports that provide an integrated summary of the research that has been carried out on a particular topic. However, the program has apparently had difficulties with those efforts; no reports have been publicly released.

Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×

The committee believes that such synthesis reports can be very valuable, both for presenting a coherent summary of what is known about a particular topic and for targeting gaps in knowledge. The appropriate type of synthesis document will, however, depend on the intended audience. In many cases, because the STAR projects represent only a part of the important research being done on the topic, such synthesis reports would need to include the research undertaken by other ORD components and perhaps other research centers. To reduce possible concerns about the objectivity or independence of these reports, they should not be prepared by EPA staff. The committee believes that supporting the preparation of synthesis reports is an important improvement that should be made in the STAR program.

Have the program’s results been used in EPA, state, or international decision-support documents?

The STAR program has not been in existence long enough to be able to document the extent to which its research results are being used in supporting new regulations and other environmental-management decisions. Even for the projects that have been completed, there is often a substantial delay between the time that the research is completed and the agency’s decision to undertake rule-making or other actions to address the issues being studied.

The committee received no information about any efforts by or plans for the STAR program to collect information on the extent to which its results are being used in decision-support documents. Because that is one of the primary purposes of the STAR program, it is reasonable to expect the program to collect information about its success in fulfilling it, at least with respect to the agency’s own decisions. Attempting to monitor the use of STAR results by other agencies and organizations would be more difficult.

The committee recommends that the STAR program consider developing a mechanism for monitoring the use of its research results in criteria and other documents that support agency environmental-management decisions. For instance, EPA should attempt to keep track of the number of times STAR-funded research projects are cited in decision-support documents. However, an independent expert review of such an analysis would be necessary to assess the value or impact of this research in decision making.

Have the research results in one or more subjects significantly improved the scientific foundation for decision making?

As indicated earlier, the committee received anecdotal evidence that

Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×

identified topics in which STAR research had resulted in groups of peerreviewed publications of immediate use in understanding causes, exposures, and effects of environmental pollution, such as those cited for STAR in endocrine disruptors, PM, and ecologic assessment (see Boxes 5-1, 5-2, and 5-3). Additional time may be needed to view the full influence of the research.

This metric goes to the heart of relevance for a mission-related agency such as EPA. Research that improves the scientific foundation for environmental decision making incorporates both core and problem-driven research, and it incorporates the research performed not only by STAR but by ORD as a whole. Accordingly, this metric looks at each of the major subjects addressed by the STAR program and asks whether the results of the STAR-funded research—in combination with research performed elsewhere—filled a critical knowledge gap, provided an important decisionsupport tool, or otherwise improved the ability to target and manage an environmental effect. For some mature issues, this metric could include evidence that research already has improved risk-assessment and risk-management decisions. For example, the BOSC review of the STAR program (EPA/SAB/BOSC 2000) suggested tracking evidence that STAR research has enabled the agency to manage hazards that had not previously been identified as conferring significant risks, implement more cost-effective remedies for known hazards, or reduce the stringency of regulations for hazards that are found to be less serious than previously thought.

The committee concludes, on the basis of the data that it was provided (see Boxes 5-1, 5-2, and 5-3), that some STAR research efforts have already significantly improved the scientific foundation for decision making. However, the committee recommends that the program initiate a more coherent plan for identifying such instances.

Can a link between STAR research and improved protection of human health and ecologic systems be identified?

This type of question, focused on the ultimate purpose of government programs, is being asked of all their activities. The committee recognizes that establishing a link between research and protection of (or improvement in) human health or ecologic conditions is extraordinarily difficult, because temporal lags are long, with different types of research affecting human health and ecologic conditions in very different time frames, and because monitoring of human health and ecologic conditions is spotty. However, the ultimate mission of EPA is to “protect human health and to safeguard the natural environment—air, water, and land—upon which life depends.”

Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×

It is appropriate, therefore, to assess this metric periodically as a touchstone for STAR in the context of ORD research, even while recognizing that it will be difficult to document successes.

Performance

In its guidelines regarding program performance, OMB emphasizes metrics that show success in achieving “identifiable results” in keeping with a schedule of “multi-year R&D objectives with annual performance outputs and milestones” (OSTP/OMB 2002). Performance, however, also includes how a program achieves it goals. Is it an open, transparent process devoid of conflicts of interest and biases? Are procedures clearly established and predictable? Is it well integrated into the agency’s other research efforts, and does it expeditiously communicate its results to potential users?

Some of those questions were addressed in connection with quality and relevance metrics. Others are addressed here.

Process Metrics

Is the STAR budget appropriate to fulfill the program’s mission?

The committee reviewed a substantial amount of information about the details of the STAR budget but less about the appropriateness of the available funding to meet the program’s mission.

After growing rapidly during the program’s early years, the STAR budget has remained relatively constant, around $100 million per year for the last several years. Research costs have increased somewhat during this period, resulting in a decline in the amount of research that can be supported by the program. The lack of growth in the STAR budget, however, reflects the lack of growth in EPA’s total research budget. In fact, the proportion of the research budget allocated to the STAR program has increased slightly and now stands at about 18% of the ORD budget.

The STAR program has leveraged its resources by forming partnerships with other agencies that have similar research interests. EPA reports that STAR partnerships with other federal and private organizations leveraged funds in such a way as to permit 35% more grants to be funded than would have been possible with EPA resources alone (P. Preuss, EPA, Washington, D.C., personal commun., March 19, 2002). About half the STAR annual announcements are funded jointly with other federal agencies (P. Preuss,

Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×

EPA, Washington, D.C., personal commun., March 18, 2002). (See Chapter 2 for additional information on joint funding.)

EPA’s joint funding with other federal agencies provides several obvious advantages, including the funding of a larger amount of research that is tied directly to the agency’s mission and the pursuit of a broader research portfolio. One example of those benefits is the 2001 RFA on epidemiologic studies on the effects of exposures to endocrine disruptors that was issued jointly by EPA, National Cancer Institute (NCI), and National Institute for Occupational Safety and Health (NIOSH). EPA funded the studies focused on specific endocrine-disrupting chemicals of interest (such as phthalates and polybrominated diphenyl ether), NCI funded the cancer studies, and NIOSH funded the occupational studies (E. Francis, EPA, Washington, D.C., personal commun., August 6, 2002). EPA commented that its greatest challenge with the joint funding efforts is to ensure that the RFAs fully meet EPA’s mission and research needs and that the agency does not simply fund more but less-relevant research (P. Preuss, EPA, Washington, D.C., personal commun., June 6, 2002).

The committee concludes that the STAR program budget—which funds a mix of individual investigator awards, centers, and fellowships—represents an appropriate proportion of the ORD portfolio because it provides research capabilities that EPA may not have in-house or that complement the agency’s intramural research. However, the committee is concerned that the STAR budget has not kept pace with general inflation or the rising costs of research and education. The lack of growth in EPA’s total research budget may be preventing the agency from establishing the firm information base it needs to support its environmental-management efforts. In the future, an expert committee may want to tackle the question of whether the budget of the STAR program, and more generally the entire R&D apparatus of EPA, combined with other federal R&D expenditures, is sufficient to provide the level of research and technology necessary to provide the resources and knowledge base for the future.

Is the program effectively complementing ORD’s other research efforts?

The committee heard several presentations about the agency’s researchplanning processes, and it interviewed STAR project officers responsible for the PM, endocrine-disruptors, and ecologic-indicators programs about their efforts to ensure that STAR supported research is complementary to other research being sponsored by EPA.

EPA has adopted a multilevel, comprehensive research-planning pro-

Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×

cess that has a major goal of ensuring that the agency’s diverse research efforts complement one another. Topics assigned to the STAR program are ones that ORD’s internal research capabilities are not able to address effectively in a timely manner or for which independent analyses are desired. The STAR program draws on experts outside the agency to address mission-related research questions. EPA is thus able to develop and foster relationships with researchers to address agency needs without greatly expanding in-house staff. The agency is able to use those researchers when necessary to respond to issues that are not within the expertise of in-house scientists or when required to address emerging problems.

The discussions with the STAR project officers and other EPA officials within ORD indicated that STAR-funded research complements and enhances EPA’s in-house research. For instance, for the endocrine-disruptors program, STAR funding is used to fund research in topics not being addressed in-house. The STAR program has funded exposure assessment and epidemiologic studies on endocrine disruptors—subjects on which in-house laboratories have not focused. Similarly, for the ecologic-indicators program the majority of research on the development of indicators is funded through the STAR program, and in-house research is focused on the “proof of concept” or “implementation” of the indicators in the field (B. Levinson, EPA, Washington, D.C., personal commun., August 5, 2002).

The STAR program further attempts to ensure and promote complementarity through its progress-review workshops, which include all STARsupported principal investigators and representatives of all the other ORD research facilities working on a common topic.

The committee concludes that the STAR program is well coordinated with the rest of the agency’s research efforts and is a necessary asset of these efforts. The STAR program allows EPA to access resources that are not available in-house, thereby substantially expanding the scientific capabilities available to assist the agency in formulating its environmental-management initiatives.

Is the program well balanced?

The committee heard presentations from several NCER officials about the balance that the STAR program maintains between different types of research, including core and problem-driven, ecologic and health, and center and individual-investigator awards.

As indicated in Table 2-1, the STAR program maintains a diverse re-search portfolio. Much of the research is focused on particular issues of

Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×

immediate interest to the agency, but some addresses more basic, “core,” or possible future environmental issues. Much of the research addresses human health issues, but this is balanced by ecologic and socioeconomic topics. Some research is undertaken by individual investigators, but multidisciplinary research centers are increasingly important.

The program appears to face a tension between maintaining a balance among different subjects of research and addressing the most important issues that the rest of the agency is attempting to deal with. To some extent, those distinctions are not as meaningful as they might appear. For instance, much of what is termed ecologic research actually elucidates the processes by which environmental stressors affect both humans and other ecosystem attributes. Many of the issues of balance were addressed in the discussion under the metric Is the STAR portfolio appropriately mixed between core and problem-driven research and between human health and ecologic research? in the section on relevance, discussed earlier.

The question of whether the program should depend primarily on individual investigator awards or continue to increase the proportion of the budget allocated to supporting research centers appears to depend on the specific issues being addressed. Individual investigator grants may be better in particular instances: when the research topic is narrowly defined, for core research, and for exploratory research. Research-center grants are effective for addressing complex, interdisciplinary problems.

The committee concludes that the program is reasonably balanced and that it is important for it to continue to maintain such a balance to address the agency’s overall mission.

Does the program award grants expeditiously?

Through presentations by EPA staff and interviews with STAR project officers, the committee obtained substantial information about the process that the STAR program uses in awarding grants. That information is summarized in Chapter 2.

It takes 1-2 years from the initial announcement of the RFA to the time when grants are awarded (see Table 2-7). To someone inexperienced with federal government procedural requirements, the grant-awards process might appear unduly bureaucratic and protracted. However, there appear to be few opportunities for reducing this processing time substantially without compromising the program’s efforts. For instance, processing time ensures that the RFAs are widely disseminated and that the award process avoids any appearance of conflict of interest. However, the committee

Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×

learned that NCER is hiring additional staff to assist in the grant awards process to shorten processing time.

Evaluating the efficiency of the process would be much easier if the program established performance benchmarks that indicated the duration of the various steps in an expeditious grants-award process and kept track of the extent to which the program met the benchmarks. In addition to the time required to complete the various steps in awarding a grant (see Table 2-7), the benchmarks could include the number of progress-review meetings grantees participate in, how long it takes to publish a project’s annual report on the STAR Web site, and the number of projects being completed according to the original schedule. The benchmarks should be established by using information obtained from other research-sponsoring organizations and EPA’s own experience in managing the STAR program. Such benchmarking provides more useful information than process times alone, because it incorporates a comparison of process times with reasonable expectations.

The committee concludes that NCER has established a reasonably expeditious process for awarding grants but recommends that the STAR program establish benchmarks that measure a successful grants-management process and that it keep track of the percentage of the program’s grants that meet these benchmarks.

Does the program have a process to demonstrate the communication of individual grant results in the professional literature?

The STAR program collects information on the publications of its investigators and makes this information readily available and searchable on its Web site. However, the information appears to be collected primarily during the period when the investigators are being supported by STAR grants and not necessarily after grants have been completed.

The committee commends the STAR program for making the bibliometric information it collects readily available to the public. The committee encourages STAR to continue to gather and update bibliometric information from its investigators, even after the research has been completed.

Is there a process in place for reviewing the performance of individual investigators and research centers?

The STAR program does not have a formal process in place for assessing the performance of its principal investigators and research centers. The

Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×

program’s project officers are primarily responsible for monitoring individual projects and for ensuring that they comply with grant requirements, but not for evaluating the content of the research.

However, the program has established mechanisms for peer review of research performance. For individual investigators these mechanisms include requiring that researchers submit annual progress reports that are published on the NCER Web site and participate in progress review meetings. Additional efforts to monitor the performance of individual grantees might not be worth while. Not only would they impose additional costs on the agency and the researchers, but the grant would probably be largely completed by the time any problems were identified. The main concern of the program would appear to be to avoid awarding additional grants to investigators who had a history of performing badly. Interviews with project officers indicated that the program has no mechanism for collecting such information.

In addition to the requirements regarding annual reports and progress review meetings, the grants to research centers also require that the centers establish external advisory committees, comprising academics and government officials, who meet annually to provide guidance and evaluate research progress. In addition, EPA’s Scientific Advisory Board did conduct a review of the PM centers and could undertake other similar reviews (EPA/SAB 2002; see Appendix B).

The committee concludes that the processes EPA has established for providing peer review of the performance of individual investigators and research centers is adequate. Such peer review procedures may not ensure performance, but it is highly unlikely that the costs and disruptions that additional review efforts would impose on the agency and the researchers would be justified by any resulting performance improvements.

Product Metrics

Is the program funding relevant research that otherwise would not be funded?

On the basis of interviews with agency staff and researchers who have received STAR support, the committee concludes that much of the missionrelated research conducted through individual investigator and center awards would not have been possible without the STAR program, inasmuch as these research projects are not funded by other agencies. For instance, EPA is one of the few agencies that provide extramural funding for examin

Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×

ing the effects of endocrine disruptors on wildlife (E. Francis, EPA, Washington, D.C., personal commun., August 6, 2002). In addition, STAR’s ecologic-indicators program is the primary source of support for research on the development of water-quality indicators for biologic monitoring (B. Levinson, EPA, Washington, D.C., personal commun., August 4, 2002).

Does the program have a schedule for the products it intends to produce, and how well does it adhere to that schedule?

Although no such schedule was presented to the committee, OMB has requested that agencies prepare such a schedule in its guidelines for evaluating research programs under GPRA (OSTP/OMB 2002), and such specification of program outputs is a normal component of budget preparations. Those documents, however, are not usually made available to the public.

Because of the STAR program’s heavy reliance on its Web site for communicating its activities to the public, placing such a schedule on its Web site might substantially enhance the efficiency of the program’s communication efforts. The committee, on the basis of discussions with EPA officials, learned that EPA program offices may not always be as aware of recent postings on NCER’s Web site as they should be. If STAR listed its expected products on NCER’s Web site and allowed users of the site to indicate that they would like to be notified when an expected product became available, the likelihood of being informed about issues could be substantially increased.

The committee recommends that the program post a schedule of expected products on its Web site and allow members of the public to indicate whether they would like to be notified when products become available. This not only would provide the public with an opportunity to observe the performance of the program but also would improve the efficiency of its communication efforts.

To what extent are site-specific studies designed to be replicated at other locations?

Some of the STAR research efforts are focused on developing analytic tools or measures of environmental health for particular geographic areas. A major purpose of such grants is to develop approaches that could be adopted in other locations.

The committee was presented with information about the extent to which biologic sampling protocols were replicated at 822 reference sites in

Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×

13 western states—work supported by the U.S. Forest Service and EPA (Gilman 2002). The STAR program also supported a pilot program in the mid-Atlantic states to “develop methods to transfer STAR grant results to environmental decision-makers, test these methods in the mid-Atlantic region, and evaluate the feasibility of using the methods in other regions” (Bradley 2002). At the time the committee was briefed on this effort, limited results were available, although the program had identified a number of regional and local decision-makers who “have found the results of the STAR program valuable” (Bradley 2002).

The pertinent questions that reviews of such projects should be addressing are the extent to which the research projects were intended to be replicable, what efforts have been made to promote such replications, the extent to which replications have occurred, and what steps could be taken to promote their occurrence. The committee recommends that the program continue its efforts to collect information to answer those questions.

FELLOWSHIP PROGRAM

The STAR fellowship program is a small component of the overall STAR program whose goals and objectives differ from those of the main research grants program. The fellowship program is an important contribution to the nation’s effort to train and “encourage promising students to obtain advanced degrees and pursue careers in environmentally related fields” (EPA 2002b) and to develop the next generation of environmental scientists. The program is the only federal fellowship program designed exclusively for students pursuing advanced degrees in environmental sciences and engineering. It is highly competitive: only 10% of applicants receive funding. Because the fellowship program has been important in encouraging and maintaining a strong interest in environmental science and engineering, the committee considers that the program should be continued and funded.

Until 2002, the STAR program was funding 100-125 fellows per year. However, the president’s FY 2003 budget did not contain funding for the program, so no new fellowships were awarded in 2002. (NOTE: The FY 2003 Omnibus Appropriations Act, signed February 2003, appropriated $9.75 million for the STAR fellowship.)

Although the program publishes on the NCER Web site information about all the students receiving fellowships, it does not gather systematic information to track the status of past and currently funded fellows to assess the impact of the STAR program on their careers. To gather information on

Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×

the influence of the fellowship program, the committee contacted more than 100 STAR fellows who were initially funded in 1995 and 1996 and who would have completed their graduate work. Of the fellows contacted, over 95% indicated high satisfaction with the program. Additional information gathered by the committee permitted it to evaluate the following metrics.

Does STAR have a process for ensuring the selection of high-quality fellows?

The committee gathered information on how the STAR program selects fellows through discussions with people within EPA’s peer-review division.

The STAR program’s process for selecting high-quality fellows ensures the competitiveness of the program. The program publishes the announcement for the fellowships widely, including posting it on the NCER Web site and distributing it via e-mail to obtain a large pool of applicants. Prospective fellows submit applications that are evaluated in an independent peerreview process that is similar to the one for reviewing individual investigator and center grants (see Chapter 2 for details). Fellowship applications are evaluated according to criteria that includes their academic and employment records, course of proposed research, and potential for success. To receive continued funding, STAR fellowship recipients are required to remain in good academic standing. A fellowship may be terminated if the EPA project officer determines that the fellow is not performing up to the standards of the program.

On the basis of the wide dissemination of the fellowship applications, the peer-review process used for selecting applicants, and the low percentage of applicants who receive support, the committee concludes that the STAR fellowship program ensures a process for selecting high-quality applicants.

What percentage of fellowship recipients obtain their advanced degrees?

Although the program apparently does not collect the information required to assess this metric, the committee’s contacts with more than 100 STAR fellows who received funding in the early years of the program indicated that nearly all have already completed their research and graduated in their degree programs. That is a testament to the quality of the process for selecting fellows and to the success of the program. The committee suggests that if the fellowship program continues, EPA may want to collect metrics on the fellows, including the number graduating and positions held, to document the success of the program.

Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×

How many fellowship recipients who have completed their graduate work are working in environmental science?

The program has not collected the information required to assess this metric, but nearly 90% of the fellows contacted by committee members were employed in the environmental science field. About 10% of those were working in government, over 55% chose to remain in academe, and the remainder were working in industry, in consulting, or in nonprofit organizations. Many stated that the fellowship was extraordinarily valuable in assisting them to advance their work in environmental science (see Box 5-4).

The committee suggests that if the fellowship program continues, EPA may want to collect information on the careers pursued by fellowship recipients to document the success of the program.

How many fellowship recipients have completed their degree programs with at least one peer-reviewed publication?

Nearly 80% of the STAR fellows contacted by committee members indicated that they had at least one peer-reviewed publication as a result of their research funded through the fellowship program. That is an indication of the quality of the fellows and the quality of their research. The committee suggests that if the fellowship program continues, EPA may want to collect information on publications and other products of fellowship recipients to document the success of the program.

CONCLUSIONS

• The committee conducted an evaluation of the quality, relevance, and performance of the STAR program, as set forth in the recent OMB research and development criteria, using metrics that grew out of its review of information available from EPA and of metrics used by EPA and other organizations. The metrics, which are both quantitative and qualitative, assisted the committee in forming judgments regarding the scientific merit of the program and its impact on the agency.

• The committee was able to evaluate the program’s process better than its results. Evaluation of research results requires a substantial lapse in time, in that it takes 3-5 years, or more, from the initiation of laboratory or field experiments until the analysis and publication of research results. Considerably more time must elapse to view the impact of the published research on the scientific and regulatory community. Advances in research

Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×

BOX 5-4 Responses from STAR Graduate-Student Fellows Regarding the Value of the STAR Fellowship

“Yes, it was valuable in the sense that it helped my mentor pay me as a graduate student. It was valuable to me as an initial award on my CV that I have followed up with additional fellowships throughout the years. This gives me a track record of receiving funding. This track record has and will continue to assist me in my future endeavors to obtain funding/jobs. I believe there are not that many opportunities for graduate students to obtain independent funding and the STAR program was valuable in that sense.”

“Yes, The STAR fellowship gave me the freedom to expand my thesis topic into very interesting directions that led to several publications in diverse journals.”

“It was an invaluable resource. It freed me from teaching so that I could focus on my Ph.D. research. I believe I was able to finish sooner and investigate more than I would have otherwise. It enabled me to buy textbooks to learn about the field that I was working in. It enabled me to attend conferences that I would not have otherwise attended. By attending these conferences I met others in the field, including other graduate students I am still in contact with, and I gained confidence in myself and my abilities. After graduation, I received a National Science Foundation postdoc and went to Rensselaer Polytechnic Institute (RPI) for three years. Without the EPA fellowship, I might not have had the confidence or the ability to apply for the postdoc and write a successful proposal.”

“Yes, very much so. I am presently working on the same system answering additional questions raised during my Ph.D. Not only did the fellowship allow me to complete my dissertation, it has in part spawned an entire research program in the evolution of development of stickleback here at the university. We presently have approximately 10-12 people working on this project, and are collaborating with labs at SUNY Stony Brook, Stanford, and Clark University.”

“It was very valuable; in particular having a research budget gave me a lot more independence and flexibility in designing and carrying out my dissertation research. The program is also well known and respected. When I interviewed for jobs after graduation, I think my having received the STAR fellowship strengthened my position considerably.”

tend not to be accomplished by the completion of individual research grants, but rather through the combined impacts of multiple research projects on a specific field.

Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×

• The committee recognizes that there have already been a substantial—some might say excessive—number of reviews of the STAR program. Most have focused on the administration and operation of the program. The committee is concerned that reviewing STAR too frequently has the potential to be damaging, in that it may divert necessary financial and personnel resources away from the program. The committee therefore recommends that STAR and ORD consider an evaluation structure for conducting future reviews by independent panels of experts, comprising individuals with the appropriate scientific, management, and policy backgrounds. Expert reviews are the best method of evaluating the quality of a research program, and having a structured framework would probably reduce the number of ad hoc, unplanned, and uncoordinated reviews.

• The STAR program funds important research that is not conducted or funded by other agencies. The STAR program has also made commendable efforts to leverage funds through establishment of research partnerships with other agencies.

• Although it is still too early for comprehensive evaluations of the research results of the STAR program, some STAR research efforts have already substantially improved the scientific foundation for decision making and the results produced by STAR investigators have been widely published in peer-reviewed journals.

• It is appropriate for EPA’s research efforts to incorporate a balance between core and problem-driven research and a balance between ecologic and health-effects research. STAR research improves the knowledge base required to make sound environmental decisions, and this includes both core and problem-driven research. A balance between human-health and ecologic research is necessary, particularly because much of what is termed ecologic research actually elucidates the processes by which environmental stressors affect both humans and other ecosystems.

• The committee encourages the STAR program to continue funding research that explores future environmental problems within its overall research portfolio. Research devoted to potential environmental threats may help to avoid or reduce the impact of such threats or at the very least put into place the scientific capacity to address them.

• Although the STAR program has used several methods to report on the results of individual grants and centers, it has yet to produce documents that summarize the “state of the science” or provide a synthesis of research results and describe how the results of a group of grants have moved scientific understanding forward. The production of such reports, using outside experts where appropriate, can be extremely useful for targeting gaps in

Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×

knowledge and communicating the state of the science to the program’s diverse users and audiences. The committee considers the increased production of such reports to be an important improvement that should be made in the STAR program. The appropriate type of state of the science or research synthesis document will depend on the intended audience. Because the STAR supported research often complements that being done elsewhere in ORD, and sometimes in other agencies, the integration and synthesis of research results is a larger issue that in many cases cannot be taken solely by the STAR program and must be addressed by ORD or EPA.

• The STAR program has been commendably aggressive in experimenting with innovative approaches to communicating the results of its funded research to a wide variety of users and audiences, but its success in these efforts has been uneven. EPA and the STAR program have various mechanisms for communicating with the STAR user community. However, the STAR program has not developed an effective strategy for communicating to a wider user community, including state, tribal, local, and international environmental agencies and the public; most of the emphasis has been on the scientific community and the program offices. In some cases, the effective dissemination of results should be primarily STAR’s responsibility. In other cases, STAR’s contributions will be only one component of a larger research effort, and the primary dissemination responsibility should lie within ORD or EPA.

• The fellowship program is an important and valuable component of the STAR program for EPA and the nation. It ensures a continuing supply of graduate students in environmental science and engineering who provide a strong foundation for the nation’s environmental research and management efforts. The program has been important in encouraging and maintaining strong interest in environmental science and engineering.

RECOMMENDATIONS

• The committee recommends that NCER institute a structured system of program-level reviews as its primary mechanism for evaluating the STAR program. The improved information-collection efforts (discussed in Chapter 5) should be used to support such reviews.

• The committee recommends that STAR and ORD continue to work to produce state-of-the-science and research-synthesis documents. These are important for identifying critical information gaps and communicating the state of knowledge on a particular issue to the many users and audiences interested in this information.

Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×

• The committee commends EPA for its efforts to communicate with its diverse users and audiences and recommends that STAR and EPA continue and, where appropriate, expand such outreach efforts. The likely audiences of research results should be identified early in the research planning process, explicitly identified in RFAs, and considered throughout the research implementation process; and a coherent strategy should be developed for disseminating research results when they become available.

• STAR program funding should be maintained at 15-20% of the overall ORD budget, even in budget-constrained times. However, budget planners should clearly recognize the constraints of not having inflation escalators to maintain the level of effort of the entire program.

• EPA should continue its efforts to attract “the best and the brightest” researchers to compete for STAR funding.

• Given the nation’s continuing need for highly qualified scientists and engineers in environmental research and management, the STAR fellowship program should be continued and funded.

REFERENCES

Blanck, H.M., M. Marcus, P.E. Tolbert, C. Rubin, A.K. Henderson, V.S. Hertzberg, R.H. Zhang, and L. Cameron. 2000. Age at menarche and tanner stage in girls exposed in utero and postnatally to polybrominated biphenyl. Epidemiology 11(6):641-647.

Bradley, P. 2002. Pilot Program Overview. Presentation at the First Meeting on the Review of EPA’s Research Grants Program, March 19, 2002, Washington, DC.

Bryan, E. 2002. Peer Review Process for EPA’s National Center for Environmental Research’s Science to Achieve Results Program. Presentation at the Third Meeting on the Review of EPA’s Research Grants Program, June 6, 2002, Washington, DC .

Butler, T.J., G.E. Likens, and B.J.B. Stunder. 2001. Regional-scale impacts of Phase I of the Clean Air Act Amendments in the USA: The relation between emissions and concentrations, both wet and dry. Atmos. Environ. 35(6):1015-1028.


Cass, G.R., L.A. Hughes, P. Bhave, M.J. Kleeman, J.O. Allen, and L.G. Salmon. 2000. The chemical composition of atmospheric ultrafine particles. Philosophical Transactions: Mathematical, Physical and Engineering Sciences 358 (1775):2581-2592.

Clarke, R.W., P.J. Catalano, P. Koutrakis, G.G. Murthy, C. Sioutas, J. Paulauskis, B. Coull, S. Ferguson, and J.J. Godleski. 1999. Urban air particulate inhalation alters pulmonary function and induce pulmonary inflammation in a rodent model of chronic bronchitis. Inhal. Toxicol. 11(8):637-656.

Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×

Cooke, J.B., and D.E. Hinton. 1999. Promotion by 17beta-estradiol and beta-hexa-chlorocyclohexane of hepatocellular tumors in medaka, Oryzias latipes. Aquat. Toxicol. 45(2):127-145.

Deegan, L.A., J.T. Finn, and J. Buonaccorsi. 1997. Development and validation of an estuarine biotic integrity index. Estuaries 20(3):601-617.


EPA (U.S. Environmental Protection Agency). 1998a. Ecological Research Strat-egy. EPA/600/R-98/086. Office of Research and Development, U.S. Environ-mental Protection Agency, Washington, DC [Online]. Available: http://www. epa.gov/ordntrnt/ ORD/WebPubs/final/eco.pdf [accessed Jan. 13, 2003].

EPA (U.S. Environmental Protection Agency). 1998b. Research Plan for Endo-crine Disruptors. EPA/600/R-98/087. Office of Research and Development, U.S. Environmental Protection Agency, Washington, DC [Online]. Available: http://www.epa.gov/ORD/WebPubs/final/revendocrine.pdf [accessed Feb. 20, 2003].

EPA (U.S. Environmental Protection Agency). 1999. Airborne Particulate Matter Research Strategy. EPA/600/R-99/045. Office of Research and Development, U.S. Environmental Protection Agency, Washington, DC. [Online]. Available: http://www.epa.gov/ORD/resplans/Pmstrat7.pdf [accessed Feb. 20, 2003].

EPA (U.S. Environmental Protection Agency). 2000. Evaluation Report: A Deci-sion Making and Valuation for Environmental Policy Interim Assessment. National Center for Environmental Research, Office of Research and Develop-ment, U.S. Environmental Protection Agency [Online]. Available: http://es. epa.gov/ncer/science/economics/reviews.html [accessed Jan. 13, 2003].

EPA (U.S. Environmental Protection Agency). 2002a. Report to ORD’s Board of Scientific Counselors, Self-Study Update. Prepared by the National Center for Environmental Research, Office of Research and Development, U.S. Environ-mental Protection Agency. January 2002.

EPA (U.S. Environmental Protection Agency). 2002b. Fall 2002 Science to Achieve Results Fellowships for Graduate Environmental Study. 2002 RFA. National Center for Environmental Research, Officer of Research and Devel-opment, U.S. Environmental Protection Agency.

EPA/BOSC (U.S. Environmental Protection Agency Board of Scientific Counsel-ors). 1998. Program Review of the National Center for Environmental Re-search and Quality Assurance (NCERQA). Final Report of the Ad Hoc Sub-committee on the Review of NCERQA. Board of Scientific Counselors, Office of Research and Development, U.S. Environmental Protection Agency, Wash-ington, DC. April 30, 1998.

EPA/NSF (U.S. Environmental Protection Agency and National Science Founda-tion). 2000. Interim Assessment for the Decision Making and Valuation for Environmental Policy Grants Program. Final Report. Prepared for National Science Foundation and U.S. Environmental Protection Agency, by Aspen Systems Corporation. April 17, 2000.

EPA/SAB (U.S. Environmental Protection Agency Science Advisory Board). 2001. The Science to Achieve Results (STAR) Water and Watersheds Grants

Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×

Program: An EPA Science Advisory Board Review. A Review by the Ecolog-ical Processes and Effects Committee (EPEC) of the EPA Science Advisory Board. EPA-SAB-EPEC-02-001. Science Advisory Board, U.S. Environmen-tal Protection Agency, Washington, DC [Online]. Available: http://www.epa. gov/science1/fiscal02.htm [accessed Jan. 13, 2003].

EPA/SAB (U.S. Environmental Protection Agency Science Advisory Board). 2002. Interim Review of the Particulate Matter (PM) Research Centers of the USEPA: An EPA Science Advisory Report. A Review by the PM Research Centers Interim Review Panel of the Executive Committee of the U.S. EPA Science Advisory Board (SAB). EPA-SAB-EC-02-008. Science Advisory Board, U.S. Environmental Protection Agency, Washington, DC. May 2002 [Online]. Available: http://www.epa.gov/science1/fiscal02.htm [accessed Jan. 13, 2003].

EPA/SAB/BOSC (U.S. Environmental Protection Agency Advisory Board and Board of Scientific Counselors). 2000. A joint SAB/BOSC report: Review of the Science to Achieve Results (STAR) program. EPA-SAB-EC-00-008. Science Advisory Board, Board of Scientific Counselors, U.S. Environmental Protection Agency [Online]. Available: http://www.epa.gov/sab/pdf/ec0008. pdf [accessed Jan. 13, 2003].

GAO (U.S. General Accounting Office). 2000. Environmental Research: STAR Grants Focus on Agency Priorities, But Management Enhancements Are Possi-ble: Report to the Chairman, Subcommittee on VA, HUD, and Independent Agencies, Committee on Appropriations, House of Representatives. GAO/RCED-00-170/B-142370. U.S. General Accounting Office, Washington, DC.

Gilman, P. 2002. Presentation at the First Meeting on the Review of EPA’s Re-search Grants Program, March 19, 2002, Washington, DC.


Hawkins, M.B., J.W. Thornton, D. Crews, J.K. Skipper, A. Dotte, and P. Thomas. 2000. Identification of a third distinct estrogen receptor and reclassification of estrogen receptors in teleosts. PNAS 97(20):10751-10756.

Hughes, C., W. Foster, S. Chan, L. Platt, S. Thompson, S. Hubbard, A. DuBose, and L. Tyrey. 2001. Extrapolation of rodent studies on amniotic fluid contam-inants to human populations. Human and Ecological Risk Assessment 7(5):979-1002.


Jenkins, R., R.A. Angus, H. McNatt, W.M. Howell, J.A. Kemppainen, M. Kirk, and E.M. Wilson. 2001. Identification of androstenedione in a river containing paper mill effluent. Environ. Toxicol. Chem. 20(6):1325-1331.

Juliano, J., and M.D. Sobsey. 1997. Simultaneous concentration of Crypto-sporidium, bacteria and viruses from water by hollow fiber ultrafiltration. In: Proceedings of the 1997 Water Quality Technology Conference, American Water Works Association, Denver, CO, 1997.


Locci, A.B., and J.F. Koonce. 1999. A theoretical analysis of food web constraints on walleye dynamics in Lake Erie. Pp. 497-510 in State of Lake Erie: Past,

Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×

and Future, M. Munawar, T. Edsall, and I.F. Munawar, eds. Leiden, The Netherlands: Backhuys Publishers.

Lovett, E, R.W. Clarke, R.L. Verrier, P. Koutrakis, J. Lawrence, J.M. Antonini, and J.J. Godleski. 1999. Rat cardiovascular dysfunction prior to death during exposure to concentrated ambient air particles. The Toxicologist 48(1-S):297.

Michalek, J.L., and J.L. Colwell. 2000. Monitoring development in Southwest Florida (1973-1995) using Landsat data and a hybrid change detection tech-nique. Natural Areas Journal April 2000.


NRC (National Research Council). 1998. Research Priorities for Airborne Particu-late Matter. 1. Immediate Priorities and a Long-Range Research Portfolio. Washington, DC: National Academy Press.

NRC (National Research Council). 1999a. Evaluating Federal Research Programs: Research and the Government Performance and Results Act. Washington, DC: National Academy Press.

NRC (National Research Council). 1999b. Research Priorities for Airborne Partic-ulate Matter. 2. Evaluating Research Progress and Updating the Portfolio. Washington, DC: National Academy Press.

NRC (National Research Council). 2001. Research Priorities for Airborne Particu-late Matter. 3. Early Research Progress. Washington, DC: National Acad-emy Press


OSTP/OMB (Office of Science Technology and Policy/Office of Management and Budget). 2002. FY 2004 Interagency Research and Development Priorities. Memorandum for the Heads of Executive Departments and Agencies, from John Marburger, Director, Office of Science and Technology Policy, and Mitchell Daniels, Director, Office of Management and Budget, The White House, Washington, DC. May 30, 2002.


Pope, C.A., R.T. Burnett, M.J. Thun, E.E. Calle, D. Krewski, K. Ito, and G.D. Thurston. 2002. Lung cancer, cardiopulmonary mortality, and long-term exposure to fine particulate air pollution. JAMA 287(9):1132-1141.

Preuss, P.W. 2002. National Center for Environmental Research, History, Goals, and Operation of the STAR Program. Presentation at The First Meeting on Review EPA’s Research Grant Program, March 18, 2002, Washington, DC.


Voinov, A., R. Costanza, L. Wainger, R. Boumans, F. Villa, T. Maxwell, and H. Voinov. 1999. Patuxent landscape model: Integrated ecological economic modeling of a watershed. Environmental Modelling and Software 14(5):473-491.


Xing, L., W.J. Welsh, W. Tong, R. Perkins, and D.M. Sheehan. 1999. Comparison of estrogen receptor alpha and beta subtypes based on comparative molecular field analysis (CoMFA). SAR QSAR Environ. Res. 10(2-3):215-237.


Zanobetti, A., and J. Schwartz. 2001. Are diabetics more susceptible to the health effects of airborne particles? Am. J. Respir. Crit. Care Med. 164(5):831-833.

Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×
Page 108
Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×
Page 109
Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×
Page 110
Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×
Page 111
Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×
Page 112
Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×
Page 113
Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×
Page 114
Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×
Page 115
Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×
Page 116
Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×
Page 117
Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×
Page 118
Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×
Page 119
Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×
Page 120
Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×
Page 121
Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×
Page 122
Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×
Page 123
Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×
Page 124
Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×
Page 125
Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×
Page 126
Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×
Page 127
Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×
Page 128
Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×
Page 129
Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×
Page 130
Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×
Page 131
Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×
Page 132
Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×
Page 133
Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×
Page 134
Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×
Page 135
Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×
Page 136
Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×
Page 137
Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×
Page 138
Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×
Page 139
Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×
Page 140
Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×
Page 141
Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×
Page 142
Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×
Page 143
Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×
Page 144
Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×
Page 145
Suggested Citation:"5. Taking the Measure of STAR." National Research Council. 2003. The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program. Washington, DC: The National Academies Press. doi: 10.17226/10701.
×
Page 146
Next: Appendix A: Biographic Information on the Committee to Review EPA's Research Grants Program »
The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program Get This Book
×
Buy Paperback | $59.00 Buy Ebook | $47.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The report favorably reviews the U.S. Environmental Protection Agency's competitive research grants program, finding that it has yielded significant new findings and knowledge critical for EPA's decision-making process. Established in 1995, the grants program was designed to enable the nation's best scientists and engineers to explore new ways to safeguard the environment and protect public health. The program awards about $100 million a year in grants and fellowships to independent investigators, multidisciplinary teams, and graduate students at universities and nonprofit institutions.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!