The Measure of STAR Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program (2003) / Chapter Skim
Currently Skimming:
5. Taking the Measure of STAR
5. Taking the Measure of STAR
Pages 108-146
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 108... ...
The process metrics are used to evaluate the adequacy of the operation or procedures of the STAR program, and the product metrics are used to evaluate the outputs ofthe program, such as the number of publications or the influence or effect the program has had or may have. The committee evaluated the STAR fellowship program independently from the grants program because it is a small part of the STAR program and operates somewhat independently.
|
From page 109... ...
The set of metrics used by the committee in evaluating the STAR grants program is presented in Table 5-1. The goals and objectives of the fellowship program differ from the grants program, and it is evaluated more briefly at the end of this chapter.
|
From page 110... ...
Do bibliometric and citation analyses demonstrate excellence in the program's research? RELEVANCE Process Does the STAR portfolio support EPA's mission, GPRA goals, and ORD's strategic plans, research strategies, and multiyear plans?
|
From page 111... ...
Can a link between STAR research and improved protection of human health and ecologic systems be identified? PERFORMANCE Process Is the STAR budget appropriate to fulfill the program's mission?
|
From page 112... ...
makes a substantial effort to reach out to a broad scientific community and to recruit the most capable scientists. NCER disseminates its RFAs widely through its Web site, the Fec'7eraI Register, announcements at professional meetings, and e-mail distributions to individuals or institutions that sign up on the STAR Web site.
|
From page 113... ...
The program selects peer reviewers from a large number of sources, including people who have served on previous panels, keyword searches of databases (such as, Community of Science, the National Library of Medicine's PUBMED, and faculty listings) , keyword searches of NCER's peerreview panelist information system, input from project officers and program-off~ce scientists, and lists of the attendees of pertinent technical conferences (Bryan 2002~.
|
From page 114... ...
Investigators are required to submit annual progress reports that describe the research being undertaken and its progress. The progress reports are reviewed by STAR project officers, and summaries are posted on the NCER Web site.
|
From page 115... ...
As mentioned in Chapter 2, numerous reviews have been done on the processes and operation of the STAR grants program, and not on the products of the grants (EPA/BOSC 1998; EPA 2000; EPA/NSF 2000; EPA/SAB/BOSC 2000; GAO 2000; EPA/SAB 2001; EPA 2002a)
|
From page 116... ...
The investigator mix also included young investigators who will be the leaders of the future. The committee notes that EPA's rate of funding of individual investigator and center awards tends to be lower than that of other federal grants programs, such as those sponsored by NSF and NIH; this reflects the competitiveness ofthe program.
|
From page 117... ...
The committee was presented with several examples of STAR-supported research efforts that had made significant contributions to scientific understanding in particular topics (see Boxes 5-1, 5-2, and 5-3~. For example, STAR-sponsored research in endocrine disruptors, particulate matter, and ecologic assessment has resulted in peer-reviewed groups of publications of immediate interest in understanding causes of, exposures to, and effects of environmental pollution.
|
From page 118... ...
As indicated in Table 4-1, the bibliometric analysis conducted for the committee indicated that the citation rate of publications that result from STARsupported research is similar to that of other research publications in the topics for which the analyses were undertaken. As stated in Chapter 4, the committee considers that bibliometric and citation analyses are important quantitative metrics for gauging the quality of research but cautions that these types of analyses have many limitations.
|
From page 119... ...
Taking the Measure of STAR 119
|
From page 120... ...
" In the committee's view, mission relevance requires that STAR research improve the knowledge base required to identify environmental issues and make sound environmental decisions. The current STAR portfolio includes both core research (research that provides an understanding of the structure and function of environmental systems, the effects of human perturbations on those systems, and the resulting effects on human health and quality of life)
|
From page 121... ...
Although the agency is the primary audience for the STAR program,
|
From page 122... ...
The committee heard presentations from several EPA representatives regarding the agency's research planning process, discussed this process with other NCER staff, and reviewed information relevant to this metric that was available on EPA's Web site. ORD invests a substantial amount of effort in the research planning process to ensure that the STAR portfolio will be relevant to EPA's mission and enhance ORD's ability to meet its strategic goals.
|
From page 123... ...
In addition, review of all the RFAs issued during FY 1999, 2000, and 2001 shows that the topics addressed in the RFAs are mission-relevant and related to the EPA and ORD strategic plans. Therefore, on the basis ofthe information available, the committee concludes that the STAR portfolio does support EPA's mission and GPRA goals and ORD's strategic plans, research strategies, and multiyear plans.
|
From page 124... ...
commented explicitly on the relevance of the STAR program to the EPA program offices, but this was not a "planned review." The research planning processes and the agency's budgeting process provide an element of such reviews in that the program's primary audience, EPA's program offices, has a substantial opportunity to comment on the relevance ofthe program's research end to influence its funding. However, those activities may not fall into the category of"external reviews" as defined by OMB, and they do not include audiences outside the agency.
|
From page 125... ...
However, the program has not yet effectively developed a strategy for communicating to its wider user community; most ofthe emphasis has been on the scientific community and the EPA program offices. Any user who lacks the sophistication or incentive to visit EPA's Web site and wade through individual grant reports or research publications is unlikely to benefit directly from the program.
|
From page 126... ...
If it addresses both human health and ecologic effects, how should it be cIassified, particularly if much of what is termed ecologic research actually elucidates the processes by which environmental stressors affect both humans and other ecosystem attributes? The committee did not attempt to conduct a comprehensive review of all of STAR's individual grants and center research plans to determine whether such balances existed.
|
From page 127... ...
Committee members obtained additional information from interviews with NCER staff. As indicated in Chapter 2, the STAR program produces two types of reports, STAR Research Capsules and STAR Integrated Topical Searches, t h a t p r o v i ~ e s u m m a r i e s o f i n ~ i v i ~ u a ~ r e s e a r c h e f f o r t s a ~ ~ r e s s i n g a p a r t i c u ~ a r topic.
|
From page 128... ...
In many cases, because the STAR projects represent only a part of the important research being done on the topic, such synthesis reports would need to include the research undertaken by other ORD components and perhaps other research centers. To reduce possible concerns about the objectivity or independence ofthese reports, they should not be prepared by EPA staff.
|
From page 129... ...
The committee concludes, on the basis of the data that it was provided (see Boxes 5-1, 5-2, and 5-3) , that some STAR research efforts have already significantly improved the scientific foundation for decision making.
|
From page 130... ...
The lack of growth in the STAR budget, however, reflects the lack of growth in EPA's total research budget. In fact, the proportion of the research budget allocated to the STAR program has increased slightly and now stands at about 1 SILO of the ORD budget.
|
From page 131... ...
Is the program effectively complementing ORE 's other research efforts? The committee heard several presentations about the agency's researchplanning processes, and it interviewed STAR project officers responsible for the PM, endocrine-disruptors, and ecologic-indicators programs about their efforts to ensure that STAR supported research is complementary to other research being sponsored by EPA.
|
From page 132... ...
The committee heard presentations from several NCER officials about the balance that the STAR program maintains between different types of research, including core and problem-driven, ecologic and health, and center and individual-investigator awards. As indicated in Table 2-1, the STAR program maintains a diverse research portfolio.
|
From page 133... ...
Does the program awar~grants expeditiously? Through presentations by EPA staff end interviews with STAR project officers, the committee obtained substantial information about the process that the STAR program uses in awarding grants.
|
From page 134... ...
However, the information appears to be collected primarily during the period when the investigators are being supported by STAR grants and not necessarily after grants have been completed. The committee commends the STAR program for making the bibliometric information it collects readily available to the public.
|
From page 135... ...
For individual investigators these mechanisms include requiring that researchers submit annual progress reports that are published on the NCER Web site and participate in progress review meetings. Additional efforts to monitor the performance of individual grantees might not be worth while.
|
From page 136... ...
Because of the STAR program's heavy reliance on its Web site for communicating its activities to the public, placing such a schedule on its Web site might substantially enhance the efficiency ofthe program's communication efforts. The committee, on the basis of discussions with EPA officials, learned that EPA program offices may not always be as aware of recent postings on NCER's Web site as they should be.
|
From page 137... ...
FELLOWSHIP PROGRAM The STAR fellowship program is a small component of the overall STAR program whose goals and objectives differ from those of the main research grants program. The fellowship program is an important contribution to the nation's effort to train and "encourage promising students to obtain advanced degrees and pursue careers in environmentally related fields" (EPA 2002b)
|
From page 138... ...
On the basis of the wide dissemination of the fellowship applications, the peer-review process used for selecting applicants, and the low percentage of applicants who receive support, the committee concludes that the STAR fellowship program ensures a process for selecting high-quality applicants. Whatpercentage offellowship recipients obtain their advanced degrees?
|
From page 139... ...
The committee suggests that if the fellowship program continues, EPA may want to collect information on publications and other products of fellowship recipients to document the success of the program. CONCLUSIONS · The committee conducted an evaluation of the quality, relevance, and performance of the STAR program, as set forth in the recent OMB research and development criteria, using metrics that grew out of its review of information available from EPA and of metrics used by EPA and other organizations.
|
From page 140... ...
740 tend not to be acco~hshcd by Sac colon of lndlvl~1 rcsc~ch ~~ but r~cr Cough tic co~lncd l-acts oF~blplc rcsc~ch projccts on ~ spcclAc Acid.
|
From page 141... ...
STAR research improves the knowledge base required to make sound environmental decisions, and this includes both core and problem-driven research. A balance between human-health and ecologic research is necessary, particularly because much of what is termed ecologic research actually elucidates the processes by which environmental stressors affect both humans and other ecosystems.
|
From page 142... ...
However, the STAR program has not developed an effective strategy for communicating to a wider user community, including state, tribal, local, and international environmental agencies and the public; most of the emphasis has been on the scientific community and the program offices. In some cases, the effective dissemination of results should tee primarily STAR's responsibility.
|
From page 143... ...
· Given the nation's continuing need for highly qualified scientists and engineers in environmental research and management, the STAR fellowship program should be continued and funded. REFERENCES Blanck, H.M., M
|
From page 144... ...
1999. Airborne Particulate Matter Research Strategy.
|
From page 145... ...
2000. EnvironmentalResearch: STAR Grants Focus on Agency Priorities, But Management Enhancements Are Possible: Report to the Chairman, Subcommittee on VA, HUD, and Independent Agencies, Committee on Appropriations, House of Representatives.
|
From page 146... ...
l 999b. Research Priorities for Airborne Particulate Matter.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.