The Measure of STAR Review of the U.S. Environmental Protection Agency's Science To Achieve Results (STAR) Research Grants Program (2003) / Chapter Skim
Currently Skimming:
4. Measure for Measure
4. Measure for Measure
Pages 91-107
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 91... ...
Finally, bibliometric analysis, a common form of metric, is discussed in relation to the STAR program. Appendix C contains examples of metrics used by federal agencies, academe, and state governments in evaluating their research programs.
|
From page 92... ...
Quantitative metrics tend to be more useful at lower levels of evaluation, when information tends to be more discrete, such as a review of a specific grant or center, but become less useful as one evaluates higher levels of integration, such as a review of the entire STAR program. Qualitative metrics have the advantage of being multidimensional, that is, of comprising an intricate and complex set of measures.
|
From page 93... ...
In a crude way, process metrics represent internal program assessments and product metrics represent external program assessments. The emphasis on product metrics is understandable, inasmuch as it is important that programs focus on what they are accomplishing, but process metrics are also important, particularly in warning of possible problems in a research program.
|
From page 94... ...
It is most cost-effective to use information that is being (or should be) collected to support continuing effective program management.
|
From page 95... ...
Typically, the project officer is responsible for carrying out the process review at this level, which usually includes such issues as ensuring that the work is being conducted on schedule by the appropriate investigators, that annual and other reports are being submitted as required, and that other federal administrative requirements are being satisfied. A product review is also sometimes carried out for individual research projects.
|
From page 96... ...
. This is often the most efficient level for conducting product reviews focusing on the substance of the research.
|
From page 97... ...
It is concerned less with how the program operates internally than how it is related to the broader institution and how effective it is in responding to the information needs ofthe organization and other potential users ofthe research results. The fourth level addresses such issues as whether the research organization has properly identified its clients, how well the research planning process and the definition of specific research topics include the perspectives of potential users, how effective the research organization is in keeping potential users "plugged in" to the research as it progresses, and how well it disseminates research results to potential users.
|
From page 98... ...
For example, in basic research if you measure relatively unimportant indicators, such as the number of publications per researcher instead of the quality of those publications, you will foster activities that may not be very productive or useful to the organization. A successful performance assessment program will both encourage positive behavior and discourage negative behavior.
|
From page 99... ...
Word references are typically based on key words; citations of multiauthor articles tend to be truncated to two or three authors; and in highly collaborative, cross-disciplinary applied research (such as that sponsored by the STAR program) , results are published in diverse journals for instance, research addressing environmental causes of childhood asthma can appear in journals dealing with buildings, general medicine, toxicology, epidemiology, molecular biology, agriculture, or social science (Geisler 2000~.
|
From page 100... ...
states that the "consensus among the critics is that the metric has some merit, but its value as a 'stand-alone' metric is doubtful." The committee agrees with that assessment and recommends the use of bibliometric analysis only to support expert reviews; review by a group knowledgeable about a specific research topic will assist in placing the results of bibliometric analysis in the context of the current state of scientific knowledge. A bibliometric analysis commissioned by the committee analyzed results of grants awarded in response to two requests for applications: 10 ecologic-indicators grants and eight endocrine-disruptor grants funded by STAR in 1996.
|
From page 101... ...
To that end, it requires each agency to produce three documents: a strategic plan that establishes long-term goals and objectives for a 5-year period, an annual performance plan that translates the goals of the strategic plan into annual targets, and an annual performance report that demonstrates whether an agency's targets have been met. Federal research agencies have developed various planning processes in response to GPRA.
|
From page 102... ...
In February 2002, OMB proposed preliminary investment criteria that could be used for evaluating federal basic-research programs. The criteria quality, relevance, and leadership were a combination of criteria suggested by COSEPUP and by the Army Research Laboratory (ARL)
|
From page 103... ...
From Table 4-3, it is evident that OMB's R&D criteria are not separate from those of COSEPUP, ORD, or STAR, but rather comprise many ofthese other criteria or goals. Examining the OMB criteria in this context underscores the fact that the criteria encompass the objectives of EPA's mission and fall within the research criteria and goals established by COSEPUP and STAR.
|
From page 104... ...
104 Cal o 5 .
|
From page 105... ...
There tends to be little targeting of specific research topics except in broad terms, such as nanotechnology. Many ofthe evaluations are based on surveys of participating institutions and on data routinely collected at the state level, such as numbers of students enrolled in institutions of higher learning.
|
From page 106... ...
Thus, product reviews are necessary to ensure that a program is producing high-quality results. · The committee recommends that the STAR program consider establishing a structured schedule of expert reviews that has four levels: level 1, individual research projects; level 2, topics or groups of research projects; level 3, the entire STAR program; and level 4, the question of how the STAR program is related to the broader institutions of ORD and EPA.
|
From page 107... ...
1999. Evaluating Federal Research Programs: Research and the Government Performance and Results Act.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.