Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 119
6
Quality and Impact Assurance
in the REE Agencies
Ensuring the high quality, or excellence, of research is a key element of
research management. But in mission-oriented research, such as that conducted
in the US Department of Agriculture (USDA) Research, Education, and Economics
(REE) mission area, impact assurance is equally important. Both quality assurance
and impact assurance are necessary follow-up activities for relevance assurance,
which was discussed in Chapter 4. The REE agencies have implemented several
changes to strengthen quality assurance during the last decade. Measuring
impacts is more difficult but has been accomplished for some dimensions. This
chapter considers quality and impact assurance processes in the REE agencies,
evidence regarding research impact, and how the processes can be improved to
address impacts of the research opportunities outlined in Chapter 3.
QUALITY ASSURANCE
Research quality the degree of excellence of research compared with other
work being conducted in a field rests on a foundation of scientific merit.
Research quality is best evaluated by professional peers selected for their exper-
~ According to the National Academies' Committee on Science, Engineering, and Public Policy,
"there are at least two aspects of quality—one absolute and one relative. The absolute aspects are
related to the quality of the research plan, the methods by which it is being pursued, its role in educa-
tion when conducted at a university, and the importance of its results to its sponsor, either obtained or
expected. The relative aspects pertain to its leadership at the edge of an advancing field. Although
the leadership aspect is generally important, the results might in some cases be of great importance to
an agency albeit not at the leading edge of a field" (NRC, 1999a).
119
OCR for page 120
120
FRONTIERS IN AGRICULTURAL RESEARCH
tise in the field being assessed, experience, and objective judgment. This
approach is known as peer review. Peer-reviewed science includes the peer
review of discoveries before publication or patenting and the evaluation of quality
of a research proposal, including the creativity of the idea, the technical sound-
ness and appropriateness of the experimental design, the relationship to other
scientific results from the literature, the record of the scientist or the scientific
team, and the likelihood of scientific advances or practical applications or impacts.
Peer review of research proposals is commonly used in allocating competitive
funding to ensure quality in research design. Quality assessment of research
outputs can include peer review of manuscripts, discoveries, research programs,
and the quality of the publication in which the research results are published.
Research quality depends heavily on research inputs, including the quality of the
scientists conducting the research.2 Rigorous evaluation systems for the scien-
tists conducting the research and incentive structures for rewarding high-quality
work and creative, independent thinking can contribute to ensuring high-quality
outcomes.
REE Quality-Assurance Mechanisms
The committee considered the REE agencies in light of metrics of quality
and mechanisms for quality assurance. Generally, the committee identified a
variety of quality-review and evaluation processes that are in place for all research
projects and programs in the REE system; they are summarized in Table 6-1.
According to the strategic plan and performance plans of ARS (USDA, 1999,
2000a), each of roughly 1,100 research projects undergoes external merit-based
peer review before new or renewed activities are begun. To ensure quality in the
workforce, all ARS employees, including the scientific employees, are subject to
annual performance reviews, and permanent scientists undergo a review of their
progress on a 3- to 5-year cycle (discussed later in this chapter). A series of
national programs reviews is designed to ensure the quality, relevance, effective-
ness, and productivity of the work being done in each national program.
As mandated by the 1998 Agricultural Research, Extension, and Education
Reform Act (AREERA) (US Congress, 1998), CSREES works with state part-
ners receiving formula funds to develop 5-year plans of work, which are reviewed
2The committee notes that some private-sector institutions organize research to yield high-quality
outcomes by emphasizing the hiring of high-quality researchers. For example, at 3M, high-quality
staff are hired and given an ample endowment and flexibility to work on anything that is of interest to
them (Arndt, 2002).
3ARS has recently restructured how it organizes and manages its national research programs
(USDA, 1999). It has aggregated its research projects into 22 national programs that are guided by
multidisciplinary teams of national program leaders. The national programs focus the work of the
agency on reaching the goals defined in the ARS strategic plan.
OCR for page 121
QUALITY AND IMPACT ASSURANCE IN THE REE AGENCIES
TABLE 6-1 Summary of REE Quality-Assurance Mechanisms
121
Agency
Mechanism
Agricultural Research
Service (ARS)
Office of Scientific Quality Review (established in 1999)
External peer review of projects before implementation
Five-year-cycle review of national programs
Annual reviews (project level and national-program level)
Location reviews (research units)
Review of quality of individual scientists annual performance
reviews for all ARS employees and 3- to 5-year Research Position
Evaluation System peer reviews for senior scientists
· Solicitation of input on quality of ARS science from peer scientists
and users
Cooperative State · Annualreview of individual projects
Research, Education, · Annual review of research programs
and Extension Service · Peer review of research proposals for competitive programs
(CSREES) · Review of output from and input into special grants (earmarks)
· External program review of the National Research Initiative by the
National Research Council (NRC, 2000)
Economic Research · Peer review (internal and external) of all published material
Service (ERS) · Rewards for productivity and high-quality performance (cash awards
and promotions)
· Internal peer-review system for social-science positions, the
Economist Position Classification System (established in 2000)
· External program review by the National Research Council (NRC,
l999b)
National Agricultural · Accuracy review using historical track records that compare forecasts
Statistics Service with final, market-derived numbers for production-related reports;
(NASS) analysis of sampling errors and nonsampling errors
· Analysis of each step of data collection, processing, and estimation of
statistics to evaluate the quality and accuracy of NASS reports
· Comparison of estimates with data sources outside the agency
· External technical review
Source: Data provided by REE agencies, 2001.
annually for quality (USDA, 1998~. In addition to quality review, these annual
reviews also address accomplishments of research with respect to strategic goals
and objectives, multistate activities, integrated research and extension, joint
activities, and stakeholder input. CSREES is working to jointly establish and
implement formal program-evaluation protocols, including expert assessments,
with university and other partners and collaborators, through its Office of Plan-
ning and Accountability (USDA, 2000b). Reviews of academic departments at
land-grant universities are also required by CSREES and occur on a 5-year cycle.
OCR for page 122
22
FRONTIERS IN AGRICULTURAL RESEARCH
CSREES-administered competitive-grants programs, such as NRI and IFAFS,
subject proposals for individual awards to an external merit-review process.
CSREES post-award management procedures ensure that funds are expended
according to the proposed plan of work, that progress reports are received and
published in the Current Research Information System (CRIS) database, and that
site visits and other oversight measures are performed as necessary. CSREES
also increasingly requires its awardees to present results of their work at national
and international scientific symposia. In addition to individual award evaluation,
the programs are reviewed for quality as a whole each time a new request for
abstracts is published. Special grants awarded to a particular institution can be
reviewed before funds are awarded, and outputs of the special grant research can
be subjected to peer review. An external review of a CSREES-administered
competitive-grants program, the National Research Initiative, was conducted by
the National Research Council in 2000 (NRC, 2000~.
According to its annual performance plan, strategic plan, and annual perfor-
mance report (USDA, 2000e, 2000f, 2001b), ERS systematically evaluates the
quality of its work and considers the factors that affect quality. As part of that
process, ERS conducts peer reviews before analysis is released, and the agency's
successful contributions to professional conferences and journals test the appro-
priateness and rigor of the research methods in its analyses with respect to disci-
plinary standards. The National Research Council provided oversight for a 2-year
review of the ERS program that was completed in 1998 (NRC, l999b). In
response to the report recommendations, ERS has taken a number of important
actions, including the creation of an internal peer-review system. On the basis of
recommendations of the Research Council study, ERS conducts broad reviews of
critical aspects of its programs. It also initiated a collaborative university-ERS
effort to measure the impacts of social-science research. The results of this analy-
sis prove helpful to the agency in considering how to measure impacts and thereby
the quality of its research.
According to its strategic plan and performance reports, NASS relies heavily
on customer satisfaction surveys and end-user meetings more related to rel-
evance assurance than to quality assurance to assess products and services. (See
Chapter 4 for a discussion of relevance assurance mechanisms.) NASS reports
that it uses historical track records to compare crop estimates and forecasts pub-
lished during the growing season with the end-of-season final estimates to evalu-
ate the accuracy of crop estimates and forecasts. NASS also uses external data
sources to check the accuracy of its estimates. The NASS strategic plan reports
that internal analysis of each step of data collection, processing, and estimation of
production and price statistics is conducted to ensure quality (USDA, 2000g) and
that professional standards are used in all major survey activities, including
sampling frame development, sample design, questionnaire design and pretest-
ing, data collection, analysis of sampling and coverage errors, nonresponse
analysis, imputation of missing data, weighting, and variance estimation (USDA,
OCR for page 123
QUALITY AND IMPACT ASSURANCE IN THE REE AGENCIES
123
2002a). NASS does hire consultants from academe as consultants on data-
collection issues, and periodic reviews by outside panels of technical experts are
used to evaluate quality and reproducibility (USDA, 2001 d, 2002a). However, it
is not clear from the strategic planning and performance documents that these
external reviews are conducted regularly (USDA, 2000g, 2001c, 2001d).
The committee was able to draw some general conclusions about the effec-
tiveness of REE quality-assurance mechanisms and their outcomes.
Peer Review
Internal peer reviews of individual projects are conducted annually and at
project completion in all the REE agencies. Research programs receive periodic
reviews, generally at 5-year intervals. The committee found that reviews tended
to report intermediate outputs, publications, presentations, patents, cultivars, and
breeds developed. Although such activities provide useful information, they are
by their nature directed more toward quantity than quality of the effort. External
peer review of project proposals and products is also occurring in the REE
agencies.
Intramural Funding
In response to previous studies critical of the lack of peer review of research
proposals in the ARS system, major changes were introduced in 1999, as required
by the 1998 AREERA (US Congress, 1998~. These changes mandated periodic
(usually 5-year) peer review of all research-project plans and creation of an Office
of Scientific Quality Review (OSQR). Review by scientists outside ARS is in-
volved in the process. Unsatisfactory research plans are not approved and must
be rewritten; in some cases, the project must be terminated.4 However, as of
40ffices of the area directors manage the postpanel activities of project plans receiving a "major
revision" or "not feasible" action class. Projects falling into these categories may be dealt with in
three ways (frequency of management action as of August 2002 is shown in parentheses): (1) prepare
a revised project plan for a re-review by the panel within 3 months (95%), (2) completely rewrite the
plan and have it peer-reviewed again by a new combination of reviewers about a year later (4%),
(3) terminate the project (less than 1%). Re-review by the original panel is the most common action.
A complete rewrite of the project plan and fresh peer review are most often the option chosen if the
reason for the poor score was the absence of key scientific expertise (usually a result of a vacant
position). The original panel reviewers are asked to provide a second review in such cases. Finally,
projects may be terminated if there are extenuating personnel issues or an inability to correct the
problems. Successful objectives from ARS projects that have been terminated because of overall
poor peer reviews were transferred to a complementary research project. The scientists involved were
either reassigned or placed under other personnel actions. Although grade level is not affected by the
reassignment, a scientist may lose his or her status as a lead scientist on a project that is terminated,
without a change in salary. The termination of projects is usually chosen only after a second review
of the proposal results in a judgment of "not feasible" or "major revision required."
OCR for page 124
24
FRONTIERS IN AGRICULTURAL RESEARCH
August 2002, OSQR reports that only five of several hundred projects have been
terminated; none of the terminations has resulted in the removal of a scientist.
The impact of an unsatisfactory research proposal on a scientist's performance
evaluation is handled case by case, depending on the nature of the problem.
The program appears to have encouraged the development of improved
research proposals, which does not necessarily guarantee that the quality of
research in a given unit will be improved but suggests that the quality of the
overall research program should increase with time. However, the low rate of
project termination resulting from either proposal review or performance evalua-
tion suggests limited accountability of the system. The major research costs
(salary for ARS scientists and laboratory personnel) constitute fixed expenses for
a unit regardless of whether specific research proposals have been approved, and
the procedure does not represent competition or ranking among proposals for
these funds. The national program leader for each research program category is
responsible for priority-setting and quality assurance but does not have the
requisite authority to move personnel or budget to various research centers to
ensure those outcomes.
The committee considered results of reviews of six national programs that
underwent peer review from February 2000 to August 2001 (Table 6-2~. The
most important evidence from Table 6-2 is that 23% of the program reviews
asked for major revisions. Of the 94 review-panel members, 12 were employed
by government agencies other than ARS,69 were university faculty members, 12
were industrial or private consultants, and one was employed by ARS. The
makeup of the review panels demonstrates that ARS programs are subjected to
outside review.
External peer-review mechanisms are also in place in ERS, where all pub-
lished materials are reviewed by experts for appropriateness for the category of
publication. ERS's FY 2000 performance report states that published research
meets peer-review standards for all five goals in every case (USDA, 2000e). ERS
also has an internal peer-review process in place for research proposals.
At NASS, in the currently narrowly defined focus of NASS efforts (see Chap-
ter 4), the quality-assurance processes are focused on the value of reports to
particular end users. Customer satisfaction surveys and end-user meetings
provide little information on the relative value of different types of estimates or
information. In addition to end-user feedback, NASS quality assurance would
also benefit from regular peer review of survey and estimation (forecast) methods
by academic and other government-agency statisticians to ensure that they reflect
the newest analytic techniques.
The committee considered REE's intramural research quality-assurance
mechanisms against the mechanisms for quality assurance used in two other fed-
eral intramural research programs those of the National Institutes of Health
(NIH) and the Environmental Protection Agency (EPA). Intramural research at
NIH has been subject to external scientific review since 1956. NIH's Office of
OCR for page 125
QUALITY AND IMPACT ASSURANCE IN THE REE AGENCIES
TABLE 6-2 Results of February 2000-August 2001 Review of Six ARS
National Programs
125
Action Needed after Review
National Program
(NP)
Number of
Projects
Reviewed Revisiona Revisionb RevisionC Revisions Feasiblee
No Minor Moderate Major Not
Manure and Byproduct 21 1 10 7 2 1
Utilization (NP 201)
Food Safety (NP 108) 62 1 21 21 18 1
Soil Resources and 33 8 14 6 5 0
Management (NP 206)
Plant Biological and 36 1 13 8 13 1
Molecular Processes
(NP 302)
Animal Health (NP 103) 35 l ~
Water Quality (NP 201) 31 7 12
v 15 9 2
8 4 0
Total 218 19 78 65 51 5
a No revision required. No revision is required, but minor changes to project plan may be made.
b Minor revision required. Project plan is basically feasible as written but requires some revision to
increase quality.
c Moderate revision required. Project plan is basically feasible as written but requires moderate
revision of one or more objectives, perhaps involving changes in experimental approaches, to increase
quality. Project plan may also need rewriting for greater clarity.
~ Major revision required. Substantial revision of one or more objectives is necessary, but project
plan should then be sound and feasible.
e Not feasible. Project plan has major flaws or deficiencies and cannot be simply revised to produce
sound project. If project is not terminated, complete redesign and rewriting are required.
Source: ARS, Office of Scientific Quality Review, 2001.
Intramural Research reported to the committee that NIH has a rigorous, largely
retrospective, review system in place in its intramural research programs. In
contrast with the review of extramural grants, which mainly assesses the quality
of proposed research, the work of all principal investigators is reviewed mainly in
retrospective fashion, in which the research program is evaluated in toto for its
overall goals, quality of research, and long-term objectives, based in specific
criteria (Box 6-1) (USDHHS, 2002~. In the case of new investigators, more
emphasis is placed on future plans. Principal investigators are either tenured or
on tenure track, designations conferred only after rigorous searches, peer review,
selection processes, and internal and external reviews of research programs. The
OCR for page 126
126
FRONTIERS IN AGRICULTURAL RESEARCH
OCR for page 127
QUALITY AND IMPACT ASSURANCE IN THE REE AGENCIES
127
OCR for page 128
28
FRONTIERS IN AGRICULTURAL RESEARCH
success rate for achieving tenure is about 65%, so 35% of the scientists who enter
a 6-year tenure-track period do not compete successfully for scientific research
resources (or leave for other reasons). Independent research support is provided
to all principal investigators, and their progress is evaluated at least every 4 years
by groups of outside experts, constituted as boards of scientific counselors. The
emphasis of the outside reviews is mainly on past accomplishments, although
future plans are presented to the review teams as well. Reviewers develop written
recommendations to increase, decrease, or hold constant the resources assigned
to principal investigators; these recommendations are acted on by the scientific
program leaders. During the past 5-year period in one of NIH's major institutes,
7% of principal investigators lost all research support, and 25% of principal
investigators had resources reduced. Resources of other principal investigators
were increased or held at the same levels, leaving room for expansion of new
initiatives and programs.
The committee also considered mechanisms for peer review of intramural
research at EPA. Peer review occurs at multiple levels. For example, EPA's
National Health and Environmental Effects Research Laboratory, in the Office of
Research and Development, conducts reviews of each of nine divisions, review-
ing research strategies and multiyear implementation plans, cross-cutting research
programs, specific scientists undergoing promotion, and investigator-initiated re-
search proposals. A variety of review processes are used, including ad hoc panel
reviews, internal review by EPA experts, Federal Advisory Committee Act
reviews, and reviews by ad hoc panels comprising a mixture of discipline-specific
external experts and internal standing committees. All major scientific or techni-
cal work products also undergo review by internal and external experts before
their release.
FINDING: The committee commends ARS and ERS for establishing
peer-review processes. The ARS peer-review process assists researchers
in producing higher-quality proposals, which are a necessary, but not an
exclusive, component of higher-quality research. However, the ARS
peer-review system appears to reward excellent research performance
adequately but may not adequately exclude poor research performance,
given the noncompetitive (unranked) nature of the peer-review process,
the extremely low rates of project termination, and the lack of impact of
poor performance in proposal peer review on personnel grade level.
RECOMMENDATION 7: The REE intramural research system should
strengthen quality control for poor research performance. Mechanisms
used at other federal intramural research agencies, including the
redirection of human or financial resources when quality is poor, could
be implemented.
OCR for page 129
QUALITY AND IMPACT ASSURANCE IN THE REE AGENCIES
129
The decision to terminate a project or a position should be made after a broad
review of all aspects of a research program, including research inputs and outputs.
Formula Funding
CSREES uses review mechanisms to ensure the quality of a diverse portfolio
of research, education, and extension programs with universities and other orga-
nizations. In some institutions, formula funds are built into base salary budgets
of individual faculty; in others, the funds are distributed as specific research grants
to individual faculty. The 1998 AREERA (US Congress, 1998) mandated mecha-
nisms to ensure that each proposal undergoes merit or peer review to determine
its quality before funding. Institutions eligible for formula funds are now required
to document in their plans of work "a description of the merit and/or peer review
process, . . . the selection of reviewers with expertise relevant to the effort, and
appropriate scientific and technical standards" (USDA, 1998~. The effectiveness
of these policies in achieving improved accountability, however, remains unclear.
Progress in formula-funded projects is assessed by CSREES through the receipt
of interim and terminal CRIS reports.
Competitive Grants
In the CSREES competitive-grant programs, the National Research Institute
(NRI), Initiative for Future Agriculture and Food Systems (IFAFS), and the Fund
for Rural America (FRA) competitive programs have used a competitive merit-
based, peer-review process to ensure quality. A National Research Council
review of the NRI concluded that the quality of scientific work in this program
was high (NRC, 2000~.
Special Grants
CSREES special grants are added to the annual budget of the agency by
congressional action. They are not subject to merit-based peer review and are
directed to specific locations. CSREES can only ensure that special-grant pro-
grams represent the best quality in the institution or program by subjecting to
peer review the output of special grants. The proposal or input of special grants
can also be reviewed before the awarding of funds. In some cases, CSREES does
not release the funds until the institution has hired a person with the necessary
expertise or partnered with another institution that has the necessary expertise.
FINDING: Several recent REE research programs (including the NRI,
FRA, and IFAFS) are based on competitive peer-reviewed funding
mechanisms, which make important contributions to ensuring the qual-
ity of the proposed research. Merit-based or peer-review mechanisms
OCR for page 135
QUALITY AND IMPACT ASSURANCE IN THE REE AGENCIES
Patents
135
Patents are the products of invention and can indicate that research results
are being accepted and used in the US economy. USDA received an average of
57 patents per year from 1996 to 2000 (see Chapter 5, Table 5-2~.
Social Rate of Return
The social rate of return on public research expenditures is the economist's
preferred tool for assessing impact; it provides a bottom-line estimate in the form
of a rate of return. The return can be compared with the social opportunity cost of
public funds. Sometimes it is difficult, however, to obtain the time profile of net
social benefits of research studies.7 Society will be best served if it invests its
research resources into projects that yield a sizable positive social marginal rate
of return.
Societal benefits in the form of rates of return from agricultural research
have been extensively reviewed (see Tables 6-5 and 6-6~. Rates of return are high
but vary widely. Evenson's summary of rates of return on public aggregates
agricultural research investments shows a median real social rate of return from
126 research studies of 45%; 66% were in the range of 21-80%. The research
studies summarized by Evenson generally estimate the market value of improve-
ments in agricultural productivity (this results in lower consumer prices or lower
costs for producers over time) and then compare that value with the costs of
investments in agricultural research. Alston et al. (2000) also published a formal
and extensive analysis (292 studies, 1,886 rate-of-return observations), which
econometrically accounts for the observed variation in the rates of return. That
study showed that the mean of the measured rates of return to research (averaging
over 1,144 observations) was 99.6% with a mode of 46%. Both meta-studies
support the finding that investments in agricultural research have generated high
and sustained returns to society.
7When individual projects are evaluated, it generally takes several years after a study is completed
before the social cost-benefit calculations can be made.
Public denotes a combination of state and federal, and the aggregate value is only a proxy for REE
research. Assessment of the impact of research supported by USDA formula and competitive funds
at universities is difficult. Individual universities decide how the formula funds are to be allocated
among competing needs, including research, education, and outreach. Federal funds from REE are
most often combined with other funds (from state, private, and other federal agencies), and attribution
of the individual contributions is nearly impossible with today's accounting procedures. Returns
from aggregate values seem most relevant here because they include successful and failed projects
and provide a better picture of the whole enterprise than rates of return for particular studies. In the
case of the SAKS, in the period 1927-1980, when these rates of return were calculated, most funding
came from state or federal sources, not from the private sector (see footnote 11).
OCR for page 136
136
FRONTIERS IN AGRICULTURAL RESEARCH
TABLE 6-5 Internal Rates of Return from US Public-Sector Agricultural
Research
Study
Method
Period
Internal Rate
of Return,a % Region
Peterson and Fitzharris (1977) Project evaluation
Peterson and Fitzharris (1977) Project evaluation
Peterson and Fitzharris (1977) Project evaluation
Peterson and Fitzharris (1977) Project evaluation
Norton and Paczkowski (1993) Project evaluation
Norton and Paczkowski (1993) Project evaluation
Griliches (1964) Statistical method
Latimer (1964) Statistical method
Evenson (1968) Statistical method
Cline (1975) Statistical method
Bredahl and Peterson (1976) Statistical method
Bredahl and Peterson (1976) Statistical method
Bredahl and Peterson (1976) Statistical method
Bredahl and Peterson (1976) Statistical method
Lu et al. (1979) Statistical method
Lu et al. (1979) Statistical method
Evenson (1979) Statistical method
Evenson (1979) Statistical method
Evenson (1979) Statistical method
Evenson (1979) Statistical method
Evenson (1979) Statistical method
Knutson and Tweeten (1979) Statistical method
White et al. (1978) Statistical method
Davis (1979) Statistical method
Davis and Peterson (1981) Statistical method
Davis and Peterson (1981) Statistical method
Davis and Peterson (1981) Statistical method
Davis and Peterson (1981) Statistical method
Welch and Evenson (1989) Statistical method
White and Havlicek (1982) Statistical method
Braha and Tweeten (1986) Statistical method
Evenson (1989) Statistical method
Alston et al. (1998a) Statistical method
Chavas and Cox (1992) Statistical method
Makki et al. (1996) Statistical method
Makki and Tweeten (1993) Statistical method
Oehmke (1996) Statistical method
Oehmke (1996) Statistical method
Yee (1992) Statistical method
Norton et al. (1992) Statistical method
1937-1942
1947-1952
1957-1962
1957-1972
1949-1979
1949-1989
1949-1959
1949-1959
1949-1959
1939-1948
1937-1942
1947-1957
1957-1962
1967-1972
1938-1972
1939-1972
1868-1926
1927-1950
1948-1971
1948-1971
1948-1971
1949-1972
1929-1977
1949-1959
1949
1954
1959
1964, 1969, 1974
1969
1943-1977
1959-1982
1950-1982
1930-1990
1930-1990
Pre-1930
1930-1990
1931-1985
1987
50
51
49
34
58
58
25-40
n.s.b
47
41-50
56
51
49
34
24-31
23-30
65
95
130
93
95
28-47
28-37
66-100
100
79
66
37
55
7-36
47
43
17-31
28
27
93
Negative
11.6
49-58
30
VA
VA
South
North
West
aThe internal rate of return is a discount rate at which the present value of a series of investments is
equal to the present value of the returns on those investments.
bn.s.= not significant.
Source: Adapted from Evenson (2001). Economic Impacts of Agricultural Research and Extension
in Agricultural Economics Volume la Agricultural Production, B.L. Gardner and G.C. Rausser, eds.
Amsterdam: Elsevier.
OCR for page 137
QUALITY AND IMPACT ASSURANCE IN THE REE AGENCIES
TABLE 6-6 Internal Rates of Return from US Extension
137
Internal Rate of
Study Period Extension Variable Return,a %
Huffman (1974)b 1959-1974 Extension staff/farm 16
Huffman (1976)b 1964 Staff days/farm 110
Evenson (1979)b 1971 Expenditure/region 100+
Huffman (1981)b 1979 Extension days/county 110
Evenson (1994)b 1950-1972 Expenditure/state Crops 101
Evenson (1994)b 1950-1972 Expenditure/state Livestock 89
Evenson (1994)b 1950-1972 Expenditure/state All 82
Huffman and Evenson (1993)C 1950-1982 Crops 40.1
Huffman and Evenson (1993)C 1950-1982 Livestock (negative)
Huffman and Evenson (1993)C 1950-1982 All 20.1
aThe internal rate of return is a discount rate at which the present value of a series of investments is
equal to the present value of the returns on those investments.
bEvenson, R. 2001. Economic impacts of agricultural research and extension. In Agricultural Eco-
nomics Volume la. Agricultural Production. B.L. Gardner and G.C. Rausser, eds. Amsterdam:
Elsevier. p. 593.
CHuffman, W.E., and R.E. Evenson. 1993. Science for Agriculture: A Long-Term Perspective. Ames,
IA: Iowa State University Press, p. 245.
Other general observations over the 1950-1982 period include a much higher
rate of return from preinvention or pretechnology research than from applied
research (Evenson, 2001), a higher rate of return from applied crop research than
from applied livestock research (Huffman and Evenson, 1993), and a lower rate
of return as the share of public agricultural research funding by federal contracts,
grants, and cooperative agreements increased (Huffman and Just, 1994~.
High rates of return are consistent with growth in total factor productivity.9
Jorgenson and Stiroh (2000) show that the US agricultural sector ranks third
among 37 US industries in aggregate total factor productivity growth over 1958-
1996 at 1.2%. New total factor productivity estimates by ERS for US agriculture
show an average annual rate of 2% for 1950-1998. Evenson (2001) describes
how a continuous investment in public agricultural research of 1% of output per
year will contribute 0.76% per year to total factor productivity growth when the
real rate of return is 40%.
9A total-productivity or multifactor productivity index measures the quantity of output produced
compared with the quantity or cost of all the measurable inputs used to produce it (Ahearn et al.,
1998). Growth in productivity indicates the growth in output unaccounted for by changes in mea-
sured inputs and is typically ascribed to investments in R&D, education, infrastructure, and econo-
mies arising from increasing the scale of production.
OCR for page 138
138
FRONTIERS IN AGRICULTURAL RESEARCH
The high rates of return from public agricultural research summarized by
Evenson (2001) pertain to investments made in publici° agricultural research from
1927 to 1980. Alston et al. (2000), using econometric evidence after accounting
for factors that cause the reported rates of return to vary among studies, estab-
lished that the rate of return over the last 2 decades has not declined.
It is important to discuss the inherent limitations of using social rate of return
as an impact indicator. For example, the estimates of rate of return do not account
for possible off-farm environmental costs or benefits (Jorgenson and Stiroh,
2000~. In addition, rate of return does not account for the full economic costs and
benefits for farms of different sizes and for agriculture-dependent communities
(for example, new technologies may enhance productivity of larger farms rela-
tively but contribute to the decline of smaller farms and the rural communities
that were once sustained by these farms) (Swanson, 1988; US Congress OTA,
1986~.
FINDING: The social rate of return on past public agricultural research
investments in the period 1950-1982 has been very high. The rate of
return over the last 2 decades has not declined. Social rate of return has
limitations in accounting for full environmental and social costs of re-
search.
Environmental, Economic, and Health Outcomes
Aside from the long-run benefits of productivity enhancement measured
through social rate of return, documented quantitative examples of the impact of
REE research on other outcomes are scarce. Box 6-3 provides several illustrative
examples of successful research that provided benefits to the environment, health,
or safety that are difficult to quantify on a national scale. As the examples show,
successful impact of specific research projects may be obvious, but difficult to
summarize in comparable measures. That is particularly true for environmental
benefits, human health benefits, or potential social benefits. Although the research
frontiers identified in this report will result in such benefits, they may be difficult
to quantify or compare among research goals and projects.
teethe private-sector share of the total support for SAESs was 7% in 1960, 9.2% in 1980, and 13.2%
in 1990. Before 1960, disaggregated data for the private-sector contribution are not available, but we
know that the sum of the private-industry contribution and other federal-government (non-USDA)
resources was not greater than 14.8% in 1900, 29.8% in 1930, and 22.6% in 1940. Thus, USDA
appropriations and state-government appropriations predominated as the source of support of the
SAESs (85% in 1900, 70.2% in 1920, 77.4% in 1940, 79.8% in 1960, and 72.5% in 1980) during the
period in which these rates of return were calculated (Huffman and Evenson, 1993).
OCR for page 139
QUALITY AND IMPACT ASSURANCE IN THE REE AGENCIES
139
OCR for page 140
140
FRONTIERS IN AGRICULTURAL RESEARCH
Monitoring and Communicating Impact
Mechanisms for tracking research investments and impacts are important
both for informing internal decision-making on future research investments and
for communicating with and improving accountability to the public. The com-
mittee considered a variety of mechanisms used by REE to track and communi-
cate its performance and impact. These are considered in detail in Appendix G.
This report has identified a need to broaden the REE agency focus on needs
of and impacts on new, nontraditional stakeholders and to be more strategic in
OCR for page 141
QUALITY AND IMPACT ASSURANCE IN THE REE AGENCIES
141
setting priorities for investments to meet national goals (Chapters 1 and 4~. To
achieve these goals, REE will need to improve and expand existing performance-
based systems for monitoring success. More-effective and more user-friendly
tracking systems will contribute to improved self-evaluation and reporting of
progress to groups outside REE. Electronic media are an increasingly critical and
strategic means for communicating impacts and research results to the general
public and should be a focal point for development and expansion. ERS's recent
redesign of its Web site to be more accessible and understandable to the public-
through provision of links to nontechnical summaries, technical abstracts, data,
and publications is an excellent model for how positive change could occur in
this regard throughout the REE mission area.
RECOMMENDATION 8: REE agencies should develop and adopt ways
of measuring the national, long-term impacts of their research on the
environment, human health, and communities. The tools should include
measures and indicators that are influenced by agricultural research or
that can be attributed to research outcomes, including how research sup-
ports the needs of action agencies. REE should strive to achieve greater
transparency in communicating these impacts through timely electronic
publishing of peer-reviewed results and through greater efforts to inter-
pret these results for a general audience.
The committee envisions that monitoring capability and development of
indicators would occur in parallel at two levels. First, monitoring capability could
be developed to show how REE research has changed in focus, relevance, quality,
leadership, and accountability (NRC, 1999a). For example, REE could track
progress toward meeting national goals by requiring that each major research
program or initiative establish performance objectives and measures for evaluat-
ing progress toward meeting national objectives, on the basis of some assessment
of adoption and/or implementation of research findings in practical applications
and possibly of how such adoption led to beneficial changes. In addition, REE
could track performance by keeping a comprehensive account of where research
funded or conducted by REE has made a critical contribution to advancing under-
standing, policies, and practices in each of the research frontiers recommended
by this study.
A second level of monitoring capability could be developed to show how
food, agricultural, natural, and human systems are changing. Because changes in
such indicators cannot be directly attributed to research, this class of indicators
cannot provide a direct measure of research performance, but it can be used to
help target future research directions. Such indicators might include nutritional
indicators, such as the healthy eating index measuring overall nutritional quality
of the American diet (Kennedy et al., 1999), and ecologic indicators, including
nutrient runoff and soil organic matter (NRC, 2000~. Indicators should be selected
after a set of defining criteria has been established, which might include a well-
OCR for page 142
42
FRONTIERS IN AGRICULTURAL RESEARCH
understood conceptual basis, reliability, applicability on clear temporal and spatial
scales, accuracy, sensitivity, precision, robustness, skill and data requirements,
data-quality requirements, archiving capability, international compatibility, costs,
benefits, and cost effectiveness (NRC, 20001. As discussed in Chapter 3, system-
atic research on the environmental, social, and community impacts of REE
research would inform this process.
SUMMARY
This chapter has considered quality-assurance and impact-assurance pro-
cesses and their outcomes in the REE agencies and has provided recommenda-
tions for improving the effectiveness of these processes. The use of peer review
as a quality-assurance mechanism for research inputs and outputs was discussed
with regard to intramural research, formula-funded research, special grants, and
competitive grants. REE staff-performance evaluation systems and reward and
incentive programs were also considered as mechanisms for ensuring the delivery
of high-quality science. In general, REE has strong and evolving quality-
assurance mechanisms in place, and REE scientists produce research that is of
high quality. However, human or financial resources should be redirected when
research is of poor quality in intramural research.
Pnmary, intermediate, and longer-term impacts of agricultural research were
descnbed. Much progress has been made in documenting research impact in the
traditional dimensions associated with improved productivity. Because in the
future more REE research will be directed at providing new kinds of benefits,
monitoring of research impact will require new outcome measures. The impor-
tance of mon~tonng, measunng, and communicating research investments and
their impacts was discussed, and changes in monitoring capability, in develop-
ment of indicators, and in communication of results were recommended.
REFERENCES
Ahearn, M., J. Yee, E. Ball, and R. Nehring. 1998. Agricultural Productivity in the United States.
Agricultural Information Bulletin, No. 740. Washington, DC: Economic Research Service, US
Department of Agriculture. Available online at http://www.ers.usda.gov/publications/aib740/.
Alston, J.M., M.C. Marra, P.G. Pardey, and T.J. Wyatt.2000. Research returns redux: A meta-analysis
of the returns to agricultural R&D. Australian Journal of Agricultural and Resource Economics
44(2): 185-215.
Amerman, R.C., G. Larson, and M. O'Neill. 2001. USDA Water Quality Initiative. Presentation to
National Research Council Committee on Opportunities in Agriculture, Public Workshop, May
22-23. Washington, DC.
Araji, A.A., F.C. White, and J.F. Guenthner. 1996. Returns from Potato Research: Accounting for
State and Regional Effects. Agricultural Experiment Station, Research Bulletin 152. Moscow,
ID: University of Idaho.
Arndt, M. 2002. 3M: A lab for growth? Business Week, Jan. 21.
OCR for page 143
QUALITY AND IMPACT ASSURANCE IN THE REE AGENCIES
143
Bostom, A.G., I.H. Rosenberg, H. Silbershatz, P.F. Jacques, J. Selhub, R.B. D'Agostino, P.W.F.
Wilson, and P.A. Wolf. 1999a. Nonfasting plasma total homocysteine levels and stroke inci-
dence in elderly persons: The Framingham study. Annals of Internal Medicine 131:352-355.
Bostom, A.G., H. Silbershatz, I.H. Rosenberg, J. Selhub, R.B. D'Agostino, P.A. Wolf, P.F. Jacques,
and P.W.F. Wilson. l999b. Nonfasting plasma total homocysteine levels and all-cause and
cardiovascular disease mortality in elderly Framingham men and women. Archives of Internal
Medicine 159:1077-1080.
Evenson, R.E. 2001. Economic Impacts of Agricultural Research and Extension. Pp. 575-628 in
Handbook of Agricultural Economics, Vol. 1: Agricultural Production, B.L. Gardner and G.C.
Rausser, eds. Amsterdam: Elsevier.
Frisvold, G., G.K. Agnew, and P. Baker. 2002. Effects of Insect Growth Regulators on Insecticide
Use and Costs in Arizona Cotton. Proceedings of the Beltwide Cotton Conferences 1. Memphis,
TN: National Cotton Council.
Huffman, W.E., and R.E. Evenson. 1993. Science for Agriculture: A Long-Term Perspective. Ames,
IA: Iowa State University Press.
Huffman, W.E., and R.E. Just. 1994. Funding, structure, and management of public agricultural re-
search in the United States. American Journal of Agricultural Economics 76(November):744-
759.
ISI (Institute for Scientific Information) 2001. ScienceWatch: Trends and Performance in Basic Re-
search. July/August. ISI Essential Science Indicators, 1991-2001. Philadelphia, PA: Institute
for Scientific Information.
Jorgenson, D.W., and K.J. Stiroh. 2000. US economic growth at the industry level. American Eco-
nomic Review 90(May):161-167.
Kennedy, E., S.A. Bowman, M. Lino, S.A. Gerrior, and P.P. Basiotis. 1999. Diet quality of Ameri-
cans: Health eating index. Chapter 5 in America's Eating Habits: Changes and Consequences,
E. Frazao, ed. Agriculture Information Bulletin No. 750. May. Washington, DC: Economic
Research Service, US Department of Agriculture.
NAS (National Academy of Sciences). 2001. Membership Directory. July 2001. National Academy
of Sciences of the United States of America.
NRC (National Research Council). 1999a. Evaluating Federal Research Programs: Research and the
Government Performance and Results Act. Washington, DC: National Academy Press.
NRC (National Research Council). l999b. Sowing the Seeds of Change: Informing Public Policy in
the Economic Research Service. Washington, DC: National Academy Press.
NRC (National Research Council). 2000. National Research Initiative: A Vital Competitive Grants
Program in Food, Fiber, and Natural-Resources Research. Washington, DC: National Academy
Press.
Selhub, J., P.F. Jacques, A.G. Bostom, R.B. D'Agostino, P.W.F. Wilson, A.J. Belanger, D.H. O'Leary,
P.A. Wolf, E.J. Schaefer, and I.H. Rosenberg. 1995. Association between plasma homocysteine
concentrations and extracranial carotid-artery stenosis. New England Journal of Medicine
332:286-291.
Selhub, J., P.F. Jacques, P.W.F. Wilson, D. Rush, and I.H. Rosenberg. 1998. Vitamin status and
intake as primary determinants of homocysteinemia in an elderly population. Journal of the
American Medical Association 270(20):2693-2698.
Seshadri, S., A. Beiser, J. Selhub, P.F. Jacques, I.H. Rosenberg, R.B. D'Agostino, P.W.F. Wilson, and
P.A. Wolf. 2002. Plasma homocysteine as a risk factor for dementia and Alzheimer's disease.
New England Journal of Medicine 346(7):76-73.
Swanson, L. 1988. Agriculture and Community Change in the US: The Congressional Research
Reports. Boulder, CO: Westview Press.
US Congress. 1998. P.L. (Public Law) 105-185. Agricultural Research, Extension, and Education
Reform Act of 1998.
OCR for page 144
144
FRONTIERS IN AGRICULTURAL RESEARCH
US Congress, OTA (Office of Technology Assessment). 1986. Technology, Public Policy, and the
Changing Structure of American Agriculture. OTA F-285. Washington, DC: US Government
Printing Office.
USDA (US Department of Agriculture).1992. Subduing the Screwworm. Agricultural Research. July.
Washington, DC: Agricultural Research Service, US Department of Agriculture.
USDA (US Department of Agriculture). 1998. Guidelines for Peer and Merit Reviews. Washington,
DC: Cooperative State Research, Education, and Extension Service, US Department of Agri-
culture.
USDA (US Department of Agriculture). 1999. Agricultural Research Service Strategic Plan: Working
Document 1997-2002. Washington, DC: Agricultural Research Service, US Department of
Agriculture. Available online at http://www.nps.ars.usda.gov/mgmt/stratpln/1999/background.
cam.
USDA (US Department of Agriculture). 2000a. Agricultural Research Service FY 2000 and 2001
Annual Performance Plans. Washington, DC: Agricultural Research Service, US Department of
Agriculture.
USDA (US Department of Agriculture). 2000b. Cooperative State Research, Education, and Exten-
sion Service FY 2000 and 2001 Annual Performance Plan. Washington, DC: Cooperative State
Research, Education, and Extension Service, US Department of Agriculture.
USDA (US Department of Agriculture). 2000c. Current Research Information System (CRIS) Fund-
ing Summaries, Table A, FY 2000. Washington, DC: US Department of Agriculture. Available
online at http://cristel. csrees. usda. gov/star/OOtablea. pdf.
USDA (US Department of Agriculture). 2000d. Economist Position Classification System. Decem-
ber 2000. Washington, DC: Economic Research Service, US Department of Agriculture.
USDA (US Department of Agriculture). 2000e. Economic Research Service FY 2000 Annual Perfor-
mance Report. Washington, DC: Economic Research Service, US Department of Agriculture.
Available online at http://www. ers. usda. gov/AboutERS/ersannualperformance. pdf.
USDA (US Department of Agriculture). 2000f. Economic Research Service Strategic Plan, 2000-
2005. October 11. Washington, DC: Economic Research Service, US Department of Agriculture.
Available online at http://www. ers. usda. gov/AboutERS/ersstrategicplan. pdf.
USDA (US Department of Agriculture). 2000g. GPRA Strategic Plan. Washington, DC: National
Agricultural Statistics Service, US Department of Agriculture. Available online at http://
www. usda.gov/nass/nassinfo/strat-2005.pdf.
USDA (US Department of Agriculture). 2000h. Peer Review of ARS Research Project Plans. Office
of Scientific Quality Review, Agricultural Research Service. November 22. Washington, DC:
Agricultural Research Service, US Department of Agriculture. Available online at http://
www. ars. usda. gov/osqr/OAManual . pdf.
USDA (US Department of Agriculture). 2001a. Data submitted to the National Research Council
Committee on Opportunities in Agriculture. Washington, DC: Economic Research Service, US
Department of Agriculture.
USDA (US Department of Agriculture). 2001b. Economic Research Service FY 2002 Annual Perfor-
mance Plan and Revised Plan for FY 2001 (July 2001). Washington, DC: Economic Research
Service, US Department of Agriculture. Available online at http://www.ers.usda.gov/AboutERS/
ersperformance_plan. pdf.
USDA (US Department of Agriculture). 2001c. National Agricultural Statistics Service FY 2002 and
Revised FY 2001 Annual Performance Plans. Washington, DC: National Agricultural Statistics
Service, US Department of Agriculture. Available online at http://www.usda.gov/nass/nassinfo/
nass-app-02-Ol.pdf.
USDA (US Department of Agriculture). 2001d. National Agricultural Statistics Service. FY 2000
Annual Program Performance Report. Washington, DC: National Agricultural Statistics Ser-
vice, US Department of Agriculture. Available online at http://www.usda.gov/ocio/ar2000/
aprpdf/arnass.pdf.
OCR for page 145
QUALITY AND IMPACT ASSURANCE IN THE REE AGENCIES
145
USDA (US Department of Agriculture). 2002a. Information Quality Guidelines. Washington, DC:
National Agricultural Statistics Service, US Department of Agriculture. Available online at http:
//www. usda.gov/nass/nassinfo/infoguide.htm.
USDA (US Department of Agriculture). 2002b. Research Position Evaluation System. Washington,
DC: Agricultural Research Service, US Department of Agriculture. Available online at http://
www. afm. ars. usda. gov/rpes/.
USDHHS (US Department of Health and Human Services). 2002. Orientation Guidelines for Boards
of Scientific Counselors. Office of the Director, National Institutes of Health. Washington, DC:
National Institutes of Health, US Department of Health and Human Services.
Representative terms from entire chapter:
statistical method
