Click for next page ( 120


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 119
6 Quality and Impact Assurance in the REE Agencies Ensuring the high quality, or excellence, of research is a key element of research management. But in mission-oriented research, such as that conducted in the US Department of Agriculture (USDA) Research, Education, and Economics (REE) mission area, impact assurance is equally important. Both quality assurance and impact assurance are necessary follow-up activities for relevance assurance, which was discussed in Chapter 4. The REE agencies have implemented several changes to strengthen quality assurance during the last decade. Measuring impacts is more difficult but has been accomplished for some dimensions. This chapter considers quality and impact assurance processes in the REE agencies, evidence regarding research impact, and how the processes can be improved to address impacts of the research opportunities outlined in Chapter 3. QUALITY ASSURANCE Research quality the degree of excellence of research compared with other work being conducted in a field rests on a foundation of scientific merit. Research quality is best evaluated by professional peers selected for their exper- ~ According to the National Academies' Committee on Science, Engineering, and Public Policy, "there are at least two aspects of qualityone absolute and one relative. The absolute aspects are related to the quality of the research plan, the methods by which it is being pursued, its role in educa- tion when conducted at a university, and the importance of its results to its sponsor, either obtained or expected. The relative aspects pertain to its leadership at the edge of an advancing field. Although the leadership aspect is generally important, the results might in some cases be of great importance to an agency albeit not at the leading edge of a field" (NRC, 1999a). 119

OCR for page 119
120 FRONTIERS IN AGRICULTURAL RESEARCH tise in the field being assessed, experience, and objective judgment. This approach is known as peer review. Peer-reviewed science includes the peer review of discoveries before publication or patenting and the evaluation of quality of a research proposal, including the creativity of the idea, the technical sound- ness and appropriateness of the experimental design, the relationship to other scientific results from the literature, the record of the scientist or the scientific team, and the likelihood of scientific advances or practical applications or impacts. Peer review of research proposals is commonly used in allocating competitive funding to ensure quality in research design. Quality assessment of research outputs can include peer review of manuscripts, discoveries, research programs, and the quality of the publication in which the research results are published. Research quality depends heavily on research inputs, including the quality of the scientists conducting the research.2 Rigorous evaluation systems for the scien- tists conducting the research and incentive structures for rewarding high-quality work and creative, independent thinking can contribute to ensuring high-quality outcomes. REE Quality-Assurance Mechanisms The committee considered the REE agencies in light of metrics of quality and mechanisms for quality assurance. Generally, the committee identified a variety of quality-review and evaluation processes that are in place for all research projects and programs in the REE system; they are summarized in Table 6-1. According to the strategic plan and performance plans of ARS (USDA, 1999, 2000a), each of roughly 1,100 research projects undergoes external merit-based peer review before new or renewed activities are begun. To ensure quality in the workforce, all ARS employees, including the scientific employees, are subject to annual performance reviews, and permanent scientists undergo a review of their progress on a 3- to 5-year cycle (discussed later in this chapter). A series of national programs reviews is designed to ensure the quality, relevance, effective- ness, and productivity of the work being done in each national program. As mandated by the 1998 Agricultural Research, Extension, and Education Reform Act (AREERA) (US Congress, 1998), CSREES works with state part- ners receiving formula funds to develop 5-year plans of work, which are reviewed 2The committee notes that some private-sector institutions organize research to yield high-quality outcomes by emphasizing the hiring of high-quality researchers. For example, at 3M, high-quality staff are hired and given an ample endowment and flexibility to work on anything that is of interest to them (Arndt, 2002). 3ARS has recently restructured how it organizes and manages its national research programs (USDA, 1999). It has aggregated its research projects into 22 national programs that are guided by multidisciplinary teams of national program leaders. The national programs focus the work of the agency on reaching the goals defined in the ARS strategic plan.

OCR for page 119
QUALITY AND IMPACT ASSURANCE IN THE REE AGENCIES TABLE 6-1 Summary of REE Quality-Assurance Mechanisms 121 Agency Mechanism Agricultural Research Service (ARS) Office of Scientific Quality Review (established in 1999) External peer review of projects before implementation Five-year-cycle review of national programs Annual reviews (project level and national-program level) Location reviews (research units) Review of quality of individual scientists annual performance reviews for all ARS employees and 3- to 5-year Research Position Evaluation System peer reviews for senior scientists Solicitation of input on quality of ARS science from peer scientists and users Cooperative State Annualreview of individual projects Research, Education, Annual review of research programs and Extension Service Peer review of research proposals for competitive programs (CSREES) Review of output from and input into special grants (earmarks) External program review of the National Research Initiative by the National Research Council (NRC, 2000) Economic Research Peer review (internal and external) of all published material Service (ERS) Rewards for productivity and high-quality performance (cash awards and promotions) Internal peer-review system for social-science positions, the Economist Position Classification System (established in 2000) External program review by the National Research Council (NRC, l999b) National Agricultural Accuracy review using historical track records that compare forecasts Statistics Service with final, market-derived numbers for production-related reports; (NASS) analysis of sampling errors and nonsampling errors Analysis of each step of data collection, processing, and estimation of statistics to evaluate the quality and accuracy of NASS reports Comparison of estimates with data sources outside the agency External technical review Source: Data provided by REE agencies, 2001. annually for quality (USDA, 1998~. In addition to quality review, these annual reviews also address accomplishments of research with respect to strategic goals and objectives, multistate activities, integrated research and extension, joint activities, and stakeholder input. CSREES is working to jointly establish and implement formal program-evaluation protocols, including expert assessments, with university and other partners and collaborators, through its Office of Plan- ning and Accountability (USDA, 2000b). Reviews of academic departments at land-grant universities are also required by CSREES and occur on a 5-year cycle.

OCR for page 119
22 FRONTIERS IN AGRICULTURAL RESEARCH CSREES-administered competitive-grants programs, such as NRI and IFAFS, subject proposals for individual awards to an external merit-review process. CSREES post-award management procedures ensure that funds are expended according to the proposed plan of work, that progress reports are received and published in the Current Research Information System (CRIS) database, and that site visits and other oversight measures are performed as necessary. CSREES also increasingly requires its awardees to present results of their work at national and international scientific symposia. In addition to individual award evaluation, the programs are reviewed for quality as a whole each time a new request for abstracts is published. Special grants awarded to a particular institution can be reviewed before funds are awarded, and outputs of the special grant research can be subjected to peer review. An external review of a CSREES-administered competitive-grants program, the National Research Initiative, was conducted by the National Research Council in 2000 (NRC, 2000~. According to its annual performance plan, strategic plan, and annual perfor- mance report (USDA, 2000e, 2000f, 2001b), ERS systematically evaluates the quality of its work and considers the factors that affect quality. As part of that process, ERS conducts peer reviews before analysis is released, and the agency's successful contributions to professional conferences and journals test the appro- priateness and rigor of the research methods in its analyses with respect to disci- plinary standards. The National Research Council provided oversight for a 2-year review of the ERS program that was completed in 1998 (NRC, l999b). In response to the report recommendations, ERS has taken a number of important actions, including the creation of an internal peer-review system. On the basis of recommendations of the Research Council study, ERS conducts broad reviews of critical aspects of its programs. It also initiated a collaborative university-ERS effort to measure the impacts of social-science research. The results of this analy- sis prove helpful to the agency in considering how to measure impacts and thereby the quality of its research. According to its strategic plan and performance reports, NASS relies heavily on customer satisfaction surveys and end-user meetings more related to rel- evance assurance than to quality assurance to assess products and services. (See Chapter 4 for a discussion of relevance assurance mechanisms.) NASS reports that it uses historical track records to compare crop estimates and forecasts pub- lished during the growing season with the end-of-season final estimates to evalu- ate the accuracy of crop estimates and forecasts. NASS also uses external data sources to check the accuracy of its estimates. The NASS strategic plan reports that internal analysis of each step of data collection, processing, and estimation of production and price statistics is conducted to ensure quality (USDA, 2000g) and that professional standards are used in all major survey activities, including sampling frame development, sample design, questionnaire design and pretest- ing, data collection, analysis of sampling and coverage errors, nonresponse analysis, imputation of missing data, weighting, and variance estimation (USDA,

OCR for page 119
QUALITY AND IMPACT ASSURANCE IN THE REE AGENCIES 123 2002a). NASS does hire consultants from academe as consultants on data- collection issues, and periodic reviews by outside panels of technical experts are used to evaluate quality and reproducibility (USDA, 2001 d, 2002a). However, it is not clear from the strategic planning and performance documents that these external reviews are conducted regularly (USDA, 2000g, 2001c, 2001d). The committee was able to draw some general conclusions about the effec- tiveness of REE quality-assurance mechanisms and their outcomes. Peer Review Internal peer reviews of individual projects are conducted annually and at project completion in all the REE agencies. Research programs receive periodic reviews, generally at 5-year intervals. The committee found that reviews tended to report intermediate outputs, publications, presentations, patents, cultivars, and breeds developed. Although such activities provide useful information, they are by their nature directed more toward quantity than quality of the effort. External peer review of project proposals and products is also occurring in the REE agencies. Intramural Funding In response to previous studies critical of the lack of peer review of research proposals in the ARS system, major changes were introduced in 1999, as required by the 1998 AREERA (US Congress, 1998~. These changes mandated periodic (usually 5-year) peer review of all research-project plans and creation of an Office of Scientific Quality Review (OSQR). Review by scientists outside ARS is in- volved in the process. Unsatisfactory research plans are not approved and must be rewritten; in some cases, the project must be terminated.4 However, as of 40ffices of the area directors manage the postpanel activities of project plans receiving a "major revision" or "not feasible" action class. Projects falling into these categories may be dealt with in three ways (frequency of management action as of August 2002 is shown in parentheses): (1) prepare a revised project plan for a re-review by the panel within 3 months (95%), (2) completely rewrite the plan and have it peer-reviewed again by a new combination of reviewers about a year later (4%), (3) terminate the project (less than 1%). Re-review by the original panel is the most common action. A complete rewrite of the project plan and fresh peer review are most often the option chosen if the reason for the poor score was the absence of key scientific expertise (usually a result of a vacant position). The original panel reviewers are asked to provide a second review in such cases. Finally, projects may be terminated if there are extenuating personnel issues or an inability to correct the problems. Successful objectives from ARS projects that have been terminated because of overall poor peer reviews were transferred to a complementary research project. The scientists involved were either reassigned or placed under other personnel actions. Although grade level is not affected by the reassignment, a scientist may lose his or her status as a lead scientist on a project that is terminated, without a change in salary. The termination of projects is usually chosen only after a second review of the proposal results in a judgment of "not feasible" or "major revision required."

OCR for page 119
24 FRONTIERS IN AGRICULTURAL RESEARCH August 2002, OSQR reports that only five of several hundred projects have been terminated; none of the terminations has resulted in the removal of a scientist. The impact of an unsatisfactory research proposal on a scientist's performance evaluation is handled case by case, depending on the nature of the problem. The program appears to have encouraged the development of improved research proposals, which does not necessarily guarantee that the quality of research in a given unit will be improved but suggests that the quality of the overall research program should increase with time. However, the low rate of project termination resulting from either proposal review or performance evalua- tion suggests limited accountability of the system. The major research costs (salary for ARS scientists and laboratory personnel) constitute fixed expenses for a unit regardless of whether specific research proposals have been approved, and the procedure does not represent competition or ranking among proposals for these funds. The national program leader for each research program category is responsible for priority-setting and quality assurance but does not have the requisite authority to move personnel or budget to various research centers to ensure those outcomes. The committee considered results of reviews of six national programs that underwent peer review from February 2000 to August 2001 (Table 6-2~. The most important evidence from Table 6-2 is that 23% of the program reviews asked for major revisions. Of the 94 review-panel members, 12 were employed by government agencies other than ARS,69 were university faculty members, 12 were industrial or private consultants, and one was employed by ARS. The makeup of the review panels demonstrates that ARS programs are subjected to outside review. External peer-review mechanisms are also in place in ERS, where all pub- lished materials are reviewed by experts for appropriateness for the category of publication. ERS's FY 2000 performance report states that published research meets peer-review standards for all five goals in every case (USDA, 2000e). ERS also has an internal peer-review process in place for research proposals. At NASS, in the currently narrowly defined focus of NASS efforts (see Chap- ter 4), the quality-assurance processes are focused on the value of reports to particular end users. Customer satisfaction surveys and end-user meetings provide little information on the relative value of different types of estimates or information. In addition to end-user feedback, NASS quality assurance would also benefit from regular peer review of survey and estimation (forecast) methods by academic and other government-agency statisticians to ensure that they reflect the newest analytic techniques. The committee considered REE's intramural research quality-assurance mechanisms against the mechanisms for quality assurance used in two other fed- eral intramural research programs those of the National Institutes of Health (NIH) and the Environmental Protection Agency (EPA). Intramural research at NIH has been subject to external scientific review since 1956. NIH's Office of

OCR for page 119
QUALITY AND IMPACT ASSURANCE IN THE REE AGENCIES TABLE 6-2 Results of February 2000-August 2001 Review of Six ARS National Programs 125 Action Needed after Review National Program (NP) Number of Projects Reviewed Revisiona Revisionb RevisionC Revisions Feasiblee No Minor Moderate Major Not Manure and Byproduct 21 1 10 7 2 1 Utilization (NP 201) Food Safety (NP 108) 62 1 21 21 18 1 Soil Resources and 33 8 14 6 5 0 Management (NP 206) Plant Biological and 36 1 13 8 13 1 Molecular Processes (NP 302) Animal Health (NP 103) 35 l ~ Water Quality (NP 201) 31 7 12 v 15 9 2 8 4 0 Total 218 19 78 65 51 5 a No revision required. No revision is required, but minor changes to project plan may be made. b Minor revision required. Project plan is basically feasible as written but requires some revision to increase quality. c Moderate revision required. Project plan is basically feasible as written but requires moderate revision of one or more objectives, perhaps involving changes in experimental approaches, to increase quality. Project plan may also need rewriting for greater clarity. ~ Major revision required. Substantial revision of one or more objectives is necessary, but project plan should then be sound and feasible. e Not feasible. Project plan has major flaws or deficiencies and cannot be simply revised to produce sound project. If project is not terminated, complete redesign and rewriting are required. Source: ARS, Office of Scientific Quality Review, 2001. Intramural Research reported to the committee that NIH has a rigorous, largely retrospective, review system in place in its intramural research programs. In contrast with the review of extramural grants, which mainly assesses the quality of proposed research, the work of all principal investigators is reviewed mainly in retrospective fashion, in which the research program is evaluated in toto for its overall goals, quality of research, and long-term objectives, based in specific criteria (Box 6-1) (USDHHS, 2002~. In the case of new investigators, more emphasis is placed on future plans. Principal investigators are either tenured or on tenure track, designations conferred only after rigorous searches, peer review, selection processes, and internal and external reviews of research programs. The

OCR for page 119
126 FRONTIERS IN AGRICULTURAL RESEARCH

OCR for page 119
QUALITY AND IMPACT ASSURANCE IN THE REE AGENCIES 127

OCR for page 119
28 FRONTIERS IN AGRICULTURAL RESEARCH success rate for achieving tenure is about 65%, so 35% of the scientists who enter a 6-year tenure-track period do not compete successfully for scientific research resources (or leave for other reasons). Independent research support is provided to all principal investigators, and their progress is evaluated at least every 4 years by groups of outside experts, constituted as boards of scientific counselors. The emphasis of the outside reviews is mainly on past accomplishments, although future plans are presented to the review teams as well. Reviewers develop written recommendations to increase, decrease, or hold constant the resources assigned to principal investigators; these recommendations are acted on by the scientific program leaders. During the past 5-year period in one of NIH's major institutes, 7% of principal investigators lost all research support, and 25% of principal investigators had resources reduced. Resources of other principal investigators were increased or held at the same levels, leaving room for expansion of new initiatives and programs. The committee also considered mechanisms for peer review of intramural research at EPA. Peer review occurs at multiple levels. For example, EPA's National Health and Environmental Effects Research Laboratory, in the Office of Research and Development, conducts reviews of each of nine divisions, review- ing research strategies and multiyear implementation plans, cross-cutting research programs, specific scientists undergoing promotion, and investigator-initiated re- search proposals. A variety of review processes are used, including ad hoc panel reviews, internal review by EPA experts, Federal Advisory Committee Act reviews, and reviews by ad hoc panels comprising a mixture of discipline-specific external experts and internal standing committees. All major scientific or techni- cal work products also undergo review by internal and external experts before their release. FINDING: The committee commends ARS and ERS for establishing peer-review processes. The ARS peer-review process assists researchers in producing higher-quality proposals, which are a necessary, but not an exclusive, component of higher-quality research. However, the ARS peer-review system appears to reward excellent research performance adequately but may not adequately exclude poor research performance, given the noncompetitive (unranked) nature of the peer-review process, the extremely low rates of project termination, and the lack of impact of poor performance in proposal peer review on personnel grade level. RECOMMENDATION 7: The REE intramural research system should strengthen quality control for poor research performance. Mechanisms used at other federal intramural research agencies, including the redirection of human or financial resources when quality is poor, could be implemented.

OCR for page 119
QUALITY AND IMPACT ASSURANCE IN THE REE AGENCIES 129 The decision to terminate a project or a position should be made after a broad review of all aspects of a research program, including research inputs and outputs. Formula Funding CSREES uses review mechanisms to ensure the quality of a diverse portfolio of research, education, and extension programs with universities and other orga- nizations. In some institutions, formula funds are built into base salary budgets of individual faculty; in others, the funds are distributed as specific research grants to individual faculty. The 1998 AREERA (US Congress, 1998) mandated mecha- nisms to ensure that each proposal undergoes merit or peer review to determine its quality before funding. Institutions eligible for formula funds are now required to document in their plans of work "a description of the merit and/or peer review process, . . . the selection of reviewers with expertise relevant to the effort, and appropriate scientific and technical standards" (USDA, 1998~. The effectiveness of these policies in achieving improved accountability, however, remains unclear. Progress in formula-funded projects is assessed by CSREES through the receipt of interim and terminal CRIS reports. Competitive Grants In the CSREES competitive-grant programs, the National Research Institute (NRI), Initiative for Future Agriculture and Food Systems (IFAFS), and the Fund for Rural America (FRA) competitive programs have used a competitive merit- based, peer-review process to ensure quality. A National Research Council review of the NRI concluded that the quality of scientific work in this program was high (NRC, 2000~. Special Grants CSREES special grants are added to the annual budget of the agency by congressional action. They are not subject to merit-based peer review and are directed to specific locations. CSREES can only ensure that special-grant pro- grams represent the best quality in the institution or program by subjecting to peer review the output of special grants. The proposal or input of special grants can also be reviewed before the awarding of funds. In some cases, CSREES does not release the funds until the institution has hired a person with the necessary expertise or partnered with another institution that has the necessary expertise. FINDING: Several recent REE research programs (including the NRI, FRA, and IFAFS) are based on competitive peer-reviewed funding mechanisms, which make important contributions to ensuring the qual- ity of the proposed research. Merit-based or peer-review mechanisms

OCR for page 119
QUALITY AND IMPACT ASSURANCE IN THE REE AGENCIES Patents 135 Patents are the products of invention and can indicate that research results are being accepted and used in the US economy. USDA received an average of 57 patents per year from 1996 to 2000 (see Chapter 5, Table 5-2~. Social Rate of Return The social rate of return on public research expenditures is the economist's preferred tool for assessing impact; it provides a bottom-line estimate in the form of a rate of return. The return can be compared with the social opportunity cost of public funds. Sometimes it is difficult, however, to obtain the time profile of net social benefits of research studies.7 Society will be best served if it invests its research resources into projects that yield a sizable positive social marginal rate of return. Societal benefits in the form of rates of return from agricultural research have been extensively reviewed (see Tables 6-5 and 6-6~. Rates of return are high but vary widely. Evenson's summary of rates of return on public aggregates agricultural research investments shows a median real social rate of return from 126 research studies of 45%; 66% were in the range of 21-80%. The research studies summarized by Evenson generally estimate the market value of improve- ments in agricultural productivity (this results in lower consumer prices or lower costs for producers over time) and then compare that value with the costs of investments in agricultural research. Alston et al. (2000) also published a formal and extensive analysis (292 studies, 1,886 rate-of-return observations), which econometrically accounts for the observed variation in the rates of return. That study showed that the mean of the measured rates of return to research (averaging over 1,144 observations) was 99.6% with a mode of 46%. Both meta-studies support the finding that investments in agricultural research have generated high and sustained returns to society. 7When individual projects are evaluated, it generally takes several years after a study is completed before the social cost-benefit calculations can be made. Public denotes a combination of state and federal, and the aggregate value is only a proxy for REE research. Assessment of the impact of research supported by USDA formula and competitive funds at universities is difficult. Individual universities decide how the formula funds are to be allocated among competing needs, including research, education, and outreach. Federal funds from REE are most often combined with other funds (from state, private, and other federal agencies), and attribution of the individual contributions is nearly impossible with today's accounting procedures. Returns from aggregate values seem most relevant here because they include successful and failed projects and provide a better picture of the whole enterprise than rates of return for particular studies. In the case of the SAKS, in the period 1927-1980, when these rates of return were calculated, most funding came from state or federal sources, not from the private sector (see footnote 11).

OCR for page 119
136 FRONTIERS IN AGRICULTURAL RESEARCH TABLE 6-5 Internal Rates of Return from US Public-Sector Agricultural Research Study Method Period Internal Rate of Return,a % Region Peterson and Fitzharris (1977) Project evaluation Peterson and Fitzharris (1977) Project evaluation Peterson and Fitzharris (1977) Project evaluation Peterson and Fitzharris (1977) Project evaluation Norton and Paczkowski (1993) Project evaluation Norton and Paczkowski (1993) Project evaluation Griliches (1964) Statistical method Latimer (1964) Statistical method Evenson (1968) Statistical method Cline (1975) Statistical method Bredahl and Peterson (1976) Statistical method Bredahl and Peterson (1976) Statistical method Bredahl and Peterson (1976) Statistical method Bredahl and Peterson (1976) Statistical method Lu et al. (1979) Statistical method Lu et al. (1979) Statistical method Evenson (1979) Statistical method Evenson (1979) Statistical method Evenson (1979) Statistical method Evenson (1979) Statistical method Evenson (1979) Statistical method Knutson and Tweeten (1979) Statistical method White et al. (1978) Statistical method Davis (1979) Statistical method Davis and Peterson (1981) Statistical method Davis and Peterson (1981) Statistical method Davis and Peterson (1981) Statistical method Davis and Peterson (1981) Statistical method Welch and Evenson (1989) Statistical method White and Havlicek (1982) Statistical method Braha and Tweeten (1986) Statistical method Evenson (1989) Statistical method Alston et al. (1998a) Statistical method Chavas and Cox (1992) Statistical method Makki et al. (1996) Statistical method Makki and Tweeten (1993) Statistical method Oehmke (1996) Statistical method Oehmke (1996) Statistical method Yee (1992) Statistical method Norton et al. (1992) Statistical method 1937-1942 1947-1952 1957-1962 1957-1972 1949-1979 1949-1989 1949-1959 1949-1959 1949-1959 1939-1948 1937-1942 1947-1957 1957-1962 1967-1972 1938-1972 1939-1972 1868-1926 1927-1950 1948-1971 1948-1971 1948-1971 1949-1972 1929-1977 1949-1959 1949 1954 1959 1964, 1969, 1974 1969 1943-1977 1959-1982 1950-1982 1930-1990 1930-1990 Pre-1930 1930-1990 1931-1985 1987 50 51 49 34 58 58 25-40 n.s.b 47 41-50 56 51 49 34 24-31 23-30 65 95 130 93 95 28-47 28-37 66-100 100 79 66 37 55 7-36 47 43 17-31 28 27 93 Negative 11.6 49-58 30 VA VA South North West aThe internal rate of return is a discount rate at which the present value of a series of investments is equal to the present value of the returns on those investments. bn.s.= not significant. Source: Adapted from Evenson (2001). Economic Impacts of Agricultural Research and Extension in Agricultural Economics Volume la Agricultural Production, B.L. Gardner and G.C. Rausser, eds. Amsterdam: Elsevier.

OCR for page 119
QUALITY AND IMPACT ASSURANCE IN THE REE AGENCIES TABLE 6-6 Internal Rates of Return from US Extension 137 Internal Rate of Study Period Extension Variable Return,a % Huffman (1974)b 1959-1974 Extension staff/farm 16 Huffman (1976)b 1964 Staff days/farm 110 Evenson (1979)b 1971 Expenditure/region 100+ Huffman (1981)b 1979 Extension days/county 110 Evenson (1994)b 1950-1972 Expenditure/state Crops 101 Evenson (1994)b 1950-1972 Expenditure/state Livestock 89 Evenson (1994)b 1950-1972 Expenditure/state All 82 Huffman and Evenson (1993)C 1950-1982 Crops 40.1 Huffman and Evenson (1993)C 1950-1982 Livestock (negative) Huffman and Evenson (1993)C 1950-1982 All 20.1 aThe internal rate of return is a discount rate at which the present value of a series of investments is equal to the present value of the returns on those investments. bEvenson, R. 2001. Economic impacts of agricultural research and extension. In Agricultural Eco- nomics Volume la. Agricultural Production. B.L. Gardner and G.C. Rausser, eds. Amsterdam: Elsevier. p. 593. CHuffman, W.E., and R.E. Evenson. 1993. Science for Agriculture: A Long-Term Perspective. Ames, IA: Iowa State University Press, p. 245. Other general observations over the 1950-1982 period include a much higher rate of return from preinvention or pretechnology research than from applied research (Evenson, 2001), a higher rate of return from applied crop research than from applied livestock research (Huffman and Evenson, 1993), and a lower rate of return as the share of public agricultural research funding by federal contracts, grants, and cooperative agreements increased (Huffman and Just, 1994~. High rates of return are consistent with growth in total factor productivity.9 Jorgenson and Stiroh (2000) show that the US agricultural sector ranks third among 37 US industries in aggregate total factor productivity growth over 1958- 1996 at 1.2%. New total factor productivity estimates by ERS for US agriculture show an average annual rate of 2% for 1950-1998. Evenson (2001) describes how a continuous investment in public agricultural research of 1% of output per year will contribute 0.76% per year to total factor productivity growth when the real rate of return is 40%. 9A total-productivity or multifactor productivity index measures the quantity of output produced compared with the quantity or cost of all the measurable inputs used to produce it (Ahearn et al., 1998). Growth in productivity indicates the growth in output unaccounted for by changes in mea- sured inputs and is typically ascribed to investments in R&D, education, infrastructure, and econo- mies arising from increasing the scale of production.

OCR for page 119
138 FRONTIERS IN AGRICULTURAL RESEARCH The high rates of return from public agricultural research summarized by Evenson (2001) pertain to investments made in publici agricultural research from 1927 to 1980. Alston et al. (2000), using econometric evidence after accounting for factors that cause the reported rates of return to vary among studies, estab- lished that the rate of return over the last 2 decades has not declined. It is important to discuss the inherent limitations of using social rate of return as an impact indicator. For example, the estimates of rate of return do not account for possible off-farm environmental costs or benefits (Jorgenson and Stiroh, 2000~. In addition, rate of return does not account for the full economic costs and benefits for farms of different sizes and for agriculture-dependent communities (for example, new technologies may enhance productivity of larger farms rela- tively but contribute to the decline of smaller farms and the rural communities that were once sustained by these farms) (Swanson, 1988; US Congress OTA, 1986~. FINDING: The social rate of return on past public agricultural research investments in the period 1950-1982 has been very high. The rate of return over the last 2 decades has not declined. Social rate of return has limitations in accounting for full environmental and social costs of re- search. Environmental, Economic, and Health Outcomes Aside from the long-run benefits of productivity enhancement measured through social rate of return, documented quantitative examples of the impact of REE research on other outcomes are scarce. Box 6-3 provides several illustrative examples of successful research that provided benefits to the environment, health, or safety that are difficult to quantify on a national scale. As the examples show, successful impact of specific research projects may be obvious, but difficult to summarize in comparable measures. That is particularly true for environmental benefits, human health benefits, or potential social benefits. Although the research frontiers identified in this report will result in such benefits, they may be difficult to quantify or compare among research goals and projects. teethe private-sector share of the total support for SAESs was 7% in 1960, 9.2% in 1980, and 13.2% in 1990. Before 1960, disaggregated data for the private-sector contribution are not available, but we know that the sum of the private-industry contribution and other federal-government (non-USDA) resources was not greater than 14.8% in 1900, 29.8% in 1930, and 22.6% in 1940. Thus, USDA appropriations and state-government appropriations predominated as the source of support of the SAESs (85% in 1900, 70.2% in 1920, 77.4% in 1940, 79.8% in 1960, and 72.5% in 1980) during the period in which these rates of return were calculated (Huffman and Evenson, 1993).

OCR for page 119
QUALITY AND IMPACT ASSURANCE IN THE REE AGENCIES 139

OCR for page 119
140 FRONTIERS IN AGRICULTURAL RESEARCH Monitoring and Communicating Impact Mechanisms for tracking research investments and impacts are important both for informing internal decision-making on future research investments and for communicating with and improving accountability to the public. The com- mittee considered a variety of mechanisms used by REE to track and communi- cate its performance and impact. These are considered in detail in Appendix G. This report has identified a need to broaden the REE agency focus on needs of and impacts on new, nontraditional stakeholders and to be more strategic in

OCR for page 119
QUALITY AND IMPACT ASSURANCE IN THE REE AGENCIES 141 setting priorities for investments to meet national goals (Chapters 1 and 4~. To achieve these goals, REE will need to improve and expand existing performance- based systems for monitoring success. More-effective and more user-friendly tracking systems will contribute to improved self-evaluation and reporting of progress to groups outside REE. Electronic media are an increasingly critical and strategic means for communicating impacts and research results to the general public and should be a focal point for development and expansion. ERS's recent redesign of its Web site to be more accessible and understandable to the public- through provision of links to nontechnical summaries, technical abstracts, data, and publications is an excellent model for how positive change could occur in this regard throughout the REE mission area. RECOMMENDATION 8: REE agencies should develop and adopt ways of measuring the national, long-term impacts of their research on the environment, human health, and communities. The tools should include measures and indicators that are influenced by agricultural research or that can be attributed to research outcomes, including how research sup- ports the needs of action agencies. REE should strive to achieve greater transparency in communicating these impacts through timely electronic publishing of peer-reviewed results and through greater efforts to inter- pret these results for a general audience. The committee envisions that monitoring capability and development of indicators would occur in parallel at two levels. First, monitoring capability could be developed to show how REE research has changed in focus, relevance, quality, leadership, and accountability (NRC, 1999a). For example, REE could track progress toward meeting national goals by requiring that each major research program or initiative establish performance objectives and measures for evaluat- ing progress toward meeting national objectives, on the basis of some assessment of adoption and/or implementation of research findings in practical applications and possibly of how such adoption led to beneficial changes. In addition, REE could track performance by keeping a comprehensive account of where research funded or conducted by REE has made a critical contribution to advancing under- standing, policies, and practices in each of the research frontiers recommended by this study. A second level of monitoring capability could be developed to show how food, agricultural, natural, and human systems are changing. Because changes in such indicators cannot be directly attributed to research, this class of indicators cannot provide a direct measure of research performance, but it can be used to help target future research directions. Such indicators might include nutritional indicators, such as the healthy eating index measuring overall nutritional quality of the American diet (Kennedy et al., 1999), and ecologic indicators, including nutrient runoff and soil organic matter (NRC, 2000~. Indicators should be selected after a set of defining criteria has been established, which might include a well-

OCR for page 119
42 FRONTIERS IN AGRICULTURAL RESEARCH understood conceptual basis, reliability, applicability on clear temporal and spatial scales, accuracy, sensitivity, precision, robustness, skill and data requirements, data-quality requirements, archiving capability, international compatibility, costs, benefits, and cost effectiveness (NRC, 20001. As discussed in Chapter 3, system- atic research on the environmental, social, and community impacts of REE research would inform this process. SUMMARY This chapter has considered quality-assurance and impact-assurance pro- cesses and their outcomes in the REE agencies and has provided recommenda- tions for improving the effectiveness of these processes. The use of peer review as a quality-assurance mechanism for research inputs and outputs was discussed with regard to intramural research, formula-funded research, special grants, and competitive grants. REE staff-performance evaluation systems and reward and incentive programs were also considered as mechanisms for ensuring the delivery of high-quality science. In general, REE has strong and evolving quality- assurance mechanisms in place, and REE scientists produce research that is of high quality. However, human or financial resources should be redirected when research is of poor quality in intramural research. Pnmary, intermediate, and longer-term impacts of agricultural research were descnbed. Much progress has been made in documenting research impact in the traditional dimensions associated with improved productivity. Because in the future more REE research will be directed at providing new kinds of benefits, monitoring of research impact will require new outcome measures. The impor- tance of mon~tonng, measunng, and communicating research investments and their impacts was discussed, and changes in monitoring capability, in develop- ment of indicators, and in communication of results were recommended. REFERENCES Ahearn, M., J. Yee, E. Ball, and R. Nehring. 1998. Agricultural Productivity in the United States. Agricultural Information Bulletin, No. 740. Washington, DC: Economic Research Service, US Department of Agriculture. Available online at http://www.ers.usda.gov/publications/aib740/. Alston, J.M., M.C. Marra, P.G. Pardey, and T.J. Wyatt.2000. Research returns redux: A meta-analysis of the returns to agricultural R&D. Australian Journal of Agricultural and Resource Economics 44(2): 185-215. Amerman, R.C., G. Larson, and M. O'Neill. 2001. USDA Water Quality Initiative. Presentation to National Research Council Committee on Opportunities in Agriculture, Public Workshop, May 22-23. Washington, DC. Araji, A.A., F.C. White, and J.F. Guenthner. 1996. Returns from Potato Research: Accounting for State and Regional Effects. Agricultural Experiment Station, Research Bulletin 152. Moscow, ID: University of Idaho. Arndt, M. 2002. 3M: A lab for growth? Business Week, Jan. 21.

OCR for page 119
QUALITY AND IMPACT ASSURANCE IN THE REE AGENCIES 143 Bostom, A.G., I.H. Rosenberg, H. Silbershatz, P.F. Jacques, J. Selhub, R.B. D'Agostino, P.W.F. Wilson, and P.A. Wolf. 1999a. Nonfasting plasma total homocysteine levels and stroke inci- dence in elderly persons: The Framingham study. Annals of Internal Medicine 131:352-355. Bostom, A.G., H. Silbershatz, I.H. Rosenberg, J. Selhub, R.B. D'Agostino, P.A. Wolf, P.F. Jacques, and P.W.F. Wilson. l999b. Nonfasting plasma total homocysteine levels and all-cause and cardiovascular disease mortality in elderly Framingham men and women. Archives of Internal Medicine 159:1077-1080. Evenson, R.E. 2001. Economic Impacts of Agricultural Research and Extension. Pp. 575-628 in Handbook of Agricultural Economics, Vol. 1: Agricultural Production, B.L. Gardner and G.C. Rausser, eds. Amsterdam: Elsevier. Frisvold, G., G.K. Agnew, and P. Baker. 2002. Effects of Insect Growth Regulators on Insecticide Use and Costs in Arizona Cotton. Proceedings of the Beltwide Cotton Conferences 1. Memphis, TN: National Cotton Council. Huffman, W.E., and R.E. Evenson. 1993. Science for Agriculture: A Long-Term Perspective. Ames, IA: Iowa State University Press. Huffman, W.E., and R.E. Just. 1994. Funding, structure, and management of public agricultural re- search in the United States. American Journal of Agricultural Economics 76(November):744- 759. ISI (Institute for Scientific Information) 2001. ScienceWatch: Trends and Performance in Basic Re- search. July/August. ISI Essential Science Indicators, 1991-2001. Philadelphia, PA: Institute for Scientific Information. Jorgenson, D.W., and K.J. Stiroh. 2000. US economic growth at the industry level. American Eco- nomic Review 90(May):161-167. Kennedy, E., S.A. Bowman, M. Lino, S.A. Gerrior, and P.P. Basiotis. 1999. Diet quality of Ameri- cans: Health eating index. Chapter 5 in America's Eating Habits: Changes and Consequences, E. Frazao, ed. Agriculture Information Bulletin No. 750. May. Washington, DC: Economic Research Service, US Department of Agriculture. NAS (National Academy of Sciences). 2001. Membership Directory. July 2001. National Academy of Sciences of the United States of America. NRC (National Research Council). 1999a. Evaluating Federal Research Programs: Research and the Government Performance and Results Act. Washington, DC: National Academy Press. NRC (National Research Council). l999b. Sowing the Seeds of Change: Informing Public Policy in the Economic Research Service. Washington, DC: National Academy Press. NRC (National Research Council). 2000. National Research Initiative: A Vital Competitive Grants Program in Food, Fiber, and Natural-Resources Research. Washington, DC: National Academy Press. Selhub, J., P.F. Jacques, A.G. Bostom, R.B. D'Agostino, P.W.F. Wilson, A.J. Belanger, D.H. O'Leary, P.A. Wolf, E.J. Schaefer, and I.H. Rosenberg. 1995. Association between plasma homocysteine concentrations and extracranial carotid-artery stenosis. New England Journal of Medicine 332:286-291. Selhub, J., P.F. Jacques, P.W.F. Wilson, D. Rush, and I.H. Rosenberg. 1998. Vitamin status and intake as primary determinants of homocysteinemia in an elderly population. Journal of the American Medical Association 270(20):2693-2698. Seshadri, S., A. Beiser, J. Selhub, P.F. Jacques, I.H. Rosenberg, R.B. D'Agostino, P.W.F. Wilson, and P.A. Wolf. 2002. Plasma homocysteine as a risk factor for dementia and Alzheimer's disease. New England Journal of Medicine 346(7):76-73. Swanson, L. 1988. Agriculture and Community Change in the US: The Congressional Research Reports. Boulder, CO: Westview Press. US Congress. 1998. P.L. (Public Law) 105-185. Agricultural Research, Extension, and Education Reform Act of 1998.

OCR for page 119
144 FRONTIERS IN AGRICULTURAL RESEARCH US Congress, OTA (Office of Technology Assessment). 1986. Technology, Public Policy, and the Changing Structure of American Agriculture. OTA F-285. Washington, DC: US Government Printing Office. USDA (US Department of Agriculture).1992. Subduing the Screwworm. Agricultural Research. July. Washington, DC: Agricultural Research Service, US Department of Agriculture. USDA (US Department of Agriculture). 1998. Guidelines for Peer and Merit Reviews. Washington, DC: Cooperative State Research, Education, and Extension Service, US Department of Agri- culture. USDA (US Department of Agriculture). 1999. Agricultural Research Service Strategic Plan: Working Document 1997-2002. Washington, DC: Agricultural Research Service, US Department of Agriculture. Available online at http://www.nps.ars.usda.gov/mgmt/stratpln/1999/background. cam. USDA (US Department of Agriculture). 2000a. Agricultural Research Service FY 2000 and 2001 Annual Performance Plans. Washington, DC: Agricultural Research Service, US Department of Agriculture. USDA (US Department of Agriculture). 2000b. Cooperative State Research, Education, and Exten- sion Service FY 2000 and 2001 Annual Performance Plan. Washington, DC: Cooperative State Research, Education, and Extension Service, US Department of Agriculture. USDA (US Department of Agriculture). 2000c. Current Research Information System (CRIS) Fund- ing Summaries, Table A, FY 2000. Washington, DC: US Department of Agriculture. Available online at http://cristel. csrees. usda. gov/star/OOtablea. pdf. USDA (US Department of Agriculture). 2000d. Economist Position Classification System. Decem- ber 2000. Washington, DC: Economic Research Service, US Department of Agriculture. USDA (US Department of Agriculture). 2000e. Economic Research Service FY 2000 Annual Perfor- mance Report. Washington, DC: Economic Research Service, US Department of Agriculture. Available online at http://www. ers. usda. gov/AboutERS/ersannualperformance. pdf. USDA (US Department of Agriculture). 2000f. Economic Research Service Strategic Plan, 2000- 2005. October 11. Washington, DC: Economic Research Service, US Department of Agriculture. Available online at http://www. ers. usda. gov/AboutERS/ersstrategicplan. pdf. USDA (US Department of Agriculture). 2000g. GPRA Strategic Plan. Washington, DC: National Agricultural Statistics Service, US Department of Agriculture. Available online at http:// www. usda.gov/nass/nassinfo/strat-2005.pdf. USDA (US Department of Agriculture). 2000h. Peer Review of ARS Research Project Plans. Office of Scientific Quality Review, Agricultural Research Service. November 22. Washington, DC: Agricultural Research Service, US Department of Agriculture. Available online at http:// www. ars. usda. gov/osqr/OAManual . pdf. USDA (US Department of Agriculture). 2001a. Data submitted to the National Research Council Committee on Opportunities in Agriculture. Washington, DC: Economic Research Service, US Department of Agriculture. USDA (US Department of Agriculture). 2001b. Economic Research Service FY 2002 Annual Perfor- mance Plan and Revised Plan for FY 2001 (July 2001). Washington, DC: Economic Research Service, US Department of Agriculture. Available online at http://www.ers.usda.gov/AboutERS/ ersperformance_plan. pdf. USDA (US Department of Agriculture). 2001c. National Agricultural Statistics Service FY 2002 and Revised FY 2001 Annual Performance Plans. Washington, DC: National Agricultural Statistics Service, US Department of Agriculture. Available online at http://www.usda.gov/nass/nassinfo/ nass-app-02-Ol.pdf. USDA (US Department of Agriculture). 2001d. National Agricultural Statistics Service. FY 2000 Annual Program Performance Report. Washington, DC: National Agricultural Statistics Ser- vice, US Department of Agriculture. Available online at http://www.usda.gov/ocio/ar2000/ aprpdf/arnass.pdf.

OCR for page 119
QUALITY AND IMPACT ASSURANCE IN THE REE AGENCIES 145 USDA (US Department of Agriculture). 2002a. Information Quality Guidelines. Washington, DC: National Agricultural Statistics Service, US Department of Agriculture. Available online at http: //www. usda.gov/nass/nassinfo/infoguide.htm. USDA (US Department of Agriculture). 2002b. Research Position Evaluation System. Washington, DC: Agricultural Research Service, US Department of Agriculture. Available online at http:// www. afm. ars. usda. gov/rpes/. USDHHS (US Department of Health and Human Services). 2002. Orientation Guidelines for Boards of Scientific Counselors. Office of the Director, National Institutes of Health. Washington, DC: National Institutes of Health, US Department of Health and Human Services.