6
Administration of Research, Information, and Policy Analysis

The management and organization of research, information, and policy analysis in the Economic Research Service (ERS)—or any policy support research agency—should be designed to obtain the best possible outcome in evaluating the program of the agency. This chapter endeavors to carry through this exercise in its broadest outline. It begins by outlining the services that ERS is called on to perform to carry out its mission and the way these services and their clientele have changed rapidly in recent years. At the same time, the universe of potential suppliers of the services ERS provides has greatly expanded, and several alternatives to a strictly intramural research and information program have emerged. The chapter then goes on to examine what are likely to be important attributes of most of the research, information, and policy analysis services provided by ERS and the prospects for favorable evaluation in these dimensions under different models of administration. It applies the principles of program evaluation, developed in the previous chapter, to the problem of delivering research, information, and analysis in support of economic policy making in the U.S. Department of Agriculture (USDA).

This chapter emphasizes how research, information, and policy analysis should be administered in ERS, leaving open questions of who should be doing what. How ERS should be organized, and where it best fits within USDA, are taken up in Chapter 7.

Changing Markets for Research and Information

The substantive mission of ERS derives from the economic policy responsibilities of USDA. This has been true since the organization of its predecessor



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 116
6 Administration of Research, Information, and Policy Analysis The management and organization of research, information, and policy analysis in the Economic Research Service (ERS)—or any policy support research agency—should be designed to obtain the best possible outcome in evaluating the program of the agency. This chapter endeavors to carry through this exercise in its broadest outline. It begins by outlining the services that ERS is called on to perform to carry out its mission and the way these services and their clientele have changed rapidly in recent years. At the same time, the universe of potential suppliers of the services ERS provides has greatly expanded, and several alternatives to a strictly intramural research and information program have emerged. The chapter then goes on to examine what are likely to be important attributes of most of the research, information, and policy analysis services provided by ERS and the prospects for favorable evaluation in these dimensions under different models of administration. It applies the principles of program evaluation, developed in the previous chapter, to the problem of delivering research, information, and analysis in support of economic policy making in the U.S. Department of Agriculture (USDA). This chapter emphasizes how research, information, and policy analysis should be administered in ERS, leaving open questions of who should be doing what. How ERS should be organized, and where it best fits within USDA, are taken up in Chapter 7. Changing Markets for Research and Information The substantive mission of ERS derives from the economic policy responsibilities of USDA. This has been true since the organization of its predecessor

OCR for page 116
agency, the Bureau of Agricultural Economics (BAE), in 1922. In the intervening years, USDA's policy mandate has become increasingly diverse and complex, and in consequence so have the research and information programs in ERS. Today, there are many institutions and individuals providing research and information similar to that provided by ERS, and the organization of these activities takes a variety of forms. Increasingly Diverse Responsibilities and Providers At its inception, the BAE concentrated almost entirely on production agriculture, with a dominant focus on understanding farm firms and the farm sector as a subset of the economy. By the time of its demise in 1953, local and regional microeconomic issues related to farm income and rural welfare had been added. Marketing research, labor economics, and welfare issues were added in the 1950s, and the analysis of foreign agricultural markets, environmental quality, nutrition, and rural economic development in the 1960s. During the 1970s and 1980s, ERS took on additional tasks in support of trade negotiations. Despite the recent move away from central control and toward freer markets in agriculture, multiple policy goals continue to generate issues that demand new information and economic analysis in support of decision making. These demands are well reflected in the list of specific ERS functions and accomplishments presented in Box 5.1. In recent decades, the number of ERS personnel has declined, having reached its peak in the 1970s and early 1980s. At that time, the production of indicators alone accounted for about a third of the ERS budget. Often two or three professionals, and in the case of major commodities many more, were assigned to the same market. The responsibilities of these individuals included research, economic modeling, market organization, data base design, and the production of secondary data. Today one individual is assigned to each market, with responsibilities limited to indicators, Situation and Outlook reports, and occasional help with staff analyses. An increasingly large share of ERS staff is responsible for research and staff analyses on the wide variety of topics documented in Box 5.1. At the time of the inception the BAE, some agricultural economics departments had been in existence since the early part of the century, but many were only beginning to be established in the land grant colleges and universities. A few early institutions produced most of the BAE economists; it was about the time of World War II or afterward before the total number of agricultural economists on college and university faculties exceeded the number of agricultural economists in the BAE. Today, several federal agencies, including the National Science Foundation and the National Institutes of Health, have extramural research programs in economics. Government-sponsored economic research takes place in a wide variety of institutions, including universities, government agencies, federally funded research and development centers, wholly funded academic research institutes (for example, the Food and Agricultural Policy Research Insti-

OCR for page 116
tute centered at the University of Missouri and Iowa State University, and the Institute for Research on Poverty at the University of Chicago, Northwestern University, and the University of Wisconsin), private foundations, and for-profit private consulting firms. Seventy-five years ago, the BAE was the dominant employer of agricultural economists, and the intramural agency research program was the only model for policy support research. Today, the ERS research program remains almost entirely intramural, but it is one employer of agricultural economists among many, and there are several alternatives to the intramural agency research model. Whereas the BAE was, in its early days, the dominant employer of agricultural economists and virtually the sole supplier of economic information and research in agriculture, today ERS must compete with others in hiring professionals and in the provision of many kinds of research and information—and do so across a much wider range of professions and problems. Some Alternative Models for Economic Policy Support Other models exist in government, in the United States and elsewhere, for organizing research in support of policy analysis. The following three examples include a special operating agency within government a nonprofit research institute outside government, and an internal research management agency relying primarily on outside expertise. Australian Bureau of Agricultural and Resource Economics Over the past decade, Australia has changed the structure of its government-sponsored agricultural economics research. The Australian Bureau of Agricultural and Resource Economics (ABARE) was formed in 1987 when two much older units, the Department of Primary Industries (DPI) and Department of Resources and Energy (DRE), were combined. As part of this merger, the Bureau of Agricultural Economics from DPI was combined with the Bureau of Resource Economics from DRE to form ABARE. ABARE's activities are similar to those of ERS. It provides a combination of intermediate and long-term research, short-term analysis for policy decisions, and data collection and development of indicators. Reporting lines for ABARE differ from those of ERS. ABARE is professionally independent, but its executive director reports directly to the minister of primary industries and the minister of resources, who have broad mandates addressed by ABARE research. In addition to reports and other publications that are distributed widely, ABARE also provides briefings to the ministers on issues addressed by its research program. About 60 percent of the budget of ABARE is provided as a direct appropriation from the Australian parliament. As a professionally independent entity, its research reports are not subject to clearance by any Australian government

OCR for page 116
agency. The balance of its budget comes from a combination of private-sector, regional, and local government units and other federal ministries. Its research priorities are driven by funding sources. ABARE is gradually breaking the ties between its staff salaries and those of the Australian government. ABARE research reports are well regarded and generally meet high professional standards. Its annual reports document several score of research programs, and it tracks performance indicators for each of them. However, its research independence from the economic interest of some of its private sponsors has recently been questioned, particularly its reports on global climate change, which were funded by Australian energy interests. Some of ABARE's analyses of trade policy have been criticized as too simplistic. Although ABARE provide economic policy analysis for agriculture, its operating environment differs sharply from that of ERS. In particular, the Australian economy depends heavily on exports of agricultural commodities and natural resources. Australian policy is intensely free trade, with weaker competing interests for this policy than is the case in the United States. Furthermore, since the Australian government is parliamentary in form, ABARE is not subject to different policy positions taken in the executive and legislative branches. Food and Agricultural Policy Research Institute The Food and Agricultural Policy Research Institute (FAPRI) at the University of Missouri and Iowa State University is a nonprofit research institute established in 1984. It is affiliated with five other universities as well, drawing on a broad range of expertise from these institutions to conduct a research program that centers on the production and consumption of agricultural products, product prices, farm income, financial risk and risk management, and foreign agricultural trade and policies. (All of these research topics are addressed by ERS as well.) FAPRI receives a direct annual appropriation from Congress and some state legislatures, and it is free to raise its own funds from the private sector, foundations, and foreign governments. It is not considered a federal agency and its staff are not constrained by Office of Personnel Management (OPM) rules. Much of FAPRI's work is based on its commodity and trade econometric models, together with its industry expertise, in which it clearly has a long-term interest. FAPRI also generates secondary data—both indicators and reports—similar to those provided by ERS. It responds to congressional inquiries with staff analyses in much the same way that ERS reports to the secretary of agriculture. The development of a long-term, independent program of research at FAPRI is hampered by the year-to-year nature of its core congressional funding. FAPRI has been able to devote resources to the long-term refinement of the models it uses in policy analysis. In particular, FAPRI's research, other than model development, has tended to be in response to policy questions well along in the decision process.

OCR for page 116
Office of Program Analysis and Evaluation The office of Program Analysis and Evaluation (PAE) in the U.S. Department of Defense (DOD) has a broad mandate for research and analysis related to the DOD budget. Some of the topics it has addressed include health care for military and dependent personnel, schooling of dependent children, state-by-state impacts of DOD budget changes, and development and maintenance of a model for retirement of military personnel. Professional staff in PAE prepare staff analyses for the DOD secretary, frame questions for intermediate and long-term research, and write and administer research contracts and requests for proposals. There is almost no intramural research. PAE contract primarily with federally funded research and development centers (FFRDCs), but also with private-sector research firms. It has long-term relationships with several FFRDCs, and these organizations have in turn made long-term investments in models that we used to support PAE staff analyses. Although the primary relationship with these external vendors involves the development and completion of multiyear research contracts, FFRDCs are also able to respond to occasional specific research and modeling requests within one month. In addition, they have been used to conduct highly classified research. Involvement in political decision making is entirely the responsibility of internal PAE staff. The FFRDCs utilized by PAE have developed long-term relationships with several federal departments, including Labor and Health and Human Services as well as DOD. Relative to federal agencies and universities, FFRDCs are able to marshal concentrations of very talented professionals for specific assignments, in large part because they are not subject to Office of Personnel Management rules or academic tenure agreements. At the same time, their volume of research and long-term relationships enable them to make multiyear commitments to their core professional staff. The critical component of the PAE model is its small, internal staff of professionals. These individuals must understand the practical nature of policy questions and frame the relevant questions for economic research. They must also develop and nurture long-term relationships with a variety of vendors, with due attention to incentives for their contractors to perform research that is relevant and meets high standards. Critical Attributes of ERS Services The programs and staff of any agency should be managed for the best possible outcome. If there are well-articulated standards for evaluation, then it is possible to work prospectively from these standards to the appropriate administration of research, information, and policy analysis. Of the four dimensions of evaluation described in Chapter 5, the most important one for this process is the

OCR for page 116
attributes of the services that ERS provides. Although specifics of these attributes will vary with the service provided, there are four that drive the agency-wide administration of programs: (1) timeliness—delivering research, information, and staff analysis when it is most effective required; (2) relevance—providing research and information that assists decision making; (3) quality—meeting the analytical standards of the relevant scientific disciplines and applied professions; and (4) credibility—ensuring that research is and is perceived to be dispassionate and independent of political influence. None of these attributes stands alone. All are likely to interact in clients' evaluation of ERS services and therefore in the ultimate success of ERS as a research and information agency in support of public economic policy. Timeliness Although timeliness is an important attribute in staff analyses, information, and research, the nature of its importance is quite different in each of these three functions. Staff analyses bring information to bear on very specific policy questions as they arise in real time. ERS, like all agencies in support of economic policy, stands ready to provide staff analyses on demand as they are required. These analyses may be requested anywhere from a day to several weeks before the chief economist or the secretary of agriculture testifies before a congressional committee, before the secretary makes a speaking tour, or immediately following a cabinet meeting. In some cases, the agency may literally need to provide information on a moment's, notice—say, to answer the secretary's questions about a written brief as the secretary rides to the airport. Deadlines for staff analyses are typically clear, and it is very unusual for ERS or my similar agency to miss a deadline. The quality, relevance, and credibility of the information provided are the dimensions on which these analyses are more likely to vary. Information and secondary data are provided by ERS on a regular schedule, The development and publication of indicators and the publication of Situation and Outlook reports, are all subject to known deadlines. Determining the frequency with which this soft of information should be provided, as well as the level of detail, is an important management concern for ERS. The timeliness of research is an extremely important attribute in any agency in support of public economic policy. Policy decisions are made on a daily basis. The evolution of critical policy questions is never perfectly predictable and often difficult to discern. Regardless of how predictable or unpredictable the policy process is, the specific decisions that actually constitute policy are often made under great time constraints. When those in policy support positions, for example agency head, are asked to provide information pertinent to these decisions, there is no time to synthesize pertinent research, much less carry out the research itself.

OCR for page 116
Since research in support of policy questions cannot be conducted on a just-in-time basis, those providing research in support of Public economic policy must do their best to anticipate these questions and direct a program of research that is most likely to provide the best support when, questions arise. When a research director can anticipate the timing and content of policy questions with accuracy, the course of action is clear. Ideally, the agency can then respond to the question, at the time it is posed, with a dispassionate report meeting high analytical standards prepared before the heat of debate. In the more common case of highly imperfect foresight, those guiding research most seek to position research projects or syntheses of research and data development so that, taken together, they provide an up-to-date knowledge base in the important subject matters an which the agency's policy responsibility and mission depend. Relevance The Programs of a research and information agency in support of public economic policy must be driven by the need, of its clients. In the case of ERS, the chief client is the secretary of agriculture. An effective research and information program provides the infrastructure that enables staff analyses to meet high standards for analytical expertise and accuracy in directly answering the question posed. Achieving and maintaining relevance in ERS or any similar agency is the product of a series of accomplishments demanding intellectual and managerial skills of high order. Since research programs and the development of good secondary data to require significant lead times, it is essential for those managing research and information programs to be in constant contact with public and private decision makers. This can be done directly, through personal interaction and reading, and indirectly, by following the critical press. Advisory panels of outside experts and regular contact with foreign counterparts and industry leaders can also play a role in keeping fully abreast of emerging and future policy issues. Determining the appropriate mix of programs in the light of evolving policy questions, budgetary constraints, and available skills is fundamental to maintaining the relevance of research and information programs. Feedback from staff analysts to managers can be an important ingredient in assessing and maintaining the relevance of programs—in determining, for example, information and research that would regularly improve responses if it were available, and in establishing that information once widely demanded and still Produced is no longer used in internal staff work. Having identified emerging or future policy issues, identifying the research required to inform the likely policy decisions requires keen analytical skills on the part of managers. Framing the appropriate question and recognizing the institutional, political, and other dimensions of the policy context in the economic analysis we critical to relevant research. Not only can mistakes at this stage

OCR for page 116
produce research that is irrelevant, but also the irrelevance of the research may further obscure the real issues when the time for a decision arrives. It may also lead to substantial investments in new research, when in fact answers to the appropriate question may have already existed in the literature and only a synthesis of existing results was called for. With reference to the models for research and information agencies in support of public economic policy discussed above, this task is regarded as being of singular importance by the managers of ABARE and the PAE office of DOD. Quality The research and information program of an agency in support of public economic policy can be judged by disciplinary and professional standards. In the case of ERS, these will often be the standards of the discipline of economics for research and that of statistics for providing secondary data, as well as the professional standards of applied policy analysis. For many services, the point of evaluation for the client is most likely to be a comparison of the quality of the ERS product with that of another, actual or hypothetical, supplier of the service, subject to similar constraints on timeliness, budget, and relevance of the response to the question posed. For managers in a research and information agency, there is therefore an implied efficiency standard of quality in all of the agency's work: for a given allocation of budget and a specified time period, and for the given analytical framing of the question, it should not be possible to obtain higher-quality research and information than that currently being delivered. An ongoing task of my agency is to examine its system for procuring research and information with respect to this efficiency standard of quality. Since disciplinary knowledge and standards constantly advance, current knowledge must be incorporated in the provision of research and information. For intramural work, new knowledge can be infused by hiring new, recently trained staff, by employing visitors with up-to-date skills, and by continuing the education of existing staff. For extramural work, meeting current disciplinary standards should be made an important part of the scoring function for the award of contracts and grants. The standards of publication in peer-reviewed academic journals are at best indirectly related to the attribute of quality of research in a research and information agency in support of public economic policy. Although regular journal publication of research provided by the agency is a reliable indicator of the quality of the research, it may also reflect an undue emphasis on methodological innovation at the expense of relevance. Failure to publish in such journals may indicate that, in addressing a relevant question in a timely way, methodological innovation was not necessary or was not possible given constraints. Using such standards fails to recognize that the appropriate criterion is one of comparison: given all of the other relevant constraints, is the research and information delivered of the highest

OCR for page 116
attainable quality? Meeting the latter, relevant standard of quality is an important key to successful evaluation of the agency's research program, and to its credibility and reputation in the long run. Credibility Establishing and maintaining the credibility of a research and information agency in support of public economic policy is in the long-term interest of the agency and of the department of which it is a pact. Credibility within the department is primarily the responsibility of agency management. Credibility beyond the department requires the support of the secretary as well as appropriate management of the agency. Within the department, senior agency managers, or their designated liaisons, must he able to answer questions posed by decision makers quickly and in plain language. This requires a thorough understanding of the question being posed, the framing of the relevant research question, and how well agency research and information respond to the questions. These qualities we distinct from those the agency seeks in its senior scientists and research investigators. If the agency regularly meets, high standards, for timeliness, relevance, and quality, then its findings will have a respected seat at the table when decisions are made. A reputation for such standards makes it more likely the agency findings that contravene the policy position of USDA or the administration will be given a hearing. Conversely, findings that support policy will be regarded seriously, even by those who hold opposing views. The credibility and reputation of the agency can carry beyond the department. At one extreme, if the agency is known only for providing findings selectively and after the fact in support of positions already established for political reasons, and if the department is known to censor findings that would contravene its position, then the agency's research will carry no weight. Even when dispassionate, anticipatory research in fact supports USDA's position, that fact will be of little advantage for USDA or the administration in dealing with Congress, the public, and other countries. At the other extreme, if agency research meets high standards for quality and relevance and is made public without regard to established political positions, then it will carry considerable weight outside the department and may be used effectively. As Practical matter, this latter situation can be approached only if research meets the standard for timeless set forth above—namely, if it take place in anticipation of a future policy decision, and its credibility is sealed by public delivery of a report prior to the policy debate. To the extent that the agency can demonstrate that it identified and framed the problem for analysis It that Political considerations did not enter findings of fact, its credibility will be enhanced. This requires some distancing of those provide counsel to the secretary from those who carry out the research. In the short run, this insulation

OCR for page 116
of research findings from the political process can be awkward and even on occasion painful, but in the long run—as discussed in Chapters 3 and 4—it is in the best interests of the agency, the department, and the nation. Principles for Research and Information Administration The responsibility of ERS is to provide research and information in support of public economic policy to its clients and in particular to the secretary. In the long run, there are several, mutually competitive, ways that ERS could meet this responsibility. Within the framework for evaluation described in Chapter 5, ERS must examine these possibilities with respect to their prospective impact on the eventual evaluation of its work. Principle of Competition The principle that services should be supplied competitively is a fundamental premise of our economic system, including government procurement. It is recognized in the Competition in Contracting Act of 1984, and it has long been the foundation of most publicly sponsored scientific research. Against this principle, there are always powerful forces pushing in the direction of a mandated sole-source supply for services. In the public sector, these forces repeatedly manifest themselves in legislation earmarking the creation of a research institute here or a laboratory there, and in rules and institutions that favor continued relations with established suppliers. A competitive supplier of credible research services, in particular, must constantly integrate new ideas in order to continue supplying those services. There is pressure for both individuals and organizations to reach beyond their immediate area of expertise to gain a competitive edge by bringing to bear new results from related fields. This process has been a hallmark of the public program of grant research in science and medicine administered in the United States over the past 50 years. Suppliers of research services with permanent, sole-source awards have no such incentives to reach out, and they often become isolated within narrow fields using methods that are increasingly outdated, As emphasized in Chapter 5, comparison and competition are central to the process of evaluation. Eventually, clients for services will make choices among alternative providers. A successful manager will recognize this fact and look ahead to clients' competitive comparisons in deciding how services will be delivered. To expect any less, in a government agency, would be an abuse of public trust. Managers of research and information agencies in support of public economic policy, in particular, must have the discretion to choose widely from among potential vendors of the services they are responsible for providing to their clients, and they must exercise that discretion. Much of the balance of this report

OCR for page 116
addresses procedures and institutions to facilitate such discretion, while ensuring that the decisions of research and information managers reflect the interests of their clients rather than the interests of vendors. Potential Impediments to Competitive Procurement No organization should ever be given a permanent source award for the provision of a service. Decisions to provide sole-source must be defended on a recurring basis, beginning from the presumption that services should be procured competitively. Extended conversations with agency heads and former agency heads revealed three reasons why an agency may decide not to undertake competitive procurement. Transaction Costs. If the transaction costs of setting up the competition are large relative to the size the service being procured, then it may be more cost-effective to provide the service on a sole-source basis. The procurement could be from an outside vendor, from another agency, or from within the agency itself. The unit appropriate for procuring services is therefore an important management decision: at one extreme, agencies make small grants and contracts for well under $ 100,000 for specific deliverables; at the other, agencies make multimillion-dollar periodic commitments to federally funded research and development centers for wide-ranging research over multiyear periods. Timeliness. If timeliness is critical, then it may be essential that services be maintained within the agency, to be available instantaneously. This case is most compelling when information is regularly needed on demand at a level of detail that can be provided only at the career staff level, as opposed to economic advisers appointed to serve their client. For example, if the secretary regularly requires information on less than a few hours' notice on such topics as the impact of current drought conditions on farm foreclosures in Texas in the past month, then permanent intramural staff may indeed be the only alternative. However, instant counsel on more general economic matters could be obtained from an under secretary or subject to OPM rules, and research into the impact of weather-related risk on farm finance could be undertaken by a variety of individuals and organizations outside ERS. Confidentiality. The need to maintain secrecy may, or may not, be a reason not to close service to competitive procurement. With respect to the substantive work in ERS, use of privileged information for personal gain is the main concern. Acting on advance knowledge of this new information could yield substantial personal fortunes and is forbidden for the same reason that insider trading is a felony. Before ERS and the National Agricultural Statistics Service release new data and attendant forecasts of crop yields and production, great physical security is provided in the final stage: the so-called look-up, in which analysts are not permitted to look-up area in the final hours before the processing

OCR for page 116
USDA, and the providers of research and information, is likely to vary with the characteristics of alternative suppliers of research and information services and with the nature of their research. Competitive procurement can go far to address two of the most important aspects of the widespread perception that the quantity and quality of ERS products are not what they should be, given the number of professionals in ERS and the size of its budget. The first factor, identified in Chapter 4, is that the USDA mandate in economic policy demands a broader range of professional skills than is now present in ERS or is ever likely to be present in the permanent staff of an intramural research agency. Moving away from the permanent sole-source award model means that the best qualified professionals are candidates in providing research and information needed to inform policy decisions. The second factor, also identified in Chapter 4, is that salaries and other dimensions of career opportunities in ERS are less attractive relative to alternatives than they were at one time. USDA and ERS have limited discretion over salaries and some dimensions of career opportunities for their own employees, but through competitive procurement they have access to a much wider universe of professionals, many of whom are not subject to the same limitations on salaries and career opportunities. Alternative Suppliers of Research and Information Services The evaluation process set forth in Chapter 5, which forms the basis for administration of research, information, and policy analysis services, entails comparison of suppliers and potential suppliers of services with respect to service attributes valued by clients. At the inception of the BAE in 1922, these services could be supplied only by an intramural research agency. There are now several alternatives. Private Sector Chapter 2 noted that, if information is nonrival and nonexcludable, then it is a pure public good. Most of the information produced and used by ERS is nonrival—that is, it is not diminished when it is used by an individual or organization. The information is also excludable—that is, individuals can be charged for their use of it: for example, abstracts of ERS reports are available at no additional cost through the Internet, but copies of full reports require payment. Similarly, the information contained in Situation and Outlook reports is nonrival but is excludable and is therefore not a pure public good. What information, if not produced by ERS, would be produced in the private sector? The answer to this question is surely neither none nor all. Changes in technology have substantially lowered the costs of producing secondary information and have greatly increased the options for making it widely available but excludable. The possibilities for delivery to a large group of clients at positive but low cost have devel-

OCR for page 116
oped rapidly. In the light of these developments, the questions of user fees and the likely structure of the market for the kind of information ERS provides, if it were to either charge user fees or withdraw completely, needs to be examined carefully and dispassionately. Intramural Research Agency ERS provides research and information almost solely from its staff of about 550 employees, including over 300 professionals.1 Other research agencies use a combination of intramural and extramural research—this is true even of the National Security Agency, which operates a small extramural research program. This report documents the important global considerations in the choice of vendors, but it does not enter into this choice at the service-by-service level appropriate to actual management. On a service-by-service basis, ERS might benefit from the experience of other agencies in their distribution of work between intramural and extramural vendors. Federally Funded Research and Development Centers (FFRDCs) Thirteen federal departments or agencies utilize 37 FFRDCs (NSF, 1998b). In each case, the research sponsor and the FFRDC enter into a relationship that is long-term but not permanent. The relationship is reviewed periodically. Substandard performance can in principle lead to the termination of the relationship. FFRDCs develop and maintain models, surveys, and other infrastructure in support of the mission of their client agency. They hire professionals who often spend major portions of their careers in the FFRDC. Many FFRDCs compete directly with the very best academic institutions to retain key senior personnel. They are able to move personnel between assignments and on occasions respond rapidly to short-term requests, in ways that universities and university-based research institutes usually cannot. They are not subject to OPM rules. The relationship between an agency and an FFRDC requires careful management. The agency and the FFRDC must establish a working, long-term relationship. At the same time, this relationship cannot degenerate into a permanent sole-source contract, with personnel moving back and forth between the agency and the FFRDC. The possibility that the FFRDC would lose a contract given substandard performance must be real. 1   A recent departure from this mode of operation took place in March 1998, with the request for proposals ''Food Assistance and Nutrition Research Program: Studies of Households Who Leave the Food Stamp Program." This initiative was undertaken as a consequence of the 1998 appropriations bill for USDA, which consolidated all research and evaluation studies of the USDA's food assistance programs under ERS and increased the budget for these studies.

OCR for page 116
University-Based Research Institutes University-based research institutes have some of the same characteristics as FFRDCs. They have long-term relationships based on multiyear contracts with the sponsoring agencies, but at regular intervals—at least 5 years, rarely more than 10—there are serious recompetitions that the incumbents lose with some regularity. Their work is circumscribed by the policy responsibilities of the sponsoring agency. Compared with FFRDCs, the emphasis is more on long-term research, less on the development of infrastructure, and university-based research institutes rarely provide direct support for day-to-day staff analyses. They typically draw heavily on the academic expertise of their university base and employ flexible arrangements to involve outstanding senior scholars on a part-time basis. The relationship between an agency and a university-based research institute requires careful management. Although day-to-day involvement in policy is not the reason for using a university-based research institute, its research must ultimately be relevant to the policy responsibilities of the agency. The agency's sponsorship must not simply become scholarship and stipend support for open-ended academic research. Grant and Contract Research Grant and contract research can be used to bring concentrations of specific talent to bear on particular research questions. The universe of potential investigators is greater than under any of the other modes of procurement, and specific individuals or groups may be matched well with specific research questions. Success in using grant and contract research requires careful attention to the framing of the question, the solicitation for proposals, the evaluation of proposals, and the maintenance of long-term relations with vendors. The framing of the question is critical. As discussed previously, identifying the research questions relevant to the anticipated policy question requires skill and careful consideration. Agency research administrators often draw on the expertise of agency clients and potential vendors in framing the question. This can serve to familiarize vendors with the pertinent policy questions and provide clients administrators with some idea of what research may be able to contribute to policy analysis within time and resource constraints. Occasionally agencies have held short conferences for this purpose. As administrators develop ongoing relations with the community of potential vendors, they improve their own management decisions by drawing on talent beyond their agency. The proposal solicitation must make clear the terms of evaluation to prospective suppliers. For example, if the research being procured is in area in which there has been considerable academic work but a dearth of findings relevant for a particular kind of policy decision, then this must be made clear in the solicitation

OCR for page 116
so as to avoid the impression that a reworking of the academic literature will suffice. The solicitation should clarify the relative weight attached not only to the proposal itself, but also to the characteristics of the vendor. In particular, substantial weight should be given to the past performance of the proposer, if the proposer has previously been a vendor for the agency. This provides a proper set of incentives for the supplier who is selected to carry out the research, for once this is done the supplier is in a monopoly position with respect to the agency for that project. In general, the evaluation criteria for a research and information agency in support of public economic policy will be quite different from the criteria used by agencies that fund external basic research, including the National Institutes of Health, the Agricultural Research Service, and the National Science Foundation. These agencies tend to emphasize quality with loose restrictions on the area of research, whereas for an agency like ERS the relevance to a specific policy context and timeliness of the work is often critical. Wholesale adoption of basic research procurement models is bound to be inappropriate. In most cases, the evaluation of proposals involves considerably more than just selection by agency administrators. The same group involved in framing the question will typically have important contributions to the evaluation of proposals—including knowledge of agency clients, qualified experts, and potential vendors free of conflict of interest. The best proposals often raise issues that were not foreseen at the time the solicitation was drawn up: new ways of framing the question may be suggested, unforeseen difficulties with methodology or data may be uncovered, and new and untried techniques may be proposed. In some cases, the agency may obtain substantially improved research by reframing the proposal on the basis of this new information, and it may wish to enter into agreements with more than one vendor. It is important to develop long-term relationships with good suppliers. On one hand, good suppliers should be able to rationalize a long-term investment in skills specific to agency needs, knowing that there will not be capricious decisions about programs and vendors. This is especially important when the work demands that those doing the research make specific investments in a project that cannot be transferred to other projects. On the other hand, all suppliers should understand that all work is subject to recompetition, and that poor performance will result in the loss of the contract. RECOMMENDATION 6-2. Choices among alternative vendors of research and information in support of public economic policy should be based on prospects for favorable evaluation of the services provided, as well as on the costs of the services. The critical attributes established in program evaluation provide the framework for choice among vendors. No single model of choice among vendors is appropriate for all programs. In particular, the methods used by the National Science Foundation, the National Institutes of

OCR for page 116
Health, the Agricultural Research Service, and the National Research Initiative will not be suited to many ERS programs and should not be presumed to be appropriate to any. Organizing Research, Information, and Policy Analysis Many of the considerations taken up in this chapter apply to most research and information agencies in support of public economic policy. There are specific features of ERS and its history that bear on the application of these principles in organizing its research, information, and policy analysis services. ERS in the Wider Research and Information Community For most of its history ERS has been closely tied in many ways to the land grant institutions and the departments of agricultural economics in those universities. From the late 1920s to 1983, when some ERS professional staff were posted outside Washington, most postings were to land grant institutions. The research and information mandate of ERS, as reflected in Box 5.1, still includes quite a few topics found in the portfolio of the traditional agricultural research establishment, but many other topics—financial institutions, nutrition, climate change, auction mechanism design, price indices, food assistance, etc.—are widely studied by economists and other social scientists in many universities, research institutes, and the private sector. The principles of evaluation set forth in Chapter 5 imply that ERS should develop relations with a wider community and in its research program it should encourage new combinations of ideas from all pertinent disciplines. RECOMMENDATION 6-3. In moving to a system of competitive procurement from a system of permanent sole-source retainers, the long-term commitment of ERS to competition should be conveyed to all potential suppliers of services. If ERS seeks to develop long-term relations with the entire community of potential suppliers, then this fact must be conveyed credibly. USDA and the land grant institutions have strong ties with Congress dating to the time when a majority of Americans lived in rural, rather than urban, areas. Many of these ties remain today. Congress makes direct appropriations to USDA for state research, extension, and forestry, which USDA in turn distributes to the states according to congressionally defined formulas. The extension system, in turn, extends literally to every county in the country. The early rationale for this system can be traced largely to the reluctance and inability of states to invest in research, the location-specific nature of agriculture, the diversity of agricultural markets and products, and the historically atomistic nature of production in agri-

OCR for page 116
culture. It has played a major role in the abundance and low cost of agricultural products in the United States. But these same factors worked against competition between suppliers of research service and their application. The necessity for genetic adaptation to specific ecosystems dictated that hybrid corn appropriate for Missouri would not compete with that developed in Michigan, and pest management in California could not be taken wholesale to Wisconsin. The politics of agriculture reinforced unconditional commitments of research funding to states and institutions. In the period since World War I, state-level investments in agricultural research have grown to exceed that of the federal government, creating a very different incentive for federal appropriations to states (Alston and Pardey, 1996). A significant portion of the benefits from any one state's research investment now spill over to other states, reducing the incentive for states to invest to an optimum level nationally (Alston and Pardey, 1996). Exceptionally high levels of return to agricultural investment persist, supporting the conclusion that the nation has been and continues to under invest in such research (Alston and Pardey, 1996; Huffman and Evenson, 1993). Without adequate compensation to the states for their loss of benefits, this will continue. This decentralized model is less applicable to the substantive economic research mandate of ERS in the 1990s and the economic policy questions that will confront the secretary of agriculture or the functional equivalent in the foreseeable future. It is essential that ERS reach out to a wider community of research and information providers. In moving to a model in which the presumption is in favor of competitive procurement, it is essential that the competitive mode be taken seriously by the secretary, the executive branch, and the Congress. A permanent sole-source contract has no virtue simply because it is granted externally. Although long-term, consistent support for the principle of competition and the importance of the attribute of quality must start in the secretary's office, it must also be sustained by similar understanding of the commitment in the executive branch and the Congress. This requires some agreement among these parties on a common set of expectations and rules to govern the role and performance of ERS. Currently this understanding is at best incomplete, with ERS sometimes caught between inconsistent expectations. RECOMMENDATION 6-4. Support for the principle of competition and the necessary attributes of quality, credibility, timeliness, and relevance by the secretary's office, the executive branch, and the Congress is essential in moving to a system of competitive procurement. It is important to fund projects and people rather than institutions, and to do so subject to periodic evaluation by qualified reviewers. Mutual understanding by political leaders in the executive branch and the congressional leadership of both parties of what is entailed by quality, credibility,

OCR for page 116
timeliness, and relevance is essential to the effectiveness of ERS or any agency conducting research in support of policy analysis. If the expectations of the executive and congressional branches with regard to these points conflict, then the work of the agency will be less than effective and the budget of the agency is very likely to suffer. Lack of a clear set of common expectations has plagued the performance of ERS for some time. Staff Analysis Staff analysis is the link between the ERS research and information programs and the policy decisions that these programs support. Close contact between staff analysis leaders and policy makers is required to ensure that the entire ERS program remains relevant to the substantive economic policy mandate of USDA. Information that must be provided on a very short-term basis—often a few days or less—requires that those providing the information be immediately available. Staff analysts must also be closely involved in guiding the ERS research and information program, including the assessment of future policy questions, the framing of questions for investigation, and the organization and supervision of research, because staff analysts are the first line of contact with policy decisions. Thus, leadership in staff analysis requires a sophisticated combination of analytical and management skills. The important attributes of credibility and relevance in staff analysis and the need for staff analysis leadership to participate in overseeing research and information programs indicate that this function must be provided by a permanent, skilled group within ERS. RECOMMENDATION 6-5. To provide immediate support for its economic policy decisions, USDA should maintain a permanent core of staff analysts. The size and composition of this group should reflect the level of detail and timeliness required in support of the economic policy mandate of USDA, and it should be reviewed from time to time as the mandate evolves. The leadership of this group must provide a combination of management and analytical skills essential to the administration of the research and information programs of ERS. ERS should regularly invigorate this group by means of visiting scholars, sabbaticals, internships, or similar programs, to maintain the contact of staff analysts with the wider research community. Information Historically there has been consistent demand for the information services of ERS (and the BAE preceding it). These services include Situation and Outlook reports and other information combining some analysis with data. (A more de-

OCR for page 116
tailed description is provided at the start of Chapter 5.) These services began with a focus on commodity markets but are being slowly extended to support public and private decision making in newer areas of USDA responsibility, including the environment and natural resources. Much of the strongest support for these services has come from smaller users, many of whom have limited analytic capacity to interpret data. There is a demonstrated private-sector demand for these information services, which will remain even if traditional price support programs vanish. There will also be other public policy uses for this information. Essentially all of this information is both nonrival and excludable. It is being produced by ERS, but not because it is a pure public good. Clients for this information include private-sector decision makers and other branches of government, as well as the secretary. Extensive price support programs for a large number of agricultural commodities require detailed projections of supply, demand, and prices in order to estimate the budgetary and farm income implications of price supports. Under the 1996 legislation these programs are being changed, with the possibility of further reductions or even elimination in 2002, the next farm legislation renewal date. If the programs are eliminated, then much of the secondary information produced by ERS will not be as vital to USDA as it once was for budget forecasting. However, there could be other reasons for maintaining public provision of this information. The regulation of agricultural production for environmental and other purposes is increasing and could require this type of secondary information and forecasts to analyze the costs and benefits of regulation on agricultural markets and income before regulations are implemented. Also, if there are market inefficiencies caused by asymmetric information (as discussed in Chapter 2), then public provision of market data and forecasts may improve market efficiency. Different types of secondary data preparation and analysis may also be needed as public priorities change. For instance, with food and nutrition program spending far exceeding government spending on farm price supports, more secondary data and analysis on food consumption patterns and expenditures may prove valuable in examining future changes in food stamp and child nutrition programs. The attributes of these secondary data and information important for policy analysis within USDA and other government agencies and to other clients would then drive the analysis of the question: Should this information be produced internally at ERS, should it be produced by other vendors under contract to ERS (as indicated in Recommendation 6-1), or should it be left to the private sector? The frequency and detail with which this information should be produced is also subject to prospective evaluation within the framework set forth in Chapter 5. RECOMMENDATION 6-6. The secondary data preparation and analysis programs of ERS should be evaluated within the framework outlined by the panel, including consultations with clients. On the basis of this evalua-

OCR for page 116
tion, a long-term plan should be drawn up, including new and discontinued services. The plan should indicate which of the services provided will be produced in ERS, which will be procured from other vendors, and which will be left to the private sector. The plan should include the anticipated impact on clients and the projected impact on the USDA budget. Research In the evaluation framework, the attributes of services are the dimensions along which comparisons are made. For ERS services generally, and for its intermediate and long-term research in particular, these attributes are likely to include timeliness, relevance, quality, and credibility. The previous discussion has indicated how these attributes are interwoven in the case of research. In particular, research that is undertaken well before a policy debate reaches a crescendo, and thus is available before policy positions are formed, is more credible than research produced just in time by one of the parties to the debate. The same is true of research that is conducted by the best-qualified investigators, directly addresses the policy issues at hand, and is not subject to clearance by either political or private interests. An arm's length relationship between the sponsor of intermediate and long-term research and the investigation process itself has important ultimate advantages for the sponsor and other policy makers, as well as for the taxpayers who make the work possible. Clear and public framing of the questions for investigation including perhaps key assumptions to be made, followed by unimpeded skilled scientific investigation, provides the most propitious environment for research in support of informed public economic policy. If there is to be rational dialogue on questions of economic policy, especially in a charged, partisan political environment, some common ground rules for fact finding are in order in both USDA and the Congress. ERS, and other research and information agencies in support of public economic policy, can do much to bring this about. It is in their long-term interests to do so. RECOMMENDATION 6-7. USDA should support the integrity of its intermediate and long-term research programs in support of economic policy, while retaining the prerogative to disagree with research findings. These programs should be conducted with the clear objective that peer-reviewed research findings may be published by the investigators independently and without prior approval by USDA, and with the clear understanding that USDA does not necessarily endorse the findings of any research program. The attribute of quality, discussed at length above, requires that ERS consider the widest feasible group of vendors to provide intermediate and long-term research and to use imagination in the vehicles for this research. For example, in

OCR for page 116
anticipation of an emerging policy issue, ERS can convene one- or two-day meetings in which the relevant issues are conveyed to groups of highly skilled investigators, and ERS managers and staff analysts can then further pinpoint the analytical issues for intermediate or long-term research. These same groups, or individuals within these groups, can also be engaged by ERS as peer reviewers of the scientific content of intermediate and long-term research. Such review would be an important component of quality control, and would be consistent with the support for the integrity of these programs addressed in Recommendation 6-7. ERS, or any other research and information agency in support of public economic policy, can bring more skilled and varied talent to bear at critical junctures in its work, than it could ever contemplate retaining as permanent staff. The attribute of quality points strongly in the direction of using external vendors for most long-term research. The attribute of credibility reinforces this conclusion. The advantages of arm's length relationships for reaching critical or sensitive conclusions are well understood. Blue-ribbon panels, independent fact-finding investigators, and studies by the National Research Council all underscore this understanding. By contrast, intramural research conducted by permanent staff in a government agency subject to clearance, and studies by private firms carried on internally or by consultants for hire face overwhelming odds against their credibility in any political charged policy debate. Research in support of public policy depends for its credibility, in great part, on its conduct by those with a greater interest in the quality of the work than in the substance of the conclusion. If ERS can credibly assure professionals—whether permanent employees or external vendors—that their work will not be subject to political interference, then it can more readily attract top talent, thereby further increasing the quality of its product. RECOMMENDATION 6-8. Vendors for intermediate and long-term research programs in support of economic policy should be sought from the widest possible universe of qualified investigators and organizations. Intermediate and long-term research conducted by all vendors, including ERS staff, must be subject to the understanding that their peer-reviewed research findings may be published without prior approval by USDA. As previously discussed, this organization of research requires ERS managers and staff analysts to ensure that their investigators understand the relevant policy questions motivating the research. This includes knowledge of institutions, the policy context, and data, as well as methodology. These attributes of good research can be made clear in solicitations and evaluations. (These requirements must be met whether research is conducted internally or externally.) The advantages with respect to quality and credibility of research that is administered internally but conducted externally outweigh these and other transaction costs in most cases.

OCR for page 116
Conclusion The economic policy mandate of the USDA has grown, in the twentieth century, from issues of specific economic concern to American farmers, to complex and critical economic questions involving social welfare, food security and safety, environmental change, international relations, and other issues affecting every-one in the world. The Department of Agriculture and the Economic Research Service have an opportunity to reach out to a correspondingly wide research and information community to bring to bear the best minds in pursuit of dispassionate solutions to these problems. This requires both a reconsideration of the deployment of information and research resources in support of public economic policy and a reaffirmation of this commitment of resources as a matter of policy. In this context, ERS can make key contributions in addressing these issues in the next century. Implementing the principle of competition is an important initiative USDA and ERS can undertake in meeting the challenges faced now and in the next decade. The research and information required to support the rapidly widening and challenging policy mandate of USDA cannot be produced to high standards by any single permanent staff of professionals, including ERS. Flexibility in selecting providers of research and information products is essential. When ERS staff are selected, the principle of competition will ensure both the actual and perceived quality and professionalism of the work. Consistent with the principle of competition, a consistent policy of insulation of economic research and fact finding from political intervention will make ERS more attractive to motivated and capable professionals, and will further improve perceptions of the quality of ERS products.