The Funding of Social Knowledge Production and Application: A Survey of Federal Agencies (1978)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

2 The Framework of Social Knowledge Production and Application WHY A NEW FRAMEWORK? The Study Project on Social Research and Development was given a broad mandate to look at a variety of activities that have all come to be associated with social R&D in the minds of governmental policy makers and the academic community. However, many of those activities—such as program evaluation and general purpose social statistics—fall outside the traditional definition of research and devel- opment. The traditional R&D framework as well as existing R&D data do not adequately capture the wide diversity of activities supported by the federal government that are aimed at either producing or applying knowledge about social problems. A new framework, developed by the Study Project, is presented on pages 12 and 13. The Study Project found it useful to think of the diverse activities as falling into two clusters—one centering around the production of new knowledge and the other centering around the appli- cation of that knowledge. The framework is based on the premise that there are many ways to produce and apply knowledge about social problems. The Study Project felt that previous attempts tended to focus too narrowly on research and neglect other legitimate means of gaining knowledge.1 Those previous studies also tended to ignore al- 'The only exception is the 1964 American Enterprise Institute study. The Federal Gov- ernment in Behavioral Science: Fields, Methods and Funds (contained in U.S. Congress. House, Committee on Government Operations, The Use of Social Research in Federal 10

The Framework \ 1 together the application of knowledge. The framework developed by the Study Project enables one to look at the entire system of both knowledge production and knowledge application—not just one por- tion of that complex system. The Study Project found the traditional R&D approach lacking in several other respects. First, the categories of research (basic and applied) and development seem to more accurately reflect perceptions of the physical sciences and technological R&D. Second, the distinc- tion between basic and applied research in the social and behavioral sciences has long been a difficult one. Third, much of the application of knowledge to solving problems does not and cannot occur through the development of products and processes. The category of "develop- ment" therefore provides little insight into a discussion of the social R&D system. For these reasons, the traditional R&D framework did not capture the many dimensions and concerns that gave rise to the Study Project's work. A major goal of the Study Project survey was to better describe the diverse set of activities associated with social R&D and social prob- lems. Particularly troublesome was the concept of "development" when applied to the social sciences and social problems. Development has long posed problems for those interested in federal support of the social sciences. The 1968 BASS report2 did not confront the question of development directly, but instead estimated that 1 percent of all devel- opment was probably "social." Since development is considered to be a nondisciplinary activity, it has not been possible to know how much of the dollar total reported to NSF and OMB as "development" was related to the social sciences or social problems. In 1968, the Reuss Committee summed up the problem well: In seventeen years, the Foundation has not decided if it is, or is not, possible to spend any money on "development" in the social sciences (a category that absorbs more than half of all agency expenditures in other areas of "R" and "D") but, while NSF continues to make up its mind, some agencies do, and others do not, report "demonstration" projects as "social science research."3 Domestic Programs, 90th Congress, 1st Session, Part I, 1967). This study used a very broad definition of both social science and research and included general purpose statis- tics. Using this broad definition, the study reported that federal expenditures exceeded $210 million in 1963. *A study sponsored by the National Research Council and the Social Science Research Council, called The Behavioral and Social Sciences: Outlook and Needs, estimated that in 1967 the federal government spent $388 million on basic research, applied research, and development. 3U.S. Congress, House, Committee on Government Operations, The Use of Social Re- search in Federal Domestic Programs, 90th Congress, 1st Session, Part 1. 1967.

DEFINITION OF SOCIAL KNOWLEDGE PRODUCTION AND KNOWLEDGE APPLICATION ACTIVITIES Knowledge Production Research Research is systematic, intensive study directed toward greater knowledge or understanding of the subject studied. Social research includes basic, applied, or policy research that studies either the behavior of individuals, groups, or institutions or the effects of policies, programs, or technologies on behavior, t Demonstrations for Policy Formulation A demonstration is a small-scale program undertaken in an operational setting for a finite period of time to test the desirability of a proposed course of action. A demonstration for policy formulation is undertaken to learn new information about the outcomes and administrative feasibility of a proposed action. Social ex- periments are included in this category. •Program Evaluation Program evaluation is evaluation that seeks to syste- matically analyze federal programs (or their components) to determine the extent to which they have achieved their objectives. A distinguishing factor of program evaluation is that national operating programs (or their components) are evaluated for the use of agency decision makers in mak- ing policy or program decisions. Program evaluation is defined as a man- agement tool; more general types of evaluation studies (activities fre- quently labeled evaluation research) were judged not to be oriented to management or decision making and were categorized as research.! •General Purpose Statistics General purpose statistics include either current or periodic data of general interest and use. A characteristic of general purpose statistics is that many of the specific users and uses are unknown. These statistics provide all levels of government and the private sector with information on a very broad spectrum of social, economic, and demo- graphic topics. Statistics that are collected for the specific purpose of providing research data in a specific area of inquiry have been categorized as research.! 12

Knowledge Application * Demonstrations for Policy 1mplementation A demonstration is a small- scale program undertaken in an operational setting for a finite period of time to test the desirability of a proposed course of action. A demon- stration for policy implementation is undertaken to promote the use of a particular action. This type of demonstration does not attempt to gen- erate new information but instead attempts to apply existing knowledge. Development of Materials The development of materials consists of the systematic use of knowledge and understanding gained from research to produce materials. Examples of such materials are educational curriculum materials or methods, testing instruments, and management or training curricula. Such materials are used in a variety of educational, training, or testing settings.t •Dissemination Dissemination consists of activities undertaken by research managers or others to promote the application of knowledge or data re- sulting from social knowledge production activities.t Dissemination activities include: Publication and distribution of scientific and technical information resulting from social research; Documentation, reference, and information services (information retrieval systems); Research syntheses written for the use of practitioners and decision makers; Technical assistance to practitioners to disseminate knowledge; Support of conferences to disseminate information; and Creation of dissemination networks and consortia. *The asterisked categories fall outside the definition of research and development used by the National Science Foundation and the Office of Management and Budget. This knowledge production and knowledge application framework can thus be viewed as containing social R&D and related activities. tThese definitions are similar to those used by the National Science Foundation and the Office of Management and Budget. 13

14 SCOPE OF THE SURVEY Development, as it relates to the social sciences and social problems, thus was a "black box" for the Study Project. It was clear that large amounts of funds were being reported to the NSF Federal Funds survey as development by agencies directly concerned with social problems, but little was known about the types of activities included under devel- opment. This problem is well illustrated by two agencies, the Office of Education and the Office of Human Development Services (previously called the Office of Human Development) in the Department of Health, Education, and Welfare. For fiscal 1976, both agencies reported almost 90 percent of their total R&D obligations as "development."4 The Study Project wanted to find out what type of activities were classified as development by those agencies—demonstrations, curriculum de- velopment, or other types of activities. From its inception, the Study Project felt that one of its major tasks was to explore the concept of development and to better understand the types of activities categorized as development by agencies concerned with social prob- lems. For all these reasons, the Study Project decided to use a framework of knowledge production and knowledge application. It was felt that this framework is more descriptive of the activities actually being con- ducted and funded by the federal government in the area of social problems. The Study Project felt it necessary to collect its own data within this framework in order to probe the nature of the activities classified as research and development by federal agencies. Another major factor in the decision to develop a new framework was the desire to include program evaluation and general purpose statistics in the survey. The inclusion of program evaluation reflects two major judgments. First, such evaluations have provided substan- tial insight into the effect of government policies and, to some extent, into the behaviors of citizens and institutions. Program evaluations have come to play an important role in our perceptions of social prob- lems. Second, program evaluations frequently use social science meth- odology and represent substantial scientific endeavors. In addition, they have clearly come to be associated with the concept of social R&D, even though such activities are not included in the definition of research and development. The collection, analysis, and publication of social statistics represent the oldest commitment of the federal government to activities that the «The figure cited for the Office of Human Development Services was based on volume 25 of the NSF Federal Funds survey. In volume 26, the fiscal 1976 figures were revised to 55 percent for research and 45 percent for development.

The Framework 15 Study Project has characterized as social R&D. General purpose statis- tics enable us to describe and characterize our society. Beyond that, statistical data are the basis for a substantial amount of social research that seeks to provide insight into the behavior of society, including, for example, the economic statistics that underlie models of the economy, the data used to study migratory trends, and the information needed to study patterns of fertility and shifts in the patterns of family structure. DEFINITIONS OF CATEGORIES USED IN THE SURVEY THE CONCEPT OF "SOCIAL" Social R&D consists of research and development and related activities con- cerned with understanding and alleviating social problems. It is intended to include such activities as the production or application of knowledge concern- ing the behavior of individuals, groups, or institutions or the effects of policies, programs, or technologies on behavior. This definition excludes biomedical or technological developments in which only minor attention is given to social or individual impacts. Under this definition, a project that assesses an existing technological capability for its impact on behavior would be considered social, while a project primarily attempting to develop a new technology would not. As might be imagined, the definition leaves considerable room for judgment, but the Study Project found that interviewers and agency personnel could reach agreement after some discussion on the distinc- tion between social and nonsocial. The boundary issue was not as big a problem as originally anticipated. CATEGORIES OF KNOWLEDGE PRODUCTION Research Research is systematic, intensive study directed toward greater knowledge or understanding of the subject studied. Social research was defined as including basic, applied, or policy research that studies either the behavior of individuals, groups, or institutions or the effects of policies, programs, or technologies on behavior. The first part of this definition is similar to the one used by NSF and OMB. The Study Project, however, did not attempt to make the distinc- tion between basic, applied, or policy research—a decision that was greeted with enthusiasm by the agency staff. The distinction between basic and applied is frequently difficult to make in any field but seems to be particularly difficult in the behavioral and social sciences.

16 SCOPE OF THE SURVEY While research is typically reported by discipline, the Study Project was interested in activities related to problems. Because of this, we included research on social problems that was carried out in disciplines other than social science and psychology. The definition of social re- search cited above emphasizes the problem being studied rather than the discipline studying it. Thus, the Study Project total for research activities is higher than the NSF figure for total (basic and applied) research in the social sciences and psychology. The Study Project research total includes some multidisciplinary research that is categorized by NSF in its various NEC (not elsewhere classified) categories. (For data on federal obligations by academic discipline, NSF provides the best data available.) Over 85 percent of the Study Project's research total was reported to the NSF Federal Funds survey. The remaining 15 percent was not re- ported, because either the individual agencies spending the funds do not report to NSF or the activities were not considered "research" by the agencies.5 Policy Formulation Demonstrations A demonstration is a small-scale pro- gram undertaken in an operational setting for a finite period of time to test the desirability of a proposed course of action. A demonstration for policy formula- tion is undertaken to learn new information about the outcomes and adminis- trative feasibility of a proposed action. Social experiments are included in this category. This definition was developed by the Study Project. Policy formula- tion demonstrations are funded to generate new information relating to the cost, the demand, and the institutional compatibility of a particular action or concept in a local setting. In general, the knowledge to be gained from policy formulation demonstrations has two dimensions: the impact and the feasibility of the proposed action. The impact of a programmatic concept is the capacity of the concept or approach to produce the desired effect on a target population as well as the unexpected medical, social, economic, or demographic external 'Fourteen agencies do not report any of their activities as either "research" or "devel- opment" to the NSF Federal Funds survey. The 14 agencies are: Appalachian Regional Commission, Commission on Civil Rights. Equal Employment Opportunity Commis- sion, Council of Economic Advisers, Council on Environmental Quality. Council on Wage and Price Stability, Federal Mediation and Conciliation Service, Federal Power Commission, Federal Reserve System, International Trade Commission. Interstate Commerce Commission, National Foundation on the Arts and Humanities, the National Center for Productivity and Quality of Working Life, and the Securities and Exchange Commission. Although these agencies do not report any R&D expenditures, some of their activities clearly fit our definition of social knowledge production and application.

The Framework 17 effects. Appraisal of the political and administrative feasibility of the programmatic concept is also studied. Feasibility indicates the capacity of the political and administrative system to bear the new programma- tic concept. It also indicates whether the new concept can be provided in a cost-effective way. When the concept of policy formulation demonstrations was ex- plained to agency personnel, it was quickly understood. Approxi- mately 50 percent of this category was reported to NSF as development; another 25 percent was reported as research;6 the remaining 25 percent was not reported at all. It is clear that each agency handles this type of demonstration differently. However, over 75 percent of all policy for- mulation demonstrations was categorized by agency personnel as fit- ting the traditional definition of research and development. Program Evaluation Program evaluation is evaluation that seeks to systemat- ically analyze federal programs (or their components) to determine the extent to which they have achieved their objectives. A distinguishing factor of pro- gram evaluation is that national operating programs (or their components) are evaluated for the use of agency decision makers in making policy or program decisions. Program evaluation is defined as a management tool. More general types of evaluation studies (activities frequently labeled evaluation research) were judged not to be management- or decision-oriented and have been categorized as research. This definition borrows heavily from the one developed by the Evaluation and Program Implementation Division of OMB.7 The line between research and evaluation appears to be very thin in many agen- cies. Because the term evaluation is used very loosely across the gov- ernment, many traditional research activities are called evaluation. In its definition the Study Project emphasized the evaluation of federal programs; the number of "evaluation" projects reported to the Study Project was thus reduced considerably. While some agencies have activities labeled "program evaluation," most do not. When activities are clearly labeled program evaluation, they do not seem to be reported to the NSF survey. "Because 25 percent of all policy formulation demonstrations were reported to NSF as research, roughly $50 million could be added to the Study Project research total if "research" policy formulation demonstrations were counted as research. This would bring the Study Project research total to over $750 million in fiscal 1977. Social experi- ments are the type of policy formulation demonstrations most closely linked to research and were categorized by the Study Project as policy formulation demonstrations and not research. 'See OMB. "Evaluation Management: A Background Paper." May 1975.

18 SCOPE OF THE SURVEY The Study Project figure of $68 million for program evaluation may strike some readers as too low. In September 1977, OMB published a figure of $200 million for evaluation.8 There are two likely explanations for this discrepancy. First, the OMB figure appears to include funds for research activities that have been inappropriately categorized as evalu- ation. When asked for evaluation dollars only, most agencies tend to use a broad definition of evaluation and include dollars categorized by the Study Project as research. Second, the OMB figure may include management efficiency evaluations, which are not aimed at assessing the impact of federal programs. Efficiency studies were excluded from the Study Project definition. General Purpose Statistics General purpose statistics include either current or periodic data of general interest and use. A characteristic of general purpose statistics is that many of the specific users and uses are unknown. These statistics provide all levels of government and the private sector with informa- tion on a very broad spectrum of social, economic, and demographic topics. Statistics that are collected for the specific purpose of providing research data in a specific area of inquiry have been categorized as research. This definition is the one used by the Statistical Policy Division of OMB." The Study Project excluded program or administrative data from the survey because they are collected as part of an agency's adminis- trative and operating responsibilities and do not serve a knowledge production function. Statistics falls clearly outside the definition of research and devel- opment used by the federal government. As expected, general purpose statistics are largely not reported to NSF as R&D. There are several exceptions, notably longitudinal and research-oriented data collections by nonstatistical agencies. CATEGORIES OF KNOWLEDGE APPLICATION Policy Implementation Demonstrations A demonstration is a small-scale program undertaken in an operational setting for a finite period of time to test the desirability of a proposed course of action. A demonstration for policy implementation is undertaken to promote the use of a particular action. This type of demonstration does not attempt to generate new information but in- stead attempts to apply existing knowledge. In contrast to policy formulation demonstrations, policy implemen- tation demonstrations imply a conscious decision on the part of federal "See OMB, "Resources for Program Evaluation Fiscal 1977," September 1977. "See OMB, Statistical Services of the United States Government, 1975, p. 23.

The Framework 19 officials to promote the application of an already accepted practice. The generation of new knowledge is not sought. The decision to pro- mote a given program or practice may be based on the knowledge gained by a policy formulation demonstration, or that stage may be skipped because no further testing or knowledge generation is consid- ered necessary by policy makers. Policy implementation demonstrations can be termed "show and tell" demonstrations, whose purpose is simply to demonstrate in a real environment the institutional and procedural changes that can be made and draw attention to exemplary practices or program delivery methods. As with policy formulation demonstrations, the concept of a policy implementation demonstration was quickly understood by agency per- sonnel. With few exceptions, the individuals surveyed agreed with the distinction between the two types of demonstrations. Over two-thirds of the funds obligated for policy implementation demonstrations are not reported as either research or development to the NSF Federal Funds survey. In general, then, this type of demon- stration seems clearly not to be considered R&D by federal agencies. However, the one-third of the funds for policy implementation demon- strations reported to the NSF as development indicates that the classifi- cation of demonstrations and development in social areas is ambiguous at best. Development of Materials The development of materials consists of the sys- tematic use of knowledge and understanding gained from research to produce materials. Examples of such materials are educational curriculum materials or methods, testing instruments, and management/training curricula. Such mate- rials are used in a variety of educational, training, or testing settings. In contrast to policy implementation demonstrations, over two- thirds of the total funds for this activity is reported to NSF as develop- ment. Development of materials is one of the three Study Project categories (research and policy formulation demonstrations are the other two) that fit the traditional definition of research and develop- ment. This category represents the one area of social science activity that can truly be termed "development," as tangible products are pro- duced. Dissemination Dissemination activities consist of action undertaken by re- search managers to promote the application of knowledge or data resulting from the conduct of social knowledge production activities. This group of activities encompasses the dissemination of information in recorded or other

20 SCOPE OF THE SURVEY communicable form that presents the status, progress, or results of social knowledge production activities. Dissemination activities include: o Publication and distribution of scientific and technical information o Documentation, reference, and information services (information retrieval systems) o Research synthesis written for the use of practitioners and decision makers o Technical assistance to practitioners to disseminate knowledge o Support of conferences to disseminate information and innovations o Creation of dissemination networks and consortia A variety of activities have been subsumed under this heading, all of which clearly fall outside the definition of R&D. NSF has classified scientific and technical information (or STINFO) as a "related activity" to R&D. The Study Project definition of dissemination goes far beyond the concept of STINFO. Hence the funds reported to NSF as STINFO account only for a small portion of the dissemination total in the Study Project survey. Other dissemination activities included in the survey are: Publication and distribution of social scientific and technical infor- mation. This group of activities includes the publication and distribu- tion of journals, technical reports, conference proceedings, mono- graphs, and other materials that disseminate the results of social R&D. Documentation, reference, and information services (information retrieval systems). This group of activities includes library and refer- ence services as well as specialized information center services that serve to disseminate and make available the results of social R&D. Research syntheses for the use of practitioners and decision makers. These are studies funded by an agency to synthesize a set of research studies or findings in a given problem area. The final result of these efforts is often directed at practitioners in the field or governmental decision makers. Technical assistance to disseminate knowledge. This is assistance either funded or provided by federal research managers to promote knowledge utilization by personal contact with practitioners or deci- sion makers. Support of conferences to disseminate information and innovations.

The Framework 21 Creation of dissemination networks and consortia. Some funding agencies have attempted to build a capacity for dissemination at the state or local level. A network or consortium is created to disseminate knowledge to practitioners or decision makers in a given field. The Study Project's dollar total for dissemination is probably on the low side. The obligations for dissemination represent only separate, identifiable projects aimed at dissemination. If funds for dissemination are included in a research grant, agencies are unable to separate funds for dissemination from funds for the research grant itself. In the case of general purpose statistics, funds for dissemination are included in the agency appropriations for "statistical collection" and thus cannot be isolated. More than 50 percent of the Study Project total for dissemination activities, nearly $165 million, is funded by the Agricultural Extension Service. After extensive interviews with the agency staff, it was de- cided that the activities of the service were largely social and fit the definition (cited above) of technical assistance used by the Study Proj- ect. The remaining activities of the service include the dissemination of scientific agricultural information and were excluded from the Study Project survey totals. SUMMARY ANALYSIS OF STUDY PROJECT CATEGORIES Survey interviews found that three of the seven knowledge production and application categories—research, policy formulation demonstra- tions, and development of materials—were categorized as either re- search or development by agency personnel. Those three categories were usually reported to the NSF Federal Funds survey. Study Project interviews found that the other four categories were generally consid- ered outside the traditional definition of R&D and not reported to the NSF survey. The only exception was policy implementation demonstra- tions, which was treated inconsistently by agency staff. Based on these observations, one can estimate a total for "social R&D" in contrast to "social knowledge production and application." The three categories included in the traditional definition of R&D give a total of approximately $1 billion for social R&D in fiscal 1977. The four "related activities" categories total an additional $874 million. Adding the figures for "social R&D" and "related activities," one gets a total of $ 1.8 billion for social knowledge production and application in

22 SCOPE OF THE SURVEY fiscal 1977. Aggregate Study Project survey totals are presented in more detail in Chapter 3. CLASSIFICATION BY POLICY AREAS The Study Project devoted much time to developing the policy areas used in the survey. Instead of adapting functional categories already in use by OMB or NSF, the Study Project developed a set of categories that are closer to those recently proposed by the General Accounting Office and the House Budget Committee and that incorporate elements of both existing and proposed systems of classification.10 Faced with the choice of developing a classification system using either broad policy areas or more detailed ones, the Study Project opted for broad categories and kept subcategories to a minimum. Broad policy areas were selected, first, because a detailed classifica- tion system was deemed more appropriate for project-level data, and the Study Project survey was based on program-level data. Second, policy areas were an important but secondary concern of the Study Project survey; greater emphasis was placed on collecting data by the knowledge production and application categories. Like most classifications of policy areas, the Study Project categories are far from perfect. Many R&D projects and programs can easily fall into several policy areas. Most have both a primary and secondary policy focus. A related problem is that the predominant mission of an agency tends to dominate. Thus, "R&D" programs funded by the Department of Transportation, for example, tend to be categorized as "transportation" even though the secondary focus of those programs or projects might be the environment or employment patterns. All problems notwithstanding, the data collected on policy areas by the Study Project provide some sense of the relative investments in social knowledge production and application in different policy areas. Rather than attempt to "define" the 12 policy areas used in the survey, the Study Project instead wrote broad descriptions of each policy area. 10Explanations of the GAO and House Budget Committee categories can be found in General Accounting Office. Need for a Government-wide Budget Classification Struc- ture for Federal Research and Development Information. PAD-77-14 and PAD 77- 14A. March 3. 1977; and House Budget Committee, "Working Paper of the House Budget Committee: Recommendations for Improving the Budget Functional Categories," Draft, July 2, 1976 (an earlier version of this paper appeared in the Congressional Record. H 929. February 10. 1976).

The Framework 23 listing the topics or issues most likely to come under each heading. The Study Project did not define a distinct policy area for "defense" since it felt that defense activities fell outside its definition of social prob- lems. However, the Department of Defense was included in the survey and its activities are classified under the appropriate policy areas. HUMAN RESOURCES Health Health is one of the two policy areas for which data were collected by subcategories: health education; health care delivery and services; prevention and control of health problems; mental health; substance abuse prevention and rehabilitation; food and nutrition; and miscel- laneous. (Biomedical research was excluded as falling outside "social R&D.") Education The education category includes knowledge production and application activities on: preschool education (day care, etc.); elementary, sec- ondary, and higher education; vocational and occupational education; education for the handicapped; basic research on education; educa- tional service delivery (education finance and school administration); adult education; and cultural affairs. Health education activities were included in the health category, and science education activities were included in the science and technology base category. Employment and Training The employment and training category includes knowledge production and application on: job training or retraining programs; the delivery of training programs; employment statistics; equal employment opportu- nities; programs aimed at upgrading skills and increasing participation and usefulness in the labor force; pension programs; etc. Vocational education was classified in the education policy area. The proposed budget classification of the General Accounting Office contains this category, whereas existing classification systems do not. Social Services and Income Security The social services and income security category includes knowledge production and application on: the delivery of social services; rehabili-

24 SCOPE OF THE SURVEY tation services; legal services; research and demonstrations on target populations (children, the elderly, minorities); unemployment insur- ance; retirement and disability insurance; public assistance and income supplements (food stamps); veterans' benefits; and the delivery of in- come security programs. Knowledge production and application ac- tivities on housing assistance were included in the housing and com- munity development category. COMMUNITY RESOURCES Economic Growth The economic growth category consists primarily of general purpose economic and demographic statistics and research on fiscal, monetary, and tax policy. Research on productivity, economic development, and business and commerce is also included. This is the only policy area in the Study Project's classification that does not have a comparable budget function in any of the proposed budget classifications. Transportation The transportation category includes knowledge production and appli- cation on: transportation safety; public transportation systems; trans- portation patterns; and the socioeconomic aspects of transportation programs and policies. The small amount of research on telecommuni- cation policy was included in this category. Housing and Community Development The housing and community development category includes knowl- edge production and application on a wide range of related topics: rural housing and development; disaster prevention and relief; area and re- gional development; housing economics and finance; housing assist- ance programs; community growth; land use control techniques; inter- governmental relations; and revenue sharing. Law Enforcement and Justice The law enforcement and justice category includes knowledge produc- tion and application on: the criminal justice system (police, courts, corrections); federal law enforcement; the prevention and causes of crime; drug law enforcement; etc.

The Framework 25 International Affairs The international affairs policy area includes knowledge production and application on: international development; foreign trade; and arms control and disarmament. Some of the activities of the Agency for International Development (AID) were categorized in this policy area, but other AID activities were categorized in education, health, or other policy areas as appropriate. NATURAL RESOURCES Natural Resources and the Environment The natural resources and environment category includes knowledge production and application on the social aspects of: recreational re- sources; conservation and land management; pollution control and abatement; environmental regulations; water resources; etc. Technological research on improving the environment was excluded. Energy Development and Conservation The energy development and conservation category includes knowl- edge production and application on the social aspects of: energy con- servation; the regulation of energy; energy modeling; supply and de- mand studies; etc. SCIENCE AND TECHNOLOGY BASE This category consists primarily of the science education, science pol- icy, and the basic social science research activities of NSF. Basic social science research in other departments was also included in this cate- gory. CLASSIFICATION BY ORGANIZATIONAL LOCATION The Study Project was also interested in finding out the organizational location of units that support social knowledge production and applica- tion activities. It did this by classifying the types of organizations in which social knowledge production and application funding programs were located. The following categorization was used in analyzing social knowledge production and application funding programs:

26 SCOPE OF THE SURVEY Associated with operating programs. These offices have pro- grammatic responsibility to administer federal programs—for exam- ple, the Food and Nutrition Service (Agriculture); the Economic De- velopment Administration (Commerce); the Office of Education (Health, Education, and Welfare); the National Park Service (Interior); and the Federal Highway Administration (Transportation). Associated with policy-making offices. These offices frequently have oversight responsibility for a number of federal programs or have staff advisory responsibility for nonprogrammatic federal policies—for example, the Office of the Assistant Secretary for Planning and Evalua- tion (Health, Education, and Welfare); the Advisory Commission on Intergovernmental Relations; the U.S. Commission on Civil Rights; and the Council of Economic Advisers. Associated with agencies whose primary mission is R&D funding. For example, the National Institute of Education (Health, Education, and Welfare); the Agricultural Research Service (Agriculture); the Na- tional Center for Health Services Research (Health, Education, and Welfare); and the National Science Foundation. Associated with agencies whose primary mission is the collection and/or analysis of statistics. For example, the Statistical Reporting Service (Agriculture); the Bureau of the Census (Commerce); the Na- tional Center for Education Statistics (Health, Education, and Wel- fare); and the National Criminal Justice Information and Statistical Service (Law Enforcement Assistance Administration/Justice). CLASSIFICATION BY GOALS OR AUDIENCE The Study Project also sought to find out more about the goals and audiences of social knowledge production and application activities. To describe these goals and audiences, the Study Project used pro- grams as the unit of analysis and devised a classification of the objec- tives of the funding agencies: o the improvement of federal programs; o the improvement of federal policies; o the creation and provision of knowledge and developed programs or materials to nonfederal audiences—knowledge for third parties;

The Framework 27 o the general advancement of knowledge concerning individual and social behavior without specific concern for application; and o the collection and analysis of statistics. This categorization proved to be substantially more difficult and judgmental than the categorization according to organizational location and function. For example, the National Institute of Mental Health (NIMH) supports considerable basic disciplinary research as well as research centered on a variety of social and mental health problems. It also supports a significant amount of research that is intended to be useful to practitioners in community mental health centers, social ser- vice agencies, and third parties generally. Although some of the rhetoric surrounding the NIMH program emphasizes the latter activity, the Study Project concluded the predominant function of NIMH to be the advancement of knowledge and classified the agency accordingly." In agencies with more than one goal or audience, the Study Project sought to classify the predominant one. ADDITIONAL SOURCES OF INFORMATION RESEARCH AND DEVELOPMENT—GENERAL National Science Foundation Surveys of Science Resources Series (publication list available from National Science Foundation, Washington, D.C. 20550). Research and Development Report, vols. I. 2, and 3. published by the American Associ- ation for the Advancement of Science (available from AAAS. 1776 Massachusetts Av- enue, N.W., Washington, D.C. 20036). Science Indicators 1976, Report of the National Science Board, U.S. Government Print- ing Office. Washington. D.C., 1977. U.S. Congress, House, Committee on Science and Technology, Subcommittee on Domestic and International Science Planning and Analysis. Statutory Provisions Re- lated to Federal Research and Development, vols. I and 2, 94th Congress. 2nd ses- sion. DEMONSTRATIONS Baer, W. S.. et z\.. Analysis of Federally Funded Demonstration Projects: Final Report, R-1926-DOC. Rand Corp., Santa Monica, Cal., April 1976. Glennan, T. K.. et al., The Role of Demonstrations in Federal R&D Policy, Rand Report R-2288-OTA, Rand Corp., Santa Monica. Cal.. May 1978 (report prepared for the Office of Technology Assessment and available from OTA). "Clearly, the Study Project would have gained added information on goals and audi- ences by carrying the analysis to the level of individual projects if this had been feasible.

28 SCOPE OF THE SURVEY Hayes, C. D., "Toward a Conceptualization of the Function of Demonstrations," in National Research Council, Case Studies in the Management of Social R&D: Selected Issues, edited by T. K. Glennan, National Academy of Sciences, Washington, D.C., 1978. Riecken, H. W., and Robert F. Boruch, eds.. Social Experimentation: A Method for Planning and Evaluating Social Intervention, Academic Press, New York, 1974. PROGRAM EVALUATION General Accounting Office. Federal Program Evaluations, A Directory for the Congress, 1976 Sourcebook Series, U.S. Government Printing Office, Washington, D.C., 1976. Schmidt, R. E., et al.. Serving the Federal Evaluation Market, Urban Institute, Washington, D.C., 1977 (for a list of Urban Institute papers on program evaluation, write Urban Institute, 2100 M Street, N.W., Washington, D.C. 20037). A Symposium on the Use of Evaluation by Federal Agencies, vol. I and 2, a report of the Metrek Division, Mitre Corp., 1820 Dolley Madison Boulevard, McLean. Va. 22101. STATISTICAL COLLECTION National Research Council, Setting Statistical Priorities, Report of the Panel on Meth- odology for Statistical Priorities, Committee on National Statistics, National Academy of Sciences, Washington, D.C., 1976. Office of Management and Budget, Federal Statistical Directory, 25th ed., U.S. Gov- ernment Printing Office, Washington, D.C., 1976. Office of Management and Budget, Statistical Services of the United States Government, revised ed., U.S. Government Printing Office, 1975. Statistical Reporter, monthly publication of the Office of Federal Statistical Policy and Standards, Department of Commerce (single copies or subscriptions available from the U.S. Government Printing Office). DISSEMINATION Directory of Federal Technology Transfer, 2nd ed.. Executive Office of the President, Federal Coordinating Council for Science, Engineering and Technology, U.S. Gov- ernment Printing Office, June 1977. Putting Knowledge to Use: A Distillation of the Literature Regarding Transfer and Change, Human Interaction Research Institute, 10889 Wilshire Boulevard, Los Angeles, Cal. 90024. U.S. Congress, House, Committee on Science and Technology, Subcommittee on Domestic and International Scientific Planning and Analysis, Intergovernmental Dis- semination of Federal Research and Development Results, 94th Congress, 1st session.