3
Improving the Current System in the Short Term

Improving statistical practices takes time and money, but the Division of Science Resources Statistics (SRS) of the National Science Foundation (NSF), like other federal agencies, has a limited budget for its statistical activities, most of which must be devoted to accomplishing the day-to-day tasks associated with collecting, processing, and disseminating data to users. The panel therefore advises SRS to make only a few modest improvements to the current system of surveys in the short term (over the next four years) and to spend most of its resources for improvement following a longer term strategy of adopting new technology to extract data automatically from administrative databases that we expect to become available in the next few years.

Our short-term recommendations and the rationales for them are detailed in this chapter. Longer term recommendations follow in Chapter 4. In Chapter 6, the pathway for leading to these changes is described, beginning with the incremental improvements outlined here and progressing to the stage at which the current surveys can be replaced with a system based on administrative records.

Our recommendations for short-term improvements in the two surveys address five major areas: (1) reform of the taxonomies of fields of science and engineering (S&E), (2) SRS’s relationships with the responding agencies, (3) the adequacy of SRS’s data collection authorities, (4) the timeliness of data collection and reporting, and (5) survey technology. Each is discussed below.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 31
3 Improving the Current System in the Short Term I mproving statistical practices takes time and money, but the Division of Science Resources Statistics (SRS) of the National Science Foundation (NSF), like other federal agencies, has a limited budget for its statistical activities, most of which must be devoted to accomplishing the day-to-day tasks associated with collecting, processing, and disseminating data to users. The panel therefore advises SRS to make only a few modest improve- ments to the current system of surveys in the short term (over the next four years) and to spend most of its resources for improvement following a longer term strategy of adopting new technology to extract data automati- cally from administrative databases that we expect to become available in the next few years. Our short-term recommendations and the rationales for them are detailed in this chapter. Longer term recommendations follow in Chapter 4. In Chapter 6, the pathway for leading to these changes is described, begin- ning with the incremental improvements outlined here and progressing to the stage at which the current surveys can be replaced with a system based on administrative records. Our recommendations for short-term improvements in the two surveys address five major areas: (1) reform of the taxonomies of fields of science and engineering (S&E), (2) SRS’s relationships with the responding agen- cies, (3) the adequacy of SRS’s data collection authorities, (4) the timeli- ness of data collection and reporting, and (5) survey technology. Each is discussed below. 

OCR for page 31
 DATA ON FEDERAL RESEARCH AND DEVELOPMENT INVESTMENTS REFORM OF THE TAXONOMIES OF FIELDS OF SCIENCE AND ENGINEERING Practical Aspects of the Selection of a Taxonomy For a taxonomy to be useful for research and development (R&D) data, it must not only include categories that are meaningful to users of the data but also be suitable for data collection purposes. Users should expe- rience the categories as being reflective of their reality. At the same time, the taxonomy should reflect the way organizations providing the data are organized and staffed for the support and the performance of R&D. The panel is aware that not all supporters and performers of R&D clas- sify their activities in the same way and that this fact introduces a certain tension in deciding whether a taxonomy is “suitable.” For example, a mis- sion agency might classify a bioinformatics research activity as life science, whereas a funded performer might report the same activity as computer science. For this reason, and others, no single taxonomy will satisfy all. However, for purposes of collecting data on research and development statistics in a consistent manner across federal government agencies, it is necessary to establish a common taxonomy that will be useful to the largest number of data providers and users. In the longer term, a provision can be made for tailoring structures that meet the specific needs of providers and users by flexibly categorizing administrative records as outlined in Chapter 5. Box 3-1 lays out some general issues in creating a taxonomy of activity in research and development. SRS has made a good start on identifying and analyzing issues with the taxonomy now used. As described in Chapter 2, SRS has recently held workshops and conducted interviews with federal agency respondents to delve into issues surrounding the taxonomy used in the federal funds survey. In these workshops, the agencies have provided SRS with practical com- ments from their perspective as respondents. For example, some agencies have reported that their databases do not collect or store data in a way that would permit easy reporting by fields of S&E, and that the current taxonomy is not relevant to the way in which they manage and track their R&D programs (Macro International, 2008, p. 8). Users of the federal funds data who participated in various discus- sions over the past few years, including the workshop held by our panel, seem nearly unanimous in their view that the current taxonomy has short- comings. Among other criticisms, users point to the fact that new fields of science are not included, there is no way to report interdisciplinary research, and more detailed breakdowns are needed. These criticisms focus on the need for additional categories in the current taxonomy, rather than a wholesale change in the taxonomy itself. The same users caution against

OCR for page 31
 IMPROVING THE CURRENT SYSTEM IN THE SHORT TERM BOX 3-1 Issues in Creating a Taxonomy of Activity in Research and Development A taxonomy groups together entities according to their common char- acteristics. The English word derives from the Greek τάξις, taxis (mean- ing order or arrangement), and νόμος, nomos (meaning law or science). Used originally for biological classification of living and extinct beings, the term now refers to any classification scheme. There is no perfect or ideal classification, Lenoir and Beghtol (2004) caution, only better or worse schemes for particular purposes, contents, users, and contexts. Units and levels of analysis also vary by specificity or “granularity,” from shallow analysis dividing entities into large aggregates to deep analysis subdividing them into smaller units. Moreover, taxonomies are never finished and so become outdated. Since the late 19th and early 20th centuries, taxonomies of knowl- edge in the West have been dominated by a system of disciplinarity that distinguishes domains of specialized inquiry by their particular subjects and objects of study. The federal funds survey, for example, categorizes fields of science and engineering into eight major types: (1) life sci- ences, (2) psychology, (3) physical sciences, (4) environmental sciences, (5) mathematics and computer sciences, (6) engineering, (7) social sci- ences, and (8) other sciences “not elsewhere classified” (n.e.c.). Each major type, in turn, contains subtypes. Physical sciences, for example, encompass astronomy, chemistry, and physics. Over the course of the 20th century, the scope and size of disciplinary domains has expanded with the growing number of specialties and subfields, turning many of them into large groups of disciplines that encompass a broad range of identifiable and in some cases autonomous specialties. The social sciences illustrate the challenge of classification. The mainstream disciplines of social sciences are anthropology, economics, political science, psychology, and sociology. Yet, Neil Smelser advises, describing social sciences solely with reference to the “big five” disci- plines distorts reality in two ways (Smelser, 2003). First, under those headings, subareas of investigation rely on variables and explanations outside the commonly understood scope of social sciences. Geopolitics, sociobiology, behavioral genetics, and behavioral neuroscience all appeal to nonsocial and nonpsychological explanatory variables and explana- tions. Second, another range of disciplines could be labeled behavioral and social scientific, although not entirely so. Demography might be con- sidered a separate social science or part of sociology, economics, and anthropology. Archaeology might be classed as part of anthropology or continued

OCR for page 31
 DATA ON FEDERAL RESEARCH AND DEVELOPMENT INVESTMENTS BOX 3-1 Continued an independent social science. Geography, history, psychiatry, law, and linguistics present similar complications for taxonomy. So do relations with the intersecting fields of genetics, behavior, and society; behavioral and cognitive neurosciences; psychiatry; health; gender studies; reli- gious studies; expressive forms; environmental/­ecological sciences and technology studies; area and international studies; and urban studies, planning, and public policy. Strict assignment to one category of inquiry or another would vary according to the criteria used (Smelser, 2003). The growth of multidisciplinary and interdisciplinary modes of research has further complicated classification, a challenge amply evident in the n.e.c. category of the federal funds survey. The category of “not else- where classified” is large and amorphous and lumps together a plurality of developments, including new subfields, single-discipline projects for which a separate field has not been assigned, emergent fields, estab- lished interdisciplinary fields, cross-cutting initiatives, “problem-focus” areas of research, and miscellaneous “other.” It also fails to discrimi- nate multidisciplinary juxtapositions of different disciplinary approaches from interdisciplinary approaches that integrate separate disciplinary data, methods, tools, concepts, and theories, as well as comprehensive transdisciplinary frameworks that posit a new conceptual synthesis or theoretical framework (Klein, 2009). Authors and users of taxonomies also have different views of how they should be constructed. Birger H jørland distinguishes four fundamental methods of classification: empiri- cism (based on observations and inductions), rationalism (principles of pure reason, deductions), historicism (context and development), and pragmatism (analysis of goals, values, and consequences in both sub- ject and object) (Hjørland, 2008). The greater plurality and complexity of knowledge today has resulted in three major views of current taxonomy. A first group continues to use standardized classification schemes based on a limited number of broadly aggregated categories, lumping together diverse practices. A second group advocates widening the broad cat- egories, with the possibilities of adding a few more major categories and using a “hierarchy of preference” approach that allows splitting into highly aggregated (2-3 digit) levels and distributing percentages of emphasis and time into more than one discipline. A third group supports a more open, flexible, dynamic, and transactional approach, depicting research in a network representation that allows for greater granularity and employs techniques of semantic mapping, web- and text-mining, controlled thesauri, and tag clouds, and Internet-based, user generated taxonomies (“folksonomies”). The work of the second and third groups inform this study and its recommendations.

OCR for page 31
 IMPROVING THE CURRENT SYSTEM IN THE SHORT TERM making significant changes to the taxonomy, for fear of introducing discon- tinuity in the historical data it provides. Associating a government-wide taxonomy for R&D funding with the disciplinary structure of academia is no easy matter. The academic com- munity is most often organized around departments for the purpose of instruction, but research activities are often multidisciplinary, engaging more than one discipline and department. Federal mission agencies tend to organize their applied R&D activities around broad national challenges, such as energy efficiency or space exploration, or around technology areas, such as nanotechnology, biotechnology, or information technology. Thus, updating the taxonomy to make it more relevant for communicating with the academic community might work for basic research, but it will not contribute much when classifying applied R&D programs and projects. One promising approach is to use federal agency records to quantify the extent of interdisciplinary research. Under the auspices of the National Academies’ Keck Futures Initiative, a team from the Georgia Institute of Technology performed keyword searches of awards databases at NSF, the National Institutes of Health, and the Department of Energy to estimate the amount of interdisciplinary research being supported. Over the time period covered by the research (1999-2003), the team found apparent growth in interdisciplinary research (Yasitis et al., 2004). Using agency contract and award information as a source of information on fields of S&E is explored at greater length in Chapter 4. SRS currently uses slightly different taxonomies for the federal funds survey and the academic R&D expenditures survey, and the taxonomies utilized in these surveys have differences from the standard taxonomy in OMB Directive 16 (see Table 2-1). Ideally, classification of R&D funding by disciplines as reported by the supporters of that R&D should be the same as reported by the recipients of those funds. Uniformity of definitions and classifications between surveys is one means of achieving that goal. Practically, some differences in taxonomies are likely to persist because it may be difficult to collect the same level of detail from different respon- dent groups, or there may be differences in the uses to which the taxonomic information is put. That said, because of the need to compare the results of the two surveys, SRS would benefit from using the same taxonomy for the federal funds survey and the academic R&D expenditures survey. Alternative Taxonomies The previous National Research Council (NRC) studies discussed in Chapter 2 have laid out the problems associated with an aging taxonomy of R&D spending. Our panel was specifically asked to make a recommenda- tion about updating Directive 16 issued by the U.S. Office of Management

OCR for page 31
 DATA ON FEDERAL RESEARCH AND DEVELOPMENT INVESTMENTS and Budget (OMB) in 1978—the taxonomy that was the original basis of the classification of fields of S&E for the federal funds survey. Several alternative taxonomies could replace or supplement the OMB Directive 16 taxonomy. In considering options for a new taxonomy, the panel looked not only at OMB Directive 16, but also at taxonomies used internationally. Two relevant international standards are “Recommenda- tions Concerning the International Standardization of Statistics on Science and Technology” of the United Nations Educational, Scientific, and Cul- tural Organization (UNESCO) (United Nations Educational, Scientific, and Cultural Organization, 1978) and “Proposed Standard Practice for Surveys on Research and Experimental Development,” called the Frascati manual, of the Organisation for Economic Co-operation and Development (OECD) (Organisation for Economic Co-operation and Development, 2002). The UNESCO-recommended fields of S&E are shown in Box 3-2. Like OMB Directive 16, the UNESCO fields have not been updated since 1978. However, these old taxonomies were considered when the international community developed the more recent OECD Frascati manual classifica- tion, which uses the most current categories in its taxonomy of fields of S&E. As shown in Box 3-3, the Frascati manual classification provides a use- ful system for organizing subactivities under the major fields. The Frascati manual recognizes that need for disaggregation within the major fields may differ from country to country, stating in the instructions, “While the major fields of science and technology are clearly defined, the level of disaggrega- tion within each component field is left to each country” (Organisation for Economic Co-operation and Development, 2002, p. 66). Adoption of the Frascati taxonomy is not a full solution, however. Although newer than Directive 16, the Frascati manual suffers from some of the same limita- tions, in that it fails to accommodate multidisciplinary fields and has no procedures for periodic updating. Two classification systems, one in widespread use and the other in devel- opment, have the advantage of permitting portrayal of multidisciplinary and interdisciplinary fields. Both the U.S. Department of Education’s Classifica- tion of Instructional Programs (CIP) and the NRC’s Taxonomy of Fields (see Box 3-4) are designed to support the collection of information from educational institutions, yet they may have wider application and attributes that commend them for consideration as alternative classification structures to OMB Directive 16. The CIP was originally developed by the National Center for Education Statistics (NCES) in 1980 to provide a taxonomy to support the tracking,

OCR for page 31
 IMPROVING THE CURRENT SYSTEM IN THE SHORT TERM BOX 3-2 UNESCO Fields of Science and Technology Natural sciences, including astronomy, bacteriology, biochemistry, biology, botany, chemistry, computer sciences, entomology, geology, geophysics, mathematics, meteorology, mineralogy, physical geogra- phy, physics, zoology, other allied subjects. Engineering and technology, including engineering proper, such as chemical, civil, electrical, and mechanical engineering, and specialized subdivisions of these; forest products; applied sciences such as geodesy, industrial chemistry, etc.; architecture; the science and technology of food production; specialized technologies or interdisciplinary fields, e.g., systems analysis, metallurgy, mining, textile technology, other allied subjects. Medical sciences, including anatomy, dentistry, medicine, nursing, obstetrics, optometry, osteopathy, pharmacy, physiotherapy, public health, other allied subjects. Agricultural sciences, including agronomy, animal husbandry, fisheries, forestry, horticulture, veterinary medicine, other allied subjects. Social sciences, anthropology (social and cultural) and ethnology, demography, economics, education and training, geography (human, economic, and social), law, linguistics, management, political science, psychology, sociology, organization and methods, miscellaneous social sciences, and interdisciplinary, methodological, and historical science and technology activities relating to subjects in this group. Humanities, arts (history of the arts and art criticism, excluding artistic research of any kind), languages (ancient and modern languages and literature), philosophy (including the history of science and technology), prehistory and history, together with auxiliary historical disciplines, such as archaeology, numismatics, paleography, etc., religion, other fields and subjects pertaining to the humanities, and interdisciplinary, methodologi- cal, historical, and other science and technology activities relating to the subjects in this group. SOURCE: Adapted from United Nations Educational, Scientific, and C ultural Organization (1979).

OCR for page 31
 DATA ON FEDERAL RESEARCH AND DEVELOPMENT INVESTMENTS BOX 3-3 Frascati Manual Taxonomy of Fields of Science and Technology 1. Natural Sciences 1.1. athematics and computer sciences [mathematics and other M allied fields: Computer sciences and other allied subjects (soft- ware development only; hardware development should be classi- fied in the engineering fields)] 1.2. hysical sciences (astronomy and space sciences, physics, other P allied subjects) 1.3. hemical sciences (chemistry, other allied subjects) C 1.4. arth and related environmental sciences (geology, geophysics, E mineralogy, physical geography and other geosciences, meteorol- ogy and other atmospheric sciences, including climatic research, oceanography, vulcanology, paleoecology, other allied sciences) 1.5. iological sciences (biology, botany, bacteriology, microbiology, B zoology, entomology, genetics, biochemistry, biophysics, other allied sciences, excluding clinical and veterinary sciences) 2. Engineering and Technology 2.1. ivil engineering (architecture engineering, building science and C engineering, construction engineering, municipal and structural engineering and other allied subjects) 2.2. lectrical engineering, electronics [electrical engineering, elec- E tronics, communication engineering and systems, computer engi- neering (hardware only) and other allied subjects] 2.3. ther engineering sciences (such as chemical, aeronautical and O space, mechanical, metallurgical and materials engineering, and their specialized subdivisions; forest products; applied sciences, such as geodesy, industrial chemistry, etc.; the science and tech- nology of food production; specialized technologies of interdisci- plinary fields, e.g., systems analysis, metallurgy, mining, textile technology and other allied subjects) 3. Medical Sciences 3.1. asic medicine (anatomy, cytology, physiology, genetics, pharmacy, B pharmacology, toxicology, immunology and immunohematology, clinical chemistry, clinical microbiology, pathology) 3.2. linical medicine (anesthesiology, pediatrics, obstetrics and gyne- C cology, internal medicine, surgery, dentistry, neurology, psychiatry, radiology, therapeutics, otorhinolaryngology, ophthalmology)

OCR for page 31
 IMPROVING THE CURRENT SYSTEM IN THE SHORT TERM 3.3. ealth sciences (public health services, social medicine, hygiene, H nursing, epidemiology) 4. Agricultural Sciences 4.1. griculture, forestry, fisheries and allied sciences (agronomy, A animal husbandry, fisheries, forestry, horticulture, other allied subjects) 4.2. eterinary medicine V 5. Social Sciences 5.1. sychology P 5.2. conomics E 5.3. ducational sciences (education and training and other allied E subjects) 5.4. ther social sciences [anthropology (social and cultural) and eth- O nology, demography, geography (human, economic and social), town and country planning, management, law, linguistics, political science, sociology, organization and methods, miscellaneous social sciences and interdisciplinary, methodological, and historical science and technology activities relating to subjects in this group. Physical anthropology, physical geography and psychophysiology should normally be classified with the natural sciences] 6. Humanities 6.1. istory (history, prehistory and history, together with auxiliary his- H torical disciplines such as archaeology, numismatics, paleography, genealogy, etc.) 6.2. anguages and literature (ancient and modern) L 6.3. ther humanities [philosophy (including the history of science O and technology), arts, history of art, art criticism, painting, sculp- ture, musicology, dramatic art excluding artistic research of any kind, religion, theology, other fields and subjects pertaining to the humanities, methodological, historical and other science and technology activities relating to the subjects in this group] SOURCE: Organisation for Economic Co-operation and Development (2002, p. 67).

OCR for page 31
0 DATA ON FEDERAL RESEARCH AND DEVELOPMENT INVESTMENTS BOX 3-4 National Research Council Taxonomy of Fields Life Sciences Animal Sciences Biochemistry, Biophysics, and Structural Biology iology/­Integrated Biology/­Integrated Biomedical Sciences B (Note: Use this field only if the degree field is not specialized.) Cell and Developmental Biology Ecology and Evolutionary Biology Entomology Food Science Forestry and Forest Sciences Genetics and Genomics Immunology and Infectious Disease Kinesiology Microbiology Neuroscience and Neurobiology Nursing Nutrition Pharmacology, Toxicology, and Environmental Health Physiology Plant Sciences Public Health Emerging Fields: Bioinformatics Biotechnology Systems Biology Physical Sciences, Mathematics, and Engineering Applied Mathematics Astrophysics and Astronomy Chemistry Computer Sciences Earth Sciences Mathematics Oceanography, Atmospheric Sciences, and Meteorology Physics Statistics and Probability Aerospace Engineering Biomedical Engineering and Bioengineering Chemical Engineering

OCR for page 31
 IMPROVING THE CURRENT SYSTEM IN THE SHORT TERM Civil and Environmental Engineering Computer Engineering Electrical and Computer Engineering Engineering Science and Materials (not elsewhere classified) Materials Science and Engineering Mechanical Engineering Operations Research, Systems Engineering, and Industrial Engineering Emerging Fields: Computational Engineering Information Science Nanoscience and Nanotechnology Nuclear Engineering Social and Behavioral Sciences Agricultural and Resource Economics Anthropology Communication Economics Geography Linguistics Political Science Psychology Public Affairs, Public Policy, and Public Administration Sociology Emerging Fields: Criminology and Criminal Justice Science and Technology Studies Urban Studies and Planning Arts and Humanities American Studies Classics Comparative Literature English Language and Literature French and Francophone Language and Literature German Language and Literature continued

OCR for page 31
 DATA ON FEDERAL RESEARCH AND DEVELOPMENT INVESTMENTS BOX 3-4 Continued Language, Societies, and Cultures History History of Art, Architecture, and Archaeology Music (except performance) Philosophy Religion Spanish and Portuguese Language and Literature Theatre and Performance Studies Emerging Fields: Feminist, Gender, and Sexuality Studies Film Studies Race, Ethnicity, and Post-Colonial Studies Rhetoric and Composition SOURCE: Taxonomy of Fields (http:/­/­sites.nationalacademies. org/­PGA/­Resdoc/­PGA_044521). assessment, and reporting of fields of study and program completion.1 This tracking is accomplished through the Integrated Postsecondary Edu- cation Data System (IPEDS) completions survey, which is submitted by postsecondary institutions that receive Title IV federal funding. This survey summarizes the number of completions by field of study across the full spec- trum of school offerings. The CIP provides the lists of the field of study. Although the CIP includes fields other than S&E, one obvious advan- tage of associating the fields of S&E taxonomy for reporting R&D data to the CIP is that it is frequently updated to stay current with educational offerings. The CIP was updated in 1985, 1990, and 2000. NCES is cur- rently updating the 2000 CIP with the goal of releasing an updated version in June 2009. These updates are based on rigorous procedures.2 A disad- 1 Memorandum, Michelle Coon, “Update of the Classification of Instructional Programs (CIP),” National Center for Education Statistics, August 11, 2008. 2 One process involves examining data from the IPEDS completion survey to identify insti- tutions that produced the largest number of completions for a specific CIP code. The course catalogs for these institutions are then mapped onto the existing CIP and examined to find instructional programs that did not fit into the existing CIP. Stakeholders and coordinators are involved in this process to identify instructional programs that are not currently represented in

OCR for page 31
 IMPROVING THE CURRENT SYSTEM IN THE SHORT TERM vantage of using the IPEDS process to drive updates of the taxonomy of fields of S&E in the SRS surveys is that it could lead to unanticipated and undesirable changes in the reporting of R&D support. If OMB and SRS were to more closely align the federal funds fields with the CIP fields, some modifications of the latter would be in order. The CIP taxonomy includes a greater number of categories than are needed for the SRS surveys, so the SRS taxonomy might need to select and combine CIP categories. The burden of this task should be minimal, since SRS currently provides respondents to another of its R&D surveys—the academic R&D expenditures survey—with a cross-walk between the CIP taxonomy and the taxonomy of fields of S&E. The NRC Taxonomy of Fields is the most recently developed tax- onomy for classifying fields of S&E. Developed to support collection of data on research doctorate programs, it is based on the classification of fields used in the Doctorate Records File, which is also maintained by SRS. The criteria for inclusion of a field in this taxonomy are tied to doctoral program production—that is, fields are included that have produced a total of at least 500 Ph.Ds in the past 5 years with participation by at least 25 universities. One advantage of the NRC classification structure is that it attempts, whenever possible, to specifically include multidisciplinary, cross-disciplinary, and transdisciplinary fields and to make provision for emerging fields exhibit- ing significant growth. Thus, interdisciplinary fields—such as neuroscience, biomedical engineering, and American studies—are included, and emerg- ing fields—such as bioinformatics, biotechnology, systems biology, compu- tational engineering, information science, science and technology studies, feminist studies, race and ethnic studies, and rhetoric and composition—also make the list (National Research Council, 2003, pp. 19-20; see also National Research Council, 2006). The NRC system was designed to depict academic research programs, so it would fit well with the federal support survey. However, it was not designed to support collection of R&D data, so it would have to be modi- fied to serve as a general-purpose R&D taxonomy. Need for Historical Continuity The need to modernize the taxonomy of fields of S&E must be balanced with the need for historical continuity of the data series that are based on the existing taxonomy. Data series based on consistent definitions of fields the CIP, and non-NCES data sources, such as the College Board’s Annual Survey of Colleges and SRS’s Survey of Earned Doctorates, are reviewed in order to identify new instructional programs. Finally, input from federal agencies that use IPEDS completions data is solicited.

OCR for page 31
 DATA ON FEDERAL RESEARCH AND DEVELOPMENT INVESTMENTS of S&E go back many years: Several of the series published by field in Sci­ ence and Engineering Indicators (National Science Board, 2006) have been consistently published since the early 1980s. Because of this consistency, it is possible to trace, for example, the dramatic increase in federal obligations for life science fields over the past four decades. The need for historical continuity suggests caution in proposing changes in the fields of S&E taxonomy. Changes should be incremental, and care should be taken to either carry forward the current taxonomy, even as data are published using a new taxonomy, or to develop cross-walks between any new taxonomy and the old one so as to minimize disruption to the historical data series. Abrupt changes to the taxonomy could confuse data users. Recommendation 3-1: The Division of Science Resources Statistics, in the near term, should make the changes necessary to improve the comparability of the federal funds taxonomy and the taxonomy for the academic research and development expenditures survey and should focus on the medium- and long-term changes the panel recommends. SRS’S RELATIONSHIPS WITH RESPONDING AGENCIES SRS collaborates with the responding agencies through regular work- shops and has initiated several efforts to study reporting issues. The 2008 report by Macro International is the most recent example. In that report, Macro International staff contacted respondents by email to solicit informa- tion and schedule face-to-face meetings for in-depth interviews. Although the information provided in the report is of high quality and ultimately very useful, the contracted investigators faced many challenges with regard to access to the agency respondents. The Macro International staff reported difficulties in identifying appropriate high-level personnel in each agency and in obtaining cooperation in some. Furthermore, the panel observes that SRS would have benefited from the social professional capital that can be built through an ongoing program of having its own staff conduct structured in-depth interviews and discussions with agency staff about the surveys, data, and reporting issues. Building relationships between SRS staff and agency respondents is critical to ensuring goodwill, understanding how the agency internal reporting systems interface with the survey question- naire, improving reporting, and ultimately obtaining high-quality data. The panel concludes that SRS would benefit from building better direct relationships with agencies responding to the federal funds and federal support surveys. The core competency and management capacity to maintain relationships with other agencies is an inherent governmental

OCR for page 31
 IMPROVING THE CURRENT SYSTEM IN THE SHORT TERM responsibility, and SRS loses an opportunity to improve practices when it relies on contractors to facilitate and manage these relationships. Senior- level SRS staff involvement with responding agencies should go a long way toward demonstrating to agencies that SRS considers the data to be important and that it values their input. Recommendation 3-2: The Division of Science Resources Statistics should devote staff and resources to managing relationships with responding agencies directly, relying less on contractors to maintain those relationships. ADEQUACY OF THE DATA COLLECTION AUTHORITIES We considered whether SRS needs more explicit statutory authority to collect data from federal agencies in a way that would improve reporting. The panel notes that agencies are already required to respond to the federal funds and federal support surveys (Congressional Research Service, 2000, p. 1). The panel concludes that restating SRS’s statutory authority would be unlikely to affect how agencies maintain their data, so difficulties in responding to the questionnaire would continue. Instead, SRS would benefit from pursuing better relationships with responding agencies, reminding them regularly of the importance of the survey and the usefulness of the data and working with them to make the survey forms as easy as possible to complete. This activity could be initi- ated as one of the major activities on the path toward modernized collec- tion of R&D spending data. A time schedule for this activity is suggested in Chapter 6. Some cosmetic changes in the collection forms might be helpful. For example, in contrast to its companion federal support survey and surveys that go out to the public, letters to respondents, survey forms, and instruc- tions for the federal funds survey do not provide respondents with any background on the law under which the data are collected, nor do they address the important uses of the information. It is important for SRS to regularly remind agencies about the authority they have and importance of the survey with each call for data. To accomplish this end, information about authorities and uses could be featured prominently on the survey form, in the instructions, and on the associated website. Recommendation 3-3: The Division of Science Resources Statistics should ensure that all questionnaires and email solicitations sent to respondents provide information on its data collection authority and on the important uses of the data.

OCR for page 31
 DATA ON FEDERAL RESEARCH AND DEVELOPMENT INVESTMENTS TIMELINESS OF DATA COLLECTION AND REPORTING The inability of SRS to release the data in a timely way limits the use- fulness of the federal funds survey and the federal support survey data to policy makers and interest groups. As previously noted, the delays result largely from the current practice of delaying publication of results until reports are received from all reporting agencies and the long delays of some agencies in assembling and forwarding the data. Some of the agencies that have delayed transmittal of their data in the past are quite large, and the publication of the estimates without their contribution could severely distort the data. For example, the two slowest reporters in 2006 were the National Institutes of Health and the National Aeronautics and Space Administration, together accounting for about one-third of federal R&D expenditures.3 SRS could consider alternatives to this practice. Alternatives include reporting incomplete information earlier, providing a preliminary report with imputation for late respondents, or designing a simple schedule to the form that could be completed more easily and quickly as the basis for publishing a preliminary report. The panel reluctantly concludes that SRS should stick with current practice. Each of these alternatives has pros and cons. Incomplete data might be misinterpreted, and preliminary totals would necessarily be on the low side because of missing data. Imputation for late respondents would introduce a new source of error and could make the data less accurate. Some data users hold that the potential for increased error due to omission or imputation is less desirable than more timely publication. Developing a new and simpler schedule to elicit early reporting would be a major new activity for SRS. In consideration of these issues, the best course of action at this time would be to continue the current policy of delaying publica- tion until all reports are received, aggressively pursue better relationships with the agencies to encourage more timely response, and devote scarce resources to the improvements the panel recommends. Recommendation 3-4: The Division of Science Resources Statistics (SRS) should maintain its current approach to data reporting, which is to wait for receipt of reports from all respondents before publishing the data. SRS should continue to report complete data without imputa- tion for missing reports and data elements. The agency should focus on working directly with respondents to find ways to improve the timeli- ness of their response to the surveys. 3 Presentation of John Jankowski, SRS, the Workshop on Modernizing the Infrastructure of the National Science Foundation Federal Funds Survey, September 5, 2008.

OCR for page 31
 IMPROVING THE CURRENT SYSTEM IN THE SHORT TERM SURVEY TECHNOLOGY Both the federal funds survey and the federal support survey have been early adopters of a web-based reporting system to facilitate response. However, not all respondents use the web to report, and the current web forms do not tap the full potential of current web-based survey methods. For example, at present the focus is on filling in the blanks in the reporting instrument. Little attention has been paid to developing an online survey instrument that is user-friendly and reduces respondent burden by using skip patterns and automatically populating responses. Some possibilities for process improvement include the automatic entry of zero values to subquestions when a response to one broad question is “no.” For example, if the agency respondent reports that it does not support research in non- profit institutions, the subsequent subquestions should automatically be populated with zero values, instead of requiring the respondent to enter a zero for each item. At the same time, SRS could consider tailoring the web survey for each agency based on prior knowledge and prior reports and with extensive col- laboration with the agency. One goal could be to include only those data items that could be expected to be included in the agency submissions with relative ease and accuracy and that would be familiar to the agencies. This tailoring of the collection of data does not have to be applied to all agencies at once; SRS could begin with the larger ones. Recommendation 3-5: The Division of Science Resources Statistics should invest in creating more user-friendly web surveys, possibly tailored to each agency, to replace current web versions of the paper surveys.