2
History and Assessment of SRS

Evolution of SRS Data Programs

Federal data collection programs date from the first decennial census, required by the U.S. Constitution and conducted in 1790. Federal statistics expanded during the nineteenth century to include agricultural statistics beginning in 1840, income statistics in 1866, education statistics in 1867, and labor statistics beginning in 1884. In the twentieth century, the federal statistical system evolved further. The Bureau of the Census and the Bureau of Labor Statistics (BLS) became bureaus in the new Department of Commerce and Labor in 1903 and BLS was later transferred to the new Department of Labor in 1913. The collection of health statistics was added to the federal statistical portfolio in the early twentieth century and later evolved into the National Center for Health Statistics. Since World War II, policy concerns have led to the creation of the Bureau of Economic Analysis (as the Office of Business Economics in 1953), the Energy Information Agency (1977), the Bureau of Justice Statistics (1979), and the Bureau of Transportation Statistics (1991) (NRC 1997).

The federal collection and dissemination of information on science and technology resources has grown with the federal role in our nation's research and development (R&D) activities. The National Science Foundation Act of 1950 created the Foundation that year with a mission "to promote the progress of science; to advance the national health, prosperity, and welfare; and to secure the national defense." That Act also made NSF a statistical agency. As amended, it requires NSF:

to provide a central clearinghouse for the collection, interpretation, and analysis of data on scientific and engineering resources, and to provide a source of information for policy formulation by other agencies of the federal government.

These activities were initially carried out by staff in NSF's Office of the Director until the Division of Science Resources Studies (SRS) was created to carry out this mandate. SRS is a division within the NSF Directorate on Social, Economic, and Behavioral Sciences (SBE).



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 17
Measuring the Science and Engineering Enterprise: Priorities for the Division of Science Resources Studies 2 History and Assessment of SRS Evolution of SRS Data Programs Federal data collection programs date from the first decennial census, required by the U.S. Constitution and conducted in 1790. Federal statistics expanded during the nineteenth century to include agricultural statistics beginning in 1840, income statistics in 1866, education statistics in 1867, and labor statistics beginning in 1884. In the twentieth century, the federal statistical system evolved further. The Bureau of the Census and the Bureau of Labor Statistics (BLS) became bureaus in the new Department of Commerce and Labor in 1903 and BLS was later transferred to the new Department of Labor in 1913. The collection of health statistics was added to the federal statistical portfolio in the early twentieth century and later evolved into the National Center for Health Statistics. Since World War II, policy concerns have led to the creation of the Bureau of Economic Analysis (as the Office of Business Economics in 1953), the Energy Information Agency (1977), the Bureau of Justice Statistics (1979), and the Bureau of Transportation Statistics (1991) (NRC 1997). The federal collection and dissemination of information on science and technology resources has grown with the federal role in our nation's research and development (R&D) activities. The National Science Foundation Act of 1950 created the Foundation that year with a mission "to promote the progress of science; to advance the national health, prosperity, and welfare; and to secure the national defense." That Act also made NSF a statistical agency. As amended, it requires NSF: to provide a central clearinghouse for the collection, interpretation, and analysis of data on scientific and engineering resources, and to provide a source of information for policy formulation by other agencies of the federal government. These activities were initially carried out by staff in NSF's Office of the Director until the Division of Science Resources Studies (SRS) was created to carry out this mandate. SRS is a division within the NSF Directorate on Social, Economic, and Behavioral Sciences (SBE).

OCR for page 17
Measuring the Science and Engineering Enterprise: Priorities for the Division of Science Resources Studies Today, SRS is organized functionally into data collection, analysis, and dissemination units. As shown in Figure 2-1, the SRS Director is assisted by a Deputy Director and a Chief Mathematical Statistician. The chief statistician is responsible for setting statistical standards for data collection and analysis across SRS activities and for working to enhance the performance of the division and its staff with respect to these standards. The Human Resources Statistics Program (HRS) and the Research and Development Statistics Program (RDS) have responsibility for primary data collection and acquisition within SRS on personnel, education, and R&D funding and performance. HRS and RDS also produce data tabulations and reports for dissemination to the public. Two additional units focus primarily on analysis activities. The Science and Engineering Indicators Program is responsible for coordinating SRS activities in support of the National Science Boards' biennial publication of Science and Engineering Indicators, as well as periodic reports on international comparisons for science and technology resources indicators. The Indicators program also administers a survey on public attitudes toward science and technology. The Integrated Studies Program has responsibility for producing special analyses on science and engineering resource topics of current significance to policymakers and the scientific community. Integrated Studies has, for example, recently published a series of "Issue Briefs" on a variety of personnel and R&D issues. The Integrated Studies Program has also been charged with developing a new survey on innovation. The division is supported by the Information Services Group that oversees publication of SRS reports, both in hard copy and via the SRS web site. Figure 2-1 Organization of the Division of Science Resources Studies.

OCR for page 17
Measuring the Science and Engineering Enterprise: Priorities for the Division of Science Resources Studies SRS Data Collection Programs The data collection, acquisition, and analysis activities that SRS administers have grown over the past half century to address the changing data and information needs of policymakers, managers, educators, and researchers in the science and technology policy arena. The science and engineering enterprise was significantly reshaped during and following World War II to include a substantial public role to meet the nation's defense needs, achieve national science and technology goals, and improve the nation's health. To make informed decisions within this new context, federal policymakers required data on and analysis of science and engineering resources issues. To meet these needs, NSF launched its first data collection activities on science and engineering personnel and R&D funding in the early 1950s. As policymakers confronted new issues and problems, the portfolio of data collection activities in SRS grew. Today, SRS data are widely used, yet mostly to address specific issues in narrow topic areas rather than broader questions concerning the level and allocation of resources to the science and engineering enterprise. For SRS to meet these broader needs, it will need to make further efforts to link and integrate its data sets. Statistics on Science and Engineering Human Resources Collection of data on Ph.D. scientists and engineers first emerged from a conversation in 1945 between Vannevar Bush, then director of the U.S. Office of Scientific Research and Development, and M.H. Trytten, director of the National Research Council's Office of Scientific Personnel. Drs. Bush and Trytten believed that many of the Ph.D. scientists who had distinguished themselves during World War II had received their undergraduate degrees from small colleges. Trytten proposed to collect data on the undergraduate mentors of Ph.D. scientists to recognize "those devoted individuals who have done the outstanding work of teaching in the sciences in colleges . . .where the facilities at their disposal are notably meager" (NRC 1951). With initial funding from the Office of Naval Research, the NRC undertook this effort. Trytten's office found that while it was difficult to precisely identify professors who mentored Ph.D. scientists when they were undergraduates, useful data could be obtained on the colleges and universities that awarded bachelor's degrees and Ph.D.s to research doctorate holders. This effort led to the fielding, in 1957, of the NSF-funded Survey of Earned Doctorates (SED)—now a staple of the SRS Human Resources Statistics program. The SED collects data on the field of study, demographic characteristics, educational history, and future plans of new doctorate recipients. Additional data on graduate students and postdoctoral fellows has been collected since 1966 through the Survey of Graduate Students and Postdoctorates in Science and Engineering (GSPSE). This survey obtains data from colleges and universities on the number of enrolled graduate students in science and engineering by institution and their distribution by field, enrollment status (part-time/full-time), level (first year graduate student, other graduate student, postdoctoral fellow), financial support, and demographic characteristics. In the act establishing the National Science Foundation in 1950, the U.S. Congress recognized the importance of maintaining current data on the nation's pool of scientists and engineers. In the NSF Act, Congress directed the Foundation to maintain "a register of scientific and technical personnel and in other ways provide a central clearinghouse for information covering all scientific and technical personnel in the United States, including its territories and

OCR for page 17
Measuring the Science and Engineering Enterprise: Priorities for the Division of Science Resources Studies possessions." In response to this directive, the NSF worked in 1952 with the Federal Security Agency to develop the National Register of Scientific and Engineering Personnel (NRSEP) and then took over its maintenance in 1953. The purpose of the Register, in part, was to monitor the nation's supply of scientific and engineering personnel in preparation for a "national emergency." The NRSEP contained information about the location, fields of specialization, and work activities of scientists and engineers trained to different degree levels (NRC 1989). In the 1970s, NSF replaced the National Register with a data system based on periodic sample surveys of the nation's scientific and engineering personnel. In response to calls for a more comprehensive look at the nation's science and engineering personnel resources, NSF had, along with other agencies, sponsored a Postcensal Manpower Survey 1 in 1962. A decade later, in 1972, NSF sponsored another postcensal survey, the Professional, Technical, and Scientific Manpower Survey, to provide core data on scientists and engineers in lieu of the discontinued NRSEP. This was supplemented, in 1974, with a New Entrants Survey, designed to obtain data on those who had obtained science and engineering degrees in the United States since the 1970 Census. To cover the population of Ph.D.-level scientists and engineers, NSF also sponsored the first Survey of Doctorate Recipients (SDR) in 1973. In the 1980s, NSF created the Scientific and Technical Personnel Data System (STPDS), drawing on these three surveys. In the late 1980s, NSF asked the NRC's Committee on National Statistics to convene a study panel to explore the national need for and future characteristics of a data system on science and engineering personnel. In its 1989 report. Surveying the Nation's Scientists and Engineers, this panel recommended that the three personnel surveys be better integrated (NRC 1989). Following the recommendations of this report, SRS transformed STPDS into the integrated Scientists and Engineers Statistical Data System (SESTAT), which improved the data quality and integration of the three surveys by modifying sample design, data collection procedures, and questionnaire content for each survey. SESTAT draws data from the National Survey of College Graduates (a postcensal survey), the Survey of Recent College Graduates (a new entrants survey), and the SDR. Question wording on the SDR was revised during the 1993 expansion of the SDR questionnaire to improve comparability of SDR data with data in the rest of the system (Cox, Mitchell, and Moonesinghe 1998b). While the integration of the content and design of the three personnel surveys in SESTAT has been an impressive success, HRS could undertake further work to improve the comparability and integration of data in its entire set of surveys. For example, the SED and SDR both collect data from doctorate recipients on type of employer, work activities, number of dependents, and disability status but the questions and response categories differ substantively on the two surveys. The addition of a question on starting salary to the SED and a question on current field of science or engineering to the SDR would enhance the complementarity of the two data sets. Today, as shown in Box 2-1, the Human Resources Statistics Program (HRS) is responsible for data on graduate education for scientists and engineers and data on the nation's science and engineering workforce. HRS conducts the two annual surveys that track science and engineering enrollments and degrees at the graduate level: the Survey of Earned Doctorates (SED) and the Survey of Graduate Students and Postdoctorates in Science and Engineering (GSPSE). HRS also collects workforce statistics on scientists and 1   A postcensal survey uses the U.S. census as a sampling frame to then obtain more detailed data on respondents.

OCR for page 17
Measuring the Science and Engineering Enterprise: Priorities for the Division of Science Resources Studies engineers derived primarily from the three biennial personnel surveys that the SESTAT system draws on. Further, HRS acquires data on occupational employment from the U.S. Bureau of Labor Statistics and uses data from surveys conducted by the National Center for Education Statistics, including the Integrated Postsecondary Education Data System(IPEDS) Completions Survey, the National Postsecondary Student Aid Survey, and the National Postsecondary Faculty Survey. The Integrated Studies Program also acquires data on the immigration of scientists and engineers from the U.S. Immigration and Naturalization Service. Box 2-1. Principal Surveys in the SRS Human Resources Statistics Program. The annual Survey of Graduate Students and Postdoctorates in Science and Engineering collects data from all institutions in the United States offering postbaccalaureate programs in science and engineering (11,597 graduate departments in 601 institutions in 1997). Information is collected on student status, demographic characteristics, and major source of federal support. In 1994 the universe comprised doctorate-granting and master's-granting institutions. The annual Survey of Earned Doctorates is a census of all individuals who earn research doctorates from U.S. institutions each school year. It collects information about the field of study, demographic characteristics, educational history, financial support during graduate school, and immediate career plans of these new Ph.D.s. There were 42,683 such research doctorate recipients from July 1997 to June 1998. The biennial National Survey of College Graduates, based on a sample of about 50,000 individuals who reported on their 1990 Census returns that they had at least a bachelor's degree in science and engineering, collects data on the careers of these individuals. The biennial National Survey of Recent College Graduates (also known as the New Entrants Survey) uses a 2-stage probability sample of approximately 25,000 individuals who have received science or engineering bachelor's and master's degrees in the years since the 1990 Census to track the early development of their science and engineering careers. The biennial Survey of Doctorate Recipients, which samples about 50,000 science and engineering doctorate holders, collects longitudinal data on their professional careers. SRS also obtains data from the Bureau of Labor Statistics' Occupational Employment Survey. Source: http://www.nsf.gov/sbe/srs/educatio.htm.

OCR for page 17
Measuring the Science and Engineering Enterprise: Priorities for the Division of Science Resources Studies Research and Development Statistics The Research and Development Statistics Program (RDS) is responsible for surveys, studies, reports and analyses on the funding and performance of research and development (R&D) in the United States. (See Box 2-2 for definitions of research and development.) RDS conducts surveys on R&D funded and performed by government, industry, universities, and nonprofit organizations. (See Box 2–3.) Evolution of the Data Collection Activities of the SRS Human Resources Statistics Program 1945 Vannevar Bush meets with M.H. Trytten of the NRC's Office of Scientific Personnel and asks for data on the B.A. origins of Ph.D. recipients—leads to the NRC's Doctorate Records Project 1950 Congress establishes the National Science Foundation and directs it to maintain the National Register of Scientific and Engineering Personnel (NRSEP) 1952 Federal Security Agency develops the NRSEP 1953 NSF takes responsibility for operating the NRSEP 1957 Doctorate Records Project establishes the annual Survey of Earned Doctorates with funding from NSF and other agencies 1962 NSF and other agencies sponsor the Postcensal Manpower Survey 1966 Annual Survey of Graduate Students and Postdoctorates in Science and Engineering established 1966 National Center for Education Statistics begins Completions Survey as part of its Higher Education General Information Survey 1968 Start of trend data available on Immigrant Scientists and Engineers from INS records 1970 Last registration for the National Register of Scientific and Engineering Personnel (NRSEP) held 1972 NSF conducts the Professional, Technical, and Scientific Manpower Survey providing core data on scientists and engineers in lieu of the discontinued NRSEP 1973 Survey of Doctorate Recipients first fielded to cover the doctoral portion of the NRSEP 1974 New Entrants Survey designed to supplement the Manpower Survey begun 1975 Survey of Graduate Students and Postdoctorates in Science and Engineering begun 1977 Data first available from BLS's triennial Occupational Employment Statistics Survey 1982 Postcensal Survey of Scientists and Engineers fielded, providing core of the Scientific and Technical Personnel Data System (STPDS) 1982 Congressionally-mandated report Women and Minorities in Science and Engineering first published (later expanded to include women, minorities, and persons with disabilities) 1993 The STPDS is replaced by the Scientists and Engineers Statistical Data System (SESTAT)—with data from the National Survey of College Graduates, the Survey of Recent College Graduates, and the Survey of Doctorate Recipients

OCR for page 17
Measuring the Science and Engineering Enterprise: Priorities for the Division of Science Resources Studies The foundation for the RDS program was laid in the 1950s when NSF first collected data on federal R&D funding and R&D performance. In 1953, NSF established the Survey of Federal Funds for Research and Development, which collects data on R&D obligations made by federal agencies. NSF also began to collect data on R&D performance in 1953 when it funded the first Survey of Industrial Research and Development. The Bureau of Labor Statistics (BLS) fielded the first Industrial R&D Survey for NSF; administration of the survey was later transferred to the U.S. Census Bureau. The same year, NSF conducted the first of six occasional surveys of R&D performance by nonprofit institutions. In 1954, NSF conducted the first of three occasional small-scale surveys of R&D at major universities. In the 1960s and 1970s, NSF expanded the data it collected on public support for science and engineering. First, NSF deepened the data collected on federal R&D spending. The Survey of Federal Funds for R&D was made stronger by expanding detailed fields of science and engineering in 1960. In 1973, federal obligations for research to universities and colleges by agency and detailed field were also added to this survey. Collection of R&D funding by budget function was also expanded in 1961. Second, NSF established the congressionally-mandated Survey of Federal Support to Universities and Colleges in 1965 to better understand the role of the federal government in supporting academic research and development. Selected information on nonprofit organizations was added to that survey in 1968. Third, to round out data on public support, NSF conducted a survey of R&D support by local governments in 1966 and by state governments in 1967. Other occasional surveys of state support for R&D have also been conducted since. Box 2-2. Definitions of Research and Development The National Science Foundation uses the following definitions in its research and development funding and performance surveys: Basic Research The objective of basic research is to gain more comprehensive knowledge or understanding of the subject under study, without specific applications in mind. In industry, basic research is defined as research that advances scientific knowledge, but does not have specific immediate commercial objectives although it may be in fields of present or potential commercial interest. Applied Research Applied research is aimed at gaining the knowledge or understanding to meet a specific, recognized need. In industry, applied research includes investigations oriented to discovering new scientific knowledge that has specific commercial objectives with respect to products, processes, or services. Development Development is the systematic use of the knowledge or understanding gained from research directed toward the production of useful materials, devices, systems, or methods, including the design and development of prototypes and processes. Source: National Science Board, Science and Engineering Indicators—1998, 4–9.

OCR for page 17
Measuring the Science and Engineering Enterprise: Priorities for the Division of Science Resources Studies At the same time, NSF expanded and regularized the data it collected on R&D performance. Adding to the Survey of Industrial Research and Development already in the field, NSF began fielding a Survey of Science and Engineering Activities at Universities and Colleges in 1964. This was replaced in 1972 by the annual Survey of Research and Development Expenditures at Universities and College. Also, from 1970 to 1991, SRS had an Industrial Panel on Science and Technology with about 80 members to obtain quick, qualitative information on issues in industrial R&D. Congress mandated the collection of additional data on R&D performance and infrastructure in the 1980s and early 1990s. NSF first fielded the congressionally-mandated National Survey of Academic Research Instruments and Instrumentation Needs in 1983. In 1986, NSF also implemented the first of what is now the congressionally mandated Survey of Science and Engineering Research Facilities at Colleges and Universities. In 1990, Congress also required NSF to maintain the Master List of Federally Funded Research and Development Centers (FFRDCs). Box 2–3. Principal Surveys in the SRS Research and Development Statistics Program. The annual Survey of Federal Funds for Research and Development collects information about the characteristics and geographic distribution of all federal R&D funding from the approximately 100 federal agencies and subagencies that obligate funds for R&D. The annual Survey of Federal Support to Universities, Colleges, and Nonprofit Institutions collects information about federal obligations to individual academic and nonprofit institutions by the 15 federal agencies that provide virtually all federal R&D funds to academic institutions. The annual Survey of Research and Development in Industry collects information on R&D expenditures and employment of scientists and engineers from a nationally representative sample of about 23,000 companies (starting with the 1992 survey), including both manufacturing and non-manufacturing companies. The annual Survey of Research and Development Expenditures at Universities and Colleges collects data from a sample of about 460 institutions of higher education that grant science and engineering (S&E) degrees and perform a minimum level of separately budgeted R&D. The biennial Survey of Scientific and Engineering R&D Facilities at Colleges and Universities collects and analyzes data on the availability, condition, need, cost, and funding sources of facilities from a sample of 303 research-performing colleges and universities. Source: http://www.nsf.gov/sbe/srs/rdstati.htm.

OCR for page 17
Measuring the Science and Engineering Enterprise: Priorities for the Division of Science Resources Studies In the 1990s, RDS implemented several initiatives designed to improve or expand the data it collects on government, industrial, and nonprofit R&D funding and performance. In 1992, SRS revised its sample design for the Survey of Industrial Research and Development (RD-1) to improve its measurement of industrial R&D. In particular, the sample was increased and modified to better capture R&D performed by small and non-manufacturing firms. In 1994, the Census Bureau conducted for SRS a pilot innovation survey fielded to 1,000 companies. In 1998, SRS fielded the Survey of Science and Engineering R&D Funding and Performance by Non-Profit Organizations. The last survey of this nature had been conducted in 1973. SRS has also recently funded a survey carried out by the Battelle Memorial Institute of state funding for research and development. In 1998, SRS suspended administration of the Survey of Academic Research Instruments and Instrumentation Needs because of low demand for data from this survey and the need to deploy its resources elsewhere. Each RDS data collection activity was developed to address a narrow topic rather than to serve as a piece of a cohesive R&D funding and performance data system. Recently, SRS has examined how the results from different R&D funding surveys compare and has discovered discrepancies in funding and performance estimates among its surveys. For example, SRS estimates that federal funding for R&D performed by industry is $31.4 billion based on data from the Survey of Federal Funds for Research and Development, but just $23.9 billion based on data from the Survey of Industrial Research and Development. Further work to improve comparability—even the integration—of these surveys would improve their analytic value. Evolution of the Data Collection Activities of the SRS Research and Development Statistics Program 1953 The Bureau of Labor Statistics conducts the first Survey of Industrial Research and Development 1953 Establishment of Survey of Federal Funds for Research and Development 1953 SRS conducts the first of six occasional surveys of R&D performance by nonprofit institutions (the last collected 1973 data) 1953 SRS conducts first of three occasional small-scale surveys of science and engineering R&D at major universities 1957 Conduct of the Survey of Industrial Research and Development is transferred to the Census Bureau 1960 Expansion of detailed fields of science in the physical and social sciences reported in the Survey of Federal Funds for R&D 1961 Expanded collection of Federal R&D Funding by Budget Function 1964 Biennial Survey of Science & Engineering Activities at Universities and Colleges begins 1964 Congressionally-mandated, annual Survey of Federal Support to Universities and Colleges is first fielded 1964 SRS conducts survey of R&D support by local governments

OCR for page 17
Measuring the Science and Engineering Enterprise: Priorities for the Division of Science Resources Studies Evolution of the Data Collection Activities of the SRS Research and Development Statistics Program (continued) 1964 First of several occasional surveys of R&D support by State Governments (latest survey in 1996) 1967 First known publication of National Patterns of R&D Resources 1968 Selected information on nonprofit organizations added to what is now the Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions 1970 Industrial Panel on Science & Technology established (approximately 80 members) to obtain qualitative data on Industrial R&D. 1972 Annual Survey of Research and Development Expenditures at Universities and Colleges replaces Survey of Science & Engineering Activities at Universities and Colleges. 1973 Federal obligations for research to universities and colleges by agency and detailed S&E field added to the Survey of Federal Funds for Research and Development 1981 SRS sponsored surveys of industrial innovation begun by John Hansen, Christopher Hill, and James Stein 1983 Congressionally-mandated, triennial National Survey of Academic Research Instruments and Instrumentation Needs begun 1983 SRS publishes yearly Highlights providing estimates of R&D expenditures in the year ahead; estimates based on input from the Industrial Panel on Science and Technology 1986 Congressionally-mandated Science & Engineering Research Facilities at Doctorate-Granting Institutions established (later becomes Science and Engineering Research Facilities at Colleges and Universities) 1990 Congress requires NSF to maintain Master List of FFRDCs (Federally Funded Research and Development Centers) 1991 Industrial Panel on Science & Technology discontinued 1992 National Survey of Academic Research Instruments and Instrumentation Needs becomes biennial 1992 Sample design for the Survey of Industrial Research and Development revised to better capture R&D performed by small and non-manufacturing firms 1994 Bureau of the Census conducts a pilot innovation survey of 1,000 companies for SRS 1995 Survey of Federal Funds and Survey of Federal Support first provide data on the Department of Defense's Science and Technology budget 1997 Formal collection of the Survey of Academic Research Instrumentation and Instrumentation Needs suspended 1998 SRS fields the Survey of Science and Engineering R&D Funding and Performance by Nonprofit Organizations (last conducted in 1973)

OCR for page 17
Measuring the Science and Engineering Enterprise: Priorities for the Division of Science Resources Studies Data Publication, Integration, and Analysis Data Publication SRS published about 30 reports in 1998 and about 50 in 1999. These include detailed statistical tables, data and issue briefs, periodic reports, and special reports: Detailed Statistical Tables: reports containing an extensive collection of tabulated data from each of SRS's surveys Data Briefs and Issue Briefs: short reports highlighting results from recent surveys and analyses Periodic reports, such as Science and Engineering Indicators; National Patterns of R&D Resources; Women, Minorities, and Persons With Disabilities in Science and Engineering; and International Science and Technology Data Update Special reports, such as Undergraduate Origins of Recent Science and Engineering Doctorate Recipients and International Resources for Science and Technology In addition to publications, SRS disseminates some of its data in electronic format. SRS has developed a web site, highly regarded by individuals interviewed for this study, that provides on-line access to tabulations, reports, and two SRS databases, WebCASPAR (Computer-Assisted Science Policy Analysis and Research system) and SESTAT (Scientists and Engineers Statistical Data System). WebCASPAR is a database system containing multiyear data on science and engineering resources at individual academic institutions and by field. SRS draws on data for WebCASPAR from several of SRS's academic surveys plus information from a variety of other sources, including the National Center for Education Statistics. SESTAT is an integrated data system that draws on SRS personnel surveys to provide data on the employment, educational and demographic characteristics of scientists and engineers in the United States. WebCASPAR and SESTAT provide examples of ways in which SRS data or data sets can be integrated or linked and future SRS efforts to link its data within and across its programs would facilitate deeper analysis of the rich data sets that SRS has created. SRS also makes its survey data sets available for researchers to use in studying science and engineering resources issues in further depth, though within the restrictions of confidentiality requirements. Like other federal agencies, SRS is required to comply with the provisions of the Privacy Act of 1974 that require that confidentiality of records on individuals be maintained. Further, SRS also provides its own assurances to survey respondents that data will not be disclosed that would permit their identification. Thus, while SRS may release its data sets to outside researchers, the division strips identifying information (name, address, social security number) from its data sets before doing so and, in some instances, SRS suppresses fields or recodes data to further protect confidentiality. When recoding or field suppression would be so extensive that it is not cost-effective to produce a data file, SRS attempts to devise alternate means for providing data to researchers. Data Analysis Data analysis in SRS is carried out across the HRS, RDS, Integrated Studies, and Indicators Programs. HRS and RDS program staff prepare data briefs that announce key highlights from surveys at the time survey data are released or shortly thereafter. Staff in

OCR for page 17
Measuring the Science and Engineering Enterprise: Priorities for the Division of Science Resources Studies these programs also prepare substantial periodic reports, such as National Patterns of R&D Resources and Women, Minorities, and Persons with Disabilities in Science and Engineering that draw on data from surveys in R&D funding and performance and human resources, respectively. The reorganization of SRS in 1997 created an Integrated Studies Program to promote more integrated analysis of SRS data sets. Staff in this program conduct analyses for the biennial Science and Engineering Indicators and for publication in topical Issue Briefs. Box 2-4 displays Issue Briefs published by SRS through May of fiscal year 1999 by research area. These provide data and analysis on a variety of important topics. Box 2-4. SRS Issue Briefs Published in Fiscal Year 1999 (through May 1999). R&D Funding and Performance What Is the State Government Role in the R&D Enterprise? (May 26, 1999) What is the Federal Role in Supporting Academic Research and Graduate Research Assistants? (April 16, 1999) How Has the Field Mix of Federal Research Funding Changed Over the Past Three Decades? (February 17, 1999) How Has the Field Mix of Academic R&D Changed? (December 2, 1998) What are the Sources of Funding for Academically Performed R&D? (December 23, 1998) Venture Capital Investment Trends in the United States and Europe (October 16, 1998) U.S. Inventors Patent Technologies Around the World (February 24, 1999) Science and Engineering Labor Market Will Small Business Become the Nation's Leading Employer of Graduates with Bachelor's Degrees in Science and Engineering? (March 4, 1999) Degrees and Occupations in Engineering: How Much Do They Diverge? (December 31,1998) How Much Does the U.S. Rely on Immigrant Engineers? (February 11, 1999) What Follows the Postdoctorate Experience? Employment Patterns of 1993 Postdocs in 1995 (November 27, 1998) Science and Engineering Education Does the Educational Debt Burden of Science and Engineering Doctorates Differ by Race/Ethnicity and Sex? (April 16, 1999) Retention of the Best Science and Engineering Graduates in Science and Engineering (February 23, 1999) Have Forms of Primary Financial Support for S&E Graduate Students Changed During the Past Two Decades? (December 4, 1998) Has the Use of Postdocs Changed? (December 2,1998) SOURCE: http://www.nsf.gov/sbe/srs/issuebrf/ib.htm

OCR for page 17
Measuring the Science and Engineering Enterprise: Priorities for the Division of Science Resources Studies Science and Engineering Indicators In addition to compiling data from its surveys and other quantitative data sources for its own publication, SRS assists in the production of the National Science Board's (NSB's) biennial Science and Engineering Indicators report. Beginning with Science Indicators—1972, NSB has sought to publish a volume that gauges the status of the science and engineering enterprise and its contributions to national goals and the national welfare. Congress later amended the National Science Foundation Act of 1950 to require the NSB ''to render to the president, every evennumbered year, a report, for submission to Congress, on indicators of the state of science and engineering in the United States" (NSB 1996). To date, the NSB has published thirteen volumes in this series, the latest being Science and Engineering Indicators—1998. Indicators describes elementary and secondary science and mathematics education, postsecondary science and engineering education, the science and engineering workforce, research and development funding patterns, academic R&D performance, and industrial R&D. The report places each of these subjects in international context. SRS fields, on a biennial basis, a Survey of Public Attitudes and Public Understanding that monitors both public attitudes toward science and technology and the public's level of scientific understanding and policy preferences on selected issues. The results of this survey are published in the Indicators report. Each volume also includes a chapter on a special topic. In 1998, Indicators included a chapter on the economic and social significance of information technology. The contents of the 1998 volume are outlined in Box 2-5. Assessing SRS Data collection activities at NSF have grown over the last half century into the Division of Science Resources Studies—a small federal statistical agency with a range of programs centered on the topic of science and engineering resources. In the 1990s, SRS managers have worked to upgrade statistical standards, build staff expertise, revise surveys to increase data relevance, better coordinate data sets, improve the timeliness of data release, and expand data analysis. While this has been progress, the division's capabilities must be developed further in order to produce substantial benefits for the science and technology policymakers, planners, educators, and researchers. Operations and Resources SRS has not historically been viewed as a federal statistical agency by NSF. Yet the SRS staff of about forty are called on to collectively carry out each of the major functions of a federal statistical agency: data collection and acquisition, quality assurance, preparation of tabulations and public use data files, data analysis, publication of reports, and data and report dissemination. Generally speaking, SRS carries out these activities well, but there is room for improvement in statistical quality, staff expertise, data relevance and timeliness, and depth and breadth of analysis. These areas are discussed in the sections below. While SRS's mandate is narrowly focused around a particular topic, science and engineering resources, its budget of $14.5 million per year (FY 1999 estimated) is small relative to other agencies with similarly

OCR for page 17
Measuring the Science and Engineering Enterprise: Priorities for the Division of Science Resources Studies Box 2-5. National Science Board, Science and Engineering Indicators—1998, Contents. Chapter 1: Elementary and Secondary Education Student Achievement Curriculum and Instruction Teachers and the Profession of Teaching Chapter 2: Higher Education in Science and Engineering Worldwide Increase in S&E Educational Capabilities Characteristics of U.S. Higher Education Institutions Undergraduate S&E Students and Degrees in the United States Graduate S&E Students and Degrees in the United States International Comparisons of S&E Training in Higher Education Chapter 3: Science and Engineering Workforce Labor Market Conditions for Recent S&E Degree-Holders Selected Characteristics of the S&E Workforce S&E Job Patterns in the Service Sector Scientists and Engineers in an International Context: Migration and R&D Employment Projected Demand for S&E Workers Chapter 4: U.S. and International Research and Development: Funds and Alliances National Trends in R&D Expenditures R&D Patterns by Sector Inter-Sector and Intra-Sector Partnerships and Alliances Government R&D Support International Comparisons of National R&D Trends Chapter 5: Academic Research and Development: Financial and Personnel Resources, Integration With Graduate Education, and Outputs Financial Resources for Academic R&D Academic Doctoral Scientists and Engineers Integration of Research with Graduate Education Outputs of Scientific and Engineering Research Chapter 6: Industry, Technology, and Competitiveness in the Marketplace U.S. Technology in the Marketplace International Trends in Industrial R&D Patented Inventions Venture Capital and High-Technology Enterprise New High-Tech Exporters Summary: Assessment of U.S. Technological Competitiveness Chapter 7: Science and Technology: Public Attitudes and Public Understanding Interest in Science and Technology Understanding of Scientific and Technical Concepts Attitudes Toward Science and Technology Policy Issues Sources of Scientific and Technical Information Chapter 8: Economic and Social Significance of Information Technologies Information Technologies Impacts of IT on the Economy IT, Education, and Knowledge Creation IT and the Citizen

OCR for page 17
Measuring the Science and Engineering Enterprise: Priorities for the Division of Science Resources Studies specific objectives, such as the Bureau of Justice Statistics. As seen in Table 2-1, the FY 1999 budgets of the other major statistical agencies range in size from $25.0 million in the Bureau of Justice Statistics to $1.35 billion for the U.S. Census Bureau. In general, NSF should see SRS as a federal statistical agency and should support the division in its efforts to meet fully those standards for statistical agencies regarding independence, professional staffing, data quality, and data analysis. We recommend that NSF provide SRS with additional staff by increasing the number of full-time equivalent (FTE) positions allotted to the division and that SRS actively engage external researchers through visiting fellowships and external grants. Additional staff and the engagement of outside researchers will increase the breadth of activities the division may undertake as well as the depth of skills brought to these activities. Further, since SRS's budget is substantially smaller than those of other agencies with a specific policy focus its resources may need to be increased given the growing importance of its subject area and our recommendations for new processes and data collection activities as discussed in Chapters 3, 4 and 5. Any budget increase, however, must be based on an informed analysis of the allocation of financial and staff resources across SRS activities and on a clear sense of priorities among current and proposed activities. Table 2-1 Fiscal 1999 (estimated) and 2000 (requested) Budgets for Major Federal Statistical Agencies (millions of dollars) Statistical Agency Department or Agency 1999 (estimated) 2000 (request) Bureau of the Census: current programs Commerce 156.1 166.9 Bureau of the Census: periodic programs (censuses) Commerce 1193.8 2914.8* Bureau of Economic Analysis Commerce 43.1 49.4 Bureau of Labor Statistics Labor 398.9 420.9 National Agricultural Statistics Service Agriculture 104.0 100.6 Economic Research Service Agriculture 65.8 55.6 National Center for Education Statistics Education 104.0** 117.5** National Center for Health Statistics HHS 94.6 109.6 Energy Information Administration Energy 70.5 72.6 Bureau of Justice Statistics Justice 25.0** 32.5** Bureau of Transportation Statistics Transportation 31.0   31.0 Statistics of Income Division, Internal Revenue Service Treasury 28.8   30.9 Science Resources Studies Division NSF 14.5** 14.9** * Developed before the Supreme Court decision on sampling. ** Funding levels shown for the National Center for Education Statistics, the Bureau of Justice Statistics, and the Science Resources Studies Division do not include salaries and expenses from other departmental sources. Source: Council of Professional Associations on Federal Statistics (Spar 1999); National Science Foundation FY 2000 Budget Request.

OCR for page 17
Measuring the Science and Engineering Enterprise: Priorities for the Division of Science Resources Studies Statistical Agency Practices In Principles and Practices for a Federal Statistical Agency, the Committee on National Statistics has delineated a useful set of standards against which SRS—as a statistical agency—may be assessed. Here we highlight SRS operations relative to standards for data quality, staff expertise, data relevance, data linkages, and analytical activity to suggest areas that need additional improvement. Data Quality The Committee on National Statistics specifies that to assure commitment to data quality a federal statistical agency should: develop an understanding of the validity and accuracy of its data and convey the resulting measures of uncertainty and sources of error to users undertake ongoing quality assurance programs to improve data validity and reliability and to improve the processes of gathering, compiling, editing, and analyzing data use modern statistical theory and sound statistical practice in all technical work (NRC 1992) Following from these, a federal statistical agency must fully describe its data and comment on their relevance to specific major uses. As described in an NRC report on the Bureau of Transportation Statistics, the agency should "describe the methods used, the assumptions made, the limitations of the data, the manner by which data linkages are made, and the results of research on the methods and data. Measures of uncertainty should be provided to users, and statistical standards should be published to guide professional staff in the agency as well as external users" (NRC 1997a). SRS has a good track record of improving data quality and meeting statistical standards in the recent past, but should take additional steps to ensure that standards are met across SRS operations. For example, SRS has undertaken substantial work in the 1990s to upgrade data quality for the three personnel surveys in the SESTAT system. In addition, SRS has required contractors to provide detailed methodology reports. To this end, the National Research Council, for example, began producing a methods report for the Survey of Earned Doctorates beginning with the 1990–1991 survey year. SRS could improve data quality and information about it further by taking at least three additional steps and possibly others. First, it should require that all of its contractors provide methods reports that address data quality standards. The contractor for the Survey of Public Attitudes, for example, does not currently provide a methodology report that details measures of data quality. Second, SRS should continue recent efforts to provide staff with professional development opportunities for improving their statistical skills, so they can better implement quality assurance programs. Third, SRS should continue to develop and strengthen a program of methodological research, undertaking rigorous analysis of the data collected to assess the quality of the data relative to concepts it is supposed to measure. Professional Staff and External Expertise Given the small size of SRS and its need to accomplish each of the major tasks of a federal statistical agency, SRS staff are required to perform multiple functions. In larger statistical agencies, each staff person would have a more focused role. In SRS, staff are called on to perform several roles at a time. For example, survey officers not only supervise survey contractors, they also conduct data analyses, write data briefs, and

OCR for page 17
Measuring the Science and Engineering Enterprise: Priorities for the Division of Science Resources Studies contribute to reports. Likewise, program directors must perform their managerial duties, but also continue to analyze data and write reports. To meet the expectations of the NSB and other constituents, SRS staff must have adequate expertise in both the theory and practice of statistics and in the disciplines relevant to the analysis of data on science and engineering resources (NRC 1992). As of spring 1999, SRS has a staff of 42, very small compared to other federal statistical agencies whose staff sizes range into the thousands. Many staff have expertise or experience in statistical methods, survey management, economics, science policy, education, and international comparisons of science and engineering. However, staff skills in areas such as statistical methods, generally speaking, should be improved and the range of staff expertise should be expanded. First, SRS may augment its staff expertise through professional development activities. SRS has begun identifying areas in which staff require additional training to improve statistical methods. These areas could be addressed in-house through individual training, group discussions with staff on statistical techniques (perhaps "brown bag" discussions scheduled during the lunch hour), and review of staff work. Staff may also upgrade skills by attending relevant graduate courses in statistical methods and data analysis, participating in short courses offered through continuing education programs, and staying active in relevant professional associations. Some statistical agencies make explicit their commitment to staff development by specifying professional development goals for each of their staff as part of their performance reviews. Such a process could ensure that staff take necessary steps to acquire and maintain appropriate skill sets (NRC 1997a). SRS should also develop a staffing plan that allows it to augment staff expertise, particularly in key areas. Earlier in the 1990s, NSF reduced the number of SRS staff and the staffing level of the division has remained flat since, even though demands on the division have increased. This reduction in SRS personnel compounded other cuts in the resources available to NSF for data analysis. In the 1970s and 1980s much use was made of SRS data in analyses by NSF's Division of Policy Research and Analysis (PRA) and its predecessors. When NSF terminated this office, the resources that had been available to PRA were never fully made available to SRS. Together, these reductions have left NSF and the science and engineering community where they are today—short on staff resources for both data collection and analysis. Within the cap on its number of full-time equivalent (FTE) staff, SRS should use staff turnover as a means to attract skilled staff and meet needs for expertise in statistical methods and important analytical subject areas. New hires in HRS and RDS should have strong skills in statistical methods and subject matter knowledge. New hires in the Indicators or Integrated Studies programs should have strong analytical skills, subject matter knowledge, and the ability to draw on diverse data to provide meaningful analyses of science and engineering resources. We would also like to see NSF allow the number of SRS staff to grow so that the division may better meet its constituents' expectations. This would allow SRS to more rapidly acquire the skilled staff it needs, broaden the range of its activities, and improve the quality of its work. Finally, because its staff is small, SRS cannot have expertise in all subject areas for which it could conceivably be called upon to provide data and analysis. SRS needs to develop a more interactive relationship with external researchers to increase the division's range of professional capabilities. SRS currently invites outside experts to attend workshops and serve on advisory panels. By also establishing programs to more actively engage these experts, SRS could expand the breadth of research it undertakes and create

OCR for page 17
Measuring the Science and Engineering Enterprise: Priorities for the Division of Science Resources Studies opportunities for interaction between SRS and external data users, producing insights that would benefit SRS staff, researchers, and the relevance and quality of the data. Several statistical agencies have visiting fellows programs in which distinguished statisticians or other researchers work in the agency for a specified period of time. NSF has a long history of experts rotating into the agency on a temporary basis and this practice could be extended to SRS. Also, many statistical agencies have programs that provide grants to external researchers to use their data. SRS has the authority to administer such a program, but has not issued a general announcement since the mid-1980s because of budget constraints. SRS has awarded external grants since, but only on an ad hoc basis. Grant sizes have ranged generally from $50,000 to $150,000. Today, the division needs to focus additional effort on data analysis and revitalizing a program of external grants would increase the use and analysis of SRS data, especially if targeted toward underutilized data such as those in SESTAT. Data Relevance SRS is a federal statistical agency that exists to serve the information needs of policymakers, program administrators, planners, educators, and researchers in the science and engineering community. It should, therefore, meet the standard for data relevance set by the Committee on National Statistics: "a federal statistical agency must be in a position to provide information relevant to issues of public policy" (NRC 1992). Indeed, SRS's data are useful to its constituents and widely used. SRS data are published in the National Science Board's biennial Science and Engineering Indicators, an important reference for federal science and technology policy makers. They are also used widely in policy reports on basic research, graduate education, and scientific and technological workforce issues. Yet, in the 1990s, changes in the science and engineering enterprise have left SRS lagging behind the data needs of its constituents and issues have emerged for which SRS has not been able to provide adequate data. (Examples of these issues in graduate education, the labor market for scientists and engineers, and R&D funding are detailed in Chapters 4 and 5.) The Committee on National Statistics states that "an agency's mission should include responsibility for assessing the needs for information of its constituents" (NRC 1992). SRS has taken some steps to assess user needs, such as fielding a customer survey on a periodic basis and contracting with the NRC to produce this report. SRS has administered customer surveys in 1996 and 1999 that seek to ascertain customer satisfaction with various aspects of SRS data and publications. However, SRS needs to take additional steps to develop and implement means for reviewing, updating, and supplementing its data collection and acquisition activities on an ongoing basis to meet current information needs. To keep its data relevant, SRS must substantially and continuously improve and supplement its survey instruments and data analysis. To achieve this, SRS must strengthen its dialogue and interactions with policymakers, academic researchers and other data users to capitalize on their insights, expertise, and analytic capabilities. Means for accomplishing this include establishing advisory committees for each survey; holding a series of workshops on emerging issues in the science and engineering enterprise; improving outreach with constituent groups; more purposeful dissemination of publications; and the promotion of data use through easier access to data and programs to more deeply involve external researchers in SRS data analysis. SRS must also develop internal processes to convert the feedback it receives from stakeholders in these activities into changes in its surveys and issues for analysis. Improvements in data collection

OCR for page 17
Measuring the Science and Engineering Enterprise: Priorities for the Division of Science Resources Studies almost inevitably come at a higher cost; however, since SRS resources are limited, it is critical that priorities for change be based on both the relevance of the data and the costs involved. To improve the currency of its data, SRS must also continue its recent efforts to substantially reduce the period of time between the reference date and data release date for each of its surveys. Means for reducing survey processing time and the time spent in releasing data include incentives for timely response, increased use of the Internet for data collection, and early release of key indicators. Finally, SRS should employ quick response surveys to obtain information more rapidly on pressing issues and use qualitative methods as a complement to periodic surveys in order to more deeply investigate poorly understood issues. Data Linkages The Committee on National Statistics states that an effective statistical agency promotes data linkages in order to enhance the value of data sets and their analytic power (NRC 1992). SRS's portfolio of data collection activities has grown over the past half century as individual surveys have been established to provide information on specific pieces of the science and engineering enterprise. SRS has only recently begun to manage its surveys as components of a more integrated data system and would increase the depth and usefulness of its data by pursuing this further. SRS should find ways beyond SESTAT and WebCASPAR to link data sets. For example, creating linkages between its R&D funding and human resources data programs would give a better overall picture of resources available for science and engineering. Standardizing its science and engineering field taxonomies and revising questions to improve comparability across survey instruments are critical steps that would facilitate this process. SRS should not limit itself to these steps, however, but should also find ways to allow researchers to link its data to those from other data sources, public and private. It should also encourage standardization in university data collection on the career paths of science and engineering graduates and continue to play a lead role in collecting, coordinating, and standardizing international S&E resource data. Data Analysis The Committee on National Statistics also stipulates that "an effective statistical agency should have a research program that is integral to its activities" (NRC 1992). Analysis of substantive issues for which data were compiled has two goals. First, it provides information on these issues through issue briefs and reports to policymakers, and other constituents. Second, analysis will indicate limitations in an agency's data and thus guide how its surveys could be redesigned to improve concepts and fill data gaps. SRS currently provides data analysis through data briefs, issue briefs, periodic reports, and special reports. The public release of SRS survey data is announced through the publication of a data brief that highlights one important trend from the data set and thus seeks to couple the release to an important current issue. SRS has recently increased the number of issues briefs it publishes, seeking to bring SRS data to the analysis of a specific, narrowly focused science and engineering resources issue. SRS also has an important relationship with the National Science Board in the production of a major statistical report, Science and Engineering Indicators. SRS is responsible for compiling data from its surveys and other quantitative data sources and producing the Indicators report under the guidance and on behalf of the NSB.

OCR for page 17
Measuring the Science and Engineering Enterprise: Priorities for the Division of Science Resources Studies SRS should carefully consider how it may best engage in and support research on science and engineering resources in the future, seeking greater analytic use of its data. We support the recent effort of SRS to expand its analytical program and the publication of topic-specific issue briefs. As noted above, we also urge SRS to engage outside experts and researchers in research activities using SRS data. The division has not realized the potential benefits of more fully interacting with outside researchers and leveraging their expertise. With a relatively small investment in grants to outside researchers or a program to bring researchers into SRS on a temporary basis, SRS would benefit from an increase in the breadth and depth of data analysis. Such a program also allows SRS analytical flexibility, as specific researchers may be engaged based on their expertise as substantive issues change. Finally, NSB and SRS should develop a long-term plan for restructuring Science and Engineering Indicators. Individuals interviewed for this study as well as science and technology policymakers recently interviewed by SRS following the publication of the last Indicators volume suggest at least three possible futures for Indicators. The first of these, of course, is maintaining the status quo. As currently conceived, Indicators provides a wealth of information on science and engineering resources in the United States, and increasingly, in an international framework, which benefits both the NSB and SRS. It provides the NSB a means for highlighting important science and engineering issues. It also provides SRS a means for showcasing its data in a high profile report considered by many an essential reference for quantitative information by science and technology policymakers. The second is for the NSB to reduce the amount of policy analysis in the volume and concentrate on the data presented. Those with this perspective believe that the policy analysis presently in Indicators is not very useful, while the data are. The third is for the NSB to make the document more focused on policy issues and less on data. Individuals holding this point of view suggest that Indicators would have a greater impact if it were smaller, less redundant with other SRS publications, and offered policy insights built on important indicators. We believe that Science and Engineering Indicators should be smaller and more policy focused. Indicators would have more impact on science and technology policy if it focused on bringing analysis to a small set of indicators on issues driving the future of the science and engineering enterprise. There should be a sharper division between the work of a policymaking body such as the National Science Board and the work of a federal statistical agency such as SRS. Much tabular material in Indicators is redundant with other publications of SRS data, which could be referenced in Indicators and also linked via hypertext when published on the Internet. Substantial SRS resources—especially staff resources—which are now devoted to the production of this volume, would be freed for other analytic activities if the report were refocused.