Measuring the Science and Engineering Enterprise: Priorities for the Division of Science Resources Studies (2000)

The Division of Science Resources Studies (SRS) of the National Science Foundation (NSF) is one of fourteen agencies in the federal statistical system, as represented on the Interagency Council on Statistical Policy. SRS is charged with providing data and analyses on various policy areas for public-and private-sector constituents. As mandated by the National Science Foundation Act of 1950 as amended, SRS is ''to provide a central clearinghouse for the collection, interpretation, and analysis of data on scientific and engineering resources, and to provide a source of information for policy formulation by other agencies of the federal government." To fulfill this mandate, SRS collects and acquires data on national patterns of research and development funding and performance, and on the education and careers of scientists and engineers.

To keep its data and analysis relevant to policymakers, educators, managers, and researchers, SRS should continually review and update the concepts it seeks to measure, and revise its survey instruments, survey operations, and data analysis as needed to keep them current. To achieve this, SRS must strengthen the frequency and intensity of its dialogue and interactions with its data users and develop internal processes to convert the feedback it receives from these stakeholders into changes in its surveys and analyses. As a key element of this strategy, SRS should create advisory committees for SRS surveys that would assist SRS in establishing priorities for future change. SRS should also actively engage external researchers in both the development of its surveys and the analysis of its data to broaden the range of expertise that is brought to bear on them. SRS should further increase the analytic value of its data by finding ways to improve comparability and linkages among its data sets and by continually improving the timeliness with which its data are released. All of these strategies—creating a dialogue, instituting advisory committees, engaging external researchers, linking data, and improving timeliness—should be used in addressing important, current science and engineering resources issues that are described briefly below and more fully in the report. Finally, we expect SRS to meet accepted standards for statistical agencies in independence, professional staffing, data quality, and data analysis.

Today, SRS is organized functionally into data collection, analysis, and dissemination units. Two statistical programs—the Human Resources Statistics Program (HRS) and the Research and

Development Statistics Program (RDS)—have responsibility for primary data collection and acquisition within SRS on personnel, education, and research and development (R&D) funding and performance. HRS and RDS also produce data tabulations and reports for dissemination to the public. Two additional units—the Science and Engineering Indicators Program, responsible for supporting the National Science Boards' biennial Science and Engineering Indicators, and the Integrated Studies Program, responsible for producing special analyses on science and engineering resource topics—focus primarily on analysis activities. The division is supported by the Information Services Group, which oversees publication of SRS reports, both in hard copy and via the SRS web site.

To be effective in this role in the long-term, SRS must ensure the ongoing relevance of the information it provides through its portfolio of data collection and analysis activities. Indeed, recent developments in the science and engineering enterprise have suggested the need for SRS to update its portfolio. Patterns in industrial R&D investment have changed. For example, industry's share of R&D funding in the United States has increased from about half in 1980 to almost two-thirds today. Service sector R&D, which accounted for just 5 percent of industrial R&D in the early 1980s, now makes up almost 25 percent of it. Federal R&D spending since the end of the Cold War has been characterized by decreases in defense R&D and an acceleration in the long-term growth of federal spending on biomedical research. Similarly, patterns in the education and careers of scientists and engineers have also changed. For example, new science Ph.D.s in some fields have encountered difficulties in the job market in the 1990s, with increased numbers of recent Ph.D.s in a series of postdoctoral positions and others working outside their fields of study. Since the early 1990s, more than half of doctorate-level scientists and engineers work in positions outside of academia. Still other developments suggest the structure of the science and engineering enterprise is changing. Recent data indicate an increasing number of intra-and inter-sectoral R&D alliances. Anecdotal information suggests that an increasing percentage of cutting-edge research is multidisciplinary in nature. New fields, such as biotechnology and information technology, have changed our perceptions of the way scientific research is conducted and translated into innovation.

In light of these developments, SRS asked the National Research Council (NRC) to undertake a review of its portfolio of data collection and analysis activities and to assist SRS in revising these activities to better meet the information needs of policymakers, managers, educators, and researchers. In response, the NRC created the Committee to Assess the Portfolio of the Science Resources Studies Division of NSF to identify gaps in NSF surveys and provide prioritized recommendations for addressing them.

Given the breadth of its charge and the short time frame for this study, the committee sought to identify key science and technology resource issues to guide its recommendations in an efficient, yet informative manner. The information for this study, gathered in 1998, included: (1) presentations to the committee by senior SRS managers on their operations and data collection challenges, and later, individual interviews with these managers; (2) focus groups with SRS staff; (3) reports of SRS customer surveys undertaken in 1994 and 1996; (4) interviews with 42 individuals in government, industry, and academia to solicit their views on current issues in

science and engineering and the data available to address them; (5) a two-day NRC "Workshop on Data to Describe Resources for the Changing Science and Engineering Enterprise;" and (6) the conclusions and recommendations from a number of recent reports on issues in science and technology policy. The committee also looked at recent data trends to substantiate the importance of issues raised from these sources.

Since updating the portfolio of a federal statistical agency to provide relevant data is an ongoing process, the committee examined how the operations of federal statistical agencies, such as SRS, may be structured to produce continuous renewal of data collection activities. We found that while SRS is considerably smaller than other major federal statistical agencies in budget authority, the division carries out each of the major functions of a federal statistical agency: data collection and acquisition, quality control, preparation of standard data tabulations, data analysis, publications, and data dissemination. Thus, we focused our study on standards for federal statistical agency operations regarding data quality, staff expertise, relevance of concepts measured, data linkages, and timeliness that suggest areas of continued improvement for SRS. However, we highlighted the last three—appropriateness of concepts and their measurement, ability to link data, and data currency —as critical dimensions of data relevance that allow statistical agencies to best inform policymaking.

We have two overarching recommendations that should be considered our highest priorities for SRS and these encourage the division to strengthen its dialogue with its stakeholders and increase its interactions with external researchers. Other recommendations follow that provide an immediate agenda for the dialogue and interactions.

Recommendation 1. To keep its data relevant and maintain data quality and analytic capacity, SRS should adopt a strategy of continuous review and renewal of the concepts it seeks to measure, and revise its survey instruments, survey operations, and data analysis as needed to keep them current. To achieve this, SRS must strengthen the frequency and intensity of its dialogue and interactions with data users, policymakers, and academic researchers and develop internal processes to convert the feedback it receives from these stakeholders into changes in its surveys and analyses. A key element of this strategy is the creation of advisory committees for SRS surveys that would assist SRS in establishing priorities for future change.

To expand the range of surveys that benefit from advisory committees, we strongly recommend the creation of such a committee for the Survey of Industrial Research and Development. We also recommend that the existing Special Emphasis Panel (i.e., advisory committee) for the Doctorate Data Project (DDP) advise SRS on the Survey of Earned Doctorates (SED), the content of all three SRS personnel surveys, and the design of the Scientists and Engineers Statistical (SESTAT) system. This panel already provides SRS advice on the SED and the Survey of Doctorate Recipients (SDR) and should also provide advice on the National Survey of College Graduates (NSCG) and the National Survey of Recent College Graduates (NSRCG).

A statistical agency should work with data users to define the concepts that it will

measure to meet users' information needs. These concepts, and how to measure them, should be continuously reviewed and updated as issues change and as analysis reveals alternate measures that better capture information that is useful to constituents. To operationalize this ongoing process of reviewing and updating data concepts, SRS must increase the frequency and intensity of its dialogue and interactions with data users, policymakers, and academic researchers to capitalize on their insights, expertise, and analytic capabilities. To generate direct interaction of this sort, SRS should establish advisory committees for each of its surveys. These committees will help SRS keep survey content up-to-date and establish priorities for future change. Such committees should be constituted or expanded to cover the Survey of Industrial R&D and all three personnel surveys in the SESTAT system. The SRS Breakout Group of the Directorate for Social, Behavioral, and Economic Sciences (SBE) Advisory Committee and the advisory committees for individual surveys should work together to review and assist in the implementation of our recommendations and set priorities among them and other proposals for change. SRS may also generate dialogue and interaction by holding workshops on emerging issues, improving outreach with constituent groups through booths at conferences, disseminating publications more purposefully, enhancing customer service, and administering a periodic customer survey. SRS should convert the feedback it receives from stakeholders in these ways into means for producing the information its constituents seek. SRS may revise survey instruments, add special modules to instruments to collect data for one survey cycle, establish quick response panels, or employ or sponsor qualitative research as a complement to periodic surveys in order to obtain information more rapidly or more deeply on poorly understood issues.

Recommendation 2. SRS should more actively engage outside researchers in the analysis of SRS data on current science and engineering resources issues. This may be accomplished by allowing researchers to work at SRS as visiting fellows and by establishing an external grants program. SRS should also monitor and summarize research using its data.

Since the division is limited in its staff size, it cannot have expertise in the full range of subject areas upon which it may be called for data and analysis. SRS and its constituents would benefit from a more interactive relationship between SRS and external researchers, such as university-based researchers, who focus on science and engineering resources issues. The division would expand its analytical range and promote data use by bringing researchers into SRS through a visiting fellows program and by providing grants to researchers who utilize SRS data. SRS should also monitor and summarize research using its data, particularly for its personnel surveys (the National Survey of College Graduates, the National Survey of Recent College Graduates, and the Survey of Doctorate Recipients), which are currently underutilized.

Recommendation 3. SRS and the National Science Board (NSB) should develop a long-term plan for Science and Engineering Indicators so that it is smaller, more policy focused, and less duplicative of other SRS publications to free SRS resources for other analytical activities.

NSB and SRS should develop a long-term plan for restructuring Science and Engineering Indicators. We believe that Science and Engineering Indicators should

be smaller and more policy focused. Indicators would have more impact on science and technology policy if it focused on bringing analysis to a small set of indicators on issues driving the future of the science and engineering enterprise. There should be a sharper division between the work of a policymaking body such as the National Science Board and the work of a federal statistical agency such as SRS. Indicators is redundant with other publications of SRS data which could be referenced in Indicators and also linked via hypertext when published on the Internet. Substantial SRS resources—especially staff resources—which are now devoted to the production of this volume, would be freed for other analytic activities if the report were refocused.

Recommendation 4. SRS should increase the analytic value of its data by improving comparability and linkages among its data sets, and between its data and data from other sources. Standardizing its science and engineering field taxonomies and other questions across survey instruments is a critical step in this process. Resolving discrepancies in results from different surveys is another.

The ability to link data sets increases the breadth and depth of data, and thereby the ability of analysts to use them to address current issues. SRS's portfolio of data collection activities has been established over the past half century incrementally as individual surveys have been established to provide information on specific pieces of the science and engineering enterprise. To increase the analytic power of its data, SRS should find ways to integrate its data sets further. SRS should continue to improve comparability in questions and response categories across surveys, particularly with regard to questions on field of science and engineering. SRS should continue to investigate discrepancies in survey results among its R&D funding and performance surveys and implement changes in survey instruments and operations to address and resolve them. The division should also consider establishing a committee to develop a design for a new, integrated R&D data system that would also account for the apparent increase in the number of intra-and inter-sectoral R&D partnerships. SRS must also develop means for linking its R&D investment data and its human resources data, and help coordinate the data gathering activities of others to improve data availability. Finally, SRS should seek effective ways to allow researchers to link its data with other data sources, public and private.

Recommendation 5. SRS must substantially reduce the period of time between the reference date and data release date for each of its surveys to improve the relevance and usefulness of its data.

In the case of SRS, the timeliness with which data are released is determined by the time that elapses between the reference date in each survey and the date on which survey data are released. The consensus of SRS's data users is that this period of time is generally too long for SRS surveys and that SRS needs to address the length of time taken for survey follow-up, data processing, and the public release of data. To improve the currency of its data, SRS must continue its recent efforts to substantially reduce the period of time between the reference date and data release date for each of its surveys. Means for accomplishing this goal include using incentives for timely response, increased use of the Internet for data collection, and early release of key indicators.

Recommendation 6. SRS should be seen as a federal statistical agency, and should be supported in its efforts to meet fully those standards set for federal statistical agencies for independence, professional staffing, data quality, and data analysis.

Recommendation 7. SRS's budget is substantially smaller than those of other federal statistical agencies and may need to be increased given the growing importance of its subject area and our recommendations for new processes, data collection activities, and additional studies. Any budgetary increase must be based on clearer information from SRS on its allocation of internal staff and financial resources across its surveys and other activities and on a clearer sense of priorities among current and future surveys and activities, as developed in coordination with advisory committees for its individual surveys and the SRS Working Group of the SBE Advisory Committee.

NSF should see SRS as a federal statistical agency and support the division's efforts as it strives to fully meet standards set for federal statistical agencies regarding independence, professional staffing, data quality, and data analysis. As an important component of this, NSF should provide SRS with additional staff by increasing the number of full-time equivalent (FTE) positions allocated to the division.

SRS's budget is substantially smaller than those of other federal statistical agencies and may need to be increased given the growing importance of its subject area and our recommendations for new processes, data collection activities, and additional studies. We did not have access to sufficiently detailed budget data to conduct a cost-benefit analysis either of the items we recommend or of existing components of the SRS portfolio. Thus, we have not been able to further prioritize our recommendations, nor have we been able to suggest trade-offs between new and existing activities.

Science and engineering, a $247 billion¹ a year enterprise of individuals and institutions who carry out research and development in the United States, plays a central role in the advancement of our knowledge-intensive economy and affects the daily lives of Americans in myriad ways. To keep its data relevant for answering today's questions on science and engineering resources, SRS needs to keep its data collection portfolio in synchrony with changes in the science and engineering enterprise. SRS should investigate under-addressed issues in graduate education, the labor market for scientists and engineers, and R&D funding and performance, and use the results of these investigations to revise its surveys and analytic activities.

Recommendation 8. SRS should revise its data collection on issues in graduate education and the job market for new Ph.D.s to better address issues on financial support for graduate students, completion of graduate school, and the transition of new Ph.D.s into employment. SRS must carefully study whether fielding a new longitudinal survey of beginning graduate students, now under consideration, is feasible and cost-effective before committing to such a survey. However, the division should revise the Survey of Earned Doctorates (SED) to

include questions on progress toward degree completion and job market experiences and it should seek to assist professional societies and universities in the collection of standardized data on the job market for new Ph.D.s.

In the face of a difficult job market for recent science Ph.D.s in some fields in the 1990s, policymakers, educators, and analysts have expressed concerns about the efficacy of certain types of support for graduate students and about the outcomes of graduate education. To better understand these issues, they have requested additional data on graduate school completion and attrition, career expectations, educational experiences and skills acquired, student financial support, and the effect of these on graduate school and career outcomes. We recommend that SRS analyze and quickly disseminate the retrospective data it collected through the 1997 SDR on the graduate school and job market experience of Ph.D.s who received their degrees between June 1990 and June 1996. These data should address some of the concerns of policymakers, educators, and analysts in this area. We also recommend that SRS revise the Survey of Earned Doctorates to obtain data on progress through graduate school, perhaps by adding a question asking respondents for the date when all Ph.D. requirements except for the dissertation were completed. We recommend that SRS institute ongoing collection of data about the job market experience of Ph.D.s prior to degree receipt by adding questions on this subject to the SED while also continuing to augment its own data collection by assisting others who are collecting data in this area. We do not, however, recommend that the SED, or a longitudinal survey of graduate students, if fielded, be used to collect data on "skills" obtained by graduate students.

SRS is currently in the development stage for a new longitudinal survey of beginning graduate students, designed to be responsive to calls for additional data on education and job market experiences of graduate students. SRS should carefully consider the feasibility and costs of developing and administering such a survey. Based on our current understanding, we question the wisdom of such a survey. SRS should weigh whether the issues raised warrant ongoing national data collection from graduate students, examine its ability to collect high-quality national data on attrition and packages of financial support through a longitudinal survey, determine the cost effectiveness of conducting such a survey, and consider alternative sources of data.

Recommendation 9. SRS should revise its data collection on the labor market for scientists and engineers to belter capture the career paths of scientists and engineers. SRS should fill gaps in existing data on careers by collecting comparative data on the careers of humanities doctorates, and data on the nonacademic careers of scientists and engineers, on science and engineering field of work, and on the international mobility of scientists and engineers. The division should also work with the Special Emphasis Panel for the Doctorate Data Project to address content and design issues for the SESTAT system to be implemented in the next decade.

To facilitate deeper analysis of personnel issues, SRS should first modify its approach to the design and use of its personnel surveys. SRS should provide better career path data to analysts by making them available at a more detailed level by field. SRS should consider the options available for improving fine field analysis that is currently obstructed by small sample sizes for its personnel surveys. On a related note, science and engineering field of work,

dropped from the SDR in 1993, should be restored to the questionnaire. SRS should also exploit the longitudinal nature of its personnel surveys obtained at great expense and with a respondent burden that is difficult to justify if the data are not used longitudinally. Finally, SRS should explore opportunities for linking its personnel data to other career and productivity data, such as data sets on federal research grants, patents, and publications.

SRS should also revise its personnel surveys to better capture the career paths of scientists and engineers. It should consider adding questions to the SDR to obtain additional data on the careers of Ph.D.s who work for government agencies, private businesses, and nonprofit organizations. Such questions might focus on non-salary compensation; patenting and other productivity measures in the private sector; use of scientific background in sales, regulation, or patent law positions; and temporary work arrangements, such as contracting and consulting. SRS must also work with the National Endowment for the Humanities (NEH) and other funding sources to reinstitute the humanities component of the SDR, discontinued following the 1995 survey cycle due to budget cuts at NEH. Finally, SRS should augment the data its collects on the global dimensions of the science and engineering workforce by expanding data on foreign scientists and engineers in the United States at all levels— students, postdoctorates, and employees.

Recommendation 10. SRS should revise the data it collects on R&D funding, performance, outputs, and outcomes to improve comparability across surveys and to address structural changes in the science and engineering enterprise. SRS should begin by addressing structural changes in industrial research and development, the relationship between R&D and innovation, the apparent increase in intra-and intersectoral partnerships and alliances, and claims that interdisciplinary research is increasing. SRS should examine the costs and benefits of administering the Survey of Industrial Research and Development at the line of business level. SRS should also revise its surveys to address new concepts (e.g., the federal science and technology budget), discrepancies in results among R&D surveys, and the need to obtain better data on academic R&D facility costs.

SRS should improve the accuracy and augment range of its data on industrial R&D and innovation. Currently the Survey of Industrial Research and Development (RD-1), which is administered at the firm level, attempts to disaggregate both applied research and development by asking respondents to distribute these by product group. Firms, however, often ignore this question and the low response rate to product group has made the collected data of little use. We recommend eliminating the product group question from RD-1. As an alternate strategy for obtaining finer detail on industrial R&D, SRS should examine the costs and benefits of administering RD-1 to business units instead of firms. Currently all R&D conducted by a firm is attributed to the firm's predominant industrial category. In an economy dominated by large, multiproduct firms, line of business reporting, if feasible, may improve data by obtaining finer detail by industrial classification and geographic location. Also, current R&D expenditure data do not provide adequate information on many activities contributing to innovation. SRS should pursue plans to develop a survey of industrial innovation that addresses the manner in which science and technology are transferred among firms and transformed into new processes and products. SRS should include potential respondents and data users in the development of the survey instrument. SRS should also sponsor research on the nature of

R&D in the service sector; the web of partnerships among firms, universities, and federal agencies and laboratories in conducting R&D; and the extent and role of multidisciplinary research in science and engineering. Results of these investigations should guide SRS in revising its surveys to obtain more complete detail on their role in the science and engineering enterprise. SRS should take steps to better support analysis of the "federal science and technology budget" (FS&T) by requesting that the Department of Energy and the National Aeronautics and Space Administration specify the FS&T portion of their aggregate budget and obligation figures as does the Department of Defense. SRS should continue to investigate and reconcile discrepancies in R&D funding data obtained by its different surveys that hamper analyses of federal R&D funding. It should also complete proposed changes to its Survey of Scientific and Engineering R&D Facilities at Colleges and Universities to provide better data for assessing overhead rates at research universities and estimating future academic infrastructure needs.