Surveying the Nation's Scientists and Engineers: A Data System for the 1990s (NRC 1989)
Chapter 5: Priority Goals and Design Features of the System for the 1990s
5.1 NSF should continue to be the lead agency within the federal government for providing comprehensive data on the science and engineering personnel resources of the nation. NSF must undertake to provide the budget and staff resources and institutional support necessary to develop and maintain a personnel data system that will adequately meet the needs of the 1990s and beyond.
5.2 Other federal agencies will continue to collect data in support of their own missions that pertain to science and engineering personnel. In order to enhance data comparability and utility to the extent practicable and to reduce duplication of effort and costs, NSF should play the lead role in coordinating federal data programs on scientists and engineers. Within the framework of established federal classification schemes, NSF should encourage standardization of key questionnaire items and classification variables for science and engineering personnel across agencies.
5.3 Currently, the primary goal of the NSF data system is to provide information on the characteristics of science and engineering personnel in order to support the planning processes of government, academic, and business institutions. In the 1990s, the data system should continue to serve this goal. Specifically, the system should
Support the preparation of regular profiles of the characteristics of scientists and engineers, including their numbers, employment patterns, qualifications, utilization, and other characteristics, with separate tabulations by field, sex, and race, when feasible, and
Support the preparation of special analyses that illuminate specific policy issues and characteristics of science and engineering personnel in greater depth
In the 1990s, the data system should also serve other important goals to which NSF does not currently accord high priority:
Provide a research base for improved analysis of relevant labor markets and of flows into, out of, and within the science and engineering labor force that can pinpoint trouble spots and provide early warnings of future problems, and
Provide a database that will support basic innovative research on scientists and engineers and the science and engineering pipeline.
5.4 NSF should provide information about the full range or people who can be considered as part of the science and engineering supply. NSF should furnish information on the population of graduates in science and engineering fields, not all of whom have related work experience. NSF should also furnish information on the population of employed scientists and engineers, not all of whom were trained in science and engineering fields. NSF should discard the current screening algorithm as a means of defining the population. Instead, NSF should use definitions based on standard occupation and degree field categories, developing within these frameworks more richly detailed classifications of subgroups of scientists and engineers.
5.5 NSF should increase the research utility of the science and engineering personnel database by enriching the content of its surveys. NSF should assign priority to new or modified content items that will provide greater understanding of:
The kinds of work that scientists and engineers do and how their work is changing in response to changes in technology, organizational structure, and other factors;
The career paths that scientists and engineers follow and the factors that influence key transitions, including initial entry into the labor force, mobility across fields and sectors, and retirement; and
The productivity that scientists and engineers achieve and how their accomplishments relate to characteristics of their training, career moves, and work environment.
5.6 NSF should conduct a large Postcensal Survey of College Graduates in 1992 based on the 1990 decennial census that provides baseline information on college graduates, including those who are trained in science and engineering fields and those with employment in science and engineering occupations.
5.7 NSF should conduct a Panel Survey of Scientists and Engineers that periodically provides updated information on the population of college graduates with science and engineering degrees and that tracks the 1992 cohort of graduates with employment in science and engineering occupations, including those who were trained in other fields. The survey should also include a sample for each new graduating class that is drawn from the Prospective Graduates Surveys conducted each year of students at higher education institutions who are about to receive a bachelor's or master's degree in a science or engineering field.
5.8 NSF should continue to support the ongoing Survey of Doctorate Recipients and employ it as the major source of information on science and engineering personnel trained to the Ph.D. level. The SDR should be modified to facilitate its use with the other surveys in the NSF science and engineering personnel data system.
5.9 To the extent possible during the decade, NSF should use other federal data sources to obtain information on components of the science and engineering population that are not covered in the NSF survey system and to evaluate NSF's survey-based estimates.
5.10 Because of the importance of degree field in defining the population of scientists and engineers, the Current Population Survey should periodically include a supplement that asks respondents for major field of bachelor's and higher degrees. NSF should work for the adoption of this recommendation by the Bureau of Labor Statistics and the Census Bureau.
5.11 NSF should pursue its planned research program to develop estimates of immigration and emigration of scientists and engineers and to develop ways of incorporating such estimates into the personnel data system.
5.12 NSF should consider the study panel's recommended design for its science and engineering personnel data system in the 1990s as a package in which basic information on the population of scientists and engineers, detailed information on topics and subgroups of key analytic interest, and evaluation and augmentation of NSF's own survey estimates using other federal data sources are integral and equally important elements.
Chapter 6: Designing and Implementing the System for the 1990s
6.1 In order to conserve resources and reduce burden on higher education institutions, NSF and the National Center for Education Statistics should design a unified sampling frame and coordinate procedure for obtaining data on prospective college graduates. The two agencies should not combine their panel surveys of new graduates, however, which serve different purposes and focus on different fields of training. Key questionnaire items should be comparable in order to permit each agency to evaluate and supplement its own data with the data from the other agency.
6.2 NSF should initiate modifications to the Survey of Doctorate Recipients, specifically, in the areas of coverage, survey scheduling, sample design, and wording of key questionnaire items, that will improve comparability of the SDR data with other data in the NSF science and personnel data system.
6.3 Detailed specification of the design for the NSF personnel data system in the 1990s will require additional analysis and decision making. NSF should in the near term set a process for reaching final design decisions. This process should include:
Identifying and funding priority research and analysis projects whose results are needed to inform the design;
Establishing a group of technical experts to work with NSF staff in reaching final design decisions and to assist NSF in monitoring the operation of the system in the 1990s; and
Sponsoring workshops and in other ways seeking both to obtain input from users and to advise them of impending changes in the data system.
6.4 Toward the end of the 1990s, NSF should conduct a thorough, zero-based evaluation of the design and operation of its personnel data system to determine whether to continue the basic design of the 1990s or to change the system in important ways, The evaluation should include a review of the goals of the system and the extent to which the informational content is serving those goals.
Chapter 7: Operating the System in the 1990s
7.1 NSF should develop a quality profile for its personnel surveys that will guide the development of an effective system to monitor and maintain data quality and suggest research to learn more about sources of error in the data and to identify further possible improvements.
7.2 NSF should take advantage of the experience of other federal statistical agencies in developing quality profiles, setting quality standards, and implementing quality control programs. NSF should keep abreast of procedures and techniques that federal agencies and private survey research centers use for improving data quality, particularly of data from continuing panel surveys.
7.3 NSF should devote a significant portion of its budget each year for the personnel data system to quality review and improvement activities.
7.4 When faced with budget constraints that necessitate trade-offs, NSF should choose options for the system that minimize total error in the data, taking into account both sampling error and nonsampling error from sources such as nonresponse.
7.5 NSF should adopt the best survey practice in designing and evaluating questionnaires for its science and engineering personnel surveys.
7.6 NSF should adopt the best survey practice for its personnel surveys in the following operational areas:
Procedures for obtaining high levels of response, both through initial contact and follow-up;
Procedures for data preparation, including developing appropriate weights, imputing missing values, and editing the data for consistency.
7.7 NSF should provide resources to the Science Resources Studies Division for staff training in survey methodology and for staff to attend conferences, short courses, and other venues of continuing education. NSF should also provide resources for the staff to develop first-hand knowledge, through field visits and other means, of the many different kinds of scientists and engineers whose characteristics the personnel surveys are intended to measure.
7.8 NSF should provide the resources for the staff of the Science Resources Studies Division to have access to the personnel microdata. NSF should encourage the staff to use the data for analytical studies, particularly those that relate to data quality and methodology, and to present their findings at professional meetings and in professional journals.
7.9 NSF should include resources in its survey contracts for contractors to propose and carry through research related to understanding and improving data quality.
Chapter 8: Building a User Community
8.1 NSF should plan an extensive publication program from the 1992 Postcensal Survey, which will provide the first comprehensive look in a decade at the entire population of scientists and engineers and permit comparative analysis with other subgroups of college graduates.
8.2 As a major publication series form the continuing Panel Survey, NSF should regularly publish profiles of college graduates with science and engineering degrees that separately identify important subgroups to permit users to apply a narrow or broad definition of the population as suits their needs. Two basic tabulation series would be useful: one series that focuses on the current employment situation of people with degrees in particular science and engineering fields, and another that focuses on the educational background and work environments of science and engineering graduates who are employed in particular science and engineering occupations. NSF should also produce publications from the Panel Survey about the cohort of employed scientists and engineers (including people trained in other fields) identified in the 1992 Postcensal Survey and about new graduates in science and engineering fields.
8.3 In determining the categorization of degree field, occupation, and other variables in NSF tabulations, user needs for more information must be balanced against considerations of sampling error. NSF should set standards for the minimum size science and engineering field for which estimates will be published based on the error properties of its surveys. Conversely, NSF should seek meaningful ways to provide additional detail for larger science and engineering fields.
8.4 NSF should provide a variety of products from the personnel data system—printed reports, public use microdata files, and other computer-readable products—that serve the needs of the entire user community, ranging from those users who require a few specific numbers to those users who are engaged in extensive analysis.
8.5 NSF should implement the recommendations that are developed by the Committee on National Statistics from its recent effort to seek ways to improve research access to the Survey of Doctorate Recipients while protecting the confidentiality of individual replies.
8.6 NSF should provide complete documentation for all products made available from the personnel data system, including a comprehensive user's guide to accompany public use microdata files. Data file documentation and technical notes included in publications should emphasize the nature and likely magnitude of the errors in the data.
8.7 NSF should actively publicize the availability of public use microdata files and other products from its personnel surveys.
8.8 NSF should actively encourage and provide support to researchers for innovative studies of science and engineering personnel using survey microdata. NSF should consider for this purpose establishing a grants program to fund projects that use the personnel data.
8.9 NSF should actively solicit feedback from its users on the design, content, and quality of the data system, and on the content and format of data products. NSF should consider for this purpose establishing a user panel to provide input on a regular basis.