Executive Summary
SESTAT (the Scientists and Engineers Statistical Data System) is a system of surveys, developed and maintained by the Science Resources Statistics (SRS) division of the National Science Foundation (NSF) to provide timely information about the numbers and characteristics of scientists and engineers in the United States.
The committee examined three options presented by SRS to improve the SESTAT design: (1) conduct a new National Survey of College Graduates based on the 2000 census frame; (2) continue the existing 1990s panels; and (3) adopt a hybrid option, combining features of the 2000 census frame and the 1990 census frame options. Our primary criteria for evaluating these options were how well the design achieves and maintains an adequate response rate, how well it covers the complete population of interest, and whether sample sizes permit sampling precision that is adequate for the principal uses of the survey data. We also considered how well the designs support analyses of longitudinal data, trends, and biases. The committee was not asked, nor did it attempt, to perform an analysis of cost and implementation factors associated with various design options for SESTAT, examine the content of the surveys that support SESTAT, or revisit the question of how the population of scientists and engineers should be defined.
We conclude that the 2000 census frame option is the best choice of the three design options. We offer three primary recommendations:
Recommendation 1: Almost all of the resources allocated to the SESTAT data collection effort in 2003 should be devoted to drawing a new National Survey of College Graduates from the 2000 census and supplementing this panel with the National Survey of Recent College Graduates.
Recommendation 2: If SRS staff confirm that a targeted sample could be useful for the purpose of adjustment, SRS should consider surveying in 2003 a very small, carefully targeted subset of the current panel to study biases in the current sample, possibly to use for the purpose of adjustment.
Recommendation 3: A cost-benefit analysis should be conducted to optimize the relative allocation of resources between the National Survey of College Graduates and the National Survey of Recent College Graduates. Also, additional oversampling should be applied to capture adequate numbers for small domains for which increased interest has become apparent since the last design.
To emphasize the importance of achieving and maintaining high quality of the data, we offer a recommendation on the new sample:
Recommendation 4: The SRS should make every effort to achieve a response rate of 85 percent or higher for the recommended new sample and to retain the sample over time.
We encourage SRS to make every effort to prevent occurrence, in the new sample, of the flaws now present in the old and to conduct methodological research to evaluate the major sources of nonsampling errors, particularly nonresponse error, as well as methods for reducing their effects on the survey estimates. We also offer recommendations pertaining to increasing coverage of populations of interest (e.g., immigrant scientists and engineers and a broader range of fields relevant to science and engineering); refining the definition of goals for SESTAT; regularly monitoring SESTAT data quality; performing more in-house research at SRS; and developing an agenda for subject matter analysis and methodological research.