National Academies Press: OpenBook
« Previous: Evaluation of the Design Options
Suggested Citation:"Conclusions and Recommendations." National Research Council. 2003. Improving the Design of the Scientists and Engineers Statistical Data System (SESTAT). Washington, DC: The National Academies Press. doi: 10.17226/10571.
×

5
Conclusions and Recommendations

CHOICE AMONG DESIGN OPTIONS

The committee concludes that the 2000 census frame option is the best choice among the available design options.

Recommendation 1: Almost all of the resources allocated to the SESTAT data collection effort in 2003 should be devoted to drawing a new National Survey of College Graduates from the 2000 census and supplementing this panel with the National Survey of Recent College Graduates.

Despite the committee’s reservations about combining data from a 1990 panel and 2000 panel, we conclude that it would be valuable to try to learn more about the nonsampling errors that necessarily creep into the system as the original NSCG sample ages. In particular, information about relative biases for the 1999 and 2003 samples is important for purposes of looking at trends across the two decades. Information about noncoverage and nonresponse biases would also help in the decision-making process for future redesigns.

However, the committee is skeptical that much can be learned about the causes of nonsampling error by simply comparing estimates from the old and new panels for the same time period. Although it is possible to

Suggested Citation:"Conclusions and Recommendations." National Research Council. 2003. Improving the Design of the Scientists and Engineers Statistical Data System (SESTAT). Washington, DC: The National Academies Press. doi: 10.17226/10571.
×

estimate the difference in the biases of the two designs by conducting surveys from the 1990 and 2003 frames simultaneously, exploring the causes of the biases in either design would be quite difficult. Information that is useful for the reduction of nonsampling errors is best obtained from special evaluation studies such as cognitive laboratory investigations, reinterview surveys, record check studies, and reliability analysis. Therefore, the committee questions the usefulness of continuing the old panel indefinitely as suggested in the hybrid option.

As an alternative to continuing the old panel indefinitely, the committee suggests carrying forward a small sample of the old panel on a one-time basis if the SRS staff considers it important to do so for the following purposes: (1) using comparisons of the new and old panel estimates to generate hypotheses regarding the sources and causes of the nonsampling error that could then be explored in more detail using the special evaluation study methods noted above; and (2) using simple ratios of the estimates derived from the old and new samples in 2003 to adjust estimates from the old data series so that they can be compared with estimates from the new design. In this way, the biases in contrasts of estimates from the old and new data series due to survey design differences could be substantially reduced.

Recommendation 2: If SRS staff confirm that a targeted sample could be useful for the purpose of adjustment, SRS should consider surveying in 2003 a small and carefully designed subsample of the current panel to study biases in the current sample, possibly to use for the purpose of adjustment.

This sample could help direct efforts to evaluate nonsampling error in the new panel and would also provide a means to adjust the old data series for comparisons with the new data series.

Recommendation 3: A cost-benefit analysis should be conducted to optimize the relative allocation of resources between the National Survey of College Graduates and the National Survey of Recent College Graduates. Also, additional oversampling should be applied to capture adequate numbers for small domains for which increased interest has become apparent since the last design.

Suggested Citation:"Conclusions and Recommendations." National Research Council. 2003. Improving the Design of the Scientists and Engineers Statistical Data System (SESTAT). Washington, DC: The National Academies Press. doi: 10.17226/10571.
×

EMPHASIS ON DATA QUALITY

The committee urges emphasis on the quality of the 2000 census panel because there is no guarantee that the 2010 census will have a long form, nor that the American Community Survey will be implemented in a form that adequately supports the SESTAT database. Every effort should be made to prevent occurrence, in the new sample, of the flaws now present in the old sample. For example, it is possible that fast developing technology such as Internet communication can be used in an innovative way to provide feedback to respondents and greater incentive for them to remain in the survey. Of course, this and other enhanced follow-up efforts may increase the per-case costs of data collection.

The survey should be designed to achieve the highest response rate possible subject to cost, timing, and data quality constraints. There are no absolute standards for response rates in the survey community. However, given that SESTAT involves surveys of a well-educated population about topics that are salient to them, an 85 percent (weighted) response rate is a reasonable goal and consistent with response rates routinely obtained by the Census Bureau in other surveys. One difficulty will be finding persons enumerated in the 2000 Census after 3 or more years. Assuming a 90 percent response rate for those census persons who can be located for the survey (i.e., 90 percent combined contact/cooperation rate for locatable persons), the rate at which persons can be located would have to be around 95 percent to achieve an overall survey response rate of 85 percent. We believe this fairly ambitious goal is achievable.

Recommendation 4: The SRS should make every effort to achieve a response rate of 85 percent or better for the recommended new sample and to retain the sample over time.

Toward that end, the SRS should consider trading off sample size and, accordingly, accepting some additional degree of sampling variability to improve the quality of the data collected if promising concrete steps in that direction can be identified.

Recommendation 5: SRS should conduct methodological research to evaluate the major sources of nonsampling errors, particularly nonresponse error, as well as methods for reducing their effects on the survey estimates.

Suggested Citation:"Conclusions and Recommendations." National Research Council. 2003. Improving the Design of the Scientists and Engineers Statistical Data System (SESTAT). Washington, DC: The National Academies Press. doi: 10.17226/10571.
×

COVERAGE OF RELEVANT POPULATIONS

Although it is beyond the committee’s scope to review the appropriateness of SESTAT’s definition of the population of scientists and engineers, we recognize the valuable contributions of others working in fields that touch upon science and engineering, and that more can be learned about scientists and engineers if information is available to permit comparisons against the population of workers as a whole.

Recommendation 6: SRS should pursue a plan to carry forward non-S&E individuals in the panel, including the fields of health, S&E education, and possibly other non-S&E fields that relate to S&E. The committee encourages efforts (including the seeking of funds) to also include non-S&E education and, if possible, other non-S&E fields.

The committee agrees with SRS that immigrants are an important, undercovered, and increasing part of the S&E community, so it is important to try to include recent immigrants with foreign degrees in the sample.

Recommendation 7: SRS should investigate productive means of reaching the goal of greater inclusion of immigrants in the sample, including, if feasible and productive, cooperative work with the Immigration and Naturalization Service.

PLANNING EXPANDED SRS ACTIVITIES AND GREATER INTERACTION WITH COLLEAGUES

The committee commends the SRS for completing the weights pertaining to longitudinal data.

Recommendation 8: SRS should encourage utilization of the longitudinal nature of the SESTAT data.

Looking ahead, the committee believes that taking certain steps now will help the SRS to plan for future SESTAT surveys over the 2000 decade and beyond.

Recommendation 9: SRS should prepare a concise, clear, and complete statement of the goals for the SESTAT database. The statement of

Suggested Citation:"Conclusions and Recommendations." National Research Council. 2003. Improving the Design of the Scientists and Engineers Statistical Data System (SESTAT). Washington, DC: The National Academies Press. doi: 10.17226/10571.
×

goals should reflect the SRS’s understanding of priorities that take into account the conflicting interests that compete for resources related to the collection of the SESTAT data. SRS goals should include expanding and maintaining meaningful contact with academic colleagues and with users from all sectors, including academia, business, and government.

Recommendation 10: SRS should regularly monitor the quality of the SESTAT data and perform more in-house research to foster increased intellectual curiosity, deep knowledge of the SESTAT data, and data-driven re-design activities. To this end, SRS should develop a well-planned agenda for subject matter analysis and for methodological research.

Suggested Citation:"Conclusions and Recommendations." National Research Council. 2003. Improving the Design of the Scientists and Engineers Statistical Data System (SESTAT). Washington, DC: The National Academies Press. doi: 10.17226/10571.
×
Page 26
Suggested Citation:"Conclusions and Recommendations." National Research Council. 2003. Improving the Design of the Scientists and Engineers Statistical Data System (SESTAT). Washington, DC: The National Academies Press. doi: 10.17226/10571.
×
Page 27
Suggested Citation:"Conclusions and Recommendations." National Research Council. 2003. Improving the Design of the Scientists and Engineers Statistical Data System (SESTAT). Washington, DC: The National Academies Press. doi: 10.17226/10571.
×
Page 28
Suggested Citation:"Conclusions and Recommendations." National Research Council. 2003. Improving the Design of the Scientists and Engineers Statistical Data System (SESTAT). Washington, DC: The National Academies Press. doi: 10.17226/10571.
×
Page 29
Suggested Citation:"Conclusions and Recommendations." National Research Council. 2003. Improving the Design of the Scientists and Engineers Statistical Data System (SESTAT). Washington, DC: The National Academies Press. doi: 10.17226/10571.
×
Page 30
Next: References »
Improving the Design of the Scientists and Engineers Statistical Data System (SESTAT) Get This Book
×
Buy Paperback | $21.00 Buy Ebook | $16.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The past decade has demonstrated the utility of SESTAT, but the SESTAT design shows some deficiencies with respect to response rates, coverage of populations of interest, and its ability to support some useful analyses. To tackle those deficiencies, NSF has proposed three possible design options for improving the database and asked the National Research Council's Committee on National Statistics (CNSTAT) to form the Committee to Review the 2000 Decade Design of the SESTAT.

This is the report of that committee. It presents our understanding of the purposes and characteristics of the SESTAT, applies the criteria we believe are important for assessing design options for the database, provides our recommendation for the best approach to adopt in the 2000 decade, and offers our encouragement to NSF to pursue opportunities to improve the understanding of the numbers and characteristics of scientists and engineers in the United States.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!