5
Conclusions and Recommendations

CHOICE AMONG DESIGN OPTIONS

The committee concludes that the 2000 census frame option is the best choice among the available design options.

Recommendation 1: Almost all of the resources allocated to the SESTAT data collection effort in 2003 should be devoted to drawing a new National Survey of College Graduates from the 2000 census and supplementing this panel with the National Survey of Recent College Graduates.

Despite the committee’s reservations about combining data from a 1990 panel and 2000 panel, we conclude that it would be valuable to try to learn more about the nonsampling errors that necessarily creep into the system as the original NSCG sample ages. In particular, information about relative biases for the 1999 and 2003 samples is important for purposes of looking at trends across the two decades. Information about noncoverage and nonresponse biases would also help in the decision-making process for future redesigns.

However, the committee is skeptical that much can be learned about the causes of nonsampling error by simply comparing estimates from the old and new panels for the same time period. Although it is possible to



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 26
5 Conclusions and Recommendations CHOICE AMONG DESIGN OPTIONS The committee concludes that the 2000 census frame option is the best choice among the available design options. Recommendation 1: Almost all of the resources allocated to the SESTAT data collection effort in 2003 should be devoted to drawing a new National Survey of College Graduates from the 2000 census and supplementing this panel with the National Survey of Recent College Graduates. Despite the committee’s reservations about combining data from a 1990 panel and 2000 panel, we conclude that it would be valuable to try to learn more about the nonsampling errors that necessarily creep into the system as the original NSCG sample ages. In particular, information about relative biases for the 1999 and 2003 samples is important for purposes of looking at trends across the two decades. Information about noncoverage and nonresponse biases would also help in the decision-making process for future redesigns. However, the committee is skeptical that much can be learned about the causes of nonsampling error by simply comparing estimates from the old and new panels for the same time period. Although it is possible to

OCR for page 26
estimate the difference in the biases of the two designs by conducting surveys from the 1990 and 2003 frames simultaneously, exploring the causes of the biases in either design would be quite difficult. Information that is useful for the reduction of nonsampling errors is best obtained from special evaluation studies such as cognitive laboratory investigations, reinterview surveys, record check studies, and reliability analysis. Therefore, the committee questions the usefulness of continuing the old panel indefinitely as suggested in the hybrid option. As an alternative to continuing the old panel indefinitely, the committee suggests carrying forward a small sample of the old panel on a one-time basis if the SRS staff considers it important to do so for the following purposes: (1) using comparisons of the new and old panel estimates to generate hypotheses regarding the sources and causes of the nonsampling error that could then be explored in more detail using the special evaluation study methods noted above; and (2) using simple ratios of the estimates derived from the old and new samples in 2003 to adjust estimates from the old data series so that they can be compared with estimates from the new design. In this way, the biases in contrasts of estimates from the old and new data series due to survey design differences could be substantially reduced. Recommendation 2: If SRS staff confirm that a targeted sample could be useful for the purpose of adjustment, SRS should consider surveying in 2003 a small and carefully designed subsample of the current panel to study biases in the current sample, possibly to use for the purpose of adjustment. This sample could help direct efforts to evaluate nonsampling error in the new panel and would also provide a means to adjust the old data series for comparisons with the new data series. Recommendation 3: A cost-benefit analysis should be conducted to optimize the relative allocation of resources between the National Survey of College Graduates and the National Survey of Recent College Graduates. Also, additional oversampling should be applied to capture adequate numbers for small domains for which increased interest has become apparent since the last design.

OCR for page 26
EMPHASIS ON DATA QUALITY The committee urges emphasis on the quality of the 2000 census panel because there is no guarantee that the 2010 census will have a long form, nor that the American Community Survey will be implemented in a form that adequately supports the SESTAT database. Every effort should be made to prevent occurrence, in the new sample, of the flaws now present in the old sample. For example, it is possible that fast developing technology such as Internet communication can be used in an innovative way to provide feedback to respondents and greater incentive for them to remain in the survey. Of course, this and other enhanced follow-up efforts may increase the per-case costs of data collection. The survey should be designed to achieve the highest response rate possible subject to cost, timing, and data quality constraints. There are no absolute standards for response rates in the survey community. However, given that SESTAT involves surveys of a well-educated population about topics that are salient to them, an 85 percent (weighted) response rate is a reasonable goal and consistent with response rates routinely obtained by the Census Bureau in other surveys. One difficulty will be finding persons enumerated in the 2000 Census after 3 or more years. Assuming a 90 percent response rate for those census persons who can be located for the survey (i.e., 90 percent combined contact/cooperation rate for locatable persons), the rate at which persons can be located would have to be around 95 percent to achieve an overall survey response rate of 85 percent. We believe this fairly ambitious goal is achievable. Recommendation 4: The SRS should make every effort to achieve a response rate of 85 percent or better for the recommended new sample and to retain the sample over time. Toward that end, the SRS should consider trading off sample size and, accordingly, accepting some additional degree of sampling variability to improve the quality of the data collected if promising concrete steps in that direction can be identified. Recommendation 5: SRS should conduct methodological research to evaluate the major sources of nonsampling errors, particularly nonresponse error, as well as methods for reducing their effects on the survey estimates.

OCR for page 26
COVERAGE OF RELEVANT POPULATIONS Although it is beyond the committee’s scope to review the appropriateness of SESTAT’s definition of the population of scientists and engineers, we recognize the valuable contributions of others working in fields that touch upon science and engineering, and that more can be learned about scientists and engineers if information is available to permit comparisons against the population of workers as a whole. Recommendation 6: SRS should pursue a plan to carry forward non-S&E individuals in the panel, including the fields of health, S&E education, and possibly other non-S&E fields that relate to S&E. The committee encourages efforts (including the seeking of funds) to also include non-S&E education and, if possible, other non-S&E fields. The committee agrees with SRS that immigrants are an important, undercovered, and increasing part of the S&E community, so it is important to try to include recent immigrants with foreign degrees in the sample. Recommendation 7: SRS should investigate productive means of reaching the goal of greater inclusion of immigrants in the sample, including, if feasible and productive, cooperative work with the Immigration and Naturalization Service. PLANNING EXPANDED SRS ACTIVITIES AND GREATER INTERACTION WITH COLLEAGUES The committee commends the SRS for completing the weights pertaining to longitudinal data. Recommendation 8: SRS should encourage utilization of the longitudinal nature of the SESTAT data. Looking ahead, the committee believes that taking certain steps now will help the SRS to plan for future SESTAT surveys over the 2000 decade and beyond. Recommendation 9: SRS should prepare a concise, clear, and complete statement of the goals for the SESTAT database. The statement of

OCR for page 26
goals should reflect the SRS’s understanding of priorities that take into account the conflicting interests that compete for resources related to the collection of the SESTAT data. SRS goals should include expanding and maintaining meaningful contact with academic colleagues and with users from all sectors, including academia, business, and government. Recommendation 10: SRS should regularly monitor the quality of the SESTAT data and perform more in-house research to foster increased intellectual curiosity, deep knowledge of the SESTAT data, and data-driven re-design activities. To this end, SRS should develop a well-planned agenda for subject matter analysis and for methodological research.