Using the American Community Survey for the National Science Foundation's Science and Engineering Workforce Statistics Programs (2008)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Executive Summary A s the sponsor of the National Survey of College Graduates (NSCG), the U.S. National Science Foundation (NSF) faces a series of chal- lenges as well as opportunities associated with the elimination of the decennial census long form that has served as the basis for locating college graduates for the NSCG. In response, NSF has proposed to sample from the list of respondents to the new American Community Survey (ACS), using criteria similar to those used in past NSCG surveys. The ACS now collects information that is roughly the same as that collected on the long form—the highest degree or level of school that the respondent has completed (Question 11, ACS-1), occupational and employment characteristics, and demographic characteristics. However, conversion to the ACS will not be an easy task. One drawback of the ACS is that this survey covers a smaller number of households in a given year than the long-form sample. The substitution of the ACS for the long form for sampling purposes is considered feasible because it is possible to identify a set of households that could serve as the sample frame for the NSCG by accumulating 2-3 years of ACS households. This issue, along with several others, needs to be resolved before the Census Bureau and NSF can shift from the long form to the ACS. To assist in identifying and resolving those issues, the NSF’s Division of Science Resources Statistics asked the Committee on National Statistics to form a committee to assess the benefits of the ACS to NSF. The com- mittee was specifically charged to conduct a workshop with the objective of studying the issues involved in replacing the decennial census long  

 USING THE AMERICAN COMMUNITY SURVEY form with the ACS as a sampling frame for the NSCG and other human resources surveys and, specifically, to consider options for survey design in this new environment. The workshop had the additional objective of identifying issues for the collection of field-of-degree information on the ACS with regard to goals, content, statistical methodology, data quality, and data products. Finally, the committee was asked to consider the rel- evance and adequacy of ACS products for meeting current and emerging data needs for NSF. This report responds to that statement of work. FINDINGS AND CONCLUSIONS There are several important mandated uses embedded in the legisla- tion that established NSF that direct the periodicity of the data and the kind of detail to be provided: updates are required every 2 years, and the main subpopulations of interest are females, minorities, and disabled people. In addition to these mandated uses, NSF in recent years has faced escalating demands for science and engineering (S&E) workforce data in response to such issues as globalization, competitiveness, the role of the S&E workforce in national economic growth, the dynamic nature of S&E workforce flows, and federal interventions to improve the health of U.S. science and engineering. Those uses require a robust collection of data on S&E workers. They also call for reexamination of the kind of data that have been collected in light of those uses. This rethinking takes place in the context of an integrated database— the Scientists and Engineers Statistical Data System (SESTAT)—that in­cludes data from three surveys that together offer a comprehensive picture of the S&E workforce. It is appropriate to reconsider elements of this system, particularly the design and content of the National Survey of Recent College Graduates (NSRCG), when data on the S&E workforce with field-of-degree information become available on a flow basis from the ACS. In these and other ways, the ACS is expected to have a major effect on the NSF’s S&E workforce statistics program. The committee finds that the upcoming changes for the NSF surveys are potentially very positive for NSF. These changes provide an oppor- tunity to improve the NSCG sample and to enhance the timeliness, fre- quency, and quality of the S&E workforce data. Yet the transition to the ACS as a sample frame and as a source of data on the S&E workforce will create some challenges. There are significantly larger margins of error in ACS estimates than in estimates from the decennial census because of smaller sample sizes even when estimates are cumulated over 5 years. On balance, however, the committee concludes that the replacement of the decennial census long form with the ACS offers an opportunity for realizing NSF goals and objectives for the SESTAT Program. 

EXECUTIVE SUMMARY  The use of the ACS will afford an opportunity to review the ­SESTAT Program. This review should proceed as a priority, but any changes that stem from this review should be approached gingerly, with careful plan- ning for transitioning to the new databases and a research program to underscore longer term revisions in the program. In that regard, the com- mittee lays out and discusses three options for a research program that would provide support for decisions on the future of the SESTAT Program: (1) focusing only on data for mandated reports, (2) continuing the current range of surveys and reports, and (3) a preferred program that develops a collection program that expands the ability to analyze the S&E workforce and provides a more comprehensive, longitudinal dataset on the nation’s S&E workforce. The committee could find no compelling rationale for eliminating any of the current components of the SESTAT Program. How- ever, the availability of the ACS with a field-of-degree question affords an opportunity for fine-tuning and redirecting the surveys. The timing of this study was such that the committee did not have the benefit of the results of a content test of the field-of-degree question for the ACS. On the basis of its examination of the potential benefits of the question, the committee concludes that an open-ended question would be more useful than a categorical one, but it is mindful that this version may not meet the evaluation criteria that have been established for the content test and may generate unacceptable costs for coding and editing. CONCLUSIONS The committee has carefully considered the four options for using the ACS as a sampling frame that were identified and presented to the com- mittee by NSF staff. The committee included in its consideration a fifth option that was offered in the workshop. • Current Approach: ACS data would be used once a decade to draw a new panel for the NSCG that would have the advantage of requiring the least amount of organizational change, meaning an easier transition. This option fails to take advantage of the yearly accumulation of ACS cases, which allows the Census Bureau to oversample rare groups (e.g., minorities) that were available on the long-form census samples, so the statistical error of the estimates for these groups of interest would increase. This option would continue the current peaks and valleys in the funding pattern in which a significant infusion of new funds is required once each decade to fund the replenishment of the sample. The committee concludes that replicating the current design is not an efficient way to use the ACS. 

 USING THE AMERICAN COMMUNITY SURVEY • Selective Updates:  The once-each-decade sample draw would be modified by using the ACS in later years of a decade to update the sample for certain domains whose coverage becomes problematic as the decade progresses (e.g., recent immigrants) or for popula- tions of emerging interest. Although this option would maintain the currency of the sample by preventing coverage losses and allowing gathering data on emerging issues, it requires a peri- odic major redesign, calls for the expensive draw of a very large sample periodically from the ACS and may create data series dis- continuities. The committee concludes that the disadvantages of the selective update design outweigh the potential advantages. However, selective subsamples could be considered to supple- ment another design to enable the study of subpopulations of emerging interest. • Continuous Sample Updating:  A fresh sample from the ACS would be selected each time the NSCG is conducted, or, at least, more frequently than once a decade. This option would maintain the currency of the NSCG sample, permit oversampling of emerg- ing or special interest populations during the decade, prevent dis- continuities in the estimates, support trend analysis, and smooth out the NSF budget cycle. However, it would impose a burden on the ACS by requiring continuous access to the entire ACS sample for all years to derive the desired sample sizes for rare popula- tions. It would also reduce (or eliminate) the longitudinal feature of the ACS. The committee concludes that a freshly selected sample from the ACS each time the NSCG is conducted is not an efficient design, particularly for small populations. If rare populations were to be effectively studied, extensive and con- tinuous use of the ACS sample would be required, which might preclude use of the ACS for other survey purposes. • Rotating Sample Approach:  Select panels that represent the pop- ulation would be rotated through the NSCG sample frame. This approach incorporates all of the coverage advantages of the selec- tive updating option and has the additional advantage that the process of screening to identify scientists and engineers would be spread more evenly over time. There are obvious cost efficiencies in that replacing only a portion of the sample would smooth out data collection costs across time and avoid ballooning costs once a decade. However, this approach requires assured continuous access to the ACS sample for NSCG frame building, and the rotat- ing panels may suffer from sample attrition over time. The com- mittee concludes that the rotating sample approach is the most promising of all the NSCG design options and that a biennial 

EXECUTIVE SUMMARY  survey cycle with four or five rotating panels is the most effi- cient and cost-effective use of the ACS as a sampling frame. In addition, the committee considered a hybrid design that would implement a rotating design for rare populations only, while a cross- sectional strategy would be used for the more populated groups of inter- est. Under this option, it would be possible to accumulate a sufficiently large number of sample cases for relatively rare populations to produce precise estimates and to capture the strength of the large number of sam- ple cases to produce current estimates similar to those produced for other groups. However, there would be drawbacks: a limited ability to follow respondents over time and potential problems of panel conditioning over time. The committee concludes that a hybrid approach using a rotating design for rare populations would have the drawback of not keeping time-in-sample constant across subpopulations and thus might lead to differential levels of nonsampling bias across subpopulations. RECOMMENDATIONS There are exciting possibilities and promising venues of research and analysis of the science and engineering workforce that will be possible when the ACS with a field-of-degree item is made available. In this new environment, timeliness and efficiency gains will translate into a myriad of new opportunities that solve longstanding knowledge gaps. The commit- tee urges NSF to seize the new opportunities and offers two overarching recommendations along those lines as well as several recommendations that address specific issues associated with adding the field-of-degree question to the ACS, using the ACS as the NSCG sampling frame, and looking to the future when the ACS with a field-of-degree question is fully available for both sampling and analytical purposes. Overarching Recommendations Recommendation 7.5:  The National Science Foundation should use the opportunity afforded by the introduction of the American Com- munity Survey as a sampling frame to reconsider the design of the Scientists and Engineers Statistical Data System (SESTAT) Program and the content of its component surveys. Recommendation 7.6:  The National Science Foundation should conduct a careful assessment of internal and user priorities for studying the science and engineering workforce to capitalize on 

 USING THE AMERICAN COMMUNITY SURVEY the expanded analytical opportunities afforded by the addition of a field-of-degree question to the American Community Survey. Adding a Field-of-Degree Question to the ACS Recommendation 5.1:  The field-of-degree question on the Ameri- can Community Survey questionnaire should be the open-ended version if the Census Bureau and the National Science Foundation agree that it meets the evaluation criteria established for the content test and if an efficient coding procedure can be developed. Recommendation 5.2:  The National Science Foundation should ask the Census Bureau to conduct an additional evaluation of the field- of-degree question to assess the validity of the responses provided by respondents. As part of this evaluation, a sample of individuals should be reinterviewed to determine if they do have degrees in the fields reported. Using the ACS for the NSCG Sample Frame Recommendation 6.1:  The National Science Foundation should stipulate that the target population of people with bachelor’s degrees be defined as of the beginning of the American Commu- nity Survey year. Recommendation 6.2:  If the National Science Foundation wishes to consider continuation of the National Survey of College Graduates with the sample drawn from the American Community Survey, the agency should use a rotating panel design. Recommendation 6.3:  The National Science Foundation should work with the Census Bureau to develop plans for using the Amer- ican Community Survey as a sampling frame for a transitional period as well as for the continuing design. Recommendation 6.4:  The Census Bureau should use unswapped American Community Survey data (with sample weights) for draw- ing a National Survey of College Graduates sampling frame. Recommendation 6.5:  The National Science Foundation and the Census Bureau should initiate a program of research on imputa- tion and nonresponse treatment for missing field-of-degree and education-level responses. 

EXECUTIVE SUMMARY  Recommendation 6.6:  The National Science Foundation and the Census Bureau should sponsor a research program to explore means of permitting a sample draw from the American Community Survey for a rotation panel for the National Survey of College Graduates while preserving American Community Survey sample units for other surveys. The ACS and SESTAT in the Future Recommendation 7.1:  The National Science Foundation should use current data from the American Community Survey to evaluate the degree to which the American Community Survey with the field- of-degree question would allow for the production of mandated indicator reports in the future. Recommendation 7.2:  If the American Community Survey is selected to produce indicator reports, the National Science Founda- tion and the Census Bureau should develop a supplemental pro- gram of special, targeted surveys to obtain information on topics and groups of interest. Recommendation 7.3:  The National Science Foundation and the Census Bureau should consider establishing a continuing experimental panel program to support testing and development of techniques and methods for the National Survey of College Graduates. Recommendation 7.4:  The National Science Foundation should sponsor the development of a matched sample of American Com- munity Survey and National Survey of College Graduates respon- dents for research purposes with access provided to researchers through the Census Bureau’s Research Data Centers.