Using the American Community Survey for the National Science Foundation's Science and Engineering Workforce Statistics Programs (2008)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

5 Adding a Field-of-Degree Question to the ACS I n the past, the National Science Foundation (NSF) was constrained to using the decennial census long form to identify the sample for the National Survey of College Graduates (NSCG) for the upcoming decade as well as for analytical purposes. The salient question from the decennial census was the question on educational attainment, which when combined with other information, such as occupation, age, sex, and racial and minority status, allowed the selection of the sample members for the initial NSCG sample for the decade. People with bachelor’s or higher degrees were brought into the NSCG sample frame. The collection of information on all persons with bachelor’s degrees or higher has provided both the group of people who have science and engineering (S&E) degrees and those who do not have S&E degrees who could be asked questions about whether they worked in S&E occupations during the decade. It is important to obtain information about this group, but the need to query everyone with a bachelor’s or higher degree to identify the S&E workforce has been inefficient and has resulted in higher than necessary costs. Seizing on the opportunity afforded due to the mandatory conversion from the decennial census long form to the American Community Survey (ACS), the leadership of NSF has proposed adding a new question on the ACS that would ask respondents to identify their field of degree. This question would enable NSF to more efficiently use it to draw a represen- tative sample of all persons with S&E training at the bachelor’s or higher level, thus making them directly eligible for inclusion in the Scientists and Engineers Statistical Data System (SESTAT) target population. 41 

42 USING THE AMERICAN COMMUNITY SURVEY The Census Bureau has been able to be somewhat more flexible in including new or revised questions on the ACS than they were on the census long form. However, due to the relative newness and the ACS’s large size, decisions on adding or changing questions are not taken lightly. The Census Bureau has an extensive program of testing and refinement of potential questions and question wording in a content test program that has been a staple of the ACS since its inception. For example, the results of the content tests in 2007 will determine the content for the 2009 ACS. Before NSF can benefit from the potential sampling efficiency and lower costs of various future designs for the NSCG, the field-of-degree question must be subjected to development and testing. The committee has observed the process of development and testing of a field-of-degree question and assumes, based on current evidence, that there will be a question added to the ACS which collects field-of-degree information. Based on that assumption, this chapter summarizes the central issues in the decision as to whether the field-of-degree question should come with specified categories or be open-ended and discusses the need to systematically test the actual responses to this question when it is implemented in order to understand the validity of the data. The addition of the field-of-degree question is a rare and major opportunity that should be approached with careful planning. QUESTION DEVELOPMENT The ACS now collects data on the highest degree or level of school completed using the question shown in Box 5-1. The inquiry appears as Question 11 on the ACS “persons” questionnaire. The response catego- ries range from “no schooling completed” to professional and doctoral degrees. The use of the highest degree or level of schooling question as a screening question was the first and easiest decision. To avoid unneces- sary respondent burden and ensure data quality, the field-of-degree ques- tion would be asked only of the group of most interest, which would be most likely to provide usable information. Thus, the proposal is that only those who answer “bachelor’s degree” or higher (master’s, professional, or doctoral) would be asked about field of degree. A more complex decision concerns the design of the field-of-degree question itself. A basic tradeoff in gathering information on field of degree is that the more detailed the information, the better that samples can be allocated to domains of interest, but the higher the cost in terms of time, the greater the potential loss of data quality. Mindful of these trade­ offs, NSF and the Census Bureau have developed and are testing two 

ADDING A FIELD-OF-DEGREE QUESTION TO THE ACS 43 BOX 5-1 The ACS Highest Degree Question 11. What is the highest degree or level of school this person has COMPLETED? Mark (X) ONE box: If currently enrolled, mark the previous grade or highest degree received. No schooling completed Nursery school to 4th grade 5th grade or 6th grade 7th grade or 8th grade 9th grade 10th grade 11th grade 12th grade—NO DIPLOMA HIGH SCHOOL GRADUATE—high school DIPLOMA or the equivalent (for   example: GED) Some college credit, but less than 1 year 1 or more years of college, no degree Associate degree (for example: AA, AS) Bachelor’s degree (for example: BA, AB, BS) Master’s degree (for example: MA, MS, MEng, MEd, MSW, MBA) Professional degree (for example: MD, DDS, DVM, LLB, JD) Doctoral degree (for example: PhD, EdD) SOURCE: National Science Foundation (2007). variants to the question—one with a categorical or forced-choice design and another with an open-ended design. This research, testing, and development program has been mounted quickly. It involves multiple venues and multiple methodologies and test populations. The final results, which were not available to the panel at the time this report was prepared, will have far-reaching impact on the avail- ability and quality of data for sampling and analytical purposes. COGNITIVE RESEARCH The issues involved in the selection of the proper question format for the field-of-degree question have been carefully studied by three indepen- dent groups of researchers who recently conducted a series of coordinated 

44 USING THE AMERICAN COMMUNITY SURVEY experiments that assisted in the development of the two versions (cat- egorical and open-ended) of the questions tested in the 2007 test (Dillman et al., 2006a, 2006b; Cobb, Krosnick, and Bannon, 2006; Rothgeb and Beck, 2007). All the investigators ran into difficulties with questionnaire design and wording in one form or another but they persevered to develop plau- sible field of degree items. The results of these cognitive research efforts led to the selection of question formats and wording that were chosen for the 2007 ACS content test. The questions included are shown in Box 5-2 and Box 5-3. Asking the field-of-degree question using a list of categories requires a respondent to first accurately recall a major and then to map it into the broad list of categories offered in the question. This action involves an understanding of the categories and a link to the respondent’s major field of study. A failure at any point in this cognitive process could result in a misclassification error. The easiest way to avoid errors by respondents is to use the open-ended question. The open-ended question allows the respondent to name their major field of study and the responses are coded according to the agency’s BOX 5-2 Categorical Field-of-Degree Question This question focuses on this person’s BACHELOR’S DEGREE. In which of the following major fields did this person receive his/her BACHELOR’S DEGREE(S)? Mark (X) “Yes” or “No” box for each category. Yes No a. Biological, Agricultural, Physical, Earth, or Other Natural Sciences l__l l__l b. Health, Nursing, or Medical Fields l__l l__l c. Engineering, Computer Sciences, or Mathematical Sciences l__l l__l d. History, Arts, or Humanities l__l l__l e. Psychology, Economics, or Other Social Sciences l__l l__l f. Business or Management l__l l__l g. Education or Education Administration l__l l__l h. Some other major field - Specify l__l l__l ______________________________________________________ ______________________________________________________ SOURCE: National Science Foundation (2007). 

ADDING A FIELD-OF-DEGREE QUESTION TO THE ACS 45 BOX 5-3 Open-Ended Field-of-Degree Question This question focuses on this person’s BACHELOR’S DEGREE. Please print below the specific major(s) of any BACHELOR’S DEGREES this person has received (for example: chemical engineering, elementary teacher education, or- ganizational psychology). _________________________________________ _________________________________________ SOURCE: National Science Foundation (2007). criteria. The open-ended question also has the advantage of providing much more field-of-degree detail than the categorical question. Balanced against these advantages, the open-ended version requires expert coding. Recommendation 5.1: The field-of-degree question on the Ameri- can Community Survey questionnaire should be the open-ended version if the Census Bureau and the National Science Foundation agree that it meets the evaluation criteria established for the content test and if an efficient coding procedure can be developed. CONTENT TEST The Census Bureau has a formal process for testing proposed new content for the ACS. Through the Office of Management and Budget Interagency Committee on the ACS, the Census Bureau includes subject- matter experts and key data users from other federal agencies in iden- tifying questions for inclusion in a content test. In general, a content test evaluates alternatives for questions that show some indication of a potential problem, such as high rates of missing data, estimates that dif- fer systematically from other sources of the same information, or high sample nonresponse. In addition, a content test includes testing of new topics proposed by other federal agencies for inclusion in the ACS. The 2007 test of the field-of-degree question options was suggested and sup- ported by NSF. The 2007 field-of-degree content test was designed to test the ques- tions across the three modes of ACS data collection: mail, computer- 

46 USING THE AMERICAN COMMUNITY SURVEY assisted telephone interviewing (CATI), and computer-assisted personal interviewing (CAPI). The test questionnaire was mailed to 15,000 housing units for each of the two versions under consideration, and nonresponse follow-up was by telephone and, when necessary, in person. The field- of-degree coding system for the NSCG, which used autocoding, clerical coding, and expert coding, was adapted for the test. For quality assurance purposes, a content follow-up reinterview of a sample of the interviews was conducted to assess the reliability of the responses. In the reinterview survey, interviewers contacted respondents by telephone, attempted to speak to the original respondent, and repeated the field-of-degree questions. The Census Bureau has specified evaluation measures and decision criteria for assessing the results of the content test of the field-of-degree questions. They include comparability to other data sources, the rates of missing data, reliability, the agreement or correspondence between the versions, and departures from the current NSCG sample frame (personal communication, Jennifer Tancreto, U.S. Census Bureau). In comparing the versions across these criteria, the Census Bureau is using a decision tree that assigns most weight to the comparison with the NSCG, then the item missing data rates and reliability considered together, then the correspondence between the versions, and, finally, an assessment of the impact on the NSCG sampling frame. NSF identified several key issues that need to be resolved regard- less of the question version that is chosen (National Science Foundation, 2007, Table 4, p. 23). Some issues will affect the version that is chosen; others will affect the use of the data for sampling or analysis. There are six issues: 1. Space on the ACS  The categorical version requires approximately one-third of a column. The open-ended version requires less than one-fourth of a column. 2. Coding After Collection  The categorical version requires only limited postdata collection coding, although nonsampling error could be added by incorrect coding of the “other-specify” item. The open-ended version requires extensive, ongoing coding, which may delay data processing and final delivery time. and non­sampling error could be added by incorrect coding of the open-ended response(s). 3. Number of Fields Available  The categorical version limits the num- ber of fields to seven, with one residual category. The open-ended   The preliminary results of the content test were not available in time to be incorporated in this report. 

ADDING A FIELD-OF-DEGREE QUESTION TO THE ACS 47 version makes it possible to develop a more extensive list of fields (for analysis and NSCG sampling) than with the categori- cal version, but the possibility will depend on the level of detail provided by respondents. This is because the initial basis for the development of the coding will be the code list and coding pro- cedure used for the 2003 SESTAT survey. The field-of-degree code list for the SESTAT surveys has 144 field-of-degree codes. This list, however, has far more detail for S&E and S&E-related degree fields than for non-S&E fields. For the full evaluation of the ACS data, a greater level detail for non-S&E fields would be desir- able. (NSF will be working with the Census Bureau and other agencies to develop such a list. Ultimately, this list may become useful not only for NSCG sampling, but also for analysis of the field-of-degree data from the ACS if the open-ended version of the question is fielded.) 4. Reporting of Multiple Bachelor’s Degrees  The categorical version of the question explicitly allows for the reporting of multiple bach- elor’s degrees. The open-ended version question stem indicates that more than one degree may be reported, but it is not clear that respondents will do so. 5. Type 1 Errors  Reducing type 1 errors (i.e., checking or writing having an S&E or S&E-related bachelor’s degree when a non-S&E field-of-degree is appropriate) can have a major effect for NSCG sampling because this type of error would lead to unnecessarily sampling a case that does not have a required degree. However, this can easily be resolved during the NSCG data collection, when the case can be identified as ineligible. Type 1 errors will cause a larger problem for analysis of ACS data because there is no other information on the ACS to validate the field of degree. 6. Type 2 Errors  Reducing type 2 errors (i.e., not reporting an S&E or S&E-related field-of-degree while actually having one) is a special challenge because this type of error will lead to population under- coverage for the NSCG. Additionally, as with type 1 errors, it will cause problems for direct analysis of the ACS field-of-degree data. Working with the Census Bureau, NSF needs to attempt to estimate the prevalence of these errors in a structured research program. The form of the question on field of degree and the accuracy of the information provided will affect the gains in efficiency. For example, it is unclear how accurate reports on the field-of-degree item will be for those reporting for others in the household (proxy reports) compared with those reporting for themselves. If the field-of-degree and occupa- 

48 USING THE AMERICAN COMMUNITY SURVEY tion items can be used to accurately distinguish scientists and engineers from other college graduates, substantial gains in efficiency are possible. For NSCG sampling purposes, the most important concern is whether a degree is accurately reported as falling into an S&E, an S&E-related, or a non-S&E category. VALIDITY TESTING Analysis of the reinterviews in the 2007 test of the two field-of-degree questions will provide some information about the reliability of the responses to the item, but it cannot provide much in the way of insight into the validity of the data. There is a heightened risk in proceeding from the content test to full data collection without the benefit of a validity study of the question version that has been selected. Under current plans, the accuracy of the field-of-degree reporting will not be known until the first NSCG is conducted using a complete or partial sample from the ACS. The information from the detailed education histories that are collected as part of the NSCG, which does not have proxy reporting, can be compared with the information reported on the field-of-degree and educational attainment questions in the ACS. If it is determined that proxy reporting may lead to a quality degradation of the scientist and engineer data from the ACS, the Census Bureau is urged to conduct research on this topic as part of the validation study program. The panel believes it would be advantageous to assess the validity of the responses using the NSCG questionnaire and procedures prior to the initial fielding of the NSCG based on an ACS sampling frame. If this is not possible, then it may be advisable in drawing the first NSCG sample from the ACS to allocate part of the sample to test the efficiency of the field-of-degree item for sampling purposes. This could be done either by drawing a larger number of apparently non-S&E cases than might be done otherwise or by drawing a portion of the sample using procedures like those used with the long-form sampling frame, i.e., procedures that do not take the field-of-degree information into account. Recommendation 5.2: The National Science Foundation should ask the Census Bureau to conduct an additional evaluation of the field-   In the categorical version of the question that is now being tested, only one set of S&E- related fields (health) can be captured accurately. To identify a sample in other S&E-related fields, NSF would have to sample some of the non-S&E field-of-degree categories, as well as some non-S&E occupations. For example, to find individuals with degrees in science or mathematics teacher education (an S&E-related field), it will be necessary to sample some individuals with bachelor’s degrees in “education or education administration” as well as some secondary teachers. 

ADDING A FIELD-OF-DEGREE QUESTION TO THE ACS 49 of-degree question to assess the validity of the responses provided by respondents. As part of this evaluation, a sample of individuals should be reinterviewed to determine if they do have degrees in the fields reported. Information about the accuracy of the field-of-degree responses will be helpful in future planning for the NSCG sample. Some number of cases apparently not meeting the criteria of being a scientist or engineer (a non- S&E bachelor’s degree and a non-S&E occupation) will likely need to be drawn in any NSCG sample from an ACS sampling frame both to provide a comparison group and to account for those in non-S&E occupations with a non-S&E bachelor’s degree but an S&E or S&E-related degree at a higher level. Knowledge of error rates for the field-of-degree questions will help NSF and the Census Bureau determine how many such cases would be required. There may be additional benefits to having the field-of-degree ques- tion beyond its immediate help in making the sample more efficient. The field-of-degree question, enhanced by the use of outside information, might, over time, help sharpen the definition of the target population for the survey. This possibility is discussed further in Chapter 7.