2
Methods

The committee was charged with addressing the study questions to the extent that they could be using available data from National Institutes of Health (NIH) supplemented by interviews with minority trainees and program administrators. This chapter describes the committee’s approach to obtaining and analyzing these data. It describes the extent to which the committee was successful in obtaining useful data, problems encountered along the way, and instances in which requested data did not materialize. Indeed the committee was not able to obtain all of the data it wished to have in order to conduct its assessment. Although it met its charge to the extent feasible, the committee could not answer all of the research questions in as direct and complete a manner as it would have liked, which it now advocates for the future. The advice provided in this report, if it leads to corresponding action, will both improve the programs in the short run and facilitate more comprehensive study in the future. Indeed, extensive data-collection efforts, ongoing deliberations, and analyses allowed the committee to identify critical data elements that should be collected by NIH on a systematic basis in order to make future assessments of all NIH research training programs feasible.

Definitions

The committee began its work by defining key terms used in its work that appear throughout this report. Three terms—biomedical, trainee, and underrepresented minority—stood out as central to the task and, to be clear, they are defined here.

The term “biomedical” is used broadly in this report. It is intended to include the biological, behavioral, and clinical sciences. In certain instances, such as when drawing from official program descriptions, the phrase “biomedical and behavioral sciences” is used.

The term “trainee” is used throughout the report to describe any person engaged in higher education science studies or training, with the intention of building a career in the biomedical sciences. Trainees may be undergraduates, graduate students, postdoctoral fellows, or junior faculty. The term trainee is also used to describe an individual who is or was a participant in any NIH-supported research training program.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 29
Assessment of NIH Minority Research and Training Programs: Phase 3 2 Methods The committee was charged with addressing the study questions to the extent that they could be using available data from National Institutes of Health (NIH) supplemented by interviews with minority trainees and program administrators. This chapter describes the committee’s approach to obtaining and analyzing these data. It describes the extent to which the committee was successful in obtaining useful data, problems encountered along the way, and instances in which requested data did not materialize. Indeed the committee was not able to obtain all of the data it wished to have in order to conduct its assessment. Although it met its charge to the extent feasible, the committee could not answer all of the research questions in as direct and complete a manner as it would have liked, which it now advocates for the future. The advice provided in this report, if it leads to corresponding action, will both improve the programs in the short run and facilitate more comprehensive study in the future. Indeed, extensive data-collection efforts, ongoing deliberations, and analyses allowed the committee to identify critical data elements that should be collected by NIH on a systematic basis in order to make future assessments of all NIH research training programs feasible. Definitions The committee began its work by defining key terms used in its work that appear throughout this report. Three terms—biomedical, trainee, and underrepresented minority—stood out as central to the task and, to be clear, they are defined here. The term “biomedical” is used broadly in this report. It is intended to include the biological, behavioral, and clinical sciences. In certain instances, such as when drawing from official program descriptions, the phrase “biomedical and behavioral sciences” is used. The term “trainee” is used throughout the report to describe any person engaged in higher education science studies or training, with the intention of building a career in the biomedical sciences. Trainees may be undergraduates, graduate students, postdoctoral fellows, or junior faculty. The term trainee is also used to describe an individual who is or was a participant in any NIH-supported research training program.

OCR for page 29
Assessment of NIH Minority Research and Training Programs: Phase 3 The term “underrepresented minority” is used narrowly, focusing on those groups that have suffered a historical pattern of discrimination in the United States and are also underrepresented in science. NIH has determined these groups to be African-Americans, Hispanics, Native-Americans and Alaskan Natives, and Pacific Islanders. For analytical purposes, however, the definition has been narrowed further. While Pacific Islanders are viewed by NIH as an underrepresented minority group, this study could not include them as such, because Pacific Islanders are aggregated with Asians across the data sets. Universe of Programs The NIH supports minority-targeted research training programs at all educational and career stages—high school, undergraduate, graduate, postdoctoral, and junior faculty. Determining which of these programs the committee would focus on, in order to meet its charge and to render the study manageable, involved several steps. First, the committee decided to focus exclusively on active extramural research training programs. Retired programs were not included, so that the assessment could focus on providing useful information about existing programs. Intramural minority programs were not included because NIH was conducting an internal assessment of those programs at the time this study began. The committee cannot speak to whatever results may have come from that effort. Second, the committee conducted a census of minority research training programs and established inclusion-exclusion criteria to determine the set of programs it would assess. Conducting a census of extramural NIH minority research training programs was a staff-intensive effort. National Center on Minority Health and Health Disparities (NCMHD) staff provided the National Research Council (NRC) with a comprehensive list of minority research training programs that served as a starting point for the census. This list was updated in 2001with help from NCMHD staff, the NIH extramural training officer, 23 institute or center (IC) liaisons who were appointed by their IC directors to assist the study, and numerous health science administrators across the NIH campus. Although most ICs responded positively to the committee’s request for program data, a few ICs did not respond and are therefore unaccounted for in terms of the minority programs they support. In addition, the committee searched through the NIH website for information on minority programs, focusing on specific IC websites, IC health disparity strategic plans, and NIH Guide program announcements. The resulting census listed 79 programs utilized to greater or lesser degrees by 17 of the 27 ICs at NIH.27 Some of these programs are duplicated across ICs, such as the minority (F31) National Research Service Award (NRSA) predoctoral fellowship award, which is supported by most ICs. Nonetheless, each instance of a program was counted in the tally of total programs regardless of how many ICs participated. 27   Specific census data are found in Appendix B.

OCR for page 29
Assessment of NIH Minority Research and Training Programs: Phase 3 TABLE 2-1 Reasons for Excluding Minority Training Programs from the Study Exclusionary Rationale Number of Programs Program was less than 5 years old in 2001 11 Program was for capacity building at minority-serving institutions (MSIs) 8 Program was less than 5 years old in 2001 and was for capacity-building at MSIs 4 Program funded conferences and workshops 2 Program focused on senior faculty outside the scope of study 1 Program was originally included but random sampling for trainee interviews produced no respondents from this program 1 Program information provided by the IC was incomplete 3 Program was outside of study scope because its goal is not to increase number of minority Ph.D.s in biomedical research and program information provided by the IC was incomplete 2 Total Excluded Programs 32 While there are 79 targeted minority programs across the ICs (inclusive of duplicated mechanisms), this study examines 47 of them. The committee established inclusion-exclusion criteria for the minority research training programs in order to reach this smaller number and then reclassified them into 13 unique program categories (i.e., without duplications) based on career stage served and mechanism number.28 In general, the minority research training programs included in the study were active extramural minority research training programs that had existed for five years or longer in 2001. Table 2-1 lists the reasons programs were excluded and the number of programs excluded for each reason. In brief, programs were included or excluded for two reasons. First, a set of programs were included in the study because they met criteria that established a significant benefit to assessing them and excluded if they did not. The following criteria were used: Extramural programs in existence for at least five years by 2001 were included and those in existence for less than five years at that point in time were excluded. The five-year period ensures that for each included program a cohort of program graduates would have had enough time to advance to the next step in their training. Extramural programs were included if participation in the program had the significant potential for causing trainee progress through the pipeline. Thus, minority-targeted travel and conference awards were excluded from the study, because of their short-lived nature and the difficulty of ascribing causality or assessing their impact on trainees’ progression through the pipeline. 28   Mechanism number refers to the letter-number designation among programs (e.g., F31, T32, T35, R03).

OCR for page 29
Assessment of NIH Minority Research and Training Programs: Phase 3 Second, programs were included if they met criteria that indicated adequate information was available on the program and if there were accessible electronic data on trainees. These criteria were as follows: The NRC needed sufficient data on the programs to understand their goals and structures. For a set of programs, the ICs at NIH did not provide sufficient information to the NRC to allow the committee to understand these programs and determine whether they should be included, so they were excluded from the study. Accessible electronic data on the program would be necessary. Thus, programs focused on minority high school students, capacity-building awards (institutional awards made primarily to minority-serving institutions), and minority supplementary awards were excluded because information on the trainees in these programs was not available to the committee in electronic form. In addition to determining which targeted programs to include, the committee identified programs that would provide for trainee comparison groups. These programs were those not targeted to a specific racial or ethnic group. To the extent possible, the nontargeted programs included in the study were matched closely with targeted programs, in terms of both their sponsoring institutes and the years during which the programs were offered. When this was not possible, the closest proxy was identified. However, there are situations in which a nontargeted comparison program does not exist, such as with all minority undergraduate training programs and the R03 Minority Dissertation Research Grant program. Table 2-2 lists the minority-targeted training programs that fall within the study parameters established by the committee and which were used to draw a sample of minority trainees. Nontargeted comparison programs are also listed in the table. From the nontargeted programs, two comparison groups were identified, which are also indicated in Table 2-2. The first comparison group is comprised of minority trainees who were trained in a nontargeted program. The second comparison group consists of trainees who do not fall within the study’s definition of underrepresented minority, hereafter referred to as “nonminority” trainees. For undergraduate trainees, there are no comparison groups because nontargeted undergraduate training programs do not exist. Consequently, the committee examined seven clusters of trainees: Undergraduate minority trainees in minority-targeted programs, Graduate minority trainees in minority-targeted programs, Graduate minority trainees in nontargeted programs, Graduate nonminority trainees in nontargeted programs, Postdoctoral/faculty minority trainees in minority-targeted programs, Postdoctoral/faculty minority trainees in nontargeted programs, and Postdoctoral/faculty nonminority trainees in nontargeted programs.

OCR for page 29
Assessment of NIH Minority Research and Training Programs: Phase 3 TABLE 2-2 Training Programs Examined in the Study Training Level Minorities in Targeted Programs Minorities in Untargeted Programs Nonminorities in Untargeted Programs Undergraduates (R25) Bridges to the Baccalaureate (T34) Undergraduate Student Training Academic Research (T34) Career Opportunities in Research   Graduate trainees (R25) Bridges to the Doctorate (R03) Dissertation Research Grant     (F31) NRSA Predoctoral Fellowship (T32) NRSA Institutional Research Training Grant (T35) NRSA Short-Term Institutional Research Training Grant (F31) NRSA Predoctoral Fellowship (T32) NRSA Institutional Research Training Grant (T35) NRSA Short-Term Institutional Research Training Grant (F31) NRSA Predoctoral Fellowship (T32) NRSA Institutional Research Training Grant (T35) NRSA Short-Term Institutional Research Training Grant Postdoctoral or faculty (T32) NRSA Institutional Research Training Grant (T32) NRSA Institutional Research Training Grant (T32) NRSA Institutional Research Training Grant   (T35) NRSA Short-Term Institutional Training Grant (T35) NRSA Short-Term Institutional Training Grant (T35) NRSA Short-Term Institutional Training Grant (K01) Mentored Research Scientist Development (Postdoctoral) (K01) Mentored Research Scientist Development (Postdoctoral) (K01) Mentored Research Scientist Development (Postdoctoral) (K08) Mentored Clinical Scientist Development (K08) Mentored Clinical Scientist Development (K08) Mentored Clinical Scientist Development (K01) Mentored Research Scientist Development (Junior Faculty) (K01) Mentored Research Scientist Development (Junior Faculty) (K01) Mentored Research Scientist Development (Junior Faculty) NOTE: NRSA = National Research Service Award

OCR for page 29
Assessment of NIH Minority Research and Training Programs: Phase 3 Approach to Data Collection The study committee sought data that would allow it to answer the questions in its charge and provide evidence to support findings and recommendations about the NIH minority research training programs included in the assessment. The committee identified four types of data that could provide input to the study: Existing data on science and engineering higher education and, more specifically, on NIH trainees in the biomedical sciences, Computer assisted telephone interviews (CATIs) with trainees who were supported by these NIH research training programs, Interviews with individuals who administer these programs at recipient colleges and universities; they are hereafter referred to as trainee “program administrators at recipient institutions” (PARIs), and Interviews with NIH staff who administer these programs on behalf of their NIH institute or center; they are hereafter referred to as “program administrators at the ICs” (PAICs). The committee decided to use all of these sources to the extent possible. The committee determined to mine existing data to the extent they were readily available in electronic data files. The committee proceeded to conduct simultaneous trainee, administrator, and staff interviews with the assistance of an NIH contractor. Existing Data Data Sources The committee identified the following six extant data sources that could be used assess the education or research outcomes of persons who received training funds from NIH: NIH Information for Management, Planning, Analysis, and Coordination (IMPAC) system. This file contains application and award information for many (but not all) extramural programs, including grants, contracts, and cooperative agreements. Grants and grantee data are extracted from the Statement of Appointment Form (PHS 2271); however, not all data elements can be reliably extracted and converted into electronic format. NIH Trainee and Fellows File (TFF). This file contains data on all individuals who received training support from NIH and other Public Health Service agencies since these programs began in 1938. The file contains information on fellows (i.e., recipients of individual awards) from the IMPAC system and on trainees from the Trainee Appointment File (TAF). The TFF can be linked with data files such as the Doctorate Records File (DRF) and the Association of American Medical College’s (AAMC’s) Medical School Graduation Questionnaire. NIH Computer Retrieval of Information on Scientific Projects (CRISP) collection. This file contains information on research projects and programs funded by the Department of Health and Human Services (DHHS) from 1972 to the present.

OCR for page 29
Assessment of NIH Minority Research and Training Programs: Phase 3 CRISP contains administrative data from IMPAC along with project abstracts and indexing terms. Doctorate Records File (DRF). The DRF is a census of research doctorate recipients from U.S. universities since 1920. Data on individuals receiving doctorates prior to 1958 was collected retrospectively. Since 1958, DRF data has been collected through the Survey of Earned Doctorates (SED) which obtains data on the social, demographic, and educational characteristics of new Ph.D.s and their plans for the year following graduation. AAMC’s Medical School Graduation Questionnaire. This file is a census of graduates of American medical schools that has been collected since 1978. It contains demographic characteristics, assessments of clinical experiences, and data on graduate satisfaction with the medical school experience. Prior assessments of minority-targeted programs conducted by ICs within NIH. The committee utilized four assessments of undergraduate programs—three by the National Institute of General Medical Sciences (NIGMS) and one by the National Institute of Mental Health (NIMH). In addition, the committee examined two assessments of graduate training programs both conducted by NIGMS. The committee viewed the TFF as an especially rich data source. The TFF contains historic data on more than 400,000 trainees who at one time or another received NIH funding. This file includes records on undergraduates, graduate students, and postdoctoral trainees, both for short courses and for multiyear and multicenter program projects. Importantly, the TFF can link multiple numbers and types of awards for recipients. It captures the following data elements reliably: type of grant, identification (ID) or serial number, name, gender, Social Security number, address, date of birth, citizenship, race or ethnicity, dates of award, award amount, and degree sought. In addition, the TFF can be linked to the DRF and the AAMC Medical School Graduation Questionnaire. Outcomes of Data Queries Although these data sets had the potential for providing the committee with a wealth of data, in actuality, the data available to the committee were of limited value and utility. First, the committee learned that the level of item nonresponse was very high for many of the variables in the TFF, including data on race or ethnic origin. For example, before 1992, nearly 75 percent of persons who received predoctoral funding from NIH were not classified by race or ethic origin in the TFF. Since 1992, the nonresponse rate for data on race or ethnicity of predoctoral trainees has been reduced to 14 percent. Consequently, even if the committee were able to access TFF files, data from early years could not be used to classify the vast majority of trainees by race or ethnic origin. There was also a high nonresponse rate for a number of other variables including gender, Social Security number, date of birth, degree sought, and permanent address. Finally, four data elements included on the paper form had not been extracted and entered into the TFF or

OCR for page 29
Assessment of NIH Minority Research and Training Programs: Phase 3 IMPAC II files, including name of institution, address and phone number of institution, name of program director, and specialty boards. Second, the committee learned early in its tenure that NIH interpreted the Privacy Act (P.L. 09-25-0112) as prohibiting the NRC study committee and staff from accessing unit record data from the TFF and any other database that contained individual trainee data because these data were deemed sensitive. Therefore, the committee was told that all data queries for trainee data, TFF data, and data linking TFF data to external data sources (e.g., the DRF) would necessarily be handled by an intermediary contractor designated by NIH. Although the committee understands and accepts the reason that NIH should and must protect the privacy of individual trainees, the requirement for an intermediary precluded some potentially fruitful avenues of study. During the information-gathering phase of the study, the committee developed a series of queries and requests for tabulation of TFF data and from external sources data that can be linked to the TFF. The committee then submitted these requests to the NIH data contractor. Unfortunately this process was far more difficult, cumbersome, and time-consuming than anticipated. The free flow of data and information that had been sought either did not emerge or the deliverables did not meet the committee's specifications. In some cases they were not provided at all. Early in its deliberations, the committee submitted eight queries, designed to assist the committee in developing instrumentation and framing the directions of additional inquiry, as follows: Provide aggregate tabulation of the racial or ethnic identity of the all persons who received initial predoctoral funding between 1970 and 1999 by year. Identify targeted undergraduates who received NIH funding and match these persons with the file of persons who received initial predoctoral funding by year. Match the targeted undergraduates with persons in the DRF and provide all DRF data. Identify the participants in the comparison programs and link them to DRF data by year of degree. Match targeted undergraduates who received NIH funding, with AAMC data on graduation from medical school by year of graduation. Match the file of persons who received initial predoctoral funding between 1970 and 1999 with AAMC data on medical school graduates by year of graduation from medical school. Match the participants in the six comparison programs with the AAMC medical school data by year of graduation from medical school. Match all persons who received minority-targeted K01 awards between 1970 and 1999 by year with the DRF database a retrospective search and then prospectively match with the CRISP database to identify existence of R01 awards. The committee did receive data on the ethnic identity of persons who received initial predoctoral funding between 1970 and 1999. However, because of the very high levels of missing ethnic-identity data, the committee was unable to use these data to assess whether NIH minority training and research programs work.

OCR for page 29
Assessment of NIH Minority Research and Training Programs: Phase 3 In addition, the committee obtained responses to five additional data requests: DRF data for persons who received initial predoctoral funding between 1970 and 1999, The number of undergraduate trainees who ever received NIH predoctoral funding, DRF data for undergraduate trainees, AAMC matched data for undergraduate trainees, and Data linking K01 awards and the DRF. The committee was disappointed with the results of the data queries. The DRF data on persons who received NIH predoctoral funding were interesting, but because the type and level of funding could not be identified, the results were of little value to the committee’s task. The committee did find that there were 103,970 persons who received NIH predoctoral funding. Of these, 41,438 matched against the DRF. Unfortunately, the NIH contractor did not use an updated version of the DRF in responding to the query. The number of matches between NIH predoctoral funding and the DRF fell off rapidly beginning with 1993. This corresponds roughly with NIH’s increase in developing funding vehicles that target minorities. Consequently, the committee found that the query did not contribute to its assessment. The outdated DRF data also plagued the results of the query that sought to match minority undergraduates participating in minority-targeted programs. Nevertheless, the match of undergraduate trainees with NIH predoctoral funding was promising—out of 6,614 targeted undergraduates, 3,914 received NIH predoctoral funding. However, when they were matched against the DRF, only 277 had received a Ph.D. and only 164 had received an M.D. degree as indicated by AAMC data. The committee believes that these results are in doubt in light of the contractor’s reliance on outdated DRF data. Finally, the committee was puzzled by the data linking minority-targeted K01 awards made between 1970 and 1999 with the DRF. Only 575 matches were found, and only 288 of K01 award recipients were also in the DFR. The remaining 287 K01 awardees presumably were M.D.s. The K01 and R01 match was never made. Interestingly, according to the DRF, about two-thirds of the 287 Ph.D.s who received these minority-targeted K01 awards were white. Following the completion of the four surveys, the committee developed a second series of nine data queries. This second series of queries was developed in order to clarify data issues generated by the four surveys of trainees. Gender, citizenship, and ethnic origin of the 5, 371 trainees included in the sampling plan for the four surveys, classified by program. Number of undergraduate trainees ever funded in seven defined programs and aggregated data on the gender, ethic identity, and citizenship by program and year.

OCR for page 29
Assessment of NIH Minority Research and Training Programs: Phase 3 Number of undergraduate trainees in the seven programs who ever received predoctoral funding, cross-classified by undergraduate program, predoctoral funding mechanism, and year of initial predoctoral funding. Undergraduate trainees from the seven programs matched against the DRF and classified by undergraduate program and year of initial undergraduate funding. Undergraduate trainees from the seven programs matched against AAMC data classified by undergraduate program and year of initial undergraduate funding. Predoctoral funding recipients between 1970 and 1999, classified by funding mechanism, whether the program was minority-targeted, and year of initial funding. Cross-classification of the listing above by gender and ethnic origin. Initial predoctoral funding recipients between 1970 and 1999, classified by mechanism number (i.e., K01, F31, T32, etc.) and year, matched against the DRF including all DRF data. Predoctoral funding recipients, classified by mechanism number and year, matched against AAMC data including data on age at graduation, gender, ethnic origin, years between matriculation and graduation, and type of medical program. At the time this report went into peer review, the committee had received answers to none of these queries because of the NIH sponsor’s considerable delay in processing contract paperwork that could have allowed the NIH data contractor to run the queries. As a result, some analyses of trainee characteristics and program outcomes that would have been very informative to the study were not completed and other analyses had to be tailored to the data provided rather than in response to the committee's needs. For example, the TFF to DRF match was not provided in the manner specified by the committee and efforts to work with the NIH contractor to create a data set that could be used by the committee did not materialize. Thus, the committee had to forego the use of outcomes data that might have provided more insight about individual and program success. The committee believes that more and better analyses of extant data would likely have emerged if the TFF and other trainee data could have been accessed directly or if the NIH data could have been supplied directly to the committee, bypassing a third-party contractor. The pattern of request and response between the committee and the NIH data contractor was slow and cumbersome. The ability to analyze data and tailor new queries based on that analysis was absent. The duration between a data request and the response from the NIH data contractor was unnecessarily long, sometimes taking many months for a single request. Consequently, the committee was unable to analyze much of the data that it had planned to use. The committee believes that a rich data resource, that could serve the NIH well in assessing the state of training and research of minorities, lies fallow.

OCR for page 29
Assessment of NIH Minority Research and Training Programs: Phase 3 Trainee Interviews The committee concluded that direct input from trainees would provide a rich source of information for assessing the value of minority training programs to individual participants and the features of these programs that facilitate trainee success. After considering alternative methods of data collection, the committee decided that trainees from each career stage (i.e., undergraduate, graduate, postdoctoral, and junior faculty) would be contacted through a CATI protocol. However, the protocol was complicated by the fact that experiences, attitudes, and outcomes for each career stage would be divergent, thus different questions and response patterns were required. The committee therefore elected to prepare four distinct CATI instruments, each of which was tailored to a specific career stage. The committee also determined that the study would include only trainees whose initial year of funding fell between 1970 and 1999; there were no minority-targeted programs prior to 1970. Trainees whose funding began after 1999 were not included because the committee believes it is necessary for at least five years to elapse from the time of initial funding in order to assess progress. CATI Instruments The committee identified the process and outcome variables it should examine as it assessed the success of NIH minority training programs. Because of the different educational levels of the four cohorts of trainees, the committee concluded that the outcome variables should be consistent with the career stage of trainees at the time he or she was funded by one of the programs under study. For example, trainees selected to participate in the interview from the undergraduate strata were asked to recount their experiences while they were undergraduates, regardless of whether they had subsequently been funded for training at the graduate or postdoctoral levels. The 10 categories of questions included in the CATI instruments are listed below: Trainee demographics, Educational expectations, Current educational status, Plans immediately following degree completion, Expectations of program participation, Career goals, Sources of financial support, Research and laboratory experience, Experience with mentor and/or laboratory principal investigator, and Overall assessment of the training program. The CATI instrument developed for graduate trainees can be found in Appendix C. The other three CATI questionnaires, although similar to the graduate trainee questionnaire have important distinctions. These questionnaires can be found on the National Academies web site at http://www7.nationalacademies.org/BHEW/NIHMinorityCodebook.pdf.

OCR for page 29
Assessment of NIH Minority Research and Training Programs: Phase 3 TABLE 2-3 Number of Trainees in Program Universe, 1970-1999 Training Level Minorities in Targeted Programs Minorities in Untargeted Programs Nonminorities in Untargeted Programs Race Unknown in Untargeted Programs Total Number of Trainees Undergraduate trainees 6,614 n/a n/a n/a 6,614 Graduate trainees 4,115 856 6,364 5,145 16,480 Postdoctoral or faculty 247 738 7,905 5,487 14,377 Total trainees 10,976 1,594 14,269 10,632 37,471 NOTE: n/a = not applicable. Trainee Universe, Sample, and Response Building on the census of minority-targeted programs conducted earlier, the committee, assisted by the NIH contractor, constructed a census of the trainee universe. What might have been a straightforward task, if NIH data sets had the right variables, turned out to be extremely time-consuming as the TFF file does not indicate whether a trainee is in a minority-targeted or nontargeted program. Consequently, the only way that this information could be obtained for each trainee was through a time-intensive process that began with the NIH data contractor providing lists of grant numbers for each program and time period by IC. These lists of grant numbers were then distributed to the IC representatives who used “in-house” databases at their disposal to determine which of the grant numbers pertained to minority-targeted awards. This strategy was ultimately successful, but it took months and required much follow up with IC representatives. The trainee universe was 37,471, as shown in Table 2-3. Once the universe had been determined, the committee established a trainee sampling plan and derived the trainee sample in collaboration with the NIH data contractor. The sampling plan designed by the committee oversampled programs with low numbers of trainees in order to increase their representation among trainee interviews. In the case of very small programs, the entire trainee population was used and every attempt was made to contact these trainees. The total trainee sample size was 5,371. As a target for interview completion, the committee selected a cell size of 100 trainees for each of the 10 combinations of trainee level and program focus as shown in Table 2-3 to ensure adequate power to demonstrate important differences.

OCR for page 29
Assessment of NIH Minority Research and Training Programs: Phase 3 Trainee Interview Protocol The NIH data contractor identified 12 interviewers and supervisors experienced in conducting government-sponsored survey research to assist this study. Each received training specific to this study that addressed the following issues: administration of the trainee CATI protocol, respondent location activities and procedures, the purpose and design of the trainee protocol and its design, relevant characteristics of the sample population, data collection procedures, and an item-by-item review of the four survey instruments. The strategies used to locate trainee respondents are described below: Obtain trainee-identification data consisting of name and Social Security number for each trainee in the sample (n = 5,371). Match these identification data against the data files of two credit bureaus (Experian and Accurint) to obtain address and telephone number for all trainees in the sample. Randomly select and call trainees from each career stage, minority-targeted or non-targeted program category, and minority status, until a quota of 100 completed interviews per cell was achieved. In total, 5,371 trainees were matched against the Experian and Accurint databases. Over two-thirds of these queries (n = 3,628) produced neither an address nor a telephone number. For the remainder of the trainee sample (n = 1,743), letters of introduction signed by the NIH director were mailed to the addresses obtained from these searches. The introductory letter explained the purpose and importance of the study, encouraged sampled individuals to participate, and invited them to provide updated contact information by returning a prepaid postcard to the NIH data contractor. The CATI sample file was updated with telephone information obtained from returned postcards. In addition, a proprietary contact information management system was used to locate individuals for whom no telephone number or an incorrect number was initially obtained. The system includes data gathered from the U.S. Postal Change of Address database. These efforts were necessary because of initial difficulties in locating individuals who may have dropped out of school, graduated many years ago, left the field, or been otherwise difficult to find. Interviews were conducted between February 11 and May 10, 2004. The data collection protocol followed by interviewers is summarized below. Attempts were made to contact trainees at both home and work telephone numbers as these became available. Attempts were made between the hours of 9 a.m. and 9 p.m. local (trainee) time. No maximum number of attempted contacts was set for any trainee. The median number of attempts required to complete an interview was between four and five calls; however, up to 70 calls were made to a few trainees. A minimum of nine attempts was made on every verified telephone number in order to complete an interview with a respondent at that number, with the following exceptions:

OCR for page 29
Assessment of NIH Minority Research and Training Programs: Phase 3 Some sample records in subgroups with high numbers of completed responses received fewer attempts due to the decision to concentrate on difficult subgroups. Respondents requesting a set appointment for an interview might be contacted fewer than nine times to comply with their request. Respondents located very late in the interviewing period (i.e., fewer than nine days left) received fewer attempts. No more than one attempt per day was made to contact any individual unless specifically requested by the trainee. Trainees who stated explicitly that they did not want to participate in the study were not contacted again. Trainees who were reluctant to participate were contacted subsequently until they gave an explicit refusal or the interview was otherwise resolved. Those identified as “resistant” respondents were contacted less frequently. Despite assurances from the NIH data contractor that all CATI operators were proficient and skillful in conducting CATI interviews, at least one interviewer had difficulty reading and correctly pronouncing the terms used in the survey. NRC staff were monitoring the first few calls when this happened and witnessed the frustration of a trainee respondent, who was a busy M.D., when the interviewer could not read or correctly pronounce essential elements of the survey. The respondent, in this case, nearly hung-up on her. Interview Outcomes Table 2-4 shows the distribution of the sample size and various sampling outcomes by trainee level. The outcomes for the CATI process can be summarized as follows: Unfortunately, the first strategy for locating trainee respondents did not prove to be effective. In total, 5,371 trainees were matched against the Experian and Accurint databases. More than two-thirds of these queries (n = 3,628) produced neither an address nor a telephone number. Of the 1,743 trainees for whom telephone and address information was obtained, 792, or 45 percent, were never contacted despite repeated attempts. Of the 951 persons contacted, 21 failed the screener (wrong person) and were dropped from the sampling frame, 141 trainees refused to participate in the interview, and 50 did not complete the interview. The final sample of 739 completed interviews included 83 trainees who were funded as undergraduates, 328 trainees funded as graduate trainees, and 328 trainees funded after receiving an M.D. or a Ph.D.

OCR for page 29
Assessment of NIH Minority Research and Training Programs: Phase 3 TABLE 2-4 Number in Sample Size and Sampling Outcomes for NIH Minority Research and Training Survey by Trainee Level Trainee Level Total in Sample Number with Phone or Address Data Number Contacted Failed Screener Refused or Terminated Interview Completed Interview Undergraduate 1,006 248 122 4 35 83 Graduate 2,464 759 415 9 78 328 Postdoctoral or junior faculty 1,901 736 414 8 78 328 Total 5,371 1,743 951 21 191 739 The committee is disappointed with these outcomes, because the percentage of trainees for whom no contact information could be obtained is sufficiently large that a sampling bias has been introduced and the external validity of the survey has been challenged. Given this, the committee determined the following: The sampling bias was sufficiently serious that the survey data could not be used in an explicitly quantitative manner in this report. The 739 completed interviews provided sufficient information to allow the data to be assessed, however, and used in a qualitative manner. Thus, data are reported using nonspecific terms such as “a majority of respondents said” or “a minority of respondents said.” Such phrases should not be equated with statistical significance. The 739 completed interviews provided a wealth of information through a series of open-ended responses, eliciting trainee opinions about the best and worst features of their programs, suggestions for changes to the programs, and a variety of experiences with them. The committee believes that this information was important and useful and, therefore, drew on this information to describe and assess the programs and their value to participants. Program Administrator Interviews The committee concluded that direct input from training program administrators at recipient institutions (PARI) and program administrators at NIH institutes and centers (PAIC) would be vital and necessary to the evaluation of these programs. After consideration of alternative methods of data collection, the committee decided that both administrator cohorts would be contacted through an unstructured, ethnographic interview process—by telephone for PARIs and in person for PAICs. The flexibility

OCR for page 29
Assessment of NIH Minority Research and Training Programs: Phase 3 inherent in ethnographic interviewing was expected to enable the interviewer to quickly target relevant topic areas and probe for information germane to the study charge. All interviews were conducted by NIH data contractor staff who are specially trained in ethnographic interviewing procedures. Interview Process The committee developed interview schedules for the ethnographic interviews of PARIs and PAICs that pose questions in five broad domains: History of the program and its role within the host institution. Processes to identify and recruit trainees. Trainee experiences with the program. Administrative program issues. Program evaluations or assessments. As was the case with the trainee CATI interviews, these interview schedules were tailored to their audiences. The two interview schedules provided a way to assess these five important domains from two different vantage points—that of the training programs themselves and that of the NIH ICs. The NIH data contractor conducted a pretest of these instruments consisting of three interviews with training program administrators and IC representatives. The pretest resulted in changes that were incorporated into the ethnographic instruments and approved by the committee. Copies of these ethnographic interview guides used for the unstructured interviews are presented in Appendix D. The committee coordinated the programs used for the trainee sample with those used for the sample of training program administrators, so that there would be three triangulated sources of information for each program—trainees, PARIs, and PAICs. For the PARI interviews, a sample of recipient institutions (e.g., universities, colleges) was randomly selected from the population of eligible institutions using a selection protocol that identified institutions as either high producers or low producers of underrepresented minority degrees The 47 PARI interviews were distributed among programs as follows: 18 targeted undergraduate programs. 18 targeted graduate or postdoctoral programs. 6 untargeted graduate or postdoctoral programs. 1 targeted postdoctoral or junior faculty program. 4 targeted institutional awards that covered all levels.

OCR for page 29
Assessment of NIH Minority Research and Training Programs: Phase 3 PAIC interviews were identified through random selection and include the following: 2 targeted undergraduate programs. 12 targeted graduate or postdoctoral programs. 4 targeted postdoctoral or junior faculty programs. 2 untargeted postdoctoral or junior faculty programs. 2 institutional awards that covered all levels. Interview Outcomes A total of 47 PARI and 22 PAIC interviews were completed during the winter and spring of 2004. The interviews were transcribed and prepared as structured text files. No quantitative analyses of these interviews were intended. Staff reviewed these files and provided the committee with key observations that were used to inform its analyses at the undergraduate, graduate, postdoctoral, and junior faculty levels as well as the chapter on perspectives of NIH administrators. Throughout both interview processes, however, interviewers did not always adhere to the study protocol. This was especially true for the PAIC interviews. In some instances, interviewees were told to speak about minority training programs in general, rather than focusing on a specific program or set of programs as the protocol dictated. Despite the committee’s request that interviewers review each transcript carefully before transmittal to the committee, it appears that this was also not done with diligence. Thus, despite having taped these many hour-long conversations, the interviews were frequently inaudible and not interpretable by the committee. As a result, potentially valuable information was lost. Unfortunately, time and budgetary constraints prohibited the recovery of these lost data.

OCR for page 29
Assessment of NIH Minority Research and Training Programs: Phase 3 This page intentionally left blank.