Click for next page ( 204


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 203
C Stakeholder Survey Methods To obtain stakeholder input, the Human Spaceflight Public and Stakeholder Opinion Panel conducted a survey of several key stakeholder groups. The survey was conducted by NORC at the University of Chicago. This appendix describes the methods used to conduct the survey. SAMPLING FRAME After initial informal exploratory discussions with a variety of experts and stakeholders, the panel, in consultation with the committee, developed a list of stakeholders customized to meet the needs of this project (Table C.1). To build a sampling frame for the survey, the panel identified leadership positions within each of the stakeholder groups of interest, and then identified the individuals occupying these positions. For example, in the case of the industry group, this included CEOs and Presidents of corporations that are members of the Aerospace Industries Association, Commercial Spaceflight Federation, and the American Institute of Aeronautics and Astronautics. In the case of space scientists, the sampling frame included members of the National Academy of Sciences, Institute of Medicine, and National Academy of Engineering, with an interest in space, as well as officers and Board Directors of professional associations, such as the American Astronautical Society and American Astronomical Society. Because the committee’s charge was to make recommendations for a sustainable program spanning the next couple of decades, the views of the younger generation were particularly important to capture. To develop a sampling frame of young space scientists, the panel assembled lists of American Institute of Aeronautics and Astronautics early career and graduate student award winners, National Science Foundation postdoctoral fellowship winners, and NASA Aeronautics Graduate Scholarship Program winners from the past three years. A description of the lists used to generate the sampling frames for each of the stakeholder groups is provided in Table C.1. The approach described yielded an overall sampling frame of over 10,000 individuals. Within each of the groups, a systematic random sample was drawn, for an overall sample of 2,054 cases. Duplicate records within each group were removed before sampling. Those individuals who appeared in the final sample for more than one group were flagged, for the purposes of the analyses, as members of each of the groups they were sampled from, but they did not receive duplicate requests to complete the survey. Because “NASA’s stakeholders” are not a clearly defined population and because we selected the stakeholders using sampling frames that were reasonable and convenient rather than comprehensive, the results from this survey cannot be generalized to all stakeholders. In addition, we have not made any attempt to weight the data to compensate for differences in the sizes of the various groups or overlaps in their composition. As a result, combining the responses from each group does not represent the universe of all of NASA’s stakeholders. Nevertheless, the sample from each frame was a probability sample and we believe this methodology provides a broader and more diverse perspective on stakeholder views than a nonprobability sample would. PREPUBLICATION COPY—SUBJECT TO FURTHER EDITORIAL CORRECTION C-1

OCR for page 203
TABLE C.1 Lists Used to Generate Sampling Frame for the Stakeholder Survey Stakeholders Description ECONOMIC/ Aerospace Industries Association INDUSTRY Commercial Spaceflight Federation American Institute of Aeronautics and Astronautics Aerospace States Association House Committee on Science, Space and Technology, Subcommittee on Space individuals with an industry affiliation who testified during the past three years Senate Commerce, Science and Transportation Committee, Science and Space Subcommittee individuals with an industry affiliation who testified during the past three years SPACE SCIENTISTS National Academy of Sciences members from relevant fields AND ENGINEERS National Academy of Engineering members from relevant fields Institute of Medicine members who indicated an interest in space related research NASA Advisory Council and Committees Aerospace Safety Advisory Panel American Astronautical Society American Geophysical Union American Astronomical Society American Society for Gravitational and Space Research House Committee on Science, Space and Technology, Subcommittee on Space, space scientists who testified during the past three years Senate Commerce, Science and Transportation Committee, Science and Space Subcommittee, space scientists who testified during the past three years YOUNG SPACE American Institute of Aeronautics and Astronautics winners of early career and SCIENTISTS AND student awards, past three years ENGINEERS National Science Foundation postdoctoral fellowship winners in relevant fields, past 3 years NASA Aeronautics Graduate Scholarship Program winners, past three years PREPUBLICATION COPY—SUBJECT TO FURTHER EDITORIAL CORRECTION C-2

OCR for page 203
Stakeholders Description OTHER SCIENTISTS National Academy of Sciences members from non-space related fields AND ENGINEERS National Academy of Engineering members from non-space related fields Institute of Medicine members who did not indicate an interest in space-related research National Science Board American Association for the Advancement of Science Science magazine editorial board House Committee on Science, Space and Technology, Subcommittee on Space non-space scientists who testified during the past three years Senate Commerce, Science and Transportation Committee, Science and Space Subcommittee non-space scientists who testified during the past three years HIGHER Deans and heads of graduate departments in relevant fields from research EDUCATION universities and doctoral/research universities (Carnegie codes 15, 16, 17) SECURITY/ Department of Defense, Defense Policy Board DEFENSE/ FOREIGN POLICY National Academy of Sciences Air Force Studies Board National Academy of Sciences Board on Army Science and Technology National Academy of Sciences Naval Studies Board Department of Defense Federally Funded Research and Development Centers House Committee on Science, Space and Technology, Subcommittee on Space individuals with a defense background who testified during the past three years Senate Commerce, Science and Transportation Committee, Science and Space Subcommittee individuals with a defense background who testified during the past three years PREPUBLICATION COPY—SUBJECT TO FURTHER EDITORIAL CORRECTION C-3

OCR for page 203
Stakeholders Description SPACE WRITERS Planetarium directors AND SCIENCE POPULARIZERS National Association of Science Writers members who indicated “astronomy/space” as one of their areas of expertise Individuals with NASA social media credentials from @NASASocial Twitter lists House Committee on Science, Space and Technology, Subcommittee on Space testifiers who are space writers or science popularizers, past three years Senate Commerce, Science and Transportation Committee, Science and Space Subcommittee testifiers who are space writers or science popularizers, past three years SPACE Explore Mars ADVOCATES Mars Society National Space Society Planetary Society Space Foundation Space Frontier Foundation Space Generation Students for the Exploration and Development of Space 100 Year Starship SpaceUp House Committee on Science, Space and Technology, Subcommittee on Space, testifiers with a space advocacy background, past three years Senate Commerce, Science and Transportation Committee, Science and Space Subcommittee testifiers with a space advocacy background, past three years SURVEY QUESTIONNAIRE The objective of the stakeholder survey was to provide input to the committee on stakeholder views of human spaceflight, and specifically:  The rationales traditionally provided for human spaceflight and any new or emerging rationales  The importance of human spaceflight in the context of tradeoffs and alternatives  The consequences of discontinuing NASA’s human spaceflight program PREPUBLICATION COPY—SUBJECT TO FURTHER EDITORIAL CORRECTION C-4

OCR for page 203
 The characteristics and goals of a worthwhile and feasible program for the near future (e.g. next two decades) The survey instrument was developed by the panel in consultation with the committee. The goal was to develop standardized questions that could be both self-administered (via a paper questionnaire or web) and administered by a trained interviewer (via phone). The final questionnaire is included in Appendix C. To minimize response order effects (respondent’s potential tendency to favor answer options appearing toward the beginning of self-administered lists and toward the end of lists administered by phone), two versions of the questionnaire were produced, reversing the order of the response items in 3, 4, 5, and 11. Respondents were randomly assigned to receive either version A or version B of the questionnaire, consistently across administration modes. (The version included in Appendix D is version A of the mail questionnaire). DATA COLLECTION To carry out the data collection, the panel selected NORC at the University of Chicago. Data collection began on September 17, 2013 and concluded on November 6, 2013. To maximize response rates, the survey was conducted in three administration modes – mail, web and telephone – all monitored by a case management system that tracked the status and outcomes of all sampled stakeholders across modes. This ensured that sample members were not contacted once they had completed the survey. It also controlled the number of emails received by sample members who had not completed the survey so as to limit unnecessary burden. As a first step, a paper questionnaire was sent to all sample members where a mailing address was available. The questionnaire was accompanied by a cover letter signed by the Chair of the NRC, along with a postage paid envelope for returning the completed questionnaire. Follow-up mailings were sent to cases where the only contact information after several rounds of locating attempts was a mailing address. In addition, repeat mailings were sent to a small number of sample members who requested another mailed questionnaire when contacted by a telephone interviewer. Near the end of data collection, reminder letters were mailed via FedEx to the remaining sample members who had not yet responded within the Industry and Young Space Scientist strata to help boost participation among these groups. These letters included the link to the web survey, individual PINs, and the toll-free number that sample members could use to complete the survey by phone. Receipt control and data entry systems were used to input responses received via mail with 100 percent data entry verification. Following the initial mailings and emails, all non-responding sample members with a located telephone number were contacted by phone. Telephone interviewers were selected and trained from the existing group employed by NORC’s telephone center in Chicago, IL. Prior to interviewing, all interviewers passed a certification test to demonstrate that they were well-versed in the project’s purpose and survey administration. A system was also implemented to receive inbound calls via a toll-free number. A web link to this mode of the survey was sent to all sample members with a known email address. Sample members accessed the web survey using an individualized PIN assigned at random during the compilation of the sample. A link to the web survey was also included in the initial paper mailing, through regularly scheduled email blasts, and by request. Telephoning began on October 1, 2013. Follow-up calls were made on a regular basis to all non- completers. Exemptions were made for those who refused to participate. NORC’s telephone center supervisors monitored interviewers throughout the data collection process. The research staff continuously monitored results for signs of any complications with the data collection process. The research staff also monitored the progress of the survey on a daily basis, tallying the number of completed surveys by mode and by strata, as well as the total number of completed PREPUBLICATION COPY—SUBJECT TO FURTHER EDITORIAL CORRECTION C-5

OCR for page 203
surveys. Careful monitoring of the transition of cases between modes of contact was critical throughout. Any cases where it was determined that the sample member lived outside the United States, was deceased or incapacitated/unable to complete the survey were considered not eligible. Six survey questions requested open-ended responses that required coding by the research staff. The answers were coded into categories developed by the panel. Two coders independently coded 100 cases completed early in data collection and inter-coder reliability was calculated. The resulting kappa statistics are shown in Table C.2 and indicating the coding was adequately reliable. These cases were selected proportionally from the eight stakeholder groups. NORC survey managers and the coding supervisor debriefed the panel on this initial coding process. Then, upon the completion of data collection, all open-ended responses were coded in compliance with guidelines delineated during the initial batch of open-ends. By the end of the data collection period 1,104 individuals, or 54 percent of the initial sample, completed the survey. The AAPOR Response Rate 3, which adjusts for ineligible cases, was 55.4 percent (Table C.3). In an effort to reduce the differences in response rates among the groups, at the end of the field period an additional mailing was sent via FedEx to nonresponders in the groups with the lowest response rates. To some extent, the differences in the final response rates reflect differences in the contact information available for the different groups, including whether efforts to reach the sample member were likely to be screened by a gatekeeper, which was especially likely in the case of the industry group. PREPUBLICATION COPY—SUBJECT TO FURTHER EDITORIAL CORRECTION C-6

OCR for page 203
TABLE C.2 Inter-Coder Reliability Calculated Based on the First 100 Cases Coded Question Number Kappa Statistic Q1A 0.82 Q1B 0.73 Q2A 0.87 Q2B 0.75 Q6 0.73 Q18 0.88 TABLE C.3 Number of Completed Cases and Response Rates Simple AAPOR Number of Response Response Original Complete Rate Rate 3 Stratum Name Sample Cases (%) (%) Economic/industry 384 104 27 28.6 Space scientists and engineers 395 261 66 67.1 Young space scientists and engineers 195 90 46 49.7 Other scientists and engineers 396 201 51 51.3 Higher education 399 294 74 74.1 Security/defense/foreign policy 110 71 65 66.4 Space writers and science popularizers 99 53 54 56.4 Space advocates 96 46 48 51.7 TOTAL 2,054 1,104 54 55.4 PREPUBLICATION COPY—SUBJECT TO FURTHER EDITORIAL CORRECTION C-7