National Academies Press: OpenBook
« Previous: Appendix B: Methodological Notes About the Public Opinion Data
Suggested Citation:"Appendix C: Stakeholder Survey Methods." National Research Council. 2014. Pathways to Exploration: Rationales and Approaches for a U.S. Program of Human Space Exploration. Washington, DC: The National Academies Press. doi: 10.17226/18801.
×

C

Stakeholder Survey Methods

To obtain stakeholder input, the Human Spaceflight Public and Stakeholder Opinion Panel conducted a survey of several key stakeholder groups. The survey was conducted by NORC at the University of Chicago. This appendix describes the methods used to conduct the survey.

SAMPLING FRAME

After initial informal exploratory discussions with a variety of experts and stakeholders, the panel, in consultation with the committee, developed a list of stakeholders customized to meet the needs of this project (Table C.1). To build a sampling frame for the survey, the panel identified leadership positions within each of the stakeholder groups of interest, and then identified the individuals occupying these positions. For example, in the case of the industry group, this included CEOs and Presidents of corporations that are members of the Aerospace Industries Association, Commercial Spaceflight Federation, and the American Institute of Aeronautics and Astronautics. In the case of space scientists, the sampling frame included members of the National Academy of Sciences, Institute of Medicine, and National Academy of Engineering, with an interest in space, as well as officers and Board Directors of professional associations, such as the American Astronautical Society and American Astronomical Society.

Because the committee’s charge was to make recommendations for a sustainable program spanning the next couple of decades, the views of the younger generation were particularly important to capture. To develop a sampling frame of young space scientists, the panel assembled lists of American Institute of Aeronautics and Astronautics early career and graduate student award winners, National Science Foundation postdoctoral fellowship winners, and NASA Aeronautics Graduate Scholarship Program winners from the past 3 years.

A description of the lists used to generate the sampling frames for each of the stakeholder groups is provided in Table C.1.

The approach described yielded an overall sampling frame of over 10,000 individuals. Within each of the groups, a systematic random sample was drawn, for an overall sample of 2,054 cases. Duplicate records within each group were removed before sampling. Those individuals who appeared in the final sample for more than one group were flagged, for the purposes of the analyses, as members of each of the groups they were sampled from, but they did not receive duplicate requests to complete the survey. Because “NASA’s stakeholders” are not a clearly defined population and because we selected the stakeholders using sampling frames that were reasonable and convenient rather than comprehensive, the results from this survey cannot be generalized to all stakeholders.

Suggested Citation:"Appendix C: Stakeholder Survey Methods." National Research Council. 2014. Pathways to Exploration: Rationales and Approaches for a U.S. Program of Human Space Exploration. Washington, DC: The National Academies Press. doi: 10.17226/18801.
×

TABLE C.1 Lists Used to Generate Sampling Frame for the Stakeholder Survey

Stakeholders Description
Economic/industry

Aerospace Industries Association

Commercial Spaceflight Federation

American Institute of Aeronautics and Astronautics

Aerospace States Association

House Committee on Science, Space and Technology, Subcommittee on Space individuals with an industry affiliation who testified during the past 3 years

Senate Commerce, Science and Transportation Committee, Science and Space Subcommittee individuals with an industry affiliation who testified during the past 3 years

Space scientists and engineers

National Academy of Sciences members from relevant fields

National Academy of Engineering members from relevant fields

Institute of Medicine members who indicated an interest in space-related research

NASA Advisory Council and Committees

Aerospace Safety Advisory Panel

American Astronautical Society

American Geophysical Union

American Astronomical Society

American Society for Gravitational and Space Research

House Committee on Science, Space and Technology, Subcommittee on Space, space scientists who testified during the past 3 years

Senate Commerce, Science and Transportation Committee, Science and Space Subcommittee, space scientists who testified during the past 3 years

Young space scientists and engineers

American Institute of Aeronautics and Astronautics winners of early career and student awards, past 3 years

National Science Foundation postdoctoral fellowship winners in relevant fields, past 3 years NASA Aeronautics Graduate Scholarship Program winners, past 3 years

Other scientists and engineers

National Academy of Sciences members from non-space-related fields

National Academy of Engineering members from non-space-related fields

Institute of Medicine members who did not indicate an interest in space-related research

National Science Board

American Association for the Advancement of Science

Science magazine editorial board

House Committee on Science, Space and Technology, Subcommittee on Space non-space scientists who testified during the past 3 years

Senate Commerce, Science and Transportation Committee, Science and Space Subcommittee non-space scientists who testified during the past 3 years

Higher education

Deans and heads of graduate departments in relevant fields from research universities and doctoral/research universities (Carnegie codes 15, 16, 17)

Security/defense/foreign policy

Department of Defense, Defense Policy Board

National Academy of Sciences Air Force Studies Board

National Academy of Sciences Board on Army Science and Technology

National Academy of Sciences Naval Studies Board

Department of Defense Federally Funded Research and Development Centers

House Committee on Science, Space and Technology, Subcommittee on Space individuals with a defense background who testified during the past 3 years

Senate Commerce, Science and Transportation Committee, Science and Space Subcommittee individuals with a defense background who testified during the past 3 years

Suggested Citation:"Appendix C: Stakeholder Survey Methods." National Research Council. 2014. Pathways to Exploration: Rationales and Approaches for a U.S. Program of Human Space Exploration. Washington, DC: The National Academies Press. doi: 10.17226/18801.
×
Stakeholders Description
Space writers and science popularizers

Planetarium directors

National Association of Science Writers members who indicated “astronomy/space” as one of their areas of expertise

Individuals with NASA social media credentials from @NASASocial Twitter lists

House Committee on Science, Space and Technology, Subcommittee on Space testifiers who are space writers or science popularizers, past 3 years

Senate Commerce, Science and Transportation Committee, Science and Space Subcommittee testifiers who are space writers or science popularizers, past 3 years

Space advocates

Explore Mars

Mars Society

National Space Society

Planetary Society

Space Foundation

Space Frontier Foundation

Space Generation

Students for the Exploration and Development of Space

100 Year Starship

SpaceUp

House Committee on Science, Space and Technology, Subcommittee on Space, testifiers with a space advocacy background, past 3 years

Senate Commerce, Science and Transportation Committee, Science and Space Subcommittee testifiers with a space advocacy background, past 3 years

In addition, we have not made any attempt to weight the data to compensate for differences in the sizes of the various groups or overlaps in their composition. As a result, combining the responses from each group does not represent the universe of all of NASA’s stakeholders. Nevertheless, the sample from each frame was a probability sample and we believe this methodology provides a broader and more diverse perspective on stakeholder views than a nonprobability sample would.

SURVEY QUESTIONNAIRE

The objective of the stakeholder survey was to provide input to the committee on stakeholder views of human spaceflight, and specifically:

  • The rationales traditionally provided for human spaceflight and any new or emerging rationales
  • The importance of human spaceflight in the context of tradeoffs and alternatives
  • The consequences of discontinuing NASA’s human spaceflight program
  • The characteristics and goals of a worthwhile and feasible program for the near future (e.g., next two decades)

The survey instrument was developed by the panel in consultation with the committee. The goal was to develop standardized questions that could be both self-administered (via a paper questionnaire or web) and administered by a trained interviewer (via phone). The final questionnaire is included in Appendix D. To minimize response order effects (respondent’s potential tendency to favor answer options appearing toward the beginning of self-administered lists and toward the end of lists administered by phone), two versions of the questionnaire were produced, reversing the order of the response items in 3, 4, 5, and 11. Respondents were randomly assigned to receive either version A or version B of the questionnaire, consistently across administration modes. (The version included in Appendix D is version A of the mail questionnaire).

Suggested Citation:"Appendix C: Stakeholder Survey Methods." National Research Council. 2014. Pathways to Exploration: Rationales and Approaches for a U.S. Program of Human Space Exploration. Washington, DC: The National Academies Press. doi: 10.17226/18801.
×

DATA COLLECTION

To carry out the data collection, the panel selected NORC at the University of Chicago. Data collection began on September 17, 2013, and concluded on November 6, 2013.

To maximize response rates, the survey was conducted in three administration modes—mail, web and telephone—all monitored by a case management system that tracked the status and outcomes of all sampled stakeholders across modes. This ensured that sample members were not contacted once they had completed the survey. It also controlled the number of emails received by sample members who had not completed the survey so as to limit unnecessary burden.

As a first step, a paper questionnaire was sent to all sample members where a mailing address was available. The questionnaire was accompanied by a cover letter signed by the Chair of the NRC, along with a postage paid envelope for returning the completed questionnaire.

Follow-up mailings were sent to cases where the only contact information after several rounds of locating attempts was a mailing address. In addition, repeat mailings were sent to a small number of sample members who requested another mailed questionnaire when contacted by a telephone interviewer. Near the end of data collection, reminder letters were mailed via FedEx to the remaining sample members who had not yet responded within the Industry and Young Space Scientist strata to help boost participation among these groups. These letters included the link to the web survey, individual PINs, and the toll-free number that sample members could use to complete the survey by phone.

Receipt control and data entry systems were used to input responses received via mail with 100 percent data entry verification.

Following the initial mailings and emails, all non-responding sample members with a located telephone number were contacted by phone. Telephone interviewers were selected and trained from the existing group employed by NORC’s telephone center in Chicago, IL. Prior to interviewing, all interviewers passed a certification test to demonstrate that they were well-versed in the project’s purpose and survey administration. A system was also implemented to receive inbound calls via a toll-free number.

A web link to this mode of the survey was sent to all sample members with a known email address. Sample members accessed the web survey using an individualized PIN assigned at random during the compilation of the sample. A link to the web survey was also included in the initial paper mailing, through regularly scheduled email blasts, and by request.

Telephoning began on October 1, 2013. Follow-up calls were made on a regular basis to all non-completers. Exemptions were made for those who refused to participate. NORC’s telephone center supervisors monitored interviewers throughout the data collection process.

The research staff continuously monitored results for signs of any complications with the data collection process. The research staff also monitored the progress of the survey on a daily basis, tallying the number of completed surveys by mode and by strata, as well as the total number of completed surveys. Careful monitoring of the transition of cases between modes of contact was critical throughout. Any cases where it was determined that the sample member lived outside the United States, was deceased or incapacitated/unable to complete the survey were considered not eligible.

Six survey questions requested open-ended responses that required coding by the research staff. The answers were coded into categories developed by the panel. Two coders independently coded 100 cases completed early in data collection and inter-coder reliability was calculated. The resulting kappa statistics are shown in Table C.2 and indicating the coding was adequately reliable. These cases were selected proportionally from the eight stakeholder groups. NORC survey managers and the coding supervisor debriefed the panel on this initial coding process. Then, upon the completion of data collection, all open-ended responses were coded in compliance with guidelines delineated during the initial batch of open-ends.

By the end of the data collection period 1,104 individuals, or 54 percent of the initial sample, completed the survey. The AAPOR Response Rate 3, which adjusts for ineligible cases, was 55.4 percent (Table C.3). In an effort to reduce the differences in response rates among the groups, at the end of the field period an additional mailing was sent via FedEx to nonresponders in the groups with the lowest response rates. To some extent, the differences

Suggested Citation:"Appendix C: Stakeholder Survey Methods." National Research Council. 2014. Pathways to Exploration: Rationales and Approaches for a U.S. Program of Human Space Exploration. Washington, DC: The National Academies Press. doi: 10.17226/18801.
×

TABLE C.2 Inter-Coder Reliability Calculated Based on the First 100 Cases Coded

Question Number Kappa Statistic
Q1A 0.82
Q1B 0.73
Q2A 0.87
Q2B 0.75
Q6 0.73
Q18 0.88

TABLE C.3 Number of Completed Cases and Response Rates

Stratum Name Original Sample Number of Complete Cases Simple Response Rate (%) AAPOR Response Rate 3 (%)
Economic/industry  384  104 27 28.6
Space scientists and engineers  395  261 66 67.1
Young space scientists and engineers  195    90 46 49.7
Other scientists and engineers  396  201 51 51.3
Higher education  399  294 74 74.1
Security/defense/foreign policy  110    71 65 66.4
Space writers and science popularizers    99    53 54 56.4
Space advocates    96    46 48 51.7
TOTAL 2,054 1,104 54 55.4

in the final response rates reflect differences in the contact information available for the different groups, including whether efforts to reach the sample member were likely to be screened by a gatekeeper, which was especially likely in the case of the industry group.

Suggested Citation:"Appendix C: Stakeholder Survey Methods." National Research Council. 2014. Pathways to Exploration: Rationales and Approaches for a U.S. Program of Human Space Exploration. Washington, DC: The National Academies Press. doi: 10.17226/18801.
×
Page 188
Suggested Citation:"Appendix C: Stakeholder Survey Methods." National Research Council. 2014. Pathways to Exploration: Rationales and Approaches for a U.S. Program of Human Space Exploration. Washington, DC: The National Academies Press. doi: 10.17226/18801.
×
Page 189
Suggested Citation:"Appendix C: Stakeholder Survey Methods." National Research Council. 2014. Pathways to Exploration: Rationales and Approaches for a U.S. Program of Human Space Exploration. Washington, DC: The National Academies Press. doi: 10.17226/18801.
×
Page 190
Suggested Citation:"Appendix C: Stakeholder Survey Methods." National Research Council. 2014. Pathways to Exploration: Rationales and Approaches for a U.S. Program of Human Space Exploration. Washington, DC: The National Academies Press. doi: 10.17226/18801.
×
Page 191
Suggested Citation:"Appendix C: Stakeholder Survey Methods." National Research Council. 2014. Pathways to Exploration: Rationales and Approaches for a U.S. Program of Human Space Exploration. Washington, DC: The National Academies Press. doi: 10.17226/18801.
×
Page 192
Next: Appendix D: Stakeholder Survey Mail Questionnaire (Version A) »
Pathways to Exploration: Rationales and Approaches for a U.S. Program of Human Space Exploration Get This Book
×
Buy Paperback | $67.00 Buy Ebook | $54.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The United States has publicly funded its human spaceflight program on a continuous basis for more than a half-century, through three wars and a half-dozen recessions, from the early Mercury and Gemini suborbital and Earth orbital missions, to the lunar landings, and thence to the first reusable winged crewed spaceplane that the United States operated for three decades. Today the United States is the major partner in a massive orbital facility - the International Space Station - that is becoming the focal point for the first tentative steps in commercial cargo and crewed orbital space flights. And yet, the long-term future of human spaceflight beyond this project is unclear. Pronouncements by multiple presidents of bold new ventures by Americans to the Moon, to Mars, and to an asteroid in its native orbit, have not been matched by the same commitment that accompanied President Kennedy's now fabled 1961 speech-namely, the substantial increase in NASA funding needed to make it happen. Are we still committed to advancing human spaceflight? What should a long-term goal be, and what does the United States need to do to achieve it?

Pathways to Exploration explores the case for advancing this endeavor, drawing on the history of rationales for human spaceflight, examining the attitudes of stakeholders and the public, and carefully assessing the technical and fiscal realities. This report recommends maintaining the long-term focus on Mars as the horizon goal for human space exploration. With this goal in mind, the report considers funding levels necessary to maintain a robust tempo of execution, current research and exploration projects and the time/resources needed to continue them, and international cooperation that could contribute to the achievement of spaceflight to Mars. According to Pathways to Exploration, a successful U.S. program would require sustained national commitment and a budget that increases by more than the rate of inflation.

In reviving a U.S. human exploration program capable of answering the enduring questions about humanity's destiny beyond our tiny blue planet, the nation will need to grapple with the attitudinal and fiscal realities of the nation today while staying true to a small but crucial set of fundamental principles for the conduct of exploration of the endless frontier. The recommendations of Pathways to Exploration provide a clear map toward a human spaceflight program that inspires students and citizens by furthering human exploration and discovery, while taking into account the long-term commitment necessary to achieve this goal.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!