As one of the leading federal science, technology, engineering, and mathematics (STEM) agencies of the United States, the National Aeronautics and Space Administration (NASA) has an important role to play in the landscape of STEM education. Education programs in NASA’s Science Mission Directorate (SMD) are one of the major ways that the agency has engaged the public in the excitement of NASA’s science missions. In 2015, NASA SMD created the Science Activation (SciAct) Program to increase the overall coherence of SMD’s education efforts; to support more effective, sustainable, and efficient use of SMD science discoveries for education; and to enable NASA scientists and engineers to engage more effectively and efficiently in the STEM learning environment with learners of all ages. As SciAct transitions into its second round of funding, it is important that NASA take stock of the status of the portfolio and consider how it might be improved. To this end, NASA asked the Board on Science Education at the National Academies of Sciences, Engineering, and Medicine to conduct a review of the SciAct portfolio. In response to this request, the Board on Science Education convened a committee of experts to
Assess the SMD’s Science Activation (SciAct) Program’s efforts toward meeting the following objectives: (1) enable STEM education, (2) improve U.S. scientific literacy, (3) advance national education goals, and (4) leverage efforts through partnerships. The review will provide an independent, authoritative forum for identifying and discussing SciAct issues in Earth
and Space Science related to NASA SMD’s SciAct Program and will include the following:
Concise written assessment of the status of the SMD Science Activation Program including feedback on improving the program. The assessment will be based on evidence gathered by the committee at its in-person and virtual meetings and on established principles for evidence-based science education as summarized in previous reports from the Board on Science Education. The committee’s assessment final report may include findings and conclusions related to management of and priorities for the next phase of the program, including the identification of any gaps in the SciAct approach, given new advances in science education pedagogy and recent Decadal recommendations. The assessment will be subject to review in accordance with the National Academies institutional policies.
To accomplish this task, the committee reviewed documentation of the SciAct Program including descriptions provided by NASA, the solicitation for proposal, and evaluation reports from individual projects in the portfolio. We also heard input from NASA staff associated with the SciAct Program and from awardees.
HISTORY, VISION, AND OBJECTIVES OF THE SciAct PROGRAM
SciAct represents a new approach to education and outreach in SMD. Previously, SMD supported its education and public outreach efforts by allocating at least 1 percent of the budget of each science mission to education and outreach (known as the 1% model), by offering awards through solicited or unsolicited proposals (e.g., Global Learning and Observation to Benefit the Environment [GLOBE] Implementation Office), or by including education and outreach as elements of a major research-enabling program (e.g., suborbital science). Although the 1 percent model had many strengths, one consequence was a lack of coordination across mission activities, leading to duplications of efforts and potentially inequitable distribution of educational resources. Additionally, it was not clear that the education and outreach activities were responsive to the needs of educators and learners nor that they helped to advance broader national goals for STEM education. The new approach taken in SciAct is designed to address these issues. The vision of the SciAct Program is
to share the story, the science, and the adventure of NASA’s scientific explorations of our home planet, the solar system, and the universe beyond, through stimulating and informative activities and experiences created by experts, delivered effectively and efficiently to learners of many backgrounds via proven conduits, thus providing a return on the public’s investment in NASA’s scientific research.
A central focus of this vision is to leverage NASA’s science discoveries for education and enable NASA scientists and engineering to engage more effectively with learners of all ages. To achieve this vision, SciAct has identified four primary objectives:
- Enable STEM education. SciAct intends to support STEM education through the mobilization of NASA’s existing resources and assets. Practically, this means that awardees are tasked with translating NASA assets into educational use by creating new materials that can be used by learners or educators, and into programs that will allow learners and students to engage with NASA assets.
- Improve U.S. scientific literacy. Similar to the enable STEM education objectives, NASA awardees are expected to translate NASA assets so that they support the development of participants’ science literacy. SciAct currently measures science literacy via the continuation of the Science and Engineering Indicators Survey, which measures one specific aspect of science literacy.
- Advance national education goals. SciAct is pursuing this objective primarily through activities that provide professional development to in-service educators, provide authentic science experiences for learners, target specific underrepresented populations either through direct engagement or by providing resources and professional development to educators in areas with high percentages of these groups.
- Leverage efforts through partnerships. SciAct pursues this objective by supporting awardees in their partnerships in local communities, while also developing collaborations across the portfolio.
Further, SciAct documentation defines NASA assets and resources as including:
- exciting science and engineering content that engages audiences and motivates them to learn more;
- subject matter experts (SMEs), including scientists and engineers, who ask compelling scientific questions and then find ways to answer them within the environment of space;
- real-life participatory and experiential opportunities (includes student collaborations, e.g., suborbital balloon experiments and other student launch opportunities); and
- other science programs in NASA’s infrastructure (e.g., GLOBE, Night Sky Network).
To better understand the links among the vision, objectives and portfolio activities, the committee developed a logic model that built on the
existing SciAct Program model, but provides more detail about potential connections among NASA assets, SMEs, the activities of individual projects, project outcomes, and portfolio-level outcomes. The committee used this logic model to inform its review of SciAct’s vision and objectives. The committee concludes
CONCLUSION 1: The National Aeronautics and Space Administration has a unique role to play in the science, technology, engineering, and mathematics education landscape, but the current four objectives for the Science Activation Program are too broad and do not appropriately reflect that role.
CONCLUSION 2: The four current Science Activation Program objectives are general enough to inform a vision for the program, however they lack specific, actionable targets. As currently stated, the objectives are so broad that they obscure a clear understanding of how awardees’ contributions aggregate toward desired outcomes.
CONCLUSION 3: Improving science literacy at the national level is one of the four Science Activation (SciAct) Program objectives. We do not have evidence that there is a centrally agreed-upon definition of science literacy across the projects. While, the approach to measuring science literacy at the national level that SciAct is currently using reflects one approach to measuring science literacy, it does not fully reflect the most up-to-date conceptualizations of science literacy.
CHARACTERIZING THE PROJECTS IN THE SciAct PORTFOLIO
Currently, 24 awards (or projects) are still in progress that constitute the SciAct Phase 1 portfolio. Awardees will have an option to extend their funding by 5 years into Phase 2 of the program (potentially at different levels) beginning in 2020, and opportunities for new projects will be afforded through SMD’s Research Opportunities in Space and Earth Sciences (ROSES) grant solicitation. The current suite of projects addresses topics from the four primary NASA science disciplines: heliophysics, earth science, planetary science, and astrophysics. The awardees include museums and science centers, universities, a community college, a K–12 school district, research institutes, and educational foundations. The 24 projects engage a variety of audiences, including families, K–12 students and teachers, adults, children, and teens in formal education, informal education, and community settings. Overall, about half of SciAct projects engage learners in informal learning environments, such as museums and out-of-school programs, and half engage learners in formal educational settings, primarily
through K–12 schools and teachers. One-third of the SciAct projects have created digital resources for learners, both exclusively and as part of more comprehensive projects.
Across the portfolio, projects have established partnerships with scientific experts, educational experts, community organizations, professional organizations, museums and other informal learning institutions, K–12 school districts, universities and colleges, and multimedia platforms as a means of creating and disseminating learning programs and resources. Furthermore, some projects have cross-collaborated in a variety of capacities, whether to broaden their dissemination efforts or to leverage each other’s expertise in using NASA data/resources and developing learning resources. Because SciAct has identified leveraging partnerships as one of its primary objectives, these partnerships and collaborations are a high priority for the SciAct Program administration; and metrics about partnerships and cross-collaboration are documented as a part of each project’s monthly report to NASA SMD.
NASA SMD has strongly encouraged all projects to use NASA SMEs in some capacity within their projects. SMEs are engaged in SciAct projects in a variety of roles, from providing scientific and technical expertise to educational teams to actively participating in SciAct programming by sharing stories, presenting scientific and technical information, and leading program activities. SciAct projects are using NASA scientific content, data, science mission activities, or technologies as the basis for their educational programs and resources. The wide geographic distribution of programs, activities, and available materials makes SciAct projects well positioned to be a valuable asset to communities, which are local to their respective institutions.
Each project has an outside evaluator who works with the project leadership to evaluate the project’s activities and whether they are producing the intended outcomes. Currently, there is no comprehensive evaluation at the portfolio level.
In examining the nature of the projects in the current SciAct portfolio, the committee concludes that:
CONCLUSION 4: The National Aeronautics and Space Administration (NASA) has developed a portfolio of diverse projects reaching a broad range of communities across the United States that utilize NASA’s resources.
CONCLUSION 5: The Science Activation (SciAct) Program has placed an emphasis on no longer funding education work attached to individual missions. This has eliminated redundancy, but it has also resulted in some missions not being represented comprehensively. The current SciAct portfolio is not consistently incorporating assets from new missions.
CONCLUSION 6: In general, the Science Activation approach has enabled development of partnerships with groups external to the National Aeronautics and Space Administration (NASA) with expertise in education and learning that provide added value for NASA.
CONCLUSION 7: The Science Activation approach to evaluation focuses on evaluating individual projects without adequate attention to how evaluation can inform the whole portfolio. Project evaluators support individual projects and are surfacing important insights that could benefit the portfolio. Among the evaluators, there is interest in contributing to a broader understanding of what is working well, what can be improved, and where there are opportunities that can be further leveraged across the portfolio. However, there are limits, given the current design and program resources, to how much this is possible.
CONCLUSION 8: Interactions among the projects to date have allowed principal investigators and evaluators to share ideas across projects in ways that were unanticipated in the original design. These kinds of collaborations can be built upon and strengthened in the future.
EXAMINING PROGRESS TOWARD SciAct’S VISION AND OBJECTIVES
The committee examined the work of the 24 awards in the SciAct portfolio in greater detail in order to understand how each project contributes to the larger objectives of the portfolio. Because the current portfolio objectives are very broad, the committee focused on four key themes that are linked to the stated objectives but are closer to the actual activities of the projects:
- enhancing STEM learning,
- leveraging NASA assets,
- broadening participation, and
- developing networks.
Projects employ a variety of approaches to supporting STEM learning and leveraging NASA’s assets. They also vary in how explicit they are in articulating the assumptions about how learning occurs that undergird the design of their project activities. The committee also noted that there does not appear to be a portfolio-wide strategy for how NASA’s assets should be used, especially in the case of SMEs. The existing program model SciAct employs does not clearly show how NASA assets (i.e., content, SMEs, existing infrastructure) and research on learning come together in the design of
project activities (e.g., outreach events, educator professional development, and infrastructure resources), and also how these activities are organized to support teaching and learning in ways that are expected to lead to positive outcomes in STEM education.
SciAct’s mission is to deliver activities and experiences to learners of many backgrounds and to leverage scientist-educator partnerships that have demonstrated diverse, broad, and deep national education and communications impact. It is clear that the existing SciAct portfolio reaches large groups of learners across different regions, abilities, age groups, and race and ethnicities, among other facets of diversity. While reach is one dimension of broadening participation, there are other facets of the work that are equally important to consider, such as whether projects are fully accessible to all learners, regardless of their abilities. From the committee’s investigation, it is unclear whether appropriate attention is being given to other dimensions of broadening participation, such as inclusion, equity, and accessibility.
The committee also found that the SciAct portfolio is currently functioning as a dissemination network, wherein NASA resources and project innovations are disseminated to learners directly. However, the committee observed that the current SciAct Program also has elements that may support its capacity to function as a learning network that enhances the existing awardees’ activities and affords opportunities to share what is learned with the larger community of scholars and developers in STEM education. Building this kind of network would require additional investments to create the infrastructure needed to support ongoing communication and collaboration among projects.
In summary, in examining the SciAct portfolio’s progress toward the current objectives, the committee concludes
CONCLUSION 9: While science, technology, engineering, and mathematics learning is in the foreground as a Science Activation goal, there is no explicit link between theories of learning and how the National Aeronautics and Space Administration assets are used (e.g., transmission models, inquiry-based practices). There is a range of design intervention strategies that are used across the portfolio. Each project uses different theories of learning in its project design and often that theory of learning is not made explicit.
CONCLUSION 10: Current research on learning emphasizes the importance of learner-centered and community-centered instructional design and practices. Awardees have had uneven success at mobilizing the National Aeronautics and Space Administration assets while also being responsive to the needs of learners and communities.
CONCLUSION 11: Given the portfolio’s emphasis on the value of subject matter experts, the portfolio lacks a coordinated effort to incorporate evidence-based practices in translating their expertise in developing and implementing educational materials and learning experiences (e.g., translating datasets, engaging in public outreach).
CONCLUSION 12: While broadening participation is a stated intention of Science Activation (SciAct), it is not clearly defined, nor is there evidence that awardee activities have uniformly had an impact in this area. Integrating goals related to broadening participation throughout SciAct projects would require explicit assessment beyond counting the numbers of participants from various groups.
CONCLUSION 13: The projects that are part of the Science Activation portfolio use a variety of design strategies to translate the National Aeronautics and Space Administration’s assets—subject matter experts, media assets, scientific instruments, datasets, etc.—to support learning in science, technology, engineering, and mathematics. However, there are limited mechanisms for gathering, synthesizing, and sharing these innovations across the portfolio or for learning from cases of success or failure.
RECOMMENDATIONS FOR MOVING INTO PHASE 2 OF THE SciAct PROGRAM
The Committee to Assess Science Activation was charged with two primary tasks: (1) assessing SciAct’s progress toward meeting its four primary objectives, and (2) offering feedback on improving the SciAct Program. The committee was struck by the considerable value of the SciAct portfolio of investments in the national landscape of efforts to support STEM learning and engagement. The scope and diversity of SciAct projects are reaching a wide swath of learners across the country, and projects are employing a myriad of strategies for engaging potential participants. At the same time, the committee noted several ways that the SciAct Program can be improved. Phase 2 of the program offers a natural inflection point for initiating a reflection process and determining how existing projects might be improved and what additional projects might augment the current portfolio in strategic ways. One key issue is to refine the objectives for the SciAct portfolio and better articulate the connections among NASA’s assets, the specific activities of the projects and the desired outcomes for projects and the portfolio as a whole. The committee concludes
CONCLUSION 14: The Science Activation Program is at an important inflection point in its history. The second phase of the program presents
an opportunity for iterative improvement and refocusing on both the individual project level and the portfolio as a whole.
CONCLUSION 15: While continuing existing awards may allow for continuity and support an environment of collaboration and partnership among existing awardees, lack of competition or opportunities to fund new projects may stifle the evolution of the portfolio.
Based on its conclusions and with an eye toward continuing and enhancing the good work that is already under way, the committee offers several specific recommendations for moving forward.
RECOMMENDATION 1: Science Activation (SciAct) should go through a visioning process that brings the portfolio up to date with current research on learning and design, the new federal science, technology, engineering, and mathematics (STEM) plan, and evidence-based approaches to broadening participation. This process should also consider how SciAct fits within and contributes to the larger STEM education ecosystem and should provide the foundation for developing actionable and measureable portfolio goals.
RECOMMENDATION 2: Science Activation should articulate how it expects that the portfolio will leverage National Aeronautics and Space Administration assets, how partnerships and networks will be built, and how these actions will lead to desired, measurable outcomes.
RECOMMENDATION 3: Science Activation (SciAct) must consider whether the development of a coordinated learning network of awardees across its portfolio is a program priority. If it is a priority, then the program must provide the necessary infrastructure to support a more active network of projects. At the very least, SciAct needs to develop more systematic mechanisms for projects to share best practices and learn from successes and failure.
RECOMMENDATION 4: Science Activation should use the opportunity provided by Phase 2 to reflect on the current portfolio within the context of the new vision, goals, and logic model. This process should critically review and guide existing projects, be explicit about the rationale and criteria for including new projects, and consider how best to integrate them into the existing portfolio. One important area for consideration is how to ensure that underserved communities receive more focused attention in the next phase of the program.
RECOMMENDATION 5: Science Activation (SciAct) should deepen its commitment to broadening participation by using evaluation measures that go beyond counting numbers of individuals who represent specific groups. In order to do this, SciAct must identify ways that the portfolio as a whole could draw upon and implement evidence-based strategies for broadening participation.
RECOMMENDATION 6: Science Activation should build ongoing opportunities for dialogue with the National Aeronautics and Space Administration Science Mission Directorate’s missions and scientists.
RECOMMENDATION 7: Science Activation should create an independent mechanism to obtain ongoing, real-time advice from individuals with expertise in learning and design, the larger policy context of science, technology, engineering, and mathematics (STEM) education, partnering with local communities, broadening participation in STEM, and science content relevant to the missions of the National Aeronautics and Space Administration Science Mission Directorate. Among other responsibilities, these experts should inform the new visioning and planning process.