3

Characterizing the Current Portfolio

The Science Activation (SciAct) portfolio includes 24 current projects that together are intended “to increase the overall coherence of the Science Mission Directorate (SMD) science education program leading to more effective, sustainable, and efficient utilization of SMD science discoveries and learning experiences and to meet overall SMD science education objectives.” According to the Cooperative Agreement Notice (CAN) for SciAct, fundamental to achieving this outcome is to “enable National Aeronautics and Space Administration (NASA) scientists and engineers to engage more effectively with learners of all ages” (National Aeronautics and Space Administration, 2015).

In this chapter, we provide a detailed overview of the SciAct Program, with particular attention to the program’s design and outputs. First, we describe the committee’s understanding of how the SciAct portfolio operates by articulating a logic model that builds on our description of SciAct’s four top-level science education objectives (see Chapter 2). Additionally, this chapter highlights common features across the 24 sponsored projects; how the portfolio addresses broadening participation in science, technology, engineering, and mathematics (STEM); the current approach to program evaluation, and collaboration within the current portfolio. The chapter concludes with a summary of the current characteristics of the program and considerations for continuous improvement.

THE LOGIC OF THE SciAct PORTFOLIO

In order to understand the implementation of the current SciAct portfolio, the committee first attempted to understand how SciAct is designed to work toward its four top-level objectives (described in Chapter 2). As part of this work, the committee used all available evidence to create a logic model that explains how the components of SciAct are intended to function in pursuit of its stated objectives. Logic models are a common way to illustrate the linkages among program goals, objectives, and activities. For the overall SciAct portfolio, a logic model can serve as a guide for both planning and assessing the role of individual projects in meeting high-level goals and long-term impacts. A critical component of any logic model, and project planning overall, is ensuring that goals and objectives are measurable and relevant to the planned activities.

The current iteration of SciAct is designed for awardees to work in collaboration, in order to strengthen the outcomes of the overall portfolio. Based on its understanding of the available evidence, the committee designed the following logic model to illustrate the strategy for utilizing SciAct awardees as the mediating agents for transforming NASA assets into activities that serve the desired objectives (see Figure 3-1).

As the logic model in Figure 3-1 depicts, the SciAct awardees leverage NASA people and products (i.e., the logic model inputs) to develop educational programming and products (i.e., the logic model activities in the top box) for a variety of learner audiences. These activities result in learner experiences and resources (i.e., the logic model outputs) that are designed to achieve project-specific outcomes (i.e., the logic model short-term outcomes in the top box). At the SciAct portfolio level, project-specific evaluations, sharing of experiences among projects, and topical working groups (i.e., the logic model activities in the bottom box) result in increased dissemination of SciAct project products and portfolio-wide evaluation data (i.e., the logic model outputs in the lower box). Dissemination and evaluation data are intended to lead to better educational products across all projects and also to build a network of awardees to strengthen the portfolio as a whole (i.e., the logic model short-term outcomes in the bottom box).

Both the project-level and portfolio-level outcomes should further the logic model’s long-term impacts, which in this case map to the four NASA SMD top-level science education objectives from the CAN. Individual projects have external evaluators who characterize the linkage between project outputs and short-term outcomes. While these outcomes are all related to the overall SciAct Program objectives (i.e., long-term impacts), the four objectives as stated are quite broad. As a result, it is difficult to measure these long-term outcomes in a meaningful way, and it is difficult to demonstrate a causal link between the short-term outcomes and any ob-

served changes in the long-term outcomes given the relatively modest size of the SciAct Program. To clearly communicate expectations from portfolio projects and assess progress toward meeting high-level objectives, the program must employ realistic long-term objectives that link to clear, concise, and measurable outcomes.

DESCRIBING THE CURRENT PROJECTS

In order to understand how the work of SciAct awardees aggregates toward meeting SciAct’s objectives, it was incumbent upon the committee to learn about the work of the individual projects. While a full investigation of the efficacy of the awards was beyond the scope of this project (see Chapter 1 of this report), the committee did endeavor to characterize themes and commonalities across awardees (for an award-by-award description of the portfolio, see Appendix A). In this section, we describe the array of projects in the portfolio with an eye toward painting a picture of SciAct’s on-the-ground presence.

SciAct was initiated in 2015 with 27 funded projects that address topics from the four primary NASA science disciplines: heliophysics, earth science, planetary science, and astrophysics. SciAct projects are distributed among the different science disciplines as follows:

• astrophysics: 3,
• earth science: 6,
• space science: 7, and
• cross-disciplinary: 8.

The SciAct projects focus on engaging a variety of audiences, including families, K–12 students and teachers, adults, children, and teens in formal education, informal education, and community settings.¹ The awardees include museums and science centers, universities, a community college, a K–12 school district, research institutes, and educational organizations and foundations. Overall, about 50 percent of SciAct projects engage learners in informal learning environments (such as museums, libraries, summer camps, out-of-school programs, in-home, and neighborhood or community spaces), and 50 percent of SciAct projects engage learners in formal educational settings, primarily through K–12 schools and teachers. One-third of the SciAct projects have created digital resources for learners, both exclusively and as part of more comprehensive projects. For example, a project led by the WGBH Educational Foundation, Bringing the Universe to America’s Classrooms, creates media-based educational materials for K–12 audiences.

___________________

¹ See https://science.nasa.gov/infographic.

The project led by the Space Telescope Science Institute, NASA’s Universe of Learning, is enabling educational use of astrophysics mission data and is providing participatory experiences by creating multimedia and immersive experiences, as well as designing exhibits and community programs and providing professional learning experiences for pre-service educators. Almost all of the projects list 11- to 18-year-olds as their target audience, and about one-half of the projects (15 out of 24) also cite 5- to 10-year olds among their targets. Additionally, about 50 percent of SciAct projects provide professional development opportunities and/or resources for K–12 educators. For a description of SciAct products, see Box 3-1.

All of the SciAct projects have established partnerships with scientific experts, educational experts, community organizations, professional organizations, museums and other informal learning institutions, K–12 school districts, universities and colleges, and/or multimedia platforms as a means of creating and disseminating learning programs and resources. Furthermore, some projects have cross-collaborations in a variety of capacities, whether to broaden their dissemination efforts or to leverage each other’s expertise in using NASA data/resources and developing learning resources. These partnerships and collaborations are a high priority for the SciAct Program administration, and metrics about partnerships and cross-collaboration are documented as a part of each project’s monthly report to NASA SMD.

A primary component of all SciAct projects is that they leverage NASA assets and infrastructure, including but not limited to

• NASA science content and data,
• NASA space and airborne platforms,
• NASA subject matter experts (SMEs) (i.e., scientific and technical personnel),
• NASA Wavelength online catalog (see http://nasawavelength.org),
• independent product review and assessment (see http://nasareviews.strategies.org),
• NASA Volunteer Networks (see http://solarsystem.nasa.gov/nnw/home.cfm),
• NASA Scientific Visualization Studio (see http://svs.gsfc.nasa.gov),
• NASA Eyes on Solar System and related products (see http://eyes.nasa.gov),
• NASA 3D Resources (see http://nasa3d.arc.nasa.gov),
• Space Grant (see http://www.nasa.gov/offices/education/programs/national/spacegrant/about/index.html),
• GLOBE (see http://www.globe.gov),
• Earth to Sky with National Parks (see http://earthtosky.org),
• Astronomy Picture of the Day (see http://apod.nasa.gov),

• NASA Mapping and Modeling Project (see http://www.lmmp.nasa.gov),
• other NASA communications infrastructure, including Web and social media sites.

SciAct has strongly encouraged all projects to use NASA SMEs in some capacity within their projects, but the extent of that engagement varies considerably across projects. Notably, regular project reporting to SMD by each project requires a listing of SMEs that participated in the project during the specified time period. SMEs consist of NASA-employed or NASA-supported scientists and engineers who have participated in NASA science missions and thus are doing, or have done, NASA science and engineering research and/or mission implementation. SMEs are engaged in SciAct projects in a variety of roles, from providing scientific and technical expertise to educational teams to actively participating in SciAct programming by sharing stories, presenting scientific and technical information, and leading program activities. This use of human capital gives NASA a unique ability to utilize its “funds of knowledge” by pipelining the scientific discoveries made by NASA scientists and engineers to the public.

SciAct strongly encourages all its projects to be “enabled by NASA’s research and missions” and to use NASA resources that are already being produced, thus contributing to furthering NASA’s overall educational mission. Most of the SciAct projects are using NASA scientific content, data, science mission activities, or technologies as the basis for their educational programs and resources. Five projects are leveraging the GLOBE Program, an international science and education program that provides students and the public worldwide with the opportunity to participate in data collection and the scientific process by developing new GLOBE applications. One SciAct project is focused solely on collecting national data on scientific literacy and thus is not directly leveraging specific NASA assets or infrastructure.

As noted in previous chapters, the SciAct Program effectively replaces educational programming that was previously funded as a part of each science mission. This approach enables NASA SMD to manage and evaluate educational activities more cohesively, and it eliminates duplicate projects. However, the committee notes that SciAct does not currently maintain a real-time way to engage the ongoing discoveries and excitement associated with NASA’s science missions (see Chapter 4 for a discussion of the value of NASA’s science missions). While the committee believes that the move away from mission-specific education efforts allows for an approach to STEM education that is able to address learners’ needs more directly, it is important to also highlight that there are additional educational opportuni-

ties that could emerge from engaging with NASA’s science missions in an ongoing way.

The NASA brand gives the agency access to many national and local partners and stakeholders to assist in reaching broad and diverse audiences of all ages. NASA employs many of the world’s leading researchers in heliophysics, astrophysics, planetary science, and earth sciences and engages in some of the most ambitious research to date. SciAct projects are expected to leverage these assets for educational and public engagement activities to meet the overarching program goals. Furthermore, once SciAct projects were selected and awarded, each project essentially became another NASA asset for others to leverage. Thus, the SciAct projects are encouraged to collaborate as part of a network of educational projects, capitalizing on one another’s strengths, to achieve SciAct Program goals (for more on how SciAct functions as a network, see Chapter 5). As the committee heard in the presentation of Kristen Erickson’s, NASA’s SMD director, this breadth of partnerships brings new external education providers and other entities into the “NASA family,” which expands the kinds of expertise in NASA’s extended wheelhouse.

SciAct’s employment of the Cooperative Agreement funding mechanism gives principal investigators (PIs) and NASA the ability to adapt within the project timeline to make changes as they learn what topics are piquing public interest and what activities and resources are most effective. This adaptive approach gives the program members the ability to be nimble, which has proven to be another asset for SciAct.

SciAct’s infrastructure (including its internal web communication portal, SMDEPO.org) allows it to coordinate efforts across the country and between research groups. Considering the geographic spread of SciAct projects, SciAct can reach a wide range of audiences across the United States. SciAct projects area based as far west as California, as far north as Alaska, as far east as Maine, and as far south as Texas, with numerous locations in between. This variety of locations gives SciAct the ability to reach diverse populations based on race, ethnicity, and socioeconomic status. For more information on this infrastructure, see descriptions of cross-portfolio communication in Appendix C of this report (see http://www.nap.edu/25569).

The wide geographical distribution of programs, activities, and available materials holds potential for SciAct to serve as a valuable asset to communities, which are local to their respective institutions. People in some locations may be engaged by solar physicists (heliophysics), while another group of participants may meet with scientists who look for evidence of past life on Mars or likelihood of life on Europa or Titan (planetary science).

BROADENING PARTICIPATION IN THE SciAct PORTFOLIO

Broadening participation in STEM is a stated priority for the SciAct Program, though it is not one of the four main objectives. The 2013 Federal Science, Technology, Engineering, and Mathematics (STEM) Education Five-Year Strategic Plan, drafted by the National Science and Technology Council (NSTC) Committee on STEM Education (CoSTEM), included the priority area “Better Serve Groups Historically Underrepresented in STEM Fields: Increase the number of students from groups that have been underrepresented in STEM fields that graduate with STEM degrees in the next 10 years and improve women’s participation in areas of STEM where they are significantly underrepresented.” Moreover, the 2018 CoSTEM Strategic Plan maintains and re-emphasizes this focus on broadening participation in STEM. The SciAct CAN notes this CoSTEM priority, and it is included as part of the top-level SMD science education objective “advance national education goals.”

Despite this emphasis, only 9 of the 24 currently active SciAct projects include some diversity metrics in their evaluation plans and reports to SMD. The metrics reported by these 9 projects include number of resources accessible to underrepresented groups, number of events in underrepresented communities, increase in percentage of participants enrolled in STEM degree programs, and increase in Spanish-language resources. In some cases, the metrics are not specific (e.g., “adequately serve groups historically underrepresented in STEM fields by making educational resources more accessible to diverse learners throughout the country”). Fifteen of the 24 currently active SciAct projects do not include diversity metrics in their evaluation plans, although some of these projects do report numbers of underrepresented participants in their programs or other similar metrics in their monthly and/or annual reports. Overall, the metrics reported vary by project and there do not appear to be targets established by SMD to better understand what success looks like for broadening participation from SMD’s perspective.

We know from current research that increasing diversity alone does not lead to improved outcomes without substantive and deliberate efforts for inclusion (Sherbin and Rashid, 2017). Diversity coupled with inclusion can increase students’ self-confidence and self-efficacy, thus having a positive impact on their classroom performance (Ruggs and Hebl, 2012). As SMD continues to refine the SciAct Program, it is important to think about how attending to other aspects of broadening participation like inclusion (as well as supporting equity and attending to accessibility) could expand the number of people who both participate in SciAct and engage in STEM generally. Also, rethinking how projects can deliberately address diversity, equity, inclusion, and accessibility (DEIA) in their programs and evaluations will

strengthen the overall SciAct portfolio in addressing national educational goals and DEIA overall. The National Science Foundation’s (NSF’s) Inclusion Across the Nation of Communities of Learners of Underrepresented Discoverers in Engineering and Science Program may be a source for best practices for both programming and evaluation efforts within a DEIA context. For further considerations on broadening participation in SciAct, see Chapter 5 of this report.

CURRENT APPROACH TO EVALUATION

Current SciAct awardees are required to have an external evaluation plan as part of their collaborative agreement. SciAct provides awardees with a checklist of required evaluation documents. The plan needs to describe the audience and need for their project, what project activities will be studied and how outcomes will be used in that process; a logic model for the plan; and the evaluation questions, design, and methods. Since each individual awardee develops its own project and evaluation, there is not currently a system for standardization and collaboration on approaches and metrics. This gives projects the opportunity to create a customized plan that best serves the individual project, and allows individual projects to set their own standards for evaluation and project improvement.

This openness to individual interpretation is mirrored in the wide variation among the annual progress reports, which range from 2 to more than 200 pages. These individualized evaluation reports make it challenging to draw conclusions across projects that could inform understanding of the performance of the portfolio as a whole. Indeed, even if it were possible to draw conclusions across evaluation reports, SciAct does not currently have a strategy for synthesizing and disseminating any emergent findings across projects. So while the committee heard presentations from project evaluators eager to contribute to a more comprehensive understanding of the portfolio, in the absence of a formal way to capture their insights, there is no incentive for evaluators to continue this work outside of their own interest.

Awardees are also required to submit an annual “quad chart” as part of the report on their progress. Quad charts provide a templatized snapshot that includes the project summary and participants, current opportunities and risks, and changes to the project plan. In addition, there are several ways for projects to report outcomes, including a mapping of evaluator outputs and measures to SciAct goals, specifically to the top-level objectives and SME interactions. These charts provide a quick, standardized way to assess the performance of the portfolio as a whole at a general level, but the diversity of projects and simplicity of metrics makes it difficult to assess the depth of the work and/or compare across projects to increase cohesion within the network.

The committee recognizes that project evaluators can serve many functions, depending on (among other things) what stage of implementation a project is in, as well as how the evaluation is designed. Evaluators may act as researchers, investigating a scholarly research question; alternatively, evaluators may act as auditors looking to see the fidelity or efficacy of project implementation. When the committee heard presentations from the individual evaluators, several PIs noted that they felt deeply supported by their evaluators as they implemented their awards. The relationship appears to provide PIs with an outside but deeply knowledgeable perspective that can be called upon to provide counsel on project design choices and other strategic concerns in a manner akin to a formative evaluation. Though this collaboration was initially unintended in the design of the SciAct portfolio, PIs were emphatic in their presentations to the committee that the support and real-time counsel of their evaluator partners was invaluable to their work.

As described above, the committee conducted a review of the program portfolio and presentations and considered the role of project evaluators as part of this process. As part of this work, the committee found that current evaluation efforts could potentially be strengthened if SciAct considered

• additional measures that provide a more detailed and refined view of the work,
• a selection of consistent measures and methods that could be used by individual projects and that could be aggregated across the portfolio, and
• sharing of project evaluation results in ways that enhance the utilization of resources across the network (rather than just as a report-out).

COLLABORATION WITHIN THE CURRENT PORTFOLIO

The SciAct portfolio consists of cooperative agreements with individual awardees who are encouraged to think and work like a collective—for example, by leveraging each other’s resources and by sharing tools. The 2017 total solar eclipse served as a unifying opportunity to collaborate given that many projects had planned programming around this event. The national popularity of the event was seen by SciAct leadership as a chance to sustain public interest in NASA-related content, as well as a way to connect awardees around resources. SciAct would like to use the total solar eclipse that will be visible in the United States in 2024 as another opportunity to coordinate efforts among awardees. Coordination around an astronomical event provides a natural way for groups to feel unified and connected, whether they are directly in collaboration or not. The common thread of

NASA-related content means that there are likely collaborators within the portfolio, despite potential differences in project audiences and programs.

Awardees are in regular communication with SciAct leadership through methods that include monthly and annual reporting, quarterly scorecards to PIs, and at least one annual face-to-face session. This provides an opportunity for SciAct leadership to identify potential points of collaboration, as well as communicate desired changes and improvements in a timely manner. Efforts to foster communication among awardees include the annual PI meeting and an internal community Web portal.² This portal serves as a place to share project information, upload monthly reports, and schedule notable events. Posts can be made public for additional dissemination. The portal serves as a communication tool for working groups, which have formed around areas of special interest, such as education technology, girls in STEM, and visualization. The platform supports forum-style posts and commenting, and several affinity groups use other tools for collaboration, such as Google docs.

The annual PI meeting is framed less around individual project reporting and more around professional development for awardees. There is built-in collaboration time throughout the schedule. This time is likely strengthened by pre-identifying desired points of emphasis, such as citizen science, women in STEM, and collaborative tools. One outcome of the 2018 annual PI meeting was the creation of a community of practice for the evaluators of each project. This nascent group has served as a professional learning opportunity for individual evaluators, and its promise for using the collective knowledge to enhance individual projects (or the entire portfolio) is recognized but not yet realized.

SUMMARY

As described above, SciAct has designed a portfolio of awards that are utilizing a diversity of approaches in order to support STEM education across the United States. As a result of this diversity, SciAct is supporting a range of creative ways to use and engage NASA’s considerable assets. Through implementation of their projects, awardees have the potential to both expand NASA’s reach into new communities and bring underrepresented groups into the NASA enterprise. This commitment to supporting projects as they attempt to meet the needs of specific communities is a clear strength of SciAct’s design: In allowing local needs to drive the direction of the awards, SciAct is more likely to maintain both its relevance and its potential for impact.

___________________

² See https://smdepo.org.

CONCLUSION 4: The National Aeronautics and Space Administration (NASA) has developed a portfolio of diverse projects reaching a broad range of communities across the United States that utilize NASA’s resources.

CONCLUSION 5: The Science Activation (SciAct) Program has placed an emphasis on no longer funding education work attached to individual missions. This has eliminated redundancy, but it has also resulted in some missions not being represented comprehensively. The current SciAct portfolio is not consistently incorporating assets from new missions.

CONCLUSION 6: In general, the Science Activation approach has enabled development of partnerships with groups external to the National Aeronautics and Space Administration (NASA) with expertise in education and learning that provide added value for NASA.

CONCLUSION 7: The Science Activation approach to evaluation focuses on evaluating individual projects without adequate attention to how evaluation can inform the whole portfolio. Project evaluators support individual projects and are surfacing important insights that could benefit the portfolio. Among the evaluators, there is interest in contributing to a broader understanding of what is working well, what can be improved, and where there are opportunities that can be further leveraged across the portfolio. However, there are limits, given the current design and program resources, to how much this is possible.

CONCLUSION 8: Interactions among the projects to date have allowed principal investigators and evaluators to share ideas across projects in ways that were unanticipated in the original design. These kinds of collaborations can be built upon and strengthened in the future.

This page intentionally left blank.