6

Recommendations for Science Activation

The committee was charged with two primary tasks: (1) assessing the Science Activation (SciAct) Program’s progress toward meeting its four overarching objectives, and (2) offering feedback on improving the SciAct Program. To carry out these tasks, the committee gathered and reviewed evidence from multiple sources, including official documents from SciAct, testimony from SciAct leadership and stakeholders, and online resources for SciAct awardees. Throughout this process, committee members attempted to bring their expertise in their respective fields to bear on their understanding of the current state of SciAct organization and planning. Based on the evidence, its members’ expertise, and the conclusions presented throughout this report, the committee offers, in this chapter, several recommendations for SciAct to consider. We begin this discussion by offering some insight into two areas of possible action for SciAct to consider in advance of Phase 2 of the portfolio: the need to articulate a portfolio-wide logic model and the value and utility of a portfolio-wide evaluator. We then proceed to a discussion of our specific recommendations and, where appropriate, highlight some additional critical considerations for SciAct leadership as it plans for the future.

In the process of formulating its recommendations, the committee was struck by the considerable value of the SciAct portfolio of investments in the national landscape of efforts to support science, technology, engineering, and mathematics (STEM) learning and engagement. The scope and diversity of SciAct projects are reaching a wide swath of learners across the country, and awardees are employing myriad strategies for engaging potential participants. The recommendations offered here are intended to

support this good work, so that SciAct can continue to amplify and leverage the assets the National Aeronautics and Space Administration (NASA) has to offer the American public. The committee believes these recommendations, if implemented, will enable SciAct to maximize its contributions at the national and local levels and have an impact on the STEM education ecosystem more broadly.

Before presenting our recommendations, we outline some considerations for midcourse reflection on the program, based on the assessment in Chapters 4 and 5.

CONSIDERATIONS FOR MIDCOURSE PROGRAM REFLECTION

The committee’s review of the SciAct Program suggests that there is a natural inflection point for initiating a reflection process as the program transitions from its first phase of funding to the second phase. Such a process can mitigate possible risks associated with a noncompetitive cycle of funding and provide an opportunity for midcourse corrections on the part of individual projects and the program as a whole. The current articulated SciAct objectives can serve as a north star for the program, driving the program’s overall direction; however, as discussed in Chapter 2, the grain size of these goals is too large to facilitate a useful, constructive assessment of the program’s overall impact, the individual awardees’ progress, the potential design changes that could improve individual projects’ effectiveness, and the SciAct Program overall.

Articulating a Logic Model

A well-articulated logic model informs the design, planning, implementation, and evaluation of programs, from small projects to large and complex initiatives such as SciAct. Moreover, it can be an extremely effective tool for facilitating the kind of reflection in which SciAct might engage in at this point in its life cycle. As we discussed in Chapter 3, logic models are a common way to illustrate the linkages among program goals, objectives, and activities. For the overall SciAct portfolio, a logic model can serve as a guide for both planning and designing the portfolio, as well as a tool for assessing the effectiveness of the portfolio and the role of individual projects in achieving high-level goals and long-term impacts. Further, developing a program’s logic model, when done jointly by all stakeholders, including program leaders, evaluators, and participants, creates a common understanding of:

• how change will occur,
• what resources are required,

• how roles and responsibilities are distributed,
• what assumptions are at play,
• what organizational and contextual factors must be attended to, and
• what indicators will be used to assess the extent to which measurable outcomes have been achieved.

This process can sometimes cause frustration when it is perceived as “taking time to think,” which takes time away from “doing the work,” (Reisman and Gienapp, 2004), but without a clear explication of the steps toward the intended outcomes, it is difficult to determine whether outcomes have been achieved, and, importantly, where and how to improve. Moreover, engaging all stakeholders in the process, especially program participants, is another mechanism for broadening participation in its full meaning: diversity, equity, inclusivity, and accessibility.

For a program as complex as SciAct, where multiple awardees are operating projects that are driven by their own understanding of how they believe their work will achieve desired outcomes, developing a shared portfolio-level logic model offers an opportunity to understand the breadth and depth of SciAct in a concrete way, a device for comparing and aggregating outcomes, and a tool for identifying common challenges and potential solutions (Morariu, 2012). The logic model presented in Chapter 3 more closely reflects the underlying relationships among the inputs, activities, outputs, short-term outcomes, and long-term impacts of the SciAct Program. Additionally, the logic model invites discussion of the ways in which the program is functioning as expected, where it can be strengthened, and the degree to which it is achieving its goals. Owing to being too general, the current model in use by SciAct has limited capacity to provide this type of evaluative feedback.

As noted in Chapter 4, however, individual awardees can also benefit from more clearly articulating how they believe their projects will achieve desired learning outcomes—in effect, developing their own, project-specific logic models. Individual awardee logic models would necessarily look different, as awardees are using a panoply of different interventions to achieve different immediate outcomes, but they should all be designed in pursuit of shared high-level goals and long-term impacts. A process for engaging awardees in articulating their own logic models would include the following five stages (ActKnowledge and Aspen Institute Roundtable on Community Change, 2004):

1. Identifying Goals and Assumptions

   Identify clear, concrete, and measurable goals, and identify the preconditions necessary to achieve their goals.

2. Backwards Mapping and Connecting Outcomes

   Sometimes root cause analysis or levels of change, in this stage the steps needed to arrive at the specified outcomes are clearly spelled out, in particular defining the expectations, assumptions, and features of the change process.

3. Developing Indicators

   Determine how to measure the implementation and effectiveness of the initiative, e.g., who is changing, how many are expected to succeed, how much of a change is intended, and how long this change process should take.

4. Identifying Interventions

   Determine what activities will enable the change process identified in Step 2 to achieve the desired goals as defined by the indicators.

5. Writing the Narrative

   This final step translates the initiative from the logic model into ordinary language. This clarifies understanding and highlights the most important components.

Though briefly described, it is possible to see how engaging in the exercise of articulating a logic model could create opportunities to understand and improve awardees’ projects, as well as the SciAct Program, while at the same time laying the groundwork for a comprehensive approach to program evaluation.

Value and Utility of a Portfolio-Level Evaluation

Currently, each SciAct project includes an evaluation team that can support learning and improvement within that individual project. While these individual project evaluators are tasked with characterizing the linkages between project outputs and short-term outcomes, because the current SciAct objectives are so broad, it is nearly impossible for evaluators to draw a line between the awards and portfolio’s objectives. Despite this challenge, the committee heard from evaluators who are eager to contribute to a more comprehensive understanding of the portfolio’s collective impact. As a result, the committee believes it is important for SciAct to consider the value and utility of a cross-portfolio evaluation.

An ongoing portfolio-level evaluation and assessment team could address a range of goals, including but not limited to:

• deepening understanding of how individual projects are aligning with SciAct goals;
• improving portfolio-level coherence throughout all stages of project planning, implementation, and assessment;

• surfacing unique and promising trends within the portfolio;
• identifying areas for improvement on a continual basis;
• representing one mechanism for feedback between SciAct network members and SciAct leadership;
• developing potential models for understanding SciAct’s impact;
• directly facilitating participation routines across the network that support participatory knowledge building, problem and goal identification, and impact assessment approaches.

The independent evaluators that are part of the current portfolio mean that substantial cross-disciplinary expertise is present in the portfolio. The presence of NASA scientists, educational designers, community-based education organizations, and education experts in the form of program evaluators also represents a significant opportunity for coordinated learning and improvement across projects. Capitalizing on these existing resources offers one pathway into ossifying connections across individual projects as well as quantifying portfolio impact.

A Pathway Forward for a Logic Model and Evaluation

As mentioned in Chapter 2, the committee believes that the current four SciAct objectives are more appropriate for informing a vision for the program. A vision is “an aspirational description of what the organization would like to achieve in the mid- or long-term” (Business Dictionary, 2019). Thus, if SciAct’s current objectives became the vision, and another set of overarching goals and objectives that are more specific to SciAct’s logic model were developed, the program would have a touchpoint that could support collective progress (i.e., providing a basis for understanding progress and for sharing advances throughout the network). In their present form, one advantage of the current objectives is that they are sufficiently high level, such that all of the projects can address them in some way. Another advantage is that these objectives enjoy widespread support across multiple sectors and stakeholder groups, and these advantages would be retained even if the four objectives were to be elevated to the portfolio’s vision.

A mid-level set of program goals and objectives that are actionable, measurable, and feasible given the scope of the SciAct resources, and are aligned with individual project goals will support better collaboration and program reporting. Currently, project evaluation plans try to align individual projects’ desired outcomes with the four SciAct objectives. With this approach to evaluation, it is not possible for the program to synthesize actual impacts across the program, except to discuss numbers of people served. Establishing a mid-level set of overarching goals and objectives will provide a basis for discussing impacts as a whole.

Helping the network to adjust to new mid-level goals and an articulated logic model might benefit from the vantage point of a program-wide evaluator. A program evaluator would work across all of the projects to gather and synthesize data to inform the program, to suggest improvements, and to define impact. Awardees have articulated that an overall evaluation must be strategic in order to be useful rather than burdensome. The current project evaluators are working in a group to share practices and to understand the opportunities to move the whole program forward. Those who are doing this work are providing a valuable service to the program, but they are also constrained by the extent to which this program-level effort is supported within their individual projects.

The committee also acknowledges that a robust portfolio-level evaluation would require a substantial allocation of new resources, and for that reason, the committee stops short of explicitly recommending that SciAct pursue that investment in full. However, because of the potential benefits described, the committee believes that SciAct needs to weigh its utility against its associated financial and personnel-based costs. In Recommendation 7 below, the committee offers suggestions for how the SciAct portfolio could pursue immediate, real-time feedback.

RECOMMENDATIONS

The committee offers the following recommendations to Science Activation. Where appropriate, the committee follows these recommendations with additional critical considerations that SciAct leadership should attend to as it plans for the future.

RECOMMENDATION 1: Science Activation (SciAct) should go through a visioning process that brings the portfolio up to date with current research on learning and design, the new federal science, technology, engineering, and mathematics (STEM) plan, and evidence-based approaches to broadening participation. This process should also consider how SciAct fits within and contributes to the larger STEM education ecosystem and should provide the foundation for developing actionable and measureable portfolio goals.

As described earlier in this report, the committee finds that NASA, and SciAct in particular, have a unique role to play in the STEM education landscape in the United States, especially given the tremendous value of NASA’s assets. However, because the four overarching objectives of SciAct in their current form are broad, it is difficult to know exactly what part of this STEM education landscape SciAct hopes to represent and impact. Evidence presented to the committee suggests that SciAct also hopes to pursue ancillary goals that

are not formally captured in its current high-level objectives, such as broadening participation in STEM, which is also a priority area in the current national agenda for STEM education (National Science and Technology Council, 2018). If these are goals of the portfolio, they need to be stated explicitly.

The visioning process we propose would allow the SciAct Program to clarify its role in the STEM education landscape; articulate the specific contributions the program can make in the larger STEM ecosystem; and identify a set of actionable, measurable goals for the portfolio.

RECOMMENDATION 2: Science Activation should articulate how it expects the portfolio will leverage National Aeronautics and Space Administration assets, how partnerships and networks will be built, and how these actions will lead to desired, measurable outcomes.

A clearly defined vision and accompanying objectives are not sufficient conditions for knowing whether or not the work of the SciAct portfolio is progressing toward specific aims. To make claims about progress, SciAct needs to thoroughly describe how it expects the work of its awardees to aggregate toward its objectives for the portfolio. As noted earlier in this chapter, developing a well-articulated logic model is a common approach for specifying how program activities lead to specific outcomes and objectives. To this end, it is necessary to think carefully about what mechanisms need to be in place to help ensure that NASA’s assets are accessed and utilized by awardees effectively, as well as what tools and scaffolds are available to support awardees’ attempts to build partnerships both in their communities and across awards. What, exactly, does SciAct expect to happen in the interactions between NASA assets and awardees, and how can the program’s design and implementation achieve specific targets? Upon completion of a logic model, it should be clear how SciAct expects the work of its portfolio (both collectively and through individual projects) to make measurable progress toward the program’s desired outcomes and objectives.

Critical Considerations for Recommendations 1 and 2. In the process of clarifying its vision, objectives, and portfolio logic, SciAct can make use of contemporary frameworks for the development of STEM learning ecosystems, as described in Chapter 4. Because SciAct awards are already integrated into the national STEM education landscape, it may be worthwhile to consider how to foreground strategies that connect learning across formal and informal education settings. These strategies are particularly valuable when they are successful in connecting in-school and out-of-school STEM learning experiences for youth in grades K–12. To the extent that SciAct can learn from practices being used by others to connect informal and formal learning, the program may find itself better positioned to leverage its work in ways that are more broadly impactful.

Additionally, in the process of clarifying its vision, objectives, and portfolio logic, SciAct may want to consider consistency of language across all portfolio materials, including its Cooperative Agreement Notice, evaluation guidance documents, and project plans. As discussed in Chapter 5, clarity on what constitutes a goal of the portfolio versus a specific, measurable objective toward that goal will help awardees focus both the design and implementation of their awards.

RECOMMENDATION 3: Science Activation (SciAct) must consider whether the development of a coordinated learning network of awardees across its portfolio is a program priority. If it is a priority, then the program must provide the necessary infrastructure to support a more active network of projects. At the very least, SciAct needs to develop more systematic mechanisms for projects to share best practices and learn from successes and failures.

Throughout this study, the committee heard testimony that a strength of the SciAct portfolio is that it has created an opportunity for awardees to learn from and share with one another. Though this opportunity has the potential to enhance the impact of SciAct, the portfolio of awards cannot be expected to function as a network unless specific scaffolds are built into the supporting structure of the portfolio. For example, offering a space to collaborate is not enough to ensure that collaboration occurs.

In its review of the evidence, the committee was struck by how important it is for awardees to be able to learn from one another’s work, and we believe that the efficacy of both the overall portfolio and individual projects could be improved if awardees were offered incentives to and structures within which to collaborate. As described in Chapter 5, different kinds of networks function in different ways and require different kinds of supports. As part of delineating a more highly specified logic model (see Recommendations 1 and 2), SciAct should consider what kind of network would be most effective for helping to achieve the program’s objectives, and then take steps to build in the appropriate supports.

RECOMMENDATION 4: Science Activation should use the opportunity provided by Phase 2 to reflect on the current portfolio within the context of the new vision, goals, and logic model. This process should critically review and guide existing projects, be explicit about the rationale and criteria for including new projects, and consider how best to integrate them into the existing portfolio. One important area for consideration is how to ensure that underserved communities receive more focused attention in the next phase of the program.

As stated in Conclusion 15 in Chapter 5, “while continuing existing awards may allow for continuity and support an environment of collaboration and partnership among existing awardees, lack of competition or opportunities to fund new projects may stifle the evolution of the portfolio.” Indeed, SciAct is at a critical moment in its history: having made a tremendous shift from the 1 percent model to its current program design, it has succeeded in implementing a first phase of funding that supports a broad diversity of exciting projects. While the committee understands the impulse to continue the current state of funding as is, we believe that continuing these awards without any level of scrutiny could stifle innovation and stagnate progress for both individual awards and the portfolio as a whole. Awardees should be offered an opportunity to engage with clear, updated expectations for their success, and SciAct should consider places where current funding fails to address the needs of specific audiences and should identify a set of consistent selection criteria for new awardee applicants. This process could be undertaken as a community rather than as a top-down assessment process, with project leaders working collectively with SciAct leadership to align project-level design and goals with SciAct’s current or re-envisioned overall objectives and logic model.

RECOMMENDATION 5: Science Activation (SciAct) should deepen its commitment to broadening participation by using evaluation measures that go beyond counting numbers of individuals who represent specific groups. In order to do this, SciAct must identify ways that the portfolio as a whole could draw upon and implement evidence-based strategies for broadening participation.

If broadening participation in STEM is indeed one of SciAct’s primary objectives, the portfolio needs to specifically consider how it is reaching and engaging underserved communities. As described in Chapter 4, research has demonstrated the value of intentionally attending to the specific learning needs identified by participant communities, so that project participants feel both welcomed and represented. SciAct awardees should be encouraged to identify ways of leveraging the assets their participant communities bring to their participation, and carefully consider issues of representation in the design and implementation of their projects.

Critical Considerations for Recommendation 5. The issue of extending SciAct funding opportunities should be considered in light of the program’s priorities around broadening participation. SciAct needs to understand clearly and comprehensively what communities it hopes to reach, why, and how; exactly whom its awardees are already reaching; and how those communities understand their STEM learning and engagement needs. Based on this understanding, SciAct should evaluate the extent to which its portfolio is

poised to meet the needs of these communities, and assess whether and how new funding opportunities should be opened to fill specific gaps. The committee notes in particular that despite the considerable reach and resources of minority-serving institutions—and historically black colleges and universities in particular—SciAct has no formal relationships with these institutions.

Specific to supporting the efforts of individual awards in their pursuit of broadening participation, the committee notes that there is a substantial body of literature devoted to providing evidence-based strategies for program designers invested in supporting diversity, inclusion, equity, and accessibility in their projects. Across literatures, several key points emerge that will be important for SciAct to foreground as it supports awardees in these efforts. First, in the design and implementation of their projects, awardees should carefully consider their assumptions about the communities with which they are working, and endeavor to avoid “deficit” thinking when making decisions about engaging with participants, especially as those decisions apply to the work of underrepresented populations (Burgstahler, 2009). SciAct should encourage awardees to actively consider issues of power—and how power differentials may impact participants’ experiences and learning—in the design and implementation of their projects, and support projects as they try to minimize power differentials as much as possible (Elmesky and Tobin, 2005). In effect, SciAct should expect that awardees will actively consider the various assets that their target participant populations bring to the table, and attempt to leverage those assets in the design and implementation of their projects (National Academies of Sciences, Engineering, and Medicine, 2018b).

RECOMMENDATION 6: Science Activation should build ongoing opportunities for dialogue with the National Aeronautics and Space Administration Science Mission Directorate’s missions and scientists.

There is no doubt that NASA’s missions and scientists represent one of its most promising assets for sparking interest in STEM and ensuring that the most current, exciting science makes its way to the American public. However, while it is important that SciAct optimize use of NASA resources, it is also critical to ensure that the appropriate scientific themes are drawn from the missions’ work, and that those themes directly support SciAct’s science education and learning objectives. The committee finds that SciAct is uniquely poised to capitalize on the opportunities presented by the missions and their scientists, and the transition to Phase 2 of the program provides an opportunity to consider mechanisms for engaging them more effectively.

Because NASA missions occur in real time with cutting-edge technologies and exciting phenomena, they hold the potential to build public enthusiasm for science in tangible, immediate ways. SciAct awardees could

benefit from having a mechanism by which to take advantage of these discrete, important moments in time to engage their participant communities. While some of these moments can be planned for in advance, some occur without substantial forewarning. This would likely require that SciAct build into its portfolio flexible funding opportunities that leverage the real-time nature of certain missions.

Additionally, if opportunities to interact with missions in an ongoing way were built into the portfolio, it could help missions understand the science education and learning objectives of SciAct, and scientists could become aware of opportunities to be involved with the program. Because SciAct is committed to meeting the needs of learners, it is critical that NASA scientists understand not only what those needs are, but also how they can and should aim to support those needs.

Critical Considerations for Recommendation 6. If the involvement of mission scientists as subject matter experts (SMEs) continues to be a SciAct priority, SciAct needs to better understand how SMEs are involved in this work. To the extent that SciAct is committed to helping SMEs engage with the public, it needs to ensure that SMEs are engaging in evidence-based practices for communication and education when working directly with participants in SciAct activities. To this end, SciAct leadership and awardees need to fully understand strategies and issues associated with communicating science to the public, and ensure that SMEs who have direct contact with public audiences are effectively trained and supported. SciAct may want to consider how to coordinate its efforts strategically to support SMEs across its entire portfolio.

In order to effectively build an ongoing dialogue between SciAct and SMD missions, it is important to first determine who the appropriate stakeholders are—both at the mission level and within SciAct—who could capitalize on more consistent communication. Once those individuals (or types of individuals) have been identified, these stakeholders will need to determine what type of dialogue would meet everyone’s needs most effectively. Several questions will need to be resolved: What should be accomplished through regular communication, and how will SciAct and mission stakeholders know if these things have been achieved? How often and through what channels should stakeholders be in communication? What are strategies that will enable SciAct to leverage exciting, unplanned discoveries or events to support the ongoing work of awardees? The committee suggests one mechanism for helping SciAct address these questions in Recommendation 7 below.

As the committee synthesized the evidence for this study, it became clear that one challenge facing SciAct is the potential friction between its twin ambitions of (1) optimizing NASA’s assets and (2) meeting the learning needs of local communities. While it is certainly possible that NASA’s

existing assets may map perfectly to what communities need to support learning, this is by no means guaranteed. Indeed, it is the job of awardees to translate these assets into resources and experiences that address the needs of their participants. In doing so, however, awardees will likely need to make choices as to what to prioritize as they design and implement their projects. The committee believes that by building ongoing opportunities for dialogue with NASA’s missions and scientists, SciAct will be better positioned to balance the potential tension of pursuing these two aims.

RECOMMENDATION 7: Science Activation should create an independent mechanism to obtain ongoing, real-time advice from individuals with expertise in learning and design, the larger policy context of science, technology, engineering, and mathematics (STEM) education, partnering with local communities, broadening participation in STEM, and science content relevant to the missions of the National Aeronautics and Space Administration Science Mission Directorate. Among other responsibilities, these experts should inform the new visioning and planning process.

The committee recognizes the considerable effort that has gone into transforming SciAct into the impressive portfolio of awards that it is today. To sustain and expand this progress, SciAct should create a mechanism to obtain ongoing, expert input so that portfolio strategy can be informed by trustworthy, real-time feedback. Such a group would be able to assist SciAct leadership as it plans for the next phase of funding, but would also be available to advise around more immediate design and implementation issues.

Critical Considerations for Recommendation 7. The committee is sympathetic to the fact that the SciAct team is accomplishing a tremendous amount with a small staff. Drawing on outside expertise is an important strategy for supplementing, enhancing, and clarifying the goals and objectives of the SciAct portfolio. Independent advisors bring expertise that can help with the visioning process proposed in Recommendation 1 and the development of the logic model proposed in Recommendation 2, and help situate the distinctive work of SciAct within the larger STEM education ecosystem, which should include federal and state STEM education policy. Further, individuals with specific expertise in the science content relevant to SMD’s missions should be able to help SciAct be strategic in its approach to building a mutually beneficial dialogue between SciAct stakeholders and SMD’s missions and scientists. The committee notes that there are multiple strategies for how to facilitate obtaining expert feedback of this type, such as the procuring of individual consultants with specialized expertise, engaging a set of active awardees and project-level evaluators, or forming an independent group of relevant experts.

RECOMMENDATION 7a: With input from these experts, Science Activation (SciAct) should consider whether and how a portfolio-level evaluation could strengthen the focus of the program and ensure that projects in the portfolio are effectively meeting overarching SciAct Program goals and objectives.

On the issue of a portfolio-level evaluation, ultimate decisions around the SciAct vision and objectives, as well as its logic model, are likely to inform whether or not a portfolio-level evaluation has utility. The committee sees the potential for multiple outcomes here. Based on the vision and objectives identified for SciAct, it is possible that the individual project evaluators could be mobilized to better inform SciAct leadership about the progress of the portfolio. It is also possible that an outside entity could offer clearer insight into whether the portfolio’s awards are truly aggregating toward achievement of the portfolio’s objectives. In either case, experts that have been engaged in the process of informing the development of SciAct’s vision and objectives would be well positioned to determine whether a portfolio-level evaluator would be a valuable investment.

This report represents the committee’s best efforts to describe the current state of and best pathways forward for the SciAct portfolio. Based on our understanding of the evidence, we believe that with effort directed at clarifying its vision, objectives, and logic model, the SciAct portfolio can be an invaluable resource in the national effort to support STEM education and engagement. SciAct is uniquely positioned to leverage NASA’s considerable assets, and it is our belief that the SciAct portfolio holds tremendous potential for supporting the needs of diverse learners nationwide.

This page intentionally left blank.