Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
6 Conclusions and Recommendations T his chapter presents the committeeâs conclusions and recommen- dations on NASAâs overall K-12 science, technology, engineering, and mathematics (STEM) program and on the agencyâs seven core headquarters projects; it also provides answers to the four specific questions asked in the congressional charge for this study. The committeeâs conclu- sions and recommendations are based on the materials and testimony out- lined in Chapter 1 and informed by the scientific, engineering, educational, and evaluation expertise of its members. Given the short period available for the study, the complexity of NASAâs education and public outreach activities, and the limited evaluative information available, the committeeâs answers are based primarily on the expertise of its members. The first section of this chapter deals with strategies for NASAâs pro- gram in K-12 education. The second section deals with the core projects that were the specific focus of our study. The third section provides recom- mendations for improving evaluation of programs and projects. The final section presents our answers to the four questions from Congress. STRATEGIES FOR NASAâS PROGRAM IN K-12 STEM EDUCATION NASA has a broad mandate to engage in the expansion of human knowledge, as stated in its founding legislation, in subsequent legislation reauthorizing the agency, and as emphasized in its current (2006) strategic education framework. Education and public outreach play a part in fulfill- ing this mission. Furthermore, as a federal science agency supported with public money, NASA has a responsibility to provide citizens with a return 112
CONCLUSIONS AND RECOMMENDATIONS 113 on their investment. For NASA, this responsibility is fulfilled in part by ensuring that the discoveries, knowledge, and information that result from its science, engineering, and exploration programs are effectively shared with the public. One of the ways that this sharing takes place is through NASAâs education programs. To successfully implement its mandate in education, NASA needs a clear view of the programâs goal and strategies for stability, project goals, and partnerships. Overarching Program Goal NASAâs role in K-12 STEM education is both motivated and con- strained by NASAâs overall mission as a science, engineering, and space exploration agency. The National Science Foundation and the U.S. Depart- ment of Education share the lead federal roles in K-12 STEM education and are responsible for the primary federal investment in these activities. Thus, NASA, like other federal science agencies, has an important but complementary role in K-12 STEM education. The assets NASA brings to this role come from the agencyâs contributions in science and technology made through the work in the mission directorates: Science, Aeronautics Research, Exploration Systems, and Space Operations. Conclusion: The primary strengths and resources that NASA brings to K-12 STEM education are its scientific discoveries, its technology and aeronautical developments, and its space exploration activities, as well as the scientists, engineers, and other technical staff that make up its work- force and the unique excitement generated by flight and space exploration. Because engineering and technology development are subjects that are not well covered in K-12 curricula, projects aimed at inspiring and engaging students in these areas are particularly important. Recommendation 1â NASA should continue to engage in education activities at the Kâ12 level, designing its Kâ12 activities so that they cap- italize on NASAâs primary strengths and resources, which are found in the mission directorates. These strengths and resources are the agencyâs scientific discoveries; its technology and aeronautical developments; its space exploration activities; the scientists, engineers, and other technical staff (both internal and external) who carry out NASAâs work; and the unique excitement generated by space flight and space exploration. Recommendation 2â The exciting nature of NASAâs mission gives par- ticular value to projects whose primary goal is to inspire and engage studentsâ interest in science and engineering, and NASAâs education portfolio should include projects with these goals. Because engineering
114 NASAâS ELEMENTARY AND SECONDARY EDUCATION PROGRAM and technology development are subjects that are not well covered in K-12 curricula, projects aimed at inspiring and engaging students in these areas are particularly important. Recommendation 3â NASA should provide opportunities for teachers and students to deepen their knowledge about NASA-supported areas of science and the nature of science and engineering through educa- tional activities that engage them with the science and engineering carried out by the mission directorates. Program Stability NASAâs education and public outreach activities at the K-12 level are broad in scope and varied in regard to the level of effort and involvement. The current portfolio includes projects that originated through different mechanisms and in different places in the agency. Over the past 5 years, the education priorities and management struc- ture have changed multiple times. For example, in the 13-month period between September 2005 and October 2006, there were four different assistant or acting assistant administrators for education. NASAâs education program has also faced a downward trend in the budget and specifically for K-12 STEM education activities. The reduction in funds is due in part to reductions in budget allocations from Congress as well as redistribution of funds within the agency. The program has also been negatively affected by increases in the number and dollar value of congressional earmarks for projects that have been designated for the headquarters Office of Education without a concomitant increase in budget. These earmarksâfrom $19 million in fiscal 2005 to $39 million in fiscal 2006âlimit the headquarters Office of Educationâs ability to make judg- ments about resource allocation that are based on an overall strategy for the Elementary and Secondary Program and on the merits and needs of individual projects. In addition, the percentage of mission funds that must be allocated to education and public outreach in the Science Mission Direc- torate was recently reduced as the result of an agency-level decision. Conclusion: Although some projects have existed for many years, NASAâs K-12 STEM education portfolio has experienced rapidly shifting priorities, fluctuations in budget, and changes in management structure that have undermined the stability of programs and made evaluation of effectiveness challenging, if not impossible. The increasing number of congressional earmarks has contributed substantially to this problem.
CONCLUSIONS AND RECOMMENDATIONS 115 Recommendation 4â NASA should strive to support stability in its education programs, in terms of funding, management structure, and priorities. Project Goals NASAâs portfolio of K-12 STEM education projects will always face a tension between reaching smaller numbers of teachers and students in a deep and sustained way and reaching relatively large numbers of t Â eachers and students in a less intense or sustained way. As implied in R Â ecommendationsÂ 2 and 3, both goals are appropriate for NASAâs educa- tion program. The challenge is to make sure that a given project is designed with a plan that is realistic for the desired reach and intensity and to main- tain an appropriate balance across the portfolio. The new strategic framework for education represents an effort to bring some order to the agencyâs overall portfolio. It articulates a set of program goals and objectives toward which all projects should aim. These goals are broad and cannot all be accomplished by any one project. Indeed, they are so broad that they cannot be accomplished by NASA education efforts alone, but need to be viewed as goals towards which NASAâs efforts should contribute as part of a national agenda. Currently, in the case of the seven core Office of Education projects, the goals and objectives articulated for each project in the individual project descriptions reflect very closely the overall goals for the entire K-12 STEM education program. This is unreal- istic, given the breadth and generality of the overall goals, and may be mis- leading to both program managers and participants about what any given project can accomplish. For example, long-term professional development is inappropriately defined as anything lasting more than 2 days. Thus, there is a need to better articulate and focus the goals and objectives of individual projects and develop a portfolio of projects that in its entirety achieves the program goals for K-12 STEM education. Conclusion: Many of the projects within the headquarters Office of Edu- cationâs K-12 STEM education program do not have clear, realistic, and appropriately defined project-level goals and objectives that reflect the resources available and the target audiences for them. Recommendation 5â NASA should take a more intentional approach to portfolio development than it has to date so that individual projects are well defined and have clear and realistic goals and objectives given their target audiences. Management of the resulting portfolio should include periodic review of the balance of investment across projects.
116 NASAâS ELEMENTARY AND SECONDARY EDUCATION PROGRAM Partnerships Given NASAâs primary focus on science, engineering, and technology, the agency employs a large staff with expertise in those areas, while the number of agency staff with primary expertise in education is limited. However, the design and implementation of effective K-12 STEM educa- tion projects requires substantial expertise in education, including knowl- edge of pedagogy, curriculum development, professional development, and evaluation. NASAâs technical staff cannot be expected to have sufficient expertise in K-12 STEM education to allow them to develop effective education Âprojects on their own. Thus, the scientists and engineers in the agency need to work in concert with experts in education, often from outside the agency, in order to achieve the appropriate mix of expertise in science, engineering and edu- cation to design and implement effective education projects. Partnerships with educational organizations can also provide a mechanism for leveraging the reach and impact of NASAâs projects. The use of partnerÂships to provide both expertise and mechanisms for dissemination was explicitly supported in the Office of Space Scienceâs approach to education and outreach, and is reflected in the agencyâs 2006 strategic framework for education. The K-12 STEM education projects in NASA and in other federal sci- ence agencies that the committee found especially promising usually rested on partnerships between agency science and engineering experts and outside experts in education. In space science at NASA, some supplementary cur- riculum materials were developed in partnerships with the Great Explora- tions in Math and Science project of the Lawrence Hall of Science; in earth science, outreach for the EarthKAM project was developed in partnership with the Technology in Education Research Center (TERC). These part- nerships appear to have been successful. The broker-facilitator and forum model supported through the Office of Space Science successfully supported the use of such partnership approaches to leverage the educational Âproducts developed in the missions, as well as facilitating cooperation among dif- ferent projects with related science goals in developing coordinated educa- tional efforts. The headquarters Office of Education projects reviewed by the commit- tee do not consistently capitalize on this kind of partnership. For example, there are several existing, successful models of school reform with which NASA Explorer Schools could have partnered to provide resources in STEM education and enhance the reform work through the draw of the NASA âbrand,â but such partnerships do not appear to have been widely sought. Likewise, DLN modules are generally developed by NASA scientists and engineers and do not appear to involve partnerships with individuals or organizations with expertise in curriculum development. In some cases
CONCLUSIONS AND RECOMMENDATIONS 117 where NASA education projects have had successful partnerships, these have not been sustained due to changes in management and/or program direction. For example, neither the EarthKAM outreach through TERC, nor an earlier effective partnership with the Council of State Science Super- visors, are currently supported. Partnerships can leverage NASAâs investment in education by ensuring that projects are designed with knowledge of research and best practice in education, aligned to the needs of the constituency the projects intend to serve, and effectively disseminated and integrated in school planning and curricula. For some projects, NASA is appropriate in the lead role, princi- pally for projects that are chiefly aimed at inspiring students or teachers by exposing them to NASAâs science and engineering achievements and chal- lenges. Even for these projects, however, design and implementation can be improved by appropriate partnerships. For other projects, such as school improvement, NASA brings a valu- able resource for enhancing student engagement in science, but it can best be used as part of a coordinated school improvement effort, with lead partners who can provide the educational and organizational support and long-term stability that are necessary for such work. The key is to select partners that bring the relevant educational expertise (curriculum develop- ment, professional development, pedagogy, district and state school system knowledge, evaluation, etc.) or that are positioned to leverage a project for broad audiences. Conclusion: NASA scientists and engineers have the expertise to introduce teachers and students to the processes of science and engineering and to the cutting-edge research related to science and engineering activities at NASA. However, to be effective in K-12 STEM education, they need to work in concert with professionals who have specific expertise in education. Conclusion: Partnerships with educational organizations can provide opportunities to leverage NASAâs projects through use of established infra- structures for dissemination. Conclusion: Examples of projects in NASAâs K-12 education portfolio that reflect knowledge of best practice usually involve a partnership between NASA and other individuals or organizations that bring a proven expertise in education. Conclusion: When designing and implementing programs and projects, NASA has not consistently and strategically built, sustained, and leveraged long-term links with the K-12 education system at the district and state l Âevels, with broader science and mathematics education-related organiza-
118 NASAâS ELEMENTARY AND SECONDARY EDUCATION PROGRAM tions, and with experienced curriculum development, professional develop- ment, and evaluation organizations. Recommendation 6â NASA program and project planning and execu- tion should make better and more consistent use of opportunities to involve education stakeholders, to partner with individuals and orga- nizations that can provide expertise in education, and to connect to the existing infrastructure for K-12 STEM education. Recommendation 7â NASAâs partnerships in education should be designed in light of the specific objectives of each project. NASA can play a lead role in projects intended to inspire and engage students and should use strategic partnerships to leverage the impact of such projects. For projects designed to affect schools, through work with students, teachers, or curriculum materials, NASA should work in partnerships with organizations that complement NASAâs science and engineering expertise with education-specific expertise and avenues of dissemination. All partnerships should begin during the early stages of project design. Role of the Headquarters Office of Education The scientific and technical work related to missions is undertaken at the NASA centers, affiliated universities and research institutions, and industry contractors. As a result, the scientific and engineering expertise that is closest to the work of the missions is at the centers, with investiga- tors at universities and research institutions, and with mission contractors. Given this pattern, the committee supports the recent shift in management of specific projects to the centers following the 2006 strategic framework and the altering of the role of the headquarters Office of Education to coordination and oversight of the portfolio. The committee assumes that the specific activities carried out by the headquarters Office of Education will include â¢ ensuring the sharing of good practices among NASAâs education programs and projects; â¢ supporting information dissemination and shared technology, in part through maintaining a user-friendly and regularly updated website that can enable teachers and students to readily find NASA education projects and materials; â¢ ensuring adequate data collection and evaluation to assess the q Â uality of the programs delivered;
CONCLUSIONS AND RECOMMENDATIONS 119 â¢ acting as a broker among the centers and programs to ensure that excellent educational projects or strategies developed by one center or program are appropriately shared; â¢ advocating and planning for the inclusion of appropriate educa- tional activities in the programs of the four operating directorates (Space Operations Mission Directorate, Exploration Systems Mis- sion Directorate, Science Mission Directorate, and Aeronautics Research Directorate); and â¢ coordinating with the efforts of other federal agencies. Three other roles are critical for the headquarters Office of Education. One is to monitor the balance of the education portfolio and to maintain the appropriate mix of projects. The second is to ensure that education projects are informed by research-based best practice ideas in education and are appropriately evaluated. The third is to ensure that projects leverage the science and engineering expertise of the agency through carefully chosen partnerships. Coordination with the efforts of other federal agencies is also an important role for the headquarters Office of Education. Recommendation 8â The NASA headquarters Office of Education should focus on leadership and advocacy for inclusion of education activities in the programs of NASAâs four operating directorates, quality assurance, internal coordination, and coordination with other agencies and organizations. In the development of new education projects, the office should partner closely with the mission directorates or centers and consult with external education experts. Use of Information and Communications Technology The committee found that K-12 STEM education projects were often using information and communications technology that is outdated (see Chapter 4). For example, the Aerospace Education Laboratory used by SEMAA is expensive and may not be the most cost-effective way to achieve the project goals of inspiring and engaging students through NASAâs science and engineering. The agency does not have a consistent effort to periodi- cally review project designs to determine whether advances in technology could be exploited to make them more cost efficient or to disseminate them more broadly. The committee also found that the official NASA website for education is difficult to navigate and does not provide easily accessible and in-depth information about the full range of NASAâs K-12 STEM education port- folio. Links to mission-related materials are especially hard to find. Given
120 NASAâS ELEMENTARY AND SECONDARY EDUCATION PROGRAM that the website is the most likely point of entry for many first-time users of NASAâs educational materials, a user-friendly and coherent website is particularly important. Finally, the central database for education, the NASA Education Evalu- ation Information System (NEEIS), is cumbersome to use and is in need of updating and revision. It has hampered the efforts of external evalua- tors and likely undermines any projectâs use of data to inform continuous improvement. Conclusion: Projects use the information and communications techÂnology that was current at the time of inception; they do not make efforts to peri- odically update technology. Continued use of outdated technology can lead to inefficiencies in use of project funds. Recommendation 9â NASA should make better use of current and emerging information and communications technology to provide broader and more user-friendly access to NASA materials, to support NASAâs K-12 STEM education activities, to extend the reach of NASAâs education activities, and to maintain a centralized data system. Recommendation 10â NASA should periodically review each project to determine whether its components are the most cost-effective uses of resources, given current information and communications technology alternatives. CORE PROJECTS OF THE HEADQUARTERS OFFICE OF EDUCATION This section presents the committeeâs conclusions and recommenda- tions for three of the seven core projects in the headquarters Office of Edu- cation: NASA Explorer Schools (NES); the Aerospace Education Services Project (AESP); and the Science, Engineering, Mathematics and Aerospace Academy (SEMAA). The other four core projects the committee was asked to review are either in early stages of implementation (Educator Astronaut Project), not yet active (INSPIRE), or currently being revised (Education Flight Projects and Digital Learning Network): For this reason, the com- mittee does not consider it appropriate to draw definitive conclusions or make recommendations for these projects.
CONCLUSIONS AND RECOMMENDATIONS 121 NASA Explorer Schools The committee had concerns about the scope of the NASA Explorer Schools (NES) project, given the resources and expertise in education that NASA, acting on its own, can provide to the schools involved. The current model for the NES project shares some features with models for content- specific whole-school reform, such as the use of a school-based leadership or action team to guide the project, the involvement of families in the school community, and building local support. However, NES lacks key features needed for the goal of broad reform, principally use of well-aligned cur- riculum, instruction, and assessment that are standards based; sustained professional development and teacher learning communities; and focused attention on student learning. In addition, the level of resources that NASA is able to provide to individual schools and the duration of schoolsâ involve- ment (3 years) are both insufficient for producing deep and lasting changes in STEM instruction. Thus, NES is unlikely to be able to achieve the whole- school curricular reform in STEM education that participating schools are anticipating. Conclusion: The NASA Explorer Schools project appears to promise support for whole-school STEM reform but does not have the capacity, resources, or duration to do so. Recommendation 11â The NASA headquarters Office of Education, in collaboration with project managers from the NASA Explorer Schools, should rethink the model for NES given its time, personnel, and budget resources. NASA should not have a leadership role in comprehensive school STEM reform efforts. However, by partnering with other suc- cessful reform efforts, NASA can bring valuable additional resources to support and enhance that work. The Aerospace Education Services Project The Aerospace Education Services Project is one of the longest running projects in NASAâs K-12 STEM education program and has enjoyed an enthusiastic group of supporters who continue to use its services. Recent external evaluations of the project documented the potential strengths of having regional specialists in place to develop ties to local educational organizations. However, the strength of those ties depends on the skills of individual specialists and how long they stay in the positions. The over- all effectiveness of the project would likely be improved if it were fully integrated and coordinated with state and local education agencies. The
122 NASAâS ELEMENTARY AND SECONDARY EDUCATION PROGRAM project would also benefit from identifying the reasons for high turnover of specialists and consider ways to achieve greater stability in staffing these positions. The amount and kind of training provided to specialists also appears to be insufficient. Currently, specialists participate in yearly professional development sessions to update their knowledge of NASA missions. They do not receive support to develop expertise in designing and providing professional development, yet much of their work is in this area. Moreover, the strategy for responding to requests for AESP services appeared to be on a first-come, first-served basis. A more systematic priority system that is designed strategically to reach desired audiences and expand the network of schools and teachers that use NASAâs resources is desirable. Conclusion: AESP specialists need more frequent and more in-depth oppor- tunities to learn about the science and engineering related to the missions, especially because this content can change continuously over the life of a mission. Specialists may also need more support to maintain and update their expertise in education, particularly in areas they may not have had the opportunity to develop as classroom teachers, such as designing and providing professional development for teachers. Recommendation 12â The AESP project should be designed so as to better integrate and coordinate services with state and local education agencies. Recommendation 13â Specialists in the Aerospace Education Services Project should receive more intensive and more frequent training to ensure they have sufficient understanding of the science and engineer- ing issues related to the educational products that they are expected to disseminate. They also should receive professional development in aspects of education in which they may not have developed expertise as teachers, such as providing professional development for teachers. Recommendation 14â The AESP project managers, in collaboration with the NASA Office of Education, should set priorities for providing services to teachers and schools other than doing so on a first-come, first-served basis. The Science, Engineering, Mathematics and Aerospace Academy The committee commends SEMAA for its focus on underserved and underrepresented populations of students and on inspiring their interest
CONCLUSIONS AND RECOMMENDATIONS 123 in science and engineering. The project has developed a number of good strategies for reaching students and their families and has worked hard at raising matching funds to leverage the dollars provided by NASA and to provide ongoing student opportunities at SEMAA sites after NASA funding terminates. The committee notes two aspects of the project that need attention. First, related to the broad concern about the cost-effective use of technol- ogy (see above), the committee questions whether the aerospace education laboratories use up-to-date technology and whether putting a lab at each SEMAA site is cost effective in terms of the projectâs intended outcomes. For example, the committee thought that computer simulations might offer an alternative and much cheaper flight simulator experience. Second, SEMAAâs menu of curriculum enhancement opportunities can better reflect the science and engineering of current missions. To do so would require periodic updating of the projectâs offerings. In addition, some more sys- tematic follow-up efforts to investigate the longer term impact of SEMAA participation would be valuable for the project. Conclusion: SEMAA is an excellent project for reaching the intended par- ticipants. The use of an after-school project to reach underserved popula- tions and inspire their interest in science and engineering appears to be an effective strategy. Conclusion: The projectâs use of technology, particularly the aerospace education laboratories, needs to be reconsidered. Recommendation 15â The SEMAA project manager, in collaboration with NASA headquarters Office of Education, should assess whether the Aerospace Education Laboratory is the most cost-effective way to achieve project goals. The outcome of this assessment should guide revision of the projectâs model. Recommendation 16â The SEMAA menu of activities should be updated periodically to reflect current NASA science and engineering activity. These updates should be carried out in partnership with orga- nizations that have expertise in curriculum development and with input from agency scientists and engineers. PROGRAM AND PROJECT EVALUATION Evaluation is an essential strategy for maintaining an effective Âportfolio of programs and projects. The challenges of carrying out appropriate evalu-
124 NASAâS ELEMENTARY AND SECONDARY EDUCATION PROGRAM ations of NASAâs K-12 STEM education projects, and of its overall pro- gram, are large; most federal science agencies engaged in education are struggling to meet similar challenges. Ideally, evaluation should be an inte- gral part of NASAâs education program, incorporating both internal and external mechanisms with varying degrees of formality. At a fundamental level, successful evaluation entails approaching the portfolio with a critical eye. A âculture of evaluationâ would mean that education staff and project managers regularly and systematically review projects with an eye toward continual improvement and that data are gathered with the intent of using them to guide that improvement. In this context, the headquarters Office of Education needs an overall evaluation plan for the K-12 education program and its projects. Such a plan would help to identify the appropriate questions that address program and project goals and outline the mechanisms by which results of evaluation would inform project implementation. NASA does not have this kind of overall evaluation culture and plan. Evaluations of individual projects have not been done systematically and are of uneven quality. There is little evidence that the results of project evaluations have guided decisions about projects. The overall evaluation plan needs to address how well the program as a whole is achieving its stated goal to âattract and maintain students in STEM disciplines.â Such a plan will necessitate longitudinal studies of samples of students participating in a variety of NASA-based K-12 activi- ties. Such studies are difficult and expensive; NASA may wish to consider whether this need can be served by participation in some more general or cross-agency longitudinal studies of student attitudes to and participation in STEM disciplines. The overall evaluation plan also needs to address questions regarding the outcomes of individual projects. Given resource constraints, external evaluations of individual projects can be scheduled on a cyclical basis, with high priority given to projects intended to have the greatest impact on student engagement and learning and to projects that face important questions about activities, participants, staffing, funding, or organization. The overall quality of the external evaluations conducted on NASAâs K-12 STEM education projects has not been high. As discussed in Chap- terÂ 5, these evaluations contained flaws in design, data quality, analysis, and interpretation that undermined confidence in the results. In most cases, NASA had used an external evaluator with the appropriate expertise, but the evaluator was not involved in early data collection decisions and had to work with whatever data had been collected by the projects. Additional mechanisms to draw on expertise in evaluation in education would be appropriate, such as expert review of proposed evaluation plans and data collection at the initiation stage of a project or advisory panels to offer
CONCLUSIONS AND RECOMMENDATIONS 125 periodic advice on the overall evaluation plan and on evaluations of indi- vidual projects. NASA now collects data on the numbers of K-12 teachers and students participating directly in NASA-sponsored events and on participantsâ reac- tions to these events. NASA divides activities for teachers into short- and long-term categories, although the agency does not appear to record actual program length or number of repeat participants. The currently collected data can become part of a NASA information system on its education and public outreach activities and may be useful for project monitoring. For evaluation purposes, however, data are also needed on the con- ditions under which the project is conducted, such as characteristics of participants and staff; frequency of activities; materials used; and repeat participants as the basis for analysis that can identify conditions associated with better outcomes or what audiences are actually being served. Such information can help to improve projects and their implementation. Conclusion: NASA lacks an effective overall plan for evaluation of its K-12 portfolio of projects that includes definition of measurable project goals and objectives, framing of the purposes of evaluations and key questions, and a plan for how information from the evaluations will inform the design and implementation of projects. Conclusion: Effective project design and management requires that a proj- ectâs goals, desired outcomes, and evaluation questions be aligned. This was generally not the case for the seven core headquarters Office of Education projects reviewed by the committee. Conclusion: Data are needed to serve dual purposes: project monitoring and fiscal due diligence, and program evaluation. Current data collection systems are structured primarily for the former. The current system for data collection, NEISS, is difficult to use and focuses mainly on collecting descriptive data such as counts of participants and participant self-reports. Such data are important for monitoring project activities, but are not s Â ufficient for conducting evaluation of the effectiveness of projects. Conclusion: In recent years many projects have been subject to rapidly changing directives that shifted project goals and activities from year to year. With ongoing changes in focus, it is difficult, if not impossible, to design an evaluation that can capture project impacts across multiple years. Recommendation 17â NASA should develop an overall evaluation plan for its K-12 education program and projects and allocate the resources needed to implement the plan.
126 NASAâS ELEMENTARY AND SECONDARY EDUCATION PROGRAM Recommendation 18â For portfolio management, the NASA evalu- ation plan should include some cross-project evaluations as well as project-specific evaluations. Recommendation 19â NASA should plan the scale, design, and fre- quency of each project evaluation so that it aligns to the scale and goals of the project and to the nature of the decisions that need to be made. Recommendation 20â NASA should use evaluation findings to inform project design as well as project improvement. To do so, NASA should establish mechanisms to connect evaluations to program and project decisions. Recommendation 21â Data and record keeping should be planned to facilitate both project monitoring and evaluation needs. Recommendation 22â All NASA evaluations should meet professional standards for evaluation. NASA should take advantage of external evaluation expertise to ensure that such standards are met. ANSWERS TO THE QUESTIONS FROM CONGRESS These responses draw on the conclusions and recommendations above, as well as on material discussed in detail in the preceding chapters. We wish to note that the timeline of this study did not allow the committee to collect original data on the projects. Rather, we relied on secondary sources such as annual reviews, reports from external evaluations, and data provided by NASA. The committeeâs conclusions should be interpreted in view of these constraints. Question 1: What is the effectiveness of the K-12 STEM education program in meeting its defined goals and objectives? The projects are somewhat effective at raising awareness of the science and engineering of NASA missions and generating studentsâ and teachersâ interest in STEM. As presently configured, they cannot be shown to be effective at enhancing learning of STEM content and providing in-depth experiences with the science and engineering of the missions. We also note that there are program elements that do not align with research-based best practice in education (see Chapter 4). NASAâs K-12 STEM education program would be well served if Âprojects consistently drew on expertise in education through partnerships with
CONCLUSIONS AND RECOMMENDATIONS 127 educational organizations and agencies to guide project development and implementation. We recommend (above) that the headquarters Office of Education adopt an approach to managing the K-12 program that includes periodic review of project implementation and impact, with the intent of revising individual projects or adjusting the balance of projects in the port- folio when necessary. At NASA, as is the case in other federal science agencies involved in education, few projects have been formally evaluated, and none has been evaluated rigorously. Thus, there are few data across projects on which to base conclusions about effectiveness. NASA does not have a coherent overall plan for evaluation, nor for how to use results of evaluation to inform both overall and project-specific decisions about program design and implementation. All of these factors made it difficult for the committee to form an accurate assessment of the effectiveness of NASAâs K-12 educa- tion activities in meeting their high-level goals. Carrying out a rigorous evaluation of the overall program has been fur- ther complicated at NASA because rapidly shifting priorities, fluctuations in budget, and changes in management structure have undermined the stability of projects and made evaluation of effectiveness virtually impossible. Most of these shifts and fluctuations are due to factors outside of the control of the NASA headquarters Office of Education, including budget reductions, congressional earmarks, and administrator turnover. Question 2: What is the adequacy of assessment metrics and data collection requirements available for determining the effectiveness of individual projects? Given that the overarching goals for education at NASA are extremely broad, appropriate metrics are difficult to develop and program effective- ness is difficult to assess. Individual projects have taken on these broad goals rather than developing specific goals and objectives that are appro- priate to the design and scope of each project. This lack of project-specific goals makes it difficult to measure project impact. Data collection efforts, common to all projects, consist chiefly of counts of sessions offered, numbers of attendees, and immediate feedback from them. This information is insufficient to evaluate the effectiveness of Âprojects or of the program as a whole. Large-scale continuing projects should develop project effectiveness measures. After a program is established, it should undergo periodic outcome evaluation conducted by external evaluators. The current data collection system, NEISS, is geared more toward accounting and tracking numbers of participants reached than toward evaluation. It does include some measures of participant satisfaction, but it does not collect extensive information related to outcomes. In addition,
128 NASAâS ELEMENTARY AND SECONDARY EDUCATION PROGRAM the current data collection system appears to be cumbersome to use, for both external evaluators and program staff (see Chapter 5). Moreover, project staff do not appear to consistently use data collected by the system to inform the continued improvement of the project. Thus, NEEIS serves a reporting and project-tracking function, but it does not support effective evaluation. Evaluation of effectiveness is further complicated by the challenge of evaluating projects that focus on inspiration and engagement. NASA has appropriately focused many of its activities on inspiring and engaging stu- dents in order to encourage them to become interested in and to pursue careers in science and engineering. These outcomes are perhaps even more difficult to assess than growth in student or teacher learning. Assessing lasting impact requires long-term records and comparison groups, but no systematic long-term impact records are available for NASA projects, even for those that have existed for a substantial amount of time. Tracking past project participants can be difficult and expensive and likely beyond the capacity of individual projects. In addition, it is unrealistic to expect projects that provide only short-term experiences for teachers or students to maintain such records. Thus, the committee recommends that the headquarters Office of Education consider a program-level longitudinal study that includes participants from those projects intended to have long- term impacts, such as SEMAA and INSPIRE. Finally, NASA does not have a coherent overall plan for evaluation and for how results of evaluation should inform program and project design and implementation. The external evaluations that were reviewed by the committee were nearly all based on project data collection about and from p Â roject participants collected at the time of their participation. In some cases, these evaluations were used to inform ongoing project improvement, but this was not uniformly the case. In addition, the quality of external eval- uations was not consistently high: they generally had many shortÂcomings in their approach to data collection, validity of measures, methods of data collection, sampling procedures, response rates, and analytic methods, and, in some cases, even in interpretation of results. Question 3: What is the state of the funding priorities in the K-12 education program, including a review of the funding level and trend for each major component of the program, to include an assessment of whether available resources are consistent with meeting identified goals and priorities? NASA supports K-12 STEM education through funds directly received by the headquarters Office of Education, as well as by mandating that a percentage of funds from individual science missions be designated for
CONCLUSIONS AND RECOMMENDATIONS 129 education activities. In this way, the agency is demonstrating its strong com- mitment to supporting STEM education. Because NASAâs K-12 STEM activities originate in different administra- tive units in the agency, it is difficult to track all of the funding. Funds for the headquarters projects are well documented and come out of the line item budgeted for education. This amount has declined from $230Â mil- lion in 2003 to $153 million in 2008, and program planning has been significantly affected by an increasing number of earmarks taken out of the education budget. Such funding uncertainties, coupled with management changes, have made it difficult for the agency to maintain a consistent approach and to appropriately match its program elements and their goals to the available funding (see Chapter 2). For the projects that originate in the headquarters Office of Educa- tion, NASA has formulated broad high-level goals that are not always commensurate with the resources of the projects. The intention appears to be that each component of NASAâs education program should contribute in working toward these broad goals, as part of a general national effort. In practice, the manner in which this effort has been undertaken in recent years is to concentrate a good fraction of the programâs resources on a limited number of schoolsââthose selected as NASA Explorer Schools. For example, the AESP providers, originally intended as a dissemination network to serve a broad audience, have been substantially realigned to provide needed human resources to the NES project. Similarly, the Digital Learning Network (DLN) has been chiefly used as a part of NES. However, even with the contributions from other projects, NES does not bring sufficient resources, of either financial or human capital, to achieve the type of schoolwide improvement in science and mathematics learning envisaged by the projectâs goals. The amount budgeted for K-12 education activities, and the human resources in the NASA education Â projects, are simply not sufficient for NASA, by itself, to undertake a nationally signifi- cant initiative focused on systemic improvement in STEM education, nor for engaging in whole-school curriculum reform activities. With regard to information and communications technology, NASA could make better use of modern information and communications tech- nology to provide broader access to its educational resources and to make efforts to do so more cost effective. The AESP and DLN projects in par- ticular seem well suited to capitalizing on modern information and com- munications technology. Similarly, the committee recommends that SEMAA evaluate whether the aerospace education laboratories represent current technology and are cost efficient for achieving that projectâs goals. A sizable proportion of funding for K-12 STEM education projects in NASA originates in the mission directorates, particularly the Science M Â ission Directorate (SMD), which had an allocation of about $25 million
130 NASAâS ELEMENTARY AND SECONDARY EDUCATION PROGRAM in 2006. The amount spent on K-12 education programs in SMD is roughly equivalent to total for all projects in the Office of Education. Historically, 1â2 percent of the budgets for science and exploration missions have been allocated to education and public outreach (including undergraduate, graduate, and postdoctoral education, as well as informal education and public information efforts). The recommended amount has been reduced to 0.5 percent, which nonetheless still creates substantial funding for SMD projects because a single mission budget can be as much as hundreds of millions of dollars. The committee was not charged with reviewing SMDâs educational activities in detail. Finally, NASA does not appear to have budgeted sufficient funds for a thorough evaluation of projects. However, the committee was unable to gather systematic information regarding how much has been spent on evaluation because funds for evaluation are not separated in program or project budgets. Nor was the agency able to provide the committee with cost information for the external evaluations conducted in previous years. Question 4: What is the extent and the effectiveness of coordination and collaboration between NASA and other federal agencies that sponâ sor science, technology, and mathematics education activities? NASA has participated in federally coordinated activities, such as the Federal Coordinating Council for Science, Engineering and Technology (FCCSET) and the Academic Competitiveness Council (ACC) and has shared information about the agencyâs education programs and their impact with these cross-agency groups. NASA has also coordinated with other federal agencies, such as the National Science Foundation, the Department of State, and the National Oceanic and Atmospheric Administration (NOAA) on a small number of education initiatives. However, NASA does not system- atically coordinate its activities with other federal agencies or interact with other federal agencies to draw on expertise related to the design of projects. This lack of coordination and collaboration is not unique to NASA. Indeed, the disconnected nature of STEM education activities between federal agen- cies engaged in such work was identified by the FCCSET panel convened in the mid-1990s, and echoed in the recent report by the ACC. There have been a limited number of cross-agency projects (generally those based in a specific science mission), in which NASA has demonstrated good collaboration with other agencies. GLOBE is one such example, where NASA, NSF, and the Department of State worked together to estab- lish program goals and contracted with educational experts to develop active earth science learning opportunities related to the earth-observing satellite program. In addition, NASA helped fund the national K-12 stan- dards in both science (the National Science Education Standards, National Research Council, 1995) and technology (International Technology Educa-
CONCLUSIONS AND RECOMMENDATIONS 131 tion Association, 2000), in partnership with other federal agencies. When a mission project has cross-agency support (such as the Gamma Ray Large Area Space Telescope, supported by NASA and the Department of Energy), the project-related education and public outreach work has had support from both agencies for a coordinated project. The committee concludes that collaboration between NASA and other agencies on education is most effective when it is driven by shared interests in the science and technology that are the focus of the work. The committee suggests that consideration be given to developing a mechanism for federal science agencies to exchange knowledge about suc- cessful K-12 STEM education efforts. However, although some coordina- tion at the federal level could be valuable, especially in regard to the most effective use of resources, at the project level coordination with state and local education agencies and the relevant national organizations can be equally important. It does not appear that the expertise of such groups is being effectively used either to plan or to implement NASA education programs and projects. CONCLUDING NOTE NASA makes significant contributions to K-12 STEM education by providing access to its expertise in science, engineering, technology, and space exploration. It is uniquely positioned to inspire and engage students in STEM subjects and to expose teachers and students to the nature of sci- ence and engineering through exposure to the agencyâs missions. The com- mittee respects NASAâs intentions and applauds many aspects of existing projects. However, as our review and evaluation show, the current K-12 STEM education program does not fully take advantage of NASAâs unique and valuable educational resources. Steps need to be taken to give the K-12 STEM program and its constituent projects greater impact through sustained partnerships, more effective use of technology, and a culture of ongoing program improvement that includes both internal formative evalu- ation and periodic external evaluation. The committeeâs recommendations outline more specifically the steps the agency can take to improve its K-12 STEM education projects. The K-12 STEM education program in the headquarters Office of Edu- cation is to be commended for its efforts to inspire and engage students in science and engineering and to position its projects so that they can best serve students from underrepresented groups. The Science Mission Directorate programs are to be commended for their close integration with the science missions of NASA and for their use of partnerships to bring educational expertise into their work. A balance of both types of work should be contin- ued, and each should learn from the best practices of others both inside and outside the agency.