Conclusions and Recommendations
This chapter presents the committee’s conclusions and recommendations on five aspects of education programs at the National Oceanic and Atmospheric Administration (NOAA): (1) the agency’s role in education, (2) its education goals and outcomes, (3) the composition and management of its education portfolio, (4) its education evaluation practices, and (5) the impact of its education efforts. The committee’s conclusions and recommendations are based on the materials and testimony outlined in Chapter 1, the data and analysis presented in Chapters 2-5, and informed by the scientific, engineering, educational, and evaluation expertise of its members.
NOAA’S ROLE IN EDUCATION
The national need to educate the public about the ocean, coastal resources, atmosphere, and climate and to support workforce development in related fields is well established. The federal government has an important role in addressing these needs as part of the national effort to improve science, technology, engineering, and mathematics (STEM) and environmental education. Although NOAA’s focus is unique, many other federal agencies also address the country’s education and workforce needs related to the ocean, coastal resources, atmosphere, and climate. It is challenging to coordinate these activities in a cohesive way, making the best use of the assets of each agency as well as the infrastructure and capabilities outside the federal government.
NOAA’s specific role in education is influenced by multiple factors. As a federal agency, it is responsible for the conservation and management of coastal and marine resources to meet the nation’s economic, social, and environmental needs. It also promotes understanding and prediction of changes in the Earth’s environment. NOAA’s role in education is tightly connected to its mission, and some education programs are defined by various congressional mandates. NOAA must fulfill these responsibilities in the context of a national effort to improve learning and understanding of STEM implemented at state and local levels in both formal and informal settings for learning, and at the same time carry out effective environmental education.
Conclusion I.1: The America COMPETES Act assigns NOAA responsibility for advancing and coordinating mission-related STEM education and stewardship efforts and for participating in interagency education efforts (Chapter 1: Overview and Introduction).
NOAA’s role in education has been recognized for more than 30 years, as evidenced by the legislative mandates to engage in education activities given to its individual operating branches and programs. This role was underscored and strengthened by the America COMPETES Act, which gives the agency a broad mandate to engage in and coordinate education initiatives in oceanic, Great Lakes, climate, atmospheric, and environmental sciences, as well as other fields related to its mission. NOAA has long conducted many types of education efforts, including grant making, creating partnerships with minority-serving colleges and universities, local hands-on learning experiences, and developing national science literacy frameworks. While its overall mission statement does not include education, dedication to education is underscored by the agency’s vision to develop a society that is informed on issues related to its mission, and education objectives are implicit in each of the agency’s four broad goals.
Conclusion I.2: NOAA is unique among federal agencies in its focus on stewardship and on oceanic, coastal, Great Lakes, atmospheric, and climate sciences. However, its mission overlaps with and complements the missions of other federal agencies (Chapter 2: NOAA’s Role in the Education Landscape).
Several federal agencies share overlapping responsibilities for national programs in science, engineering, exploration, and stewardship related to oceans, the atmosphere, climate, and the environment. These agencies also develop and support programs in STEM education and environmental education that are similar to NOAA education programs. In addition, there is a significant infrastructure and human capacity outside the federal government for conducting scientific research, developing technology, and
supporting education in areas related to NOAA’s mission. Thus, NOAA needs to establish where its strengths lie, set priorities for how it engages in different education initiatives, and coordinate its efforts with those of other federal agencies.
Conclusion I.3: NOAA has assumed a coordinating role on interagency groups convened to support coherence and collaboration across federal agencies involved in science education related to its mission (Chapter 1: Overview and Introduction; Chapter 4: Overview and Critique of NOAA’s Education Programs).
In the past decade, interagency coordination groups have been established to support coherence and collaboration through efforts, such as those to develop literacy frameworks that lay out the key ideas in oceanic, atmospheric, climate, and environmental sciences. NOAA is an active participant in these groups and has assumed leadership in some of them. For example, NOAA cochairs both the U.S. Climate Change Science Program’s Education Interagency Working Group and the Ocean Research Resources Advisory Panel’s Interagency Working Group on Ocean Education.
Conclusion I.4: NOAA can fulfill its role in education by leveraging its modern and groundbreaking technologies and discoveries; research equipment; data; scientists, engineers, researchers, and other technical staff; stewardship and management of natural resources; specialized education expertise; partnerships; and connections to local, regional, national, and international stakeholders and natural resource managers (Chapter 2: NOAA’s Role in the Education Landscape).
NOAA is one of the key federal agencies engaged in stewardship of the coasts and oceans. The resources it manages provide vast and important education opportunities, and management of these environments provides the agency with connections to the surrounding communities and organizations concerned with environmental issues. NOAA brings to education the cutting-edge science, engineering, and technology (e.g., ocean research vehicles, satellite systems, data systems, data collecting buoys, etc.) that it produces through intramural activities or supports through funding of external scientists and engineers. Technology, scientific resources, and data systems provide opportunities for students and citizens to see scientists, engineers, and researchers in action; gain a scientific understanding of the natural world; and participate in research, data collection, and analysis.
Conclusion I.5: NOAA’s role in education is shaped by the distributed nature of its education efforts across the line offices and the Office of Education, small education staff, and small education budget (Chapter 2: NOAA’s Role in the Education Landscape).
Education programs are run by five of the agency’s six line offices (the Office of Program Planning and Integration does not run education programs) and the Office of Education. The line offices (some of which have individual education mandates) and the Office of Education can act independently, sometimes even in competition with each other. Usually, an individual or a small team implements the majority of education programs at the local level. Most local NOAA staff can dedicate only a portion of their time to education, because many of them also have communication, extension, research, or exploration responsibilities. NOAA’s education budget is also relatively small in comparison to other federal agencies engaged in STEM education, such as the U.S. Department of Education, the National Science Foundation, the National Aeronautics and Space Administration, and the U.S. Department of Energy. These limited resources make it next to impossible for NOAA to accomplish its ambitious education goals without strategic partnerships.
Conclusion I.6: The scope of both the stewardship and science topics related to NOAA’s mission is global in nature and involves partnerships at the local, state, federal, and international levels. NOAA can make more effective use of the agency’s global and international programs in its domestic education activities and foster education and stewardship through collaborative work with international partners (Chapter 2: NOAA’s Role in the Education Landscape).
NOAA’s mission includes topics that are inherently global, such as climate change and ocean observing, which makes it essential to establish and maintain strategic partnerships to accomplish its ambitious education goals. Clear education goals, planning, and strategic use of NOAA resources are critical aspects of effective partnerships.
Conclusion I.7: NOAA can play a supporting role in state and local education. Its education efforts are more likely to be productive if they align with national and local education needs, because education activities and products that do not consider the needs of the potential audiences are less likely to be used (Chapter 2: NOAA’s Role in the Education Landscape).
Education programs need to focus on productive partnerships to support local and state education systems, while promoting NOAA’s education and stewardship mission. As part of developing an implementation plan, NOAA will therefore need to make better use of assets and programs that already exist, such as academic research and education programs, sanctuaries and other protected areas managed by NOAA as well as other federal and state agencies, and even entertainment media, such as movies, radio, and computer and video games. In addition, NOAA can provide international, national, and state agencies with resources that clearly provide
insight into the essential concepts and important principles of science and environmental education that should be woven into standards and curriculum expectations. NOAA’s contribution to formal and informal education can be significant if the agency shares new research and makes this knowledge available and understood by the public.
Based on our conclusions regarding NOAA’s role in education, the committee makes the following recommendations.
Recommendation I.1: NOAA should fulfill its role in education through the use of:
agency and external expertise in science, engineering, technology, and education; cutting-edge scientific research and exploration activities; internationally collected data sets; and advances in technology and engineering;
place-based assets that directly connect local issues to national and global science and stewardship issues: marine sanctuaries, estuarine research reserves, fisheries activities, and other natural resources protected and managed by federal, state, and local entities;
partnerships with local and state education infrastructure, academic institutions, government agencies, business and industry, and private-sector and nonprofit organizations; and
the agency’s global science and international partnerships.
Recommendation I.2: In order to adequately address the mismatch between its available resources and its ambitious education agenda, NOAA should better align and deploy its resources. This may require the termination of certain activities and programs that, based on appropriate evaluation, do not directly and effectively contribute to its education and stewardship goals.
Recommendation I.3: Within the constraints of NOAA’s mandates in education, the agency should continually evaluate where it leads, collaborates, follows, or declines to participate in partnerships with others. These decisions should be guided by consideration of the agency’s role, assets, resources, and priorities in education and the strengths and missions of other agencies, institutions, and organizations engaged in education.
II. EDUCATION GOALS AND OUTCOMES
Two goals are outlined in the NOAA education strategic plan for 2009-2029: (1) to advance the environmental literacy of the nation and (2) to promote a diverse workforce in oceanic, coastal, Great Lakes, weather, and climate sciences. At this time, NOAA is developing a strategic implementa-
tion plan to specify how it will accomplish these goals. The strategic plan lists six outcomes under the environmental literacy goal:
The use of research on effective environmental and science education.
That educators understand and use environmental literacy principles.
That educators, students, and the public engage in inquiry-based learning.
That opportunities are available for lifelong engagement in science and environmental education.
That agencies collaborate on education activities.
That there is coordination across NOAA in education, extension, outreach, training, and communication.
Three outcomes are listed under the promoting a diverse workforce goal:
A diverse and qualified pool of applicants pursue opportunities for career development in NOAA mission-critical disciplines.
NOAA employees support activities to disseminate NOAA scientific and stewardship work.
A diverse pool of individuals who enter career paths at NOAA and in other related organizations.
The plan describes the issues and topics that need to be addressed to reach its goals and provides strategies to accomplish the outcomes under each goal. The plan also emphasizes the importance of partnerships to reach its education goals and outcomes.
Conclusion II.1: The education strategic plan has a number of strengths:
Appropriate goals of supporting environmental literacy and workforce development;
A commitment to developing education programs informed by evidence about effective practices;
A call to contribute to the body of knowledge on effective education practices in fields related to NOAA’s mission;
A call to develop partnerships with appropriate agencies, institutions, and organizations;
An emphasis on including more members of historically underrepresented groups in fields critical to NOAA’s mission; and
An emphasis on the use of ocean, coastal, and other place-based resources as unique and valuable assets for learning (Chapter 3: The Education Portfolio and Effective Practices).
The current education strategic plan is a step forward from the previous one. It provides more detailed description of outcomes and outlines strategies to bring about those outcomes. It also describes the importance of effective education practices, as well as issues and topics to be addressed in reaching the goals. Given the limited education resources of the agency, it is critical that the plan stresses the need to create partnerships and connections to local, regional, national, and international stakeholders and natural resource managers to achieve its education goals.
Conclusion II.2: The current education strategic plan also has weaknesses that need to be addressed:
Some of the nine outcomes are strategies for accomplishing the two overarching goals rather than measurable changes that would be expected as the result of reaching either goal.
There is no environmental literacy outcome related to addressing the needs of underserved populations or addressing issues of equity.
There is no environmental literacy outcome related to stewardship, yet many of the local education activities as implemented have stewardship goals.
There is no clear articulation of the specific workforce needs, nor is there clear articulation or a plan to provide a robust assessment of the workforce needs in fields critical to NOAA’s mission.
There is no mechanism for local education staff from different education programs to share effective practices and lessons learned.
There is no specific guidance or strategy for involving or drawing on NOAA expertise in engineering, science, and other fields.
There is no specific guidance to NOAA employees engaged in education activities regarding fostering of external partners to connect to the national STEM infrastructure and human capacity (Chapter 3: The Education Portfolio and Effective Practices).
These issues make it difficult to develop education programs that align the agency’s education goals and objectives. Alignment would ensure that NOAA can capitalize on its assets. Alignment of goals and objectives would also support the agency’s ability to conduct evaluations that produce data to show whether the agency is meeting its educational goals. These data are fundamental to decision making regarding the development of new programming and the management of existing programming.
To address these issues, NOAA needs to address three problems with the outcomes in the strategic education plan. First, outcomes are typically thought of as measurable changes or absolute levels of performance that
can be expected as a result of the efforts made to reach a goal (i.e., what does it look like when environmental literacy is approached or reached and what constitutes a significant step along the path to acceptable diversity in the workforce). However, only three of the six environmental literacy outcomes and two of the three workforce development outcomes specify expectations; the other outcomes describe strategies or processes that might contribute to reaching the goals.
Second, while diversity is a focus of the workforce goal, there is no mention of diversity or broadening participation in the environmental literacy goal. There is a need to include an outcome focused on reaching out to underserved populations in the environmental literacy goal to address the national need to expand understanding of and interest in the science and stewardship issues related to NOAA’s mission among K-12 students, adults, and the public.
Third, there is no environmental literacy outcome related to stewardship, despite the fact that stewardship is major element of environmental literacy. An outcome focused on stewardship is needed because it is the primary goal of many local education activities, and it would encourage the measurement of attitude and behavior change in program evaluations.
It is unclear how NOAA can accomplish its goal of supporting the creation of a “world-class” workforce without a clear understanding of its own and the nation’s workforce needs in areas that are critical to its mission. Workforce needs are difficult to predict in general, and particularly difficult to do so in interdisciplinary areas, such as those critical to NOAA’s mission. The agency needs a clearer estimate of workforce needs to guide the scope and direction of its initiatives.
Scientists and engineers have the expertise to introduce teachers and students to the processes of science and engineering, as well as to the cutting-edge research related to science and engineering activities that are connected to NOAA’s mission. However, they need to work in concert with professionals who have specific expertise in learning, education, or behavior change and modification. The implementation of the education strategic plan or education implementation plan needs to provide guidance and require that these connections occur. Similarly, the implementation plan needs to provide guidance on fostering internal and external partnerships. Both are highlighted as important aspects of NOAA’s education strategic plan, but there is very little detail regarding how either will be accomplished.
Conclusion II.3: The use of the term “NOAA science” in the strategic plan is confusing. It is unclear whether this term is meant to refer to the science conducted by NOAA scientists, the research or the results of research funded by NOAA, or any science conducted on topics related to NOAA’s mission (Chapter 3: The Education Portfolio and Effective Practices).
It may be that the term “NOAA science” is used as a convenient shorthand term in describing broadly NOAA-related science. However, we are particularly concerned about narrow interpretations of the term as applied to NOAA’s broader science education, stewardship, and workforce development goals. A narrow interpretation may lead to an exclusive focus on the research funded by NOAA, thereby constraining the agency’s activities. Education activities aimed at environmental literacy for the public should draw on a broad body of scientific knowledge to avoid hampering public environmental literacy, which is a likely result of using a narrower body of knowledge. The committee is also concerned that the term may encourage a blurred line between activities focused on education and activities that are more appropriately defined as public relations or agency branding. This is not to say that there is evidence that the line is blurred in current activities, but rather that the use of the term “NOAA science” may lead to a blurring of this line in the future.
To address these concerns, the committee makes four recommendations.
Recommendation II.1: NOAA education programs should formally address broadening participation of underrepresented groups as an important outcome through all phases, from the initial stages of planning through implementation and evaluation. The environmental literacy goal, in particular, should include outcomes related to reaching out to underserved and underrepresented communities.
Recommendation II.2: To reach NOAA’s environmental literacy goal, the Education Council should develop its implementation plan and future revisions of the education strategic plan to:
clarify how it will capitalize on scientific findings, engineering advances, and stewardship activities that relate broad national priorities to local concerns to engage individuals of all ages in education;
articulate how NOAA education programs will draw on the scientific, engineering, research, and other expertise accessible within the agency as well as in the broader community;
address the mismatch between the lack of an outcome related to stewardship and the focus on stewardship outcomes in local programs;
consistently define outcomes as measurable concepts that allow an assessment of whether a goal is being reached, to clearly distinguish outcomes on audiences (impact) from outputs of activities; and
provide more opportunities for local and regional education staff from all education programs to share effective practices and lessons learned.
Recommendation II.3: To achieve the workforce development goal, the education strategic plan, the education implementation plan, or both should call for periodic assessment of the current and anticipated needs in fields critical to NOAA’s mission to guide investment in appropriate workforce development activities.
Recommendation II.4: NOAA education programs should draw from current and relevant scientific and engineering advances regardless of what agency, institution, or organization they are originated or funded by.
COMPOSITION AND MANAGEMENT OF THE EDUCATION PORTFOLIO
NOAA supports a wide range of education programs for varied audiences that include K-12, postsecondary, graduate, and informal education activities with local, regional, national, and international scope. NOAA has developed professional development programs, classroom materials, curricula, museum exhibits, place-based learning experiences, literacy documents, and other products. The audiences for these programs include teachers, students, scientists, and the public. A coherent, coordinated education portfolio is needed to achieve goals effectively and efficiently, to share successful strategies for engaging and teaching different audiences, to pool resources to support synergistic activities, to develop cross-discipline activities, and to sustain consistent education strategies.
Conclusion III.1: Management of a federal education portfolio is complicated, and NOAA has characteristics that make this particularly challenging:
The fact that education programs are distributed across the agency.
The overall management structure was not designed with education responsibilities in mind.
A broad, overarching mission that includes a number of scientific areas, environmental stewardship, and commerce and transportation issues.
Responsibilities that overlap and need to be coordinated with other federal agencies.
The fact that development and implementation of nearly all of its education programs took place prior to the creation of the coordinating structures (the Education Council and the Education Office).
The fact that some NOAA education programs have individual legislative mandates that guide their education activities (Chapter 2: NOAA’s Role in the Education Landscape; Chapter 3: The Education Portfolio and Effective Practices).
Creating and managing a strategically balanced education portfolio is no small task. Education programs are enacted by five of the six line offices in NOAA (the Office of Program Planning and Integration does not enact education programs) and the Office of Education. In addition, an unquantified amount of education support is provided through scientific research grants and contracts. Individual offices have separate mandates and often have local components with local control. Education programs are managed differently across these offices as a result of the resources available, separate missions, and different mandates. The differences in management structures, missions, and education mandates are obstacles to developing a cohesive and coordinated education portfolio. The Education Council needs to encourage the coordination of the agency’s education programs to establish and monitor the agency’s education portfolio.
Conclusion III.2: While relatively new, the Education Council, led by the Office of Education, serves an essential, high-level internal coordinating function. The Education Council led the development of the education strategic plan and is developing the collaborative working relationships necessary for implementation. However, the Education Council does not have budgetary or institutional control over the education efforts of NOAA line and program offices; this limits its effectiveness in carrying out NOAA’s education mandate (Chapter 3: The Education Portfolio and Effective Practices).
The Education Council is the primary means that NOAA has developed to address the challenges in managing its education portfolio. In a short period of time, the Education Council has shown promise in bringing coherence and coordination to the agency’s education portfolio by developing the strategic plan and encouraging a common evaluation framework. The committee is concerned that the Education Council may not have the needed power to enforce difficult decisions regarding the priorities, focus, and components in the portfolio in the near future—specifically, if the decisions require some line offices to end or restructure some of their education programs.
Providing more power to the Education Council does not come without concerns. It is not clear whether top-down decisions by the council will be perceived as supportive of the local education offices, which currently rely heavily on the enthusiasm, creativity, and good will of the local staff and their partners. Management structures or decisions that negatively impact the enthusiasm, creativity, and good will of local staff and partners could have far-reaching impacts on the success of the programs. Thus, any decisions about the power of the council must be weighed against the possibility of creating disharmony or a lack of enthusiasm through its decisions and decision-making process. In addition, the Education Council’s decision-
making process would be well served by honoring input from local staff of each of the line offices.
Conclusion III.3: The NOAA education portfolio has developed in the absence of an overarching strategic direction and without a system to monitor or catalogue activities. It is therefore difficult to assess its composition, balance, and impact. In addition, there is no clear institutional mechanism to streamline or shift the portfolio (Chapter 4: Overview and Critique of NOAA’s Education Programs).
Although we received a great deal of information, the committee gained only a partial understanding of the education portfolio because the agency had no systematic process for collecting and cataloguing information about activities supported and carried out in the regional and local offices. It was therefore difficult to assess many aspects of the portfolio’s balance. During our review, the agency began to develop such a process for collecting this information. Clearly, NOAA now understands that an adequate system to monitor and catalogue its education portfolio is needed to make informed decisions about portfolio balance.
The system to monitor and catalogue current and future portfolio balance should include evaluation criteria and evaluation procedures, continuing review of the entire portfolio to enable reprioritization, and new strategic directions. It should support the development of a dynamic portfolio in which programs may be discontinued if no longer needed or effective, to make room for new programs that may more effectively achieve goals and outcomes, or that allow NOAA to address new strategic goals and outcomes. Developing figures or tables, such as those presented in Chapter 4 (see Figure 4.2 and Table 4.2), could be useful in understanding and tracking portfolio balance.
Developing a coordinated and cohesive education portfolio is further complicated by the lack of a mechanism to bring the local education staff from different programs together to share effective practices across education programs. The strategic plan mentions the need for internal coordination to support education activities; however, the coordination being discussed is at the Education Council level. Promoting connections among local education staff can be just as valuable in creating internal coordination.
In addressing these concerns, NOAA will have to grapple with the level of centralized control that is optimal and information exchange systems that connect people who face similar issues. The agency must weigh whether it would be better off with a single, coordinated education program, or a set of loosely coupled programs run by the various offices, or whether a middle ground can be established. In making decisions about how to manage and monitor its education programs, NOAA should be striving to create an
education portfolio that continues to address the needs of the communities it serves in an efficient and coordinated manner. In addition, the agency should have access to information needed to understand the composition of its programs, to make strategic decisions about the direction of its programs, and to evaluate the impact of its programs.
Conclusion III.4: To date, NOAA’s education programs have been more focused on ocean or coastal concepts and issues than on climate and atmospheric concepts and issues. There are emerging efforts to bring greater attention to climate and the atmosphere across the agency’s education activities (Chapter 4: Overview and Critique of NOAA’s Education Programs).
Although we were unable to obtain information about all of NOAA’s education programs, it is clear that the majority of activities focus on issues related to ocean literacy. The imbalance with respect to atmospheric and climate literacy is an issue acknowledged by NOAA education staff. The Climate Office, along with the Education Council, is working to bring greater attention to climate and atmospheric issues.
With regard to the composition and management of NOAA’s education portfolio, the committee makes the following recommendation.
Recommendation III.1: NOAA should develop and implement a system to monitor and catalogue its education portfolio and guide decisions regarding what programs should be developed, continued, modified, or ended. In balancing the portfolio, the Education Council should
increase attention to climate and atmospheric science education programs to complement the current focus on ocean science. These programs should emphasize the strong connections and interactions among the ocean, the atmosphere, the land, and human and nonhuman species;
provide purposeful attention to both STEM learning and stewardship goals so as to enable synergies; and
make decisions based on national education needs, the education priorities of the agency, and a clear picture of its education portfolio.
EDUCATION EVALUATION PRACTICES
The challenges of carrying out appropriate evaluations of education projects are large. Most federal science agencies are struggling to meet these challenges, and NOAA is giving increasing attention to evaluating its education projects. Although evaluation is primarily the responsibility of each individual education and outreach program, the Education Council
provides leadership and guidance for education evaluation activities across the agency. The strategic education plan 2009-2029 highlights the need for more comprehensive evaluation.
Conclusion IV.1: NOAA has conducted evaluations of a small proportion of its educational activities, and these evaluations are limited in scope and quality. The evaluations tend to focus on intermediate rather than long-term outcomes and on participant opinion, feedback, beliefs, and knowledge. They usually do not address outcomes related to attitudes or behavior, and they generally lack control or comparison groups. Summative evaluations have been carried out on a very small proportion of education activities, and there has been little consideration of evaluation that would allow NOAA to recalibrate the education portfolio to effectively meet the agency’s educational goals (Chapter 4: Overview and Critique of NOAA’s Education Programs; Chapter 5: Current Evaluation Framework and Existing Evaluation Efforts).
While NOAA education programs have conducted many evaluations, they have resulted in very few data that illustrate the impact of the programs. For example, many evaluations include only self-reported information about participant enjoyment, satisfaction, or perceived impact. Also, even when appropriate there were rarely control or comparison groups in the evaluations we reviewed. Evaluation has historically not been used by the agency to collect data to assess whether the overarching goals and outcomes in the education strategic plan are being accomplished.
One major issue is that evaluation, particularly when focusing on impact at the program level, can be expensive. A rule of thumb is that at least 5 percent of a projects budget should be devoted to summative evaluation. Formative evaluation should be part of program design, and its cost is part of the program. However, the education efforts in NOAA are operating on a shoestring budget, and taking 5 percent of the budget away from a project’s operating costs may negatively impact the project’s implementation and be insufficient to conduct a meaningful evaluation.
Given its limited overall funds, it is critical that NOAA develop a plan for wisely allocating funds for evaluation (so as not to compromise programming). To achieve the greatest return on its limited resources, evaluations of individual projects can be scheduled on a cyclical basis, with high priority given to projects intended to have the greatest impact on environmental literacy and workforce needs and to projects that face important questions regarding activities, participants, staffing, funding, or organization. In addition, partnerships can be developed to minimize the financial burden of the evaluation process on NOAA. Program effectiveness should be determined by conducting formative evaluations as part of the project or program design process.
Conclusion IV.2: The Education Council is increasing its emphasis on evaluation and moving toward comprehensive program evaluation by adopting the Bennett Targeting Outcomes of Programs (TOP) model. This approach may help implement a more strategic and coherent approach to education evaluation efforts across the agency, serving as an important tool to align programs with NOAA’s education goals. However, as with most evaluation models, the TOP model does not include specific guidance regarding the implementation of evaluations or how to design high-quality evaluations (Chapter 5: Current Evaluation Framework and Existing Evaluation Efforts).
The adoption of a uniform model, such as the Bennett TOP model, as a framework to guide evaluation strategies and practices is a useful step that illustrates the positive influence of the Education Council. NOAA education programs need to continue to refine their evaluation expertise, contract with appropriate evaluators to design and implement evaluations, and use evaluation results to improve education activities.
Conclusion IV.3: The collection of data for project monitoring and evaluation purposes is uneven across the education portfolio. A set of guidelines to systematically collect consistent data for these purposes is needed (Chapter 4: Overview and Critique of NOAA’s Education Programs; Chapter 5: Current Evaluation Framework and Existing Evaluation Efforts).
Data are needed for several purposes, including project monitoring, fiscal due diligence, and program evaluation. NOAA needs a systematic way of collecting data for each purpose to ensure that data are comparable across programs and initiatives.
Education evaluation practices were being augmented during our review of NOAA’s education program. To ensure that these practices continue to develop in a positive direction, the committee makes the following recommendations.
Recommendation IV.1: The Education Council should continue to improve the evaluation expertise of its education program managers, contract with external evaluators for summative evaluation, and require the incorporation of the most appropriate and rigorous evaluation strategies during program development to guide design, continual improvement, and delivery of its education programs.
Recommendation IV.2: The Education Council should increase the emphasis on high-quality evaluations. Summative evaluations should focus on the program outcomes related to learning and stewardship, not only satisfaction with education experiences, and should use the most appropriate and rigorous evaluation designs.
Recommendation IV.3: The Education Council should consider developing a number of approaches to inform strategic portfolio management and how evaluation findings can be used to inform decisions about portfolio balance.
Recommendation IV.4: Education programs should evaluate internal collaboration among line offices and between education and operational and scientific staff, as well as the quality of external partnerships with other agencies, institutions, organizations, and the broader STEM communities.
EVIDENCE OF IMPACT
Many of the education programs have not been in existence long enough to allow any definitive impact assessments. Furthermore, because of the problems with the outcomes defined in the strategic plan and the lack of rigorous evaluation programs, the Education Council has not yet developed the capability to analyze the impact of the larger portfolio of education programs. In general, there is little comparative, long-term, or causal data about individual programs or across programs to enable such analyses.
Conclusion V.1: Although NOAA has created a large number of education initiatives with its limited education budget, there is evidence of impact for only a small proportion of them. The majority of initiatives have collected information only on their scope, reach, and participant satisfaction, which are not sufficient to judge impact. On the basis of the available evidence, all that can be said about that the impact of such programs is that they are positively perceived by the participants and, as yet, serve a small portion of the population (Chapter 4: Overview and Critique of NOAA’s Education Programs).
There is limited information regarding the impact of NOAA’s education programs—a situation that is not uncommon among federal agencies. However, NOAA’s ability to make good use of its education resources and assets to engage in a substantial number of education activities is impressive. Most partnerships with educational organizations, other agencies, and institutions with complementary STEM expertise have enhanced the reach and impact of NOAA’s education efforts. These partnerships have often yielded additional expertise, educational tools, mechanisms for dissemination, and matching funds.
Conclusion V.2: Although the current strategic education plan calls for the use of research-based education practices, current education activities do not consistently follow what is understood about effective education practices in the United States and abroad (Chapter 3: The Education Port-
folio and Effective Practices; Chapter 4: Overview and Critique of NOAA’s Education Programs).
There is a growing body of literature regarding effective practices in formal and informal science education, behavior change, reaching under-served populations, and workforce preparation. This literature can be used to support the development of science education programs that are likely to be successful.
To encourage the development and implementation of effective programs to address the goals outlined in NOAA’s education strategic plan, the committee makes the following recommendation.
Recommendation V.1: NOAA education staff should draw on evidence from education research, evaluations of NOAA programs, and external education expertise to identify and implement effective practices for supporting education activities.
People are NOAA’s most valuable assets. The education staff is dedicated and passionate about addressing areas related to its mission. They have developed diverse education activities for a wide range of audiences and regions. While many of the conclusions of this committee address issues with NOAA’s education efforts, the agency and its education staff are to be commended for their historic commitment to education, which precedes the agencywide congressional mandate on education. The agency’s current education strategic plan is a significant improvement over the previous one. We hope that our recommendations help NOAA continue to improve its education efforts.