Click for next page ( 2


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 1
Summary T he National Sea Grant College Program (NSGCP or Sea Grant) was created nearly 40 years ago and has matured into a state-federal partnership with a distinctive role and management structure. Sea Grant is a nationwide network (administered through the National Oce- anic and Atmospheric Administration [NOAA]) of 301 individual Sea Grant programs2 based at some of the nation's top universities. The NSGCP engages this network in conducting scientific research, educa- tion, training, and extension projects designed to increase assessment, development, utilization, and conservation of coastal resources by pro- viding assistance to promote responsive research and training activities and to broaden knowledge and techniques (National Sea Grant College and Program Act, 1966 [P.L. 89688]).3 The NSGCP has been a main source of funding in the United States for activities in marine policy, and thus far has been a major contributor to the issues of aquaculture, biotechnology, coastal communities and econo- mies, coastal natural hazards, ecosystems and habitats, fisheries, marine science literacy, seafood science and technology, urban coasts and inva- sive species. The program also supports students at all levels of the edu- 1Not including the 3 programs in development stages. 2For the purpose of this report, all 30 programs will be referred to as "individual Sea Grant programs." Previous Sea Grant literature has also used the term "state program" or "Sea Grant college/institute." 3See Appendix C for Sea Grant legislation. 1

OCR for page 1
2 EVALUATION OF THE SEA GRANT PROGRAM REVIEW PROCESS cational system and has supported education and training of many ma- rine and Great Lakes scientists, resource managers, and policy specialists through its three fellowship programs, including the John A. Knauss Marine Policy Fellowship, the Sea Grant/NOAA Fisheries Graduate Fel- lowship, and the Sea Grant Industry Fellowship Program. In 1993, the Under Secretary of Commerce for Oceans and Atmo- sphere requested the National Academies review and evaluate the NSGCP as part of an effort to prepare for the then pending National Sea Grant College Program Reauthorization Act of 1998 (P.L. 105-160). The resulting 1994 report, A Review of the NOAA National Sea Grant College Program, recommended several actions, including strengthening the strategic plan- ning process at the national level, clarifying the roles and responsibilities of the National Sea Grant Office (NSGO) and individual program person- nel as well as the National Sea Grant Review Panel (NSGRP), and carry- ing out systematic, periodic reviews of the individual programs (National Research Council [NRC], 1994). THE CURRENT STUDY In partial response to the 1994 report, the Director of the NSGCP (referred to as "National Director" throughout this report) requested that the National Sea Grant Review Panel establish a process for evaluating each individual program once over a four-year review cycle. These re- views are carried out through a series of site visits, each of which usually involves 4 to 7 recognized experts in marine science and policy, who focus on a uniform set of performance criteria, using a standardized set of benchmarks and indicators. This evaluation process has evolved through time, in response both to experience gained during its execution and to evolving expectations of Congress. The National Sea Grant College Pro- gram Act Amendments of 2002 (P.L. 107299) directed NOAA to contract with the National Academies to carry out a review of the evaluation process and make appropriate recommendations to improve its overall effectiveness. Statement of Task The Committee on the Evaluation of the Sea Grant Program Review Process (the Committee) was charged with assessing new procedures adopted by the NSGCP since the publication of the 1994 NRC report to determine their impacts. During this study, the Committee assessed the impact of the new procedures and evaluation process on Sea Grant as a whole. Among the areas considered were the quality of the work pro-

OCR for page 1
SUMMARY 3 duced by the program; its responsiveness to national, regional, and local needs; and the quality of its leadership, management, and reputation. Specifically, the Committee was asked to examine: (1) Effectiveness of major changes instituted in response to the recom- mendations of the 1994 NRC report with regard to individual program performance and quality. (2) Effectiveness of program review procedures with regard to accu- racy, accountability, and enhancement of individual program perfor- mance. Both the previous and current review procedures (adopted in 2003 in response to the Sea Grant Act of 2002) will be assessed as specified below: Review the effectiveness of the evaluation and rating system in deter- mining relative performance of programs with regard to manage- ment and quality of research, education, extension, and training activities; Evaluate whether there have been improvements in programs as a result of the evaluation process; Evaluate the 2003 review procedures for their ability to meaningfully segregate individual programs into five categories based on com- petitive scores; and Compare the effectiveness of the previous and 2003 review proce- dures with regard to the dual objectives of maximizing the quality of each program and of rating programs relative to each other for the purpose of determining performance-based funding. (3) Assessment of the usefulness and fairness of metrics developed to evaluate programs with different operational constraints, resources, and local priorities. Evaluate metrics for relevance and clarity; Determine whether metrics provide a quantitative measure of quality of performance; and Assess whether metrics improve consistency and objectivity of re- views from different teams evaluating a diverse portfolio of indi- vidual Sea Grant programs. The Committee was also asked to make recommendations for im- proving the overall effectiveness of the evaluation process to ensure fair- ness, consistency, and enhancement of performance.

OCR for page 1
4 EVALUATION OF THE SEA GRANT PROGRAM REVIEW PROCESS IMPACT OF CHANGES IN RESPONSE TO 1994 REPORT Following the 1994 NRC report, the NSGO instituted a number of changes in an effort to improve the overall program and the manner in which individual programs are evaluated. Although strategic planning within the NSGCP needs to be improved, the adoption of a formal strate- gic planning process at the national program level, as recommended in the 1994 report, is prima facie an improvement over earlier practice. In addition, there is a consensus among the directors of individual Sea Grant programs that the evaluation process instituted in 1998 in partial response to the 1994 report has led to improvements in their programs, despite the fact that many within this group are openly critical of some aspects of the process. Finally, several members of the Committee have first-hand, long- term experience with the Sea Grant program and it is their considered opinion that the changes instituted since 1994 have strengthened the over- all program. As with the Sea Grant directors, the opinions of even knowl- edgeable individuals cannot be taken as objective indicators; but, the una- nimity of response to this issue--particularly in light of differences of opinions on other issues--suggests that real improvements have occurred. EFFECTIVENESS OF POST-2002 EVALUATION As mentioned, the process established by the NSGO in 1998 (and modified periodically since) to evaluate program performance appears to have led to improvements in the overall program. However, several areas of concern remain. Since the reauthorization of the program in 2002, pro- gram evaluation within Sea Grant has evolved to serve two, theoretically related purposes: (1) identifying areas for improvement in individual pro- grams, and (2) rating and ranking of individual programs for the purpose of competitively awarding merit and bonus funds (as stipulated by Con- gress in the 2002 legislation, P.L.107299). These purposes are related insofar as competition for funds serves as an incentive to the individual programs to improve. The evaluation process as it has been performed since 2003, however, appears to be more appropriately structured to achieve the narrow goal of ranking programs and distributing competi- tive funds. For the overall program to improve--and, in particular, for it to become (and be seen to become) a truly national program--there is need for NSGO to strengthen its ability to facilitate and coordinate efforts of the individual programs. Perhaps the foremost concern about the Sea Grant evaluation process is the reliance by the NSGO on periodic assessments as the primary, if not only, means of evaluation and oversight. Despite the general high quality of the information they provide, the overreliance on periodic assessments

OCR for page 1
SUMMARY 5 undermines the role that the NSGO could play in continued improve- ment of the individual programs and in the administration and coordina- tion of the national program. The periodic assessments themselves rely heavily on information collected during quadrennial visits by Program Assessment Teams (PATs) overseen by the NSGRP. As the members of PATs and the NSGRP are not Sea Grant employees, the preponderance of program oversight is actually external. The Director of the National Sea Grant College Program, under supervision of the Secretary of Com- merce and in consultation with the National Sea Grant Review Panel and the individual Sea Grant programs, should strengthen the ability of the National Sea Grant Office to carry out meaningful, ongoing in- ternal assessment in order to complement periodic, external assessment currently taking place. STRATEGIC PLANNING The importance of strategic planning in program development, imple- mentation, and evaluation was emphasized in the 1994 NRC report. Spe- cifically, the 1994 report recommended that "State Sea Grant Directors [individual Sea Grant Program directors] and the Director of the NSGCP [National Director] must cooperate to develop a single strategic plan ar- ticulating a shared vision and strategies which must be fully integrated into, and reflective of, NOAA's strategic plan." Although strategic plan- ning at the national level, as carried out by the NSGO, meets this recom- mendation, the same cannot be said at the state level. More effort is needed to ensure that all of the individual Sea Grant programs develop strategic plans that dovetail with the national plan, while addressing local and state challenges they may be uniquely equipped to address. Since 1994, a number of high-level reviews, such as the recent report from the U.S. Commission on Ocean Policy, have identified the most pressing problems in marine policy. These reviews have repeatedly em- phasized the need to identify and address issues at the proper regional scale. The state and federal partnership NSGCP represents would seem to be well suited to addressing these intermediate-scale problems, as federal coordination and support for local and state efforts is generally an impor- tant component to effective regional action. To ensure that strategic plan- ning reflects a shared vision, representatives of the NSGO should partici- pate in the local strategic planning process and the strategic plan should serve as the basis upon which the individual Sea Grant program is evalu- ated. Steps should be taken by the Director of the National Sea Grant College Program, under supervision of the Secretary of Commerce and in consultation with the National Sea Grant Review Panel and the indi- vidual Sea Grant programs, to strengthen strategic planning at both the

OCR for page 1
6 EVALUATION OF THE SEA GRANT PROGRAM REVIEW PROCESS national and individual program level. The strategic plans of the indi- vidual programs and the national program should represent a coordi- nated and collective effort to serve local, regional, and national needs. As discussed in Chapter 4, actions by the NSGO should include: developing and implementing a process to assist individual pro- grams in strategic planning; and creating a separate process for evaluating and approving appro- priately ambitious strategic plans for the individual programs. PERFORMANCE CRITERIA Performance criteria are a combination of quantitative and qualita- tive measures used to assess the selected activity or program, the out- comes of that program, and, in some instances, the system that program is intended to influence. In the case of assessing the effectiveness and impacts of individual Sea Grant programs, this involves assigning bench- marks to describe the expected level of performance in a particular area (such as program organization and management) and indicators to help assess the outcomes or impacts of the individual program in that area. As discussed earlier, strategic planning is a critical basis for implementa- tion, review, and evaluation of institutional programs. The Director of the National Sea Grant College Program, under supervision of the Sec- retary of Commerce and in consultation with National Sea Grant Re- view Panel and the individual Sea Grant programs, should modify the benchmarks and indicators, as needed, to ensure that the performance of each program is measured against the objectives outlined in the separately approved, program specific strategic plan called for in the previous recommendation. In addition, the current Sea Grant evaluation criteria do not recognize the importance individual programs should play in building cooperative efforts to address regional and even national scale problems. The existing benchmarks tend to encourage program development at the local scale. Furthermore, the use of the periodic assessment scores in determining merit and bonus allocations may have resulted in lower levels of coopera- tive behavior between programs, which now see themselves as pitted against one another. Encouraging programs to undertake cooperative ef- forts to address regional scale problems thus needs to be incorporated into Sea Grant evaluation criteria and given a high value. Modifying the evaluation criteria to place greater weight on coopera- tive efforts is not intended as a recommendation to increase the complex- ity of the criteria. To the contrary, the current set of scored criteria are found to be overly complex and numerous, requiring significant amounts

OCR for page 1
SUMMARY 7 of time be devoted to developing consensus scores on a large number of criteria that, in many cases, account for a small percentage of the overall score. This endeavor to achieve greater precision by increasing the num- ber of score criteria tends to inadvertently discourage efforts to produce more holistic judgments of program performance. The Director of the National Sea Grant College Program, under supervision of the Secre- tary of Commerce and in consultation with National Sea Grant Review Panel and the individual Sea Grant programs, should substantially re- duce the overall number of scored criteria by combining various exist- ing criteria, while adding cooperative, network-building activities as an explicitly evaluated, highly valued criterion. Implementation of re- vised criteria should be postponed until the beginning of the next cycle of program review (the current review cycle will conclude in late 2006). PROGRAM ASSESSMENT TEAMS AND SITE VISITS Two of the major shortcomings of the current program assessment process are the limited overlap of the PATs in membership and the inabil- ity to evaluate the entire program in less than four years. Together, these problems compromise the reliability and credibility of the annual ranking required under the 2002 Act Amendments. Both shortcomings could be alleviated to a degree by shortening the PAT site visits and focusing atten- tion during the visit on the most essential evaluation tasks. Reducing the demands on the PATs would allow members to serve on more than one team and would also allow a larger number of site visits each year. As long as the PAT process remains the primary source of information to rate and rank individual programs, steps will need to be taken to improve the reliability and credibility of the process. The Director of the National Sea Grant College Program, under supervision of the Secretary of Com- merce and in consultation with the National Sea Grant Review Panel and the individual Sea Grant programs, should shorten the duration of and standardize the PAT site visits, based on the minimum time and material needed to cover essential, standardized elements of the pro- gram assessment. If, as recommended in chapters 3, 4, and 5 of this re- port, the annual evaluation process carried out by the NSGO is modified so as to provide a reliable and credible assessment of individual pro- grams, changes to the PAT process to improve reliability will be less urgent. This would allow greater flexibility for the scope and design of PAT visits.

OCR for page 1
8 EVALUATION OF THE SEA GRANT PROGRAM REVIEW PROCESS PROVIDING COORDINATION AND FACILITATION THROUGH INFORMED, ONGOING OVERSIGHT Greater involvement and ongoing oversight by the NSGO is needed to ensure that the program as a whole continues to improve while ad- dressing, local, regional and national needs. Informed oversight is also needed to lend credibility to annual program rankings and the allocation of merit and bonus funds. The two goals of program improvement and increased credibility can be simultaneously served by a meaningful ongo- ing, annual evaluation process that complements the periodic assessment carried out during the PAT site visit. This annual evaluation process, discussed in detail in Chapter 4, would replace the current NSGO Final Evaluation Review (FE). The current FE is summarized in Chapter 2. Review material prepared for the annual review should include an effec- tive annual report, supplemented by material that demonstrates the ex- tent to which the annual activities combine to form a cohesive ongoing program of activity organized to accomplish the objectives of appropri- ately ambitious strategic plans. The Director of the National Sea Grant College Program, under supervision of the Secretary of Commerce, should rank the individual Sea Grant programs based on a program evaluation process that includes more robust, credible, and transparent annual assessments of each individual Sea Grant program. Assessment of programs that have undergone periodic assessments in the preceding year should also include consideration of the PAT reports and the indi- vidual Sea Grant program directors' responses to the PAT reports. The additional effort required of individual Sea Grant programs to provide information on an annual basis can be offset to a degree by reducing the time required to prepare materials for the periodic assessment, if most of the information required by the latter can be made up of materials sub- mitted annually. FAIRNESS IN COMPETITION This study systematically evaluated the possibility that assessments, ratings, and the subsequent ranking of program performance are influ- enced by size or age of the program, location, type of institutional admin- istration linkages, and years of experience of the program officer within the NSGO. With one exception, a statistical analysis relating program ratings with these and other factors found no significant bias. The excep- tion is a positive correlation between years of experience of the program officer with the program under evaluation and the improvement in pro- gram score during the FE. Although the changes in program score were generally fairly small, the nature of the current assessment and ranking process results in a very narrow range of program scores overall; thus, even a minute difference in the score assigned to two similarly perform-

OCR for page 1
SUMMARY 9 ing programs that straddle the boundary between bonus categories could result in a significant difference in the amount of bonus funds awarded. The Director of the National Sea Grant College Program, under super- vision of the Secretary of Commerce, should revise the calculation of bonus funding allocation relative to program rank to ensure that small differences in program rank do not result in large differences in bonus funding, while preserving or even enhancing the ability to competi- tively award bonus funds as required by the National Sea Grant Col- lege Program Act Amendments of 2002 (P.L. 107299). IMPROVING PROGRAM COHESION The NSGO does not currently play a sufficient role in ongoing pro- gram assistance, communication, and assessment, or in maintaining close ongoing working relationships with the individual Sea Grant programs. This limits the ability of the NSGO, and by extension the National Direc- tor, to "provide an appropriately balanced response to local, regional, and national needs, which is reflective of integration with the relevant por- tions of strategic plans of the Department of Commerce and of the Ad- ministration" (33 U.S.C. 1123). In order for the NSGO to more effectively administer the NSGCP and coordinate and facilitate the efforts of the individual Sea Grant programs, thus fulfilling the federal role within the Sea Grant partnership, the ca- pabilities of the NSGO should be reevaluated, and likely, enhanced. The Secretary of Commerce, in consultation with the National Sea Grant Review Panel, should take steps to ensure that sufficient human and fiscal resources are available to allow robust, ongoing, and meaningful interaction among the Director of the National Sea Grant College Pro- gram, the staff of the National Sea Grant Office, the directors of indi- vidual Sea Grant programs, and the administrators of the home institu- tions of individual Sea Grant programs. While the purpose of this study was not to provide specific recom- mendations about how the NSGO should be organized, staffed, or funded, it does seem appropriate to point out various approaches that might be considered for achieving this recommended action, without significantly expanding the size of the NSGO staff. One such approach might include establishing a small number of program officers who spend a far greater portion of their time working with a small number of individual pro- grams with common challenges than is currently possible now. Indeed, additional approaches need to be further explored. The Director of the National Sea Grant College Program, under supervision of the Secre- tary of Commerce and in consultation with the National Sea Grant Review Panel and the individual Sea Grant programs, should under- take an evaluation of how work force capabilities and other compo-

OCR for page 1
10 EVALUATION OF THE SEA GRANT PROGRAM REVIEW PROCESS nents of effective program administration could be modified within the National Sea Grant Office to enhance its ability to coordinate and facilitate the actions of individual Sea Grant programs. Based on comments made during information gathering meetings, written correspondence submitted in response to committee requests, and various NSGO and NSGRP documents, it is apparent that a number of individual program directors remain confused about key aspects of the program assessment process, the annual evaluation process, and their impacts on program rankings and funding. Although responsibility for understanding this process rests with the individual Sea Grant program directors, the NSGO has a responsibility to make sure the process is rea- sonably straightforward and understandable. As discussed in Chapter 3, there should be greater attention and clarity regarding all aspects of pro- gram assessment. The Director of the National Sea Grant College Pro- gram, under supervision of the Secretary of Commerce, should take steps to ensure that the program assessment process (both the new an- nual assessment called for in this report and the PAT review) is well described and understood by individual program directors, congres- sional staff, personnel of the Office of Management and Budget, uni- versity and state administrators, and the general public. If the recommendations put forth above are implemented, the func- tions of the annual and periodic assessments will evolve such that both will provide different and independent sources of information about the state of the Sea Grant program as a whole. This information should pro- vide important insights to the Secretary of Commerce, the National Direc- tor, and potentially Congress. Thus, there would seem to be a need to synthesize and analyze the results of these assessments every four years, including a synthesis of all periodic assessments completed during that time and a systematic review of the NSGO. Developing such a "state of the Sea Grant program" report would seem to be an obvious role for the NSGRP. The Director of the National Sea Grant College Program, act- ing under authority of the Secretary, should direct the National Sea Grant Review Panel to undertake the development of a systematic re- view of the "state of the Sea Grant program" once every four years. The review should rely extensively on information collected during the an- nual and periodic assessments, augmented with a site visit to the Na- tional Sea Grant Office, and it should focus on how the program is functioning as a whole. In addition to commenting on how the program is performing in terms of the various criteria used during the assessments, the "state of the Sea Grant program" report could identify needed changes in how the program is administered, how the assessment process is car- ried out, or other areas as deemed valuable by the Secretary of Commerce or the National Director.