Evaluation of the Sea Grant Program Review Process (2006)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

1 Introduction T he National Sea Grant College Program (NSGCP) is a network of 301 individual Sea Grant programs2 and provides funds via these programs for marine and Great Lakes applied research, education, and outreach. Sea Grant has been a major source of funding in the United States for work in areas such as marine aquaculture, shellfish disease, aquatic nuisance species, coastal and estuarine ecology, seafood safety, marine biotechnology, marine engineering, marine technology develop- ment, and marine policy. Each of the 30 individual Sea Grant programs (see Figure 1.1) facilitates communication among university researchers, 1 Of the 30 Sea Grant programs, 28 are individual Sea Grant College programs and 2 are Sea Grant Institutional programs. California and Massachusetts have two Sea Grant pro- grams each, namely the University of California (located at Scripps Institution of Oceanog- raphy) and the University of Southern California (USC) programs and the Massachusetts Institute of Technology and the Woods Hole Oceanographic Institution (WHOI) programs, respectively. USC and WHOI are the two Sea Grant Institutional programs. Two individual Sea Grant programs operate as bi-state programs (Mississippi-Alabama, Illinois-Indiana). In addition to these 30 Sea Grant programs, Pennsylvania, Vermont (Lake Champlain), and Guam (3 programs) are in the initial stages of developing a full Sea Grant program and have not yet been included in the evaluation process. Source: F. Schuler, NOAA, personal communication, 2005. 2 For the purpose of this report, each of the 30 programs will be referred to as an "indi- vidual Sea Grant program" to differentiate it from the entire network, which is referred to as the National Program or NSGCP. Previous Sea Grant literature and legislation have also used the terms "state program" and "Sea Grant college/institute." 11 

12 EVALUATION OF THE SEA GRANT PROGRAM REVIEW PROCESS FIGURE 1.1 National Sea Grant College Program Network (Guam outside of map range). Source: National Oceanic and Atmospheric Administration (NOAA). industry members, policy makers, educators, and the public. Through its outreach and extension services, scientific research results are shared with the user communities, and these groups in turn communicate their prob- lems and needs back to the researchers. Thus, Sea Grant plays an impor- tant role in identifying problems, funding potential solutions, and pro- viding educational opportunities and materials. There are Web sites3 where individual Sea Grant program directors and the general public can obtain information on the NSGCP and all funded projects. ORIGIN OF THE NATIONAL SEA GRANT COLLEGE PROGRAM The idea of a Sea Grant college program was originally put forward by oceanographer, inventor, and writer Dr. Athelstan Spilhaus at the 93rd meeting of the American Fisheries Society in 1963. Interest in the Sea 3The National Sea Grant Office website (http://www.seagrant.noaa.gov), the National Sea Grant Library (http://nsgd.gso.uri.edu/), the National Sea Grant Law Center (http:// www.olemiss.edu/orgs/SGLC/lawcenterhome.htm), the National Sea Grant Education Teacher Resource--The Bridge (http://www.vims.edu/bridge/), and the Sea Grant Media Center (http://www.seagrantnews.org/). 

INTRODUCTION 13 Grant concept grew, much of it sparked by an editorial written by Spilhaus (1964, p. 993): . . . . I have suggested the establishment of "sea-grant colleges" in exist- ing universities that wish to develop oceanic work. . . . These would be modernized parallels of the great developments in agriculture and the mechanic arts which were occasioned by the Land Grant Act of about a hundred years ago. . . . Establishment of the land-grant colleges was one of the best investments this nation ever made. The same kind of imagi- nation and foresight should be applied to exploitation of the sea. In 1965, U.S. Senator Claiborne Pell of Rhode Island introduced legis- lation to establish Sea Grant colleges on campuses nationwide as centers of excellence in marine and coastal studies. With the adoption in 1966 of the National Sea Grant College and Program Act (P.L. 89­688) (see Ap- pendix C for key Sea Grant program legislation), Congress established a federal government/academic/industry partnership supporting the "es- tablishment, development, and operation of programs by sea grant col- leges and . . . other sea grant programs designed to achieve the gainful use of marine resources" (P.L. 89­688). The development of marine resources was defined as: . . . scientific endeavors relating to the marine environment, including but not limited to the fields oriented toward the development, conserva- tion, or economic utilization of the physical, chemical, geological and biological resources of the marine environment, the fields of marine com- merce and marine engineering, the fields relating to exploration or re- search in, the recovery of natural resources from, and the transmission of energy in, the marine environment; the fields of oceanography and oceanology and the fields with respect to the study of the economic, legal, medical or sociological problems arising out of the management, use, development recovery and control of the natural resources of the marine environment [P.L. 89­688]. The term marine environment was defined in the Act as: "the oceans, the Continental Shelf of the United States, the Great Lakes, the seabed and subsoil of the submarine areas adjacent to the coasts of the United States to the depth of two hundred meters or beyond that limit" (P.L. 89­688). Title 33, Chapter 22 of U.S. Code,4 The National Sea Grant College Program Act (33 U.S.C. 1121­1131) codified P.L. 89­688, and subsequent 4The U.S. Code is the codification by subject matter of the general and permanent laws of the United States. It is divided by broad subjects into 50 titles and published by the Office of the Law Revision Counsel of the U.S. House of Representatives. 

14 EVALUATION OF THE SEA GRANT PROGRAM REVIEW PROCESS amendments (e.g., P.L. 94­461,5 P.L. 105­160, P.L.107­299) (see Appendix H for reprinting of most sections in Chapter 22). In Section 1121 of Title 33, Congress declares the following policy: . . . . The understanding, assessment, development, utilization, and con- servation of such resources require a broad commitment and an intense involvement on the part of the federal government in continuing part- nership with State and local governments, private industry, universities, organizations, and individuals concerned with or affected by ocean, coastal, and Great Lakes resources. . . . The National Oceanic and Atmo- spheric Administration, through the national sea grant college program, offers the most suitable locus and means for such commitment and in- volvement through the promotion of activities that will result in greater such understanding, assessment, development, utilization, and conser- vation. The most cost-effective way to promote such activities is through continued and increased federal support of the establishment, develop- ment, and operation of programs and projects by sea grant colleges, sea grant institutes, and other institutions, including strong collaborations between administration scientists and scientists at academic institutions [33 U.S.C. 1121]. U.S. CODE: LEADERSHIP ROLES AND RESPONSIBILITIES DEFINED Title 33, Chapter 22 of the U.S. Code defines the responsibilities of the key components of the National Sea Grant College Program, and those of various other entities within the federal government. This section defines those responsibilities. The Secretary of Commerce, acting through the Under Secretary of Commerce for Oceans and Atmosphere,6 is responsible for maintaining the National Sea Grant College Program (referred to as "NSGCP" or "Na- tional Program" throughout this report7), which is to be administered by 5P.L. 94­461 completely rewrote the Congressional statement of findings, objectives, and purposes of the National Sea Grant Program Act to reflect the extension and strengthening of the national sea grant program to promote research, education, training, and advisory service activities in fields related to ocean and coastal resources through federal support to sea grant colleges, sea grant regional consortia, and other institutions through NOAA, and to make education, training, research, and advisory services responsive to state, local, re- gional, or national needs and problems. 6The term `'Secretary'' refers to the "Secretary of Commerce, acting through the Under Secretary of Commerce for Oceans and Atmosphere" (33 U.S.C. 1122 [15]). Currently, VADM Conrad Lautenbacher is the Under Secretary of Commerce for Oceans and Atmo- sphere as well as the Administrator of NOAA. 733 U.S.C. 1122 uses "The Program" to refer to the National Sea Grant College Program. 

INTRODUCTION 15 the National Sea Grant Office (NSGO). To carry out this function, the Secretary appoints the Director of the National Sea Grant College Pro- gram ("National Director"), who, subject to the supervision of the Secre- tary, administers the NSGCP and oversees the operation of the NSGO. Thus, the Secretary is ultimately responsible for the appointment, assign- ment of duties, transfer, and compensation of "such personnel as may be necessary, to administer the Program." In addition, the Secretary, in con- sultation with the National Sea Grant Review Panel (NSGRP) and the individual Sea Grant programs (both discussed below), and acting through the National Director, establishes guidelines related to the activi- ties and responsibilities of the individual Sea Grant programs. These guidelines are the major input into the development, every four years, of a strategic plan that establishes priorities for the National Program, pro- vides an appropriately balanced response to local, regional, and national needs, and is reflective of integration with relevant portions of the strate- gic plans of the Department of Commerce and the Administration (NOAA8 ; 33 U.S.C. 1123). The National Director is appointed by the Secretary of Commerce to administer the NSGCP and oversee the operation of the NSGO. The Na- tional Director, subject to the supervision of the Secretary and in consulta- tion with the NSGRP and individual Sea Grant programs, facilitates and coordinates the development of a strategic plan every four years that establishes priorities for the National Program (33 U.S.C. Sec. 1123). In addition, the National Director encourages the establishment and growth of individual Sea Grant programs and facilitates the cooperation and co- ordination of the National Program with other Federal activities in fields related to ocean, coastal, and Great Lakes resources. The National Direc- tor is also charged with evaluating the performance of the individual Sea Grant programs and rating the programs according to their relative per- formance. Title 33 U.S.C. 1123 (d)(3)(A) prescribes that the National Di- rector rank the individual Sea Grant programs "into no less than 5 catego- ries, with each of the 2 best-performing categories containing no more than 25 percent of the programs." Title 33 U.S.C. 1123 (d)(3)(B) prescribes that the National Director, subject to the availability of appropriations, allocate funding among individual Sea Grant programs so as to: (i) pro- mote healthy competition among the individual Sea Grant programs; (ii) encourage successful implementation of the individual programs; (iii) to the maximum extent consistent with other provisions of The National Sea Grant College Program Act provide a stable base of funding for individual 8 The term "Administration" refers to the National Oceanic and Atmospheric Adminis- tration (Title 33 U.S.C. 1122 [1]). 

16 EVALUATION OF THE SEA GRANT PROGRAM REVIEW PROCESS Sea Grant programs; and (iv) encourage and promote coordination and cooperation among the research, education, and outreach programs of NOAA and those of academic institutions (individual programs). The Directors of individual Sea Grant Colleges and Institutes (re- ferred to as directors of individual Sea Grant programs in this report) are required by Title 33 U.S.C. 1126 to coordinate program activities and help set local, regional and national priorities. Thus the directors of the 30 Sea Grant programs (see Figure 1.1 for map of current Sea Grant loca- tions) play a prominent and pivotal role in carrying out the function of the National Program. In addition to overseeing the merit review of all pro- posals for grants and contracts awarded under authority provided by Title 33 U.S.C. 1124, it is the responsibility of each individual director, in consultation with the National Director and the National Sea Grant Re- view Panel, to develop and implement a program that is consistent with the guidelines and priorities established by the National Strategic Plan required by Title 33 U.S.C. 1123 (c)(1). Furthermore, each individual Sea Grant program administers a significant pool of nonfederal funds, pro- vided either as a match to federal funding, or as a grant or contract with a state or local funding source. When acting collectively through the Sea Grant Association (SGA) (to be discussed shortly), the directors of the individual Sea Grant programs are a unified voice for these institutions on issues of importance to the oceans and coasts. NOAA's National Sea Grant Office (NSGO), as mandated by Title 33 U.S.C. 1123 (a), operates under the direction of the National Director and administers funding to the individual Sea Grant programs and over- sees several national funding competitions. The NSGO also facilitates the Department of Commerce designation of Sea Grant College programs9 and oversees the program assessment process. The NSGO, in consultation with the NSGRP and individual Sea Grant programs, is responsible for the development of a strategic plan that establishes priorities for the NSGCP, provides an appropriately balanced response to local, regional, and national needs, and is reflective of integration with relevant portions of the strategic plans of the Department of Commerce and NOAA. In addition, the NSGO is responsible for managing funding competitions for National Strategic Investments; three fellowship programs (i.e., the John A. Knauss Marine Policy Fellowship, the Sea Grant/NOAA Fisheries 9Designation of an individual Sea Grant program is the official naming of an institution of higher education or confederation of such institutions as an official Sea Grant College program as bestowed by the Secretary of Commerce. Applicant institutions must meet certain eligibility requirements. 

INTRODUCTION 17 Graduate Fellowship, the Sea Grant Industry Fellowship Program); and providing national coordination and leadership for Sea Grant's research, education, extension, communications, and fiscal networks. By law, the NSGO must use no more than 5 percent of the total budget for adminis- trative costs in any given fiscal year to administer the NSGCP (33 U.S.C. 1131). The National Sea Grant Review Panel (NSGRP), as mandated by Title 33 U.S.C. 1128, comprises 15 individuals with diverse backgrounds in marine affairs. The panel, appointed by the Secretary of Commerce, is charged with advising the Secretary and the National Director concern- ing: (i) applications or proposals for, and performance under, grants and contracts awarded; (ii) the Sea Grant Fellowship Program; (iii) the desig- nation and operation of Sea Grant Colleges and Institutes, and the opera- tion of Sea Grant Programs; (iv) the formulation and application of the planning guidelines and priorities (as discussed above); and (v) "such other matters as the Secretary refers to the panel for review and advice"(33 U.S.C. 1128). In 1998, in partial response to the 1994 National Research Council (NRC) report A Review of NOAA National Sea Grant College Program and the 1997 Report on Evaluation of Sea Grant College Programs requested by the NSGO and completed by Copeland et al. (1997), the National Director, acting under the supervision of the Secretary of Commerce, exercised authority under Title 33 U.S.C. 1128(b)(5) to request that the NSGRP for- mally oversee the periodic assessment of individual Sea Grant programs (required by Title 33 U.S.C. 1123[c][2] as amended by the National Sea Grant College Program Reauthorization Act of 1998 [P.L. 105­160]). The Sea Grant Association (SGA) is a nonprofit organization com- prising the academic institutions that participate in the NSGCP (i.e., pri- marily directors and other administrators of individual Sea Grant pro- grams). Though not a formal part of the NSGCP, the SGA plays an important role in furthering the Sea Grant program concept. The SGA provides the mechanism for these institutions to coordinate the research, education, training, and outreach activities of individual Sea Grant pro- grams and to set program priorities (to enhance the economic, environ- mental, and social potential of the nation's coastal, marine, and Great Lakes resources) at both the regional and national level (SGA Brochure, available online at http://www.sga.seagrant.org). THE EVOLVING SEA GRANT PROGRAM REVIEW PROCESS AND ENABLING LEGISLATION This report is the product of the NRC study requested by Congress in P.L.107­299, sponsored by the Under Secretary of Commerce for Oceans 

18 EVALUATION OF THE SEA GRANT PROGRAM REVIEW PROCESS and Atmosphere, and completed in 2006. It is the second NRC review of the Sea Grant program. The first NRC study, requested by the Under Secretary of Commerce for Oceans and Atmosphere in 1993 and completed in 1994, reviewed and evaluated the NSGCP to provide the basis for any needed changes to the program and to provide information for NOAA as it worked with Con- gress on the then pending National Sea Grant College Program Reautho- rization Act of 1998. The statement of task for the first study focused on the entire program, and the resulting report, A Review of the NOAA Na- tional Sea Grant College Program, made several recommendations for im- proving the program overall (Box 1.1). With regard to the proposal and program review process, the 1994 NRC report suggested that the review process for research proposals be decoupled from the NSGO evaluation of individual Sea Grant programs. It also recommended that standard scientific and peer review procedures be implemented for all of the individual Sea Grant programs. The report recommended that the review process and all aspects of program imple- mentation, including administration, be streamlined prior to FY 1996. In addition, the report called for the NSGO to evaluate the success of each individual program on a four-year cycle, using, in part, retrospective in- formation on recent achievements, based on measures for each of the three areas of research, education, and outreach. Finally, it was recom- mended that the NSGRP evaluate the performance of the NSGO on the same timetable. Following the release of the first NRC study and other efforts by Congress, NOAA, and other key players, the National Sea Grant College and Program Act was reauthorized by Congress in both 1998 and 2002 (P.L. 105­160 and P.L. 107­299, respectively). Some Highlights of the National Sea Grant College Program Legislation In the 1998 reauthorization of the National Sea Grant College and Program Act, Congress made some changes to the NSGCP. Among the more notable changes was the establishment of the performance based evaluation system, or PAT review, and the direction that funded resources would be allocated to programs based, in part, on their performance. Congress enacted several new program requirements in the 2002 re- authorization of the National Sea Grant College Program. Three of these new requirements are relevant to this study (P.L. 107­299): 

INTRODUCTION 19 Box 1.1 Key Issue Areas and Recommendations from A Review of the NOAA National Sea Grant College Program (Reprinted from NRC, 1994) ISSUE 1--SEA GRANT'S POSITION WITHIN NOAA The Administrator must ensure that NSGCP has appropriate responsibility and capability for research, education and outreach across NOAA. NSGCP should be relocated within NOAA to report directly to the Office of the Administrator. ISSUE 2--SHARED VISION AND STRATEGIC PLANNING State Sea Grant directors [individual Sea Grant program directors] and the Direc- tor of the NSGO [National Director] must cooperate to develop a single strategic plan articulating a shared vision and strategies which must be fully integrated into, and reflective of, NOAA's strategic plan. Unified Sea Grant strategic planning should begin immediately so that its results can be incorporated in the Fiscal Year (FY) 1997 NOAA budget. ISSUE 3--OVERLAPPING ROLES AND RESPONSIBILITES The roles and responsibilities of the state Sea Grant directors [individual Sea Grant program directors], NSGO, and National Sea Grant Review Panel [NSGRP] must be clarified. The resultant roles and responsibilities of NSGO and NSGRP should be clarified by the NOAA Administrator prior to the 1995 reauthorization. ISSUE 4--PROPOSAL REVIEW AND PROGRAM EVALUATION The review process for research proposals should be decoupled from the NSGO evaluation of state programs prior to the 1995 reauthorization. Standard scientific and peer review procedures should be implemented for all state Sea Grant pro- grams. The review process and all aspects of program implementation, including administration, should be streamlined prior to FY 1996. NSGO should evaluate the success of each state program on a four-year cycle, using, in part, retrospective information on recent achievements, based on measures for each of the three areas of research, education, and outreach. NSGRP should evaluate the perfor- mance of NSGO on the same timetable. ISSUE 5--INTERACTIONS WITH INDUSTRY NSGO and the state Sea Grant Programs must increase their interactions with marine industry to include program policy guidance, expanded outreach and ma- rine advisory services, joint research projects, and substantial industry financial support of the Sea Grant program. Action to address this recommendation should form part of the examination of the performance of each state program. These actions should be identified in the Sea Grant strategic plan. ISSUE 6--FUNDING The committee agreed that NSGCP needs additional funding to fulfill its poten- tial. In the last decade, the purchasing power of the average research grant has declined by about one-half. A steady increase in funding is necessary if the program's potential contributions to the nation's economic and environmental health are to be realized. Any additional funds appropriated to NSGCP should be split between enhancement of meritorious state programs and support of new initiatives. 

20 EVALUATION OF THE SEA GRANT PROGRAM REVIEW PROCESS · Strategic Planning: The Secretary of Commerce was directed to develop a strategic plan every four years and to consult and coordinate with the NSGRP and individual Sea Grant programs when doing so. · New Rating and Ranking of Sea Grant Programs: The Secretary, acting through the National Director, was directed to evaluate the perfor- mance of individual Sea Grant programs and rate these programs using the priorities, guidelines, and qualifications established by the Secretary, and rank the programs according to their relative performance into no less than 5 categories and with each of the 2 best-performing categories containing no more than 25 percent of the programs. · Review the Evaluation Process: A review of the Sea Grant evalua- tion and rating process was requested by the Act. The National Acad- emies was to start this review three years from the date of enactment (enactment was November 26, 2002). The U.S. Department of Commerce was directed to have the National Academies review the effectiveness of the evaluation and rating system (under the 2002 amendment) in deter- mining the relative performance of programs of individual Sea Grant programs, and to evaluate whether the individual Sea Grant programs have improved as a result of the evaluation process. The National Acad- emies was also requested to make recommendations to improve the over- all effectiveness of the evaluation process. STUDY APPROACH AND REPORT ORGANIZATION In response to congressional mandate (P.L. 107­299), the National Academies formed a committee of experts to carry out evaluate the NSGCP review process (see Box 1.2 for specific statement of task). Study Approach Information to support the study's conclusions was gathered through direct requests and public meetings. Materials and comments were re- quested from the NSGO, the NSGRP, the SGA, and from all individual Sea Grant program directors. Three public meetings were held in: Wash- ington, D.C. (March 2­4, 2005); Rockport, Maine (June 4­5, 2005), concur- rently with the first few days of the biennial Sea Grant Week; and Ann Arbor/Detroit, Michigan (August 9­11, 2005). During those meetings, the committee heard presentations by staff of the SGA and the NSGO. Open forum sessions were held where directors of individual Sea Grant pro- grams shared concerns and observations (see Box 1.3 for some questions asked by the Committee at an open forum). The committee also had one- on-one discussions with several individual Sea Grant program directors 

INTRODUCTION 21 Box 1.2 Statement of Task This study will assess new procedures adopted by the National Sea Grant Program since the publication of the 1994 National Research Council report A Review of NOAA National Sea Grant College Program to determine their impacts. During this study, the committee will address the impact of the new procedures and evaluation process on Sea Grant as a whole, identifying constructive changes and value added to overall institutional effectiveness, responsiveness, quality of management, leadership, and reputation. As part of this assessment, the committee will examine: (1) Effectiveness of major changes instituted in response to the recommen- dations of the 1994 NRC report with regard to individual program performance and quality. (2) Effectiveness of program review procedures with regard to accuracy, ac- countability, and enhancement of individual program performance. Both the previ- ous and current (adopted in 2003 in response to the Sea Grant Act of 2002 [P.L. 107-299]) review procedures will be assessed as specified below: · Review the effectiveness of the evaluation and rating system in deter- mining relative performance of programs with regard to management and quality of research, education, extension, and training activities; · Evaluate whether there have been improvements in programs as a re- sult of the evaluation process; · Evaluate the 2003 review procedures for their ability to meaningfully segregate individual programs into five categories based on competitive scores; and · Compare the effectiveness of the previous and 2003 review procedures with regard to the dual objectives of maximizing the quality of each program and of rating programs relative to each other for the purpose of determining performance- based funding. (3) Assessment of the usefulness and fairness of metrics developed to eval- uate programs with different operational constraints, resources, and local priori- ties. · Evaluate metrics for relevance and clarity; · Determine whether metrics provide a quantitative measure of quality of performance; and · Assess whether metrics improve consistency and objectivity of reviews from different teams evaluating a diverse portfolio of state Sea Grant programs. The committee will make recommendations for improving the overall effec- tiveness of the evaluation process to ensure fairness, consistency, and enhance- ment of performance. 

22 EVALUATION OF THE SEA GRANT PROGRAM REVIEW PROCESS Box 1.3 Questions Asked of all Attendees at the NRC Committee Forum on June 5, 2005, in Rockport, ME IMPROVEMENTS Has the effectiveness of review been improved by the changes since 2002? Apart from your program, are the best programs receiving the best scores? EFFICIENCY OF EVALUATIONS Does the expense and effort justify the outcome? If Sea Grant programs need to be ranked would you prefer to use the PAT process or do you have other sugges- tions? STANDARDIZATION OF PAT REVIEWS/FINAL EVALUATION Do benchmarks adequately capture program outcomes . . . for education? . . . for extension? . . . for outreach? . . . for research? Are we measuring what we care about or caring about what we measure? Is the use of weights for the subcategories appropriate? Do NSGO staff and pro- gram officers use the same weights and benchmarks consistently throughout time and for each program? Are PAT manuals and benchmarks shared throughout your program? Is there adequate consistency of PAT teams between reviews (to individual SG directors that have served on less than one PAT)? Have you expressed interest in serving on a PAT and not been invited? SUGGESTIONS What is the primary change you would make to the program assessment? Why? ROLE OF PROGRAM OFFICER What do you think the role of the program officer should be in general? REGIONAL COOPERATION AND COLLABORATION Have there been recent efforts to stimulate collaboration between programs? Do individual directors think collaboration a good idea? What's hindering the process? Is it valued in the evaluation process? and their staff members regarding the evaluation process and its impact on the individual programs. During the two-year study, the Committee observed a number of Program Assessment Team visits to individual Sea Grant programs-- several day meetings where individual Sea Grant programs are reviewed by an assigned Program Assessment Team (detailed discussion in Chap- ter 2)--in the states of Washington (2004), Oregon (2005), Georgia (2005), Ohio (2005), New York (2005), and Massachusetts (2005, Woods Hole Oceanographic Institution program). Two representatives of the commit- 

INTRODUCTION 23 tee attended the last few days of the week-long nonpublic meeting of the NSGO (February 2005), referred to throughout this report as "NSGO Fi- nal Evaluation Review." In addition to these efforts, NRC staff met and corresponded with U.S. Office of Management and Budget staff, congres- sional staff, and NSGO staff to obtain specific information and historical data. In addition, the committee contacted each individual Sea Grant pro- gram and requested information on previous and current PATs (Cycle 1 and Cycle 2), PAT reports, director response letters, final evaluation let- ters, information on costs of preparing for and conducting PAT reviews during Cycle 2, PAT briefing materials, and answers by individual Sea Grant directors to the questions raised in a formal letter written by the chair of the Committee (see Appendix I for the text of this letter sent to each individual Sea Grant program director). Almost all (28 out of the 30 reviewed Sea Grant programs) Sea Grant programs responded to this request. Individual programs sent many of the materials from both Cycle 1 and Cycle 2 reviews, including: PAT reports, director's responses to the final evaluation letter; funding allocation letters, PAT briefing materials, and information specifically on Cycle 2 costs incurred (as requested by the Committee). In addition, individual Sea Grant program directors submitted letters and comments, information on the Topical Advisory Teams (TATs), and miscellaneous additional documents. Further, the committee reviewed all key documents written on the Sea Grant review process to date (Byrne et al., 2000; Toll et al., 2001; Duce et al., 2002; Kudrna et al., 2005;10see reference lists of these reports), the last eight years of PAT manuals (from 1998 to 2005), and many of the documents provided on the NSGO Sea Grant shared database. The find- ings and recommendations of the committee were based on all of this research and their own experience. The Structure of the Report This report attempts to identify strengths and weaknesses of the cur- rent evaluation process and suggests improvements to enhance the per- formance of the individual programs and the Sea Grant program as a whole. Chapter 2 discusses the history of the Sea Grant program review process, thereby providing context for subsequent analyses. Chapter 3 explains and critiques the current assessment process (both the PAT re- view and NSGO Final Evaluation Review) and provides recommenda- 10Executive summary of the Kudrna et al. (2005) is provided in Appendix J. 

24 EVALUATION OF THE SEA GRANT PROGRAM REVIEW PROCESS tions for improving the reliability, efficiency, and transparency of the competitive process. Chapter 4 discusses the broader need for program oversight and management and makes suggestions for how to move be- yond the periodic assessment process in an effort to strengthen NSGCP efforts to provide an appropriately balanced response to local, regional, and national needs. Finally, Chapter 5 discusses report findings and rec- ommendations as a whole, summarizing key findings and recommenda- tions from chapters 3 and 4 in an integrated narrative.