Click for next page ( R2


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page R1
Evaluation of the Sea Grant Program Review Process Committee on the Evaluation of the Sea Grant Program Review Process Ocean Studies Board Division on Earth and Life Studies

OCR for page R1
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, N.W. Washington, DC 20001 NOTICE: The project that is the subject of this report was approved by the Gov- erning Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engi- neering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance. This study was supported by Contract/Grant No. DG133R04CQ0009 between the National Academy of Sciences and the National Oceanic and Atmospheric Ad- ministration. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the organizations or agencies that provided support for the project. International Standard book Number-10 0-309-10232-4 International Standard book Number-13 978-0-309-10232-2 Front cover: (Top) Elizabeth North checks her samples of blue crab larvae captured off the mouth of Chesapeake Bay. Photo credit: Michael W. Fincham. (Bottom) Photo taken during the Michigan Sea Grant Great Lakes Education Program (GLEP) benthic studies activity. Photo credit: Steve Stewart, Michigan Sea Grant College Program. (Right) Angie Sowers, Grace Brush and Holly Bowers (left to right) are taking core samples of sediment. Photo credit: Skip Brown. (Left) Ma- rine biologists from Point Reyes Bird Observatory (PRBO) rinsing nets after cap- turing krill at the shelf-break near California's Farallon Islands. Photo credit: Ben- jamin L. Saenz, PRBO. Back cover: (Top) Charleston Progressive middle-school students bag shell on Bowens Island for the S.C. Oyster Restoration and Enhance- ment Program. Photo courtesy of S.C. Department of Natural Resources. (Right) Stephen Giovannoni shown studying harmful algae that affect shellfish. Photo courtesy of Oregon Sea Grant. (Left) Teachers learn to build their own remotely operated vehicle during the Sea Perch workshop. Photo courtesy of S.C. Sea Grant Consortium. Additional copies of this report are available from the National Academies Press, 500 Fifth Street, N.W., Lockbox 285, Washington, DC 20055; (800) 624-6242 or (202) 334-3313 (in the Washington metropolitan area); Internet, http:// www.nap.edu. Copyright 2006 by the National Academy of Sciences. All rights reserved.

OCR for page R1
The National Academy of Sciences is a private, nonprofit, self-perpetuating soci- ety of distinguished scholars engaged in scientific and engineering research, dedi- cated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its mem- bers, sharing with the National Academy of Sciences the responsibility for advis- ing the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Wm. A. Wulf is president of the National Academy of Engineering. The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine. The National Research Council was organized by the National Academy of Sci- ences in 1916 to associate the broad community of science and technology with the Academy's purposes of furthering knowledge and advising the federal gov- ernment. Functioning in accordance with general policies determined by the Acad- emy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering commu- nities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. Wm. A. Wulf are chair and vice chair, respectively, of the National Research Council. www.national-academies.org

OCR for page R1

OCR for page R1
This report is dedicated to the memory of committee member Fred Noel Spiess (1919-2006)

OCR for page R1

OCR for page R1
COMMITTEE ON THE EVALUATION OF THE SEA GRANT PROGRAM REVIEW PROCESS1 JAMES M. COLEMAN (Chair), Louisiana State University, Baton Rouge ROBERT J. BAILEY, Oregon Department of Land Conservation and Management, Salem BILLY J. (B.J.) COPELAND, North Carolina State University, Pittsboro SUSAN E. COZZENS, Georgia Institute of Technology, Atlanta KEITH R. CRIDDLE, University of Alaska, Fairbanks ELIEZER GEISLER, Illinois Institute of Technology, Chicago MICHAEL W. HOWELL, University of South Florida, St. Petersburg RICHARD C. KARNEY, Martha's Vineyard Shellfish Group, Inc., Vineyard Haven, Massachusetts GEORGE I. MATSUMOTO, Monterey Bay Aquarium Research Institute, Moss Landing, California JOAN BRAY ROSE, Michigan State University, East Lansing ANDREW R. SOLOW, Woods Hole Oceanographic Institution, Woods Hole, Massachusetts FRED N. SPIESS, Scripps Institution of Oceanography, University of California, La Jolla Staff DAN WALKER, Study Director JENNIFER MERRILL, Senior Program Officer AMANDA BABSON, Christine Mirzayan Science and Technology Policy Fellow NANCY CAPUTO, Research Associate 1 Committee and staff biographies can be found in Appendix A. vii

OCR for page R1
OCEAN STUDIES BOARD SHIRLEY A. POMPONI (Chair), Harbor Branch Oceanographic Institution, Fort Pierce, Florida LEE G. ANDERSON, University of Delaware, Newark JOHN A. ARMSTRONG, IBM Corporation (retired), Amherst, Massachusetts WHITLOW AU, University of Hawaii at Manoa ROBERT G. BEA, University of California, Berkeley ROBERT DUCE, Texas A&M University, College Station MARY (MISSY) H. FEELEY, ExxonMobil Exploration Company, Houston, Texas HOLLY GREENING, Tampa Bay National Estuary Program, St. Petersburg, Florida DEBRA HERNANDEZ, Hernandez and Company, Isle of Palms, South Carolina CYNTHIA M. JONES, Old Dominion University, Norfolk, Virginia WILLIAM A. KUPERMAN, Scripps Institution of Oceanography, La Jolla, California FRANK E. MULLER-KARGER, University of South Florida, St. Petersburg JOAN OLTMAN-SHAY, NorthWest Research Associates, Inc., Bellevue, Washington ROBERT T. PAINE, University of Washington, Seattle S. GEORGE H. PHILANDER, Princeton University, New Jersey RAYMOND W. SCHMITT, Woods Hole Oceanographic Institution, Massachusetts DANIEL SUMAN, Rosenstiel School of Marine and Atmospheric Science, University of Miami, Florida STEVEN TOMASZESKI, Rear Admiral, U.S. Navy (retired), Fairfax, Virginia ANNE M. TREHU, Oregon State University, Corvallis Staff SUSAN ROBERTS, Director FRANK HALL, Program Officer SUSAN PARK, Associate Program Officer ANDREAS SOHRE, Financial Associate SHIREL SMITH, Administrative Coordinator JODI BOSTROM, Research Associate NANCY CAPUTO, Research Associate SARAH CAPOTE, Senior Program Assistant viii

OCR for page R1
Acknowledgments T his report was greatly enhanced by the participants of the three workshops held as part of this study. The committee would first like to acknowledge the efforts of those who gave presentations at meetings: Ronald Baird, NOAA; Leon M. Cammen, NOAA; Jonathan Kramer, Sea Grant Association; Frank Kudrna, Sea Grant Review Panel; Nathaniel E. Robinson, Sea Grant Review Panel; and Francis Schuler, NOAA. These talks helped set the stage for fruitful discussions in the closed sessions that followed. The committee is also grateful to a number of people who provided important discussion and/or material for this report, these include: Brian Allee, Alaska Sea Grant, University of Alaska Fairbanks Paul Anderson, Maine Sea Grant, University of Maine, Orono Anders W. Andren, Wisconsin Sea Grant, University of Wisconsin- Madison James C. Cato, Florida Sea Grant, University of Florida, Gainesville Ruperto Chapparo, Puerto Rico Sea Grant, University of Puerto Rico, Mayaguez Barry A. Costa-Pierce, Rhode Island Sea Grant, University of Rhode Island, Narragansett Penelope D. Dalton, Washington Sea Grant, University of Washington, Seattle M. Richard DeVoe, South Carolina Sea Grant Consortium, Charleston ix

OCR for page R1
x ACKNOWLEDGMENTS Linda E. Duguay, University of Southern California Sea Grant Program, Los Angeles Louie Echols, Washington Sea Grant, University of Washington, Seattle Lawrence K. Forcier, Lake Champlain Sea Grant, University of Vermont, Burlington Nikola Garber, NOAA, Silver Spring, Maryland E. Gordon Grau, Hawaii Sea Grant, University of Hawaii, Honolulu Sami Grimes, NOAA, Silver Spring, Maryland Jeff Gunderson, Minnesota Sea Grant, University of Minnesota, Duluth Ronald Hodson, North Carolina Sea Grant, North Carolina State University, Raleigh Mark Malchoff, Lake Champlain Sea Grant, University of Vermont, Burlington Robert Malouf, Oregon Sea Grant, Oregon State University, Corvallis Jack S. Mattice, New York Sea Grant, State University of New York, Stony Brook Judith E. McDowell, Woods Hole Oceanographic Institution Sea Grant, Woods Hole, Massachusetts Russell A. Moll, California Sea Grant, University of California- San Diego, La Jolla Edward C. Monahan, Connecticut Sea Grant, University of Connecticut, Groton Amy Painter, NOAA, Silver Spring, Maryland Jonathan Pennock, New Hampshire Sea Grant, University of New Hampshire, Durham Melissa Pearson, NOAA, Silver Spring, Maryland Mac V. Rawson, Georgia Sea Grant, University of Georgia, Athens Jeffrey M. Reutter, Ohio Sea Grant, Ohio State University, Columbus Carl Richards, Minnesota Sea Grant, University of Minnesota, Duluth William L. Rickards, Virginia Sea Grant, University of Virginia, Charlottesville Donald Scavia, Michigan Sea Grant, Ann Arbor Robert R. Stickney, Texas Sea Grant, Texas A&M University, College Station William Sullivan, Illinois-Indiana Sea Grant, University of Illinois Urbana-Champaign

OCR for page R1
ACKNOWLEDGMENTS xi LaDon Swann, Mississippi-Alabama Sea Grant Consortium, Ocean Springs, Mississippi Nancy Targett, Delaware Sea Grant, University of Delaware, Newark Michael P. Weinstein, New Jersey Sea Grant, New Jersey Marine Science Consortium, Fort Hancock Charles Wilson, Louisiana Sea Grant, Louisiana State University, Baton Rouge This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with pro- cedures approved by the NRC's Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integ- rity of the deliberative process. We wish to thank the following individu- als for their participation in their review of this report: Robert A. Duce, Texas A&M University, College Station Dennis Hedgecock, University of Southern California, Los Angeles Susan M. Henrichs, University of Alaska Fairbanks, Fairbanks Ronald N. Kostoff, Office of Naval Research, Arlington, Virginia Jonathan Kramer, University System of Maryland, College Park Christopher G. Mann, Marine Aquaculture Task Force, Takoma Park, Maryland Bonnie J. McCay, Cook College of Rutgers University, New Brunswick, New Jersey Arthur R. Nowell, University of Washington, Seattle Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the conclu- sions or recommendations nor did they see the final draft of the report before its release. The review of this report was overseen by Michael K. Orbach, Duke University, appointed by the Divison on Earth and Life Studies, who was responsible for making certain that an independent examination of this report was carried out in accordance with institu- tional procedures and that all review comments were carefully consid- ered. Responsibility for the final content of this report rests entirely with the authoring committee and the institution.

OCR for page R1

OCR for page R1
Contents SUMMARY 1 1 INTRODUCTION 11 Origin of the National Sea Grant College Program, 12 U.S. Code: Leadership Roles and Responsibilities Defined, 14 The Evolving Sea Grant Program Review Process and Enabling Legislation, 17 Some Highlights of the National Sea Grant College Program Legislation, 18 Study Approach and Report Organization, 20 Study Approach, 20 The Structure of the Report, 23 2 HISTORY OF SEA GRANT PROGRAM REVIEW AND ASSESSMENT 25 The Transition: 19941998, 26 Program Review: 1998 and Beyond, 27 Program Assessment Team (External Review), 29 PAT Guidelines, 30 PAT Report and Program Directors Formal Response, 32 PAT Review Criteria, 34 The Metrics Committee, 35 Final Evaluation Process, 35 Program Performance Rating, 38 Final Report, Ranking, and Allocation of Funds, 39 Concerns with the Program Evaluation Process, 40 xiii

OCR for page R1
xiv CONTENTS 3 CRITIQUE OF THE PERIODIC ASSESSMENT PROCESS 43 Guidance Documents, 43 Program Assessment Team Visit, 50 Final Evaluation Process, 52 Credibility of PAT and FE Scoring Process, 53 Improving the Value of Assessment, 56 Improvement, 57 Distribution of Merit and Bonus Funds, 58 Potential Biases, 60 Broad Program Management, 62 Collaboration among Individual Sea Grant Programs, 63 Findings and Recommendations Regarding the Periodic Assessment Process, 64 Rethinking the Program Assessment Process, 68 4 PROGRAM OVERSIGHT AND MANAGEMENT 69 Introduction to Program Management, 70 Management and Oversight of Research and Outreach Programs, 70 The Sea Grant Review Process Compared to Other Federal Programs, 73 Implications of Review on Funding and Competition, 73 Allocation of Funds, Peer-Review, Competition, and Awards to Meritorious Projects, 74 Strategic Planning as a Program Development and Evaluation Framework, 75 Role of the National Sea Grant Office, 77 Annual and Periodic Assessment Processes as Integral Elements of Program Administration, 78 Findings and Recommendations Regarding Program Oversight and Management, 81 Sea Grant Program Administration, 82 Role of the National Sea Grant Office, 83 Strategic Planning Process, 83 Increasing Reliability and Transparency of Annual and Periodic Assessment, 84 5 MAJOR FINDINGS AND RECOMMENDATIONS 87 Impact of Changes in Response to the 1994 Report, 87 Effectiveness of Post-1998 Evaluation, 89 Strategic Planning, 91 Performance Criteria, 91

OCR for page R1
CONTENTS xv Program Assessment Team and Site Visit, 93 Providing Coordination and Facilitation through Informed, Ongoing Oversight, 94 Fairness in Competition, 94 Improving Program Cohesion, 95 REFERENCES 99 APPENDIXES A Committee and Staff Biographies 101 B List of Acronyms 109 C Key Sea Grant Legislation 111 D National Sea Grant Program Memorandum on NSGO Final Evaluation and Merit Funding, April 22, 1999 129 E Revised Policy Memorandum on NSGO Final Evaluation and Merit Funding (2005); April 8, 2005 139 F A Multivariate Analysis of Potential Biases in the Final Evaluation Scores 157 G Expected Indicators of Performance and Other Issues of Importance 165 H U.S. Code, Title 33, Chapter 22 169 I Letter from James Coleman, Chair-NRC Committee Sent to All Individual Sea Grant Program Directors 185 J Executive Summary from "Review and Recommendations: Sea Grant Program Evaluation Process" 189

OCR for page R1