National Academies Press: OpenBook

Evaluation of the Sea Grant Program Review Process (2006)

Chapter: E Revised Policy Memorandum on NSGO Final Evaluation and Merit Funding (2005); April 8, 2005

« Previous: D National Sea Grant Program Memorandum on NSGO Final Evaluation and Merit Funding, April 22, 1999
Suggested Citation:"E Revised Policy Memorandum on NSGO Final Evaluation and Merit Funding (2005); April 8, 2005." National Research Council. 2006. Evaluation of the Sea Grant Program Review Process. Washington, DC: The National Academies Press. doi: 10.17226/11670.
×
Page 139
Suggested Citation:"E Revised Policy Memorandum on NSGO Final Evaluation and Merit Funding (2005); April 8, 2005." National Research Council. 2006. Evaluation of the Sea Grant Program Review Process. Washington, DC: The National Academies Press. doi: 10.17226/11670.
×
Page 140
Suggested Citation:"E Revised Policy Memorandum on NSGO Final Evaluation and Merit Funding (2005); April 8, 2005." National Research Council. 2006. Evaluation of the Sea Grant Program Review Process. Washington, DC: The National Academies Press. doi: 10.17226/11670.
×
Page 141
Suggested Citation:"E Revised Policy Memorandum on NSGO Final Evaluation and Merit Funding (2005); April 8, 2005." National Research Council. 2006. Evaluation of the Sea Grant Program Review Process. Washington, DC: The National Academies Press. doi: 10.17226/11670.
×
Page 142
Suggested Citation:"E Revised Policy Memorandum on NSGO Final Evaluation and Merit Funding (2005); April 8, 2005." National Research Council. 2006. Evaluation of the Sea Grant Program Review Process. Washington, DC: The National Academies Press. doi: 10.17226/11670.
×
Page 143
Suggested Citation:"E Revised Policy Memorandum on NSGO Final Evaluation and Merit Funding (2005); April 8, 2005." National Research Council. 2006. Evaluation of the Sea Grant Program Review Process. Washington, DC: The National Academies Press. doi: 10.17226/11670.
×
Page 144
Suggested Citation:"E Revised Policy Memorandum on NSGO Final Evaluation and Merit Funding (2005); April 8, 2005." National Research Council. 2006. Evaluation of the Sea Grant Program Review Process. Washington, DC: The National Academies Press. doi: 10.17226/11670.
×
Page 145
Suggested Citation:"E Revised Policy Memorandum on NSGO Final Evaluation and Merit Funding (2005); April 8, 2005." National Research Council. 2006. Evaluation of the Sea Grant Program Review Process. Washington, DC: The National Academies Press. doi: 10.17226/11670.
×
Page 146
Suggested Citation:"E Revised Policy Memorandum on NSGO Final Evaluation and Merit Funding (2005); April 8, 2005." National Research Council. 2006. Evaluation of the Sea Grant Program Review Process. Washington, DC: The National Academies Press. doi: 10.17226/11670.
×
Page 147
Suggested Citation:"E Revised Policy Memorandum on NSGO Final Evaluation and Merit Funding (2005); April 8, 2005." National Research Council. 2006. Evaluation of the Sea Grant Program Review Process. Washington, DC: The National Academies Press. doi: 10.17226/11670.
×
Page 148
Suggested Citation:"E Revised Policy Memorandum on NSGO Final Evaluation and Merit Funding (2005); April 8, 2005." National Research Council. 2006. Evaluation of the Sea Grant Program Review Process. Washington, DC: The National Academies Press. doi: 10.17226/11670.
×
Page 149
Suggested Citation:"E Revised Policy Memorandum on NSGO Final Evaluation and Merit Funding (2005); April 8, 2005." National Research Council. 2006. Evaluation of the Sea Grant Program Review Process. Washington, DC: The National Academies Press. doi: 10.17226/11670.
×
Page 150
Suggested Citation:"E Revised Policy Memorandum on NSGO Final Evaluation and Merit Funding (2005); April 8, 2005." National Research Council. 2006. Evaluation of the Sea Grant Program Review Process. Washington, DC: The National Academies Press. doi: 10.17226/11670.
×
Page 151
Suggested Citation:"E Revised Policy Memorandum on NSGO Final Evaluation and Merit Funding (2005); April 8, 2005." National Research Council. 2006. Evaluation of the Sea Grant Program Review Process. Washington, DC: The National Academies Press. doi: 10.17226/11670.
×
Page 152
Suggested Citation:"E Revised Policy Memorandum on NSGO Final Evaluation and Merit Funding (2005); April 8, 2005." National Research Council. 2006. Evaluation of the Sea Grant Program Review Process. Washington, DC: The National Academies Press. doi: 10.17226/11670.
×
Page 153
Suggested Citation:"E Revised Policy Memorandum on NSGO Final Evaluation and Merit Funding (2005); April 8, 2005." National Research Council. 2006. Evaluation of the Sea Grant Program Review Process. Washington, DC: The National Academies Press. doi: 10.17226/11670.
×
Page 154
Suggested Citation:"E Revised Policy Memorandum on NSGO Final Evaluation and Merit Funding (2005); April 8, 2005." National Research Council. 2006. Evaluation of the Sea Grant Program Review Process. Washington, DC: The National Academies Press. doi: 10.17226/11670.
×
Page 155
Suggested Citation:"E Revised Policy Memorandum on NSGO Final Evaluation and Merit Funding (2005); April 8, 2005." National Research Council. 2006. Evaluation of the Sea Grant Program Review Process. Washington, DC: The National Academies Press. doi: 10.17226/11670.
×
Page 156

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Appendix E Revised Policy Memorandum on NSGO Final Evaluation and Merit Funding (2005); April 8, 2005 [Stamped Date on Original: April 8, 2005] MEMORANDUM FOR: Sea Grant Directors FROM: Ronald C. Baird Director SUBJECT: Revised Policy Memorandum on NSGO Final Evaluation and Merit Funding (2005) This policy document revises the previous, "Policy Memorandum on NSGO Final Evaluation and Merit Funding," (1999). Revisions incorpo- rate recommendations from the Toll Committee Report ("Review and Rec- ommendations: Sea Grant Program Evaluation Process Report of the Sea Grant Review Panel's Program Evaluation Committee," October 2001) and the new requirements under the ``National Sea Grant College Program Act Amendments of 2002'' (Public Law 107­299). The policy was also re- viewed for consistency with the memorandum, "Policy for the Allocation of Funds, FY2003 and Beyond," (March, 2003). 139

140 APPENDIX E INTRODUCTION To increase the overall performance and effectiveness of the Sea Grant network and associated institutional programs, the National Sea Grant Office (NSGO) tracks and analyzes the performance of each Sea Grant program. In 1998, the NSGO moved from the prospective evaluation of Sea Grant program proposals and their individual projects to a retrospec- tive evaluation of overall program performance and accomplishment. The intent of evaluation is to improve Sea Grant's overall performance while providing incentives for strong performance. This change was first rec- ommended by the Ocean Studies Board of the National Research Council in 1994 and subsequently endorsed by the institutions that comprise the Sea Grant network. The National Sea Grant College Program Act of 1998 (P.L. 105-160) codified this change and charged the Director of the Na- tional Sea Grant College Program to: ``204(3) With respect to sea grant colleges and sea grant institutes, the Director shall-- ``(A) evaluate the programs of sea grant colleges and sea grant institutes, using the priorities, guidelines, and qualifications established by the Secretary; ``(B) subject to the availability of appropriations, allocate funding among sea grant colleges and sea grant institutes so as to-- ``(i) promote healthy competition among sea grant colleges and institutes; ``(ii) encourage successful implementation of sea grant programs; and ``(iii) to the maximum extent consistent with other provisions of this Act, provide a stable base of funding for sea grant colleges and institutes; and ``(C) ensure compliance with the guidelines for merit review....'' SEA GRANT PROGRAM EVALUATION In response to the 1994 National Research Council/Ocean Studies Board Report and the 1998 Sea Grant reauthorization legislation, the NSGO in- troduced a system of performance-based reviews ("Implementation of Pro- gram Evaluation Procedures in the National Sea Grant College Program," April 20, 1998) that continue to the present. Among other things, this requires

APPENDIX E 141 (1) Program Assessment Team Evaluations and (2) NSGO Final Evalua- tion Reviews. 1. Program Assessment Team Evaluations An onsite evaluation by a Program Assessment Team (PAT) of each insti- tution responsible for administering a Sea Grant program is conducted under the auspices of the Sea Grant Review Panel. The NSGO Director will notify university officials of the upcoming PAT. The role of the PAT is to assess the performance of a Sea Grant program with respect to a standard set of evaluation criteria and benchmarks and to make recom- mendations for the improvement of the program. The PAT Report and recommendations are used primarily to improve individual program per- formance and also to provide a basis for comparison among programs over the long term. During the first cycle of PAT reviews (1998­2001), teams assigned a grade to each of the four major benchmark categories and an overall grade using appropriate weights: Organizing and Managing the Program (20%) Connecting Sea Grant with Users (20%) Effective & Aggressive Long-Range Planning (10%) Producing Significant Results (50%) As a result of the Toll Committee Report, and subsequent recommenda- tions by the Sea Grant Review Panel, the grading regime was changed for the second PAT cycle that began in 2003. Instead of marks for each of the four benchmark categories, the PAT will now provide a rating for each of 14 finer scale sub-elements under the four major benchmark categories and no overall grade. Those sub-elements and the weights assigned to each are listed below and described in detail in the PAT Manual: Organizing and Managing the Program (20%) Leadership of the Program (6%) Institutional Setting and Support (4%) Project Selection (2%) Recruiting Talent (3%) Effective and Integrated Program Components (5%)

142 APPENDIX E Connecting Sea Grant with Users (20%) Engagement with Appropriate User Communities (15%) Partnerships (5%) Effective & Aggressive Long-Range Planning (10%) Strategic Planning Process (4%) Strategic Plan Quality (4%) Implementation Plan (2%) Producing Significant Results (50%) Contributions to Science and Technology (10%) Contributions to Extension, Communications and Education (10%) Impact on Society, the Economy and the Environment (25%) Success in Achieving Planned Program Outcomes (5%) Each sub-element will be given one of four possible ratings by the PAT: · Needs Improvement--In general, performance does not reach the benchmark for this sub-element. · Meets Benchmark--In general, performance meets, but does not exceed, the benchmark for this sub-element. · Exceeds Benchmark--In general, performance goes beyond what would be required to simply meet the benchmark for this sub- element. · Highest Performance--Performance goes well beyond the bench- mark for this sub-element and is outstanding in all areas. The PAT will provide a briefing for the Sea Grant Director and appropri- ate university officials at the end of the PAT visit. The ratings are pre- sented at the debriefing and a rating sheet is also provided for the record. Following the PAT review, the chair of the assessment team provides a written PAT Report to the institution. The Sea Grant institution is encour- aged to provide the NSGO a written response to the PAT Report. The comprehensive PAT Report and the institutional response to the report will become part of the record for the institutional program and both will be considered at the NSGO Review. Actions taken after the PAT by a Sea Grant institution in response to PAT recommendations will be acknowl- edged, but will not become a factor in the current NSGO ratings. All improvements made by the Sea Grant institution after the PAT will be more properly considered in the next PAT cycle.

APPENDIX E 143 2. NSGO Final Evaluation Reviews The four-year NSGO Final Evaluation Review (henceforth, NSGO Re- view) is conducted by the NSGO in the year following the program's PAT visit (usually February). The evaluation relies primarily on the informa- tion provided by the program to the PAT, the PAT Report and ratings, and the institutional response to the PAT Report. A NSGO Final Evalua- tion Report (henceforth, NSGO Report) summarizes the findings of the NSGO performance review for that Sea Grant program over the last four- year review cycle. In addition to the report, the NSGO provides a perfor- mance rating to each Sea Grant program as part of the evaluation. The primary objective of the NSGO Review is to provide local manage- ment with an assessment of performance and specific recommendations directed toward improvement and maintenance of existing program strengths. The second objective is to assign programs to a rating category that can be used in the allocation of a partial amount of Sea Grant funds. This rating was the basis during the first cycle of reviews (1998­2002) for allocating merit funding from a $3,000,000 pool of funds set aside in the Sea Grant budget for that purpose. The seven or eight Sea Grant programs that were evaluated by a PAT in the prior calendar year are considered as a group and scheduled for NSGO Review every four years. The NSGO conducts the final evaluation during a one-week period, typically in the month of February. The criteria and benchmarks used in the NSGO Review are identical to those used by the PAT. Effort is taken to assure that all programs are evaluated in a similar manner using the same standard criteria and performance benchmarks listed above and described in detail in the PAT Manual. The NSGO Director has mandated that all NSGO technical staff partici- pate and be present for the entire review. One or more members of the National Sea Grant Review Panel, usually from the panel's executive com- mittee, also attend this meeting as observers, which is consistent with the panel's oversight responsibilities for the conduct of program evaluation. Performance Information Considered In preparation for the NSGO Review, the NSGO Program Officer pre- pares materials for distribution. The PAT Report and the SG program's response provide the primary input to the NSGO review process.

144 APPENDIX E Documents and reports are distributed to NSGO staff for study prior to the review, which include: · The overview section from the program's briefing book prepared by the Sea Grant program for its PAT Review. This section includes a program description, the response to previous PAT recommen- dations, and a description of program accomplishments and im- pacts. (See NSGO Guidelines for Program Assessment Briefing Books, May 9, 2003) · The Program Assessment Team Report as signed off by the PAT Chair · The Sea Grant program's formal written response to the PAT Re- port · The program's strategic and implementation plans Collectively, the NSGO staff also has access to documents on file for each program, part of the continuous and ongoing communications that occur between a Sea Grant program and the NSGO. Some of these materials are less generally available to the PAT and represent additional information for the NSGO to use in the evaluation process. This includes: · Annual progress reports · Omnibus proposals · Publications · Archived information on accomplishments · Trip reports and peer review panel visits by the Program Officer · Topical Assessment Team reports (if any) · Detailed Sea Grant funding information · Supportive material deemed to be relevant by the Program Officer or staff Structure of the NSGO Review The NSGO Executive Director is responsible for planning the review and for the staff preparation needed to carry out the review. NSGO Program Officers are responsible for preparing a presentation on the programs that will be reviewed. The NSGO Director facilitates the evaluation sessions during review week. Each half-day session is focused on a single program and is reviewed on its own merits and not in direct comparison with other programs. Since Program Officers play a central role in the NSGO Review, it is the NSGO's policy not to reassign Program Officers in mid-cycle, if at all

APPENDIX E 145 possible. Assignments are made with the goal of maintaining continuing associations between the federal program officer and a Sea Grant pro- gram over the review cycle, or longer. However, at times this will not be feasible due to NSGO staff turnover. To begin the review of each program, the NSGO program officer provides an overview of the program's performance since the last PAT (five years for the second cycle of reviews, but normally four years). Each program officer follows a prescribed format using a standardized presentation tem- plate that ensures consistency of the kinds of information being presented. The template follows the benchmarks and indicators of performance from the PAT Manual. Evaluation-related materials on file in the NSGO (see above) are considered where appropriate. For example, participation in national competitions and responsiveness to network-wide activities have bearing on successful performance, and often the NSGO will have a better perspective here than would the PAT. Following the Program Officer's presentation, the NSGO director facili- tates a discussion of the program. The review is structured to consider the same criteria and benchmarks addressed by the PAT. Each of the four major evaluation criteria and the 14 sub-elements are discussed in succes- sion, including the PAT findings and ratings. All PAT recommendations are reviewed. Those deemed most critical from the NSGO perspective are highlighted for inclusion in the NSGO Report. Where appropriate, PAT recommendations may be modified and additional recommendations de- veloped based on the NSGO Review. During the review process, "best management practices" are identified for subsequent promulgation to the Sea Grant network. The discussions and findings from the NSGO Review form the basis for a report that is prepared under the direction of and signed by the NSGO Director. The NSGO Report is best understood when read in conjunction with the PAT Report, which will be included with the NSGO Report when distributed. It is the NSGO's intention to complete the report and transmit it to the Sea Grant Director within 30-days of the end of the NSGO Review. While the NSGO Report findings and ratings are consid- ered final, the draft report will be sent to the program director for factual review and correction of minor errors prior to final distribution. A ten- day turnaround period is considered a reasonable time for directors to respond, but extensions may be requested. The NSGO report will be sent only to the Sea Grant program. The Sea Grant program director can de- cide how to use the NSGO Report within their university. The NSGO

146 APPENDIX E Report for a given program will be distributed to the Panel members who served on that particular PAT. Under certain circumstances, the NSGO director may elect to send a spe- cial letter of findings to the institution. These specials reports, it is ex- pected, will be used infrequently and only in highly unusual cases that warrant communication at a higher administrative level in the university. NSGO RATING DECISIONS One objective of the NSGO Review process is to provide a consistent approach to rating Sea Grant programs. The intensive, weeklong PAT evaluation by a team of experts, who interact with university officials, constituents, and government officials, provides credible information from which to judge a program's performance. The NSGO Review pro- vides an additional assessment of performance that adds to the PAT re- view in several salient ways: · Performance-relevant information available to the NSGO results, not only from the PAT process, but also from a continuous process of evaluation and dialogue between the NSGO and the Sea Grant program over the full four-year cycle. · For the NSGO Review, the institution's formal response to the PAT's findings and recommendations is available and explicitly considered. This additional input is critical information for the NSGO Review and can often provide clarifying information on program performance. · The NSGO Review provides a broad perspective across seven to eight programs each year, and across all Sea Grant programs over a four-year cycle. While the NSGO evaluates programs individu- ally, by considering a group of programs at the same time and with the same reviewers, more consistency for assigning ratings can be achieved. The rating of a program involves the use of judgment in weighing the qualitative and quantitative evidence available. Following extensive dis- cussion of a program's performance under each evaluation criteria, the NSGO staff provides their individual rating ranging from 1(highest) to 4 (lowest) for the 14 evaluation sub-elements. The NSGO director will set a minimum level of experience that will be required of new NSGO staff

APPENDIX E 147 members before they will be asked to contribute their individual rating of programs. Nonetheless, new technical staff members are expected to be present and to participate in the discussions. The NSGO rating for a program is derived from reaching a broad consen- sus (two-thirds majority) of individual NSGO staff ratings for the 14 sub- elements using the PAT results as reference. Taking into account the pro- portional weighting of each of the 14 sub-elements (e.g., Contributions to Science and Technology - 10%) and the NSGO consensus ratings for each, a program score is calculated. The NSGO final rating for the program is determined by locating a program's score along a fixed four-category rating scale for merit funding and a variable two-category rating scale for bonus funding. Merit funding and bonus funding allocations are dis- cussed in detail below. The NSGO Review is a semi-autonomous review that significantly weighs and is informed by the PAT findings and ratings. As would be expected, the findings and ratings of the PAT and the NSGO are in agreement in the large majority of cases. As a matter of policy, however, if there is not a broad consensus agreement (two-thirds majority) on a particular sub- element rating, the NSGO assigns a rating consistent with the PAT rating for that sub-element. Final ratings for the group of seven or eight programs are considered at the last session of the review week. The NSGO final ratings are reviewed and considered for adjustment, if NSGO staff offers a convincing case for reconsideration. Rating adjustments result, as for all NSGO ratings, only from a broad consensus agreement (two-thirds majority). All decisions to change a PAT rating are ultimately the final responsibility of the NSGO Director. At the conclusion of the session, all NSGO ratings are consid- ered final. The final NSGO ratings are used to assign each program to a merit-fund- ing category and are also interleaved with the last rating of all other Sea Grant programs to determine eligibility for bonus-funding categories. NSGO RATINGS AND MERIT FUND ALLOCATIONS First Cycle (1998/99­2002/03) In the first cycle of merit funding, the NSGO established a pool of funds in the Sea Grant budget to be allocated to individual Sea Grant programs on

148 APPENDIX E the basis of overall performance. The NSGO developed systematic proce- dures to rate each Sea Grant program for the purpose of allocating funds from this merit pool. During Cycle 2, merit fund allocations will continue to be made in a manner similar to Cycle 1, with slight modifications. Merit Funding Sea Grant programs that have reached institutional or college status are assigned to one of four merit categories. Categories 1 and 2 are reserved for programs that achieve the highest levels of performance. Category 3 denotes programs meeting performance benchmarks, while programs assigned to Category 4 have significant deficiencies. Programs assigned to categories 1, 2 and 3 qualify for merit pool allocations over the next four years. If a program fails to meet 20 percent or more of the weighted benchmarks, it will be considered as having "significant deficiencies" and assigned to Category 4. Programs assigned to Category 4 do not receive a merit pool allocation during the four-year period. The merit pool allocation consists of two parts: · A minimum allocation that is fixed for four-years (assuming level funding), and · A residual share component that is variable and may change each year depending upon the performance ratings of all programs that have been reviewed. Added together, these two components determine each program's merit funding allocation for a given year. The minimum allocation is a fixed percentage of the merit pool that a program can expect to receive over the course of the next four years. Assuming level funding of the merit pool, this amount will remain the same each year. The fixed minimum component for a program in Cat- egory 1 is calculated by dividing the total amount of funds in the merit pool by the number of programs (e.g. $3 million merit pool ÷ 30 programs = $100,000 per program in Category 1). A program in Category 2 and Category 3 receives a minimum allocation of 70 percent and 40 percent respectively of that received by a program in Category 1. In the above example of a $100,000 minimum allocation for Category 1 programs, the minimum component for programs in Category 2 and Category 3 would be $70,000 and $40,000 respectively. A program assigned Category 4, "sig- nificant deficiencies," would not receive a merit funding allocation. Once

APPENDIX E 149 the amount in the merit pool is determined, the fixed minimum compo- nent remains unchanged until the total merit pool amount changes. The residual share of merit funding depends upon the distribution of ratings across all programs for a given year. The amount of the merit pool that remains unallocated after meeting the minimum allocations, the "re- sidual" amount, is distributed to programs in Category 1 and Category 2 only. Category 1 programs gets twice as much of the residual as those in Category 2 programs. Category 3 programs do not receive a share of the residual. A new residual share is calculated every year. The NSGO Direc- tor could cap an award if the residual share exceeds 10 percent of the merit pool, although in practice this is unlikely to occur. Merit Funding · Category 1 = (Merit Pool ÷ 30) + 2 shares of residual · Category 2 = 70% of (Merit Pool ÷ 30) + 1 share of residual · Category 3 = 40% of (Merit Pool ÷ 30) + no share of residual · Category 4 = no merit funding Following the NSGO Review each year, the new ratings for the seven or eight programs replace their prior rating and the merit pool allocations are recalculated. Calculating the allocations each year assures that all programs have the same merit funding opportunities, regardless of the year they are reviewed. The distribution of the $3 million merit pool at the end of the first cycle is shown below.

150 APPENDIX E Cycle 1 150 100 K$ 50 - Programs Rating Merit Funding Merit Pool Allocation for Each Rating Distribution (# of Programs) Rating Minimum From Residual Merit Funding 15 "Category 1" $ 100,000 $ 25,714 $ 125,700 12 "Category 2" $ 70,000 $ 12,857 $ 82,900 3 "Category 3" $ 40,000 $ - $ 40,000 0 "Category 4" $ - $ - $ - 30 Total Merit Pool = $ 3,000,000 Total Minimum Allocations = $ 2,460,000 Total Category 1 $ 1,885,500 Total Residual Allocations = $ 540,000 Total Category 2 $ 994,800 Shares of Residual 42 Total Category 3 $ 120,000 Residual/Share $ 12,857 Total Merit Pool $ 3,000,300 Second Cycle (2003/04­2006/07) Sea Grant Reauthorization Legislation (2002) New provisions of the ``National Sea Grant College Program Act Amend- ments of 2002'' (Public Law 107­299) impose new requirements for evalu- ation of Sea Grant college and institutional programs. The law now re- quires the Director of the National Sea Grant College Program to rate such programs according to their relative performance into at least five categories, with each of the two best-performing categories containing at most 25 percent of the programs. In particular,

APPENDIX E 151 Section 3, Requirements Applicable to National Sea Grant Col- lege Program, states: (b) PROGRAM EVALUATION AND RATING.-- (1) EVALUATION AND RATING REQUIREMENT.--Section 204(d)(3)(A) of the National Sea Grant College Program Act (33 U.S.C. 1123(d)(3)(A)) is amended to read as follows: "(A)(I) evaluate the performance of the programs of sea grant colleges and sea grant institutes, using the priorities, guidelines, and qualifications established by the Secretary under subsection c), and determine which of the programs are the best managed and carry out the highest quality re- search, education, extension, and training activities; and "(ii) rate the programs according to their relative performance (as deter- mined under clause (I)) into no less than 5 categories, with each of the 2 best-performing categories containing no more than 25 percent of the pro- grams." Public Law 107­299 also requires the Secretary of Commerce to distribute all appropriations in excess of FY2003 levels to any combination of: (1) Sea Grant programs, according to their performance rating; (2) national stra- tegic investments; (3) Sea Grant program qualifying activities; and (4) Sea Grant colleges or institutes designated after this Act's enactment, but not yet evaluated. Section 7, Authorization of Appropriations, states: c) DISTRIBUTION OF FUNDS.--In any fiscal year in which the appropriations made under subsection (a)(1) exceed the amounts ap- propriated for fiscal year 2003 for the purposes described in such sub- section, the Secretary shall distribute any excess amounts (except amounts used for the administration of the sea grant program) to any combination of the following: "(1) sea grant programs, according to their rating under section 204(d)(3)(A); "(2) national strategic investments authorized under section 204(b)(4);

152 APPENDIX E "(3) a college, university, institution, association, or alliance for activi- ties that are necessary for it to be designated as a sea grant college or sea grant institute; "(4) a sea grant college or sea grant institute designated after the date of enactment of the National Sea Grant College Program Act Amendments of 2002 but not yet evaluated under section 204(d)(3)(A)." Merit Funding and Bonus Funding Combined In summary, the three key provisions of Public Law 107­299 that will affect the ratings and allocation of funds during the second cycle of re- views are: · The NSGO is required to rate programs according to their relative performance and assign programs into no less than five categories. · Each of the top two categories cannot contain more than 25 percent of the Sea Grant programs. · Appropriations above the FY2003 level can be allocated according to these ratings. In order to meet these requirements, the NSGO will adopt a two-tier approach to funding allocations related to performance evaluations. · The first tier, or "merit funding" tier, retains the framework of the Cycle 1 merit funding. All programs will continue to be assigned to a merit-funding category. Programs assigned a rating of Category 1 (highest), 2 or 3, based on the NSGO Review, will receive merit funding allocated similarly to the Cycle 1 allocation procedures. Categories 1 and 2 are reserved for programs that achieve the high- est levels of performance. Category 3 denotes programs that meet performance benchmarks. · The second tier, or "bonus funding" tier, will at times be used to allocate part or all of the funds appropriated in excess of the FY2003 appropriation. Bonus funding would go only to programs that are rated in Category 1 (best-performing category) and are rated among the top programs in "Category 1" (each of the 2 best-per- forming categories containing no more than 25 percent of the pro- grams). Currently, this would allow up to 14 programs to receive bonus funding, or up to seven programs in each of the two bonus funding categories. It would be expected that the NSGO would

APPENDIX E 153 maintain a 2:1 ratio in the "bonus pool" for the two "bonus" groups in Category 1. In combination, the three merit categories plus the two bonus categories give the five groups mandated by Congress: First group (1A)--Top "Category 1" programs (currently limited to 7) receive "Category 1" merit funds + the higher bonus Second group (1B)--Next "Category 1" programs (currently lim- ited to 7) receive "Category 1" merit funds + the smaller bonus Third Group (1C)--All other "Category 1" programs (no limit) re- ceive "Category 1" merit funds / no bonus Fourth Group--all "Category 2" programs receive "Category 2" merit funds / no bonus Fifth Group--all "Category 3" programs receive "Category 3" merit funds / no bonus Each program is assigned to a merit-funding category (Category 1, 2 or 3) that will not change over the four-year period. There is no interim grad- ing of programs in the "outyears". Programs are evaluated and rated once every four years through the PAT and NSGO process, and the program's rating is in effect for the full four years. The one change that may occur over time is a program's relative position in the new rating categories mandated by Congress, or in Sea Grant terminology, the two new bonus categories. Assignment to the two new bonus categories is dependent not only on a program's rating, but also on the distribution of the ratings of all pro- grams. Consequently, with respect to the bonus funding only, it is pos- sible for a Sea Grant program not being reviewed to be affected. The ratings of the seven or eight programs reviewed yearly may reorder the distribution of ratings across programs. Each year it is possible for a pro- gram not reviewed to move into or out of a bonus category (e.g., from Group 1B to 1C or vice versa) or to move up or down between the two bonus categories (e.g., from Group 1A to 1B or vice versa).

154 APPENDIX E Each year, the NSGO Director may add funds to the merit pool and/or the bonus pool in response to actual appropriations. In the future, were appropriations to increase substantially, increases in merit funding would be a primary mechanism for maintaining and enhancing Sea Grant's en- abling infrastructure. Currently, as many as 14 programs in Category 1 would receive bonus funding in addition to merit, in any given year. However, because Congress limits the number of bonus programs, the actual allocations would depend on the distribution of the merit ratings and the number of "Category 1" programs. As such, the merit pool would be expected to be larger relative to the bonus pool. The NSGO expects to provide preliminary notice to programs of the next fiscal year's merit funding and changes in bonus categories, if any, fol- lowing finalization of the NSGO Review. A funding letter will be pre- pared that will go to each Sea Grant program to indicate the dollar level of the merit funding allocation a program will receive in the following fiscal year, assuming level funding. The letter will also indicate whether a program's rating makes it eligible for either of the two bonus categories, along with a preliminary estimate of bonus funding for the following fiscal year, again assuming level funding. Normally, this letter will be sent 10 months or more ahead of the anniversary dates for renewal of omnibus grant awards. The hypothetical example below shows how a $3 million merit funding pool plus a $1 million bonus funding pool might be allocated assuming 20 programs have been rated in "Category 1". Cycle 2 Merit and Bonus Funding Merit Fund Bonus 250 200 150 K $ 100 50 0 Programs

APPENDIX E 155 It should be noted that it is possible that a particular group would not have any programs assigned to it. For example, if there were 14 or fewer Category 1 programs, the Third Group (1C) would have no programs assigned to it. Programs with Significant Deficiencies: The major goal of the evalua- tion process is to help programs improve. If as a result of the NSGO Review, a program is determined to have a significant number of defi- cient program elements (fails to meet 20 percent or more of the weighted benchmarks), the program will be assigned to the "Significant Deficien- cies" (Category 4) and would not be eligible to receive merit funds over the next four-year cycle. While occurring very rarely, if the NSGO Review determines that a pro- gram should receive a rating of "Significant Deficiencies," a corrective action plan will be required to address all the deficient elements. The action plan identifies any changes in goals, organization, procedures, planning, and operations that need to be implemented to correct the defi- ciency. The action plan is a joint effort of the Sea Grant institution and the NSGO. The plan should be in place within six months of notification to the Sea Grant program of the "Significant Deficiencies" (Category 4) rat- ing and the need of corrective action. Failure to fully implement a correc- tive action plan and to show significant improvement by the two-year mid-cycle mark, as determined by an assessment team, could result in a program having its core funding reduced or decertification of Sea Grant college or institutional status. In addition, the NSGO Director may also require a program rated in categories 1, 2 or 3 to submit a corrective action plan for a particular area of the program. If the NSGO Review finds that a program fails to meet the benchmark for a sub-element, the requirement for a partial corrective action plan will be identified in the NSGO Report.

Next: F A Multivariate Analysis of Potential Biases in the Final Evaluation Scores »
Evaluation of the Sea Grant Program Review Process Get This Book
×
Buy Paperback | $63.00
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF
  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!