Click for next page ( 130


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 129
Appendix D National Sea Grant Program Memorandum on NSGO Final Evaluation and Merit Funding, April 22, 1999 [Stamped Date on Original: April 22, 1999] MEMORANDUM FOR: Sea Grant Directors FROM: Ronald C. Baird Director SUBJECT: Policy Memorandum on NSGO Final Evaluation and Merit Funding Introduction The National Sea Grant Office (NSGO) guidelines set forth in the policy document of April 20, 1998, "Implementation of Program Evaluation Pro- cedures in the National Sea Grant College Program," require that the NSGO submit to each institution responsible for administering a Sea Grant Program, a Final Evaluation and Recommendation Report that summa- rizes the findings of a performance review for that Sea Grant Program over a four-year program cycle. The primary objectives of the Report are to provide local management with an assessment of performance and specific recommendations directed toward performance improvement and maintenance of existing program strengths. This memorandum es- 129

OCR for page 129
130 APPENDIX D tablishes the procedures of the NSGO in the conduct of the review and preparation of the report. In addition to the Report, and in accordance with the policy document of April 20, 1998 the NSGO is instructed to provide a performance rating to each Sea Grant program being evaluated. The rating is the basis for allo- cating merit-based funding from a pool of funds set aside in the Sea Grant budget for that purpose. This memorandum also establishes procedures and mechanics for rating determination and the allocation of merit fund- ing. NSGO Review Process General: The NSGO will conduct the final review of a Sea Grant program within one year following the Program Assessment Team (PAT) site visit. All programs are evaluated to the extent possible in a similar manner and against common performance benchmarks. It must be recognized, how- ever, that improvements in methodology in evaluating performance can be expected to be incorporated into the process over time. The final evaluation review will be conducted by the NSGO technical staff and one or more members of the Executive Committee of the Na- tional Sea Grant Review Panel. The review is expected to be completed over a one week period during February/March of each year. All pro- grams having had PAT visits in the previous calendar year will be consid- ered as a group so as to maintain a consistent four-year review cycle. For example, the NSGO review session for 1999 occurred from February 11-19 and evaluations of the eight programs having PAT reviews in calendar year 1998 were included. After each program has been evaluated, the NSGO will prepare the report that will contain the principal findings of the review and recommenda- tions for enhancement of program performance. The report and a sum- mary cover letter from the director of the NSGO will be sent to the direc- tor of each program being evaluated. The covering letter and report will also be copied to the appropriate administrators of the institution with administrative responsibility for that Sea Grant program. The report will normally be submitted prior to the end of April in each calendar year. The following sections explain in greater detail the materials used and the structure of the review process.

OCR for page 129
APPENDIX D 131 Reports and Information Considered: In preparation for the final re- view, the NSGO Program Officer prepares for distribution materials that will be considered in the review. The compendium of documents and reports made available to NSGO staff for study prior to the review are listed below. Some of these materials are not generally available to the PAT and represent additional information for the NSGO to use in the evaluation process. The PAT report along with the institution's response The program's strategic plan Pertinent descriptive material from implementation plans/omni- bus proposals etc. Annual progress reports Information on major accomplishments Trip and peer review panel reports by the Program Officer Topical Assessment Team reports (if any) Sea Grant funding information Other material deemed to be relevant by the Program Officer Review Structure: The director of the NSGO facilitates the review ses- sions. Each session is limited to a discussion of one program only. Thus each program is reviewed on its own merits and not in competition with other programs. The NSGO Program Officer provides a brief overview of the program's performance over the last four years. Information is pre- sented about the Sea Grant program's management structure and institu- tional setting, its outside advisory and strategic planning process, the programmatic areas of emphasis, major program accomplishments and outstanding issues or concerns. The NSGO final review attempts to amplify and add to the information base produced in connection with the PAT site visit. The evaluation re- port prepared by the PAT however provides a significant input into the NSGO's final evaluation. The program's response to the PAT report and any recent information received about corrective actions are also consid- ered. Evaluation-related materials on file in the NSGO, including progress reports or the response to recommendations from previous reviews, are weighed. Participation in national initiatives and responsiveness to net- work-wide activities are also considered. These bear on successful perfor- mance, and usually the NSGO has a better perspective here than the PAT. Each of the four major evaluation criteria is discussed in sequence includ- ing the PAT findings. All PAT recommendations are reviewed. Those

OCR for page 129
132 APPENDIX D deemed most critical are highlighted for inclusion in the report. Where appropriate, PAT recommendations may be modified and additional rec- ommendations developed based on the NSGO review. During the review process "Best Management Practices" are identified. These will be sum- marized for later promulgation to the Sea Grant network. The discussions and findings from the review form the basis for the report that is prepared under the direction of the director NSGO. Merit Fund Ratings and Allocations General: In accordance with NSGO guidelines, there has been established in the Sea Grant budget a pool of funds to be allocated to individual Sea Grant programs on the basis of overall performance. It is the responsibil- ity of the NSGO to develop a system to rate each Sea Grant program for the purpose of allocating funds from this merit pool. The NSGO is also charged with providing a rationale and mechanism for the disbursement of such funds in accordance with the rating system while considering both variations in the amount of funding available to the pool as well as changes in the distribution of program ratings that can occur each year. The rationale and protocols for both the rating system and allocation of merit funding are presented in the next section. Merit Categories and Decision Protocols: The evaluation of program performance involves the use of judgment in weighing the qualitative and quantitative evidence available to the NSGO. It is also recognized that better metrics for the measurement of performance will be developed over time and incorporation of these must be provided for in the rating scheme. For the purpose of merit funding, the NSGO believes it is neither possible nor advisable to create a rating system that attempts to deter- mine fine scale differences in performance among structurally dissimilar programs. Consequently, the system developed consists of only four rat- ing categories of satisfactory or better performance and one default cat- egory for unsatisfactory performance, assignment to which involves sub- stantial corrective action and could lead to reductions in future funding allocations. After the evaluation process, programs are assigned to one of the four rating categories. Ratings are based on grading of the same four criteria as the PAT evaluations and carry the same weighting percentages. The PAT assessments are significant determinants in arriving at an overall final merit rating even though the NSGO categories are not directly compa- rable to those of the PAT. For instance, programs that did not carry a PAT

OCR for page 129
APPENDIX D 133 grade of "Excellent" for the criteria "Producing Significant Results" were not rated in the highest performance categories (1 and 2) by the NSGO in the 1999 evaluations. Final program ratings then reflect a consensus from both the NSGO and outside PAT based on a collective assessment of how well a program has performed in relation to the evaluation criteria over the last four years. Categories 1 and 2 are reserved for programs that achieve the highest levels of performance. Category 3 denotes programs meeting performance benchmarks, while programs assigned to Category 4 have significant de- ficiencies. Merit Pool Categories: Programs assigned to categories 1, 2, and 3 qualify for merit pool allocations over the next four years. Programs assigned to Category 4 will not receive a merit pool allocation during the four-year period. The merit pool allocation will consist of two parts, a minimum allocation that is fixed and available annually for each of four years, and a residual share component that is variable and may change each year de- pending upon the performance ratings of all programs that have been reviewed. Added together, these two components--the minimum alloca- tion and the residual share--determine each program's merit funding allocation for a given year. The merit pool allocations are set up in this manner so that all programs will have the same merit funding opportuni- ties, regardless of the year they are reviewed. Minimum Allocation: The minimum allocation is a fixed percentage of the merit pool that a program can expect to receive over the course of the next four years. Assuming level funding in the pool this will be a constant amount annually. The fixed minimum component for a program in Cat- egory 1 is calculated by dividing the merit pool by the number of pro- grams that have been reviewed (8 in 1999, 29 after four years). A program in Category 2 and Category 3 would receive a minimum allocation of 70 percent and 40 percent respectively of that received by a program in Category 1. For example, if the total merit pool for 29 programs is $2,900,000, the minimum component would be $100,000, $70,000, and $40,000 for a program in category 1, 2, and 3 respectively. A program in Category 4 would not receive an allocation. Once the amount in the merit pool is determined, the fixed minimum component remains unchanged until the total merit pool amount changes. Such a change could result from changes in the Sea Grant program's appropriation or a change in budget strategy. Our goal is to eventually have up to 10 percent of the core funding allocated through the merit

OCR for page 129
134 APPENDIX D pool. In 1999 the funds used for regional and multiprogram activities, $2,900,000 of which are being phased into the merit pool, amount to about 7 percent of core funding. Note also that budget considerations might require reductions in the funding of the merit pool sometime in the fu- ture. Residual Share: The residual share is a variable component that depends upon how much of the merit pool remains unallocated after covering the fixed minimum allocations. It also depends on the distribution of ratings across all programs. The amount that remains after meeting the minimum allocations, the _residual_ amount, is distributed to programs in Category 1 and Category 2 only. Category 1 gets twice as much as Category 2. A new residual share is calculated every year. The Director of the NSGO could cap an award if the residual share exceeds 10 percent of the merit pool, although in practice this is unlikely to occur. Program Merit Pool Allocations: The fixed minimum allocation and the residual share are added together to determine the merit pool allocations for each program. In 1999, only eight programs had been reviewed and these will receive the merit funding allocations in FY2000. By the end of four years, all programs will have been evaluated and the review process cycle will have reached steady state. The attached charts show the details of the calculations and several hypothetical examples of how merit pool allocations are likely to change from year-to-year due to the variable re- sidual share component. Restrictions on Allocated Funds: Merit fund allocations are considered augmentations to a program's core funding level and are subject only to the normal terms and conditions that apply to all funds used in support- ing a program's core activities.

OCR for page 129
APPENDIX D 135

OCR for page 129
136 APPENDIX D

OCR for page 129
APPENDIX D 137

OCR for page 129
138 APPENDIX D Year 5 and after: (29 Programs) Year 5 and after: (29 Programs) Starting in year five, after all 29 programs have been phased in, this pro- cess continues in a similar manner. The overall merit rating distribution for 29 programs in year five is determined by replacing the year-one ratings of the first group of programs with their newer ratings set in year five. In year six, the ratings of the second group of programs from year two are replaced by their newer ratings set in year six, and so on for successive years.