Evaluation of the Sea Grant Program Review Process (2006) / Chapter Skim
Currently Skimming:
F A Multivariate Analysis of Potential Biases in the Final Evaluation Scores
F A Multivariate Analysis of Potential Biases in the Final Evaluation Scores
Pages 157-164
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 157... ...
and FE scores provide accurate assessments of program quality according to the assessment criteria, but might be subject to random errors associated with differences between Cycle 1 and Cycle 2, the number of years that particular NSGO program officers are associated with: particular Sea Grant programs; program seniority; the size of state and federal budget allocations awarded to programs; the within cycle order of review of programs; and the number of years that particular program officers have served as program officers. The general linear model that was estimated can be represented by: Cyclej, PO Continuityij, Program Maturityij, State Budgetij, FEij = f Federal Budgetij, Order of Reviewi, PO Seniorityij where Cyclej is a binary variable used to differentiate between scores awarded in Cycle 1 and Cycle 2; PO Continuity is the number of years that a particular NSGO program officer is assigned to the ith individual Sea 157
|
From page 158... ...
The initial model coefficient estimates are: Standard Coefficients Error P-value Intercept 2.723 0.608 0.000 Cycle Dummy 0.068 0.156 0.667 PO Continuity -0.087 0.038 0.029 Program Maturity -0.040 0.019 0.046 State Budget 1.17E-07 2.32E-07 0.617 Federal Budget 7.68E-08 1.11E-07 0.493 Prog Reviewed in Year 1 0.049 0.186 0.793 Prog Reviewed in Year 2 0.088 0.162 0.591 PO Experience < or = 1 year -0.277 0.411 0.505 PO Experience 2 to 3 years 0.093 0.212 0.664 PO Experience 4 to 10 years -0.117 0.146 0.428 The structure of the model can be viewed as an attempt to explain variations in FE scores for the individual programs using information or proxy information for potential sources of bias that were suggested by the individual Sea Grant program directors. Thus, if the model were to provide accurate predictions of the FE scores, there would be evidence to support the concerns of the individual Sea Grant program directors.
|
From page 159... ...
Because the probability of observing an estimate of -0.087 if the true value of this coefficient were greater that or equal to zero is 0.014, the null hypothesis can be rejected. That is, there is statistical support for the assertion that individual Sea Grant programs with long-term relationships with their program officers scored lower than programs with less program officer continuity.
|
From page 160... ...
160 APPENDIX F hypothesis can be rejected; there is statistical support for the assertion that mature programs are scored lower than newer programs. The coefficients associated with the magnitude of state and federal budgets allocated to the individual Sea Grant programs indicate that programs with larger budgets earn higher scores, but the effect is miniscule: a $1 increase in the individual program's state budget is associated with an increase of 1.17E-07 in the score, and a $1 increase in the individual program's federal budget is associated with an increase of 7.68E-08 in the score.
|
From page 161... ...
That is, there is again statistical support for the assertion that individual Sea Grant programs that have enjoyed long term relationships with their program officers scored lower (better) than programs with less program officer continuity.
|
From page 162... ...
Again, because public testimony suggested that the scores would be lower for Sea Grant Colleges and Institutes that enjoyed longer working relationships with their program officers, the null (no effect) hypothesis is that this coefficient is not significantly greater than zero.
|
From page 163... ...
The average difference between Category 1B and Category 1C is of a similar magnitude. Thus for two otherwise identical individual Sea Grant programs that deserve to be rated in Category 1A -- one with a new program officer and one with a program officer who has been with an individual Sea Grant program for 4 years -- the program with the new officer would be expected to score 0.307 points higher (worse)
|
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.