external evaluation teams in tracking the development and implementation of the program from its earliest stages. The program should include a vigorous formative evaluation, which should be carried out competitively by teams from academia or appropriate nonprofit or for-profit evaluation firms. The evaluation teams should be managed by the OIP and report directly to the director of the NSF and to the National Science Board.
In the program’s early years it would be too soon to attempt a summative evaluation, that is, to evaluate whether the program accomplished its ultimate goals or to assess its contribution to the nation. Instead, a formative evaluation strategy would focus on measures that would provide insight into whether the program’s anticipated short-term impacts were being achieved on a continuing basis, and which features of the prize contests are more or less successful.
Among the relevant indicators of positive near-term program impact could be
whether the contests attract large numbers of contestants;
whether those contestants are a more diverse group than the traditional NSF constituency;
whether, to what extent, and from what sources private funds are forthcoming to support the research activities of contestants or even to augment the prize purses;
whether prize rules and processes functioned well;
whether the contestants make related innovations that are successfully spun off from the main prizeseeking activity;
and whether the program significantly advances innovation in the area of the prize topics.
Assessments could also be made of whether public awareness of and interest in innovation or the impacts of a specific prize are enhanced by the prize program, and whether NSF’s public image is affected by its sponsorship of the program. Among the relevant program design variables to be examined are
the duration of the contest, the size of the prize purse,
whether the contests are first-past-the post or best-in-class,
whether a contest is administered by NSF or by a contractor,
and whether a contest includes prizes for achievement of intermediate objectives.
Evaluation teams could also examine whether NSF’s administration of the program is effective as measured by accomplishment of NSF’s own stated goals for the program.