In response to the third question to the panel on evaluating SARP, the committee concludes that a standard “textbook” evaluation is not appropriate for SARP because of the small size of SARP; the expectation that desired outcomes will take a considerable period of sustained effort to achieve; the multiple types and levels of decisions that can be influenced by climate information; the variety of relevant decision makers; and the multiplicity of programmatic approaches to shape decision support systems.
Because standard evaluation approaches are not appropriate for the Sectoral Applications Research Program, we recommend that evaluation questions for the Sectoral Applications Research Program be addressed by a monitoring program.
Such a monitoring approach would aim at recording and analyzing trends in metrics that are appropriate for each type of SARP activity: pilot projects, workshops, and use-inspired research. It would employ multiple metrics, some of them recording processes in SARP and some tapping outputs and outcomes. Monitoring should rely wherever possible on data that can be reliably collected without substantial time and resources. Representatives from target audiences should contribute to decisions regarding the details of data collection and surveys that could be most useful for monitoring SARP performance.