informed monitoring can minimize problems associated with the implementation of an impact evaluation.

STEPS THAT CAN BE TAKEN PRIOR TO AWARDING THE EVALUATION CONTRACT

Developing an Effective Request for Proposals (RFP)

As noted in Chapter 2, an initial step for ensuring a high-quality evaluation is a well-developed account of the questions that need to be answered and the form such answers should take to be useful to the intended audience. These considerations, in turn, have rather direct implications for the design and implementation of an impact evaluation. The usual vehicle for translating this critical background information into guidelines and expectations for the evaluation design and implementation is a Request for Proposal (RFP) circulated to potential evaluators. An RFP that is based on solid information about the nature and circumstances of the program to be evaluated should encourage prospective evaluators to plan for the likely implementation problems. For instance, a thorough RFP might prompt the applicant to provide (a) a power analysis to support the proposed number of cases; (b) evidence that supports the claim that a sufficient number of cases will be available (e.g., pilot study results or analysis of agency data showing that the number of cases that fit the selection criteria were available in a recent period); (c) a carefully considered plan for actually obtaining the necessary number of cases; and (d) a management plan for overseeing and correcting, if necessary, the process of recruitment of cases for the study.

When such background information is not provided in the RFP, it will fall to the evaluation contractor to discover it and adapt the evaluation plans accordingly. In such circumstances, the RFP and the terms of the evaluation contract must allow such flexibility. In addition, consideration must be given to the possibility that the discovery process will reveal circumstances that make successful implementation of the evaluation unlikely. Where there is significant uncertainty about the feasibility of an impact evaluation, a two-step contracting process would be advisable, with the first step focusing on developing background information and formulating the evaluation plan and the second step, if warranted, being the implementation of that plan and completion of the evaluation.

Funding agencies and evaluators have used a number of approaches to developing the information needed to formulate an instructive RFP or planning the evaluation directly. Site visits, for example, are one common way to assess whether essential resources such as space, equipment, and staff will be available to the evaluation project and to ensure that key local



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement