in measurement, data collection, analysis, and the drawing of conclusions (Shadish et al., 2001). Reliability means that different assessors would reach similar conclusions on the basis of the evaluation methods used.
The context of the MMRS program presents some special challenges in terms of evaluation of the program. First, the MMRS program involves a web of planning activities, resources, intergovernmental agreements, and exercises at multiple levels of government. This web of activities is seen in Figure 5-1. Any one of a number of policy instruments, development activities, emergency capacity functions, and follow-up activities might be evaluated, or sets of them might be evaluated. Second, any MMRS itself represents an effort to coordinate multiple entities and activities that are independently funded and that receive their authority from other sources. Third, evaluation of the MMRS program is inferential, because even after September 11, 2001, incidents of domestic terrorism have occurred in only a few cities, and so the adequacies of most MMRSs have never been tested directly. Fourth, evaluation of the MMRS program is also inferential because, of necessity, assumptions must be made about how the component parts should work together.
Evaluation can focus on a variety of entities and questions, such as the following:
Inputs. Inputs are an individual city’s resources, personnel, and political and logistic agreements committed on behalf of the MMRS.
Processes. Also known as implementation, processes would include the variety of activities designed to achieve a specific level of capacity to detect an attack, to deal with the crisis phase, and to manage the aftermath. Such activities lead to the intermediate results required to achieve preparedness. These might include, for example, growth in decision makers’ knowledge and experience with the variety of events in question, training programs under way in various units, designation and assumption of responsibilities, purchase of necessary equipment and supplies, and periodic testing of communications.
Outputs. Because true terrorist attacks are, fortunately, still rare, an assessment of the ultimate outcome of the MMRS is not likely to be available for many cities. Instead, intermediate outcomes, referred to hereafter as outputs, are more feasible and are represented by progress of various elements of the system in response to exercises, false alarms, and nonterrorism events. Immediate outputs might include, for example, the number or percentage of personnel passing specialized tests on chemical or biological weapons.