Cover Image

Not for Sale



View/Hide Left Panel
Click for next page ( 48


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 47
Guidelines for Transportation Emergency Training Exercises ference typically is conducted to ensure that MSEL implementers developed by the exercise planners do the following (after which time the final MSEL is published): Reflect the intent of the event as described in the MSEL items, Accomplish intended results, Include the correct action and information addressees, and Provide a time sequence consistent with the flow of other exercise events. Objectives are perhaps the most Final Planning Conference: A forum to review the processes and procedures for conducting the exer- important cise, final drafts of all exercise materials, and all logistical requirements. There should be no major element of the changes made either to the design or scope of the exercise or to any supporting documentation. exercise planning process. Sample agendas and other materials for supporting the planning conferences for operations-based Subsequent exercises are available on the G&T Secure Portal. HSEEP Volume IV describes these materials and evaluations and is available at http://www.ojp.usdoj.gov/odp/docs/HSEEPv4.pdf. recommended improvements A sample IPC package is available from the United States Marine Corps at http://www.certip.org/ will be based on policies/6-14meeting.html. them. The Alaska Department of Homeland Security has prepared an extensive guide for the development of both discussion-based and operations-based exercises. This guide provides useful tips and recom- mendations for organizing and conducting planning conferences. The guide supports compliance with the G&T HSEEP and is available at http://www.ak-prepared.com/homelandsecurity/exercise/full-scale/ documents/ODP%20DesksideVol%20I.doc. DESIGN AND DEVELOPMENT Building on the exercise foundation, the design and development process should focus on identify- ing objectives, designing the scenario, creating documentation, developing policies, planning exer- cise conduct, and selecting an evaluation methodology. OBJECTIVES Exercise objectives are the foundation of design and development. Exercise objectives define specific goals, provide a framework for the development of the scenario, guide development of individual orga- nizational objectives, and provide evaluation focus for the exercise. Generally, the number of exercise objectives will be limited by planners to Enable timely execution of the exercise, Facilitate design of a reasonable scenario, and Adequately support the successful completion of exercise goals. Objectives are initially prepared during concept development. More complex exercises typically have both major and supporting objectives. A major objective contributes to development of general events to present to participants for action. Supporting objectives help planners to ensure that all participants receive an adequate opportunity to assess specific capabilities. The performance addressed by the objective should have observable and measurable indicators to aid in identifying evaluation criteria. FEMA's SMART System, shown as Figure 10, is a good checklist for ensuring the completeness and accuracy of objectives. 47

OCR for page 47
Guidelines for Transportation Emergency Training Exercises A good objective is simply and clearly phrased. It is brief and easy to Simple understand. The objective sets the level of performance so that results are observable and Measurable you can tell when an objective has been reached. The objective is not too tough to achieve. It is reasonable in its commitment of Achievable resources. Realistic An objective is not only achievable, but also realistic for the exercise. Task Oriented The objective focuses on a behavior or a procedure. With respect to exercise design, each objective should focus on an individual emergency function. Many examples of scenarios are FIGURE 10 SMART SYSTEM CHECKLIST available. Transportation agencies should check with their SCENARIO local public safety and emergency A scenario provides the backdrop and storyline that drive an exercise. The first step in designing the management scenario is determining the type of threat/hazard (e.g., chemical, biological, radiological, nuclear, explo- agencies for sive, cyber, or other). Thought should be given to creating a scenario that involves local incidents and templates and local facilities and is based on exercise objectives derived from risk and vulnerability assessments con- examples tailored ducted at the transportation agency. Each type of hazard presents its own strengths and weaknesses to their local for evaluating different aspects of prevention, response, and recovery and is applicable to different exer- communities. cise objectives. The next step is to determine the venue (i.e., the facility or site) that the scenario will affect. Venue selection should be based on the type of hazard used. For example, if a nonpersistent chemical agent (e.g., sarin) is selected, the venue should not be an open-air facility (e.g., outdoor station) because of the agent's dissipating characteristics. Table 9 provides information on the characteristics of a good scenario. The Wisconsin Department of Emergency Management has released a set of 39 scenarios for use by local responders. This document provides an overview to Wisconsin's recommended program for TABLE 9 WHAT DOES A GOOD SCENARIO LOOK LIKE? Characteristic Description Appropriate The scenario must fit the Exercise objectives (for both the transportation organization and participating external agencies) Targeted incident location, in terms of geography and logical functioning of the location during transportation operations Emergency response organization's makeup and capabilities Realistic The scenario must be plausible in terms of Design-basis coherence Threat and vulnerability assessment performed by the transportation agency Credible conditions and environmental challenges Robust The scenario must be broad enough to Support all the envisioned exercise activities Provide the opportunity for all organizations to meet their objectives Engaging The scenario must Move participants to act with a high degree of involvement Be challenging and raise responders' adrenaline level 48

OCR for page 47
Guidelines for Transportation Emergency Training Exercises conducting tabletops. It is available at http://emergencymanagement.wi.gov/docview.asp?docid= 738&locid=%2018. The Environmental Protection Agency has also made available a set of scenarios simulating radiological emergencies involving nuclear power plants, Department of Energy weapons and waste storage facili- ties, and military sites. These materials can be found at http://www.epa.gov/radiation/rert/exercises.htm. Michigan State University also offers a resource page to support the development of terrorism-based scenarios at http://www.cj.msu.edu/outreach/wmd/moduleresources.htm. Sample scenarios are also available to support operations-based exercises on the G&T Secure Portal. HSEEP Volume IV provides a description of these materials and is available at http://www.ojp.usdoj.gov/ odp/docs/HSEEPv4.pdf. DOCUMENTATION The exercise plan The list below briefly describes typical products for operations-based exercises. Documentation mate- resembles the rials such as meeting minutes, presentations, agendas, and news releases have been omitted because situation manual these documents typically are created while developing the primary products. in how it is applied during an The exercise plan (EXPLAN), typically used for operations-based exercises, provides an exercise operations-based synopsis and is published and distributed prior to the start of the exercise. In addition to addressing exercise. exercise objectives and scope, the EXPLAN assigns tasks and responsibilities for successful exer- cise execution. The EXPLAN should not contain detailed scenario information, such as the hazard to be employed. This document is generally intended for exercise players and observers. A sample exercise plan template is available from FEMA at http://training.fema.gov/emiweb/downloads/ EXERCISE%20%20PLAN1.doc. Sample exercise plans to support a variety of operations-based exercises are available on the G&T Secure Portal. HSEEP Volume IV provides a description of these materials and is available at http:// www.ojp.usdoj.gov/odp/docs/HSEEPv4.pdf. The controller and evaluator (C/E) handbook supplements the EXPLAN, containing more detailed information about the exercise scenario and describing exercise controllers' and evaluators' roles and responsibilities. Because the C/E handbook contains information on the scenario and exercise admin- istration, it should be distributed only to individuals specifically designated as controllers or evaluators. The C/E handbook is generally used on smaller or limited-scope exercises of short duration in lieu of control staff instructions (COSIN) and an EVALPLAN. Its specifics may include the following: Roles and responsibilities of functional or individual controllers; Evaluation aids and checklists; A schedule for training, site set-up, exercise conduct, and critiques; An exercise safety plan; and A controller communications plan. Larger, more complex exercises may use COSIN and an EVALPLAN in place of, or in addition to, the C/E handbook. A sample C/E handbook to support a variety of operations-based exercises is available on the G&T Secure Portal. HSEEP Volume IV provides a description of these materials and is available at http://www.ojp.usdoj.gov/odp/docs/HSEEPv4.pdf. 49

OCR for page 47
Guidelines for Transportation Emergency Training Exercises Controllers and Control staff instructions (COSIN) contain the guidance that exercise controllers, simulators, and evaluators need evaluators need concerning procedures and responsibilities for exercise control, simulation, and sup- handbooks and port. The purpose of COSIN is to detail the scenario for the duration of the exercise, develop guide- instructions to lines for control and simulation support of the exercise, explain the exercise concept as it relates to direct their controllers and simulators; and establish and define the control structures, communications, logistics, activities and administration. Its level of detail will vary and can include the following: during an operations-based Exercise overview, exercise. Exercise control organization and scheme, Controller roles and responsibilities, Control communications plan, VIP/observer management plan, List of key exercise events, Short MSEL and long MSEL, and Exercise safety plan. A template for a COSIN plan is available from FEMA at http://training.fema.gov/emiweb/downloads/ CONPLAN1.DOC. Sample COSIN to support a variety of operations-based exercises are available on the G&T Secure Portal. HSEEP Volume IV provides a description of these materials and is available at http://www. ojp.usdoj.gov/odp/docs/HSEEPv4.pdf. The master scenario events list (MSEL) is a chronological timeline of expected actions and scripted events to be injected into exercise play by controllers to generate or prompt player activity. Preparing the MSEL is probably the most complicated activity required for operations-based exercises. However, this activity enables the exercise planning team to identify expected actions and to document when and how the actions should be performed, based on existing plans, procedures, and training. Use of an MSEL ensures that necessary events happen so that all objectives are met. An MSEL contains a chrono- logical list of the events that drive exercise play. The MSEL links simulation to action, enhances the exer- cise experience for players, and reflects an incident or activity that will prompt players to implement the policy or procedure being tested. A sample MSEL format is presented in Table 10. In accordance with the recommendations in Table 7, each MSEL record identifies the Designated scenario time; Event synopsis; Controller responsible for delivering inject, with C/E special instructions (if applicable); Expected action (i.e., the player response expected after an MSEL inject is delivered); Intended player (i.e., the agency or individual player for whom the MSEL inject is intended); Objective to be demonstrated (if applicable); and Notes (for controllers and evaluators to track actual events against those listed in the MSEL, with special instructions for individual controllers and evaluators). 50

OCR for page 47
Guidelines for Transportation Emergency Training Exercises TABLE 10 MSEL SAMPLE FORMAT Time Message Message Summary Expected Response (in No. minutes) -90 1 From senior controller to Exercise window is opened. transportation supervisor: Permission to open exercise window. -60 2 From senior controller to lead Event scene is isolated, and safety safety and security controllers: and security controllers begin to make Implement security and safety final check of simulations. plans. -10 3 From senior controller to all Equipment is checked and time is evaluators and controllers: noted. Communication check and time check. 00 4 From senior controller to event Exercise start: scene safety controller: Start Fire department receives fire smoke generator. alarm. Fire department receives 911 5 From senior controller to event call. scene lead controller: Sound fire Transportation control center and alarm. medical department monitor 911 call. 6 From senior controller to employee actor #1: Make 911 call "This is an exercise. We have just had an explosion at the store's loading dock. The fire alarm is sounding. I see at least three injured persons. This is an exercise." Times listed in an MSEL should reflect the time at which an inject should occur. These times should be as realistic as possible and should be based on input from functional area representatives. For exam- ple, to determine when triage and treatment should be established during the exercise, solicit input from emergency medical services (EMS) or a hospital representative. If the activity occurs sooner than antic- ipated, the time should be noted but play should not be interrupted. There are three types of injects: Contextual injects are introduced to a player by a controller to help build the contemporary oper- ating environment. For example, if the exercise objectives include information sharing, an MSEL inject can be developed to direct a controller to select an actor to portray a suspect. The inject could then instruct the controller to prompt another actor to approach a law enforcement officer and inform him or her that this person was behaving suspiciously. Expected action events are expected actions that would normally take place during this type of incident. For example, during an FSE involving a chemical agent, establishment of decon- tamination is an expected action. Contingency injects are events that should be verbally indicated to a player by a controller if the player does not discover them. For example, if a simulated secondary device is placed at an incident scene but is not discovered, a controller may want to prompt an actor to approach a player and say that he or she witnessed suspicious activity close to the device location. This should prompt the discovery of the device by the player and result in subsequent notification of law enforcement (specifically, the bomb squad). MSELs are typically produced in two formats: short and long. Short MSELs list the inject, the time, a short description, the responsible controller, and a player. These MSELs can be used as a quick 51

OCR for page 47
Guidelines for Transportation Emergency Training Exercises reference guide during exercise play. Long MSELs are used when greater detail is necessary. They include more detailed descriptions, exact quotes for injects by simulation cells (i.e., by groups acting out an element of the exercise, or SIMCELL), and descriptions of expected actions. Message injects are typically used in exercises that involve multiple simulated activities. These mes- sages are typically delivered via a SIMCELL and are used to simulate the actions, activities, and con- versations of an individual, agency, or organization that is not participating in the exercise but that would likely be actively involved during a real event. For example, in an exercise with limited scope, the state governor's office may not be playing. To simulate the activities of the governor's office during an emer- gency event, a message can be scripted to simulate notification of the mayor by the governor. That mes- sage can be delivered by phone through the SIMCELL. This script or message inject should be read by a simulator acting on behalf of the governor's office. The Department of Energy's Oak Ridge National Laboratory offers an extensive online training pro- gram for developing scenarios and MSELs. These materials, which include interactive forms and tables, can be accessed at http://www.orau.gov/emi/wbt/default.htm. As with the G&T Secure Portal, users must register to access these resources. Additional information on this registration process can be obtained by calling Oak Ridge at (865) 576-2007. Sample MSELs, supporting scenarios, and evaluation measures are also available on the G&T Secure Portal. HSEEP Volume IV provides a description of these materials and is available at http://www.ojp.usdoj.gov/odp/docs/HSEEPv4.pdf. Evaluation plans (EVALPLANs) provide evaluation staff with guidance and instructions on evalua- tion or observation methodology to be used as well as essential materials required to execute their specific functions. The EVALPLAN is a limited distribution document that evaluators use in conjunc- tion with the EXPLAN and the MSEL. Level of detail varies and can include the following: Exercise overview, Evaluation control organization, Evaluation methodology and observation techniques, Evaluator roles and responsibilities, and Evaluation communications plan. FEMA provides guidelines for the selection of evaluation methodologies at http://training.fema.gov/ emiweb/downloads/HMEEM%20R-VI%20UPDATE%20JULY%202000.DOC. FEMA provides a sample EVELPLAN that takes the user through all steps of the process. It is available at http://training.fema.gov/emiweb/downloads/evalplan.doc. Sample evaluation plans to support a variety of operations-based exercises are available on the G&T Secure Portal. HSEEP Volume IV provides a description of these materials and is available at http://www.ojp.usdoj.gov/odp/docs/HSEEPv4.pdf. POLICIES Exercise policies are developed to provide guidance or parameters of acceptable practices for designing, developing, conducting, and evaluating exercises. Policies are designed to prevent, or at least mitigate the impact of, an action that may cause bodily harm to participants, destruction of prop- erty, or embarrassment to the participants or affected community. State and local transportation agen- 52

OCR for page 47
Guidelines for Transportation Emergency Training Exercises cies should develop policies appropriate to the type of exercise that address safety, media, cancel- lation, and weather. A final activity that must occur during the development cycle is preparation of the protocols used to govern the exercise. Table 11 provides a sample list of protocols for responders, controllers, evalu- ators, and observers/VIPs. TABLE 11 EXERCISE PROTOCOLS Role Rules Responders Move participants to act with a high degree of involvement. Monitor your actions. Ensure you keep yourself safe. Follow all instructions from controllers unless the instructions place you in danger. Verbalize your thought processes. The evaluators are not mind readers, and you want credit for what you have accomplished. Speak loudly so evaluators can hear what you are saying. If you have questions on what you are observing or information that was provided, ask the controller. Evaluators are directed not to speak to you. Always state "This is an exercise" when making radio and telephone calls. If nonresponders get in your way, tell the controller. Participate in the postexercise critique. Ensure that you sign in so that you get credit for your participation. Controllers Ensure exercise safety--this is your primary duty. Do not give clues to responders. For example, "Who did you notify?" is a better question than "Did you notify the state?" The latter question tells the responder that the state has to be notified. Be responsible for suspending, restarting, and terminating. Follow the instructions from the senior controller. Inform the evaluator when you issue a contingency message. Ensure that all responders sign in to get credit for their participation in the exercise. Facilitate the responder critique immediately after the exercise. You may explain why things happened or the responder expectations, but do not comment on Conducting performance. operations-based Evaluators Assist the controllers with monitoring safety. Position yourself to observe and hear the responders' performance, but do not get exercises is like in the responders' way. producing If you have a question for responders, go through the controller. theater. To create Keep your timeline of observed performance during the exercise; evaluation of performance happens after the exercise. the final product Evaluate performance against the responder's plans and procedures, not what you of a stage play, believe the plans and procedures should be. there is both a Attend the postexercise critique. Take notes regarding what is said, and use the debrief to understand the rationale for why specific actions were performed. script for actors Do not comment on performance during the critique. and a supporting Observers and Do not ask questions of responders, evaluators, or controllers. All questions will be set of activities VIPs addressed through your escort. Stay out of the way of responders. for the stage Remain quiet. Do not critique or prompt responder performance. crew. In this Keep a professional bearing. Remember that the first people to be kicked out due analogy, existing to space restrictions are the observers and VIPs. When in doubt, ask your escort. plans, policies, and procedures (for the players) EXERCISE CONDUCT and the MSEL (for the After design and development tasks are complete, the exercise takes place. Exercise conduct details controllers) serve include set-up, presentations/briefings, facilitation/control/evaluation, and call-off procedures. as the script, and the message SET-UP injects are like the supporting The planning team should visit the exercise site on the day prior to the exercise to set up the site. On activities the day of the exercise, planning team members should arrive several hours before the scheduled start performed by the time to handle any remaining logistical or administrative items pertaining to set-up and to arrange for stage crew. registration. 53

OCR for page 47
Guidelines for Transportation Emergency Training Exercises PRESENTATIONS/BRIEFINGS Presentations and briefings are important tools for delivering information. As described in Section 4, a discussion-based exercise generally includes a multimedia presentation to present the scenario and accompany the SITMAN. An operations-based exercise may include briefings for controllers, evalua- tors, actors, players, and observers. A briefing and/or presentation is an opportune time to distribute exercise documentation, provide necessary instructions and administrative information, and answer any questions. FACILITATION/CONTROL/EVALUATION In an operations-based exercise, controllers plan and manage exercise play, set up and operate the exercise incident site, and possibly take the roles of response individuals and agencies not actually par- ticipating in the exercise. Controllers give key data to players and may prompt or initiate certain player actions (as listed in the MSEL) to ensure that objectives are met and the exercise maintains continuity. Controllers are the only participants who should provide information or direction to the players. All con- trollers should be accountable to one senior controller. If conducting an exercise requires more con- trollers or evaluators than are available, a controller may serve as an evaluator; however, this dual role typically is discouraged. Evaluators are selected from various agencies to evaluate and comment on designated functional areas of the exercise. Evaluators are chosen based on their expertise in the functional areas that they will review. Evaluators have a passive role in the exercise and only note the actions of players; they do not interfere with exercise flow. Evaluators should use exercise evaluation guides (EEGs) to record obser- vations and notes. Controllers support the exercise by both conducting functional activities (e.g., setting up a simulation smoke machine) and inserting scripted event messages into play in accordance with the exercise sce- nario. Typically, two types of messages are used in transportation exercises: action messages and con- trol messages. Action messages provide event information to players during the exercise. These mes- sages are usually scripted in the MSEL. Control messages enable controllers to keep the exercise on track and to address situations in which players did not meet specific objectives or took actions that were not anticipated in the MSEL. Specific uses of messages are presented in Table 12. CALL-OFF PROCEDURES Controllers and evaluators should always be cognizant of safety issues that may present themselves during any exercise. If the safety of participants or bystanders is in any way compromised, it is the responsibility of those conducting the exercise to suspend or terminate the exercise. Appropriate atten- tion to that call-off procedure should be part of any safety plan in any operations-based exercise. TABLE 12 ACTION AND CONTROL MESSAGES Message Type Uses Action Messages Used to provide event information (e.g., 911-calls) Used to provide earned information (e.g., description of damage when a reentry team enters a building) Used to request information from responders (e.g., news requests via telephone) Control Messages Used to start, suspend, and terminate the exercise Used to force responder actions where expectations are not met Used to insert time-sequenced data (e.g., change in predetermined weather conditions) Used to start and end simulations (e.g., turning smoke generators on and off) Used to provide instructions to controllers (e.g., remind responders to state "this is an exercise") 54

OCR for page 47
Guidelines for Transportation Emergency Training Exercises There is no standardized format for messages in transportation exercises. Messages may contain any combination of the following: Message item: Indicates the content or characteristic of the message. Message number: Indicates the message in the MSEL. From line: Indicates who issues the message. To line: Indicates the recipient(s) of the message. Time: Indicates the expected time at which the message will be issued. Method of delivery: Indicates how the information will be delivered. Subject line: Summarizes the message. This is often the exact wording in the MSEL. "This is an exercise": Notifies readers that the message is for an exercise. This statement is located before and after the message text. Special instructions: Inform the person delivering the message of the special conditions for issu- ing the message. Script or instructions: Provides information to responders based on the MSEL. Instructions to controller or actor: Indicates activities to start and end simulations, provides attached data forms, or changes simulated conditions. Note area: Provides space for controllers to note responder performance. Time of message delivery: Indicates when the message was delivered. Controllers are in a unique position to view exercise play, grasp the dynamics of an action or activity as it unfolds, and comment on what they observe. Controllers should have extensive emergency pre- paredness experience, and they should have participated previously in tabletops, drills, and other exer- cises so that they know what to expect in the way of behavior and response. In addition, they should Controllers use have observed exercises at other facilities in order to broaden their perspective and experience. They the MSEL should keep current on updates to the emergency plan. to direct communication PLAYER INTERACTION WITH CONTROLLERS AND SIMULATORS with players and to ensure that the Controllers and simulators will have constant interaction with players throughout the exercise; however, exercise stays on each interacts differently. Controllers monitor and manage exercise activities to ensure that exercise track. Simulators objectives are being met, interact with players to determine the status of ongoing activities, and com- also communicate municate with players by following the MSEL and injecting implementer messages. Controllers must with players, ensure that they do not disrupt play when communicating with players. Simulators, on the other hand, representing an play the role of nonparticipating persons or organizations. organization affected by the The MSEL is the primary document used by exercise controllers to manage the exercise and to know exercise that was when to insert event implementer messages into the exercise. The MSEL is restricted for use by con- not able to attend. trollers, simulators, and evaluators. Actors support players by Simulators communicate with players by responding to questions from the players directed to non- playing specific playing persons or organizations. Occasionally, a selected simulator may act as a surrogate for a roles, such as senior official or decision-maker. While role playing as a senior official, the simulator may interact with injured victims. players on a face-to-face basis. 55

OCR for page 47
Guidelines for Transportation Emergency Training Exercises Players have an active role in responding to an incident by either discussing (in discussion-based exer- cises) or performing (in operations-based exercises) their regular roles and responsibilities. Each player is responsible for acting on exercise messages in accordance with established procedures and for coor- dinating actions in the same manner as for an actual event. Players also must ensure that exercise con- trol is informed of actions taken and completed through follow-up voice or message exchange initiated by the player. Actors are mock victims who simulate specific roles, including injuries from a disaster, to add realism to an exercise. Actors may be made up to more realistically reflect their injuries or symptoms. This practice is referred to as moulage. A sample checklist for managing the conduct of a full-scale exercise is presented in Table 13. EVALUATION Exercise evaluation refers to the act of observing and recording exercise activity or conduct; applying the behavior or activity against exercise objectives; and noting strengths, weaknesses, deficiencies, or other observations. As evaluated practice activities, operations-based exercises provide a process for continuous improvement. Evaluation is the cornerstone of exercises; it documents strengths and opportunities for improvement in a jurisdiction's preparedness and is the first step in the improvement process. The evaluation process for all exercises includes a formal exercise evaluation, integrated analysis, and an AAR/improvement plan that should begin with exercise planning and end when improvements have been implemented and validated through subsequent exercises. The process recommended by HSEEP is presented in Figure 11. HSEEP Volume II provides extensive guidance for establishing evaluation programs and is available at http://www.ojp.usdoj.gov/odp/docs/HSEEPv2.pdf. As discussed earlier, Attachment 2 provides an exercise evaluation guide that can be used by trans- portation agencies to support evaluation of exercises. Both expected activities and performance mea- sures are identified for each of the eight mission outcomes specified by the DHS. HSEEP includes the following eight steps for evaluation: Plan/organize the evaluation. Observe the exercise and collect data. Analyze data. Develop AAR. Conduct debrief meeting. Identify improvements to be taken. Finalize AAR. Track implementation. As described in this process, exercise evaluation should address each exercise objective to answer the following questions: Was the exercise objective met? 56

OCR for page 47
Guidelines for Transportation Emergency Training Exercises TABLE 13 FULL-SCALE EXERCISE CHECKLIST Full-Scale Exercise Checklist Participants: Controller(s)--sufficient to manage all event sites Actors (mock victims)--different age groups, body types, physical characteristics Players (most functions, all levels--policy, coordination, operation, field) Evaluators Simulators--to convey messages and actions for agencies or individuals who could not participate in the exercise Safety Officer Site Selection: Adequate space for number of victims, responders, and observers Space for vehicles and equipment As realistic as possible without interfering with normal traffic or safety Credible scenario and location Scene Management: Logistics (who, what, where, how, when) Believable simulation of emergency Realistic victims Preparation of simulators to realistically portray roles Number of victims consistent with type of emergency, history of past events Types of injuries consistent with type of emergency, history of past events Victim load compatible with local capacity to handle Props and materials to simulate injuries, damage, other effects Personnel and Resources: Number of participants Number of volunteers for scene set-up, victims, etc. Types and numbers of equipment Communications equipment Fuel for vehicles and equipment Materials and supplies Expenses identified (wages, overtime, fuel, materials and supplies) Response Capability Sufficient personnel kept in reserve to handle routine nonexercise events Safety Safety addressed through development Each design team member responsible for safety in own discipline Hazards identified and resolved Safety addressed in pre-exercise briefing, simulator, and evaluator packets Each field location examined for safety issues Safety officer designated, given authority Legal Liability Legal questions of liability researched by local attorney Emergency Call-Off Call-off procedure in place, including code word or phrase Call-off procedure tested Media Role of media addressed in planning, used as a resource Media and observers considered in logistical planning If yes, what were the results? If no, what changes are necessary to achieve the objective? Determining how the exercise objectives were met allows evaluators to answer the following important questions about the transportation agency's performance: Are parts of the plan in need of revision? Is current equipment adequate? 57

OCR for page 47
Guidelines for Transportation Emergency Training Exercises FIGURE 11 HSEEP PROCESS Is additional training required? Are additional resources necessary? Are staffing levels adequate? Is the communication system vulnerable to overload? How effectively did independent agencies cooperate to resolve the problem? Evaluators record what they observe during the exercise, the hot wash sessions, and the after action review. Their objective is to describe what happened, compare it with what was supposed to happen (as scripted in the MSEL), and explain why any differences between the two occurred. Evaluators also take the lead in drafting recommendations and lessons learned from the exercise. Like controllers, eval- uators must receive special training for the exercise. For most operations-based exercises, evaluation involves the following activities: 58

OCR for page 47
Guidelines for Transportation Emergency Training Exercises The AAR is used DEBRIEF/HOT WASH to provide feedback to A debrief (for facilitators or controllers/evaluators) and/or hot wash (for players) should occur follow- participating ing both discussion- and operations-based exercises. The debrief is a forum for planners, facilitators, jurisdictions controllers, and evaluators to review and provide feedback on the exercise. It should be a facilitated on their discussion that gives each person an opportunity to provide an overview of the functional area that performance they observed and to document both strengths and areas for improvement. during the exercise. The The debrief should be facilitated by the lead exercise planner or the exercise director; results should be AAR summarizes captured for inclusion in the AAR. Other sessions, such as a separate meeting for specialized respon- what happened ders during an operations-based exercise, may be held as necessary. and analyzes performance of A hot wash occurs immediately following an operations-based exercise and gives players the opportu- the tasks nity to provide immediate feedback. It enables controllers and evaluators to capture events while they identified through remain fresh in players' minds, to ascertain players' level of satisfaction with the exercise, and to deter- the planning mine any concerns and proposed improvement items. Each functional area (e.g., fire, law enforcement, process as critical and medical) should conduct a hot wash, and each hot wash should be facilitated by the lead controller and the for that area. demonstrated capacity to The debrief and/or hot wash provides an ideal time for facilitators, controllers, evaluators, and players accomplish the to complete and submit their completed EEGs and feedback forms. Information from these forms should overall exercise be included in the AAR/improvement plan. goal. The AAR includes Information on conducting debriefs and hot washes is available on the G&T Secure Portal, which is sum- recommendations marized at http://www.ojp.usdoj.gov/odp/docs/HSEEPv4.pdf. for improvements AFTER ACTION REPORT based on the analysis, which To prepare this report, the exercise evaluation team will analyze data collected from the hot wash and/or will be addressed debrief, participant feedback forms, and other sources (e.g., plans and procedures) and compare the in the IPC. actual results with the intended outcome. The level of detail in an AAR reflects the exercise type and size. AARs describe the exercise scenario, player activities, preliminary observations, major issues, and recommendations for improvement. A sample outline for an AAR includes the following: Executive summary: -- Strengths. -- Areas for improvement. Exercise overview: -- Exercise name. -- Exercise duration. -- Exercise date. -- Exercise location. -- Sponsoring agency. -- Type of exercise. -- Funding source. -- Classification (i.e., sensitivity of information). 59

OCR for page 47
Guidelines for Transportation Emergency Training Exercises -- Scenario. -- Participating agencies. -- Number of participants. -- Exercise overview. -- Exercise evaluation. Exercise goals and objectives: -- Goal #1. -- Objectives for Goal #1. -- Goal #2. -- Objectives for Goal #2. -- Goal #3. -- Objectives for Goal #3. Exercise events synopsis: -- Scenario. -- Timeline. Analysis of outcomes for each participating agency: -- Analysis of how well the transportation agency and other participating agencies and juris- dictions performed their functions during the exercise. Examples include a few paragraphs on how well each participating agency performed its functions in response to the exercise scenario. Analysis of critical task performance: -- Analysis of how individual tasks were performed, as defined in the evaluation guides. Each task identified by the exercise planning team as critical to the response required by the sce- nario should be discussed in this section. Tasks that were performed as expected require only a short write-up that describes how the task was performed. These write-ups generally would not be followed by recommendations. For tasks that were not performed as expected, the write-up should include (1) an issue statement; (2) references to plans, procedures, and eval- uation guides; (3) a brief summary of the issue; (4) the effect of the issue on the response; and (5) an analysis of what happened or did not happen and the root causes for the variance from the expected outcome. Recommendations for improvement should also be presented to address identified issues. To facilitate tracking of recommendations and improvements, acronyms should be spelled out in each recommendation. -- Documentation of a variance from expected performance that may have resulted in an improved response or innovative approaches that were used during the response. Conclusion Improvement plan matrix 60