Train-the-Trainer Pilot Courses for Incident Responders and Managers (2013)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

16 C h a p t e r 3 This section presents the findings of a comprehensive analysis of the SHRP 2 train-the-trainer pilot courses, based on the results of the (1) Course Reaction Analysis (post-course par- ticipant and instructor feedback forms); (2) participant assess- ments; and (3) curriculum modification recommendations from the team based on observer, SHRP 2 team, FHWA, and student feedback. Findings are summarized in the following areas: (1) sufficiency of materials and instructional methods to prepare instructors; (2) course length; (3) instructor criteria; (4) achievement of learning objectives; (5) multidisciplinary emphasis of training; (6) curriculum changes; and (7) logisti- cal lessons learned. Sufficiency of Materials and Instructional Methods to prepare Instructors The team reviewed participant feedback forms from the first four pilot courses to assess this area. One key indicator of a successful course is word-of-mouth recommendation from colleague to colleague. Figure 3.1 illustrates the responses to Question 16 of the feedback form: “Would you recom- mend this course to others?” The response was overwhelm- ingly positive, with 95% of attendees indicating they would recommend this course (139 out of 146 respondents). Of the seven individuals who would not recommend the course, the responses were relatively evenly distributed across the disciplines in attendance (two transportation, two fire, and three law enforcement), indicating any negative feelings were not concentrated within a single discipline. There also did not appear to be any correlation with experience, with no negative responses among those with more than 20 years’ experience, three responses from those with 16 to 20 years, one response with 11 to 15 years, one response with 6 to 10 years, and one response with 1 to 5 years (the remaining negative or neutral response did not provide an experience level). Another critical indicator of success in the course is the ability to connect with students regarding the importance of safe, quick clearance (SQC) principles. When asked if they gained an appreciation of SQC, 95% of respondents answered positively (139 out of 146 responses). The appre- ciation was spread uniformly among disciplines, as shown in Figure 3.2. Furthermore, increased appreciation of SQC principles occurred at all experience levels. Attendees also viewed the course as a valuable time-saving exercise as it relates to training other TIM professionals. Fully 98% of respondents indicated attending the course saved them research time. As shown in Figure 3.3, when asked how many hours of research time were saved by attending the course, more than half (54%) of respondents reported a large time savings of 6 hours or more. Both TIM experts and nov- ices reported time savings. Finally, given that these were train-the-trainer pilots, an important evaluation area was the extent to which the atten- dees felt confident they could teach the curriculum to other TIM professionals after completion of the train-the-trainer course, based on materials and instruction. As shown by Fig- ure 3.4, the vast majority of attendees, 85%, felt confident they could lead the course. Again, these responses were relatively consistent across discipline, TIM experience, and training experience. The team offers the following additional findings based on participant feedback forms, as well as observer feedback. Course Length The team reviewed participant feedback forms from the first four pilot courses to assess this area. While this course was origi- nally designed to be 2.5 days, based on feedback received from the TETG in the KO meeting, the team condensed it to 2 days based on participant feedback and TETG input—1.5 days for the core TIM curriculum and 0.5 day for the train-the-trainer component of the course. However, it is important to note that, Findings and Applications

17 even though the train-the-trainer lessons were reserved to the end of the course, the instructors provided trainer-specific insights on the material throughout the course; for example, they alerted students to key teaching points or to issues that may arise in future classes. In response to the question “The duration of the training was sufficient for learning the subject matter,” 135 out of 147 respondents from the four train-the-trainer pilots agreed (Figures 3.5 and 3.6). Most concerns surrounding the suffi- ciency of the course length came from towing participants, which may reflect the fact that these participants have likely received less formal training on TIM principles than DOT, fire, and law enforcement participants. In free responses to the evaluation, many students felt the training was rushed and, for example, “could go another ½ day . . . a little overwhelming.” Others wrote that the course was too long and expressed concerned that responders from their discipline would not be able to devote 16 hours to a training course; one wrote “For the course to have an impact on firefighters, it needs to be shortened to 3 to 4 hours.” The divide in reactions to course length reinforces the need for instructors to tailor the course to their students. If a class is targeted to individuals with extensive field experience, less time may be required. However, if experience levels are more mixed (as they were in the four train-the-trainer pilots con- ducted), then the full 2 days may be necessary. In addition, instructors have the option to modularize the course so that material can be given in shorter periods and responders do not have to devote 2 full days to the course, but the content can still be covered in detail. Throughout the train-the-trainer pilots, the instructors reinforced the potential for future trainers to present the 2% 19% 25% 28% 26% None 1 to 2 3 to 5 6 to 10 10+ Figure 3.3. “Estimate the time (in hours) this training may save you on research information.” Figure 3.2. “I gained an appreciation of why quick clearance is important.” 1% 4% 30% 65% Strongly Disagree Disagree Neutral Agree Strongly Agree Figure 3.1. “I would recommend this course to others.”

18 course using a modular concept by selecting specific lessons to teach rather than the entire course. In addition, they pro- vided students with examples on how the structure of the course could be rearranged. For example, at the alumni-led pilot, the “Situational Awareness” lesson (currently Lesson 11) was moved to a position after Lesson 3 (“Arrival”) so that stu- dents had an opportunity to leave the classroom and break up the time spent on in-class lecture during the first day. Several students noted that they would use the modular concept when delivering the training; for example, one planned “to implement the modules in 1–2 hour training meetings with local [fire departments], tow companies, etc.” The modular nature of the course provides trainers with more flexibility. It decreases the concern of many responders that they cannot afford to spend a full 1.5 days on training, and it enables instructors to target lessons based on the spe- cific needs of their region. Instructor Criteria At the four train-the-trainer pilot deliveries, the instructors represented two backgrounds—one from a fire background coupled with one trainer from a law enforcement or DOT background. At the alumni-led pilot, there were nine instruc- tors representing each of these major responder disciplines. The mix of instructor backgrounds reinforced the multi- disciplinary nature of the course and ensured that multiple perspectives were heard. Students appreciated the diversity of experience instructors brought to the pilot deliveries, with Figure 3.5. “92% of participants report the duration of the training was sufficient for learning the subject matter” (four train-the- trainer pilots overall). 1% 14% 61% 24% Strongly Disagree Disagree Neutral Agree Strongly Agree Figure 3.4. “Based on the training and materials I received, I am confident that I can lead all classroom activities.” Figure 3.6. “The duration of the training was sufficient for learning the subject matter” (four train-the-trainer pilots by discipline).

19 one student noting that “representation of two different disciplines . . . was well balanced and added significantly to the delivery and acceptance of the information.” Another important aspect of instructor background was their real-world field experience, which not only enabled them to speak knowledgeably about the subject matter but also increased their legitimacy as voices of authority on the topic of TIM because they had “lived” the content of the course. For example, one student said “The instructors gained my respect on Day One because they were ‘real.’ They had managed inci- dents. They had seen the results of mismanaged incidents firsthand.” For the train-the-trainer course, the team recommended (but did not require) that invited participants meet certain eligibility criteria, shown in Table 3.1. These criteria are meant to ensure that participants have a baseline understand- ing of TIM concepts and terminology and that they are expe- rienced instructors with the ability to pass the course on to future students. Participants who did not have experience in adult training affirmed that having this background would have enriched the value of the class. It is important for poten- tial students to recognize that the train-the-trainer course does not teach them how to be an instructor in general, but how to instruct this course in particular. Students should leave the course feeling confident that with adequate prepara- tion time to familiarize themselves with the content, they have enough background knowledge as an instructor to teach the course effectively. achievement of Learning Objectives The second level of evaluation analyzed the results of the post-course student assessment to determine whether instruc- tional strategies supported learning objectives and if the mini- mum knowledge requirements were met across responder disciplines and experience levels. A full review of the course assessments can be located in Appendix E. The exam was distributed to 162 incident responders par- ticipating in one of the five pilot courses. The team primar- ily targeted incident responders from six separate disciplines to participate in the course: law enforcement, fire/rescue, Department of Towing and Recovery, emergency medical services (EMS), dispatch, and DOT. Each participant, under the guidance of the instructor, was issued a test with spe- cific instructions. However, the test was informally proc- tored; the instructors were in the room while the students were taking the exams. The exam was not held to a specific time limit. The respondents consisted of 51 representatives of law enforcement, 42 from fire/rescue, 18 from Department of Towing and Recovery, two from EMS, two from dispatch, 46 from a state DOT, and one other. Table 3.2 provides a demo- graphic profile of the total respondents. The respondents were asked to provide their years of expe- rience. Of the 162 respondents, 137 answered the question. Table 3.3 provides the experience profile based on the answers received. Table 3.1. Recommended Student Eligibility Criteria to Participate in Train-the-Trainer Course Criteria Metric TIM-related field experience Minimum of 5 years in field Willing to participate in full course Agreement to participate in course Experience as instructor Recognized as instructor in his or her specific discipline Commitment to multidisciplinary TIM Desirable/Preferred: • Member of multidisciplinary TIM task force, working group, or committee • National Incident Management System (NIMS) Training, particularly Incident Command Sys- tem (ICS) 100, 200, and 700 Table 3.2. Respondents by Discipline Discipline Number of Respondents Law enforcement 51 Fire 42 Towing 18 EMS 2 Dispatch 2 DOT 46 Other 1 Total 162

20 Figure 3.7 illustrates the overall student performance, com- pared pilot to pilot. There was minimal variation among loca- tions. Virginia’s students achieved the highest score (85.0%). Alumni-led students achieved the lowest scoring (80.4%). The lower alumni-led score was anticipated, given that (1) the alumni-led pilot was marketed to less-experienced responders than the four train-the-trainer pilot courses and (2) the alumni-led pilot was taught by recent graduates of the train- the-trainer course, whereas the train-the-trainer pilots were taught by master instructors from the team who were very familiar with the curriculum. One purpose of this assessment was to determine whether the instructional strategies supported the learning objectives. Learning for each lesson was evaluated separately. Figure 3.8 presents overall student performance by lesson, and it demon- strates that learning remained relatively consistent across the lessons. Student scores for the alumni-led pilot were generally lowest in all lessons. Lesson 2 has modest variation in scores, most likely because there were only three questions for this sec- tion. Given that, after modifications were made throughout the pilots, Lesson 2 is now designed for 20 minutes of instruction time, it may be necessary to add more questions to that lesson. Scores generally trend downward after Lesson 3, likely due to fatigue. It is important to note that the course is designed to be delivered in its entirety or in modules. In instances where the course is broken into several modules, assessment fatigue is anticipated to be less of an issue. Should the course be deliv- ered in its entirety, it is recommended to move the Field Table 3.3. Respondents by Discipline and Years of Experience Discipline 0–5 6–10 11–15 16–20 21 Total Law enforcement 12 5 12 7 7 43 Fire 7 4 7 2 19 39 Towing 3 2 4 2 2 13 EMS 0 0 1 0 1 2 Dispatch 0 0 0 0 1 1 DOT 9 4 9 7 9 38 Other 0 0 1 0 0 1 Total 31 15 34 18 39 137 Figure 3.7. Average student assessment scores across pilot locations. Figure 3.8. Average assessment scores by lesson and pilot location.

21 Activity (Lesson 11) from Day 2 to Day 1 to provide an extended classroom break on the 1st day. This move will also keep students in the classroom before the exam and should provide better continuity (i.e., students will not have to transi- tion from classroom curriculum to the field activity and then back to the classroom for assessment). Variation in absorption was evaluated to determine if con- tent presentation had an impact on students’ learning. Fig- ure 3.9 presents the average lesson scores for those that attended one of the four train-the-trainer pilots and demon- strates there is some variability in the absorption of learning at the start and end of the class. Lesson 3 received the highest score (88.7%); Lesson 9 the lowest (66.1%). Several respon- dents skipped Lesson 9 (skipped sections are not included in the analysis). Due to the volatility of curriculum modifica- tions throughout the pilot deliveries, some imbalance resulted between the curriculum and parallel exam questions for the various lessons. Figure 3.10 depicts the relative distribution of time and number of exam questions for each lesson (les- sons are noted in pie slices as well as in the key to the right). For example, Lesson 9 was designed for only 10 minutes (3%) of instruction time, yet contains seven assessment ques- tions (7% of the exam questions). Likewise, Lesson 2 currently accounts for approximately 5% of total instructional time yet 3% of exam questions, and Lesson 5 represents approximately 5% of total instruction time yet 11% of the exam question bank. A recommendation is to rebalance the exam question bank to more closely reflect the approximate “weight” of the lessons as measured by approximate lesson time. In addition, it is recommended that additional questions be added to Les- son 2 to provide a more balanced ratio of instruction time to number of assessment questions. A secondary purpose of this assessment is to determine whether the minimum knowledge requirements were met 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0% 100.0% Figure 3.9. Average assessment scores for all train- the-trainer pilots (variation in absorption). Figure 3.10. Comparison of lesson lengths and associated relative proportion of questions in exam by lesson.

22 across incident responder types and experience levels. Fig- ure 3.11 illustrates that learning is occurring across the vari- ous responder types. (Only law enforcement, fire, towing, and DOT are shown; EMS and dispatch are excluded due to smaller sample size.) It demonstrates that learning remains relatively consistent across the four disciplines. There is little variation among discipline scores in Tennessee and Montana. Student scores for the alumni-led pilot course had the largest spread between high and low discipline score (15.6 points). Figure 3.12 illustrates that learning occurred across the various experience levels in on-scene TIM response and dem- onstrates that learning remained relatively consistent across the continuum of experience in the field. The 25 students who did not identify their level of experience scored within the same level as those who did. In summary, there is a small difference in scores based on years of experience, as demon- strated by the lowest score of 79.6% for those with 6 to 10 years and the highest score of 84.4% for those with more than 21 years of experience. Overall the assessment successfully measured course perfor- mance. Learning occurred across incident responder types and experience levels. There was no major difference in student performance based on training or testing location. It is appar- ent from performance on the assessment that the instruc- tional strategies supported the learning objectives. Multidisciplinary Emphasis of Training A common theme across all five pilots in response to the ques- tion “What do you consider to be the most valuable informa- tion that you will take away from this class?” was that students understood the importance of inter-agency communication and cooperation in TIM. The overwhelming appreciation for the interdisciplinary aspect of the training course reflects that the value of inter-agency collaboration comes through strongly in the course materials and resonates with participants. The team recommends that instructors continue to emphasize the Figure 3.11. Average assessment scores by location and discipline (multidisciplinary learning). Figure 3.12. Average assessment scores across the continuum of TIM experience.

23 multidisciplinary aspect of the course and try to gather respond- ers from diverse backgrounds in their training sessions. Sample student responses to the question, “What do you consider to be the most valuable information that you will take away from this class?” include the following: • “The value of interdisciplinary training and cooperation.” • “From a fire rescue standpoint, the integration of multiple disciplines on the scene that typically in the past has not been in the forefront.” • “Been doing this training for a few years. It helps to have multidiscipline setup.” • “Each discipline’s response role from dispatch to scene to clearing.” • “Every department has assets and resources that contrib- ute to TIM. This class shows how to merge all assets and resources together for the best possible TIM scene.” Curriculum Changes At the conclusion of each pilot course delivery, feedback from attendees and observers was consolidated into a list of cur- riculum and delivery changes. While each course resulted in hundreds of comments, a review of the primary changes from each pilot delivery effectively illustrates the evolution of the course across the five pilots. Each of the five pilot deliveries resulted in specific rec- ommended changes to consider for future deliveries. These comments were made by observers and instructors from the team, class participants, and members of the project man- agement team (SHRP 2/FHWA) who attended the training. Every discrete comment was saved into a comment log with specific details regarding the lesson and slide to which the comment referred, the proposed resolution to the comment, the name of the commenter, as well the curriculum materials to which the resolution to the comment would need to be applied—the PowerPoint presentation, the instructor guide, and/or the student workbook. The team developed a com- ment log for each pilot delivery, which was adjudicated with the project management team. In addition, the com- ments were maintained in an ongoing master comment log that traced comments received, starting with the KO meet- ing and continuing through all pilot deliveries, as shown in Figure 3.13. The team implemented changes to the course materials resulting from the comments as time permitted. Due to the short time lapse between the courses (particularly between the Tennessee, Virginia, and Montana deliveries), the team priori- tized comments so that those considered the most important were addressed before the next training delivery, while others were addressed on an ongoing basis. In total, the team collected and implemented nearly 1,500 discrete content changes as a result of comments collected in the KO meeting and across the five training deliveries. A description of the type and substance of comments gathered from each pilot follows. Figure 3.13. Screenshot of course comment log from Florida pilot delivery.

24 Tennessee (Nashville) As a result of comments received during the training deliv- ered in Nashville, Tennessee, the team made 112 changes to the course curriculum. The types of changes implemented are outlined in Figure 3.14. Key curriculum modifications implemented by the team as a result of the Nashville training include • Addressed inconsistencies between the core and train-the- trainer instructor guides and the presentations. • In response to feedback that the course felt “rushed,” devel- oped a pacer guide to aid instructors by providing them timing guidance by lesson and subsection. • Further balanced quick clearance and safety messaging. • Added TIM time line graphic and NASCAR “pit stop” video to promote SQC messaging. • Advanced sources of congestion and quick clearance laws (to Lesson 0) to address these topics earlier in the course. • Added an instructor customization checklist to beginning of the instructor guide. Virginia (Richmond) As a result of comments received during the training deliv- ered in Richmond, Virginia, the team made 156 changes to the course curriculum. The types of changes implemented are outlined in Figure 3.15. Key curriculum modifications implemented by the team as a result of the Virginia training include the following: • Refreshed imagery to show more highway and fewer city street scenes, more mixed discipline scenes, and updated accident scenes. • Updated terminology to ensure it is discipline-neutral. • Removed or made recommendations for substitution of duplicative case studies or examples. Figure 3.14. Change profile for Nashville, Tennessee, pilot delivery. Figure 3.15. Change profile for Virginia pilot delivery.

25 Key curriculum modifications implemented by the team as a result of the Florida training include • Added “Kerri Crane (Indiana State Police)” secondary inci- dent video to the beginning segment of the course to rein- force quick clearance messaging. • Revised TIM diagram and expanded instructor manual guidance to better align with the Manual on Uniform Traf- fic Control Devices. • Included printed scenario instructions at each table for the hands-on tabletop activity. • Updated imagery (i.e., arrow board, deployable sign). Tennessee (Knoxville)—Alumni-led Pilot As a result of comments received during the training deliv- ered in Knoxville, Tennessee, the team made 80 changes to the course curriculum. The types of changes implemented are outlined in Figure 3.18. Key curriculum modifications implemented by the team as a result of the Knoxville training include the following: • Refreshed data such as line-of-duty death statistics. • Moved the “Situational Awareness” lesson so it follows Lesson 3 (“Arrival”) to break up lecture delivery on the 1st day with a more hands-on lesson. • Reinforced instructor notes based on observed experience with new instructors. • Created a chart to assist instructors with tabletop assign ments. • Added photography to provide visual instruction on setting up activities. • Inserted thumbnail PowerPoint images into instructor guide so students could more easily trace the content in the guide to the presentation. Montana (Helena) As a result of comments received during the training deliv- ered in Helena, Montana, the team made 153 changes to the course curriculum. The types of changes implemented are outlined in Figure 3.16. Key curriculum modifications implemented by the team as a result of the Montana training include the following: • Reinforced SQC terminology throughout. • Added instructor notes that provide context for the course content, promote uniformity of delivery, and enhance messaging. • Updated instructor guide and student workbook to match presentation. Florida (Fort Lauderdale) As a result of comments received during the training deliv- ered in Fort Lauderdale, Florida, the team made 302 changes to the course curriculum. The types of changes implemented are outlined in Figure 3.17. Figure 3.16. Change profile for Montana pilot delivery. Figure 3.17. Change profile for Florida pilot delivery.

26 of the training to potential attendees. The L32A research affirmed the value of a minimum of three e-mail correspon- dence touch points with candidate participants as well as person-to-person phone calls to support recruitment of par- ticipants. Sample invitational language is provided in Appen- dix B. The local points of contact should feel free to customize the invitational language to the local audience, provided the modifications retain the key logistical points. Reminder and follow-up correspondence by e-mail and telephone are required to secure the desired participation. Finally, having at least one high-ranking member of state or regional organiza- tions representing transportation, law enforcement, and fire be involved in the planning process helps champion and secure participation by students in the training. • Pre-Event Setup. A pre-event meeting, ideally held the after- noon before the course delivery, is also necessary to ensure that the classroom is prepared. Of particular importance is setup for the tabletop activity (Lesson 11), which can take 1–3 hours. Other planning activities outside of conference calls are also required throughout the course planning pro- cess. A final planning activity is to obtain the needed class- room learning materials and teaching props. The instructor guide provided in this curriculum delineates the checklist of needed items, and organizers are encouraged to review this well in advance of the training date. Logistical Lessons Learned The team identified the following recommendations and les- sons learned regarding workshop logistics: • Planning. A minimum of three phone calls are recommended for workshop planning: an initial call, a midterm call, and a final planning call. Ideally, the initial planning call should be held at least 8 weeks before the planned training event. Outcomes of this call include securing the venue for the training; confirming event sponsors; confirming trainers; and creating participant recruitment strategy, time line, and roles. The midterm call focuses on creating strategies to secure desired participant mix; preparing instructors; assigning roles for preparing and distributing participant packages; and initiating logistical arrangements. The final planning call involves any final recruitment actions that need to be taken to secure the desired participant mix; final- izing logistical arrangements with the venue; and assuring instructor preparation. • Participant Recruitment. As soon as possible, get involve- ment from the various local organizations because this is critical for obtaining a balanced pool of qualified attend- ees. The most successful pilot courses had strong local lead- ership that effectively conveyed the purpose and importance Figure 3.18. Change profile for Knoxville, Tennessee, pilot delivery.