Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
21 Conclusions The outcome of this research project has been the creation of a comprehensive multidisciplinary curriculum devel- oped with the input of responders from the targeted audi- ences. Additionally, a diverse group of evaluators considered to be TIM leaders extensively reviewed the curriculum. Par- ticipants received the training program well and viewed it as beneficial. Their positive response is demonstrated by the Kirkpatrick Level 1 and 2 evaluation results discussed in the section on assessment of training effectiveness in Chap- ter 3. The curriculum represents what the team considers to be the gold standard encompassing all the topics and strategies that, if successfully transferred into everyday work practices, would help achieve reliability goals while also increasing the safety of both the general public and on-scene responders. One of the critical factors in ensuring the successful nation- wide implementation of this course is ensuring that potential instructors are adequately prepared to teach the course. To help facilitate this preparation, detailed speaker notes have been added to the courseâs instructor guide and a train-the- trainer component has been developed. However, although the train-the-trainer component was delivered during the pilots, actual instructor delivery of the course by trainers who were not involved in the development of the materials has not yet been tested. Therefore, to validate the transferability of the course, the research team recommends that the train- the-trainer curriculum also be validated through targeted pilot testing. Candidate instructors would be required to demonstrate subject matter and curriculum competency as well as classroom management and facilitation skills to a spe- cific standard. Competency criteria for both a master instruc- tor and a qualified instructor could be developed. To develop these criteria, the research team recommends, as outlined in the section on additional pilot course deliveries, that a series of 10 pilots be conducted. These pilots would be divided into three phases: â¢ During Phase I, content not previously tested in a class- room (i.e., changes made in response to the final pilot of the SHRP 2 L12 effort) could be field tested and validated. During this phase, a sample of potential instructors would be selected and trained in the content to act as instructors of the program for Phase II. â¢ Phase II of the pilot series would consist of having the potential instructors conduct the training sessions with different methods used to facilitate their transition to pro- ficiency in teaching this course (as outlined in Table 4.1) and using a standardized evaluation tool against which proficiency would be measured. â¢ During Phase III, any changes that resulted from lessons learned in Phase II would be incorporated, and a final pilot would be held to validate any changes. Although the level of detail covered in the curriculum is necessarily robust and thorough, it may be difficult for some agencies to send their responders for a full 2-day course. Therefore, the research team recommends exploring other options for delivery, such as those discussed below in the sec- tions on online and discipline-specific training. Although the research team agrees that for this subject matter face-to-face training is preferable to achieve optimal learning outcomes, offering online options could open up the content to a wider audience. Of the online options sug- gested below, the team believes that creating prerequisite modules that could be completed by students before they attend the classroom portion would be the most beneficial. A combination of online and classroom learning would shorten classroom time while retaining the unique benefits of face-to-face, multidisciplinary training. Additionally, it would retain the integrity of the curriculum, allowing all C h a p t e r 4 Conclusions and Suggested Research
22 Table 4.1. Field Test Actions and Objectives Field Test Actions and Objectives Phase I Field Test 1 â¢ Content should be refined because many modifications were made after the final research field test and evaluator debriefing. â¢ Lesson times should be validated because several lessons had to be abbreviated during the initial field tests. â¢ Tasks specific to train the trainer, such as teach-backs, setup and implementation of hands-on tabletop, and situational awareness lessons, should be refined and expanded with selected students from the group. Field Test 2 â¢ Remediation of all content and timing issues should be verified. â¢ Remediation of any train-the-trainer issues identified in the first field test should be validated. Field Test 3 â¢ This would be the first field test for which all content is considered golden. â¢ Candidates who have been identified as potential instructors should be trained. This group would consist of four to six people from a variety of disciplines who have prior instructor experience in their discipline. Phase II Field Test 4 â¢ The value of mentoring could be tested by pairing the master instructor with one of the novice instructors and having them teach the class together. â¢ The session should conclude with a debriefing and lessons-learned session for the novice instructor. â¢ The novice instructor should be evaluated against a structured and controlled model. Field Test 5 â¢ Two novice instructors are primarily responsible for delivering the class, with observation and assistance, as necessary, of a master instructor. â¢ Because they have fewer primary teaching responsibilities, the master instructors would be able to better observe classroom interactions and help coach and mentor novice instructors, as necessary. â¢ The session should conclude with a debriefing and lessons-learned session for the novice instructors. â¢ Novice instructors should be evaluated against a structured and controlled model. Field Test 6 â¢ Novice instructors would lead this class alone without the presence of the master instructor. This field test would test the transferability of the course without the mentoring option. â¢ The session should conclude with a debriefing and lessons-learned session for the novice instructors. â¢ Novice instructors should be evaluated against a structured and controlled model. Field Test 7 â¢ The value of mentoring could be tested by pairing the master instructor with one of the novice instructors and having them teach the class together. â¢ The session should conclude with a debriefing and lessons-learned session for the novice instructor. â¢ The novice instructor should be evaluated against a structured and controlled model. Field Test 8 â¢ Novice instructors would lead this class alone. This field test would test the transferability of the course with instructors who had the benefit of previously working with a master instructor for their first class. â¢ The session should conclude with a debriefing and lessons-learned session for the novice instructors. â¢ Novice instructors should be evaluated against a structured and controlled model. Field Test 9 â¢ Novice instructors would lead this class alone without the presence of the master instructor. In the week before delivery, these instructors would have a virtual meeting with a master instructor to review instructional materials and obtain clarification on any pertinent issues. This field test would test the transferability of the course with limited mentoring. â¢ The session should conclude with a debriefing and lessons-learned session for the novice instructors. â¢ Novice instructors should be evaluated against a structured and controlled model. Phase III Field Test 10 â¢ In this final field test, lessons learned and content refinements should be implemented before the field test and validated during the delivery. content to be retained rather than cutting material to fit an allotted time period. Typical compression ratios when con- verting classroom training into online training are approxi- mately 50%, meaning that for every hour of classroom training, an equivalent amount of training could be taken online in 30 minutes. A full content analysis would be neces- sary to determine which portions of the content would be most suitable for conversion to an online format, but the research team believes that such an option would allow the classroom portion to be reduced to a 1-day class with 3 to 4 hours completed online before arrival in the classroom. Similarly, the research team recommends that discipline- specific training should be considered for further research. Although discipline-specific training may not promote the appreciation and understanding of other respondersâ TIM roles that is fostered by multidisciplinary training, it may be advantageous for many disciplines to insert excerpts from the course into their existing training or incorporate them into
23 44 Identify and train additional candidate trainers who, after meeting specified criteria relating to experience as both a responder and an instructor in their disciplines, would act as instructors for some of the subsequent deliveries. For ease of explanation, these instructors will be referred to as novice instructors, indicating that they are new to teaching this particular course. â¢ Phase II would involve delivering six field tests. The intent would be as follows: 44 Deliver the course using new instructors and expanded target audiences, and use a standardized model to vali- date instructor proficiency. Although all potential instruc- tors would receive the same base training, different methods would be used to facilitate their transition from novice to proficient (or qualified) TIM instructor. Methods would include the following: 4âª Having each novice paired with a master instructor for the first class; 4âª Having the master instructor observe and offer coach- ing and feedback to a pair of novices at designated points throughout the class; 4âª Having the master instructor coach the novice instruc- tors in the week before class delivery; and 4âª Offering no mentoring or coaching to evaluate the sufficiency of the instructor guide and other training materials as stand-alone materials in preparing train- ers to satisfactorily deliver this training. An independent evaluation team would monitor the courses to determine the various degrees of difficulty in delivery of all elements of the course content. The exact course lessons and specific course content that may chal- lenge future instructors would be identified. If a course lesson, a recommended delivery methodology, or a course element is consistently difficult to present, found to be confusing to instructors, or presented incorrectly during these field tests, these elements can be reevaluated, and changes made to the instructional strategies used. Feedback also would be gathered from the novice instruc- tors at the end of each class. At the conclusion of Phase II, a facilitated workshop should be conducted with the nov- ice instructors to gain insights into their experiences and garner any feedback that could be used to improve the model used to train instructors. On completion, the evaluation data from each ses- sion, as well as novice instructor feedback, should be analyzed and compared to gauge which strategy would be most beneficial for implementation of the course on a large scale. 44 No changes should be made to any course materials dur- ing this phase to enable testing a common set of materials to multiple audiences. â¢ Phase III would involve collecting all comments and rec- ommendations received during the Phase II delivery of field already established routines. The course has been designed in separate lessons to facilitate delivery in a modular fashion, although refinement of message and delivery format would likely have to be made for successful delivery of a lesson to a single discipline (or in a format that matched the require- ments of many disciplines). The research team does not consider this option to be optimal as a replacement for a multidisciplinary course, but it could serve to augment the reach and reinforce the message of the course by having the subject addressed at the discipline level. As described in Chapter 3 in the section on assessing train- ing effectiveness, the formative evaluations show that students have a positive reaction to the course and that they are able to demonstrate proficiency in the content. What is not yet known is whether students are transferring the knowledge gained back to their jobs and, if so, to what extent. The impact that this training may have on improving reliability is also not yet known. Methods of determining these parameters are dis- cussed below in the section on measuring learning transfer and results. The research team considers such measurements vital because positive outcomes provide validation that the training intervention is effectively targeting the correct areas; positive results would also encourage buy-in for the course from stake- holders. Less positive results would suggest that further inter- ventions should be considered. The section on additional pilot courses suggests the inclusion of such evaluations. Finally, as discussed in the section on leadership and over- sight for the training, the research team recommends begin- ning with central oversight and administration to allow the training and curriculum to become accepted as a national standard within its market. This same central entity would lead marketing for the training, including development and maintenance of the central website to establish a brand for the training, ensure that core messages are clear, and show that a reliable one-stop shop containing all information about this training is readily available for audiences. Suggested research: additional pilot Course Deliveries As described in the previous section, the research team rec- ommends that a three-phase approach be conducted to test the transferability of the curriculum to other instructors: â¢ Phase I would involve the first three deliveries of the field test. The intent would be as follows: 44 Further test course materials, delivery methods, and time requirements. These deliveries would follow the model used for the pilot course deliveries, with course materi- als being modified after each delivery to incorporate recommended changes.
24 deliveries to promote consistent quality in instructorsâ deliv- ery of the curriculum across the country. Candidate instruc- tors would emerge as approved, proficient instructors. The report from these field tests would yield a measure- ment and evaluation tool for the individuals designated as qualified by an agency. Such a tool would be invaluable. For the training to have the largest reach possible, regional master instructors would need to be responsible for conducting train-the-trainer classes to turn out qualified instructors who would teach the responder course at a more local level. Com- petency criteria for both a master instructor and a qualified instructor would be developed. Options to be considered include mentoring and coach- ing; these are more fully described in Table 4.1. This model provides the novice instructor with an opportunity to receive mentoring from the master instructor and also provides an opportunity for further evaluation and adherence to required criteria. It is envisioned that the field tests would be conducted in a format similar to that described in Table 4.1. Suggested research: Online training Students in both pilot sessions and independent evaluators suggested that an online version of the training would allow more responders to access the curriculum. Although feed- back was unanimous that in-person training is by far the preferred delivery mechanism because it fosters valuable relationship building among the multidisciplinary, multi- agency course participants, it is not feasible for all agencies, especially those working with volunteers. Stakeholders also pointed out that online training can be a valuable supple- mental resource to reinforce learning, even among audiences receiving in-person training. Several online format options could be considered. Audio and video presentation of material could be supplemented with video and photographic clips from the in-person train- ing. This format could include interactive questions from multidisciplinary perspectives (the answers would reflect each disciplineâs perspective to help mimic the insights gained in the in-person training format) or discussions within a multi- disciplinary group. Each lesson would include an online test. Interactive games, quizzes, and real-life scenarios as comple- ments to specific modules to focus on kinesthetic skills and references to related source materials would provide a more in-depth experience. Yet another option, live and recorded online seminar-style delivery with optional live interactive chat with the instructors, would be similar to formats offered by online universities. More specific web-based options to consider include an online TIM university, in which a media-rich online version test courses four through nine and making one final set of revisions and modifications to the course. A final course delivery then would be conducted to validate all changes and to establish the official baseline training course and supporting materials. The research team recommends that the preimplementa- tion course, or field tests, be delivered around the country, both in areas with emerging TIM initiatives and in areas that already have well-established programs. To test the course applicability and robustness, it is envisioned that the course should be offered in a mix of urban and rural areas and in multistate corridors. Stakeholder associations, such as the International Association of Chiefs of Police, the U.S. Fire Administration, the American Association of State Highway Transportation Officials, and the Towing and Recovery Asso- ciation of America, should be approached to examine oppor- tunities for sponsorship of the field tests and to discuss the possibilities of integration of the program into their training initiatives. State public safety academies may include fire, EMS, and law enforcement training or establish a level of interdisciplinary cooperation. Fire training agencies are com- fortable with the National Fire Protection Association stan- dard system, which may be helpful in introducing the concept of national TIM training. Organizations already teaching TIM concepts include the Cumberland Valley Volunteer Firemenâs Association, the Emergency Responder Safety Institute, the American Traffic Safety Services Association, the I-95 Corri- dor Coalitionâs incident management virtual training pro- gram, and the First 30 initiative. Opportunities to integrate this training into the programs these institutions offer also should be explored. Because a critical component of the success of this initia- tive is to ensure the transferability of the content and teach- ing methods, the model by which potential instructors are trained and validated as meeting the criteria deemed neces- sary to successfully teach this course should be further refined. Throughout the field tests, feedback should be gathered, for- mally and informally, from attendees and agencies and incor- porated into the course. The geographic diversity and makeup of the classes will ensure that the course caters to the needs of the responder community nationwide. Mechanisms should be developed to measure the best way to transition the novice instructors to the classroom environ- ment: options include coteaching with a master instructor, observing and being observed by a master instructor for one or more training sessions, offering some limited virtual coach- ing before the first class, and allowing the instructor to directly begin to teach, using student feedback forms and knowledge test results to identify any potential areas needing improve- ment in teaching delivery. Ideally, feedback on novice instruc- tors should be sought over the period of the first three course
25 the PowerPoint presentation to illustrate to potential instruc- tors how to perform this lesson. Suggested research: Discipline-Specific training The TIM course was developed as a multidisciplinary course intended for a multidisciplinary audience, but the day-to-day reality is that, for many disciplines, it may be advantageous to insert excerpts from the course into existing training or incorporate them into already established routines. For exam- ple, course information could be distributed during weekly fire department safety meetings or to patrol officers during briefing sessions at shift change. It is worth investigating whether the content contained in the multidisciplinary TIM course could be converted into smaller segments delivered through discipline-specific channels (e.g., fire academies) while preserving the multiagency, multidisciplinary values and emphasis of the course. Additional research should be conducted to examine whether content contained in this multidisciplinary train- ing could be extracted and act as stand-alone material aimed at a single discipline for use as part of different ini- tiatives. Although the core objectives and message of the training would remain the same, it is likely that the focus would shift depending on the audience. For example, it is anticipated that the course would focus on different areas if it were delivered to a group of 9-1-1 center operators and dispatchers as opposed to transportation or law enforce- ment personnel. Additional research should be conducted to determine the most appropriate training medium or format for each discipline. When completed, field tests aimed at discipline- specific delivery models should be conducted. These field tests would allow for analysis of the feasibility of convert- ing the course into single-discipline training and the degree to which students develop the appreciation for the multi- disciplinary, multiagency emphasis of the core competen- cies that occurs in the multidisciplinary in-person delivery environment. Such tests also would reveal which aspects of the entire course content are most relevant to specific dis- ciplines and the format that would be most useful for each discipline. It is recommended that six discipline-specific field tests be run (one for each major TIM discipline; see Table 4.2) using an instructor and students from that discipline only. Obser- vation and analysis of Kirkpatrick Level 1 (reaction) and Level 2 (learning) evaluation data gathered during these field tests from student evaluation forms and knowledge tests at the completion of the training should be used to modify each discipline-specific version of the course if discipline-specific training is determined feasible. of the course would combine live facilitated webcasts with an assigned instructor with precreated web-based lessons. The online facilitated modular sessions could be interspersed with the precreated content to give them more of a live feel. Students could be assigned to multidisciplinary groups that could go through the training together as in a traditional classroom over a predefined time period, such as 6 weeks. In this case, students would complete segments, activities, or assignments in a given time frame. The group would meet in an online chat room or community with the instructor, which would give the training more of a classroom feel. Stu- dents could be assigned to a group based on their responder discipline to retain the desired multidisciplinary flavor. If online TIM university classes were conducted regionally, they could be combined with a traditional classroom session for some activities. Another option would involve creating web-based training lessons that would be a prerequisite to attending the class- room training. Segments of content could be converted to web-based training that students would complete individu- ally to shorten the duration of the in-person classroom train- ing. Having students complete some of the introductory lessons in advance would allow instructors to focus more intensively on the multidisciplinary collaborative lessons during the available classroom time. This type of training, called blended learning, may be an attractive option for regions whose agenciesâ employees or volunteers are unable to commit to 2 full days away from the job. In contrast to the blended learning model, the entire cur- riculum could be converted to web-based training that would serve as a stand-alone equivalent option to classroom train- ing. A similar model is the Federal Emergency Management Agencyâs independent study program. The target audience would be those responders who do not have the flexibility to attend even a 1-day course or who prefer an online format. As an alternative option, selected core elements of the course could be converted. Students completing this version of the online training would benefit from the information but would not receive the same completion certificate as those taking the complete course. As a more specific example of the value of web-based instruction, a video of the situational awareness field practi- cum would permit a cost-effective presentation of the out- door field practicum content in environments in which it would not otherwise be feasible, such as for online-only audi- ences, in-person training venues experiencing inclement weather, or when responder vehicles are not available. An online video would permit some flexibility for agencies unable to participate in 2 full days of training because it would allow students to obtain the knowledge conveyed in this field segment. This video also could be used in the train- the-trainer section as a supplement to the still photographs in
26 control early rollouts of the training to ensure that the core values and key principles for long-term success of the train- ing are honored: quality, qualified trainers; multidisciplinary perspective and emphasis; and structured collection of feed- back from students both immediately after and 1 year after training. The research team also recommends that this central entity initially lead marketing for the training, including develop- ment and maintenance of the central website (the nucleus of the marketing). This approach will help to establish a brand for the training, ensure that core messages are clear, and show that a reliable one-stop shop containing all information about this training is available. This website would be the home of online training for alumni forums and communi- ties of practice and a platform for soliciting feedback from students. Distributed marketing should be strongly encouraged to raise awareness about and generate interest in the training and to attract audiences to the website to learn more. National TIM proponents would learn about leveraging key stake- holder channels, including word-of-mouth endorsements and marketing messages from associations and academies, such as the National Fire Protection Association, the National Association of State EMS Officials, and the International Association of Chiefs of Police. Marketing messages could include links from these organizationsâ websites to the central TIM website, mentions in their newsletters and at events of this training, and encouragement to their audiences to obtain this training. In addition to developing and maintaining the central website and marketing messages, the central entity would be responsible for the cost-effective development of other col- lateral such as brochures. These materials would be made available for use by national associations and regional and local agencies to raise awareness in their markets about the training. The central website would provide all stake holder organizations with immediately available marketing resources that they may not have the resources to develop on their own. As organizations acquire qualified trainers, they would be encouraged to conduct their own marketing to attract stu- dents to the training and would be welcome to cobrand or supplement these materials with others as desired. The research team recommends a variety of additional actions to help further refine, mature, and institutionalize this training curriculum in its intended market: â¢ Feedback should be solicited on the curriculum and train- ing methods from new participants immediately after they receive the training and from alumni within 1 year of the training, after they have had the opportunity to put their new knowledge into practice. This feedback could be col- lected on the website and structured by lesson; competency Measuring Learning transfer and results Although learning is the obvious expected outcome of any training, the most critical aspect of any learning interven- tion is studentsâ ability to transfer knowledge back to the job and have their behavioral changes positively affect the situation that initially prompted the learning intervention. The Kirkpatrick Level 1 (reaction) data show that studentsâ reactions to the training were positive, and the Level 2 (learn- ing) results show that they were able to demonstrate profi- ciency in the content, as discussed in Chapter 3 in the section on assessment of training effectiveness. It is not yet known whether those skills are being transferred into everyday work practices or if they are having the intended effects on improv- ing reliability. Conducting further evaluation on the effects of the course would require Level 3 and 4 evaluations. A Level 3 evaluation explicitly measures a studentâs ability to transfer learning to the job and the degree to which students have applied the training or knowledge to their jobs. It is generally conducted a few weeks to 3 months after training with a studentâs super- visor and ideally involves measuring on-the-job observable behaviors, as well as conducting interviews. A Level 4 evalu- ation measures the impact of the training on the organization or on reliability. This evaluation is achieved by measuring quantifiable changes in key performance indicators by means of interviews, questionnaires, or focus groups 3 months to 1 year after training. Leadership and Oversight for the training The research team recommends central oversight and admin- istration of this training for approximately 3 to 5 years to allow the training and curriculum to become accepted as a national standard within the designated market of multi- agency federal, state, and local TIM responders across the country. Central oversight and administration will help to Table 4.2. Discipline-Specific Training Field Tests Field Test Audience Field Test 11 Law enforcement Field Test 12 Fire and rescue Field Test 13 Emergency medical services Field Test 14 Transportation and service patrol Field Test 15 Towing and recovery Field Test 16 Notification and dispatch
27 training periods in their annual calendars, the new curricu- lum releases could be timed to align with and support these time frames. Alternatively, an annual release of updated train- ing would help stimulate anticipation, similar to commercial launches that use known timelines for the release of much- anticipated products. Market receptivity to and interest in the advantages of formally certifying trainers and students who complete the training should continue to be monitored, as well as interest in accrediting the training. Accreditation would facilitate dis- tributed (franchised) training opportunities and professional education credits for attendees. Within 3 years, as market acceptance of the idea of national TIM training grows and the curriculum has matured and evolved by extensive practi- tioner feedback, it is expected that the market will support certification and accreditation of this training. Such support is possible if the training is perceived as the representative standard of TIM practice and if practitioners who have taken the training report safer, quicker incident clearance. Certifying and accrediting authorities (or a single author- ity) should be identified, and a model for implementing certification and accreditation should be established that preserves the core value of an evolving curriculum continu- ally shaped by field practitioners as the state of the practice evolves. When a certification and accreditation model is in place, certified and accredited training delivery organiza- tions will be free to conduct their own marketing to attract students to the training. The need for a centralized website likely will become obsolete at this point. or practice; and the name, title, role, and experience level of the submitter. â¢ The training curriculum should be positioned as a liv- ing, collaborative curriculum that is updated at least annually by the community of expert TIM practitioners across the country to reflect the latest state of TIM prac- tice. A formal annual TIM summit with selected, nomi- nated SMEs, assisted by a facilitator, should systematically review the feedback collected over the course of the pre- vious year. SMEs will formally adjudicate feedback and identify specific changes to the curriculum baseline. The central administering authority will manage the curricu- lum with stringent configuration management controls to ensure traceability for any changes to the curriculum. The basis for every change will be explicitly documented to provide a knowledge base on the evolution of the train- ing curriculum. â¢ The training should also be positioned as national TIM training developed by and for TIM practitioners to stimu- late a sense of ownership and trust among the broad TIM community for this curriculum as their curriculum, spe- cifically designed to help them be as effective as possible in achieving safe, quick clearance. As updated training materials are developed and formally released, they should be available through the central website; all state and local agencies can subscribe to RSS feeds to be automatically notified when a new and updated curriculum is available. If key stakeholder organizations have standard