Leveraging Technology for Transportation Agency Workforce Development and Training (2017)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

14 chapter three State of the Practice overview To document the state of ICT-supported training efforts at state and local roadway transportation agencies, a survey of practice was conducted through the NCHRP in cooperation with the National Transportation Training Directors organization and LTAP/TTAP programs. The survey instrument was distributed to the National Transportation Training Directors organization’s members and all LTAP and TTAP centers. These efforts resulted in a total of 78 completed responses, including 40 (of 50) state DOT responses and 38 (of 58) LTAP and TTAP responses, as shown in Figure 1. These numbers represent an 80% response rate among state DOTs and a 66% response rate among LTAP and TTAP centers. TTAP responses are not shown on the map. The Alaska, Eastern, Northern Plains, and Western TTAPs responded to the survey. This chapter summarizes the findings from the survey of practices. The information is presented in a number of formats, including tables and graphs, as appropriate. The survey questions, which were distributed electronically, appear as Appendix A (only in the online version of this report), and the responses received are presented in Appendix B. In the table and figure captions, the total number of state DOT and LTAP/TTAP respondents is recorded. In the discussion, the percentage of state DOT and LTAP/TTAP responses is based on the total number of respondents (single answer questions) or the number of respondents to each answer option (multiple answer questions). Survey content The survey questions are organized into the following five categories: • ICT Usage—This section of the survey asks respondents whether they are providing ICT-related training and the types of ICT used. Agencies not providing ICT-related training were able to iden- tify reasons for not doing so before advancing to the final survey section “Future ICT Efforts” and exiting the survey. • Implementation of ICT—This section of the survey asks respondents to identify the reasons for adopting ICT-supported training, its main uses, models for implementation planning, and the implementation tasks completed. • ICT-Supported Training Design and Delivery—This section asks respondents to identify the types of content being taught by ICT and how ICT-supported training is developed and main- tained (e.g., in-house, contract, partnership, or third-party vendor). Agencies are also asked to identify the staffing and budgetary resources used to support their ICT-supported training programs. • Tracking and Evaluating Training—This section asks respondents to explain how participant registration and completion of ICT-supported training and evaluations are tracked and reported and the types of evaluations being administered after training. • Future ICT Efforts—This section of the survey asks respondents to identify any future plans the agency has for ICT-supported training. The results of the survey are presented here.

15 information and communication technologieS uSage The first section of the survey pertains to ICT usage. This section of the survey posed questions about whether agencies were using ICT and, if so, what types of ICT they were using. Agencies responding that they were not using ICT were asked to identify the reason(s) for this decision. Of the 40 state DOTs and 38 LTAP/TTAPs that responded to the survey, 34 (85%) DOTs and 17 (45%) LTAP/TTAPs reported that they are using ICT to support training (see Table 1). As shown in Table 2, the most widely used type of ICT-supported training for state DOTs and LTAP/TTAPs is web-based training, followed by computer-based training. (As noted in “Survey Limitations,” survey responses indicated some agencies use the terms “web-based training” and “computer-based training” interchangeably.) A significant number of state DOTs are also using web conference and video conference training methods (44% and 41%, respectively), although this usage is less evident with LTAP/TTAPs (12% and 18%, respectively). Mobile usage is not as widely employed (29% of state DOTs and 35% of LTAP/TTAPs), and four agencies acknowledged using the incorrect definition of “mobile training” when replying to the survey. The revised results reveal FIGURE 1 NCHRP Topic 47-04 response record for state DOTs and LTAPs. TABLE 1 AgENCIES USINg ICT LTAP/TTAP State DOT Number Percentage Number Percentage Yes 17 45 34 85 No 21 53 6 16 Of 40 state DOT and 38 LTAP/TTAP responses.

16 that six (18%) state DOTs and six (35%) LTAP/TTAPs have mobile training (i.e., training specifically designed as a mobile application and delivered by means of a mobile device). Several agencies selected “other” as a response and listed a responsive website and videos as other types of ICT-supported training being used. The responses also show that 27 state DOTs and ten LTAP/TTAPs use more than one type of ICT. The 27 agencies that reported not using ICT were asked to explain the reasoning behind this decision. Respondents were able to select as many answers as were applicable and had the option to provide a written response. As shown in Figure 2, the main reason reported was the belief that more traditional methods of training (e.g., on-the-job) are more successful. This response was selected by three state DOTs and ten LTAP/TTAPs. Ten LTAP/TTAP respondents also selected the lack of a budget and technical staff to develop or deliver ICT-supported training as reasons for not using ICT-supported training. One state DOT and eight LTAP/TTAPs reported they did not have a content management system or LMS to administer ICT-supported training, whereas eight LTAP/TTAPs reported not having the necessary hardware or software required to implement ICT-supported train- ing. Nine agencies selected “other” reasons for not using ICT to support training. The reasons pro- vided were the lack of computer access, lack of enthusiasm among the target audience for ICT training, past issues with ICT training, a lack of knowledge about ICT, and an inability to sustain ICT training. Several agencies also stated they are planning to use ICT but need to acquire or implement an LMS, evaluate ICT-supported training, or develop content. After completing this question, most respondents not providing ICT-supported training advanced to the “Future of ICT” section of the survey before exiting. FIGURE 2 Reasons for not using ICT to support training. FIGURE 2 Reasons for not using ICT to support training. TABLE 2 TyPES OF ICT-SUPPORTED TRAININg BEINg USED LTAP/TTAP State DOT Number Percentage Number Percentage Web-based training 9 53 30 88 Computer-based training 7 41 25 74 Web conference training 2 12 15 44 Video conference training 3 18 14 41 Mobile 6 35 10 29 Other 5 29 2 6 Of 34 state DOT and 17 LTAP/TTAP responses.

17 imPlementation of information and communication technologieS In this section, agencies were asked to respond to questions pertaining to their original reasons for acquiring ICT, reasons for using it to conduct training, the planning required to implement ICT, and the purpose(s) ICT currently serves. As shown in Table 3, the respondents (50% of LTAP/TTAPs and 49% of state DOTs) stated that ICT was acquired to conduct training, although a significant portion (30% of LTAP/TTAPs and 40% of state DOTs) acquired ICT to facilitate meetings and other nontraining interactions. The reasons for using ICT to support training were more varied. For this question, respondents were able to choose multiple answers and had the option to provide a written response. As Figure 3 shows, the most common reasons selected by state DOTs for using ICT to support training were to provide FIGURE 3 Reasons for implementing use of or choosing to use ICT-supported training. TABLE 3 REASON ICT WAS ORIgINALLy ADOPTED/IMPLEMENTED By THE AgENCy Of 35 state DOT and 20 LTAP/TTAP responses. LTAP/TTAP State DOT Number Percentage Number Percentage Conduct training 10 50 17 49 Facilitate meetings and other types of communication but not to conduct training 6 30 14 40 Don’t know 1 5 3 9

18 employees with training flexibility for accessing and completing training and reduce delivery costs (85% of state DOTs, respectively). LTAP/TTAPs most frequently cited providing employees with training flexibility for accessing and completing training (79% of LTAP/TTAPs). Other agencies wanted to provide opportunities for geographically diverse audiences to participate in synchronous training sessions (74% of state DOTs and 63% of LTAP/TTAPs). Three agencies selected “other” reasons for using ICT to support training. The reasons provided were to preserve knowledge and support succession planning, provide shorter “bits” of training as required by evolving training meth- odologies and learning trends, and meeting usage. As the results of the previous question indicate, agencies responded that ICT-supported training replaces content delivered through traditional means (74% of state DOTs and 53% of LTAP/TTAPs). These results are documented in Figure 4. In addition, a number of agencies reported using ICT pre- dominantly to develop supplemental training for existing training (65% of state DOTs and 63% of LTAP/TTAPs). A smaller number of agencies stated that ICT is predominantly used when developing new training courses (38% of state DOTs and 5% of LTAP/TTAPs). Agencies used different models to plan the implementation of ICT-supported training (see Fig- ure 5). Other government agencies outside of transportation were often used (59% of state DOTs and 26% of LTAP/TTAPs), as were private sector companies (56% of state DOTs and 32% of LTAP/ TTAPs) and other transportation agencies (47% of state DOTs and 26% of LTAP/TTAPs). Fifteen agencies selected “other” for this question and listed the following models for implementation: uni- versities, online research, existing internal models, and the TCCC. FIGURE 4 Use of ICT-supported training.

19 To make ICT-supported training available, agencies had to complete several tasks (see Figure 6). The main task was the acquisition of hardware and software (76% of state DOTs and 53% of LTAP/TTAPs). Many agencies also acquired an LMS (50% of state DOTs and 47% of LTAP/TTAPs). Several agen- cies also identified an ICT training provider (38% of state DOTs and 12% of LTAP/TTAPs) and hired support staff (24% of state DOTs and 24% of LTAP/TTAPs). Of the 14 agencies that selected “other” for this question, additional tasks included gaining access to an existing LMS outside of the agency, reassigning job responsibilities to existing staff, assigning new project developments, upgrading existing systems, and learning how to operate new hardware and software. FIGURE 5 Models used to plan the implementation of ICT-supported training. FIGURE 6 Actions completed to make ICT-supported training available.

20 deSigning and delivering information and communication technologieS–SuPPorted training In this section of the survey, agencies answered questions about their ICT training development activi- ties. The first questions addressed the content being taught. Following were procedural questions to determine how ICT is designed, who is involved in the development, and how training alignment with employee performance criteria and workforce development objectives is verified. Additional questions focus on the use of internal and external resources to develop and maintain ICT-supported training and the procedures followed to maintain the courses. Agencies use ICT to deliver training in every one of the content areas, as shown by Figure 7. For state DOTs, human resources and personnel training is the content delivered most often using ICT (74%). For LTAP/TTAPs, traffic safety training is the content delivered most often using ICT (65%). Other content areas for which the states regularly use ICT delivery methods are construction, maintenance, finance and administration, and emergency management (56%, 53%, 53%, and 50%, respectively). Other content areas for which the LTAP/TTAPs regularly employ ICT delivery methods are construction and maintenance (41% and 47%, respectively). The types of training provided by the respondents who selected “other” include tribal sovereignty and government, safety, executive man- dates, leadership development, information technology technical, project management, procurement, contracts, railroad safety, and land surveys. designing ict-Supported training Agencies were asked to identify how delivery methods are selected. The responses to this question are shown in Table 4. Agencies rely on the training design plan to recommend a delivery method, which takes into account audience, budget, and the type of content, among other considerations. Some agencies select ICT to deliver training when a subject matter expert (SME) requests this method specifically (20% of state DOTs and 30% of LTAP/TTAPs). One respondent reported the agency policy is to deliver all training using ICT; another reported that trainings developed with a specific partner are delivered using a specific type of ICT. Another respondent reported that in some instances the agency performs an in-class training pilot, after which learners identify content to be taught using ICT. Finally, one agency reported the process for assigning ICT as the delivery method for training is informal. FIGURE 7 Content being delivered via ICT.

21 The roles assigned to ICT-supported training development projects also show similarity across agencies. As Figure 8 shows, almost all agencies include an SME when developing ICT-supported training (90% of state DOTs and 100% of LTAP/TTAPs). The use of an instructional designer is less common but still has a significant role for most agencies (63% of state DOTs and 50% of LTAP/ TTAPs). Variations in the roles are more apparent for ICT support or technical staff (47% of state DOTs and 70% of LTAP/TTAPs), web developers (20% of state DOTs and 30% of LTAP/TTAPs), and graphic designers (17% of state DOTs and 10% of LTAP/TTAPs). Several agencies reported the use of training staff, specialists, experts, or coordinators in development projects, as well as manage- ment sponsors and human resources (which includes training). One respondent reported the agency has an employee development team responsible for producing much of the agency’s ICT-supported training. The team includes five specialists (training experts, instructional designers, etc.) who work with partners and the target audience to produce training content. In addition to what content is taught by ICT, it is important to know how well the content aligns with the intended audience’s performance criteria or workforce objectives. This allows for a greater understanding of the relevance of the content. As shown in Table 5, most state DOTs and LTAP/ TTAPs reported that the content taught by ICT was aligned (82% and 76%, respectively). LTAP/TTAP State DOT Number Percentage Number Percentage Training design plan includes a recommendation for delivery method. Recommendation takes into account audience, budget, and appropriateness of the content for specified delivery method. 5 50 18 60 SME requests training for content using ICT-supported method. If money is available to meet request, project is completed. 3 30 6 20 All of our new training is delivered via one type of ICT, so no selection criteria are needed. 1 10 0 0 Other 0 0 4 13 Don’t know 1 10 2 7 Of 30 state DOT and 10 LTAP/TTAP responses. TABLE 4 PROCEDURE FOR SELECTINg ICT DELIVERy METHODS FOR TRAININg FIGURE 8 Roles typically assigned to an ICT-supported training development project.

22 Agencies reporting that training content aligns with performance evaluations or workforce develop- ment objectives were asked how alignment is determined (see Table 6). Respondents were able to select as many answers as were applicable and could provide a written response. It appears that most state DOTs use training design plans to verify alignment. The LTAPs also use design plans, but a large number also use performance evaluations to verify alignment because such documents specify ICT-supported trainings that must be completed. Agencies documented the use of needs assessments to identify work- force development objectives, surveys, and employee input as “other” ways to verify alignment. developing or acquiring information and communication technologies–Supported training How agencies develop or acquire training appears to vary based on the type of ICT (see Figure 9). Web-based training, the most widely used type of ICT-supported training, typically is developed using in-house resources by all agencies. As shown in Figure 10, state DOTs also develop or acquire web-based training through agency partners or by hiring a contractor or consultant (63% and 40%, respectively). Third-party vendors are also used by the state DOTs to acquire web-based training, but only one LTAP/TTAP acknowledged using this option. Computer-based and video conference training had similar results, with respondents for all agen- cies using in-house resources for development (see Figures 11 and 12). State DOTs also use agency partners, contractors or consultants, or third-party vendors to acquire or develop computer-based training (36%, 28%, and 52%, respectively), but only one LTAP/TTAP respondent indicated the agency used a contractor or consultant for the same purpose. In terms of video conference training, which LTAP/TTAPs used less often to deliver training, only one agency reported purchasing training through a third-party vendor. State agencies use web conferencing much more frequently than do LTAP/TTAPs and reported using a greater variety of approaches for developing or acquiring this type of training (see Figure 13). Although using in-house resources to develop web conference training received the most responses (73% of state DOTs), the use of agency partners is common (67% of state DOTs). Purchasing web LTAP/TTAP State DOT Number Percentage Number Percentage Clearly aligned 13 76 28 82 Not aligned 4 24 6 18 Of 34 state DOT and 17 LTAP/TTAP responses. TABLE 5 ALIgNMENT OF ICT-SUPPORTED TRAININg AND TARgET AUDIENCE PERFORMANCE CRITERIA OR WORkFORCE DEVELOPMENT OBjECTIVES LTAP/TTAP State DOT Number Percentage Number Percentage Training design plan documents which criteria or objectives are addressed 5 38 17 61 Performance evaluations document specific ICT-supported trainings that must be completed 3 23 11 39 Don’t know 3 23 4 14 Other 3 23 4 14 Of 28 state DOT and 13 LTAP/TTAP responses. TABLE 6 VERIFICATION OF CONTENT ALIgNMENT WITH PERFORMANCE CRITERIA OR WORkFORCE DEVELOPMENT OBjECTIVES

FIGURE 9 Approaches to developing or acquiring ICT-supported training. FIGURE 10 Approaches to developing web-based training. FIGURE 11 Approaches to developing computer-based training.

24 conference training through a third-party vendor or developing with a hired consultant or contractor also represented a significant number of responses (53% and 40% of state DOTs, respectively). Mobile training is not widely used, and follow-up interviews indicate that several agencies that reported “mobile training” as a delivery method are referring to web-based training that is com- pleted using a mobile device (which is not the definition of mobile training used in this report). With these cautions in mind, responses to this question reflect a variety of development approaches to ICT-supported training (see Figure 14). For mobile training, LTAP/TTAPs rely mostly on in-house resources to develop products (80%), whereas state DOTs acquire these products through a third- party vendor (90%). In contrast, purchasing through a third-party vendor is one of the least-used approaches of the LTAP/TTAPs (20%). The use of contractors or consultants to develop mobile training is the same for both types of agencies (40% of state DOTs and 40% of LTAP/TTAPs). Partnering to acquire ict-Supported training It benefits transportation agencies in many instances to work with partners to develop ICT-supported training. This often allows agencies to acquire a more robust, comprehensive product than if working FIGURE 12 Approaches to developing video conference training. FIGURE 13 Approaches to developing web conference training.

25 alone. As survey results in the previous discussion indicate, agencies are using partners to develop or acquire ICT-supported training. This section seeks to identify the types of partners being sought and the nature of those relationships. The first question in this section asked respondents to identify the agencies with which they partnered to develop or acquire ICT-supported training (see Figure 15). A significant num- ber of state DOTs (67%) responded that they participate in the TCCC. For a membership fee, TCCC members gain access to all of the web-based trainings posted to the council’s LMS and can provide input to the development of new web-based training products. State DOTs also responded that they partner with community colleges, Clear Roads, and the American Public Works Association (APWA) to develop or acquire ICT-supported training. A small percentage of state DOTs and LTAP/TTAPs also partner with university transportation centers (UTCs). The state DOTs and one LTAP/TTAP also identified a large number of “other” partners, includ- ing FHWA (LTAP and state DOTs), the National Transportation Training Directors organiza- tion, NHI, E-Safety, state department of human resources development, state department of accounting and general services, AASHTO, governor’s office of employee relations, local colleges/university, state office of administration (not transportation-specific topics), state office FIGURE 14 Approaches to developing mobile device training. FIGURE 15 Agencies’ partners for the development or acquisition of ICT-supported training.

26 of management and enterprise services (human capital management division), and the Transportation Learning Network. As shown in Tables 7 and 8, training developed through partnership typically is owned by the state DOT (67%) but usually is not subject to copyright (75%). The two TTAP centers responding to this question reported that ICT-supported training products developed or acquired through a partnership are not owned by the TTAP and are not copyrighted. The final question in this section addressed how partnering agreements, either for a single or multiple ICT-supported training developments, addressed updates and maintenance of the product. As shown in Table 9, most state DOTs (52%) and one LTAP have provisions for how updates and maintenance are handled. Several respondents from state DOTs reported they did not know whether their agency’s partnering agreements specified how updates and maintenance are handled (29%). Purchasing ict-Supported training For agencies with limited staff expertise, budget, or partnering opportunities, purchasing ICT- supported training is a viable option. As with partnering agreements, agencies are often able to purchase more robust, comprehensive, ICT-supported training that they can develop in-house. Purchasing ICT-supported training through a third-party vendor can also be a good option if the pur- chasing agreement includes technical support. As shown in Figure 16, the state DOTs and LTAPs predominantly purchased from NHI and AASHTO (68% and 64% for state DOTs, 100% and 100% for LTAP/TTAPs). To a lesser extent, state DOTs used TRB, community colleges, and UTCs to purchase training (23%, 18%, and 9%, respectively). Only one LTAP indicated using TRB, and two LTAP/TTAP State DOT Number Percentage Number Percentage Yes 0 0 14 67 No 2 100 4 19 Don’t know 0 0 3 14 Of 21 state DOT and 2 LTAP/TTAP responses. TABLE 7 TRAININg OBTAINED THROUgH A PARTNERSHIP IS OWNED By THE AgENCy LTAP/TTAP State DOT Number Percentage Number Percentage Yes 0 0 1 25 No 2 100 3 75 Of 4 state DOT and 2 LTAP/TTAP responses. TABLE 8 TRAININg OBTAINED THROUgH A PARTNERSHIP IS COPyRIgHTED LTAP/TTAP State DOT Number Percentage Number Percentage Yes 1 50 11 52 No 1 50 4 19 Don’t know 0 0 6 29 Of 4 state DOT and 2 LTAP/TTAP responses. TABLE 9 PARTNERSHIP AgREEMENT INCLUDES A PLAN FOR UPDATINg AND MAINTAININg THE PRODUCT

27 indicated using APWA. “Other” third-party vendors identified by state DOTs and LTAPs included ITE, FHWA, LearnSmart®, Microsoft, American Society for Testing and Materials, equipment ven- dors, BLR, RedVector, private consultants, Regis, and state universities. State DOTs and LTAP/TTAPs that purchased ICT-supported training were equally divided in their ability to acquire the latest version of the product once the initial purchase had been made (see Figure 17). Fifty percent of state DOTs and LTAPs obtained new versions of the purchased products free of charge, whereas 50% of LTAPs and 45% of state DOTs had to purchase new versions at full price. Only one respondent from a state DOT reported the agency obtained new versions at a discounted price. One-half of all agencies did not know if the purchasing agreement entitled them to support services (see Table 10). The remaining state DOTs were almost equally divided between receiving support services and not receiving services (27% and 23%, respectively). Those agencies that do receive support services from the vendor are getting help with initial deployment of the ICT-supported training product, customer support for users, and troubleshooting for technical issues, such as integration with the LMS (see Figure 18). FIGURE 16 Third-party vendors used to acquire ICT-supported training. FIGURE 17 How agencies acquire the latest version of ICT-supported training.

28 development costs The cost associated with developing 1 h of ICT-supported in-house training as reported by most agencies falls into the range of $1,000 to $5,000 (see Figure 19). Although some state DOTs and LTAP/TTAPs had expenditures in the middle range of $5,000 to $10,000 for various ICT types, no LTAP/TTAPs and only two state DOTs reported spending $10,000 to $15,000 to develop 1 h of LTAP/TTAP State DOT Number Percentage Number Percentage Yes 1 50 6 27 No 0 0 5 23 Don’t know 1 50 11 50 Of 22 state DOT and 2 LTAP/TTAP responses. TABLE 10 PURCHASE AgREEMENT INCLUDES A PLAN FOR UPDATINg AND MAINTAININg THE PRODUCT FIGURE 19 Typical amount spent to develop 1 h of ICT-supported training, sorted by type (CBT = computer-based training, VCT = video conferencing training, WBT = web-based training, WCT = web conference training). FIGURE 18 Types of support services provided by third-party vendors.

29 ICT training. Although not documented in the survey responses, case example interviews indicate that the costs reflected in these estimates typically represent time and labor for in-house staff, such as the SME and instructional designer, to develop the training product. updating and maintaining training As with any other type of training, the cost of ICT-supported training needs to reflect more than the price to develop. There also needs to be consideration for how the training is to be updated and maintained. The questions in this section of the survey pertain to the agency’s policy and procedures for keeping training updated and maintained, both for products developed in-house and those out- sourced. In the following sections, additional information is provided on how products developed in partnership with other agencies or acquired through a third-party vendor may differ in how they are updated or maintained. The responses to the first question in this section of the survey are shown in Table 11. Respondents were asked whether a plan existed for updating and maintaining ICT-supported content developed in-house. Most agencies responded in the affirmative (67% of state DOTs and 70% of LTAP/TTAPs). If an agency responded “yes” to the previous question, the respondent was asked to describe the agency’s plan for updating ICT-supported content. As shown in Figure 20, 80% of state DOTs and 86% of LTAP/TTAPs have a specific schedule for reviewing course content and generating recommendations to update the training. Almost an equal percentage of state DOTs (75%) have no formal schedule, allowing reviews to be triggered by SME requests or low course evaluation scores. LTAP/TTAP State DOT Number Percentage Number Percentage Yes 7 70 20 67 No 2 20 8 27 Don’t know 1 10 2 7 Of 30 state DOT and 10 LTAP/TTAP responses. TABLE 11 ExISTENCE OF A PLAN TO UPDATE AND MAINTAIN ICT-SUPPORTED CONTENT DEVELOPED IN-HOUSE FIGURE 20 Agency plans for updating ICT-supported content.

30 Far fewer LTAP/TTAPs (29%) employ these methods. Several state DOTs noted that this schedule can vary depending on the department responsible for the training and the content and whether it was purchased through a third-party vendor. One state DOT noted that leadership reviews training as well. When changes are required to existing ICT-supported training content, in-house staff is usually tasked, as shown in Table 12 (93% of state DOTs and 100% of LTAP/TTAPs). State DOTs also docu- mented significant use of consultants or contractors to complete this task (40%). In one instance, a state DOT identified partnering with an educational institution in conjunction with using an internal person to make the changes. The agencies that use internal staff to complete training updates were asked to identify who these employees are by title (see Figure 21). Most employees completing updates were identified as human resources or training staff (57% of state DOTs and 40% of LTAP/TTAPs). For state DOTs, the second most likely group to complete this work is SMEs (21%), but for LTAP/TTAPs it is more likely to be an information technology (IT) department employee with sufficient technical skills (30%). One LTAP said that both training and IT work together, one state DOT said SMEs and IT work together, another said SMEs and training work together, and one said that a combination of training, SMEs, and IT work together to complete updates. The agencies that use internal staff to complete training updates were asked to identify the amount of time the in-house employee spends on training versus nontraining tasks (see Figure 22). It was almost an equal split between those who are solely responsible for training work and those who also complete nontraining related tasks (46% versus 50% for state DOTs and 50% versus 40% for LTAP/TTAPs). LTAP/TTAP State DOT Number Percentage Number Percentage In-house agency staff 10 100 28 93 Consultant or contractor 1 10 12 40 Other 0 0 3 10 Of 30 state DOT and 10 LTAP/TTAP responses. TABLE 12 WHO UPDATES ExISTINg TRAININg? FIGURE 21 Agency internal staff used to complete ICT updates.

31 The types of additional tasks assigned to LTAP/TTAP in-house staff updating ICT-supported train- ing included project management, web design, newsletter editing, assisting with registration calls and functions, supporting the Pedestrian Bike Safety Resource Center project, vehicle and building main- tenance, and desktop support. State DOT in-house staff were also assigned to system administration for the pavement management system, employee and labor relations, classification, human resources, video production, photography, career consulting, multimedia needs, roadway design, maintenance, knowledge management, database maintenance, and IT support. tracking and evaluating ict-SuPPorted training As the previous survey responses have shown, state DOTs, LTAP, and TTAPs are investing significant time and money in ICT-supported training. To determine whether that investment is having the desired impact on performance, both for the individual employee and the workforce as a whole, agencies ideally are tracking employee completion and mastery of the training, obtaining feedback, and ana- lyzing the data collected to determine the training’s impact. This is referred to as the ideal because it is recognized that not all agencies have the resources to complete these activities. This section of the survey seeks to determine the extent to which these activities are taking place at agencies and identify issues that might be prohibiting their execution. The first question in this section asks whether agencies have an LMS. As shown in Table 13, most agencies have an LMS (79% of state DOTs and 76% of LTAP/TTAPs). In addition, as shown in Table 14, most agencies are tracking completion of ICT-supported training. As shown in Figure 23, most agencies are using the agency’s LMS to track training (73% of state DOTs and 86% of LTAP/TTAPs). State DOTs are also tracking com pletion through a third-party LMS (36%) or have learners send evidence of completion (e.g., a screenshot of the final screen, a FIGURE 22 Training versus nontraining responsibilities for in-house staff. LTAP/TTAP State DOT Number Percentage Number Percentage Yes 13 76 27 79 No 4 24 7 21 Of 34 state DOT and 17 LTAP/TTAP responses. TABLE 13 AgENCIES WITH A LEARNINg MANAgEMENT SySTEM

32 FIGURE 23 How agencies track ICT-supported training. LTAP/TTAP State DOT Number Percentage Number Percentage Yes 14 82 33 97 No 3 18 1 3 Of 34 state DOT and 17 LTAP/TTAP responses. TABLE 14 AgENCIES TRACkINg COMPLETION OF ICT-SUPPORTED TRAININg completion certificate, an exam score) to their human resources department or supervisors (39% and 21%, respectively). Agencies that listed “other” methods for tracking responded that they use other internal staff and databases to track attendance and completion. Several who are using other databases said they are migrating to an LMS. One state DOT respondent reported that supplemental learning is input into the LMS by the learner and approved by the supervisor. Many agencies also evaluate learners’ mastery of content for ICT-supported training (see Fig- ure 24). To determine mastery, most agencies use online exams (70% of state DOTs and 57% of LTAP/TTAPs). A significant number of state DOTs are also using performance-based evaluations and printed exams (33% and 30%, respectively). Some agencies said they distribute knowledge checks by means of portable document formats or administer them electronically immediately after web-based training, in some cases using in-house testing software. Several agencies responded that evaluations are administered depending on the content and whether it is required. A smaller percent- age of agencies are not administering exams (18% of state DOTs and 29% of LTAP/TTAPs). To evaluate learners’ satisfaction with ICT-supported training, agencies are using online evaluations (58% of state DOTs and 57% of LTAP/TTAPs) and printed evaluations (42% of state DOTs and 21% of LTAP/TTAPs) (see Figure 25). Some agencies are conducting informal evaluations, such as using Survey Monkey or contacting supervisors about observing new skills and behavior changes. One state DOT respondent reported that evaluations are administered depending on the content and whether it is required. Only 15% of state DOTs and 14% of LTAP/TTAPs are not administering evaluations. As shown in Table 15, the administration of participant evaluations occurs in most cases whether or not the training is mandatory (86% of state DOTs and 100% of LTAP/TTAPs).

33 FIGURE 24 What agencies use to evaluate content mastery. FIGURE 25 Methods used to evaluate learners’ satisfaction with ICT-supported training. LTAP/TTAP State DOT Number Percentage Number Percentage Yes 12 100 24 86 No 0 0 2 7 Don’t know 0 0 2 7 Of 28 state DOT and 12 LTAP/TTAP responses. TABLE 15 EVALUATIONS ARE ADMINISTERED FOR BOTH MANDATORy AND NONMANDATORy TRAININg The final question in this section addressed the efficacy of ICT-supported training. Respondents selected the ICT method they believed to be most effective for each audience (see Figure 26). They then selected the ICT method they considered to be most effective for all employees. The top two responses for across-audiences were computer-based and web-based training, including the “all employees” category. For some agencies, web-based training was considered more effective: equipment operators (state DOTs and LTAP/TTAPs), regional or district managers, foremen, and inspectors (LTAP/TTAPs), and technicians (state DOTs). For other agencies, computer-based training was considered more effective: engineers (state DOTs and LTAP/TTAPs), regional or district managers, foremen, and inspectors (state DOTs). LTAP/TTAPs considered both types of ICT-supported training

34 to be equally effective for technicians and the “all employees” category. State DOTs considered web- based training to be the most effective for “all employees.” future ict effortS In the final section of the survey, respondents documented their future plans for ICT-supported training. Respondents who are not currently delivering ICT-supported training also responded to these questions. Most agencies responded that they are planning to expand the number of ICT-supported training courses offered (70% of state DOTs and 37% of LTAP/TTAPs) (see Figure 27). A significant number of LTAP/TTAPs plan to acquire or develop ICT-supported training (29%), whereas several agencies plan to acquire ICT although not necessarily for training purposes (13%). Several agencies plan to maintain their number of ICT-supported training courses but not expand the program (13% of state DOTs and 18% of LTAP/TTAPs). Only one LTAP/TTAP indicated the agency was going to reduce the number of ICT-supported training courses offered. The five state DOTs and five LTAP/TTAPs planning to acquire ICT were asked to identify the delivery system(s) they planned to acquire (see Figure 28). LTAP/TTAPs responded that they would acquire web conference training, video conference training, computer hardware, and mobile devices. Two LTAP/TTAPs said they would acquire both web conference training and video conference train- ing. One state DOT said the agency would acquire mobile devices and video conference training. One agency also noted that the agency planned to acquire an LMS. Agencies planning to develop or acquire ICT-supported training were asked to identify the type(s) of training they planned to acquire. As shown in Figure 29, web-based training is to be acquired by FIGURE 26 Most effective ICT-supported training method for each audience.

FIGURE 27 Agency ICT plans for the next 3 to 5 years. FIGURE 28 ICT delivery systems agencies plan to acquire. FIGURE 29 Types of ICT-supported training agencies plan to acquire or develop.

36 most LTAP/TTAPs (82%). LTAP/TTAPs also plan to acquire computer-based training (45%), and web conference training, mobile, and video conference training (27%, respectively). Two state DOTs plan to acquire web-based training, and one plans to acquire web conference training. In the next 3 to 5 years, 42 agencies plan to expand their ICT-supported training catalog (see Fig- ure 30). Most agencies plan to add 20 or fewer courses (68% of state DOTs and 79% of LTAP/TTAPs), whereas 32% of state DOTs and 21% of LTAP/TTAPs plan to add between 20 and 50 ICT-supported training courses. One LTAP/TTAP respondent reported reducing the agency’s ICT-supported training offerings. The agency’s reasons for doing so were because of a perception that this type of training had not been effective and evidence that the training had not been effective for some, or all, audiences. FIGURE 30 Number of ICT-supported trainings the agency intends to add in the next 3 to 5 years.