Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
46 Current Local Practices in Rail Car Training and Qualification and National Program Piloting Experience Limitations of Current Local Practices in Rail Car Training and Qualification Training systems for rail vehicle technicians throughout the public transportation industry vary both in quality and quantity. The results of training at local entities are also inconsistent and often do not meet the expectations of front- line workers and managers. Numerous discussions with the National Rail Vehicle Training Standards Committee and TCRP Project E-07 panel members highlighted these shortcomings and spawned further inquiry into the causes of the problem and potential solutions. Through these discussions, it became apparent that a comprehensive national system of qualification was needed, one in which the qualification process begins with training based on agreed-upon national training standards. At the foundation of this system of qualification is a committee structure consisting of SMEs from labor and management who jointly develop the training standards and define the quality and level of technical training needed to prepare technicians to pass assessments based on the standards. The joint com- mittee also determines which training delivery and mentor- ing methods are most effective to impart technical skills and knowledge. The capstone of the national qualification system is an apprenticeship registered with the U.S Department of Labor and a tie to college credits, all developed through TCRP Project E-07. Because equipment and procedures vary between agencies, the national program is designed to provide flex- ibility for local customization. A detailed description of the national comprehensive system of qualification was provided in Chapter 1. During the course of TCRP Project E-07 many stories about the state of rail vehicle maintenance training were shared with the project panel. Frontline workers were the most vocal about needing better and more training, and training administrators, trainers, maintenance supervisors, and other managers and union leaders echoed these sentimentsâalbeit less vocifer- ously. The National Rail Vehicle Training Standards Com- mittee discussed the limitations existing in current training systems, the need for more and better training, and the obstacles confronting achievement of these goals. The safe performance of rail vehicle maintenance and safe operation of equipment following maintenance were of great concern to the National Rail Vehicle Training Standards Com- mittee and the TCRP Project E-07 panel members. Neither can be achieved without quality training. Furthermore, assess- ing technician skills would not be fair or appropriate unless quality training was provided in advance of the assessment process. National Rail Vehicle Training Standards Committee members were polled to find out what they felt were the main obstacles to quality training and what training elements should be included in a national system of qualification. The results were not surprising. The first and most often repeated obstacles were a lack of organizational commitment to training and the lack of funds for training departments. The result of these two major stumbling blocks is that the elements of good train- ing systemsâgood courseware, quality trainers, structured mentor plans, mentor training, structured on-the-job training, time off the shop floor for training, and moreâwere under- funded or not undertaken at all in the majority of agencies. In essence, it all comes down to a lack of funding. Decisions to not fund training or pull funds from training first in a financial crisis are inextricably intertwined with public transportationâs perennially inadequate funding. This conundrum seems to be nearly universal within the industry. Including human capital within the definition of âcapitalâ would go a long way toward resolving the training system underfunding issue. The cost of quality training can be reduced by shared devel- opment of and local use of industrywide training resources that can be customized for local implementation. Joint industry- wide development of the system of qualification is a perfect example, with cost-effective local solutions arising from national training standards and curriculum to courseware, training of C H A P T E R 4
47 instructors and mentors, assessments, and apprenticeship. All of this dramatically reduces the cost to each agency and brings the knowledge of many SMEs to bear on developing courseware and training. Investing in quality training actually saves money for agencies through the reduction in unnecessary repairs, parts replacement, and faulty repairs that result once technicians are properly trained. Studies undertaken by the Transporta- tion Learning Center quantifying performance of equipment before and after training have shown that quality training for bus mechanics lessened diagnostic and repair times and increased the mean distance between failures (MDBF) in bus operations significantly. The Transportation Learning Centerâs Metrics of Success report shows that agency investments in quality training systems pay for themselves six times over within a few years and more than pay for themselves within 18 months (Transportation Learning Center 2010a). Current systems too often rely on the old idea that a new hire can learn by osmosis, simply by standing next to an expe- rienced technician. This approach often leads to a situation where true training does not readily occur and the experi- enced technician becomes the only person in the shop who may know how to do the work in a safe and proper manner. This can result in over-reliance on a single employee. An inter- esting finding during the Transportation Learning Centerâs research at one agency is that the MDBF decreases (gets worse) during the summer. After some investigation, it became appar- ent that this is because the âgo toâ people have the seniority to take vacation during the summer months, resulting in a seasonal brain-drain within the shop. A structured approach to training as part of a comprehen- sive system of qualification solves the problems associated with workplace incompetence by providing technicians with the knowledge and skills they need to correctly, efficiently, and safely perform work tasks. Training tools developed nationally by a joint committee of SMEs ensure that the train- ing content is appropriate and directly applicable to the jobs technicians are expected to perform every day on the shop floor. Training provided to all workers in a structured and effective manner makes certain that every person in the shop knows as much as experienced technicians and that all the experienced technicians working in the shop are in fact quali- fied to carry out their work. There is nothing more counter- productive in a shop than having the âgo toâ people passing along bad work habits and improper work procedures to others trying to learn their craft. Structured training, where classroom and on-the-job courseware and delivery elements are clearly defined and optimized, elevates the competence of all workers. These workers are then, collectively, in a better position to improve the quality of maintenance and repair work, improve shop safety, and pass along their insights in a more meaningful, productive, and correct manner. Key to the national process are provisions that support customized local implementation. This is based on the under- standing that each agency needs a training system that matches its own specific fleet equipment, job classifications, and pro- motion systems. With qualification frameworks and resources developed nationally by SME committees for each occupation, each transit agency can access quality materials and training delivery methods and customize them for their specific opera- tions. Each location is no longer left on its own to develop a quality training program, simply hoping that it will be effective. OEM Training Quality Found to Be Inconsistent OEM-provided training that comes packaged with new capital equipment is an important source of training, but it can fall short of the training needs of agencies for this new equipment. In a 2012 survey conducted by the Transportation Learning Center, SMEs on the National Rail Vehicle Training Standards Committee identified 50 distinct training areas on which OEMs provide training materials. The named OEMs include leading vendors in the industry. In 35 (70 percent) of these areas, the SMEs rated OEM training materials as poor or fair (see Figure 20). Among the topics covered by the OEM training materials, rail vehicle troubleshooting and com- munications were found to be the most problematic. Not one subject area had an average score that put it in the âVery Goodâ Category across all vendors (see Figure 21). System of Qualificationâ Local Implementation and Pilot Experience Throughout TCRP Project E-07, elements of the struc- tured system of qualification identified through the national work were put into action at a number of National Rail Vehicle 26% 48% 4% 22% Poor Fair Good Very Good Note: each OEM is represented by the average score of their training materials across mul ple subject areas. Sample size = 38 OEMs Figure 20. Ranking of OEM training materials.
48 Training Standards Committee member locations. These ele- ments include the application of training standards, validated courseware, proven training delivery methods, apprenticeship programs, structured on-the-job mentoring, and customiza- tion provisions to suit specific agency needs. In addition to the local implementation of these training elements, pilot locations were enlisted to administer written and hands-on assessments locally to identify and remedy potential weaknesses before rollout in the national qualifica- tion program. Doing so in advance of actual program launch allowed the assessment process to be tested and modified through National Rail Vehicle Training Standards Committee input to make it more efficient, user-friendly and fair to all participating agencies. The assessment pilot took place at six agencies: Sacramento SacRT, Santa Clara Valley Transporta- tion Authority (VTA) San Jose, LACMTA, Massachusetts Bay Transportation Authority (MBTA) Boston, GRTA, and Port Authority of Allegheny County in Pittsburgh, PA. Structured Training Application of structured training elements as identified through TCRP Project E-07 produced favorable results at many participating agencies. At UTA, national training standards developed jointly by SMEs from both labor and management were fully adopted into the agencyâs training program. The standards were also used to validate courseware to ensure that materials used for training complied with learning objectives contained within the standards. In areas where courseware was deficient, UTA used resources provided by the Transpor- tation Learning Center to locate missing elements. According to the maintenance training administrator at UTA, âWe plan to adopt all training standards recommended by the National Rail Vehicle Training Standards Committee, including the assess- ments. By using this national framework, weâve been able to achieve a clear focus, and weâre moving forward efficiently.â In early 2013, the GCRTA and its ATU Local 268 began the process of building a new rail car apprenticeship program. Labor and management collectively realized that the agency needed to do much more to prepare the next generation of rail car technicians. Rail vehicles and associated onboard equipment have grown in complexity, especially with regard to sophis- ticated electronic applications. The agency recognized that existing training methods would not be sufficient to educate the number of new hires needed to replace the many incum- bent technicians about to retire. In response to this training need, the agency set out to develop an entirely new rail car apprenticeship program utilizing the structured training ele- ments and resources of TCRP Project E-07. Management and the union agreed to work together to build an apprenticeship based on the new national system of qualification, realizing it was the most effective path they could follow. As a result of a skill gap analysis conducted by New York City Transit (NYCT), which pointed to skill deficiencies among its rail car technician workforce, the agency stepped up to add additional training in fiber optics, HVAC, and brazing and Note: each subject area is represented by the average score of the training materials provided by relevant OEMs. Very Good Good Fair Poor Figure 21. Ranking of OEM training materials by subject area.
49 welding. Training provided to enhance deficient skills was based on the national training standards. An additional skill gap analysis of rail car inspector apprentices is underway to improve training in that work area. Transportation Learning Center staff was also involved with the partnership team at NYCT to discuss potential improve- ments to its apprentice training based on TCRP Project E-07 activities, including a more systematic review of on-the-job training practices. The framework for examining the usefulness of on-the-job learning and its relationship to apprenticeship is also based on the national training standards. Flagging and safety have also been identified as critical training needs at NYCT. Classes conforming to training elements identified through TCRP Project E-07 have been put into place as a result of the needs analysis. Although SacRT will most likely never have a full appren- ticeship program because of its small size, the joint labor- management training partnership there has made every effort to streamline their training and bring it in line with the national standards. The light rail vehicle (LRV) maintenance trainer at SacRT credits participation in the National Rail Vehicle Train- ing Standards Committee work and personal relationships forged there for development of a better understanding of the materials available for training rail car technicians and the best training practices available for replication. These concepts were a direct result of participation in TCRP Project E-07 and the National Rail Vehicle Training Standards Committee pro- cess. Development of standards also gave SacRT a foundation for developing hands-on training and assessments. Participation by SEPTA allowed the agency to use much of the courseware identified through TCRP Project E-07. Additionally, National Rail Vehicle Training Standards Com- mittee participation has helped the agency exchange ideas with other locations. Working with the National Rail Vehicle Training Standards Committee and TCRP Project E-07 has helped MBTA shape and improve many of its training programs by incorporating national standards into its courses. Input from National Rail Vehicle Training Standards Committee members on the labor side helped management think through ways to make sure that the agency was fair in the way it offered training and promo- tion opportunities. Participation in TCRP Project E-07 also allowed them to ask questions during meetings and exchange ideas to help improve training. Activities are also underway at MBTA to expand its current training program to include hands-on learning and mentoring as a result of materials provided through TCRP Project E-07. Pilot Qualification Assessments Pilot test delivery was conducted at six transit agencies to validate the hands-on and written assessments and con- firm that rail car technicians could effectively demonstrate proficiencies resulting from training. For the pilot written assessments, all participating locations but one, MBTA, had technicians take the written assessments electronically via the Internet, allowing the research team to evaluate computer-based as well as pen-and-paper methods for test administration. MBTA had to use the pen-and-paper method because it does not have computer stations with web access on the property. Written assessments were administered prior to the hands-on assessments per qualification procedures. Labor and management proctors at each agency first participated in a tutorial with Transportation Learning Center staff to become thoroughly familiar with the process. Written Assessments A total of 65 validated written assessments were adminis- tered across all of the pilot locations. Another 5 were admin- istered in Current Collection, but there were not enough validated questions in this module to score the results. Of the 65 validated written assessments, 46 received a passing score, for a success rate of 71 percent. See Figure 22 for a summary of modules taken and mean scores (see table column labeled âpercentageâ) from all pilot locations. Assessment topics included Couplers, Trucks & Axles, Propulsion & Dynamic Braking, Friction Braking, and HVAC. Candidates taking the written assessments did so confiden- tially and were given a score sheet that showed their strengths and weaknesses in specific areas (see Figure 23 and 24 for example individual feedback reports). Those who passed the assessments were informed on their score sheet that the results would be kept in the Transportation Learning Centerâs database and would be used toward the official rail vehicle technician qualification program. Aggregated feedback reports were provided to the local man- agement and union team. They gave each agency a specific indication of the areas where their training program needed to be enhanced without compromising the confidentiality of individual assessment results (see Figure 25 for an example agency feedback report). For example, if score results show that several technicians at a particular agency did poorly in certain areas (e.g., suspension, doors, etc.), agency staff could then enhance that part of their training program to supply techni- cians with the needed knowledge. The courseware validation tool developed under TCRP Project E-07 could also be used to make sure that certain training materials effectively address learning objectives contained in each of the national standards. In addition to determining whether technicians have the required aptitude, written assessments provide agencies with an indication of how well their training program measures up to national standards and where to redirect their train- ing efforts.
50 Note: Overall percentage = the # of correct answers/ # of incorrect answers for the entire module. Employee Date of Assessment Written Assessment Module Strong Adequate Needs Improvement N/A >=75% <75% and >=50% <50% 77% 7 80% 5 100% 3 100% 6 100% 2 100% 2 Comments: # of Questions Pass/Fail Status Refrigeration Components Tools Congratulations, you passed this module! Your assessment result will be kept in our confidential database and may be used towards the official Rail Vehicle Technician Qualification Program once it is launched. Thank you for your participation in this pilot! Agency Name Background Knowledge Evaporators and Condensers Agency A Overall Percentage Compressor/Motors HVAC Controls January 31, 2013 84% Pass John Doe 206 - HVAC PercentageSubject Areas Module 206 - HVAC Pilot Written Assessment Feedback Figure 23. Example individual feedback report for passing assessment. Written Assessment Module Strong Adequate Needs Improvement N/A >=75% <75% and >=50% <50% 82% 16 74% 5 76% 26 78% 9 79% 9 Module 201 - Couplers Module 202 - Trucks & Axles Module 203 - Propulsion & Dynamic Braking Module 205 - Friction Braking Subject Areas Percentage # of Candidates All 65Number of Candidates Assessed Module 206 - HVAC Figure 22. Pilot location averageâwritten assessment modules.
51 Note: Overall percentage = the # of correct answers/ # of incorrect answers for the entire module. Module 206 - HVAC Pilot Written Assessment Feedback Employee Date of Assessment Written Assessment Module Strong Adequate Needs Improvement N/A >=75% <75% and >=50% <50% 54% 12 100% 3 100% 4 60% 3 100% 1 50% 2 Comments: # of Questions Pass/Fail Status Refrigeration Components Tools Sorry, you did not pass this module. Your assessment result will be kept confidential. You will have the opportunity to retake this module once the official Rail Vehicle Technician Qualification Program is launched. Thank you for your participation in this pilot! Agency Name Background Knowledge Evaporators and Condensers Agency A Overall Percentage Compressor/Motors HVAC Controls January 31, 2013 64% Fail John Doe 206 - HVAC PercentageSubject Areas Figure 24. Example individual feedback report for failing assessment. Module 206 - HVAC Pilot Written Assessment Feedback Agency Name Date of Assessment Written Assessment Module Number of Candidates Assessed 7 Strong Adequate Needs Improvement N/A >=75% <75% and >=50% <50% 85% 75% 95% 80% N/A 30% Subject Areas Percentage January 31, 2013Agency A Tools Compressor/Motors HVAC Controls Average Percentage 73%206 - HVAC Background Knowledge Evaporators and Condensers Refrigeration Components Figure 25. Sample agency aggregated feedback report.
52 Hands-On Assessments Pilot hands-on assessments followed the written assess- ments. Hands-on assessments are an important element of the qualification program because they provide technicians who may not do well on written assessments an opportu- nity to display their abilities in an actual work environment. Transportation Learning Center staff worked with the labor- management team at each agency location to take the generic hands-on tasks developed by the National Rail Vehicle Train- ing Standards Committee and customize them to suit each agencyâs equipment, work procedures, and terminology. The process confirmed that one fixed set of hands-on assessments would not be appropriate for all agencies to follow because of the differences across agencies in equipment, practices, and so forth. Transportation Learning Center staff worked with local labor-management teams to develop a series of hands-on assessment scenarios and scoring structures that remained consistent with the National Rail Vehicle Training Standards Committeeâs generic procedures yet conformed to each agencyâs equipment and practices. The pilot hands-on assessments resulted in some minor modifications to the assessment procedures, as reflected in the final versions of three documents: ⢠National Rail Car Hands-On Skills Assessment Part 1: Task Application Form ⢠National Rail Car Hands-On Skills Assessment Part 2: Evaluatorâs Worksheet ⢠Candidate Assessment Form Modifications made to these documents as a result of the pilot testing were aimed at clarifying communication with the candidates, making certain they fully understood what was expected of them prior to and during the assessments. Part 1 of the National Rail Car Hands-On Skills Assessment consists of the form that agencies will use to apply for a hands-on assessment. It begins the process of working with the Transportation Learning Center to tailor tasks to each agency location. Part 2 is the evaluatorâs worksheet to be used by the joint labor-management team to conduct the assess- ments, assess each candidateâs performance, and arrive at a final score. A third document, the Candidate Assess- ment Form, provides rail technicians planning to take the assessments with essential background information about the process to make them feel more comfortable about the assessments. The Candidate Assessment Form was a direct outcome of the piloting process because it became evident that technicians needed more up-front information. Final piloting confirmed that the material contained in all three documents, following some minor revisions, is now ready for large-scale implementation. Feedback on Pilot Experiences Pilot experiences and feedback received from each agency proved useful to fine-tuning the hands-on process. SacRT and IBEW Local 1245â Sacramento, California A management evaluator and labor evaluator made up the labor-management team at SacRT. Hands-on assessments for friction brake inspection were administered to four tech- nicians. All four had recently completed training in which conducting a brake inspection was included. Candidates were asked to conduct a brake inspection per agency procedures, verbalize those procedures, and identify planted defects. They achieved scores ranging from 60 to 100. It is noteworthy that during one of the TCRP Project E-07 pilot hands-on assess- ments a major fault identified by a technician resulted in the agency issuing a fleet defect directive, resulting in modifica- tions being made to the entire fleet to avoid serious electrical consequences. Comments from the four SacRT technicians indicated that they found the hands-on assessment exercise very helpful in identifying possible problems during brake inspections. Given the opportunity, several indicated that they would participate in future exercises. Knowing that San Jose VTA would be con- ducting assessments next, the joint labor-management team from that agency traveled to Sacramento to witness the pilot process, which put them in a better position to conduct their own assessments. VTA and ATU Local 265âSan Jose, California A management evaluator and labor evaluator administered the hands-on assessment to five technicians in the area of Current Collection, Pantograph Inspection with scores ranging from 85 to 100. Students had also recently completed training in the topic, and the assessments provided a way for the agency to gauge learning and its training program. Witnessing the hands-on assessments at SacRT in advance allowed VTA to conduct their own assessments without on-site assistance from the Transportation Learning Center. Transportation Learning Center staff worked with the labor-management team over the telephone and through emails in advance to develop the assessment tasks, a process that will be used once the program becomes operational on a national level. Comments from those taking the hands-on assessments were again generally positive: ⢠âI enjoyed the test. It helped me to figure out what I did not understand about certain problems. [We] need more frequent training.â
53 ⢠âTest was performed well and explained well.â This com- ment reveals that the steps taken by the labor-management team to more fully explain the assessment process, steps that were taken as a result of the Sacramento piloting, were successful. ⢠âGood test procedures. Good to find out what I didnât know or wasnât shown when getting taught in the beginning.â This comment highlights the assessments as a way for students to gauge their learning; allowing them to apply their training to real-world job tasks reveals any learning gaps. ⢠âA very good assessment process to improve mechanicsâ skills for younger generation.â The comment implies that younger mechanics appreciate learn-by-doing methods. The management evaluator added the following: Through the pilot, we were not only able to evaluate our current training, but also received some great ideas and feedback from the candidates that will be used to improve future training. We felt that this pilot program was extremely successful, and we are pleased to have participated in it. We are confident that this national program will continue to be a valuable tool for us to boost training of our technicians. LACMTA and ATU Local 1277â Los Angeles, California Hands-on assessments were administered to two Los Angeles rail car technicians by a management evaluator and labor evaluator. The pilot assessments consisted of perform- ing an HVAC system inspection. Both technicians had strong scoresâ85 percent or higher. Comments from the two can- didates included the following: ⢠âHands-on training helped me pass this test.â ⢠âDegree of difficulty OK for me because I work much on AC, but would be more difficult for others with limited experience.â Both comments reveal that training and expe- rience are required to pass the assessments, an indication that they were not too easy. The management evaluator, speaking about the overall project said: This committee has given me a great experience and valuable resource for any future issue that may arise. I concur with everyone that this committee has done a great job! MBTA and ATU Local 589âBoston, Massachusetts Hands-on assessments were administered to eight techni- cians by the labor evaluator and management evaluator. As part of the assessment, technicians were asked to perform coupler inspections according to local agency procedures, verbalize what they were doing, identify planted defects, and describe procedures to correct the defects. All eight had recently completed coupler training where they performed coupler inspections as part of the training. All but one achieved a perfect score of 100. The other pilot candidate, who scored 60 percent, had not completed all elements of the training due to personal reasons. The perfect score achieved by the others is an indica- tion that the training as delivered by MBTA is effective. GCRTA and ATU Local 268âCleveland, Ohio Hands-on assessments were given to three Greater Cleveland RTA technicians on conducting inspections of pro- pulsion and dynamic braking systems. The assessments were administered by two labor evaluators and two management evaluators. Unlike technicians at the other agencies, these technicians were not provided with recent training on the subject. Two of the pilot subjects who had not had recent training scored 60 and 70 percent. This points to the need for additional refresher training. Greater Cleveland RTA uses an agency-developed version of hands-on assessment to validate their training and welcomed the opportunity to pilot a nationally developed process that they helped create. The agency will change their locally devel- oped hands-on assessment process to conform to the national format. Port Authority of Allegheny County and ATU Local 85âPittsburgh, Pennsylvania A labor-management project team was formed to work on the piloting initiative. The project team reviewed and updated hands-on and written assessment documents concerning propulsion system inspections. Assessments were adminis- tered by labor and management evaluators. Six employees (including three Rail Tech As and three Rail Tech Bs) partici- pated in both written and hands-on assessments. Written assessment results showed that the Rail Techsâ knowl- edge was strong in three of the four subject areas. Test data suggested that training in one subject area, AC propulsion components, needed improvement. The assessment feedback offered valuable data analysis to identify the training needs of the workforce. The participantsâ comments concerning the assessment focused on terminology issues. The technical jargon found in the assessment at times was not consistent with the terminology used on the shop floor. Technicians participating in the hands-on assessment verified that years of training and experience are beneficial to developing a skilled workforce. All six rail technicians scored 100 percent on the hands-on assessment. As a result of the interaction between the assessment participants and the
54 labor-management evaluators during the hands-on assess- ments, the project team realized the need to update the agencyâs standard operating procedure concerning propulsion system inspections. Lessons Learned and Conclusions The pilot training and assessments allowed various elements of a structured national system of qualification to be imple- mented at several agency locations. Although not at the scale needed by the rail industry, these pilot implementations did begin the process of locally identifying the need for and ben- efits associated with a structured training program, one based on national training standards. The pilot implementations also provided a trial run of the written and hands-on assess- ment processes developed through National Rail Vehicle Train- ing Standards Committee involvement. With the important prerequisite that training be given in advance of testing, the hands-on assessments allowed technicians to demonstrate their ability to carry out work tasks in accordance with nationally recognized standards in a way that might not be realized through written testing alone. Hands-on assessments fully achieve what written assessments cannotâconfirming that as a result of their training technicians can in fact properly and safely perform job tasks that they will be responsible for on a daily basis. Whereas written tests evaluate a technicianâs knowledge, hands-on assessments prove that technicians can convert that knowledge into useful workplace skills. The pilot implementations demonstrated that incorporating both written and hands-on assessments following training are essential to a national qualification program. As revealed by many of the comments provided, labor and management representatives were generally pleased with the implementation process. For labor and management evalu- ators, many of whom served on the National Rail Vehicle Training Standards Committee to develop the process, the pilot implementations provided an opportunity to witness just how effective the qualification process could be, and how labor-management relations are strengthened as a result of that process. Having labor represented as an equal partner in the process provided assurances to union technicians that the training and assessment process was developed in a fair and impartial manner. Assessing technicians following recent train- ing on a particular subject allowed the joint labor-management teams to determine just how effective the training was at matching nationally established standards. The pilot implementations also revealed that conducting hands-on assessments, while a worthwhile program element, does take additional time and effort. Equipment needs to be secured and made available during the assessments, and evaluators need to plant defects and return the equipment to normal operating condition following each assessment. Unlike written assessments, which can be scheduled as a group activity within a relatively short period of time, hands-on assessments need to be scheduled over a longer time period because candidates participate one at a time. Additionally, the pilot implementations once again con- firmed the importance of training in the qualification pro- cess. As mentioned regarding MBTA, all candidates, except for one who did not take training because of personal reasons, achieved a perfect score. These results illustrate that training is essential to achieving the skills needed to become qualified. Other interesting findings are that some technicians without formal training scored well because of their extensive experi- ence, which supports a provision in the program that allows technicians to opt out of the training prerequisite requirement if they feel they have sufficient training. If the assessments show that this is not the case (i.e., they do not pass), they can take targeted training to acquire the knowledge and skills needed to become qualified. All pilot assessment results are kept confidential, with indi- vidual results and feedback provided to each participating technician and aggregate results and analysis provided to the joint labor-management teams at each agency. Those tech- nicians who passed the assessments will be given credits for their scores when the program is officially launched.
