Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
100 A P P E N D I X G Level 2 Training Evaluation Tool The Kirkpatrick/Philips training evaluation model was developed by Donald L. Kirkpatrick and Jack Phillips. The model consists of five levels of training evaluation: Level 1: The reaction of participants Level 2: The level of learning achieved Level 3: Changes in learner behavior Level 4: Business results derived from training Level 5: Return on investment (ROI) that the training delivers This tool is intended to assist transit agencies in conducting Level 2 training evaluations. Level 2 evaluations are used to assess how much training participants have learned. In the transit industry, these evaluations can be particularly useful for assessing whether or not a training is effective in teaching employees complex technical knowledge and skills that may be critical to core responsibilities and on-the-job performance. Level 2 evaluations can be conducted through written and web-based tests or scored observations. Written and web-based tests are useful for assessing declarative (e.g., content- based) and procedural (e.g., step-by-step-based) knowledge. Scored observations are useful for assessing specific behaviors. Reviewing the learning objectives for each knowledge, skill, or ability taught in the training course can help in determining which tool will be most effective for assessing learning. It is also possible to use a combination of testing and observations if appropriate. Level 2 evaluations should be administered both immediately before and after training takes place, to determine if the training improved employeesâ knowledge and skills. How to Develop Level 2 Evaluations When developing a Level 2 evaluation plan, it is important to first determine if the knowledge, skills, and abilities (KSAs) being assessed are covered to an equal extent in the training or if there is greater focus on certain KSAs compared to others. If each KSA is covered to an equal extent, develop the same number of test items for each (e.g., three items per knowledge, skill, and ability). For a scored observation, develop the same number of observable tasks for each KSA being assessed. If some KSAs are covered to a greater extent in the training, develop more test items or observable tasks for those KSAs compared to others.
Level 2 Training Evaluation Tool 101 Minimize reading time Write test questions concisely. Write plausible distractors Distractors are response options that are incorrect, but also plausible to a test taker who does not have complete mastery over the content that the test question is measuring. It is good practice to write distractors that correspond to the major errors that people generally make. For example: Good Distractors: Reliability is defined as: a) the consistency of a measure b) the accuracy of a measure c) the validity of a measure Bad Distractors: Reliability is defined as: a) the consistency of a measure b) a measure of taste c) a unique weather phenomenon Write response options that are inconsistent All response options should be similar in grammatical form, style, content, and length. For example, if the correct answer is a phrase, then the other response options should also be phrases. Also, all response options should be relatively the same length. Make items independent of each other One test question must not contain information that helps the test taker answer other questions. Also, avoid using the same response options for more than one test question. This may help the test taker eliminate a response option based on his or her answer to the other questions. Write trick questions Trick questions are designed to deliberately confuse and mislead the test taker. They may include Questions on trivial content: The content of all test questions must be relevant to the job and must link back to a task and/or a KSA. Questions that require test takers to make overly fine discriminations between ideas or concepts (i.e., over-specific test questions). Questions that contain window-dressing: This is information that is included in the question, but has little or nothing to do with the actual question or the content it is addressing. This only confuses the test taker and adds to the reading time. Questions that cover principles or ideas in ways that are not required for the job. Questions that are highly ambiguous. Donât: The sections below provide guidelines for developing a Level 2 written or web-based test and conducting scored observations. This is followed by a tool for developing a Level 2 evaluation plan, sample questions for a written or web-based test, and templates for conducting an observation. Test Development Guidelines Written and web-based tests typically contain multiple-choice questions. The table below contains guidelines for writing effective and appropriate multiple-choice questions. Do: Use the correct-answer format The correct-answer format has only one correct answer to the question. This is different from the best-answer format in which more than one response option may be correct, but only one is the best correct answer.
102 Transit Technical Training, Volume 2: Guide to Overcoming Barriers to Implementing Best and Innovative Training Scored Observation Guidelines Step 1: Determine the goal of the observation. Consider the learning objectives of the training course, and determine what is critical for the learner to achieve successfully. This may be a process or an outcome, depending on the training topic. Step 2: Consider logistics for conducting the observation. Consider practical and logistical limitations, such as safety issues, equipment requirements, and the ability to standardize the testing process, so all employees are observed under identical conditions. Step 3: Identify observation tasks and equipment and materials needed. Determine what tasks the learner should be able to perform. Then, develop a checklist for the observation that identifies what equipment or materials are needed; the procedures for setting up the observation; the procedures that the test administrator needs to complete with each learner during the test; and the words the test administrator should say. This will help ensure all learners undergo the same observation. Step 4: Develop an observation scorecard or scoring sheet. If assessing a process, develop a scorecard or scoring sheet identifying the exact steps the learner should take and the order in which they should be performed. Steps must be observable, objective, and have clear beginning and end points. Use pass/fail ratings for each step, and avoid testing trivial steps. If assessing an outcome, develop a scorecard or scoring sheet that identifies what aspects of the outcome should be evaluated and what constitutes acceptable and unacceptable results for each aspect. Determine if the learner can receive partial credit. Write negatively oriented items An example of a negatively oriented item is âWhich of the following is NOT required to repair X?â This item should be reworded to a positive orientation, such as âThe following is required to repair X:â The phrase âall of the below, EXCEPTâ¦â also should be avoided, as it unnecessarily increases the difficulty of the question. Include direct and indirect clues Direct clues allow the test taker to choose the correct answer without having any knowledge of the content. This occurs when the same or similar word or phrase is used in both the test question and correct answer or when a distractor response option is the direct opposite of the correct answer. Indirect clues are less obvious. An indirect clue occurs when distractor response options say the same thing but in different ways. Use ambiguous, obsolete, or slang terminology Donât:
Level 2 Training Evaluation Tool 103 Level 2 Evaluation Planning Tool Instructions: Complete the following table prior to conducting a Level 2 evaluation. First, list the learning objectives covered in the training course. Next, list the KSA(s) addressed by each learning objective. Based on the importance or emphasis placed on a given learning objective and related KSA(s) in the training course, determine how many written test questions and/or observation tasks will be developed. Learning Objective KSA(s) Addressed Written Test Questions Scored Observation Tasks Learning objective from training course KSA(s) addressed by the learning objective # of test questions # of observation tasks Sample Level 2 Test Questions 1. The average bus engine should receive an oil change after this many months. a. 3 b. 5 c. 6 d. 7 e. 18 2. A bus drive indicates the electrical system is failing because the lights on the bus flicker on and off and the dashboard instruments are not refreshing with new information. What should you assess first to determine the cause? a. Fuse and then the bus battery becauseâ¦XYZ. b. Bus battery and then the fuse box becauseâ¦XYZ. c. Wire connections and then the computer updates becauseâ¦XYZ. d. Computer updates and then the fuse box becauseâ¦XYZ. e. Manual control settings and then the bus battery becauseâ¦XYZ.
104 Transit Technical Training, Volume 2: Guide to Overcoming Barriers to Implementing Best and Innovative Training 3. There is an issue in the functionality of the connection between A and B. What test could be used to diagnose the problem? a. Blue Test becauseâ¦XYZ. b. Red Test becauseâ¦XYZ. c. Green Test becauseâ¦XYZ. d. Yellow Test becauseâ¦XYZ. e. Black Test becauseâ¦XYZ. B A D C
Level 2 Training Evaluation Tool 105 Scored Observation Template Instructions: Use the following templates to plan and perform a scored observation. Use the Administration Checklist for planning purposes. Record what task will be observed, what materials or equipment will be needed, what procedure should be followed, and instructions for observers to use. The instructions should contain a script that all observers must recite, to ensure consistency across observations. Use the Scoring Checklist when conducting an observation. Enter the test date, scorerâs name, learnerâs name, and description of the task being observed and scored. Depending on if the observation is for a process or an outcome, list the process steps or observable, objective aspects of the outcome. During the observation, record whether or not the learner has passed or failed each part of the observation. Take written notes as needed (e.g., if the learner cannot complete the test due to equipment malfunction). Administration Checklist Task: Description of task being observed Materials Needed: Material 1 Material 2 Material 3 Procedure to Administer Task: Step 1: Step 2: Step 3: Instructions for Learner: â[Instructions script to be used by observer]â Scoring Checklist Test Date: Scorer Name: Learner Name: Task: Description of task being observed Pass Fail Process Step or Outcome Aspect 1 Process Step or Outcome Aspect 2 Process Step or Outcome Aspect 3 Notes:
