perception, cognition, physiology, and psychology are buried within the individual. Team performance is even harder to measure. With team performance a number of individuals interact, and the interactions need to accommodate authority levels, role and responsibility levels, verbal and nonverbal communications, and power levels.

Human Factors Aspects of Simulation (NRC, 1985) contains comments and recommendations with respect to performance measurement. The study concluded that the development and application of performance measurement in simulator-based training and research requires that the following elements be considered:

  • Operational measures and criteria of overall system effectiveness for representative tasks and operating environments are needed.
  • Analysis of the hierarchy of goals and control strategies that operators employ in the performance of real-world tasks is needed.
  • Measurement for performance diagnosis needs to be developed.
  • Team performance measurement where the contribution of each person is to be defined and measured is difficult.
  • Automated instrumentation and performance measurement systems are not possible in all simulations, or at least for certain tasks.
  • There is no single source, or even a coherent body of literature, to which practitioners can turn to obtain useful data on performance measurement methods and simulation practice.

Performance measurement must consider three levels of relevance: concept, understanding, and performance. Concept refers to the knowledge base of the individual, the degree to which the correct analysis and response methods are known. Understanding refers to the degree to which the individual is able to accommodate and work around missing or inaccurate information to correctly analyze and respond. Performance refers to the physical acts that are observable by instructing or evaluating personnel. Performance observation is clouded further when tasks demand multiple and simultaneous responses, are complex, or are to be measured at points of significant stress (which may include extremis or near-extremis exercises).

Evaluation and Assessment Defined

In some fields the terms evaluation and assessment can be used interchangeably. For this study, the terms are given more narrow definitions. Evaluation is applied to the formal or informal review of training exercise results: the input is the training program; the output is the evaluation. In this context, evaluation is an element of the instructional design process. The evaluation can be informal or formal, subjective or objective, or both (see ''Forms of Evaluation and Assessment" below).



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement