anyone examining a set of measurement data will arrive at the same evaluative judgment. In other words, such an evaluation system would produce consistent outcomes in any situation.
Since a completely “objective” evaluation is not possible, however, the goal must be to achieve consistent results from a necessarily subjective process. That is, we must design a process that provides the same evaluative judgment based on a data set, regardless of who considers the data. This can be done through a process called controlled subjectivity.
Psychometric methods can be used to create tools for measuring faculty performance (e.g., observation checklists, student- and peer-rating forms) in a way that produces reliable data (i.e., measurements) that are as objective as possible. However, because we know that an evaluation must be subjective, the problem is how to achieve the characteristic of objectivity (i.e., consistency of conclusions based on the same data regardless of who considers them) in a necessarily subjective process.
Because subjectivity in a faculty evaluation system is unavoidable, the goal should be to limit or control its impact. To accomplish this we use a process called controlled subjectivity, which is defined as the consistent application of a predetermined set of values in the interpretation of measurement data to arrive at an evaluative judgment (Arreola, 2007).
In other words, subjectivity in an evaluation system can be controlled when an a priori agreement has been reached on the context and (subjective) value system that will be used to interpret the objective data. Thus, even though the evaluation process involves subjectivity, we can still ensure consistency in outcomes, thus approximating a hypothetical (although oxymoronic) “objective” evaluation system.
Every evaluation rests upon an implicitly assumed value or set of values. An evaluation provides a systematic observation (measurement) of the performance of interest and a judgment as to whether that performance conforms to the assumed values. If there is a good match, the performance is judged desirable and is generally given a positive or “good” evaluation. If there is a discrepancy, the performance is judged to be undesirable and is generally given a negative or “poor” evaluation.
As was noted earlier, the evaluation process implies the existence and application of a contextual system, or structure, of values associated with the characteristic(s) being measured. Thus before an evaluation system can be developed, the values of those who intend to use it must be defined and should be carefully developed to reflect the values of the institution where they will be applied. For a faculty evaluation system to reflect the values of the institution correctly, we must not only determine those values and have them clearly in mind, but we must also express them in such a way that they may be applied consistently to all individuals subject to the evaluation process.
The value system of a faculty evaluation for a unit in a larger institution must be in basic agreement with the larger value system of the institution. The first step, therefore, must be to