As mentioned earlier, a number of professional associations concerned with measurement have developed standards to guide the development and evaluation of assessment programs (American Educational Research Association, American Psychological Association, and the National Council on Measurement in Education, 1999; National Commission for Certifying Agencies, 2004; Society for Industrial and Organizational Psychology, 2003). Although the standards they have articulated in various documents are tailored to different contexts, they share a number of common features. With regard to credentialing assessments, they lay out guidelines for the process of identifying the competencies to be assessed; developing the assessment and exercises; field-testing exercises; administering the exercises and scoring the responses; setting the passing standard; and evaluating the reliability of the scores, the validity of interpretations based on the assessment results, and the fairness of the interpretations and uses of these results. From our review of these standards, we identified a set of specific questions to investigate with regard to the development and technical characteristics of the NBPTS assessments.
With regard to the identification of the material to be assessed and the development of the assessment (Question 1), we ask:
What processes were used to identify the knowledge, skills, dispositions, and judgments that characterize accomplished teachers? Was the process for establishing the descriptions of these characteristics thoughtful, thorough, and adequately justified? To what extent did those involved in the process have appropriate qualifications? To what extent were the participants balanced with respect to relevant factors, including teaching contexts and perspectives on teaching?
Are the identified knowledge, skills, dispositions, and judgments presented in a way that is clear, accurate, reasonable, and complete? What evidence is there that they are relevant to performance?
Do the knowledge, skills, dispositions, and judgments that were identified reflect current thinking in the specific field? What is the process for revisiting and refreshing the descriptions of expectations in each field?
Are the knowledge, skills, dispositions, and judgments, as well as the teaching practices they imply, effective for all groups of students, regardless of their race and ethnicities, socioeconomic status, and native language status?
With regard to the reliability and validity of the assessment results, the methods for establishing the passing score, and test fairness (Question 2), we ask: