broad topic includes several subcategories, none of which speak directly to cultural diversity (Tests at a Glance, pp. 42–43).
Comment: The three broad topics and the more detailed descriptions seem reasonable (but a content specialist could judge more appropriately the quality and completeness of the content coverage).
How were the KSAs derived and by whom? The content domain1 was determined by using a job analysis procedure that began in 1996. At the time this job analysis was conducted, subject assessments already existed for elementary and secondary levels. This study was an attempt “to determine the appropriateness of the task and knowledge statements developed for secondary school teachers [emphasis in original] of English…to middle school teachers [emphasis in original] of language arts” (ETS, February 1998, Task and Knowledge Areas Important for Middle School Teachers of Language Arts: A Transportability Study, p. 1). Thus, the job analysis was an attempt to determine the extent that a job analysis undertaken earlier for secondary teachers would apply to middle school teachers (teachers of grades 5 to 9). The job analysis methodology for secondary school English teachers was summarized briefly as being similar to other job analyses conducted for the Praxis series of tests. The procedures used for the MS:ELA test entailed (1) examining and modifying the job analysis inventory for the secondary school teachers, (2) conducting a national survey of appropriate teachers and teacher educators, and (3) analyzing the results of the survey.
The initial step—revising the job analysis inventory used for secondary school teachers—involved ETS test development staff, who made appropriate modifications in the inventory (e.g., made instructions relevant to middle grades, changed English to language arts). The importance scale was also modified to add a “not relevant” response choice to the previously used five-point scale. Thus, respondents had a six-point importance scale.
There was no external (to ETS) review of these items. The external review occurred previously when the items were developed originally for secondary school teachers.
There was no pilot survey undertaken to obtain information about the clarity of the revised instructions and content of the survey instrument. The changes in the instrument are described as being minimal; thus, it is likely that no pilot was thought necessary.
The survey instrument was mailed to 1,000 practicing middle school teachers and 500 teacher educators of middle school language arts. The response rates were not impressive. Only 364 usable responses were analyzed (24 percent). Of these 364 responses, 232 were from teachers. All those surveyed were asked to judge the importance of each of the 115 task statements (content