BOX 3–2 Survey of Standard-Setting Methods for Initial Teacher Licensure Examinations
1. There are several methods used to set performance standards, or cut scores, on tests. Which method most closely describes the method used by your state to set a passing standard for your initial teacher licensure examination(s)? If your state uses different methods for different tests, please mark all methods used:
___ Modified Angoff. For each item or exercise, panelists indicate the proportion of minimally competent examinees who would be able to answer the item satisfactorily. The standard is established by summing these proportions across all items or exercises.
___ Bookmarking. This method is used for both multiple-choice and constructed-response tests. Test items are arranged in descending order of difficulty and panelists place “bookmarks” immediately after the last item that all minimally competent examinees should be expected to know.
___ Judgmental policy capturing. This method is used for tests such as performance assessments and test batteries that contain items or exercises that have a score scale of three or more points (rather than simply right/wrong). Panelists review profiles of scores and provide judgments about the performance level of each examinee. Statistical analysis is used to yield a recommended standard for each panelist.
___ Nedelsky’s method. This method is used with multiple-choice tests. For each item, panelists eliminate all response options that the minimally competent examinee should be able to eliminate and records the reciprocal of the remaining number of responses. (If a panelist eliminates two out of five response options, three options would remain and the panelist would record “1/3”).
___ Ebel’s method. This method is used with multiple-choice tests. Test items are classified in a two-dimensional grid of item difficulty and item relevance. Each cell of the grid indicates how many items are at a given level of difficulty and relevance. Panelists indicate the proportion of items in each cell that a minimally competent examinee should answer correctly.
___ Contrasting groups. Panelists engage in discussion and agree on what constitutes minimally acceptable performance. They then identify candidates who, in their judgment, are clearly above the standard and those who are clearly below the standard. The examination is administered to both groups of examinees, score distributions are plotted for the two groups, and the performance standard is set at the point of intersection between the two score distributions.
___ Other (please describe)
Composition of the Standard-Setting Panel
2. How many panelists typically serve on your state’s standard-setting panel? ___
3. What is the composition of your state’s standard-setting panel? Please indicate the approximate percentage of your state’s standard-setting panel represented by each type of person listed below.