assessments, as are the annual state achievement tests administered for accountability purposes.
All assessments should be designed to be of high quality: to measure the intended constructs, provide useful and accurate information, and meet technical and psychometric standards. For assessments used to make decisions that have an important impact on test takers’ lives, however, these issues are critical. When assessments are used to make high-stakes decisions, such as promotion or retention, high school graduation, college admissions, credentialing, job placement, and the like, they must meet accepted standards to ensure that they are reliable, valid, and fair to all the individuals who take them. A number of assessments used for high-stakes decisions were discussed by workshop presenters, including the Multistate Bar exam used to award certification to lawyers, the situational judgment test used for admitting Belgian students to medical school, the tests of integrity used for hiring job applicants, and some of the assessment center strategies used to make hiring and promotion decisions.
For the workshop, the committee arranged for two presentations to focus on technical measurement issues, particularly as they relate to high-stakes uses of summative assessments. Deirdre Knapp, vice president and director of the assessment, training, and policy studies division with HumRRO, spoke about the fundamentals of developing assessments. Steve Wise, vice-president for research and development with Northwest Evaluation Association, discussed the issues to consider in evaluating the extent to which the assessments validly measure the constructs they are intended to measure. This chapter summarizes their presentations and lays out the steps they highlighted as fundamental for ensuring that the assessments are of high quality and appropriate for their intended uses.2 Where appropriate, the reader is referred to other sources for more in-depth technical information about test development procedures.
Defining the Construct
According to Knapp, assessment development should begin with a “needs analysis.” A needs analysis is a systematic effort to determine exactly what information users want to obtain from the assessment and how they plan to use it. A needs assessment typically relies on information gathered from surveys, focus groups, and other types of discussions
2Knapp’s presentation is available at http://www7.national-academies.org/bota/21st_Century_Workshop_Knapp.pdf [August 2011]. Wise’s presentation is available at http://www7.national-academies.org/bota/21st_Century_Workshop_Wise.pdf [August 2011].