with stakeholders. Detailed information about conducting a needs analysis can be found at http://www.needsassessment.org [August 2011].
Knapp emphasized that it is important to have a clear articulation of the construct to be assessed: that is, the knowledge, skill, and/or behavior the stakeholders would like to have measured. The construct definition helps the test developer to determine how to measure it. She cautioned that for the skills covered in this workshop, developing a definition and operationalizing these definitions in order to produce test items can be challenging. For example, consider the variability in the definitions of critical thinking that Nathan Kuncel presented or the definitions of self-regulations that Rick Hoyle discussed. In order to develop an assessment that meets appropriate technical standards, the definition needs to be detailed and sufficiently precise to support the development of test items. Test development is less challenging when the construct is more concrete and discrete, such as specific subject-matter or job knowledge.
One of the more important issues to consider during the initial development stage, Knapp said, is whether the assessment needs to measure the skill itself or simply illustrate the skill. For instance, if the goal is to measure teamwork skills, is it necessary to observe the test takers actually performing their teamwork skills? Or is it sufficient that they simply answer questions that show they know how to collaborate with others and effectively work as a team? This is one of the issues that should be covered as part of the needs analysis.
Knapp highlighted the importance of considering which aspects of the construct can be measured by a standardized assessment and which aspects cannot. If the construct being assessed is particularly broad and the assessment cannot get at all components of it, what aspects of the construct are the most important to capture? There are always tradeoffs in assessment development, and careful prioritization of the most critical features can help with decision making about the construct. Knapp advised that once these decisions are made and the assessment is designed, the developer should be absolutely clear on which aspects of the construct are captured and which aspects are not.
Along with defining the construct, it is important to identify the context or situation in which the knowledge, skills, or behaviors are to be demonstrated. Identification of the specific way in which the construct is to be demonstrated helps to determine the type of assessment items to be used.
Determining the Item Types