To be useful for an assessment, benchmarks must be “operationalized.”
The ITEA Standards for Technological Literacy (ITEA, 2000) the AAAS Benchmarks for Science Literacy (AAAS, 1993), the NRC National Science Education Standards (NRC, 1996), and especially state-specific content standards, would be logical starting points for determining the content of the assessment. All of these documents suggest “benchmark” knowledge and skills that a technologically literate individual should have. To be useful for an assessment, however, the benchmarks must be “operationalized, ” that is, the most important technology concepts and capabilities must first be identified and then made specific enough to clarify the range of material to be covered in the assessment. This is a step in the process of developing an assessment framework for technological literacy, as discussed in Chapter 3.
In addition, existing assessments may be reviewed to determine if any items are aligned with, and measure, the operationalized benchmarks. If not, technology-related content may have to be added. A review of the general guidelines for student assessments developed by ITEA may also be helpful (ITEA, 2004a).
The assessment framework must specify the emphasis, or weight, given to items in each dimension of technological literacy. The weighting process must be based on many factors, including the purpose of the assessment, the time allotted for testing, the developers’ views of the importance of each dimension of technological literacy, and expert judgments about reasonable expectations for students in these grades. Table 6-1 shows how the weighting process might work.
In this sample case, the state would derive a scale score for each student. If similar technology-related concepts were tested for more than one grade level (e.g., manufacturing processes for grades 3–5 and 6–8), the state might use cross-grade vertical scaling, which would enable scorers to compare the mastery of material by students at different grade levels. Using within-grade scaling, which is more common, the performance levels in each grade would be examined independently.
To provide scores in a useful form for policy makers and instructional leaders, the state board of education might establish performance