Tech Tally Approaches to Assessing Technological Literacy (2006) / Chapter Skim
Currently Skimming:
4 An Assessment Primer
4 An Assessment Primer
Pages 63-92
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 63... ...
The middle section focuses on what researchers have learned about cognition, that is, how people think and learn generally. The last section summarizes research on how people learn technological concepts and processes.
|
From page 64... ...
Statistically, indices of test reliability typically range from zero to one, with reliabilities of 0.85 and above signifying test scores that are likely to be consistent from one test administration to the next and thus highly reliable (Linn and Gronlund, 2000)
|
From page 65... ...
. One of the most important types of evidence for determining validity is how well the themes, wording, and format of test items relate to a specified target-content domain, which may be based on specific learning objectives, such as those spelled out in educational standards (e.g., ITEA's Standards for Technological Literacy)
|
From page 66... ...
and NRC (1996) have developed national science education standards that include references to technological literacy.
|
From page 67... ...
of learning, Theories of cognitive learning based on a constructivist approach assessments to knowledge acquisition suggest that the most valuable assessment in- must also take struments for students -- at both the K12 and post-secondary levels (i.e., into account pre-service teachers) -- are integrated with instructional outcomes and learning curriculum content.
|
From page 68... ...
Suppose, for instance, that two questions seem identical, but the first has a correct response rate of 80 percent by all groups, and the second has an 80 percent correct response rate from all groups but one. Even if the bias is not apparent, the second question should not be used in the assessment.
|
From page 69... ...
Indeed, well constructed multiple-choice items can measure virtually any level of cognitive functioning. One weakness of the selected-response format is that test takers can sometimes arrive at correct answers indirectly by eliminating incorrect choices, rather than directly by applying the knowledge intended by the test developer.
|
From page 70... ...
In assessing technological lit eracy, multiple-choice and short-answer questions might be used to mea sure facts, knowledge, and concepts related to technological literacy, as well as the types of knowledge that can be applied in different situations. However, depending on the objective of the assessment, the latter skill might also be measured by performance tasks.
|
From page 71... ...
The National Assessment of Educational Progress and the Trends in International Mathematics and Science Study use matrix-sampling techniques. So-called census testing involves giving the same test to all members of the target population.
|
From page 72... ...
In the context of the present report, cognition is a theory or set of beliefs about how people represent knowledge and develop competence in a subject domain. To test an individual's learning and knowledge, assessment designers must first understand how people learn and know things.
|
From page 73... ...
on the higher levels of the hierarchy and are interconnected with ancillary concepts and related facts on the lower levels of the hierarchy. In addition, procedures and contexts for applying knowledge are bundled with the knowledge so that experts can retrieve knowledge in "chunks" with relatively little cognitive effort.
|
From page 74... ...
Thus, novices must spend more cognitive effort looking for and spend more retrieving knowledge from memory, which leaves less short-term memory cognitive effort for high-level tasks, such as problem solving. looking for In a telling experiment by Egan and Schwartz (1979)
|
From page 75... ...
Only after considerable experience solving problems in this way do unskilled problem solvers begin to realize that this approach cannot be "generalized." At that point, they may begin to shift to concept-based problem-solving strategies. Cognitive research related to expertise raises a number of questions relevant to assessments of technological literacy: · What assumptions can be made about the conditions and time necessary to acquire technological literacy?
|
From page 76... ...
. In many studies, a failure to transfer knowledge was attributable to insufficient initial learning (e.g., Brown, 1990; Klahr and Carver, 1988; Littlefield et al., 1988)
|
From page 77... ...
. Cognitive research related to knowledge transfer has raised several questions relevant to the assessment of technological literacy: · What test items would assess the transfer of key technological principles from one context to another?
|
From page 78... ...
Conceptual Change Cognitive scientists have also examined how people form con cepts and how they give up one concept in favor of another. In science learning, for example, although no consensus has been reached on the 78 T E C H T A L L Y
|
From page 79... ...
To encourage conceptual change, then, instructors must help students both develop knowledge pieces or resources relevant to the situation and then activate students' existing productive resources they may not have considered relevant (Smith et al., 1994)
|
From page 80... ...
· How well do current theories of conceptual change in science map to what occurs in technological learning? Research on Technological Learning Technological literacy is a dynamic characteristic developed over a lifetime.
|
From page 81... ...
Concept maps by experts reflect the organization and structure of their knowledge, whereas concept maps by novices tend to reflect their less integrated, less structured knowledge. Thomson concluded that, although concept mapping might be useful for assessing student knowledge structures, a great deal of work remains to be done to validate the use of concept mapping in technology-education research.
|
From page 82... ...
Both novice and expert design tasks designers manage a range of concurrent cognitive actions, but novices lack differently. metacognitive strategies for organizing their activities (Kavakli and Gero, 2002)
|
From page 83... ...
Research on how individuals learn the engineering design process 2This section includes material adapted from Waller, 2004.
|
From page 84... ...
Taken together, these studies indicate several features of mental models: (1) they are developed initially through everyday experiences; (2)
|
From page 85... ...
Second, learning preferences not only affect the way people learn, but also how they interact with assessment instruments, and engineering concepts can be expressed in many different ways (e.g., mathematics, diagrams, analogies, and verbal descriptions)
|
From page 86... ...
If these items are used, an assessment instrument may indicate differences in conceptual understanding that actually reflect socioeconomic and/or cultural differences among study participants. References AAAS (American Association for the Advancement of Science)
|
From page 87... ...
International Journal of Science Education 21(1)
|
From page 88... ...
Interna tional Journal of Science Education 20(10)
|
From page 89... ...
International Journal of Science Education 18(8)
|
From page 90... ...
International Journal of Technology and Design Educa tion 13(1)
|
From page 91... ...
Dordrecht, The Netherlands: Kluwer Academic Publishers. Reprinted from International Journal of Technology and Design Education 7(1-2)
|
From page 92... ...
1992. Mental models of the Earth: a study of conceptual change in childhood.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.