Tech Tally Approaches to Assessing Technological Literacy (2006) / Chapter Skim
Currently Skimming:
3 Assessment as a Design Challenge
3 Assessment as a Design Challenge
Pages 41-62
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 41... ...
In short, they are engaging in design processes. Webster's dictionary defines design as "a mental project or scheme 41
|
From page 42... ...
In fact, the engineering design process is also well suited to solving any problem that is poorly or imperfectly defined, such as the problem facing the committee. The challenge of assessing technological literacy is difficult for several reasons.
|
From page 43... ...
for assessing technological literacy. And third, although not part of the committee's formal task, the committee was able to promote technological literacy by demonstrating how the engineering design process can be used to address non-engineering problems.
|
From page 44... ...
collected and analyzed existing assessment instruments to research. provide data for brainstorming sessions; (2)
|
From page 45... ...
Designers of a particular specific assessment instruments can follow the road map, keeping the solution. particular purpose or goal of that assessment in mind.
|
From page 46... ...
The committee was able to identify only a handful of technology-related assessment instru ments (see Chapter 5 for descriptions) , and none of these mapped very well to the idea of technological literacy presented in Technically Speaking.
|
From page 47... ...
Prior to the event, all participants were provided with copies of Technically Speaking and a brief summary of the project goals and objectives, and nearly all of them prepared short written statements expressing their views on the opportunities and obstacles to assessing technological literacy. An indepen dent evaluation company, The Study Group, conducted pre- and post-workshop interviews with partici pants to elicit more in-depth views on the issues.
|
From page 48... ...
As we will show in Chapter 6, when the design process is applied to the development of a specific type of assessment instrument, the constraints also become more specific. Identify Design Criteria Once the constraints have been determined, the specific criteria, or goals, for the design can be identified.
|
From page 49... ...
No single assessment instrument is likely to be effective for more than one target audience. Assessments for technological literacy should produce valid and reliable data on as many of the three dimensions of technological literacyaspossible.
|
From page 50... ...
BOX 3-4 Characteristics of Higher-Order Thinking Higher order thinking · is nonalgorithmic, with the path of action not fully specified in advance · tends to be complex, with the total path not "visible" from any single mental vantage point · yields multiple solutions, each with costs and benefits, rather than a unique solution · involves nuanced judgment and interpretation · involves the application of multiple criteria, which sometimes conflict with one another · often involves uncertainty because everything that bears on the task at hand may not be known · involves self-regulation of the thinking process · involves imposing meaning, or finding structure, in apparent disorder · is effortful, requiring considerable mental work to make elaborations and judgments SOURCE: Adapted from Resnick, 1987.
|
From page 51... ...
put on different areas of content, depending on the learning goals, the age of the test population, and other factors. For example, the Illinois Science Assessment Framework suggests that 20 percent of questions focus on student understanding of science inquiry and technological design; 60 percent on major scientific themes, such as living things, matter and A S S E S S M E N T A S A D E S I G N C H A L L E N G E 51
|
From page 52... ...
However, in all cases, there is a strong connection between the framework and relevant standards. Assessment frameworks often provide test developers with de tailed suggestions on the nitty-gritty of test construction, such as the number of test items; the number of multiple choice, short answer, and other types of questions; the relative "weight" of the parts of the assess ment; the amount of time allotted for each section; and the best type of scoring (e.g., CSAP, 2004)
|
From page 53... ...
According to the framework, technological design will be the focus of 10 percent of the 2009 NAEP items. The 1996, 2000, and 2003 NAEP mathematics frameworks included five primary content "strands" -- number sense, properties, and operations; measurement; geometry and spatial sense; data analysis, statistics, and probability; and algebra and functions -- and three cognitive domains, called "math abilities" -- conceptual understanding, procedural knowledge, and problem solving (NAGB, 2002)
|
From page 54... ...
and NRC (1996) , both of which address the nature of technology and the relationship between technology and sci ence, the committee adapted the three dimensions of technological lit eracy proposed in Technically Speaking -- knowledge, capabilities, and critical thinking and decision making2 -- as the cognitive elements in the matrix.
|
From page 55... ...
NAEP Mathematics Conceptual Procedural knowledge Problem solving Framework understanding Committee on Assessing Knowledge Capability Critical thinking and Technological Literacy decision making NOTE: The 2009 NAEP science framework departs from earlier NAEP models by recasting the cognitive aspects of science knowledge and skills. These differences make it difficult to fit the new model to previous NAEP efforts and to the committee's own work.
|
From page 56... ...
Research found that was also useful for investigating whether ideas that arose during brain research was storming sessions met the criteria and constraints of the problem at hand. helpful in every In fact, the committee found that research was helpful in every phase of the phase of the design process, from defining the problem to generating design process.
|
From page 57... ...
The com processes, no mittee was then left with the challenging task of providing guidance single proposed without the benefit of an existing model. solution meets In most design processes, no single proposed solution meets all of all of the criteria.
|
From page 58... ...
In the end, it is probably more important that assessment developers care fully assess how their design choices relate to the criteria and which criteria are most important for achieving the purpose of the assessment than that they use a formal matrix analysis. The committee analyzed the two alternative conceptualizations in light of relevant content standards.
|
From page 59... ...
The product of the committee's design process is this report rather than an actual assessment instrument. The principle impetus for refinements to the report was comments from the 11 outside reviewers of the document.
|
From page 60... ...
The road map is intended to provide a workable approach for many years, even if the environment for assessing technological literacy changes significantly. References AAAS (American Association for the Advancement of Science)
|
From page 61... ...
2005. Science NAEP 2009: Science Framework for the 2009 National Assessment of Educational Progress -- Prepublication Edition.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.