National Academies Press: OpenBook

Designing a Market Basket for NAEP: Summary of a Workshop (2000)

Chapter: 5 Using Innovations in Measurement and Reporting: Releasing a Representative Set of Test Questions

« Previous: 4 The Perspectives of NAEP's Sponsors and Contractors
Suggested Citation:"5 Using Innovations in Measurement and Reporting: Releasing a Representative Set of Test Questions." National Research Council. 2000. Designing a Market Basket for NAEP: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/9891.
×

5

Using Innovations in Measurement and Reporting: Releasing a Representative Set of Test Questions

A basic component of the NAEP market basket is the release of a representative set of test items. While NAEP has always released samples of items, under plans for the market basket, many more items would be released with the goal of representing the content domain for a specific subject. For example, the collection of items for fourth grade mathematics would consist of the appropriate number and mix of test items needed to represent the domain of fourth grade mathematics, as defined by NAEP's frameworks. The Committee on NAEP Reporting Practices was interested in the purposes that might be served by such a release of items and requested that workshop discussants consider who might use this information and how it might be used. The discussion below attempts to summarize the major points made by the speakers.

DEMYSTIFYING THE TEST

Workshop discussants remarked that an aura of mystery surrounds testing. In their interactions with the public, they have found that people question why so much time is devoted to testing and are unsure of how to interpret the results. The public is not fully aware of the material that is covered on achievement tests, the skills that students are expected to demonstrate, and the inferences about student achievement that can be drawn from test results. Moreover, the public does not always see the link between assessment programs and school reform efforts.

Suggested Citation:"5 Using Innovations in Measurement and Reporting: Releasing a Representative Set of Test Questions." National Research Council. 2000. Designing a Market Basket for NAEP: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/9891.
×

Marilyn McConachie, Illinois State Board of Education member, summarized these perceptions most succinctly, saying:

Analysis of public understanding of test results in Illinois parallels national commentary on NAEP and other large-scale testing. Put simply, the public believes too much time is spent on testing and doesn' t really understand what students know and are able to do or whether performance is good enough. These beliefs appear to erode support for state testing (and for NAEP).

According to McConachie, only when tests are “demystified” will the public understand what is being tested and why, and only then will the public support the continued gathering of this important information.

Other workshop discussants commented that public release of test items, scoring rubrics, and student work samples could serve to further public understanding of what NAEP tests. Many felt that the public is not generally aware of the difficulty level of the material covered on achievement tests today. For most people, the basis of comparison is their own school experiences, but curricula and expectations for students have changed. Joseph O'Reilly, director of assessment for Mesa, Arizona, public schools offered an example that illustrates this perception:

The public does not seem to understand the difficulty of the concepts taught today compared to when they were in school, especially in mathematics. For example, basic trigonometric angles are commonplace in algebra today but were not covered until much later in the curriculum, if at all, thirty years ago.

O'Reilly believes that the release of a large collection of items would be very useful in communicating the higher levels of expectations of tests like NAEP. Additionally, because many state testing programs cover content similar to that tested by NAEP, the information learned from NAEP's release of items could also increase understanding of state and local assessments.

STIMULATING DISCUSSION AMONG TEACHERS

Workshop speakers observed that release of a large number of representative items could be used to stimulate discussion among teachers regarding the format and content of test questions. In addition, review of the released items could facilitate discussions about ways to align local curricula and instructional practices with the material covered on the national assessment. Discussants explained that it is often difficult to draw conclusions about their students' NAEP performance because it is not clear

Suggested Citation:"5 Using Innovations in Measurement and Reporting: Releasing a Representative Set of Test Questions." National Research Council. 2000. Designing a Market Basket for NAEP: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/9891.
×

whether the material tested on NAEP is covered by their curricula or when it is covered.

The superintendents representing the First in the World Consortium provided examples of how they utilized information from their students ' participation in the Third International Mathematics and Science Study (TIMSS) to guide local curricular and instructional changes. The consortium participants did not have access to actual test items but, instead, had information on the specific topics covered and the content areas assessed. Teams of teachers from consortium schools examined the topics covered on the TIMSS assessment. They also considered when and how these topics were presented in their curricula, discussing such issues as: at what grade level the topic is first introduced in their programs; at what grade level mastery of the topic is expected; and how the topic is reinforced over the grades. Analysis of their TIMSS results, particularly of students' strengths and weaknesses, in comparison to teaching and instructional practices, allowed the participating school systems to identify needed changes in their curricula.

This use of TIMSS materials exemplifies one potential use of released NAEP materials. While the First in the World Consortium school systems did not see actual items, they had the benefit of receiving information on the topics and content areas covered by TIMSS items. For school systems and others to realize similar benefits from the released NAEP materials, items would need to be categorized into frameworks and content areas. Otherwise, teachers and other users might categorize items themselves, perhaps incorrectly, in order to make inferences about the relationships between material tested and content covered by their curricula.

Workshop discussants also suggested that it would be useful to associate the released items with the NAEP achievement level category of students expected to answer the question correctly. This matching of items with achievement levels would demonstrate the content and skills students should have mastered at each level, which would facilitate understanding of the assessment. For example, if teachers were able to view items that illustrated what students scoring at the proficient level in fourth-grade mathematics should be able to do, they would be able to adjust their teaching accordingly. Used diagnostically, this information could help students progress from below basic to basic, from basic to proficient, and from proficient to advanced.

Suggested Citation:"5 Using Innovations in Measurement and Reporting: Releasing a Representative Set of Test Questions." National Research Council. 2000. Designing a Market Basket for NAEP: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/9891.
×

ENCOURAGING IMPROVED STATE AND LOCAL TESTING

NAEP often serves as a role model for the development of state and local assessments and the policy governing those assessments. During the committee's earlier workshop on reporting district-level NAEP results, participants commented that NAEP's frameworks, its innovative item design, and its use of achievement-level reporting have greatly influenced assessments around the country (National Research Council, 1999c.) Similar observations were made at the market-basket workshop.

Participants in the market-basket workshop thought that a large-scale release of NAEP items and related test materials could potentially improve state and local assessment programs. NAEP produces high-quality items and test materials. Allowing test developers to view large amounts of NAEP test materials (test questions as well as rubrics for scoring constructed response items) could therefore have a positive effect on the quality of item design for state and local assessments. The release of high-quality NAEP materials could also help revamp classroom-based assessments. Furthermore, in their opinion, policymakers would be able to see the breadth and depth of the content and skills assessed and the grade levels at which students are expected to have mastered certain subject matter—information that could play an important role in redefining curricula. Participants emphasized, however, that for such objectives to be realized, item release would need to be both large and representative of the domain assessed.

Suggested Citation:"5 Using Innovations in Measurement and Reporting: Releasing a Representative Set of Test Questions." National Research Council. 2000. Designing a Market Basket for NAEP: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/9891.
×
Page 30
Suggested Citation:"5 Using Innovations in Measurement and Reporting: Releasing a Representative Set of Test Questions." National Research Council. 2000. Designing a Market Basket for NAEP: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/9891.
×
Page 31
Suggested Citation:"5 Using Innovations in Measurement and Reporting: Releasing a Representative Set of Test Questions." National Research Council. 2000. Designing a Market Basket for NAEP: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/9891.
×
Page 32
Suggested Citation:"5 Using Innovations in Measurement and Reporting: Releasing a Representative Set of Test Questions." National Research Council. 2000. Designing a Market Basket for NAEP: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/9891.
×
Page 33
Next: 6 Using Innovations in Measurement and Reporting: Reporting Percent Correct Scores »
Designing a Market Basket for NAEP: Summary of a Workshop Get This Book
×
Buy Paperback | $29.00 Buy Ebook | $23.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

At the request of the U.S. Department of Education, the National Research Council (NRC) established the Committee on NAEP Reporting Practices to examine the feasibility and potential impact of district-level and market-basket reporting practices. As part of its charge, the committee sponsored a workshop in February 2000 to gather information on issues related to market-basket reporting for the National Assessment of Education Progress (NAEP).

Designing a Market Basket for NAEP: Summary of a Workshop explores with various stakeholders their interest in and perceptions regarding the desirability, feasibility, and potential impact of market-basket reporting for the NAEP. The market-basket concept is based on the idea that a relatively limited set of items can represent some larger construct. The general idea of a NAEP market basket is based on an image of a collection of test questions representative of some larger content domain and an easily understood index to summarize performance on the items.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!