THE MARKET-BASKET CONCEPT
The purpose of the National Research Council's Workshop on Market-Basket Reporting was to explore with various stakeholders their interest in and perceptions regarding the desirability, feasibility, and potential impact of market-basket reporting for the National Assessment of Educational Progress (NAEP). The market-basket concept is based on the idea that a relatively limited set of items can represent some larger construct. The most common example of a market basket is the Consumer Price Index (CPI) produced by the Bureau of Labor Statistics. The CPI tracks changes in the prices paid by urban consumers in purchasing a representative set of consumer goods and services. The CPI measures cost differentials from month to month for products in its market basket; therefore, the CPI is frequently used as an indicator of change in the U.S. economy. In the context of the CPI, the concept of a market basket resonates with the general public; it invokes the tangible image of a shopper going to the market and filling a basket with a set of goods that is regarded as broadly reflecting consumer spending patterns.
The general idea of a NAEP market basket draws on a similar image: a collection of test questions representative of some larger content domain; and an easily understood index to summarize performance on the items. There are two components of the NAEP market basket, the collection of items and the summary index. The collection of items could be large
(longer than a typical test form given to a student) or small (small enough to be considered an administrable test form). The summary index currently under consideration is the percent correct score.
At present, several alternatives have been proposed for the NAEP market basket. Figure 1 provides a diagram of the various components of the market basket and shows how they relate to two alternate scenarios under which the market basket would be assembled and used. Under one sce-
nario, a large collection of items would be assembled and released publicly. To adequately cover the breadth of the content domain, the collection would be much larger than any one of the forms used in the test and probably too long to administer to a single student at one sitting. This presents some challenges for the calculation of the percent correct scores. Because no student would take all of the items, complex statistical procedures would be needed for estimating scores. This alternative appears in Figure 1 as “scenario one.”
A second scenario involves producing multiple, “administrable” test forms (called “short forms”). Students would take an entire test form, and scores could be based on students'performance for the entire test in the manner usually employed by testing programs. Although this would simplify calculation of percent correct scores, the collection of items would be much smaller and less likely to adequately represent the content domain. This scenario also calls for assembling multiple test forms. Some forms would be released to the public, while others would remain secure, perhaps for use by state and local assessment programs, and possibly to be embedded into or administered in conjunction with existing tests. This alternative appears in Figure 1 as “scenario two.”
THE COMMITTEE ON NAEP REPORTING PRACTICES
At the National Research Council (NRC), the study on market-basket reporting is being handled by the Committee on NAEP Reporting Practices. The NRC established this committee in 1999 at the request of the United States Department of Education to examine the feasibility, desirability, and potential impact of district-level and market-basket reporting practices. Because issues related to these reporting practices are intertwined, the committee is examining them in tandem. The committee 's study questions regarding district-level reporting are as follows:
What are the proposed characteristics of a district-level NAEP?
If implemented, what information needs might it serve?
What is the degree of interest in participating in district-level NAEP? What factors would influence interest?
Would district-level NAEP pose any threats to the validity of inferences from national and state NAEP?
What are the implications of district-level reporting for other state and local assessment programs?
District-level reporting was the focus of a workshop held in September 1999. The proceedings from this workshop were summarized and published (see National Research Council, 1999c).
The committee's study questions with respect to market-basket reporting are as follows:
What is market-basket reporting?
How might reports of market-basket results be presented to NAEP's audiences? Are there prototypes?
What information needs might be served by market-basket reporting for NAEP?
Are market-basket results likely to be relevant and accurate enough to meet these needs?
Would market-basket reporting pose any threats to the validity of inferences from national and state NAEP? What types of inferences would be valid?
What are the implications of market-basket reporting for other national, state, and local assessment programs? What role might a NAEP short form play?
On February 7 and 8, 2000, the committee convened the Workshop on Market-Basket Reporting to begin to address the questions outlined above regarding NAEP market-basket reporting. The committee's further consideration of both district-level and market-basket reporting will be reflected in its final report, scheduled for release in November 2000.
WORKSHOP ON MARKET-BASKET REPORTING
The workshop opened with a panel of representatives from the organizations involved in setting policy for and operating NAEP: the National Assessment Governing Board (NAGB) and the National Center for Education Statistics (NCES). Also included were individuals from the Educational Testing Service (ETS), the contractual agency that works on NAEP. Panel members talked about the perceived needs that led to consideration of the market basket and plans for conducting research on the market basket. The panel included:
Roy Truby, executive director of NAGB
Andrew Kolstad, senior technical advisor for the Assessment Division at NCES
Robert Mislevy, distinguished research scholar with ETS
John Mazzeo, executive director of ETS's School and College Services
Prior to the workshop, each panel member prepared a paper addressing questions specified by the committee. At the workshop, time was allotted for panel members to present their papers orally. The initial questions posed to these panel members and synopses of their papers and presentations appear in Chapter 4 of this report.
The committee invited individuals representing a variety of perspectives to serve as discussants at the workshop and to react to the material presented by the opening panel speakers. These discussants received copies of the speakers' papers several weeks in advance of the workshop along with a set of questions to address in their comments (see Agenda in Appendix A for these questions). Each discussant made an oral presentation during the workshop and subsequently submitted written copy of his or her remarks.
The first discussant panel responded from a policy perspective, highlighting the impact the market-basket proposal might have on state and local education policy for instruction and assessment programs. This panel included Wayne Martin, director of the Council of Chief State School Officers' State Education Assessment Center; Marilyn McConachie, a member of the Illinois State Board of Education and former member of NAGB; Carroll Thomas, superintendent of public schools in Beaumont, Texas; and Marlene Hartzman, an evaluation specialist with the public school system in Montgomery County, Maryland.
Assessment and Curriculum
A second discussant panel explored the perspectives of data users and practitioners. This panel considered the impact and uses of the market basket with respect to state and local assessment programs, curricula, and instructional practices. Panel members included Scott Trimble, director of assessment with Kentucky's Department of Education; Joseph O'Reilly, director of assessment for the public school system in Mesa, Arizona, and past
president of the National Association of Test Directors (NATD); and Ronald Costello, assistant superintendent for public schools in Noblesville, Indiana.
A third panel of discussants responded to technical and measurement issues related to the market basket. This panel comprised well-known statisticians who have worked extensively with NAEP. Panel members were Darrell Bock, a faculty fellow with the University of Chicago 's Department of Psychology; David Thissen, professor of psychology at the University of North Carolina-Chapel Hill; and Donald McLaughlin, chief scientist with the American Institutes of Research (AIR).
Content and Skill Coverage
Patricia Kenney, research associate with the Learning Research and Development Center at the University of Pittsburgh, was invited to speak with respect to the content and skill coverage of the collections of items included in the market basket. Kenney has extensive knowledge of the NAEP mathematics frameworks through her work with the National Council of Teachers of Mathematics (NCTM), her role as co-director of the NCTM NAEP Interpretive Reports Project, and her work comparing the congruence between NAEP and state assessments in North Carolina and Maryland.
First in the World Consortium
Two school district superintendents, Paul Kimmelman (of West Northfield, Illinois) and Dave Kroeze (of Northbrook, Illinois), discussed the work of the “First in the World Consortium.” This group of 20 school districts in Illinois participated in and received results from the Third International Mathematics and Science Study (TIMSS) as part of their efforts to achieve their education goals and to work toward world-class standards. Kimmelman and Kroeze described the ways in which consortium members have used TIMSS results to guide changes in instruction and assessment. Their comments provided insights into how individual school districts might use the released short-form version of the NAEP market-basket.
A Newspaper Reporter's Perspective
The committee was interested in what the newspaper-reading general public wants to know about student achievement. Committee members wanted to hear a reporter's perspective on how the press might use results from the market basket and the types of conclusions that might be drawn from the information. Richard Colvin, education reporter with the Los Angeles Times, served as a discussant at the workshop.
The Consumer Price Index
The committee also solicited information about market baskets and summary indicators as they are used in other contexts, such as the Consumer Price Index (CPI), which is most frequently cited as an analogy for the NAEP market basket. Kenneth Stewart, chief of the Information and Analysis Section at the Bureau of Labor Statistics, spoke about how the CPI is formed using a market basket of representative consumer goods and services and how the CPI influences other economic measures.
ORGANIZATION OF THIS REPORT
The purpose of this report is to capture the discussions and major points made during the market-basket workshop in order to assist NAEP's sponsors in their decision making about the feasibility, desirability, and potential impact of the NAEP market-basket proposal. The workshop permitted a considerable amount of open discussion by presenters, as well as by participants, much of which is woven into this summary report. As a summary, this report is intended to highlight the key issues identified by the various stakeholders who attended the workshop, but it does not attempt to establish consensus on findings or recommendations. The Committee on NAEP Reporting Practices will publish its final report in November 2000 that will include findings from the entire study and will offer recommendations with respect to district-level and market-basket reporting.
The concept of a NAEP market basket was first addressed by NAGB in “Redesigning the National Assessment of Educational Progress” (National Assessment Governing Board, 1996) and is again discussed in “National Assessment of Educational Progress: Design 2000-2010 Policy” (National Assessment Governing Board, 1999b). Background on these two
redesign efforts appears in Chapter 2 to provide readers with an understanding for the motivations behind the market-basket proposal.
Since the NAEP market-basket concept has often been compared to the Consumer Price Index, the information provided to workshop participants by Kenneth Stewart appears in the next chapter. Chapter 3 is intended to acquaint the reader with summary indicators in other contexts and establish background for the exploration of an analogous summary indicator for NAEP.
Chapter 4 contains synopses of the papers and presentations by NAEP's sponsoring agencies (NAGB and NCES) and test development contractor (ETS). Because this material set the stage for discussants' presentations, these summaries are provided to help the reader understand the basis for discussants' remarks.
Chapter 5, Chapter 6 through Chapter 7 highlight the comments made by discussants and other workshop participants. These chapters consider features of the market basket in relation to the NAEP redesign objectives that market-basket reporting has been conceptualized to address, that is, using innovations in measurement and reporting and simplifying NAEP's technical design (as described in Chapter 2). Because there was considerable overlap in the nature of the comments made during the workshop, Chapter 5, Chapter 6 through Chapter 7 are organized around the central issues raised during the workshop rather than according to the chronological delivery of discussants ' remarks panel by panel.
Chapter 8 concludes the report by highlighting issues to consider and resolve as NAEP's sponsors develop future plans for the market basket.