Page 1
Executive Summary
Since 1969, the National Assessment of Educational Progress (NAEP) has been assessing educational attainment across the country. Mandated by Congress, NAEP surveys the educational accomplishments of students in the United States, monitors changes in achievement, and provides a measure of student learning at critical points in their school experience. NAEP results are summarized for the nation as a whole and for individual states with sufficient numbers of participating schools and students.
NAEP's sponsors believe that NAEP could provide useful data about educational achievement below the state level. They suggest that below state results “could provide an important source of data for informing a variety of education reform efforts at the local level” (National Assessment Governing Board, 1995b). In addition, district-level reporting could provide local educators with feedback in return for their participation in NAEP, something that NAEP's sponsors believe might increase motivation to participate in the assessment. Reporting results below the state level was prohibited until 1994. The Improving America's Schools Act of 1994, which reauthorized NAEP in that year, removed the language prohibiting below-state reporting and set the stage for consideration of reporting district-level and school-level results.
At the same time, NAEP's sponsors have been taking a critical look at their reporting procedures with an eye toward improving the usefulness and interpretability of reports. An overarching principle in their recent redesign policy is to define the audience for NAEP reports and to vary the
Page 2
kind and amount of detail in reports to make them most useful for the various audiences. Accordingly, NAEP's sponsors have funded studies to examine the ways in which reports are used by policy makers, educators, the press, and others and to identify common misuses and misinterpretations of reported data.
Within the context of the redesign proposals, the idea of market-basket reporting emerged as a way to better communicate what students in the United States know and are able to do at grade levels tested by NAEP. The market-basket concept is based on the idea that a relatively limited set of items can represent some larger construct. NAEP's sponsors draw parallels between the proposed NAEP market basket and the Consumer Price Index (CPI). The proposed NAEP market basket would consist of a publicly released collection of items intended to represent the content and skills assessed. Percent correct scores, a metric NAEP's sponsors believe is widely understood, will be used to summarize performance on the collection of items.
STUDY APPROACH
At the request of the Department of Education, the National Research Council formed the Committee on NAEP Reporting Practices to address questions about the desirability, feasibility, and potential impact of implementing these reporting practices. The committee developed study questions designed to address issues surrounding district-level and market-basket reporting. Study questions focused on the:
-
characteristics and features of the reporting methods,
-
information needs likely to be served,
-
level of interest in the reporting practices,
-
types of inferences that could be based on the reported data,
-
implications of the reporting methods for NAEP, and
-
implications of the reporting methods for state and local education programs.
To gather information on these issues, the committee reviewed the literature and policy statements on these two reporting practices; invited representatives from the National Assessment Governing Board (NAGB) and the National Center for Education Statistics (NCES) to attend their meetings and present information; attended NAGB board and sub-
Page 3
committee meetings; held a discussion during the Large Scale Assessment Conference sponsored by the Council of Chief State School Officers (CCSSO); arranged for a briefing on the CPI; and convened two multiday workshops. One workshop focused on district-level reporting, the other addressed market-basket reporting.
DISTRICT-LEVEL REPORTING
NAEP's sponsors believe that reporting district-level NAEP results would support local and state education reform efforts. Their rationale is that reporting NAEP performance for school districts has the potential to enable comparisons that cannot be made based on existing assessment results: comparisons of district-level achievement results across state boundaries and comparisons of district-level results with national assessment data.
Opinions about the desirability of such data are varied. Some participants in the committee's workshop believed the information would be uniquely informative. For example, comparisons among districts with similar demographic characteristics would allow them to identify those performing better than expected and instructional practices that work well. Others were attracted to the prospect of having a means for external validation and considered NAEP to be a stable external measure of achievement for making comparisons with their state and local assessment results. Another appealing feature to workshop participants was the possibility of assessment results in subject areas and grades not tested by their state or local programs. In addition, NAEP collects background data that many states and districts do not have the resources to collect, and they would look forward to receiving reports that associate district-level performance with background and school environmental data.
Other workshop participants were wary of the ways data might be used. Officials from some of the larger urban areas maintained that they were already aware that their children do not perform as well as those from other districts. Another set of assessment results would provide yet another opportunity for the press and others to criticize them. Some expressed concern about alignment issues, noting that their curricula do not necessarily match the material tested on NAEP. Attempts to use NAEP as a means of external validation for the state assessment would be problematic when the state assessment is aligned with instruction and NAEP is not, particularly if results from the different assessments suggest different findings about student achievement.
Page 4
Given workshop participants' comments and the materials reviewed, the committee's most overriding concern about developing a program for reporting district-level NAEP results relates to districts' and states' levels of interest. Previous attempts at reporting district results (in 1996 and 1998) indicated virtually no interest in receiving district-level summaries of performance. Workshop participants' reactions were mixed, in part due to the lack of information about the goals, objectives, specifications, and costs of a district-level program. It is not clear what district-level reporting is intended to accomplish or whom the program would serve—only large urban districts or smaller districts as well. Decisions have not been made about the types of information districts and states would receive, when they would receive the information, how much it would cost, or who would pay the costs. These details need to be resolved before NAEP's sponsors can expect to gauge actual interest in receiving district-level results. Once the details are specified, then it is important to determine if there is sufficient interest to justify pursuing the program. On these points, the committee offers the following recommendations:
RECOMMENDATION: Market research emphasizing both needs analysis and product analysis is necessary to evaluate the level of interest in district-level reporting. The decision to move ahead with district-level reporting should be based on the results of market research conducted by an independent market-research organization. If market research suggests that there is little or no interest in district-level reporting, NAEP's sponsors should not continue to invest NAEP's limited resources pursuing district-level reporting.
RECOMMENDATION: If the decision is made to proceed with district-level reporting, NAEP's sponsors should develop and implement a plan for program evaluation, similar to the research conducted during the initial years of the Trial State Assessment, that would investigate the quality and utility of district-level NAEP data.
Page 5
MARKET-BASKET REPORTING
Large-Scale Release of Items
The proposal for a NAEP market basket emanates from desires to make NAEP results more meaningful and more easily understood. According to workshop participants, the large-scale release of a market-basket set of items could demystify the assessment by providing many examples of the content and skills assessed and the format of items. Review of the content and skill coverage could stimulate discussions with local educators about their curricula and instructional programs. Review of NAEP item formats could lead to improvements in the design of items used with state, local, and classroom-based assessments. In addition, public review of the released materials could enhance understanding of the goals and purposes of the national assessment and might lead to increased public support for testing. Although workshop participants were generally positive about a large-scale release of items, they noted that a large release could be overly influential on local and state curricula or assessments. For instance, policy makers and educators concerned about their NAEP performance could attempt to align their curricula more closely with what is tested on NAEP. Assessment, curricula, and instructional practices form a tightly woven system—making changes to one aspect of the system can have an impact on other aspects of the system.
Percent Correct Scores
Using percent correct scores to summarize performance on the market basket is intended to make test results more user friendly. Because nearly everyone who has passed through the American school system has at one time or another been tested and received a percent correct score, most people could be expected to understand such scores. NAEP's sponsors believe that percent correct scores would have immediate meaning to the public.
Based on workshop participants' reactions, it is doubtful that percent correct scores would be more easily understood than the achievement-level results that NAEP currently reports. Many users have become accustomed to achievement level reporting; moving to a percent correct metric would require new interpretative assistance. Further, the use of this metric presents a number of challenges. For example, it is unclear whether percent
Page 6
correct scores would indicate percent of questions answered correctly or percent of possible points—both pose complications due to the mix of multiple-choice items and constructed response tasks scored on multipoint scales. In addition, NAEP results are not currently reported on a percent correct metric.
The NAEP Short Form
With the NAEP short form, the release of exemplar items would be smaller, but an intact form would be provided to states and districts to administer as they see fit. NAEP's sponsors hope that the short form will enable faster and more understandable reporting of results. Initial plans call for a fourth-grade mathematics short form, but the ultimate plan might be to develop short forms for a variety of subjects for states to use in years when NAEP assessments are not scheduled for particular subjects. The policy guiding development of the short forms stipulates that results be reported using NAEP achievement levels.
Many workshop participants found the idea of the short form to be appealing, but their comments reflected a variety of conceptions about the characteristics of a short form. Several envisioned the short form as a set of items that could be embedded into existing assessments to link results from state and local assessments with NAEP, while others viewed the short form as a mechanism for providing more timely reporting of NAEP results. Still others see it as a means for facilitating district-level or school-level reporting. It is not clear that all of these desired uses would be supported.
These widely divergent conceptions are exacerbated by the limited policy guidance NAGB has provided. While the generality of policy statements is appropriate so that developers are not limited in the approaches they might consider to put policy into practice, the lack of detail makes the short form an amorphous concept open to a variety of interpretations. Too many details remain undecided for the committee to make specific recommendations about the short form.
CONCLUSION: Thus far, the NAEP short form has been defined by general NAGB policy, but it has not been developed in sufficient technical and practical detail to allow potential users to react to a firm proposal. Instead, users are projecting into the general idea their own desired characteristics for the short form, such as an anchor for linking scales. Some of their
Page 7
ideas and desires for the short form have already been determined to be problematic. It will not be possible for a short-form design to support all uses described by workshop participants.
Long Market Baskets versus Short-Form Market Baskets
All configurations for the market basket will involve tradeoffs. A market basket comprised of a large collection of items is more likely to be representative of the NAEP frameworks. As currently conceived, the NAEP short forms consist of approximately 30 items to be administered during a 60-minute testing period. A collection this small is unlikely to adequately represent the NAEP frameworks. Deriving results from the short form that are representative of the NAEP frameworks, technically sound, and comparable across versions of the short forms and to main NAEP results pose significant challenges. On these points, the committee makes the following recommendation.
RECOMMENDATION: All decisions about the configuration of the NAEP market basket will involve tradeoffs. Some methods for configuring the market basket would result in simpler procedures than others but would not support the desired inferences. Other methods would yield more generalizable results but at the expense of simplicity. If the decision is made to proceed with designing a NAEP market basket, its configuration should be based on a clear articulation of the purposes and objectives for the market basket.
ANALOGIES WITH THE CONSUMER PRICE INDEX MARKET BASKET
Because analogies have frequently been made between the NAEP market basket and the Consumer Price Index (CPI), the committee investigated the extent to which such comparisons hold. In considering the proposals to develop and report a summary measure from the existing NAEP frameworks, the committee realized that the purpose and construction of the CPI market basket differs fundamentally from the corresponding elements of current NAEP proposals. The task of building an educational parallel to the CPI is formidable and appears to differ conceptually from
Page 8
the current NAEP market-basket development activities. Implementing a true “market-basket” approach in NAEP would thus necessitate major operational changes. Most importantly, a large national survey would need to be conducted to determine what students are actually taught in U.S. classrooms. Survey results would be used to construct the market basket, and then students would be tested to evaluate performance on the market basket.
Furthermore, the market-basket metaphor may be inappropriate. A market basket is an odd, even jarring image in the context of educational achievement. Most people do not see education as a consumer purchase or an assortment of independent parcels placed in a shopping cart. On these points, we find:
CONCLUSION: Use of the term “market basket” is misleading because (1) the NAEP frameworks reflect the aspirations of policy makers and educators and are not purely descriptive in nature and (2) the current operational features of NAEP differ fundamentally from the data collection processes used in producing the CPI.
RECOMMENDATION: In describing the various proposals for reporting a summary measure from the existing NAEP frameworks, NAEP's sponsors should refrain from using the term “market basket” because of inaccuracies in the implied analogy with the CPI.
RECOMMENDATION: If, given the issues raised about market-basket reporting, NAEP's sponsors wish to pursue the development of this concept, they should consider developing an educational index that possesses characteristics analogous to those of the Consumer Price Index: (1) is descriptive rather than reflecting policy makers' and educators' aspirations; (2) is reflective of regional differences in educational programs; and (3) is updated regularly to incorporate changes in curriculum.
ENHANCING REPORTS
When NAEP began reporting state-level results in 1990, researchers and others expressed concerns about potential misinterpretation or misuse
Page 9
of the data. Although not all the dire predictions came true, reports of below-state NAEP results increase the potential for misinterpretation problems. Given the amount of attention that below-state results would likely receive, whether derived from main NAEP or via a NAEP short form, significant attention should be devoted to product design.
As part of our study, the committee hoped to be able to review prototypic reports for the proposed reporting methods. While some preliminary examples of district-level and market-basket reports were available, NAEP's sponsors have not made definitive decisions about the format of reports. Given the stage of report design, we conducted a review of the literature on NAEP reporting procedures and examined examples of NAEP reports. Based on these reviews, we offer suggestions and recommendations for report design.
The design of data displays should be carefully evaluated and should evolve through methodical processes that consider the purposes of the data, the needs of users, the types of interpretations, and the anticipated types of misinterpretations. User-needs analysis is an appropriate forum for determining both product design and effective metaphors for aiding in communication.
Even if the proposals for district-level and market-basket reporting are not implemented, attention to the way NAEP information is provided would be useful. The types of NAEP reports are many and varied. The information serves many purposes for a broad constellation of audiences, including researchers, policy makers, the press, and the public. The more technical users as well as the lay public look to NAEP to support, refute, or inform their ideas about students' academic accomplishments. The messages taken from NAEP's data displays can easily influence their perceptions about the state of education in the United States. We therefore recommend:
RECOMMENDATION: Appropriate user profiles and needs assessments should be considered as part of the integrated design of district-level and market-basket reports. The integration of usability as part of the overall design process is essential because it considers the information needs of the public.
RECOMMENDATION: The text, graphs, and tables of reports developed for market-basket and district-level reporting should be subjected to standard usability engineering tech-
Page 10
niques including appropriate usability testing methodologies. The purpose of such procedures would be to make reports more comprehensible to their readers and more accessible to their target audiences.
IMPLICATIONS FOR NAEP AND LOCAL EDUCATIONAL SYSTEMS
The two proposed reporting practices would provide new information that would receive attention from new audiences—audiences that may have not previously attended to NAEP results. The use of such information by policy makers, state and local departments of education, the press, and the lay public could have a significant impact on NAEP and on state and local assessment, curriculum, and instruction. In addition, these reporting methods pose challenges for NAEP's current procedures, including item development, sampling procedures, analytic and scoring methodologies, and report preparation.
NAEP has traditionally been a low-stakes assessment, but reporting results at a level closer to those responsible for instruction raises the stakes. With higher stakes comes the need to pay greater attention to security issues. In addition, motivation to do well may increase, which could affect the comparability of NAEP results across time and across jurisdictions, depending on how jurisdictions use the new results.
Introducing new products and procedures to an already complex system has significant cost and resource implications. To construct short forms and to accommodate security considerations, item development would need to be stepped up. Sampling procedures would need to be altered and additional students tested to support district-level results. Analytic methodologies would need to be adapted. The types and numbers of reports to be produced would affect report preparation, possibly increasing the length of time to release results. These factors would require fundamental changes in NAEP's processes, operations, and products.
For local education systems, the reporting practices could increase the attention on NAEP results. Current assessments might be replaced or altered to accommodate NAEP's schedule or to be modeled more closely after the NAEP frameworks and item formats. There could be efforts to align instructional programs more closely with the NAEP frameworks. If NAEP were to report percent correct scores, states and districts might consider following suit for local assessments and change to a metric that may
Page 11
not lead to improved understanding of NAEP or local test results. It is not clear that these changes would be beneficial to local education systems, and the implementation of these reporting approaches would require support systems to aid districts and states in appropriate uses and interpretations of the reported results. We therefore recommend:
RECOMMENDATION: The potential is high for significant impact on curriculum and /or assessment at the local levels. If either district-level reporting or market-basket reporting, with or without a short form, is planned for implementation, the program sponsors should develop and implement intensive support systems to assist districts and states in appropriate uses and interpretations of any such NAEP results reported.
