6

Strategies for Implementing the Committee's Recommendations for Transforming NAEP

In this report we have argued that many of the demands placed on NAEP are a function of the unfilled need for better information about many aspects of the American student population and education system. We have asserted that the nation needs a new definition of educational progress, one that provides a more comprehensive picture of education in America and one that supports the policy interests that drive NAEP's authorization. Our argument rests on the premise that student achievement on a set of large-scale assessments should not be the nation's only marker of educational progress. We argue that policy makers and the public must be informed about other important educational outcomes and about the associations between student achievement and education. We also argue that the NAEP program cannot assume the burden of providing all of the information needed on educational performance and progress and that it should focus instead on improving methods for providing high-quality information about student academic achievement.

In this final chapter, we make suggestions for reconfiguring NAEP to play a key role in a coordinated system of education indicators. We begin with an overview of committee recommendations that have procedural implications for upcoming programmatic activity. We then offer and develop suggestions for effecting the changes we recommend for NAEP. In discussing them, we review work recently undertaken by the U.S. Department of Education that lays a foundation for the assessment changes we describe. When we can, we relate our suggestions for implementation to the timelines and priorities of ongoing activity at the National Center for Education Statistics (NCES) and the National Assessment Governing Board (NAGB).



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 185
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress 6 Strategies for Implementing the Committee's Recommendations for Transforming NAEP In this report we have argued that many of the demands placed on NAEP are a function of the unfilled need for better information about many aspects of the American student population and education system. We have asserted that the nation needs a new definition of educational progress, one that provides a more comprehensive picture of education in America and one that supports the policy interests that drive NAEP's authorization. Our argument rests on the premise that student achievement on a set of large-scale assessments should not be the nation's only marker of educational progress. We argue that policy makers and the public must be informed about other important educational outcomes and about the associations between student achievement and education. We also argue that the NAEP program cannot assume the burden of providing all of the information needed on educational performance and progress and that it should focus instead on improving methods for providing high-quality information about student academic achievement. In this final chapter, we make suggestions for reconfiguring NAEP to play a key role in a coordinated system of education indicators. We begin with an overview of committee recommendations that have procedural implications for upcoming programmatic activity. We then offer and develop suggestions for effecting the changes we recommend for NAEP. In discussing them, we review work recently undertaken by the U.S. Department of Education that lays a foundation for the assessment changes we describe. When we can, we relate our suggestions for implementation to the timelines and priorities of ongoing activity at the National Center for Education Statistics (NCES) and the National Assessment Governing Board (NAGB).

OCR for page 185
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress RECAPITULATION OF THE PRIMARY RECOMMENDATIONS In Chapters 1 through 5, we discussed the creation of a coordinated system of education indicators. We made recommendations for streamlining NAEP's design, enhancing the participation and meaningful assessment of all students in NAEP, providing more informative portrayals of student performance in NAEP reports, and improving achievement-level setting. We argued that: Educational progress is not synonymous with student achievement results on NAEP. The assessment of educational progress should be reconceptualized as a broader, coordinated system of education indicators that includes, but goes beyond, measures of student achievement. Data on curriculum and instruction, academic standards, technology use, financial allocations, and other indicators of education inputs, practices, and outcomes should be collected using a range of methods and included in the coordinated system for assessing educational performance and progress. NAEP's frameworks, achievement measures, performance levels, and results are only one component of the more inclusive system we describe. Within this larger system, the sampling and administration designs for NAEP's survey-based student achievement measures should be streamlined. At the same time, the measurement of achievement by NAEP needs to be reconceptualized so that it capitalizes on contemporary research, theory, and practice in NAEP subjects in ways that support in-depth interpretations of student knowledge and understanding. The student achievement measures should be broadened beyond large-scale assessment methods. Although current instruments can be improved to better assess portions of the current NAEP frameworks, alternative methods are better suited to assessing broader conceptualizations of achievement that include complex skills. Academic achievement should be more broadly defined and measured using methods that are appropriately matched to the subjects, skills, and populations of interest. A multiple-methods design should be included in new paradigm NAEP to better assess: aspects of student achievement not well addressed by large-scale survey methods (e.g., scientific investigation, self-regulatory skills), noncore subject areas for which testing frequency generally prohibits the establishment of trend lines, subject areas (or portions of subject areas) in which not all students receive instruction (e.g., fine arts, advanced mathematics), the accomplishments of special populations (e.g., English-language learners, students with disabilities), and perhaps the educational experiences and postsecondary plans of high school seniors.

OCR for page 185
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress Furthermore, NAGB should explore and implement a new model for achievement-level setting. We made this set of arguments more concrete by providing a model for a coordinated system of education indicators. The model suggests the types and range of indicators that might be included in an integrated system. We provided it to illustrate, but not prescribe, an indicator model and repeat it here as groundwork for the discussion in this chapter. Figure 6-1 shows possible components of a coordinated system and depicts the measures of student achievement within the broader system. In the model we show new paradigm NAEP with its core NAEP and multiple-methods NAEP components. Our recommendations regarding the NAEP program have significant ramifications for the programs and plans of the National Center for Education Statistics and the National Assessment Governing Board. We organize our discussion of implementation strategies around four additional committee recommendations: First and foremost, the U.S. Department of Education and the National Center for Education Statistics should quickly initiate the activities necessary for reconfiguring NAEP and developing a coordinated system of education indicators. Responsibility for the design and implementation of the coordinated system should lie with NCES. NCES should develop the conceptual and structural framework for the system and should house, manage, and refine the data system. Within this structure and for new paradigm NAEP, the National Assessment Governing Board should have responsibility for determining what areas of student achievement are to be assessed and for setting expectations for student performance. NCES should determine how to measure the areas of achievement identified by NAGB and should develop and implement the assessments, collect and analyze the data, and report the results. The transitional contract for NAEP should be amended to address many of the recommendations put forward in Chapters 2 through 5. The more expansive changes we outline should build on these and be introduced with the generation of assessments for which new frameworks are developed. The U.S. Department of Education should quickly fund efforts to design, test, and evaluate new methods for assessing student achievement. In Chapter 1 we discussed strategies and activities to be pursued by the U.S. Department of Education and NCES in developing the coordinated system of education indicators. In the remainder of this chapter, we provide suggestions for activities that should be pursued to transform the NAEP program and create new paradigm NAEP to play a critical role in the coordinated system.

OCR for page 185
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress FIGURE 6-1 Proposed coordinated system of education indicators, including new paradigm NAEP. NOTE: TIMSS = Third International Mathematics and Science Study; NELS = National Education Longitudinal Study; ECLS = Early Childhood Longitudinal Study.

OCR for page 185
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress RECENT WORK THAT PROVIDES A FOUNDATION FOR NEW PARADIGM NAEP In August 1996 the National Assessment Governing Board adopted policies for redesigning national assessment that would "carry NAEP into the next millennium" (National Assessment Governing Board, 1996). Soon thereafter, the NAEP program began laying plans for implementing the adopted policies. Implementation planning was supported by suggestions from a series of planning grants, commissioned papers, stakeholder surveys, and public hearings. At its November 1997 meeting, NAGB passed a resolution that guides implementation of NAEP's redesign (National Assessment Governing Board, 1997). NAGB and NCES are planning a staged implementation of the redesign initiatives; they have funded a transitional procurement covering the NAEP administrations from 2000 through 2002 and a redesign procurement covering the NAEP administrations from 2003 through 2006, years during which new assessment frameworks will be introduced. The transitional contract implements as many of the redesign goals as is practicable and conducts research and development to prepare for full implementation. Additional work on redesign initiatives will be undertaken through NCES's NAEP secondary analysis program, validity studies program, and as described in this and other evaluation reports. Some of NAGB's redesign policies, if successfully implemented, will lay the groundwork for the assessment changes we describe; they are discussed in NAGB's implementation plan entitled, "Bridging Policy to Implementation: A Resolution" (National Assessment Governing Board, 1997). NAGB's priorities for future NAEP include: Combining the long-and short-term trend designs. In their policy statement, NAGB says that a plan should be developed in concert with the transitional contract to "ensure that the current cross-sectional assessment shall become the future long-term trend (replacing the current long-term trend) when new assessment frameworks are brought on-line." They ask for research and development to support a combined design. This priority parallels our recommendations for streamlining NAEP's design. Streamlining the national and state samples. Here, too, our recommendations are paralleled by NAGB's intentions; they ask for sampling plans for future assessments that are more efficient and less burdensome. State samples, they say, should be "developed to the largest extent possible by augmenting the national samples in states." Exploring market-basket reporting. NAGB has stated that secondary analysis grants or validity studies projects should be funded to examine market-basket reporting and cost options. We support these explorations to seek more easily understood metrics for describing NAEP's large-scale assessment results.

OCR for page 185
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress Issuing policy and practitioner reports. NAGB's resolution calls for issuance of simple, accurate, and jargon-free reports for practitioners and the public. We also contend that national and state policy makers should receive targeted, policy-relevant reports. Implementing multiple-methods assessments. NAGB's resolution makes some initial inroads toward the implementation of a multiple-methods program. For the year 2000, for example, they specify that hands-on science blocks should be administered to the minimum number of students needed for accurate estimation and comparability of trend data to the past. They also ask for research and development on possible simplified designs for NAEP data collections. They note that the NAEP assessment development contractor will conduct small-scale pilot studies to examine potential designs. While these initial explorations are positive steps in the direction of more closely matching assessment method with assessment purpose, our recommendations regarding multiple-methods NAEP extend well beyond these steps. Facilitating interpretive uses of NAEP data. The redesign implementation resolution states that NAEP users should have access to data in forms that support efforts to improve education. This priority is not inconsistent with the more expansive initiatives for serving the interpretive needs of educators and policy makers that we have recommended throughout this report. Again, Chapter 4 made suggestions for obtaining interpretive information from NAEP, and Chapter 1 discussed the coordinated system. The earlier mentioned planning grants, recently authored by the Educational Testing Service (Johnson et al., 1997), American College Testing (Bay et al., 1997), and the American Institutes for Research (1997), as well as work by NCES and NAGB, provide additional input on a number of issues addressed in this report. As described in Chapter 4, the ETS proposal, for example, discusses first steps for multiple-methods designs in some detail. ETS suggests modular assessment for NAEP, as had NAGB's Design/Feasibility Team in 1996 (Forsyth et al., 1996). ETS recommended partitioning the subject-area frameworks into aspects of achievement that can be efficiently assessed using large-scale surveys—consisting primarily of multiple-choice and short constructed-response items—and those more appropriately assessed with extended constructed-response tasks that rely on higher levels of engagement and development. The first set of achievements should be assessed, it stated, under the current NAEP model and the second with smaller groups of examinees, perhaps with more extensive collection of ancillary data on student background variables and on the learning practices of students (Johnson et al., 1997). The small-scale data collections would support deeper analysis and richer reporting. In their design document, ETS acknowledged that tasks in the second category may not contribute to the derivation of scaled scores, but their information value may lie "closest to the exercises themselves" (p. 4-29). ETS suggested that portions of the large-scale survey assessment

OCR for page 185
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress be given to students performing extended tasks, however, so that the relationships among assessment components could be evaluated. These proposals are reasonable first steps toward a multiple-methods NAEP. NCES efforts to expand inclusion criteria and offer a range of testing accommodations in 1996 lay the groundwork for continued effort in this area. As they currently do, NCES should continue to support and conduct research on the meaningful assessment of English-language learners and students with disabilities in NAEP, and it should implement promising procedures for enhancing inclusion, accommodation, meaningful assessment, and meaningful reports for special populations. In issuing their most recent request for proposal and in work with their advisers (the Technical Advisory Committee on Standard Setting), NAGB has stated its clear desire to consider and explore alternate models for achievement-level setting. The changes planned and being implemented for the upcoming civics and writing assessments adjust the current standard-setting model and procedures in relatively minor ways. NAGB is open, however, to a more fundamental recasting of the current process. To date, however, attractive and realistic alternatives have not been forthcoming. (Several shortcomings of the current model and suggestions for future standard-setting research were described in Chapter 5.) OPERATIONALIZING CHANGES TO THE NAEP PROGRAM The changes we recommend to the NAEP program are multifaceted and would need to be phased in over time. There is an opportunity for those responsible for NAEP to pursue strategies that enhance the likelihood of an effective transformation over the next two decades. As mentioned earlier in this chapter, the NAEP redesign plan involves a transitional contract covering assessment administration through the year 2002. During this period, tests based on existing frameworks will be administered. We see this as an excellent opportunity for effecting change and conducting some of the foundational research that should precede implementation of several of our recommendations. We recommend that NCES and NAGB make the changes described in this section in the context of activity funded under the transitional contract and in subsequent administrations of subjects under the current main NAEP and trend NAEP frameworks. Table 6-1 provides a summary of the NAEP administrations scheduled during the transitional period and as new frameworks are introduced. Some of the following changes may be possible with minor modification to awarded tasks and with later administrations under the current frameworks: Extended-response and other performance tasks should be administered to samples of students using a modular approach. NCES should further their efforts to represent all students in NAEP and

OCR for page 185
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress TABLE 6-1 Scheduled NAEP Contracts and Administrations by Framework, Subject, and Grade/Age   National NAEP     State NAEP       Trend NAEP NAEP Contract/Administration Dates Framework Subject Grade/Age Frame work Subject Grade/Age Frameworka Subject Grade/Age Transitional contract Current main Mathematics; Science Grades 4, 8, 12 Current main Mathematics; Science; Reading; Writing Grades 4, 8       Assessments administered in 2000-2002   U.S. History; Geography; Reading; Writing               Redesign contract Current main Civics; Science Grades 4, 8, 12 Current main Science; Writing Grades 4, 8 Current trend instruments Reading; Mathematics; Science Ages 9, 13, 17 Assessments administered in 2003-2006   Writing           Writing Grades 4, 8, 11   New main Mathematics; Reading Grades 4, 8, 12 New main Mathematics; Reading Grades 4, 8         New main Foreign Language; World History; Economics Grade 12             Future work New main Arts; Mathematics Grades 4, 8, 12 New main Mathematics; Science; Reading; Writing Grades 4, 8 Current trend instruments Reading; Mathematics; Science Ages 9, 13, 17 Assessments administered in 2007-2010   Science; U.S. History; Geography; Reading; Writing           Writing Grades 4, 8, 11 a Trend assessments are not based on formal framework but are collections of items administered in past assessments. SOURCE: NAEP program documents. meaningfully portray the performance of English-language learning students and students with disabilities. Improved development of items, tasks, and scoring rubrics should be undertaken within the current frameworks. These changes could be directed at better assessment of currently tested knowledge and skills and extend beyond these to more complex aspects of achievement described by the frameworks, but that are currently not adequately assessed. New materials could be included in replacement blocks for the current generation of assessments.

OCR for page 185
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress Improved coherence across the steps of development, administration, analysis, and reporting should be an immediate priority. The recently established subject-area standing committees provide a basis from which to begin the work of more coherent assessment and reporting. As they do with the National Council of Teachers of Mathematics, the NAEP program should work with disciplinary organizations to generate reports that provide in-depth analyses of student response data for individual items and groups of items. The NAEP program recently developed similar relationships

OCR for page 185
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress with various arts organizations for the fine arts assessment, and it should continue this practice to enhance the utility and visibility of NAEP products. In the resolution for redesign implementation, NAGB stated that field tests should be embedded as feasible in regular testings. We extend this idea to suggest that alternatives for streamlined sampling and administration designs for upcoming frameworks should be explored in conjunction with the 2000 and 2002 administrations. The feasibility, likely accuracy, and efficiency of alternate designs could at least partially be examined on a field-test basis in upcoming administrations. Similarly, alternate assessment formats should be field-tested in conjunction with operational testing in 2000 and 2002 for possible use with new frameworks. Furthermore, small targeted studies should be conducted to assemble some of the building blocks of multiple-methods NAEP. New constructs, item types, item families, measurement methods, sampling plans, scoring approaches, and reporting schemes could be tried in concert with the core program. Finally, alternate standard-setting models should be explored in 2000 and 2002 along with the current approach. This would allow NAGB and its contractors to explore the efficacy and outputs of possible future models. These changes are summarized in Table 6-2, which lists the modifications we propose under the transitional contract and in concert with NAEP administrations under existing frameworks. These changes would provide strong grounding for more expansive revision with subjects administered under new frameworks. More ambitious changes to the assessment program undoubtedly are better tied to the introduction of new frameworks in the year 2003 and beyond. Improvements that could be made under the redesign contract and in future work also are summarized in Table 6-2. These improvements include: Introduction of combined trend NAEP and main NAEP designs as discussed in Chapter 2 and slated for research development under the transitional procurement, Implementation of more efficient national and state sampling designs —again, as earlier described and under consideration in the next procurement, Design of a NAEP assessment system that includes large-scale survey methods and an array of smaller-scale, alternative measures, using, for example, video assessment, interviews, computer-based analyses of natural student work, and other measurement techniques, in ways that better align assessment method with assessment purpose, As earlier described, implementation of an expanded array of alternative assessments using multiple methods for noncore subject areas, special populations, and other aspects of achievement and populations not well addressed by large-scale survey methods,

OCR for page 185
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress TABLE 6-2 Proposed Introduction of Changes Associated with New Paradigm NAEP NAEP Contract/Administration Dates Framework   Proposed Introduction of Changes Transitional contract Current main Goals associated w/assessments based on current frameworks (and beyond) • Administration of extended-response and performance tasks to targeted samples Assessments administered in 2000-2002     • Continued efforts to meaningfully assess English-language learners and students with disabilities       • Improved development of items, tasks, and rubrics for replacement blocks       • Improved coherence across development, administration, analysis, reporting Redesign contract Current main   • Work with disciplinary organizations to report in-depth analyses of student response information Assessments administered in 2003-2006     • Exploration of alternatives for streamlined designs       • Research and field-testing of new item types/families, measurement methods, scoring approaches, and reporting schemes       • Exploration of new achievement-level-setting models   New main Goals associated w/assessments based on new frameworks • Introduction of combined main and trend designs       • Implementation of more efficient national and state sampling designs       • Development of assessments with multiple methods in which methods are better aligned to purposes       • Administration of alternative assessments for noncore subjects, special populations, and constructs not well measured by large-scale survey methods Future work New main   • Research and instrument development to support assessment of broader conceptions of achievement Assessments administered in 2007-2010     • Implementation of a new achievement-level-setting model

OCR for page 185
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress Research to support the assessment of broader conceptualizations of student achievement as described in Chapter 4, and Operational use of a new model for achievement-level setting. Distribution of Responsibility for New Paradigm NAEP The new paradigm NAEP we argue for in this report has several components that differ in significant ways from the existing NAEP program. In our judgment, it is important to make clear how responsibility for the components of an operational new paradigm NAEP should be distributed. The following is our recommended partitioning of responsibility across the varied facets of the program: Development of frameworks for the assessment of student achievement . As it has historically and in keeping with their authority to set policy for NAEP (P.L. 103-382, Section 412), NAGB should specify the disciplines to be assessed and develop NAEP's frameworks. NAGB should stop short of specifying methodology and technical designs for the assessments. Specification of measurement methods and development of measures for the achievement variables. This should be accomplished by NCES and its advisers and contractors. Data collection and database development. Again, NCES should accomplish this work with its contractors. Determination of performance standards for the survey-based achievement measures. NAGB and its advisers should continue in their evaluative role. Initial analysis and reporting. NCES should analyze and report data with help from advisers and contractors. The according of responsibilities for new paradigm NAEP described above entails some changes in the current legislatively authorized responsibilities of NAGB and NCES. In the current legislative authorization for NAEP, NCES, and NAGB (Improving America's Schools Act of 1994, P.L. 103-382), one of the duties of NAGB is to ''design the methodology of the [NAEP] assessment. …'' Because the implementation of new paradigm NAEP is an extraordinary design challenge—both technically and operationally —we recommend that responsibility for the design of assessment methodologies in new paradigm NAEP should rest with the body with demonstrated technical and operational expertise—NCES. CONCLUSION The U.S. Department of Education, NCES, and NAGB have an important opportunity to take the steps that are appropriate to NAEP's continued evolution and its rich history as the preeminent program for assessing academic achievement

OCR for page 185
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress in America. It is our hope that the activities we advocate, within the time frame proposed, will be seriously and vigorously pursued. NAEP cannot remain stagnant if it is to fulfill its mission of informing its constituents about important aspects of academic achievement. It is also clear that NAEP cannot fill the role of providing all information necessary and appropriate to portraying and pursuing improvements in education in America. Awareness of the complex relationship between student characteristics, teaching, learning, and achievement should mark public debate about the progress of American education. The system described in this report, of which NAEP is an integral component, would (1) support better understanding of student achievement data, (2) expand the measurement of academic achievement and other educational outcomes to include nontest-based indicators, and (3) serve policy planning by raising awareness of the complexity of the system and providing a basis for hypothesis generation about educational success and its school, demographic, and family correlates. The operational implications of the recommendations we make in this report are substantial. The development and implementation challenges of an improved NAEP within a coordinated system of education indicators will be great. It is our contention, however, that substantial efforts to these ends will result in the provision of important and useful descriptive, evaluative, and interpretive information about American academic achievement and educational progress.