Index

A

Accommodations/adaptations, 51, 88-89, 91-112 (passim), 191

Americans with Disabilities Act, 88

Goals 2000: Educate America Act, 10, 88, 96

individualized education plans, 88, 94, 99-100

Individuals with Disabilities Education Act, 88

items read orally to poor readers, 13, 95, 99, 105

test administration procedures, 91, 92, 99-100

Accountability, 2, 10, 19, 23, 27, students with disabilities/English-language learners, 88, 91, 94, 107, 109, 110-111, 112

Achievement-level descriptions, 114, 128, 134-139, 141, 152-154, 163-173 (passim)

preliminary, 119, 134-135, 165, 170, 176-179, 182, 183, 256, 257, 259, 260

Achievement levels, 2, 7, 9, 10, 17, 20, 25, 72, 73, 162-184, cost factors, 8

definitional issues, 54, 124, 137-138, 139, 169-170, 186

frameworks and assessment development process, 115, 116, 117, 121 , 124-125, 128, 134, 135-140, 143-147 (passim), 152-154, 158, 159, 160, 192, 194, 195, 196

reporting, 2, 6, 7, 25-31, 32, 33, 72, 73, 135, 162, 163-165

special studies, 16, 57, 110

use of NAEP results, 2, 31, 171, 176, 180, 182, 184, 246

see also Mathematics achievement;

Performance standards;

Reading achievement;

Science achievement;

Writing achievement

Achievement-level setting, 2, 7, 31, 48, 116, 134-135, 152, 162, 163-168, 171, 176, 179-184, 186, 246, 256-261

anchoring, 2, 170-171, 173, 177, 257-261

Angoff procedure, 165, 166, 168

NCES role, 167, 175-176, 187, 196

see also Cutscores

Administration procedures, testing, 12-13, 65

costs, 78

grade-vs age-level testing, 13, 14

individual testing, 12-13, 99

item response time, 13, 62, 99, 128, 130-131

items read orally due to poor readers, 13, 95, 99, 105

professional administrators, 13, 82, 86

state vs national NAEP, 82, 85



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 267
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress Index A Accommodations/adaptations, 51, 88-89, 91-112 (passim), 191 Americans with Disabilities Act, 88 Goals 2000: Educate America Act, 10, 88, 96 individualized education plans, 88, 94, 99-100 Individuals with Disabilities Education Act, 88 items read orally to poor readers, 13, 95, 99, 105 test administration procedures, 91, 92, 99-100 Accountability, 2, 10, 19, 23, 27, students with disabilities/English-language learners, 88, 91, 94, 107, 109, 110-111, 112 Achievement-level descriptions, 114, 128, 134-139, 141, 152-154, 163-173 (passim) preliminary, 119, 134-135, 165, 170, 176-179, 182, 183, 256, 257, 259, 260 Achievement levels, 2, 7, 9, 10, 17, 20, 25, 72, 73, 162-184, cost factors, 8 definitional issues, 54, 124, 137-138, 139, 169-170, 186 frameworks and assessment development process, 115, 116, 117, 121 , 124-125, 128, 134, 135-140, 143-147 (passim), 152-154, 158, 159, 160, 192, 194, 195, 196 reporting, 2, 6, 7, 25-31, 32, 33, 72, 73, 135, 162, 163-165 special studies, 16, 57, 110 use of NAEP results, 2, 31, 171, 176, 180, 182, 184, 246 see also Mathematics achievement; Performance standards; Reading achievement; Science achievement; Writing achievement Achievement-level setting, 2, 7, 31, 48, 116, 134-135, 152, 162, 163-168, 171, 176, 179-184, 186, 246, 256-261 anchoring, 2, 170-171, 173, 177, 257-261 Angoff procedure, 165, 166, 168 NCES role, 167, 175-176, 187, 196 see also Cutscores Administration procedures, testing, 12-13, 65 costs, 78 grade-vs age-level testing, 13, 14 individual testing, 12-13, 99 item response time, 13, 62, 99, 128, 130-131 items read orally due to poor readers, 13, 95, 99, 105 professional administrators, 13, 82, 86 state vs national NAEP, 82, 85

OCR for page 267
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress students with disabilities/English-language learners, 91, 92, 99-100 see also Group testing Administration schedules, testing, 18, 121, 191, 192-193, 195 Administrative factors, see Governance factors; School-level measures Advanced achievement levels, 2, 7, 64, 147, 152, 163, 164-171, 173 , 174, 175, 180, 183, 186, 256, 259, 260 Advanced placement, 165, 167, 169, 174 Advisory Council on Education Statistics, 53 African Americans, 44, 62 Age factors, 12, 13, 79 grade-vs age-level testing, 13, 14, 192-193 see also Grade-level factors Alexander-James Panel, 2, 15 American College Testing, 19, 168-169, 190 American Institutes for Research, 190 Americans with Disabilities Act, 88 Analysis methods, 20, 189, 193, 196 coordinated system of education indicators, 45, 49-51 (passim), 55-56 costs, 78 current NAEP, 35, 58, 62, 63-64, 65, 67, 68-70, 71, 78 frameworks and assessment development process, 116, 117 reporting metrics, 72 secondary data, 42 streamlining NAEP design, 4, 14, 62, 63-64, 65, 67, 68-70, 71, 78 , 79 students with disabilities/English-language learners, 101, 104, 108, 110 text analysis, 43, 49, 219-220 video analyses, 43, 46, 48, 194 see also Computer-based testing and analysis; Content coverage and analysis; Item analysis and development Anchoring, 2, 170-171, 173, 177, 257-261 Angoff procedure, 165, 166, 168 Applied Measurement in Education, 11 Arts achievement, 193-194 Assessment development process, 2, 4, 5-6, 33, 67, 92, 114-161, 190-191, 194, 195, 220-236 achievement levels, 115, 116, 117, 121, 124-125, 128, 134, 135-140 , 145-147, 152-154, 158, 159, 160, 192, 194, 195, 196; see also ''mathematics achievement'' and "science achievement" infra constructed-response items, 115, 122, 123, 128-129, 132, 146, 149 , 150, 153-154 contractors, 115, 116, 118, 119, 136, 148-150, 190, 192-193, 195 defined, 115 funding, 149-150, 151, 155, 191 governance, 117-118, 134-135 interpretive information, 114, 116, 137, 153, 154-157, 159, 160-161 item analysis and development, 117, 118-119, 122-123, 125, 128-137 , 142, 149-155, 160, 219-236, 256-257 knowledge and skills measures, 115, 121, 125, 126, 127, 128, 131-133, 138, 139, 140, 156, 159, 233, 256-257 large-scale assessments, 114, 125, 128, 133, 136, 147-149, 156, 158, 159, 161, 194 main NAEP, 114, 118-121, 126-128, 136, 194, 195, 259 mathematics achievement, 118, 121, 122-124, 126-127, 132, 136, 142 NCES role, 115, 116, 118, 122, 134, 148-149, 154 performance standards, 114, 115, 118, 123, 125, 133, 142-143, 146 , 149, 155, 159, 220, 257, 259-260 reading achievement, 118, 119, 120, 122, 123, 127-128, 132, 133, 136, 141-142, 219-230 reporting procedures, 114, 117, 121, 128, 135, 145-146, 151, 154-160 (passim), 221 science achievement, 116, 118, 126, 129-130, 132-133, 134, 136, 30, 132-133, 134, 136, 142, 143-144, 231-236 scoring, 114, 117, 118, 119, 125, 132, 133-134, 135, 136, 146, 149 , 150, 153-154, 156, 160, 161, 219-220, 226-227, 234-235, 257 summary scores, 116, 117, 121, 128, 139, 152, 154 task measures, 119, 128, 132, 136, 139, 142-144, 146, 149, 153-154 , 160, 219-220 see also Field tests; Item analysis and development Assessment in Transition: Monitoring the Nation's Educational Progress, 25, 124 Association of State Assessment Programs, 97

OCR for page 267
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress B Background variables, 34 instructional, 65 students, 35, 40, 42, 64, 95, 119, 122, 149, 188, 190, 220 teachers, 40, 41, 43 Balanced incomplete blocks, 62, 63, 66 Barometer information, see Descriptive information Bias, see Error of measurement Bilingual education, 91, 98-101 (passim), 104, 105 translated versions of NAEP, 95, 97, 100, 101, 111 Black persons, see African AmericansC Center for Equity and Excellence, 98 Civics, 18, 61, 136 Class size, 26, 44 Cognitive factors and theory, 6, 7, 19-20, 128, 133, 137-141, 143-146, 166 see also Knowledge and skills measures; Learning theory; Problem solving Common Core of Data, 36-37, 51 Computer-based testing and analysis, 32, 63, 148, 194 The Condition of Education, 35 Constructed-response items, 80, 136, 166, 174 frameworks and assessment development process, 115, 122, 123, 128-129, 132, 146, 149, 150, 153-154 main vs trend NAEP, 79 national NAEP, 80 nonresponse rates, 123 Content coverage and analysis, 12, 27, 48-49, 70, 79, 118 text analysis, 43, 49, 219-220 Contextual data, see Interpretive information Contracts and contractors, 17, 31, 68, 82, 116, 118, 136, 150, 166 , 168, 177, 191, 195, 196 administration of testing, state vs national, 82 Educational Testing Service, 14, 17, 31, 93, 118, 119, 148-149, 150, 190 frameworks and assessment development process, 115, 116, 118, 119 , 136, 148-150, 190, 192-193, 195 KPMG Peat Marwick LLP, 25, 66, 69, 70, 78 NAEP, redesign/transitional, 148, 175, 187, 189-196 (passim) National Computer Systems, 119 RAND, 40, 46 scheduled contracts and administration by framework, 192-193, 195 WESTAT, 17, 119 Coordinated system of education indicators, 4, 6, 8, 20, 22-55, 73, 158, 186, 187, 188 analysis methods, 45, 49-51 (passim), 55-56 costs of, 51-52, 53 curriculum, 23, 40, 41, 43, 54 data collection, 4, 6, 8, 20, 22-55, 106 Department of Education role, 23-24, 34, 50, 53-54 frameworks, 24, 33, 38, 40, 47, 51-55 (passim) funding, 23, 41, 42, 44, 53 governance, 45, 51-54 NAGB, 22-25, 31, 187 international components, 42, 43, 46, 158, 188 interpretive information, 4, 23, 24, 25, 27-55, 45, 115 knowledge and skills measures, 23-24, 47 large-scale assessments, 24, 34, 47, 51, 54, 188 NCES, 31, 34-38, 40-41, 45-46, 49-54, 187 new paradigm NAEP, 46-47, 48, 55, 84, 85, 109, 110, 146-147, 151, 158, 187-197 pedagogy, 23, 40-43 (passim), 46, 47-49, 54 reporting procedures, 25-26, 73 sampling, 45, 46, 47, 49, 50-51, 107 skills measures, 24, 47 standards, general, 23, 45 state-level factors, 45-46 students with disabilities/English-language learners, 106, 107 task measures, 23, 24, 33, 47, 54 validity issues, 40, 46 Core NAEP, 7, 47, 57, 78, 80, 82, 84, 85, 147-149, 153, 155, 158, 181-182, 187, 188, 194 students with disabilities/English-language learners, 109, 110, 112 see also Mathematics achievement; Reading achievement; Science achievement; Writing achievement Cost and cost-benefit factors, 8, 10, 78 elementary/secondary school expenditures, data on, 35

OCR for page 267
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress indicator systems, 38, 51-52, 53 NAEP design, general, 3, 8, 62, 78, 84, 149-150, 151 national/state systems separate, 66, 78, 82, 85 sampling designs, 62, 66 state-level indicators, 62, 66, 78, 82, 85 students with disabilities/English-language learners, 8, 107-108, 111 trend NAEP, 78 see also Accountability; Funding Council of Chief State School Officers, 17, 40, 46, 53, 96, 98 Council of the Great City Schools, 53 CSEI, see Coordinated system of education indicators Cultural bias, 63 Curriculum, 6 cognitive theory and, 19-20 coordinated system of education indicators, 23, 40, 41, 43, 54 high school, 80-81 NAEP design and, 12, 16, 20, 58, 79, 80-81 special studies, 16, 57 standards, 10, 179, 153 students with disabilities/English-language learners, 111 Curriculum and Evaluation Standards for School Mathematics, 10, 122 Cutscores, 64, 166, 167-171, 173-177 (passim), 180, 182, 183, 256 , 260D Databases, 50-52, 196 Data collection, general, 4, 11, 14, 20, 23, 196, 259 coordinated system of education indicators, 4, 6, 8, 20, 22-55, 106 frameworks, 115, 150 mixed methods of, 43, 45, 47-49, 54 NAGB objectives, 25 national vs state, 65 streamlining NAEP design, 4, 7, 65, 68, 73, 76, 79-82, 84, 190, 196 students with disabilities/English-language learners, 87, 96, 104 , 106, 107, 108, 112, 113 see also Large-scale assessments; Sampling Data presentation, see Reporting procedures Demographic factors, 1, 152, 19 performance measures, 27, 163 reporting, 1-2, 27, 31, 34, 163 socioeconomic status, 14, 50 poverty, 14, 44 social indicator systems, 3, 24, 38-39, 152, 153 summary scores, 14 students with disabilities/English-language learners, 89-90 see also Age factors; Gender factors; Race/ethnicity; Urban/rural factors Department of Education, 1, 3, 187 coordinated system of education indicators, 23-24, 34, 50, 53-54 English-language learners, 105, 106 students with disabilities, 106, 109 see also Contracts and contractors; National Center for Education Statistics; terms beginning "Office of…" Department of Education Organization Act, 88 Descriptive information, 3, 8, 27-32, 33, 57, 72, 114, 128, 164, 197 cost factors, 8 students with disabilities, 101 Design/Feasibility Team, 66, 72, 78-79, 190 Difficulty level, see Item difficulty Dimensionality of data, 63, 70 Disabilities, see Learning disabilities; Students with disabilities Dropouts, 57, 81, 86E Early Childhood Longitudinal Study, 36-37, 46, 110, 188 Economic factors, see Cost and cost-benefit factors; Funding; Poverty; Socioeconomic status Educating One and All: Students with Disabilities and Standards-Based Reform, 91, 94 Educational indicators, 4, 9, 10, 19, 154 summary scores, 128 see also Coordinated system of educational indicators Educational Longitudinal Study, 36-37 Educational Testing Service, 14, 17, 31, 93, 118, 119, 148-149, 150, 190 Education Commission of the States, 14

OCR for page 267
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress Education Counts, 38 Education reform, 2, 9-10, 19, 20, 25-27 (passim), 118, 153 coordinated system of education indicators, 41 see also Accountability Education Week, 172 Elementary and Secondary Education Act, 14 English-language learners, 7-8, 14, 19, 20, 26, 87-113, 134, 186, 191 accommodations/adaptations, 51, 88-89, 91-112 (passim), 191 analysis methods, 101, 104, 108, 110 bilingual education, 91, 98-101 (passim), 104, 105 translated versions of NAEP, 95, 97, 100, 101, 111 coordinated system of education indicators, 106, 107 core NAEP, 109, 110, 112 cost factors, 8, 107-108, 111 data collection, 87, 96, 104, 106, 107, 108, 112, 113 defined, 87(n. 1) demography of, 89-90 educational indicators, 88, 106, 114 field tests, 98-101 funding, 108, 109 Goals 2000: Educate America Act, 10, 88, 96 grade-level factors, 90 historical perspectives, 5, 92-98 identification/classification, 102, 104-105, 106, 113 inclusion, 45, 92, 93, 95, 97-104, 106, 109, 111-113, 191 interpretive information, 5, 87, 99, 106, 112 large-scale assessments, 88, 89, 91-93, 96, 97, 98, 104, 105, 106-109, 110, 112-113 legislation, 10, 88, 96, 106, 108 mathematics achievement, 98-101, 103, 110 NCES role, 91, 93, 94, 97, 105, 106, 109, 191-192 numbers by grade level, 90 participation, 5, 19, 57 performance standards, 87, 89, 94 reading achievement, 94, 110 regional factors, 89-90 reporting procedures, 108, 111 sampling, 101, 102, 103, 106, 107 school-level measures, 89-90 skills measures, 104-105 standards, general, 5, 87, 89, 91, 92, 94, 98-102, 109, 111-113 state role, 88, 93, 94, 97, 98, 104, 106, 109, 110 summary scores, 106 test administration procedures, 91, 92, 99-100 TIMSS, 44 translated versions of NAEP, 95, 97, 100, 101, 111 urban/rural factors, 89-90 validity issues, 88, 91, 95, 106, 109-111 see also Hispanic students; Participation, students with disabilities/English-language learners Equality of Opportunity Surveys, 50 Error of measurement, 63, 69, 82-83, 86, 118, 175 probability sampling, 62, 64, 66, 68-69, 175 Estimation procedures, 58, 76, 80, 86, 175 Evaluative information, 3, 4, 7, 8, 11, 27-33 (passim), 57 teacher evaluation of NAEP, 77-78, 129-130 see also Achievement levels; Performance standards Extended-response items, 128, 132, 136, 140, 149, 150, 155, 190-191 , 194, 220, 225, 226, 229, 230F Family factors, 27, 34, 41, 42, 44 Federal government national curriculum, 13-14 standards, general, 10 see also Department of Education; Funding; Legislation Field tests, 51, 117, 118, 119, 133-134, 136, 150, 194 reporting procedures, tables/displays, 71 Fit of items, 63, 69, 70, 101, 118, 122-123, 125, 128-132, 220, 221, 229 Focused Reports, 65 Focus on NAEP, 65 Foreign countries, see International perspectives; specific countries Foreign-language speakers, see English-language learners

OCR for page 267
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress Frameworks, 2, 4, 5-6, 17, 114-161, 165, 194, 257 achievement levels, 115, 116, 117, 121, 124-125, 128, 134, 135-140 , 145-147, 152-154, 158, 159, 160, 192, 194, 195, 196; see also other "mathematics achievement" and "science achievement" infra analysis methods, 116, 117 constructed-response items, 115, 122, 123, 128-129, 132, 146, 149 , 150, 153-154 contractors, 115, 116, 118, 119, 136, 148-150, 190, 192-193, 195 coordinated system of education indicators, 24, 33, 38, 40, 47, 51-55 (passim) data collection, 115, 150 design streamlining, 57, 58-61, 62, 65-66, 77 funding, 149-150, 151, 155, 191 governance, 117-118, 134-135 NAGB, 116, 118, 127, 194, 196 interpretive information, 114, 116, 137, 153, 154-157, 159, 160-161 item analysis and development, 117, 118-119, 122-123, 125, 128-137 , 142, 149-155, 160, 219-236, 256-257 knowledge and skills measures, 115, 121, 125, 126, 127, 128, 131-133, 138, 139, 140, 156, 159, 233, 256-257 large-scale assessments, 114, 125, 128, 133, 136, 147-149, 156, 158, 159, 161, 194 main NAEP, 114, 118-121, 126-128, 136, 194, 195, 259 mathematics achievement, 118, 121, 122-124, 126-127, 132, 136, 142 national NAEP, 59-60 NCES role, 115, 116, 118, 122, 134, 148-149, 154 performance standards, 114, 115, 118, 123, 125, 133, 142-143, 146 , 149, 155, 159, 220, 257, 259-260 reading achievement, 118, 119, 120, 122, 123, 127-128, 132, 133, 136, 141-142, 219-230 reporting procedures, 114, 117, 121, 128, 135, 145-146, 151, 154-160 (passim), 221 scheduled contracts and administration by framework, 192-193, 195 science achievement, 116, 118, 126, 129-130, 132-133, 134, 136, 142, 143-144, 231-236 scoring, 114, 117, 118, 119, 125, 132, 133-134, 135, 136, 146, 149 ,150, 153-154, 156, 160, 161, 219-220, 226-227, 234-235, 257 summary scores, 116, 117, 121, 128, 139, 152, 154 task measures, 119, 128, 132, 136, 139, 142-144, 146, 149, 153-154 ,160, 219-220 trend NAEP, 77-78, 192, 194 From Data to Information, 40, 41, 49-50 Funding, 19 coordinated system of education indicators, 23, 41, 42, 44, 53 framework and assessment development process, 149-150, 151, 155, 191 students with disabilities/English-language learners, 108, 109 see also Accountability; Contracts and contractors; Cost and cost-benefit factorsG Gender factors, 2, 68, 152, 169 Geography achievement, 18, 61, 136, 166 Goals 2000: Educate America Act, 10, 88, 96 Governance factors, 17-19 coordinated system of education indicators, 45, 51-54 frameworks and assessment development process, 117-118, 134-135 reporting, 19, 31, 72, 164, 172, 190 see also National Assessment Governing Board Government role, see Federal government; Local government; State government Grade-level factors, 13, 14, 15, 31, 172-173, 192-193 see also Age factors Graduate Record Exam, 93 Group testing, 12-13 group tasks, 47, 138H Hawkins-Stafford Elementary and Secondary School Improvement Amendment , 35 High School and Beyond, 31, 46 High school students, 44, 80-81, 85, 86, 186 dropouts, 57, 81, 86

OCR for page 267
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress Hispanic students, 44, 62, 95, 99 bilingual education, 91, 98-101 (passim), 105, 110 translated versions of NAEP, 95, 97, 100, 101, 111 Historical perspectives, 1-2, 9-15, 19, 163 achievement measures, 4 contextual data, 24 indicator systems, 38 sociopolitical context, 9-11 students with disabilities/English language learners, 5, 92-98 History, as academic subject, 18, 32, 52, 61, 136, 166I Improving America's Schools Act, 2, 10, 11, 17, 24, 88, 122, 162, 175, 196 Improving Schooling for Language-Minority Children: A Research Agenda , 91 Inclusion, 45, 92, 93, 95, 97-104, 106, 109, 111-113, 191 The Inclusion of Students with Disabilities and Limited Proficient Students in Large-Scale Assessments: A Summary of Recent Progress , 92 Indicator systems costs of, 38, 51-52, 53 general, 38-39 purposes saved by 1, 3-4, 9, 10, 27-32, 34, 35-39, 42-45, 186; see also Accountability social, 3, 24, 38-39, 152, 153 see also Coordinated system of educational indicators Individualized education plans, 88, 94, 99-100 Individual-level measures, see Student-level measures Individuals with Disabilities Education Act, 88 Individual testing, 12-13 students with disabilities/English language learners, 99 Institute on Educational Governance, Finance, Policy Making, and Management, 53 Institute on Student Achievement, Curriculum, and Assessment, 52 Instruction, see Pedagogy Instructional Reports, 65 Integrating Federal Statistics on Children, 40 International perspectives, 2, 15, 85, 110 coordinated system of education indicators, 42, 43, 46, 158, 188 sociopolitical trends, 9-10 TIMSS, 36-37, 40, 42-44, 46, 48, 110, 169, 174, 188, 259 Interpretive information, 6, 9, 19 cost factors, 8 coordinated system of education indicators, 4, 23, 24, 25, 27-55, 45, 115 frameworks and assessment development processes, 114, 116, 137, 153, 154-157, 159, 160-161 mass media, 9-10, 12, 25-30, 68, 71, 171 streamlining NAEP design, 4, 5, 57, 64, 65, 67-68, 71, 72, 73, 81 , 86, 190 students with disabilities/English-language learners, 5, 87, 99, 106, 112 see also Descriptive information; Evaluative information IRT, see Item response theory Item analysis and development, 4, 7, 14, 62-64, 69, 178-179, 192 anchoring, 2, 170-171, 173, 177, 257-261 fit of items, 63, 69, 70, 101, 118, 122-123, 125, 128-132, 220, 221, 229 frameworks and assessment development process, 117, 118-119, 122-123, 125, 128-137, 142, 149-155, 160, 219-236, 256-257 matrix sampling, 13, 62, 68 balanced incomplete blocks, 62, 63, 66 see also Field tests Item authoring, 256-257 Item difficulty, 13, 14, 100-101, 110, 119, 139, 166, 174-175, 179 , 238, 243, 245, 257 Item families, 146, 156-157, 160, 193, 237-255 Item mapping, 51, 169, 233, 257-259, 260-261 Item response theory, 62-64, 69, 72, 95, 101J Jackknife methods, 64, 69K Knowledge and skills measures, 5, 6, 12, 13, 19-20, 27, 77 coordinated system of education indicators, 23-24, 47 English-language learners, 104-105

OCR for page 267
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress frameworks and assessment development process, 115, 121, 125, 126 , 127, 128, 131-133, 138, 139, 140, 159, 233, 256-257 main NAEP, 4-5, 15-16, 77, 78-80 main vs trend NAEP, 77 see also Task measures KPMG Peat Marwick LLP, 25, 66, 69, 70, 78L Language factors, see Bilingual education; English-language learners Large-scale assessments, 3-10 (passim), 21 achievement-level setting, 182 coordinated system of education indicators, 24, 34, 47, 51, 54, 188 frameworks and assessment development processes, 114, 125, 128, 133, 136, 147-149, 156, 158, 159, 161, 194 streamlining NAEP design, 56, 57, 70, 80, 84 students with disabilities/English-language learners, 88, 89, 91-93, 96, 97, 98, 104, 105, 106-109, 110, 112-113 Learning disabilities, 88, 95, 104, 108, 110 Office of Special Education and Rehabilitative Services, 91 Learning theory, 5-6, 19-20, 128, 137, 138-141, 151, 160 LEAs, see School districts Legislation, 10 Americans with Disabilities Act, 88 Department of Education Organization Act, 88 Elementary and Secondary Education Act, 14 Goals 2000: Educate America Act, 10, 88, 96 Hawkins-Stafford Elementary and Secondary School Improvement Amendment , 35 Improving America's Schools Act, 2, 10, 11, 17, 24, 88, 122, 162, 175, 196 Individuals with Disabilities Education Act, 88 National Education Statistics Act, 52 Perkins Act, 88 students with disabilities/English-language learners, 10, 88, 96, 106, 108 Literacy, see English-language learners; Reading achievement; Writing achievement Local Education Authorities, see School districts Local factors and trends, 4 age-vs grade-level testing, 14 streamlining testing, 81-82 see also School-level measures Local government national curriculum, 13 standards, 56 Longitudinal studies, 31, 34, 36-37, 39, 42, 43, 46, 85, 110, 158 see also specific studiesM Main NAEP frameworks and assessment development process, 114, 118-121, 126-128, 136, 194, 195, 259 knowledge measures, 4-5, 15-16, 77, 78-80 mathematics achievement, 15, 17, 28-30, 65, 77-78, 80, 84, 118, 119, 125, 126-127, 132, 156, 157, 171, 192-193 national trends, 15-16, 56, 57, 58, 65 performance standards, 4-5, 15-16, 77, 78-80 reading achievement, 15, 17, 65, 80, 84, 118, 120, 125, 127-128, 133, 171, 192-193, 219 reporting, 163, 171 science achievement, 15, 17, 28-30, 65, 77-78, 80, 84, 116, 118, 121, 125, 126, 157, 168, 192-193, 259 state, 15, 16, 56, 57, 58, 65-66 students with disabilities/English-language learners, 103 trend vs, 4-5, 15-16, 58, 63, 64, 65, 73-80, 84, 85 writing achievement, 15, 17, 65, 80, 84, 192-193 Market-basket reporting, 72, 189 Mass media, 9-10, 12, 25-30, 68, 71, 171 Mathematics achievement, 7, 10, 27, 52, 58, 60, 65 English-language learners, 98-101, 103, 110 frameworks and assessment development process, 118, 121, 122-124, 126-127, 132, 136, 142

OCR for page 267
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress item families, 237-255 main NAEP, 15, 17, 28-30, 65, 77-78, 80, 84, 118, 119, 125, 126-127, 132, 156, 157, 171, 192-193 NAEP administration schedule, 18 National Council of Teachers of Mathematics, 122, 155, 193 national NAEP, 18, 60, 63, 73, 75, 77 new paradigm NAEP, 84 reporting, 28, 58, 59, 73, 171, 172 standards, 10, 166, 171, 172, 176 state NAEP, 82 students with disabilities, 103 TIMSS, 36-37, 40, 42-44, 46, 48, 110, 169, 174, 188, 259 trend NAEP, 15, 17, 18, 60, 63, 73, 75, 77, 80, 84, 102 trend vs main NAEP, 73, 75-80 (passim) see also Achievement levels; Core NAEP Matrix sampling, 13, 62, 68 balanced incomplete blocks, 62, 63, 66 Minority groups, see Race/ethnicity Mixed methods of data collection, 43, 45, 47-49, 54 Multiple assessment strategies, 4, 6, 7, 47, 57, 66, 73-80 Multiple-choice items, 12, 118, 128-129, 136, 146, 150, 174, 229, 230 guessing/nonresponse, 13, 64 item response theory, 63 main vs trend NAEP, 79 Multiple-methods NAEP, 6, 7, 8, 10, 46, 47, 66, 80, 81, 84, 85, 109, 110, 124, 147-156 (passim), 158, 181-182, 186, 188, 190, 193, 194N NAEP Facts, 65 NAEP Redesigned, 148-149 NAEP Report Cards, see Report Cards National Academy of Education, 7, 25-26, 66, 67, 122, 137 performance standards, 167, 168, 174 students with disabilities/English-language learners, 92, 94, 96, 98 National Assessment Governing Board, 1, 2, 3, 11, 17, 65-66, 185, 189 achievement-level setting, 165-166, 168-171, 173-174, 175, 180, 182, 183, 187, 191, 196, 260 coordinated system of education indicators, 22-25, 31, 187 Design/Feasibility Team, 66, 72, 78-79, 190 frameworks and assessment development process, 116, 118, 127, 194, 196 NAEP objectives, general, 24-25, 78-79, 162, 189-190 reporting, 31, 72, 164, 172, 190 state government role, 25, 65-66 National Assessment of Educational Progress, see Core NAEP; Main NAEP; Multiple-methods NAEP; National NAEP; New paradigm NAEP; State NAEP; Trend NAEP National Association of State Test Directors, 53 National Center for Education Statistics, 1-3, 17, 185, 189 achievement-level setting, 167, 175-176, 187, 196 coordinated system of education indicators, 31, 34-38, 40-41, 45-46, 49-54, 187 data collection, general, 36-37, 196 frameworks and assessment development process, 115, 116, 118, 122, 134, 148-149, 154 reporting procedures, 31, 154, 196 students with disabilities/English-language learners, 91, 93, 94, 97, 105, 106, 109, 191-192 see also Contracts and contractors National Center for History in Schools, 52 National Center for Improving Student Learning and Achievement in Mathematics and Science, 52 National Center for Research on Evaluation, Standards, and Student Testing, 52, 97, 98 National Center for Research on the Organization and Restructuring of Schools, 53 National Center for the Improvement of Early Reading Achievement, 52 National Center for the Study of Teaching and Policy, 53 National Center on Education Outcomes, 92 National Center on Increasing the Effectiveness of State and Local Education Reform Efforts, 53 National Computer Systems, 119 National Council of Teachers of Mathematics, 122, 155, 193

OCR for page 267
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress National Education Longitudinal Study, 36-37, 44, 46, 50, 51, 110 , 188 National Education Statistics Act, 52 National Governors' Association, 15, 172 National Household Education Survey, 35-36 National Longitudinal Survey of Youth, 44 National NAEP, 15-16, 56, 57, 58-60, 63, 65, 73 administration of testing, 82 constructed-response items, 80 costs, 66, 78, 82, 85 frameworks, 59-60 main NAEP, 15-16, 56, 57, 58, 65 mathematics achievement, 18, 60, 63, 73, 75, 77 multiple assessments, 73-77 pedagogy, 16, 58 reading achievement, 16, 18, 59, 73-75, 77 reporting, 1-2, 14, 71 sampling, 59-60, 62, 66 science achievement, 18, 60, 63 state vs, 65, 81-84, 85 writing achievement, 16, 18, 59 National Research and Development Center on English Learning and Achievement, 52 National Science Education Standards, 10 National trends, 5, 19, 46-46 coordinated system of education indicators, 45-46 main NAEP, 15-16, 56, 57, 58, 65 NAEP results reporting, 1-2, 14 NAGB objectives, 24 sampling, 59-63, 68, 194 standards and goals, 10 streamlining, 81-84, 189 students with disabilities/English-language learners, 110 see also National NAEP A Nation at Risk, 1, 9, 14 The Nation's Report Card, 15, 154, 172 NCES, see National Center for Education Statistics New paradigm NAEP, 46-47, 48, 55, 84, 85, 109, 110, 146-147, 151, 158, 187, 189, 194-196 Newspapers, see Mass media Non-native English speakers, see English-language learners Nonresponse rates, 64, 80O Office of Bilingual Education and Minority Languages, 91 Office of Civil Rights, 91 Office of Education Research and Improvement, 46, 91, 94 Office of Special Education and Rehabilitative Services, 91 Office of the General Counsel, 91 Open-ended items, 22, 146 see also Extended-response itemsP Participation, students with disabilities/English-language learners , 5, 19, 57, 87-89, 91-113 (passim) accommodations/adaptations, 51, 88-89, 91-112 (passim), 191 Goals 2000: Educate America Act, 10, 88, 96 Pedagogy coordinated system of education indicators, 23, 40-43 (passim), 46, 47-49, 54 national NAEP, 16, 58 teacher background, 40, 41, 43 trend NAEP, 77 video techniques, instructional methods analysis, 43, 46, 48, 194 Performance standards, 3, 17, 19, 20, 23, 44, 48, 162-184, 196 advanced level, 2, 7, 64, 147, 152, 163, 164-171, 173, 174, 175, 180, 183, 186, 256, 259, 260 defined, 162-164 demographic factors, 27, 163 frameworks and assessment development process, 114, 115, 118, 123, 125, 133, 142-143, 146, 149, 155, 159, 220, 257, 259-260 history of, 15 main NAEP, 4-5, 15-16, 77, 78-80 reporting of, 2, 6, 7, 25-31, 32, 33, 72, 73, 162, 163-165 students with disabilities/English-language learners, 87, 89, 94 see also Achievement levels; Constructed-response items; Cutscores; Extended-response items; Task measures

OCR for page 267
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress Perkins Act, 88 Policy Statement on Redesigning the National Assessment of Educational Progress, 11 Political factors, 9-11, 13-15, 19, 20, 23, 25-26, 31, 38-39 stakeholders, 3, 25, 80 see also Accountability; Education reform Poverty, 14, 44 Preliminary achievement-level descriptions, 119, 134-135, 165, 170, 176-179, 182, 183, 256, 257, 259, 260; see also Achievement-level descriptions Press, see Mass media Private School Universe Study, 36-37 Problem solving, 47, 57, 121, 127, 136-137, 138, 140, 146Q Quality Counts, 172R Race/ethnicity, 14, 50, 79, 169 African Americans, 44, 62 bilingual education, 91, 98-101 (passim), 105, 110 cultural bias, 63 Hispanics, 44, 62, 95, 99 translated versions of NAEP, 95, 97, 100, 101, 111 trend NAEP, 44, 79 RAND, 40, 46 Reading achievement, 7, 65, 166 frameworks and assessment development process, 118, 119, 120, 122, 123, 127-128, 132, 133, 136, 141-142, 219-230 items read orally due to poor readers, 13, 95, 99, 105 main NAEP, 15, 17, 65, 80, 84, 118, 120, 125, 127-128, 133, 171, 192-193, 219 NAEP administration schedule, 18 NAEP mandate, 24 national NAEP, 16, 18, 59, 73-75, 77 new paradigm NAEP, 84 oral, 16 reporting, 31-32 state-level trends, 26, 59 students with disabilities/English-language learners, 94, 110 text analysis, 43, 49, 219-220 time factors, reading passages, 122 trend NAEP, 15, 17, 18, 59, 63, 73-75, 77, 80, 84 see also Achievement levels; Core NAEP Reform of education, see Education reform Reform Up Close, 40 Regional factors and trends English-language learners, 89-90 NAEP results reporting, 1, 14, 19, 152 Report Cards, 8, 65, 71, 72, 73, 107, 121, 152, 154, 163, 170, 172 Reporting metrics, 72, 84, 86, 159, 162, 164, 171, 189 Reporting procedures, 1-2, 8, 20, 25-26, 64-65, 66, 67-68, 70-73, 86, 196 achievement levels, 2, 6, 7, 25-31, 32, 33, 72, 73, 162, 163-165 coordinated system of education indicators, 25-26, 73 costs, 78 demographic factors, 1-2, 27, 31, 34, 163 field tests, tables/displays, 71 frameworks and assessment development process, 114, 117, 121, 128, 135, 145-146, 151, 154-160 (passim), 221 governance, 19, 31, 72, 164, 172, 190 market-basket reporting, 72, 189 mathematics achievement, 28, 58, 59, 73, 171, 172 national NAEP, 1-2, 14, 71 NCES role, 31, 154, 196 performance standards, 2, 6, 7, 25-31, 32, 33, 72, 73, 162, 163-165 reading achievement, 31-32 regional trends, 1, 14, 19, 152 scoring and scoring criteria, 72, see also Summary scores standards-based, 2, 7, 25-26, 33, 162, 163-165, 167-168, 171-173, 182, 184 students with disabilities/English-language learners, 108, 111 state NAEP, 2, 10, 14, 15, 26, 31, 68, 71 timeliness, 67, 71, 152, 154 trend NAEP, 73, 163 see also Descriptive information; Evaluative information; Interpretive information

OCR for page 267
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress Rural areas, see Urban/rural factorsS Sampling, 2, 4, 11, 13, 14, 17, 20, 45, 46, 47, 49, 50-51, 56, 57, 58-63, 64, 66, 68, 86, 161 coordinated system of education indicators, 45, 46, 47, 49, 50-51, 107 cost factors, 62, 66 dropouts, 81, 86 errors, 82-83 probability sampling, 62, 64, 66, 68-69, 175 jackknife methods, 64, 69 matrix sampling, 13, 62, 68 balanced incomplete blocks, 62, 63, 66 national NAEP, 59-60, 62, 66 national trends, other, 59-63, 68, 194 state-level, 59-62, 65-66, 68, 83, 85, 194 state vs national NAEP, 83, 85 students with disabilities/English-language learners, 101, 102, 103, 106, 107 time factors, 69 see also Demographic factors; Error of measurement; Race/ethnicity Scaling, 4, 56, 66, 67, 72, 80, 101, 111, 117, 154, 163, 261 Scholastic Aptitude Test, 50, 93 School districts, 10, 82, 104 School-level measures, 27, 34, 39, 40, 41, 42, 45, 46, 47, 51 English-language learners, 89-90 Schools and Staffing Survey, 36-37, 41, 51 Science achievement, 7, 10, 27, 60, 65, 78, 80, 259 database, 50 frameworks and assessment development process, 116, 118, 126, 129-130, 132-133, 134, 136, 142, 143-144, 231-236 main NAEP, 15, 17, 28-30, 65, 77-78, 80, 84, 116, 118, 121, 125, 126, 157, 168, 192-193, 259 NAEP administration schedule, 18 national NAEP, 18, 60, 63 new paradigm NAEP, 84 standards, 10, 166, 168-171, 178-179 TIMSS, 36-37, 40, 42-44, 46, 48, 110, 169, 174, 188, 259 trend NAEP, 15, 17, 18, 60, 63, 80, 84, 259 see also Achievement levels; Core NAEP Scoring and scoring criteria, 67, 192 frameworks and assessment development process, 114, 117, 118, 119, 125, 132, 133-134, 135, 136, 146, 149, 150, 153-154, 156, 160, 161, 219-220, 226-227, 234-235, 257 levels and types of knowledge, 6 multicategory, 67 nonresponse rates, 64, 80 reporting, 72 state vs national NAEP, 82-83 see also Cutscores; Summary scores Second-language speakers, see English-language learners Sex differences, see Gender factors Social indicator systems, 3, 24, 38-39, 152, 153 Socioeconomic status, 50 poverty, 14, 44 Sociopolitical factors, see Demographic factors; Political factors Spanish-speaking students, see Hispanic students Special studies, 16, 57, 110 Special Study Panel on Education Indicators, 35, 37, 38, 40-41 Standards, general, 19, 20, 27, 126 coordinated system of education indicators, 23, 45 curriculum, 10, 179, 153 mathematics achievement, 58, 59, 73 NAEP history, 12, 15 reporting, standards-based, 2, 7, 25-26, 33, 162, 163-165, 167-168, 171-173, 182, 184 science achievement, 10, 166, 168-171, 178-179 students with disabilities/English-language learners, 5, 87, 89, 91, 92, 94, 98-102, 109, 111-113 see also Achievement-level descriptions; Achievement levels; Achievement-level setting; Performance standards; Preliminary achievement-level descriptions; Scoring and scoring criteria State Collaboratives on Assessment and Standards Project, 40, 46 State government, 19 costs, 78 curriculum reform, 13, 153 evaluation of programs, 11 NAGB and, 25, 65-66

OCR for page 267
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress national curriculum, 13 National Governors' Association, 15, 172 standards, 10, 56 students with disabilities/English-language learners, 88, 93, 94, 97, 98, 104, 106, 109, 110 State-level factors and trends, 4, 5, 23, 26, 189 age-vs grade-level testing, 14 coordinated system of education indicators, 45-46 main NAEP, 15, 16, 56, 57, 58, 65-66 reading achievement, 26, 59 sampling, 59-62, 65-66, 68, 83, 85, 194 sociopolitical factors, 10 State NAEP, 14, 15, 16, 45-46, 56, 57, 58, 65-66, 192-193 cost factors, 62, 66, 78, 82, 85 national NAEP vs, 65, 81-84, 85 objectives, 24 reporting, 2, 10, 14, 15, 26, 31, 68, 71 sampling, 59-62, 65-66, 68, 83, 85, 194 streamlining, 81-84 Student-level measures, 40, 42, 43, 45, 46 Students with disabilities, 5, 7-8, 14, 19, 20, 87-113, 134, 191 accommodations/adaptations, 51, 88-89, 91-112 (passim), 191 accountability, 91, 94 achievement levels, 88, 107, 109, 110-111, 112 administration procedures, testing, 91, 92, 99-100 analysis methods, 101, 104, 108, 110 coordinated system of education indicators, 106, 107 core NAEP, 109, 110, 112 cost factors, 8, 107-108, 111 curriculum, 111 data collection, 87, 96, 104, 106, 107, 108, 112, 113 demography of, 89-90 Department of Education role, 106, 109 NCES, 91, 93, 94, 97, 105, 106, 109, 191-192 educational indicators, general, 88, 106, 114 funding, 108, 109 Goals 2000: Educate America Act, 10, 88, 96 historical perspectives, 5, 92-98 identification/classification, 102, 104-105, 106 inclusion, 45, 92, 93, 95, 97-104, 106, 109, 111-113, 191 individualized education plans, 88 interpretive information, 5, 87, 99, 106, 112 large-scale assessments, 88, 89, 91-93, 96, 97, 98, 104, 106-109, 110, 112-113 legislation, 10, 88, 106, 108 NCES role, 91, 93, 94, 97, 105, 106, 109, 191-192 participation, 5, 19, 57, 87-88 performance standards, 87, 89, 94 reading achievement, 94, 110 reporting procedures, 108, 111 sampling, 101, 102, 103, 106, 107 standards, 5, 87, 89, 91, 92, 94, 98-102, 109, 111-113 validity issues, 88, 91, 95, 106, 109-111 see also Learning disabilities Students with limited English proficiency, see English-language learners Student/teacher ratios, 44 Summary scores, 1-2, 6 demographic trends, 14 error of measurement, 63 frameworks and assessment development process, 116, 117, 121, 128, 139, 152, 154T Task measures, 4, 12, 192 coordinated system of education indicators, 23, 24, 33, 47, 54 framework and assessment development process, 119, 128, 132, 136, 139, 142-144, 146, 149, 153-154, 160, 219-220 group, 47, 138 streamlining NAEP design, 67, 68, 71, 81, 86 see also Extended-response items Teacher background variables, 40, 41, 43 Teacher evaluation of NAEP, 77-78, 129-130 Teaching methods, see Pedagogy Technical Review Panel, 11, 166 Test administration procedures, see Administration procedures, testing Text analysis, 219-220 textbooks, 43, 49 Third International Mathematics and Science Study, 36-37, 40, 42-44, 46, 48, 110, 169, 174, 188, 259

OCR for page 267
GRADING THE NATION'S REPORT CARD: Evaluating NAEP and Transforming the Assessment of Educational Progress Time factors administration schedules, 18, 121, 191, 192-193, 195 assessment development processes, 124, 136, 159, 160 item response time, 13, 62, 99, 128, 130-131 reading passages, 122 reporting of results, 67, 71, 152, 154 sampling, 69 Trend NAEP, 4, 27, 56, 58, 65, 84, 85 achievement levels, 4-5, 15-16, 56, 58, 77, 78-80 cost factors, 78 frameworks, 77-78, 192, 194 main vs, 4-5, 15-16, 57, 58, 63, 64, 65, 73, 77, 78-80, 84, 85 mathematics achievement, 15, 17, 18, 60, 63, 73, 75, 77, 80, 84, 102 new paradigm, 47, 194 race, 44, 79 reading achievement, 15, 17, 18, 59, 63, 73-75, 77, 80, 84 reporting, 73, 163 sampling, 66 science achievement, 15, 17, 18, 60, 63, 80, 84, 259 writing achievement, 15, 17, 18, 59, 63, 80, 84 Trends in Academic Progress, 65 Trial State Assessment, 25, 94, 96-98 (passim), 174U Urban/rural factors English-language learners, 89-90 Use of NAEP results, 11, 20, 25-31, 39, 114, 181, 256 achievement levels, 2, 31, 171, 176, 180, 182, 184, 246 mass media, 9-10, 12, 25-30, 68, 71, 171 school districts, 104 stakeholders, 3, 25, 80V Validating National Curriculum Indicators, 40, 46 Validity issues, 11, 122, 153, 167-168, 173, 176-177, 182, 189, 221, 226 analysis and scaling models, 67 content, 12 coordinated system of education indicators, 40, 46 students with disabilities/English-language learners, 88, 91, 95, 106, 109-111 Validity Studies Panel, 11 Video techniques, instructional methods analysis, 43, 46, 48, 194W WESTAT, 17, 119 Writing achievement, 7, 59, 65 main NAEP, 15, 17, 65, 80, 84, 192-193 NAEP administration schedule, 18 NAEP mandate, 24 national NAEP, 16, 18, 59 new paradigm NAEP, 84 extended performance, 16 trend NAEP, 15, 17, 18, 59, 63, 80, 84 trend vs main NAEP, 79, 80 see also Core NAEP