NAEP Reporting Practices: Investigating District-Level and Market-Basket Reporting (2001)

Page 112

Page 113

Carswell, C.M., Frankenberger, S., & Bernhard, D. (1991) Graphing in depth: Perspectives on the use of three-dimensional graphs to represent lower-dimensional data. Behaviour and Information Technology, 10(6), 459-474.

Cieslak, P. (2000). Milwaukee's experience with district-level NAEP results. Paper presented at the February workshop of the Committee on NAEP Reporting Practices: Investigating District-Level and Market-Based Reporting, National Research Council, Washington, DC.

Cleveland, W.S. (1985). The elements of graphing data. Monterey, CA: Wadsworth.

Cleveland, W.S. (1993). Visualizing Data. Summit, NJ: Hobart Press.

Cleveland, W.S., & McGill, R. (1984). Graphical perception: Theory, experimentation, and application to the development of graphic methods. Journal of the American Statistical Association, 70, 531-534.

Cleveland, W.S., & McGill, R. (1985). Graphical perception and graphical methods for analyzing scientific data. Science, 229, 838-833.

Colvin, R.L. (2000). NAEP Market-basket reporting: A journalist's perspective. Paper presented at the February workshop of the Committee on NAEP Reporting Practices: Investigating District-Level and Market-Based Reporting, National Research Council, Washington, DC.

Council of Chief State School Officers. (2000). Annual survey of state student assessment programs: A summary report, fall 1999. Washington, DC: Author.

Dent, B.D. (1993). Cartography: Thematic map design (3rd ed.). Dubuque, IA: William C. Brown.

DeVito, P.J. (1997). The future of the National Assessment of Educational Progress from the states' perspective. In Assessment in transition: Monitoring the nation's educational progress. Stanford, CA National Academy of Education.

Educational Testing Service . (1997a). NAEP 1996 mathematics report for Milwaukee Public Schools grade 8: Findings from a special study of the National Asssessment of Educational Progress. Princeton, NJ: Author.

Educational Testing Service. (1997b). NAEP 1996 science report for Milwaukee Public Schools grade 8: Findings from a special study of the National Asssessment of Educational Progress. Author.

Educational Testing Service. (1998). Prepare for mathematics market basket (Chapter 11) and analyze and report on mathematics market basket booklet (Chapter 18, Task 52). In NAEP 2000: Application for cooperative agreement for the National Assessment of Educational Progress—Technical application. Author.

Feldt, L.S., & Brennan, R.L. (1989). Reliability. In R.L. Linn(Ed.), Educational Measurement 3rd ed. (pp. 105-146). New York, NY: Macmillan.

Forsyth, R.A. (1991). Do NAEP scales yield valid criterion-referenced interpretations? Educational Measurement: Issues and Practice , 10(3), 3-9, 16.

Forsyth, R., Hambleton, R., Linn, R., Mislevy, R., & Yen, W. (1996, July 1). Design and feasibility team report to the National Assessment Governing Board .

Gillian, D.J., Wickens, C.D., Hollands, J.G., & Carswell, C.M. (1998). Guidelines for presenting quantitative data in HFES publications. Human Factors, 40, 28-41.

Glaser, R., Linn, R., & Bohrnstedt, G. (1997). Assessment in transition: Monitoring the nation's educational progress. Stanford, CA: National Academy of Education.

Page 114

Haertel, E.H. (1991). Reasonable inferences for the trial state NAEP given the current design: Inferences that can and cannot be made. In Assessing student achievement in the states: Background studies. Stanford, CA: National Academy of Education.

Hambleton, R.K., Brennan, R.L., Brown, W., Dodd, B., Forsyth, R.A., Mehrens, W.A., Nellhaus, J., Reckase, M., Rindone, D., van der Linden, W.J., & Zwick, R. (2000). A response to “Setting Reasonable and Useful Performance Standards” in the National Academy of Sciences' Grading the nation's report card. Educational Measurement: Issues and Practice, 19(2), 5-14.

Hambleton, R.K., & Meara, K. (2000). Newspaper coverage of NAEP results, 1990 to 1999. In National Assessment Governing Board (Ed.), Student performance standards of the National Assessment of Educational Progress: Affirmation and improvements . Washington,DC: Editor.

Hambleton, R.K., & Slater, S.C. (1996). Are NAEP executive summary reports understandable to policymakers and educators? Paper presented at the annual meeting of the National Council on Measurement in Education, New York .

Haney, W., & Madaus, G.F. (1991). Caution on the future of NAEP: Arguments against using NAEP tests and data reporting below the state level. In Assessing student achievement in the states: Background studies. Stanford, CA: National Academy of Education.

Hartka, L., & Stancavage, F. (1994). Perspectives on the impact of the 1994 trial state assessments: State assessment directors, state mathematics specialists, and state reading specialists . In Quality and utility: The 1994 trial state assessment in reading: Background studies. Stanford, CA: National Academy of Education.

Herrmann, D., & Pickle, L.W. (1996). A cognitive subtask model of statistical map reading. Visual Cognition, 3, 165-190.

Hubel, D.H., & Weisel, T.N. (1965). Receptive fields of single neurons in two nonstiate visual areas (18 and 19) of the cat. Journal of Neurophysiology, 28, 229-289.

Hubel, D.H., & Weisel, T.N. (1979). Brain mechanisms and vision. Scientific American. 241(3), 150-162.

Jaeger, R.M. (1996). Reporting large scale assessment results for public consumption: Some propositions and palliatives. Paper presented at the annual meeting of the National Council on Measurement in Education, New York.

Jaeger, R.M. (1998, September). Reporting the results of the National Assessment of Educational Progress. Paper commissioned by the NAEP Validity Studies Panel.

Johnson, E. , Lazer, S., & O'Sullivan, C. (1997). NAEP reconfigured: An integrated redesign of the National Assessment of Educational Progress. Washington, DC: National Center for Education Statistics.

Kenney, P.A. (2000). Market basket reporting for NAEP: A content perspective. Paper presented at the March workshop of the Committee on NAEP Reporting Practices: Investigating District-Level and Market-Based Reporting, National Research Council, Washington, DC .

Kiplinger, V.L., & Linn, R.L. (1992). Raising the stakes of test administration: The impact on student performance on NAEP (CSE Technical Report No. 360). Los Angeles, CA: University of California, National Center for Research on Evaluation, Standards, and Student Testing.

Page 115

Kiplinger, V.L. , & Linn, R.L. (1995/1996). Raising the stakes of test administration: The impact of student performance on NAEP . Educational Assessment, 3, 311-333.

Kolen, M.J., & Brennan, R.L. (1995). Test equating methods and practices. New York, NY: Springer-Verlag.

Kolstad, A. (2000 ). Simplifying the interpretation of NAEP results with market baskets and shortened forms of NAEP. Paper presented at the February workshop of the Committee on NAEP Reporting Practices: Investigating District-Level and Market-Based Reporting, National Research Council, Washington, DC .

Koretz, D.M. (1991). State comparisons using NAEP: Large costs, disappointing benefits. Educational Researcher, 20(3), 19-21.

Koretz, D.M., & Deibert, E. (1995/1996). Setting standards and interpreting achievement: A cautionary tale from the National Assessment of Educational Progress. Educational Assessment, 3(1), 53-81.

Kosslyn S. (1985). Graphics and human information processing. Journal of the American Statistical Association, 80, 499-512.

Kosslyn, S. (1994). Elements of graph design. New York, NY: W. H. Freeman.

Linn, R.L., Koretz, D., & Baker, E.L. (1996). Assessing the validity of the National Assessment of Educational Progress: NAEP technical review panel white paper. Los Angeles, CA: University of California, National Center for Research on Evaluation, Standards, and Student Testing.

Lord, F.M., & Novick, M.R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.

Mazzeo, J. (2000). NAEP's year-2000 market-basket study: What do we expect to learn? Paper presented at the February workshop of the Committee on NAEP Reporting Practices: Investigating District-Level and Market-Based Reporting, National Research Council, Washington, DC.

McConachie, M. (2000). State policy perspectives on NAEP market basket reporting. Paper presented at the February workshop of the Committee on NAEP Reporting Practices: Investigating District-Level and Market-Based Reporting, National Research Council, Washington, DC.

McDonald, R.P. (1999). Test theory: A unified approach. Mahwah, NJ: Lawrence Erlbaum.

McDonnell, L.M. (1994). Policymakers' views of student assessment. Report commissioned by the Office of Educational Research and Improvement, U.S. Department of Education. Santa Monica, CA: RAND Institute on Education and Training.

Milroy, R., & Poulton, E.C. (1978). Labeling graphs for increased reading speed. Ergonomics, 22, 55-61.

Mislevy, R. (2000). Evidentiary relationships among data-gathering methods and reporting scales in surveys of educational achievement. Paper presented at the February workshop of the Committee on NAEP Reporting Practices: Investigating District-Level and Market-Based Reporting, National Research Council, Washington, DC.

Monmonier, M. (1991). How to lie with maps. Chicago, IL: University of Chicago Press.

National Assessment Governing Board. (1995a). Guidelines for the conduct of below-state NAEP assessments, draft implementation document. Washington, DC: Author.

National Assessment Governing Board. (1995b). Guidelines for the conduct of below-state NAEP assessments, policy statement. Washington, DC: Author.

Page 116

National Assessment Governing Board. (1996). Redesigning the National Assessment of Education Progress, policy statement. Washington, DC: Author.

National Assessment Governing Board. (1997). Resolution on market basket reporting, report of August 2. Washington, DC: Author.

National Assessment Governing Board. (1999a). The National Assessment of Educational Progress: Design 2000-2010. Washington, DC: Author.

National Assessment Governing Board. (1999b, November). Policy guidance on the NAEP short form. Washington, DC: Author.

National Assessment Governing Board. (1999c). Reporting and Dissemination Committee agenda of November 19. Washington, DC: Author.

National Assessment Governing Board. (1999d). Reporting and Dissemination Committee report of August 6. Washington, DC: Author.

National Assessment Governing Board. (2000a). Design and Methodology Committee agenda of August 4. Washington, DC: Author.

National Assessment Governing Board. (2000b). Mathematics framework for the 1996 and 2000 National Assessment of Educational Progress. [On-line]. Available http://www.nagb.org.

National Center for Education Statistics. (1995). Draft guidelines and technical specifications for the conduct of below-state NAEP assessments. Washington, DC: Author.

National Center for Education Statistics. (2000a, May 9 ). A brief history of NAEP participation. Paper prepared for meeting of the Design and Analysis Committee, Washington, DC: Author.

National Center for Education Statistics . (2000b, May 9). Rewards for NAEP: Proposals and consequences. Paper prepared for meeting of the Design and Analysis Committee , Washington, DC: Author.

National Research Council. (1999a). Embedding questions: The pursuit of a common measure in uncommon tests. Committee on Embedding Common Test Items in State and District Assessments. D.M. Koretz, M.W. Bertentha , & B.F. Green , (Eds.), Board on Testing and Assessment, Commission on Behavioral and Social Sciences and Education. Washington, DC: National Academy Press.

National Research Council. (1999). Grading the nation's report card: Evaluating NAEP and transforming the assessment of educational progress . Committee on Evaluation of National and State Assessments of Educational Progress . J.W. Pellegrino , L.R. Jones , & K.M. Mitchell , (Eds.), Commission on Behavioral and Social Sciences and Education. Washington, DC: National Academy Press.

National Research Council. (1999c). Reporting district-level NAEP data: Summary of a workshop. Committee on NAEP Reporting Practices: Investigating District-Level and Market-Basket Reporting. P.J. DeVito & J.A. Koenig, (Eds.), Board on Testing and Assessment, Commission on Behavioral and Social Sciences and Education. Washington, DC: National Academy Press.

National Research Council. (1999d). Uncommon measures: Equivalence and linkage among educational tests. Committee on Equivalency and Linkage of Educational Tests. M.J. Feuer, P.W. Holland, B.F. Green, M.W. Bertenthal, & F.C. Hemphill, (Eds.), Commission on Behavioral and Social Sciences and Education. Washington, DC: National Academy Press.

Page 117

National Research Council. (2000). Designing a market-basket for NAEP: Summary of a workshop. Committee on NAEP Reporting Practices: Investigating District-Level and Market-Basket Reporting. P.J. DeVito & J.A. Koenig, (Eds.), Board on Testing and Assessment. Washington, DC: National Academy Press.

Neilsen, J. (1993). Usability engineering. San Diego, CA: Morgan Kaufmann.

Norman, D.A. (1988). The psychology of everyday things. New York, NY: Basic Books.

Nunnally, J.C. (1967). Psychometric theory. New York, NY: McGraw-Hill.

O'Reilly, J. (2000). District level and market-basket reporting: A district perspective. Paper presented at the February workshop of the Committee on NAEP Reporting Practices: Investigating District-Level and Market-Based Reporting, National Research Council, Washington, DC.

Pickle, L.W., & Herrmann, D. (1994). The process of reading statistical maps: The effect of color. Statistical Computing and Statistical Graphics Newsletter, 5(1), 1, 12-16.

Pommerich, M., & Nicewander, W.A. (1998). Estimating average domain scores. Iowa City, IA: American College Testing.

Roeber, E.D. (1994). Guidelines for the use of NAEP at the district and school levels. Paper commissioned by the National Assessment Governing Board. February.

Rubin, D.B. (1987). Multiple imputation for nonresponse in surveys. New York, NY: John Wiley & Sons.

Rubin, J. (1994). Handbook of usability testing. New York, NY: John Wiley & Sons.

Rust, K. (1999). NAEP sample designs and district level reporting. Paper prepared for the National Research Council Workshop on District-Level Reporting, Washington, DC.

Selden, R. (1991). The case for district- and school-level results from NAEP. In Assessing student achievement in the states: Background studies. Stanford, CA: National Academy of Education.

Simkin, D., & Hastie, R. (1987). An information processing analysis of graph perception. Journal of the American Statistical Association, 82, 454-465.

Spence, I., & Lewandowski, S. (1989). Robust multidimensional scaling. Psychometrika, 54(3).

Stancavage, F.B., Roeber, E., & Bohrnstedt, G.H. (1992). A study of the impact of reporting the results of the 1990 trial state assessment: First report. Assessing student achievement in the states: Background studies. Stanford, CA: National Academy of Education.

Stancavage, F.B., Roeber, E., & Bohrnstedt, G.H. (1993). Impact of the 1990 trial state assessment: A follow up study. The trial state assessment: Prospects and realities: Background studies. Stanford, CA: National Academy of Education.

Stanley, J.C. (1971). Reliability. In R.L. Thorndike (Ed.), Educational measurement (2nd ed.). Washington, DC: American Council on Education.

Truby, R. (2000, February). A market basket for NAEP: Policies and objectives of the National Assessment Governing Board. Paper presented at the workshop of the Committee on NAEP Reporting Practices: Investigating District-Level and Market-Based Reporting, National Research Council, Washington, DC.

Tversky, B., & Schiano, D.J. (1989). Perceptual and conceptual factors in distortions in memory for graphs and maps. Journal of Experimental Psychology: General, 118(4), 387-398.

Page 118

U.S. Department of Education. National Center for Education Statistics. (1999). Horkay, N. (Ed.), The NAEP guide (NCES Report No. 2000-456).

Vernon, M.D. (1952). The use and value of graphical methods of presenting quantitative data. Occupational Psychology, 26, 22-24.

Wainer, H. (1997a). Improving tabular displays with NAEP tables as examples and inspirations. Journal of Educational and Behavioral Statistics, 22, 1-30.

Wainer, H. (1997b). Visual revelations: Graphical tales of fate and deception from Napoleon Bonaparte to Ross Perot. New York: Copernicus.

Wainer, H., Hambleton, R.K., & Meara, K. (1999). Alternative displays for communicating NAEP results: A redesign and validity study. Journal of Educational Measurement, 36(4), 301-335.

Williams, P.L., Reese, C.M., Campbell, J.R., Mazzeo, J., & Phillips, G.W. (1995). NAEP 1994 reading: A first look. Washington, DC: United States Department of Education.