Appendix A
The Committee’s Public Forums on Performance Levels for NAAL

Public Forum—February 27, 2004

Panelists

Cynthia Baur, U.S. Department of Health and Human Services

Beth Beuhlmann, U.S. Chamber of Commerce, Workforce Preparation

Richard Colvin, Hechinger Institute

Leslie Farr, Ohio State University

Milton Goldberg, Education Commission of the States (ECS)

Richard Long, International Reading Association

Christopher Mazzeo, National Governors Association

Gemma Santos, Miami Dade Public Schools

Tony Sarmiento, Senior Service America, Inc.

Linda Taylor, Comprehensive Adult Student Assessment System

Robert Wedgeworth, Proliteracy Worldwide

Participants

Joan Auchter, GED Testing Service/American Council on Education

Justin Baer, American Institutes for Research (AIR)

Amy Baide, Department of Homeland Security

Sandra Baxter, The National Institute for Literacy (NIFL)

Jaleh Behroozi, The National Institute for Literacy (NIFL)

Martha Berlin, Westat



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 203
Measuring Literacy: Performance Levels for Adults Appendix A The Committee’s Public Forums on Performance Levels for NAAL Public Forum—February 27, 2004 Panelists Cynthia Baur, U.S. Department of Health and Human Services Beth Beuhlmann, U.S. Chamber of Commerce, Workforce Preparation Richard Colvin, Hechinger Institute Leslie Farr, Ohio State University Milton Goldberg, Education Commission of the States (ECS) Richard Long, International Reading Association Christopher Mazzeo, National Governors Association Gemma Santos, Miami Dade Public Schools Tony Sarmiento, Senior Service America, Inc. Linda Taylor, Comprehensive Adult Student Assessment System Robert Wedgeworth, Proliteracy Worldwide Participants Joan Auchter, GED Testing Service/American Council on Education Justin Baer, American Institutes for Research (AIR) Amy Baide, Department of Homeland Security Sandra Baxter, The National Institute for Literacy (NIFL) Jaleh Behroozi, The National Institute for Literacy (NIFL) Martha Berlin, Westat

OCR for page 203
Measuring Literacy: Performance Levels for Adults Peggy Carr, Department of Education, National Center for Education Statistics (NCES) June Crawford, The National Institute for Literacy (NIFL) Elizabeth Greenberg, American Institutes for Research (AIR) Ricardo Hernandez, Office of Vocational and Adult Education (OVAE) Shannon Holmes, U.S. Conference of Mayors Eugene Johnson, American Institutes for Research (AIR) Linda Johnston Lloyd, Health Resources & Services Administration (HRSA) Michael Jones, Office of Vocational and Adult Education (OVAE) Cheryl Keenan, Office of Vocational and Adult Education (OVAE) Irwin Kirsch, Educational Testing Service (ETS) Andy Kolstad, National Center for Education Statistics (NCES) Mark Kutner, American Institutes for Research (AIR) Mariann Lemke, National Center for Education Statistics (NCES) Anne Lewis, freelance journalist Lennox McLendon, National Adult Education Professional Development Consortium Wendy Mettger, Mettger Communications Leyla Mohadjer, Westat Gerri Ratliff, Department of Homeland Security Lyn Schaefer, GED Testing Service Peggy Seufert, American Institutes for Research (AIR) Sondra Stein, The National Institute for Literacy (NIFL) Lynn Thai, Department of Homeland Security Peter Waite, Proliteracy America Dan Wagner, National Center on Adult Literacy Maria White, Department of Health and Human Services Sheida White, National Center for Education Statistics (NCES) Kentaro Yamamoto, Educational Testing Service (ETS) Representatives from State Departments of Adult Education—April 22-23, 2004 Bob Bickerton, Massachusetts Steve Coffman, Missouri Donna Cornelius, Massachusetts Cheryl King, Kentucky Tom Orvino, New York Ann Serino, Massachusetts Reecie Stagnolia, Kentucky Linda Young, Oklahoma

OCR for page 203
Measuring Literacy: Performance Levels for Adults Stakeholder Questions and Alternative Versions of Performance-Level Descriptions for the National Assessment of Adult Literacy In what ways did you use the results from the 1992 NALS? What were the strengths and weaknesses of these performance levels? To what extent did these performance levels provide you with the information that you needed? NAAL measures skills in the areas of prose, document, and quantitative literacy. To what extent is it useful and informative to have different performance level descriptions for each area? Are results from the three areas of literacy used differently? If so, how? The attachment presents three alternative versions of performance-level descriptions for the prose literacy scale. Sample 1 is simply a reformatted version of the existing performance-level descriptions with 5 levels. Sample 2 is a 4-level model, and Sample 3 is a 3-level model. Please comment on how many levels are needed. What types of decisions are made at the various levels? What are the critical distinctions that need to be made? Level Labels: The three samples present different labels for the levels. Sample 1 uses numbers (Col. 2). Samples 2 and 3 use phrases as labels (Col. 2). In addition, Sample 3 presents a narrative description of the label (Col. 3). Please comment on these alternative labels. What types of labels are useful and informative? Feel free to make suggestions for alternative labels. Level Descriptions: The three samples present different ways of describing the skills represented by the performance level. Sample 1 describes the tasks associated with the level (Col. 3). Sample 2 describes what an average respondent who scores at this level should be able to do (Col. 3). Sample 3 (Col. 4) describes what the average respondent who scores at this level is able to do and not able to do in probabilistic terms (i.e., likely, not likely). Please comment on these alternative ways of describing the skills associated with the levels. What types of descriptions are useful and informative? Feel free to make suggestions for alternative descriptions. Sample Tasks: The three samples present different ways of exemplifying the tasks respondents who score at the level should be able to do. Samples 1 and 2 (Col. 4) are similar and provide examples drawn from actual assessment. Sample 3 (Col. 5) attempts to generalize from assessment tasks to real world tasks. Please comment on the extent to which these exemplifications are useful and informative. Relationships Between Prose Scores and Background Data: Samples 2 and 3 present the relationships between NAAL scores and key real-world factors as measured on the background questionnaire. Sample 2 (Col. 5) uses societal factors (income, education, voting) and Sample 3 (Col. 6) uses reading related factors. (Please be aware that the percentages in

OCR for page 203
Measuring Literacy: Performance Levels for Adults cluded in these examples are purely hypothetical. If we were to recommend this format, the percentages would be based on analyses with NAAL data.) Please comment on the utility of this type of information.

OCR for page 203
Measuring Literacy: Performance Levels for Adults SAMPLES ARE ON THE FOLLOWING PAGES

OCR for page 203
Measuring Literacy: Performance Levels for Adults Sample 1: Five-Level Model, Based on Current PLDs for Prose Literacy (Col. 1) Level (Col. 2) Label (Col. 3) Description of tasks (Col. 4) Sample NAAL tasks associated with the level I Level 1 Most of the tasks in this level require the reader to read relatively short text to locate a single piece of information which is identical to or synonymous with the information given in the question or directive. If plausible but incorrect information is present in the text, it tends not to be located near the correct information. Locate one piece of information in a sports article. Identify the country in a reading passage. Underline sentence explaining action stated in short article. II Level 2 Some tasks in this level require readers to locate a single piece of information in the text; however, several distractors or plausible but incorrect pieces of information may be present, or low-level inference may be required. Other tasks require the reader to integrate two or more pieces of information or to compare and contrast easily identifiable information based on a criterion provided in the question or directive. Underline the meaning of a term given in government brochure on supplemental security income. Locate two features of information in sports article. Interpret instructions from an appliance warranty. III Level 3 Tasks in this level tend to require readers to make literal or synonymous matches between the text and information given in the task, or to make matches that require low-level inferences. Other tasks ask readers to integrate information from dense or lengthy text that contains no organizational aids Write a brief letter to complain about an error on a bill. Read a news article and identify a sentence that provides interpretation of a situation. Read lengthy article to identify two behaviors that meet a stated condition.

OCR for page 203
Measuring Literacy: Performance Levels for Adults     such as headings. Readers may also be asked to generate a response based on information that can be easily identified in the text. Distracting information is present, but is not located near the correct information.   IV Level 4 These tasks require readers to perform multiple-feature matches and to integrate or synthesize information from complex or lengthy passages. More complex inferences are needed to perform successfully. Conditional information is frequently present in tasks in this level and must be taken into consideration by the reader. State in writing an argument made in a newspaper article. Explain differences between two types of employee benefits. Contrast views expressed in two editorials on technologies available to make fuel-efficient cars. V Level 5 Some tasks in this level require the reader to search for information in dense text which contains a number of plausible distractors. Others ask readers to make high-level inferences or use specialized background knowledge. Some tasks ask readers to contrast complex information. Compare approaches stated in a narrative on growing up. Summarize two ways lawyers may challenge prospective jurors. Interpret a phrase from a lengthy news article.

OCR for page 203
Measuring Literacy: Performance Levels for Adults Sample 2: Four-Level Model for Prose Literacy (Col. 1) Level (Col. 2) Label (Col. 3) The average respondent who scores at this level should be able to: (Col. 4) Sample of NAAL tasks the average respondent should be able to: (Col. 5) Relationships with societal factors I Below Basic Identify letters and numbers. Point to orally presented words. Read aloud words, phrases. or short sentences. Locate specific information on a food label. Identify a specific word in an advertisement. Locate an address in an advertisement. >50% of being poor or near poor >50% chance of not having a U.S. high school diploma <50% chance of voting in most recent election II Basic Locate a single piece of information in a brief written text that uses organizational aids such as headings. Make simple inferences. Provide appropriate answers to questions that require integrating two or more pieces of information. Locate one piece of information in a sports article. Identify the country in a reading passage. Interpret brief instructions from warranty information. 25-50% of being poor or near poor 50% chance of having at least a high school diploma 50-60% chance of voting in most recent election

OCR for page 203
Measuring Literacy: Performance Levels for Adults III Intermediate Generate a response based on information that is easily identifiable in dense or lengthy text. Integrate information from dense or lengthy text. Write a brief letter to complain about an error on a bill. Read a magazine article and identify the facts that support a specific inference. 10-25% of being poor or near poor 50% chance of having at least some college 60-70% chance of voting in recent election IV Advanced Search for information that is in dense texts. Integrate or synthesize information from complex texts. Perform complex inferences. State in writing an argument made in a newspaper article. Explain the differences between two types of employee benefits. Contrast views expressed in two editorials. <10% of being poor or near poor 50% chance of having at least an associates college degree 70-80% chance of voting in recent election NOTES: NAAL background data will be analyzed to provide information about the likely relationships between literacy levels and the three societal factors included in Column 5; the current figures are for demonstration purposes only. Many of these descriptions may not generalize to English language learners (ELL). ELLs may be literate in languages other than English, and the relationships with societal factors (Col. 5) may be different for ELLs than for native English speakers. The response mode may also affect ELLs’ performance, since writing skills in English may develop at a slower pace than reading or speaking skills.

OCR for page 203
Measuring Literacy: Performance Levels for Adults Sample 3: Three-Level Model for Prose Literacy (Col. 1) Level (Col. 2) Label (Col. 3) Description of label for individuals who score at this level: (Col. 4) The average respondent who scores at this level: (Col. 5) Sample real-world tasks the average respondent at this level should be able to do: (Col. 6) Relationship to reading-oriented factors I Minimally Literate in English Are not able to independently handle most of the tasks of daily living that require literacy skills in English. Is likely to be able to identify letters and numbers; point to words when they are presented orally; orally read individual words, phrases, or short sentences. Is not likely to able to read connected text. Place signature in proper place on a form. Follow a simple recipe. Identify an address in an advertisement. >80% chance of never reading the newspaper >80% chance of needing a lot of help with printed information II Somewhat Literate in English Should be able to independently handle some of the tasks of daily living that require literacy skills in English. Is likely to be able to locate a single piece of information in a brief written text with organizational aids, such as headings; to integrate two or more pieces of information; and to compare and contrast easily identifiable information. Follow directions on a medicine bottle. Read aloud to a child in preschool or elementary grade. 40-80% chance of never reading the newspaper 40-80% chance of needing a lot of help with printed information

OCR for page 203
Measuring Literacy: Performance Levels for Adults       Is not likely to be able to integrate or synthesize information from dense text with no organizational aids.     III Fully Literate in English Should be able to independently handle most of the tasks of daily living that require literacy skills in English. Is likely to be able to integrate or synthesize information from dense or lengthy text that contains no organizational aids; and to generate a response based on information that can be identified in the text. Read and understand New York Times. Understand a proposition on a ballot. <10% chance of never reading the newspaper <10% chance of needing a lot of help with printed information NOTES: NAAL background data will be analyzed to provide information about the likely relationships between literacy levels and the reading oriented factors in Column 6; the current figures are for demonstration purposes only. These descriptions may not generalize to English language learners (ELL). ELLs may be literate in languages other than English, and their performance on real world tasks (Col. 5 and 6) may be different than for native English speakers. The response mode may also affect ELLs’ performance, since writing skills in English may deve lop at a slower pace than reading or speaking skills.