Appendix C
July 2004 Bookmark Standard-Setting Session with the 1992 NALS Data

As described in the body of the report, the Committee on Performance Levels for Adult Literacy convened two bookmark standard-setting sessions in 2004, one in July to gather panelists’ judgments about cut scores for the 1992 National Adult Literacy Survey (NALS) and another in September to collect judgments about cut scores for the 2003 National Assessment of Adult Literacy (NAAL). This appendix details how the bookmark procedure was implemented and reports results for the July session, and Appendix D presents similar information for the September session. Following the text are the background materials, which include the agenda, participant questionnaires, tables, and figures for the July session. The appendix concludes with technical details about the data files that the committee used for the standard settings; this information is provided to assist the U.S. Department of Education and its contractors with any follow-up analyses that need to be conducted with respect to the cut scores for the performance levels.

BOOKMARK STANDARD SETTING WITH THE 1992 NALS DATA

The July 2004 session was held to obtain panelists’ judgments about cut scores for the 1992 NALS and to collect their feedback about the performance-level descriptions. Several consultants assisted the committee with the standard setting, including Richard Patz, one of the original developers of the bookmark procedure.

A total of 42 panelists participated in the standard setting. Background information on the panelists was collected by means of a questionnaire (a



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 221
Measuring Literacy: Performance Levels for Adults Appendix C July 2004 Bookmark Standard-Setting Session with the 1992 NALS Data As described in the body of the report, the Committee on Performance Levels for Adult Literacy convened two bookmark standard-setting sessions in 2004, one in July to gather panelists’ judgments about cut scores for the 1992 National Adult Literacy Survey (NALS) and another in September to collect judgments about cut scores for the 2003 National Assessment of Adult Literacy (NAAL). This appendix details how the bookmark procedure was implemented and reports results for the July session, and Appendix D presents similar information for the September session. Following the text are the background materials, which include the agenda, participant questionnaires, tables, and figures for the July session. The appendix concludes with technical details about the data files that the committee used for the standard settings; this information is provided to assist the U.S. Department of Education and its contractors with any follow-up analyses that need to be conducted with respect to the cut scores for the performance levels. BOOKMARK STANDARD SETTING WITH THE 1992 NALS DATA The July 2004 session was held to obtain panelists’ judgments about cut scores for the 1992 NALS and to collect their feedback about the performance-level descriptions. Several consultants assisted the committee with the standard setting, including Richard Patz, one of the original developers of the bookmark procedure. A total of 42 panelists participated in the standard setting. Background information on the panelists was collected by means of a questionnaire (a

OCR for page 221
Measuring Literacy: Performance Levels for Adults blank questionnaire is included in Background Materials at the end of this appendix). A majority (85 percent, n = 28) had managerial responsibilities for adult education in their states or regional areas, although many panelists were instructors as well as program coordinators or directors. Most panelists worked in adult basic education (66 percent, n = 22), general educational development or GED (54 percent, n = 18), or English language instruction (51 percent, n = 17) settings. Almost half (45 percent, n = 15) reported they were very familiar with NALS prior to participating in the standard-setting activities; 42 percent (n = 14) reported that they were somewhat familiar with NALS. Only four participants (12 percent) who completed the questionnaire said they were unfamiliar with NALS prior to the standard setting. Panelists were assigned to tables using a quasi-stratified-random procedure intended to produce groups with comparable mixtures of perspectives and experience. To accomplish this, panelists were assigned to one of nine tables after being sorted on the following criteria: (1) their primary professional responsibilities (instructor, coordinator or director, researcher), (2) the primary population of adults they worked with as indicated on their resumes, and (3) the areas in which they worked as indicated on their resumes. The sorting revealed that panelists brought the following perspectives to the standard-setting exercise: adult basic education (ABE) instructor, English for speakers of other languages (ESOL) instructor, GED instructor, program coordinator or director, or researcher. Panelists in each classification were then randomly assigned to one of the nine tables so that each group included at least one person from each of the classifications. Each table consisted of four or five panelists and had a mixture of perspectives: instructor, director, researcher, ESOL, GED, and ABE. Once panelists were assigned to tables, each table was then randomly assigned to two of the three literacy areas (prose, document, or quantitative). The sequence in which they worked on the different literacy scales was alternated in an attempt to balance any potential order effects (see Table C-1). Three tables worked with the prose items first (referred to as Occasion 1 bookmark placements) and the document items second (referred to as Occasion 2 bookmark placements); three tables worked with the document items first (Occasion 1) and the quantitative items second (Occasion 2); and three tables worked with the quantitative items first (Occasion 1) and the prose items second (Occasion 2). Ordered Item Booklets For each literacy area, an ordered item booklet was prepared that rank-ordered the test questions from least to most difficult according to the responses of NALS examinees. The ordered item booklets consisted of all

OCR for page 221
Measuring Literacy: Performance Levels for Adults the available NALS tasks for a given literacy area, even though with the balanced incomplete block spiraling design used for the assessment, no individual actually responded to all test questions. The tasks were arranged in the ordered item booklets so that the question appeared first (one question per page) followed by the stimulus materials (e.g., a newspaper article, a bus schedule, a graph) and the scoring rubric. Accompanying each ordered item booklet was an item map that listed each item number and a brief description of the item. The number of items in each NALS ordered item booklet was 39 for prose literacy, 71 for document literacy, and 42 for quantitative literacy. Training Procedures Two training sessions were held, one just for the table leaders, the individuals assigned to be discussion facilitators for the tables of panelists, and one for all panelists. The role of the table leader was to serve as a discussion facilitator but not to dominate the discussion or to try to bring the tablemates to consensus about cut scores. Table leaders also distributed standard-setting materials to each table member, guided the discussions of the content and context characteristics that differentiated NALS test items from each other, led the discussion of the impact data for the final round of bookmark placements, and ensured that security procedures were followed. Table leader training was held the day before the standard setting to familiarize the table leaders with their roles, the NALS materials, and the agenda of activities for the standard-setting weekend. (The agenda for the July session is included in Background Materials at the end of this appendix.) Panelist training was held the morning of the standard setting. Richard Patz facilitated both training sessions and used the same training materials for both sessions. This helped ensure that the table leaders were well acquainted with the bookmark process. The training began with an overview of NALS (skills assessed by the tasks in the three literacy areas, administrative procedures, etc.), followed by background about the committee’s charge and the timing of its work. Panelists were told that the cut scores that resulted from the bookmark procedure would be the group’s recommendations to the committee but that it would ultimately be up to the committee to determine the final cut scores to recommend to the Department of Education. Panelists then received instruction in the elements and procedures of the bookmark method. Conducting the Standard Setting Once the training session was completed, the bookmark process began by having each panelist respond to all the questions in the NALS test

OCR for page 221
Measuring Literacy: Performance Levels for Adults booklet for their assigned literacy scale. For this task, the test booklets contained the full complement of NALS items for each literacy scale, arranged in the order that test takers would see them but not ranked-ordered as in the ordered item booklets. Afterward, the table leader facilitated discussion of differences among items with respect to knowledge, skills, and competencies required and familiarized panelists with the scoring rubrics. Panelists were expected to take notes during the discussion, which would be used in making their judgments. Panelists then received the ordered item booklets. They discussed each item and noted characteristics they thought made one item more difficult than another. The table leader distributed the performance-level descriptions.1 Each table member then individually reviewed the performance-level descriptions, the items in the ordered item booklet, the scoring rubrics, and their notes about each item and proceeded to independently place bookmarks to represent cut points for basic, intermediate, and advanced literacy; this first bookmark placement constituted Round 1. On the second day of standard setting, each table received a summary of the Round 1 bookmark placements made by each table member and were provided the medians of the bookmark placements (calculated for each table). Table leaders facilitated discussion among table members about their respective bookmark placements, moving from basic to intermediate to advanced literacy, without asking for consensus. Panelists were given just under two hours to deliberate about differences in their bookmark placements before independently making judgments for Round 2. Throughout the standard setting, staff members, consultants, assistants, and four committee members observed the interactions among the panelists as they discussed the characteristics of the items and their reasons for selecting their bookmark placements. For Round 3, each table again received a summary of the Round 2 bookmark placements made by each table member as well as the medians for the table. In addition, each table received impact data, that is, the proportion of the 1992 population who would have been categorized at the below basic, basic, intermediate, or advanced literacy level based on the table’s median cut points. After discussion of the variability of Round 2 judgments and the impact of their proposed cut points on the percentages of adults who would be placed into each of the four literacy groups, each panelist made his or her final judgments about bookmark placements for the basic, intermediate, and advanced literacy levels. This final set of judgments concluded Round 3. After Round 3, panelists were asked to provide feedback about the 1   The performance-level descriptions used in July are presented in Table 5-2 of the report.

OCR for page 221
Measuring Literacy: Performance Levels for Adults performance-level descriptions by reviewing the items that fell between each of their bookmarks and editing the descriptions accordingly. That is, the items in the booklet up to, but not including, the first bookmark described the basic literacy level. Panelists reviewed these items and revised the descriptions to better fit the items that fell within this level. They were asked to do the same for the intermediate and advanced performance-level descriptions. On the afternoon of the second day, the processes described above were repeated for the second literacy area. Round 1 was completed on the second day; Rounds 2 and 3 were completed on the third day. The standard setting concluded with a group session to obtain feedback from the panelists. Using Different Response Probability Instructions In conjunction with the July standard setting, the committee collected information about the impact of varying the instructions given to panelists with regard to the criteria used to judge the probability that an examinee would answer a question correctly (the response probability). The NALS results were reported in 1992 using a response probability of 80 percent, a level commonly associated with mastery tests. Some researchers have questioned the need for such a strict criterion for an assessment like NALS, for which there are no individual results, and recommend instead using a more moderate response probability level of 67 percent (e.g., Kolstad, 2001). The authors of the bookmark method also recommend a 67 percent response probability level (Mitzel et al., 2001). Because the issue of response probability had received so much attention in relation to NALS results, the committee arranged to collect data from panelists about the impact of using different (50, 67, or 80 percent) response probability values. Specifically, we were interested in evaluating (1) the extent to which panelists understand and can make sense of the concept of response probability level when making judgments about cut scores and (2) the extent to which panelists make different choices when faced with different response probability levels. Panelists were told that they would be given different instructions to use in making their judgments and that they should not discuss the instructions with each other. As described earlier, the panelists were grouped into nine tables of four or five panelists each. Each group was given different instructions and worked with different ordered item booklets. Three tables (approximately 15 panelists) worked with booklets in which the items were ordered with a response probability of 80 percent and received instructions to use 80 percent as the likelihood that the examinee would answer an item correctly. Similarly, three tables used ordered item booklets and instructions consistent with a response probability of 67 percent, and three tables used or-

OCR for page 221
Measuring Literacy: Performance Levels for Adults dered item booklets and instructions consistent with a response probability of 50 percent. Panelists received training in small groups about their assigned response probability instructions. The additional training session gave detailed instructions to panelists on one of three difficulty levels (50, 67, or 80 percent). These specialized instructions are summarized in Background Materials at the end of this appendix. Each table of panelists used the same response probability level for the second content area as they did for the first. Refining the Performance-Level Descriptions The performance-level descriptions used at the July standard setting consisted of overall and subject-specific descriptors for the top four performance levels. In accord with typical bookmark procedures, concrete examples of stimulus materials (e.g., newspaper articles, almanac) or types of tasks (e.g., read a bus schedule, fill out an employment application form) had been intentionally omitted from the performance-level descriptions because including specific examples tends to overly influence panelists’ judgments about the bookmark placements. Omission of specific examples allows the panelists to rely on their own expertise in making judgments. Panelists’ written comments about and edits of the performance levels were reviewed. Many panelists commented about the lack of concrete examples, saying that a few examples would have helped them. Some were concerned that NALS did not have enough items at the upper end of the spectrum for them to confidently make a distinction between intermediate and advanced categories. They also suggested edits, such as adding the modifier “consistently” to the levels higher than below basic, asked for clarification of adjectives such as “dense” versus “commonplace” text and “routine” versus “complex” arithmetic operations. In addition, the panelists raised questions about the scope of the NALS quantitative assessment and the extent to which it was intended to evaluate arithmetic skills versus functional quantitative reasoning. They also pointed out inconsistencies in the wording of the descriptions, moving from one level to the next. The committee used this feedback to rethink and reword the level descriptions in ways that better addressed the prose, document, and quantitative literacy demands suggested by the assessment items. Revised descriptions were used for the September standard-setting session. The following types of changes were made. The introduction to the descriptions was rewritten to include the phrase, “An individual who scores at this level, independently, and in English …,” reflecting the nature of the NALS and NAAL as tests of literacy in English in which examinees complete the test items with minimal or no help from the interviewer or other

OCR for page 221
Measuring Literacy: Performance Levels for Adults family members or individuals. In addition, the subject-area descriptions were revised to better reflect the range of literacy skills needed for the NALS items without specifying the types of NALS items or stimuli used. Four panelists who had participated in the July standard-setting session were invited to review the revised performance-level descriptions prior to the September standard setting, and their feedback was used to further refine the descriptions.2 Panelists’ Evaluation of the Standard Setting At the end of the July standard-setting session, panelists were asked to complete a satisfaction questionnaire (a blank questionnaire is included in Background Materials at the end of this appendix). Almost all of the participants reported that they were either very satisfied (59 percent, n = 20) or satisfied (35 percent, n = 12) with the standard-setting training, while only two participants reported that they were not satisfied with the training they received. Almost three-quarters of the participants (74 percent, n = 25) reported being very satisfied with their table interactions and discussions; roughly a quarter (26 percent, n = 9) reported that they were satisfied with the logistical arrangements. The contributions and guidance of the table leaders were perceived as mainly very satisfactory (53 percent, n = 18) or satisfactory (32 percent, n = 11). Only two participants (6 percent) indicated that their table leaders were not satisfactory. Both of these individuals wrote on their evaluations that their table leaders were overly talkative and did not facilitate discussions among the table members. The majority of comments indicated that participants thought their table leaders were well organized, adept at facilitating discussion, and kept the table members focused on the standard setting tasks. The organization of the standard-setting session was well received: over half of the participants (68 percent, n = 23) were very satisfied and 32 percent (n = 11) reported satisfaction with the session. Participants also reported being satisfied with their work during the standard setting—94 percent of the participants reported that they were either very satisfied (44 percent, n = 15) or satisfied (50 percent, n = 17) with the cut scores decided by their table, indicating a high level of participant confidence in both the process and the product of the standard-setting session. In addition, 85 percent (n = 29) and 12 percent (n = 4) reported that participation in the standard-setting session was very valuable or valuable to them, respectively. 2   The performance-level descriptions used in September are presented in Table 5-3 of the report.

OCR for page 221
Measuring Literacy: Performance Levels for Adults Besides giving feedback on the standard-setting session, panelists were also very helpful in suggesting ways in which the September standard-setting session would benefit from the perspective of those who had just completed the process. For example, the participants reflected a range of adult education areas, such as ABE, GED, and ESL. While the experiences and perspectives of these individuals were useful and appropriate for the standard-setting task, the July participants asked that the committee consider broadening the array of perspectives for the September gathering by including middle school or high school language arts teachers and professionals familiar with human relations, employment testing, or skills profiling. The July participants commented that the table discussions needed these additional perspectives to better conceptualize the range of literacy skills within the performance levels. In addition, the panelists commented that they would have liked to have seen a broader representation of community types (e.g., rural, suburban, urban) reflected in the table discussions because the needs of adult learners and their environments play a factor in program availability and access to various literacy materials represented in NALS. The committee agreed and solicited participation from members of these professional and geographic areas for the September standard setting. RESULTS OF STANDARD SETTING WITH 1992 DATA In an effort to provide results that can be fully understood and replicated, this section provides complete results from the July standard setting reported separately by literacy area. Prose Literacy A complete listing of all judgments made by each panelist who reviewed the prose literacy scale at the July standard-setting session is presented in Tables C-2A, C-2B, and C-2C respectively, for Basic, Intermediate, and Advanced. The information included in the table consists of each participant’s bookmark placement for each round, as well as the corresponding scale score.3 The table number and response probability (rp) level used by each panelist are provided, as well as an indication of whether a 3   The item parameters used for the July standard setting were those available in the public data file. The transformation constants used to convert theta estimates to scaled scores follow—prose: 54.973831 and 284.808948; document: 55.018198 and 279.632461; quantitative: 58.82459 and 284.991949.

OCR for page 221
Measuring Literacy: Performance Levels for Adults given literacy scale was reviewed by the panelist first (i.e., Occasion 1) or second (i.e., Occasion 2). Figure C-1 illustrates the bookmark placement results on the scale score metric by round and table. The top three graphs present the results for Occasion 1 (Tables 1, 4, and 7), and the bottom three graphs show the results for Occasion 2 (Tables 2, 5, and 8). The lines are differentiated by performance level to indicate panelists’ cut score recommendations: the upward-facing triangles (Δ) indicate the cut score each panelist recommended for the basic literacy performance standard, the asterisks (*) represent the intermediate literacy performance standard, and the downward-facing triangles (∇) indicate the advanced literacy performance standard. The median Round 3 placement for the table for each cut score is indicated by a standalone symbol (Δ, *, or ∇) on the right-hand side of each graph. The numbers below each graph represent the scale scores corresponding to the median basic, intermediate, and advanced literacy values for the given table. The graphs in Figure C-1 reflect panelist behavior similar to other, published, bookmark standard-setting sessions (Lewis et al., 1998). That is, as the rounds progress, the variability in bookmark placements tends to decrease, resulting in a relative convergence of bookmark location by the end of the third round. As Figure C-1 illustrates, however, convergence did not always happen, given that bookmark placement reflects individual decisions and biases. Panelists at Tables 1 and 2 used an 80 percent response probability level (rp80); Tables 4 and 5 were assigned an rp level of 67 percent (rp67); and Tables 7 and 8 were instructed to use a 50 percent response probability level (rp50). Across Tables 1, 4, and 7, there was generally more agreement among panelists in the basic and intermediate cut scores at the conclusion of the Round 3, but the final placements of the advanced cut score varied considerably. A somewhat different pattern is seen across Tables 2, 5, and 8. Panelists at Tables 5 and 8 appeared to reach consensus regarding the cut scores for the basic performance level, Table 2 participants achieved consensus on the cut scores for the intermediate level; and Table 5 achieved consensus on the cut score for the advanced level. Round 3 data from the two occasions were combined and descriptive statistics calculated. This information is reported by rp level for the prose literacy scale in Table C-3. Across performance levels and rp levels, the standard errors were lowest with the 67 percent response probability level. Document Literacy Panelists at six of the nine tables reviewed NALS items from the document literacy scale. A complete listing of all judgments made by each pan-

OCR for page 221
Measuring Literacy: Performance Levels for Adults elist who reviewed the document literacy scale at the July standard-setting session is presented in Tables C-4A, C-4B, and C-4C. Figure C-2 shows the bookmark placement results on the scale score metric for each of the three Occasion 1 (top three graphs) and Occasion 2 (bottom three graphs) cut scores by round and table. Panelists at Tables 3 and 1 used rp80, panelists at Tables 6 and 4 used rp67, and panelists at Tables 9 and 7 used rp50. Final bookmark placements for Table 9 are taken from Round 2, due to a data processing in the Round 3 results for that table. As with prose literacy, the variability of bookmark placements decreased as the rounds progressed. At all of the tables, there appeared to be more agreement with regard to the cut scores for the basic and intermediate performance levels than for the advanced level. Although some convergence in the advanced cut scores was observed as the rounds progressed, the Round 3 bookmark placements are quite disparate. Summary statistics for the Occasion 1 and Occasion 2 combined data are presented in Table C-5. Unlike the data for prose literacy, the standard error of the mean for document literacy across rp levels and performance levels was lowest for rp50 and highest for rp80. Quantitative Literacy Panelists at six of the nine tables reviewed NALS items from the quantitative literacy scale. A complete listing of all judgments made by each panelist who reviewed the quantitative literacy scale at the July standard-setting session is presented in Tables C-6A, C-6B, and C-6C. The Occasion 1 (top three graphs) and Occasion 2 (bottom three graphs) bookmark locations and corresponding scale scores reported by each panelist by round and rp level are given in Figure C-3. Panelists at Table 2 and 3 used rp80, panelists at Table 5 and 6 used rp67, and panelists at Tables 8 and 9 used rp50. Overall, panelists tended to approach consensus on the cut scores for the basic and intermediate performance levels, although this was not true for Tables 3 or 5. Considerable disparity was evident in the cut scores for the advanced level, and this variability was maintained across all three rounds. Summary statistics on the combined Occasion 1 and Occasion 2 data are given in Table C-7. The standard error was highest in the basic and advanced performance levels for rp67 and in the intermediate performance level for rp80.

OCR for page 221
Measuring Literacy: Performance Levels for Adults Results from Comparison of Different Response Probability Levels The purpose of using the different response probability instructions was to evaluate the extent to which the different response probability criteria influenced panelists’ judgments about bookmark placements. It would be expected that panelists using the lower probability criteria would place their bookmarks later in the ordered item booklets, and, as the probability criteria increase, the bookmarks would be placed earlier in the booklet. Bookmark placements are converted to scaled scores in two steps. First the item response theory (IRT) model (here, the two-parameter logistic model, or 2PL) is used to calculate the theta value at which an individual would be expected to answer the item correctly at the specified probability level (see equation 3-1 in the technical note to Chapter 3). Then the theta value is transformed to a scale score value using a linear transformation equation. Typically, the IRT model equation estimates the value of theta associated with a 50 percent probability of correctly answering an item. As described in the technical note to Chapter 3, the equation can be solved for different probabilities of a correct response. Thus, when the response probability value is 67, the theta estimate is the value at which one would have 67 percent chance of answering the item correctly. Likewise, when the response probability is 80, the theta estimates the value at which one would have an 80 percent chance of answering the item correctly. For a given item, the theta values will increase as the response probability moves from 50 to 67 to 80; the scaled scores will similarly increase. If panelists apply the different response probabilities correctly, they should shift their bookmark placements in such a way that they compensate exactly for the differences in the way the bookmark placements are translated into thetas and to cut scores. That is, ideally, panelists should compensate for the different response criteria by placing their bookmarks earlier or later in the ordered item booklet. If they are compensating exactly for the different instructions, the theta (and scale score) associated with the bookmark placement should be identical under the three different response probability instructions, even though the bookmark locations would differ. Given these expectations for panelists’ implementation of the response probability criteria, we further examined both the bookmark placements and the resulting scaled cut scores. In the body of the report, we presented the median results for the Round 3 judgments, as it is these judgments that are typically used in determining the final cut scores. Here we examine the Round 1 judgments, as these would be expected to be more independent than those made after group discussions. In addition, we look at the results separately by occasion. That is, as

OCR for page 221
Measuring Literacy: Performance Levels for Adults TABLE C-9 Summary of Round 1 Bookmark Placements and Cut Scores for Document Literacy by Response Probability (RP) Level and Occasion, July 2004 RP Level Basic Intermediate Advanced 0.80 0.67 0.50 0.80 0.67 0.50 0.80 0.67 0.50 Occasion 1 Median bookmark placement 15.0 15.0 20.0 48.0 48.0 46.0 67.0 73.0 65.0 Median cut score 217.5 193.0 182.0 276.0 252.0 221.0 360.5 378.0 279.0 N 4 5 5 4 5 5 4 5 5 Occasions 1 and 2 Median bookmark placement 12.0 12.0 18.0 42.5 45.5 46.0 64.0 70.0 67.0 Median cut score 211.0 188.5 176.0 260.5 244.5 221.0 330.0 341.5 286.0 N 8 10 9 8 10 9 8 10 9

OCR for page 221
Measuring Literacy: Performance Levels for Adults TABLE C-10 Summary of Round 1 Bookmark Placements and Cut Scores for Quantitative Literacy by Response Probability (RP) Level and Occasion, July 2004 RP Level Basic Intermediate Advanced 0.80 0.67 0.50 0.80 0.67 0.50 0.80 0.67 0.50 Occasion 1 Median bookmark placement 19.0 5.0 11.0 30.0 14.0 27.0 41.0 32.0 40.0 Median cut score 287.0 216.0 235.0 349.0 271.0 288.0 421.0 329.0 351.0 N 5 4 5 5 5 5 5 5 5 Occasions 1 and 2 Median bookmark placement 14.0 10.5 11.0 27.0 25.0 26.5 39.0 40.0 39.0 Median cut score 283.0 271.0 235.0 342.0 307.0 286.0 389.0 371.5 323.0 N 9 9 10 9 10 10 9 10 10

OCR for page 221
Measuring Literacy: Performance Levels for Adults TABLE C-11 Regression Results for Bookmark Placements for Prose Literacy, July 2004 Number of Panelists Occasion 1 Occasions 1 and 2 13 28 Basic RP50 1.65a (2.06)b 0.54 (1.25) RP80 2.15 (2.06) 1.21 (1.25) Constant 7.60 (1.37) 6.90 (0.86) R2 0.11 0.04 Intermediate RP50 1.80 (1.86) 2.51 (2.18) RP80 –1.20 (1.86) –0.49 (2.18) Constant 22.2 (1.24) 20.60 (1.50) R2 0.19 0.07 Advanced RP50 2.40 (2.64) 1.54 (2.41) RP80 –2.6 (2.64) –3.34 (2.41) Constant 35.6 (1.76) 33.9 (1.66) R2 0.24 0.14 aRegression coefficient. bStandard error.

OCR for page 221
Measuring Literacy: Performance Levels for Adults TABLE C-12 Regression Results for Bookmark Placements for Document Literacy, July 2004 Number of Panelists Occasion 1 Occasions 1 and 2 13 27 Basic RP50 1.60a (3.40)b 4.27* (2.13) RP80 –0.45 (3.61) 0.60 (2.20) Constant 16.20 (2.41) 13.40 (1.46) R2 0.03 0.16 Intermediate RP50 –4.40 (4.58) –0.08 (3.59) RP80 –2.05 (4.86) –3.30 (3.70) Constant 48.8 (3.23) 45.3 (2.47) R2 0.08 0.04 Advanced RP50 –7.0* (2.72) –0.93 (3.20) RP80 –5.15 (2.89) –6.35* (3.31) Constant 72.4 (1.92) 67.6 (2.20) R2 0.39 0.15 aRegression coefficient. bStandard error. *p < .10.

OCR for page 221
Measuring Literacy: Performance Levels for Adults TABLE C-13 Regression Results for Bookmark Placements for Quantitative Literacy, July 2004 Number of Panelists Occasion 1 Occasions 1 and 2 14 29 Basic RP50 6.60a** (2.10)b 1.40 (1.72) RP80 11.40** (2.16) 2.97 (1.72) Constant 4.20 (1.49) 9.70 (1.22) R2 0.79 0.07 Intermediate RP50 10.40* (2.87) 4.1 (3.20) RP80 12.20** (2.95) 3.37 (3.20) Constant 17.00 (2.03) 23.3 (2.27) R2 0.58 0.08 Advanced RP50 6.60* (2.05) 1.70 (2.59) RP80 7.80* (2.11) 1.40 (2.59) Constant 33.00 (1.45) 37.6 (1.83) R2 0.46 0.03 aRegression coefficient bStandard error *p < .10 **p < .01

OCR for page 221
Measuring Literacy: Performance Levels for Adults TABLE C-14 Regression Results for Cut Scores for Prose Literacy, July 2004 Number of Panelists Occasion 1 Occasions 1 and 2 13 28 Basic RP50 –11.65a (16.37)b –18.99* (8.91) RP80 33.10* (14.62) 25.70* (9.46) Constant 220.40 (10.22) 217.10 (5.85) R2 0.45 0.47 Intermediate RP50 –12.65 (11.39) –11.03 (11.54) RP80 14.60 (10.80) 14.52 (10.25) Constant 287.40 (7.27) 281.70 (8.99) R2 0.35 0.18 Advanced RP50 –5.95 (21.21) –19.00 (19.39) RP80 2.80 (28.12) –1.78 (20.62) Constant 371.20 (18.24) 361.00 (15.51) R2 0.01 0.04 aRegression coefficient. bStandard error. *p < .10.

OCR for page 221
Measuring Literacy: Performance Levels for Adults TABLE C-15 Regression Results for Cut Scores for Document Literacy, July 2004 Number of Panelists Occasion 1 Occasions 1 and 2 14 28 Basic RP50 –21.60a* (7.25)b –16.80** (4.67) RP80 22.15** (7.28) 23.20** (4.57) Constant 197.60 (5.04) 191.80 (3.10) R2 0.76 0.74 Intermediate RP50 –40.40** (10.69) –30.17** (7.17) RP80 15.85 (10.12) 14.38* (7.92) Constant 256.40 (9.44) 248.50 (6.56) R2 0.76 0.61 Advanced RP50 –78.00* (26.52) –31.72 (20.81) RP80 –11.45 (14.17) –2.50 (19.56) Constant 371.20 (4.25) 333.50 (13.33) R2 0.52 0.11 aRegression coefficient. bStandard error. *p < .10. **p < .01.

OCR for page 221
Measuring Literacy: Performance Levels for Adults TABLE C-16 Regression Results for Cut Scores for Quantitative Literacy, July 2004 Number of Panelists Occasion 1 Occasions 1 and 2 14 29 Basic RP50 22.20a** (7.37)b –8.20 (11.67) RP80 74.20** (3.66) 38.47** (10.42) Constant 215.20 (1.07) 243.20 (9.36) R2 0.92 0.46 Intermediate RP50 9.40 (10.51) –11.80 (9.92) RP80 68.20** (10.45) 36.40* (11.85) Constant 279.20 (7.02) 300.60 (8.85) R2 0.80 0.47 Advanced RP50 7.80 (18.41) –30.80* (14.97) RP80 74.80** (17.12) 31.78* (16.64) Constant 338.80 (13.63) 369.00 (13.01) R2 0.65 0.40 aRegression coefficient. bStandard error. *p < .10. **p < .01.

OCR for page 221
Measuring Literacy: Performance Levels for Adults FIGURE C-1 Prose literacy cut scores by round for participants at each table, July 2004. Symbols indicate basic (Δ), intermediate (*), and advanced (∇) cut-score judgments. Round 3 medians are depicted by standalone symbols.

OCR for page 221
Measuring Literacy: Performance Levels for Adults FIGURE C-2 Document literacy cut scores by round for participants at each table, July 2004. Symbols indicate basic (Δ), intermediate (*), and advanced (∇) cut-score judgments. Round 3 medians are depicted by standalone symbols.

OCR for page 221
Measuring Literacy: Performance Levels for Adults FIGURE C-3 Quantitative literacy cut scores by round for participants at each table, July 2004. Symbols indicate basic (Δ), intermediate (*), and advanced (∇) cut-score judgments. Round 3 medians are depicted by standalone symbols.