3
Approaches to Assessing Health Literacy

WHAT IS HEALTH LITERACY AND HOW DO WE MEASURE IT?

Lauren McCormack, Ph.D., M.S.P.H.

RTI International


RTI International is developing and testing a new measure of health literacy. The objective of this R01 project funded by the National Institutes of Health (NIH) is to create a publicly available health literacy instrument that can be used for population-based surveillance and for measuring an individual’s health literacy in intervention and research studies. In addition to the research team, there is an external panel advising the project. Specific project tasks include developing a conceptual framework, developing health literacy items, cognitively testing these items, pilot testing the items in a survey, and conducting psychometric analyses of the pilot data.

As discussed previously, existing measures of health literacy have limitations. For example, a major limitation of the Test of Functional Health Literacy in Adults (TOFHLA), the Rapid Estimate of Adult Literacy in Medicine (REALM), the Wide Range Achievement Test (WRAT), and the Ask-Me-3 is that these instruments largely measure reading ability or print literacy. The National Assessment of Adult Literacy (NAAL), as has been discussed, is not publicly available, and there is uncertainty about when the next round will be fielded.

The project team began by reviewing existing definitions of health literacy. The Office of Disease Prevention and Health Promotion defines



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 29
3 Approaches to Assessing Health Literacy WHAT IS HEALTH LITERACy AND HOW DO WE MEASuRE IT? Lauren McCormack, Ph.D., M.S.P.H. RTI International RTI International is developing and testing a new measure of health literacy. The objective of this R01 project funded by the National Institutes of Health (NIH) is to create a publicly available health literacy instrument that can be used for population-based surveillance and for measuring an individual’s health literacy in intervention and research studies. In addition to the research team, there is an external panel advising the project. Specific project tasks include developing a conceptual framework, developing health literacy items, cognitively testing these items, pilot testing the items in a survey, and conducting psychometric analyses of the pilot data. As discussed previously, existing measures of health literacy have limitations. For example, a major limitation of the Test of Functional Health Literacy in Adults (TOFHLA), the Rapid Estimate of Adult Lit - eracy in Medicine (REALM), the Wide Range Achievement Test (WRAT), and the Ask-Me-3 is that these instruments largely measure reading abil- ity or print literacy. The National Assessment of Adult Literacy (NAAL), as has been discussed, is not publicly available, and there is uncertainty about when the next round will be fielded. The project team began by reviewing existing definitions of health literacy. The Office of Disease Prevention and Health Promotion defines 

OCR for page 29
0 MeASUReS oF HeALTH LITeRACY health literacy as not simply a function of basic literacy skills, but as “dependent on individual and system factors, including communication skills of lay persons and professionals, lay and professional knowledge of health topics, culture, the demands of the healthcare and public health systems, and the demands of the situation/context” (http://www.health. gov/communication/literacy/quickguide/ factsbasic.htm). “Health liter- acy varies by context and setting and is not necessarily related to years of education or general reading ability,” according to the National Network of Libraries of Medicine (http://nnlm.gov/outreach/consumer/hlthlit. html). The Institute of Medicine (IOM) states that, “Even well-educated people with strong reading and writing skills may have trouble com- prehending a medical form or doctor’s instructions regarding a drug or procedure” (2004). Thus, the literature review supports an increas- ing recognition of the importance of context and setting when assessing health literacy. The project team adopted a slightly modified version of the Ratzan and Parker (2000) definition. Conceptual Framework and Skills-based Approach to Measurement The next step in the project was to develop a conceptual framework (see Figure 3-1). As Pleasant said earlier, a conceptual framework is critical as a foundation of measurement. An important component of this framework is the feedback loop from health-related outcomes back into skills; people learn from their experiences, and that affects their skills for the future. There is an increasing call in health care for consumer activation, con- sumer empowerment, and consumer involvement. Under these circum - stances a skills-based approach to measuring health literacy is warranted. Therefore, the approach under development will include assessments of people’s ability to use different types of health information to make informed decisions as well as the skills needed across the life course in periods of health and periods of illness. Issues addressed range from dis- ease prevention to treatment and self-management. The assessment will be based on the U.S. health care system, which means that the measure - ment process reflects current health insurance issues and care provided in public and private systems. One challenge in creating a skills-based approach in which data are collected via a computer is keeping up with technological advancements and changes in health-related materials that are used in the measurement process. The measures will cover several health literacy domains, includ- ing print (both prose and document), numeracy skills, communication (including listening, speaking, and negotiating), and information seeking or navigation. A hierarchical approach was taken to determine the mea- sures. First the skill or task was identified. Second, stimuli that enabled

OCR for page 29
DEMOGRAPHICS HEALTH LITERACY SKILLS Print Literacy (reading, writing) Numeracy Skills Communication (listening, speaking, negotiating) RESOURCES Information Seeking (navigation) Community Family Culture HEALTH INDIVIDUAL/PATIENT RELATED UNDERSTANDING HEALTH RELATED MODERATORS OUTCOMES OF STIMULUS STIMULUS Media Health Care Provider Health Care CAPABILITIES System PRIOR KNOWLEDGE CONSTRUCTS DEMOGRAPHICS: age, race, ethnicity, income, gender HEALTH RELATED STIMULUS: educational brochure, conversation with doctor STIMULUS CONSIDERATIONS RESOURCES: occupation, employment, income, social s upport, culture, language, education, literacy Health literacy demand: complexity and difficulty of stimulus, one vs. many stimuli PRIOR KNOWLEDGE: vocabulary, conceptual know ledge of health/health care Channel: print, mass media CAPABILITIES: vision, hearing, verbal ability, memory, cognitive/reasoning/processing Message: health promoting vs. disease preventing, fear appeals, narrative vs. non-narrative INDIVIDUAL/PATIENT L EVEL MODERATORS: motivation, self-efficacy, perception of outcome/behavior, attitude toward Messenger: relationship of messenger to recipient, credibility of/trust in messenger/source, outcome or behavior, coping strategies, social support, emotions, behavioral intention, trust in information/source/stimulus, communication skills of messenger decision making skills, fatalism, knowledge, decision making) FIguRE 3-1 Conceptual framework for individual health literacy. SOURCE: McCormack, 2009. 

OCR for page 29
 MeASUReS oF HeALTH LITeRACY measurements of the skill or task were selected. Finally, the mode of administration was chosen. Although some of these questions could be conducted over the telephone or in person, a web-based approach is the preferred mode at this time. The following criteria were used to identify the skills to include in the measures: • Understanding health-related concepts and terms (in writing and verbally); • Interpreting tables, charts, symbols, maps, and other visuals; • Making inferences based on available data; • Applying information to new situations; and • Using arithmetic manipulations. Criteria used for selection of stimuli included • Sufficiently related to the health of the public; • Widely applicable, balanced content; • Accessible to many subgroups (gender neutral, culturally sensitive); • Clinically important and not controversial; • Appropriate length of content; • Mixture of public- and private-sector materials; • Likely to stand the test of time; • Variety of formats/channels; • Wide range of difficulty; and • Has face validity. In developing criteria for the survey items themselves, the project team determined that prior knowledge should not be required to answer the questions. Another criterion is that there must be only one correct response, but there also have to be reasonable distractors (alternatives) that are neither too obvious nor too difficult. The questions must be independent of each other, that is, respondents should not have to get the first question correct in order to get the second question correct. Finally, the questions must include a range of difficulty and must cognitively test well. Survey Items The following are examples of stimuli and survey items that the project team is considering for the assessment. Final decisions about the stimuli and items will be based on the pilot work and assuming approvals are granted from the organization that created the stimuli. One possible stimu- lus is “Signs of a Stroke” (Figure 3-2). A few survey questions are associated with each stimulus.

OCR for page 29
 APPRoACHeS To ASSeSSInG HeALTH LITeRACY EASY: Which of the following is NOT a sign of a stroke? a) Difficulty breathing (correct response) b ) Blurred vision c) Bad headache d) Numbness on one side MEDIUM: W hich of the fo llowing is a true statement? a) Someone who is having a stroke will have all of the signs. b) Someone who is having a stroke may have only two or three signs. (correct response) c) Everyone who has a stroke dies if they are not treated right away. Figure 3-2 Signs of a stroke. FIguRE 3-2 SOURCE: McCormack, 2009. Other items in the survey require reading an article to obtain infor- mation and then answering questions based on information provided in the article. Other questions are based on short videos such as the public service announcement The Faces of Influenza, sponsored by the American Lung Association and posted on YouTube.com. There are also questions about symbols. For example, the question appearing in Figure 3-3 is about medication adherence. Bill is at the beach and taking this medicine. He should a) Continue his outdoor activities without any changes b) Not take his medicine for now c) Take the medicine before going outdoors in the morning d) Sit in the shade if he is going to be outdoors (correct response) FIguRE 3-3 Caution symbols on medication bottles. SOURCE: McCormack, 2009. Figure 3-3

OCR for page 29
4 MeASUReS oF HeALTH LITeRACY Conclusion A number of issues and challenges remain as health literacy measures are developed, including identifying skills that can be measured, select - ing appropriate stimuli and items, and assessing the trade-offs associated with different modes of administration. Another issue is how emerging technologies will allow improvement in measurement of health literacy, especially oral literacy. Additional questions include, What are the advan- tages and disadvantages of using real-world stimuli versus stimuli devel- oped for assessments? On which national surveys would health literacy items and scales best fit? How do we deal with the need for stimuli to be updated and/or changed over time? DISCuSSION Moderator: George Isham, M.D., M.S. HealthPartners One audience participant asked whether there is enough knowledge and new technology today (e.g., with the personal health record and the new health initiative measures) that one could develop a measure, be it of knowledge, skills, or function, that would take 5 minutes and that could be used to rapidly move the field forward. McCormack said many in the field would like to have a 5-minute short form instrument to measure health literacy, and one could be created eventually. A first step is creat - ing a longer form of the instrument and using psychometric and other analyses to determine which items reflect the core of the instrument, then eliminating items that contribute less. One possible model for measuring health literacy is to take an approach like the Patient Reported Outcomes Measurement Information System (PROMIS)1 for quality-of-care mea- surement. PROMIS uses a large bank of items that are rotated over time but still measure the same construct. 1 PROMIS “is an NIH Roadmap network project intended to improve the reliability, validity, and precision of PROs and to provide definitive new instruments that will exceed the capabilities of classic instruments and enable improved outcome measurement for clinical research across all NIH institutes” (http://aramis.stanford.edu/downloads/ 2005FriesCERS53.pdf). Accessed April 9, 2009.

OCR for page 29
 APPRoACHeS To ASSeSSInG HeALTH LITeRACY REFININg AND STANDARIZINg HEALTH LITERACy ASSESSMENT: ENgLISH AND SPANISH ITEM bANkS elizabeth Hahn, M.A. northwestern Uniersity The bilingual assessment of health literacy project at Northwestern is funded by the National Heart, Lung, and Blood Institute of the NIH. The project has four goals: 1. Develop English- and Spanish-language item banks for reading- related health literacy skills; 2. Evaluate the feasibility, validity, and acceptability of computer- based methods for assessment of health literacy; 3. Develop computer-adaptive testing (CAT) of health literacy in clinical settings; and 4. Evaluate the associations among health literacy, sociodemographic/ clinical characteristics, and health outcomes in primary care patients. There is a continuum in health literacy that goes from low health lit - eracy to high health literacy (Figure 3-4). The project intends to develop items that span the continuum and to make sure that for each English item on the continuum, there is a corresponding item in Spanish that sits at the same place on the continuum. To have equivalence across English and Spanish, the items must mean the same thing. There will be a bank of questions that identifies the underlying trait to be measured. The definition of the trait and the meaning of each item will be the same across all participant characteristics. If that were not the case, differences due to measurement bias could be interpreted incorrectly as real differences between groups. Item Response Theory (IRT) Item Banks Item Locations, English Low Literacy High Literacy Item Locations, Spanish FIguRE 3-4 Item response theory. Figure 3-4 SOURCE: Hahn, 2009.

OCR for page 29
 MeASUReS oF HeALTH LITeRACY Item bank A well-constructed item bank will enable development of computer- adaptive tests or creation of short forms of the test. In other words, indi - viduals could answer different questions in the item bank but, because it is known exactly where on the continuum each question is located, it will still be possible to estimate a health literacy score for each individual with good precision. The Talking Touchscreen2 (la Pantalla Parlanchina) will be adapted and used, providing those with low literacy an opportu - nity to self-administer questions by having some text on the screen read out loud. The definition of health literacy used in this project has essentially two parts: capacity and application. First, an individual must have the capacity to process and understand health-related information. He or she must then be able to apply that information in the management of her or his own health. The capacity to obtain information, which is part of other definitions of health literacy discussed earlier, is a navigation skill that is not included in this health literacy tool. Instead, the focus is on comprehending and interpreting information provided and understand- ing what an appropriate health care decision based on that information should be. Whether the patient actually implements an appropriate health care decision and related behavior is also beyond the capability of this assessment tool. The following are examples of items in this tool. All items are in English for this presentation, but there are comparable items in Spanish. Figure 3-5 shows a prose item. There is a short paragraph with text drawn from real-world documents. This is followed by a sentence with a missing word. Options are then given for the respondent to choose what to use to fill in the blank. A second type of item included in this assessment tool is a document item. There is a stimulus (in Figure 3-6 a prescription label is the stimu- lus), followed by a question that asks about the stimulus. This particular item also has sound (the respondent would click on the “talking head” in the figure) so that information can be relayed orally. The third type of item (Figure 3-7) involves a quantitative or numer- acy skill. Again, the respondent can click to have the information deliv- ered orally. All of the items have four response choices with only one correct answer. 2 “The TT [Talking Touchscreen] is a practical, user-friendly data acquisition method that provides greater opportunities to measure self-reported outcomes in patients with a range of literacy skills” (Hahn et al., 2004).

OCR for page 29
 APPRoACHeS To ASSeSSInG HeALTH LITeRACY FIguRE 3-5 Prose item. SOURCE: Hahn, 2009. Figure 3-5 fixed image FIguRE 3-6 Medications for Mr. Beta. SOURCE: Hahn, 2009. Figure 3-6 fixed image

OCR for page 29
 MeASUReS oF HeALTH LITeRACY FIguRE 3-7 Sample body mass index chart. SOURCE: Hahn, 2009. Figure 3-7 fixed images Item Testing All of the items were pilot tested with 97 English-speaking partici- pants and 134 Spanish speakers. The characteristics of the pilot test par- ticipants can be seen in Table 3-1. Most of the testing was done with paper and pencil, but the printed paper looked just like the Talking Touchscreen view will look when those components are completed. There were also research assistants present who could read the questions out loud for participants. Cognitive inter- viewing was conducted with some participants, who were shown the different types of items and then asked to describe how they would go about answering the questions. The participants were recruited mainly in primary care clinics, which are also where the ultimate calibration testing is being conducted. To obtain sufficient numbers for the pilot test, some testing was conducted at community-based organizations that provide general education develop - ment (GED), literacy tutoring, or job training. The pilot test showed that nearly all (>90 percent English, 100 percent Spanish) correctly described the steps needed to answer each type of ques-

OCR for page 29
 APPRoACHeS To ASSeSSInG HeALTH LITeRACY TAbLE 3-1 Characteristics of Pilot Test Participants English Speaking (n = 97) Spanish Speaking (n = 134) % Female 65% 75% Mean age 44.0 38.7 Hispanic ethnicity 91% 99% Race African American 60% — White 7% 22% Other 33% 78% Education < High school 43% 53% High school/GED 31% 25% College 26% 22% Not available (n = 27) — Method Paper and pencil 74% 100% Talking Touchscreen 26% — SOURCE: Hahn, 2009. tion. Participants were also asked if they felt anxious, nervous, or uncom - fortable completing this health literacy test. Only one English-speaking participant and three Spanish-speaking participants were uncomfortable or anxious. Once the participants completed the computer-based test, cognitive interviews were conducted with 25 English-speaking participants. Most reported that the test was easy to use and commented favorably on the screen design and the availability of audio. Some evidence shows that even people with high literacy skills found comprehension was aided with sound as well as the visual prompt. Participants also commented favorably on the items, even when acknowledging that some of them were difficult to answer. A large number of items are needed for a good bank of items. Ulti- mately, people will answer only a small number of items, but the pilot tested 98 English items and 127 Spanish items. Some items were elimi- nated, such as those that everyone completed correctly. Such items are not useful for measurement. The items that were left have a range of difficulty. A small number of items are at the easy and difficult ends of the range, and the bulk of the items are in the middle. A 10-item short form was developed for the pilot test and is being used in other ongoing projects. Calibration testing is under way for the final set of 90 English items and 90 Spanish items. Those items are being tested with 600 English speakers and 600 Spanish speakers who are pri- mary care patients. The analysis plan is to accomplish the following:

OCR for page 29
0 MeASUReS oF HeALTH LITeRACY sure those skills by standardized reading tests. The focus has been on understanding information. As a result, interventions primarily have been aimed at making information easier to understand by reducing the cogni - tive demand. Such work is important and must continue, but it does have its limita- tions. What has been learned from intervention studies is that improved information does improve knowledge for readers with both higher and lower reading skills. However, skilled and unskilled readers alike still struggle to use their acquired knowledge. Reading and understanding information are important parts of functional health literacy, but they offer an incomplete picture and they are insufficient to promote appropri- ate use of health services, good self-care, and improved health. The problem with focusing on academic skills and information is that, like money, one needs it. But what one really needs is not the money itself, but what the money enables one to do. Similarly, it is not really the information that patients need for health, it is what the information enables them to do and how it enables them to function, and that is the function in health literacy. A number of literacy scholars characterize types of levels or layers of literacy. Donald Nutbeam (2000, 2008) applied the work of Freebody and Luke (1990) to characterize three types of health literacy. One type is functional health literacy, defined as reading and writing associated with tasks. In this usage it is associated with literacy tasks—read the list of words, then pass the comprehension test. This is the stage of the field in its measurement of health literacy. Nutbeam also refers to functional health literacy as fundamental or basic literacy that is associated with everyday tasks. Interactive literacy is another type of literacy. It requires social skills such as listening and speaking to complete more complicated interactive tasks. Such tasks might be making an appointment, getting to the appoint - ment, describing symptoms, and listening to treatment instructions. This type of health literacy is analogous to oral health literacy. Critical reflection is the third and higher level literacy that is needed to manage one’s health. As an example, a mother goes to the pediatrician and hears that her baby should sleep on his back to avoid Sudden Infant Death Syndrome. She hears from her grandmother that the baby should sleep on his stomach to avoid aspiration. She needs critical literacy, reflec- tive literacy, to differentiate the sources of information to reconcile the conflicting advice, to manage the power differentials, to control the sleep position for her son, and to thereby manage his health. The current conceptualization and measures of health literacy miss much of this deeper meaning and purpose of literacy for health. What does this conceptualization have to do with measurement?

OCR for page 29
 APPRoACHeS To ASSeSSInG HeALTH LITeRACY One might infer from the use of the term “functional” in this model that interactive health literacy and reflective health literacy are not functional. That is not the case. It is possible to extend the idea of function to all three types of health literacy. Functional health literacy, then, becomes a con - cept that describes the practical application of a wide range of cognitive and noncognitive skills in real life, rather than a single literacy skill in a clinical setting. Functional health literacy is the outcome of intervention rather than the independent variable. It captures how people use literacy for health as patients and also as family members, workers, and citizens. It captures social capital. Measurement How can one measure the function in functional health literacy? A good example of promoting many aspects of family functioning can be found in the work of public health nurses in maternal and child health home visitation programs. These visiting nurses link disadvantaged par- ents to health care services and community resources. They provide social support, practical assistance, information, and health education. Many of these programs use an instrument called the Life Skills Progression TM (Wollesen and Peifer, 2006) to monitor parents’ progress toward higher levels of functioning. To measure functional health literacy, two scales were derived from the Life Skills Progression instrument: a Functional Healthcare Literacy Scale (Figure 3-9) and a Functional Selfcare Literacy Scale (Figure 3-10). The Functional Healthcare Literacy Scale rates parents’ use of health information and services for both parent and child. Each of the items is a Likert scale that identifies behaviors, practices, and characteristics that indicate progressive levels of function that range from dysfunction to optimal functioning. Scores greater than 4 indicate adequate to optimal functioning. One might think of this scale as a map and the items as path - ways toward optimal functioning in a health care system. Both of these assessments function in the same manner. To monitor progress, a home visitor completes the instrument on a parent at intake, every 6 months, and at end of service. The comparison of these sequential measures allows one to track progress over time and to see points of regres- sion. The data are immediately available for intervention planning. The elegance in measuring function is that it provides for solutions along with the identification of problems. One can choose to intervene on a need, which would be indicated by a low score on the left in Figure 3-10. One can also choose to intervene by building on a strength, which is indi - cated by a high score on the right. Subsequent measurement allows one to see the impact of the interventions.

OCR for page 29
 MeASUReS oF HeALTH LITeRACY FIguRE 3-9 Functional Healthcare Literacy Scale. SOURCE: Wollesen and Peifer, 2006. Reprinted by permission from Paul H. Brookes Publishing Co., Inc. Figure 3-9 fixed images FIguRE 3-10 Functional Selfcare Literacy Scale. SOURCE: Wollesen and Peifer, 2006. Reprinted by permission from Paul H. Brookes Publishing Co., Inc. Figure 3-10 fixed image

OCR for page 29
 APPRoACHeS To ASSeSSInG HeALTH LITeRACY These scales demonstrated good reliability. Validity testing is under way. Results This project was a 2-year, quasi-experimental, multicohort interven - tion study with multiple waves of measurement. The total database from seven home visitation programs has 2,532 parents. The data below are on about 1,800 of those parents. One can see in Figure 3-11 that the inter- vention worked quickly in the first six months. Parents demonstrated statistically significant linear stepwise progress over time, regardless of their reading level. Conclusion Measuring function is important in assessments of health literacy, Smith said, because measuring function captures the impact of efforts to reduce the risk of low literacy skills as well as the efforts to promote functioning directly. It allows the integration of the social determinants of health, it guides interventions, it informs practice, and it is patient centered. The Life Skills Progression method presented here could be N = 1,808 Time in service = 36 mo P < 0.001 Change in Health Care Literacy and Personal Care Literacy Over Time with Home Visitation FIguRE 3-11 Home visitation promotes parental functional health literacy. SOURCE: Smith, 2009. Figure 3-11, fixed image, type above and below is editable

OCR for page 29
4 MeASUReS oF HeALTH LITeRACY adapted for clinical use, particularly for adults with chronic conditions that require frequent visits, clinical encounters, and significant self-care. Is the instrument clinically feasible? It takes an experienced user about 5 minutes to complete both of the scales, and the data are immedi - ately available for intervention planning. There is a limitation in that the assessments were implemented in home visitation programs, a hallmark of which is that the visitor and the family build a relationship over time. Therefore, the degree to which the clinical practice environment limits relationship development could affect use of the instrument. In conclusion, Smith said, focus on function. HEALTH LITERACy AND CANCER PREVENTION: DO PEOPLE uNDERSTAND WHAT THEy HEAR? Kathleen Mazor, ed.D. Uniersity of Massachusetts Medical School The focus of this R01-funded project is on understanding spoken communication. The project team was multidisciplinary, and the research was carried out within the Cancer Research Network (CRN). The CRN is a consortium of 14 health plans around the country that cover approxi- mately 10 million enrollees (about 4 percent of the population). Not much attention has been paid to oral communication in the health literacy field. It is often said that if people cannot read, one needs to speak with them or let them listen to an audio version of the information. But do people actually understand even if they hear information? It is not just a question of hearing the words—the listener must know what the words mean and the context within which the words are spoken and be able to act on the information provided. Better measurement of oral communica - tion is important in improving health literacy. The project has three aims, although this presentation will discuss only the first aim, which is to develop and validate a psychometrically sound test of oral health literacy. The project also aims to investigate the relationship between oral health literacy and cancer prevention behaviors by comparing scores from the instrument with actual health behaviors. The third aim is to develop and test recommendations for improving oral communication about cancer prevention and screening. Measurement The first step in developing the measures for assessing oral health literacy was to specify the test blueprint. Because the assessment was administered in the CRN, the focus was on common cancers (breast,

OCR for page 29
 APPRoACHeS To ASSeSSInG HeALTH LITeRACY cervical, colorectal, general, lung, prostate, and skin), cancer prevention, and screening. Not included were factors such as diagnosis, treatment, follow-up, and survival. The blueprint also specified the context within which messages are received (e.g., media or clinical), the style of the com - munication (i.e., narrative, statistical or numeric, factual), the purpose of the communication (i.e., instruction, information, or query), and the content of the information (i.e., prevention, screening methods, or risk factors). The next step was to collect and examine messages about cancer that one might receive from different media, including television, radio, the Internet, and patient education materials. Of great importance was the need to include only those clips that contained accurate information. Fur- thermore, a variety of clips that represented the kinds of thing one might encounter in everyday life were included. The selected clips of oral communication varied in content. Some clips showed a person describing his or her cancer experience, or the experience of quitting smoking. These personal stories were identified as narraties. Another set of clips presented factual information such as the type of cancer or the stage of the cancer. Such information can be delivered in three dif- ferent ways: (1) One can simply provide information, (2) one can poten- tially provide information and then ask something about it, or (3) one can give someone an instruction with the intention that the person take action related to that information. The third step was to develop some clinical vignettes. To construct such vignettes, physicians agreed to participate in role playing, which would be audiotaped and then transcribed. A professional writer helped create the scripts, which clinicians then reviewed and revised. After that, the project team conducted its own review and revisions. The project team then produced videos of those vignettes. The next step was to construct the items for the test. Unfortunately, while there is literature on different approaches to use in developing items, there is not a great deal of literature on how to measure compre - hension. The approach the team chose to use is the sentence verification technique (SVT).3 This is a method that is used to examine comprehen- sion of text messages. The first step is to select a portion of the transcript that contains what one is trying to measure. Next, the sentence(s) is paraphrased so that the wording is changed but the meaning remains 3 “The Sentence Verification Technique (SVT) is a procedure that non-psychometricians can use to develop reading and listening comprehension tests that can be based on a wide variety of text materials” (http://www.readingsuccesslab.com/publications/ Svt%20Review%20PDF%20version.pdf). Accessed April 12, 2009.

OCR for page 29
 MeASUReS oF HeALTH LITeRACY constant. The information is also rewritten so that the wording is similar to the original sentence(s) but with a different meaning. For example, an original sentence said “overall HPV [human papillomavirus] prevalence among females in the United States, ages 14 to 59 years of age, was 26.8 percent, and that means one in four women are infected with HPV.” When the original material was para - phrased, the result was written as, “A quarter of women ages 14 to 59 are infected with HPV.” When the material was rewritten for a meaning change, the sentence read, “One in four women in the United States are infected with cervical cancer,” which is not true because only certain HPV strains develop into cervical cancer. A respondent would hear the original statement and then hear either the paraphrase or the changed-meaning statement. Then the respondent would be asked, “Is the meaning of the statement about the same as the content of the original sample, or is it different?” Testing Pilot testing is currently under way. The test is administered using a touchscreen laptop. No reading is required; everything is spoken. Instruc- tions are given at the beginning, and the test takes about an hour. Cur- rently, the test has 16 videos and 66 questions. It is in English only, which is a limitation imposed because of resources available. Participant feedback on the test is that it is user friendly, even for those unfamiliar with a computer; it is engaging and informative; it has clear instructions; and participants are not fatigued at the conclusion of the test. Once pilot testing is completed, the items will be revised and the test will then be administered to about 1,000 adults at four sites. Conclusion Results to date have shown that the comprehension of spoken mea- sures is variable, Mazor said. Measuring comprehension of spoken messages is challenging because many factors affect comprehension and all of those factors cannot be evaluated fully in a single study. DISCuSSION Moderator: Cindy Brach, M.P.P. Agency for Healthcare Research and Quality One participant said it appears that what both the assessment of func- tional health literacy described by Smith and the assessment of oral health

OCR for page 29
 APPRoACHeS To ASSeSSInG HeALTH LITeRACY literacy described by Mazor are attempting to do is to develop more authentic health literacy measures. For the future, in terms of functional health literacy, what are some of the factors that may predict whether people score high or low on a functional scale? Smith said that major predictors are likely to be self-efficacy, confidence, and social support. Sometimes having a child will galvanize one’s motivation and interest, and parents become very ready to learn and to change. Another participant commented that Smith’s measure has not focused on prediction, but rather on intervention and how one moves forward with that. It is very exciting to think about obtaining information that one can use to intervene and improve health outcomes. One participant stated that in terms of what had been presented as measures for health literacy, what was missing was a focus on measure- ment specifically related to either parents or children. The way in which health literacy is measured in adults is very different from what parents understand about taking care of their children’s health. Furthermore, measuring health literacy in children, from young children through ado- lescence, is exceedingly complex. The main social support for a child is the parent or caregiver. But as the child moves from childhood through adolescence to adulthood, there is dynamic change in whose knowledge and whose management determines health actions. Smith responded that the functional health literacy measure is designed specifically to address literacy in parents and how it affects children’s health. Ratzan suggested that a framework for health literacy could follow a life course determinant model. One of the things such a model would do is address the issue raised earlier about the lack of mea - sures of parent and child health literacy. One participant referenced the levels of intervention in the health care system discussed in the report Crossing the Quality Chasm (IOM, 2001). The discussion describes opportunities to intervene at different levels, such as at the level of the individual patient, the team, the organizational level, and the specific context or environment. These different approaches to measuring health literacy appear to be moving back and forth among these levels. At some point, it might be helpful to develop a table or graph that sets out the domains of activity and organizes the measures at dif- ferent levels within these domains. Developing such a table might help clarify which factors contribute to problems in health literacy at different levels. On the other hand, such a table might help determine which inter- ventions are likely to work at different levels. Smith responded that the Life Skills Progression Instrument and the functional health literacy measure derived from it are used at all levels. The data are rich with information for the individual level, the particular practice level, and the organizational level. The assessment looked at

OCR for page 29
 MeASUReS oF HeALTH LITeRACY seven different programs with a number of visitors in each one. Analysis of differences among sites is ongoing. Not every site achieves the same progress. Differences in program emphasis and differences in individual visitor practices create different levels of progress for the families. One participant said he appreciated the focus on functionality, which is important to incorporate into the testing of health literacy. Also impor- tant are constructs from behavioral science, such as the self-efficacy con - struct used by Smith. Were there other constructs from behavioral theories that could be suitably incorporated into measurement of functional health literacy? Smith said health belief model theories can be incorporated. One participant said Mazor appeared to be looking at oral health lit - eracy in a static way rather than taking advantage of the simple ability in an interpersonal situation to ask a question for clarification. Is that being factored into the analysis? Mazor responded that it is not because of the resource limits and constraints of the testing situation—administering a test that did not require someone to score it. It is an important piece missing from the study, but there is still value in learning whether people understand information when they hear it the first time. If one could get at the interaction effect, the participant continued, there is an opportunity to weave health literacy into the care model. The care model4 is a systems approach to health care that involves productive interactions among activated patients and informed providers. If one could weave health literacy into that model, it could be a very important way to evaluate whether those interactions have been productive. Mazor said that one will probably find that people do not understand a lot of what are fairly simple bits of interactions. It is important to know that what is said in interactions is not understood in the same way that print literacy is understood. Another participant from Health Literacy Consulting said that her experience has shown that difficulty in understanding is increased at the moment of encounter when the provider speaks English but the patient speaks English as a second language. Assessment of oral health literacy would benefit from looking at this issue. One participant asked whether there is any assessment that observes what happens during an interaction between the patient and clinician, either with a peer observation of the process, a patient exit interview, a 4“The Care Model is a population-based model that relies on knowing which patients have the illness, assuring that they receive evidence-based care, and actively aiding them to participate in their own care. . . . Effective outpatient chronic illness care is characterized by productive interactions between activated patients (as well as their family and caregivers) and a prepared practice team. This care takes place in a health care system that utilizes community resources” (http://www.tachc.org/HDC/Overview/CareModel.asp). Accessed April 12, 2009.

OCR for page 29
 APPRoACHeS To ASSeSSInG HeALTH LITeRACY doctor exit interview, or another method. Cindy Brach responded that John Hopkins University has a project where interactions are videotaped in order to study the nature of the interaction. One participant asked, for the assessment of oral health literacy tak - ing place in the CRN, are there any plans to study people’s ability to understand information under distressing conditions? For the most part, it appears these assessments are being conducted under ideal conditions. But how will patients perform when they have just been given distressing news, such as that they have cancer or that they need to come back for a second mammogram? Mazor said that issue is very important, but it is not something that is covered by the study described. An assessment conducted under stressful conditions would require different measurement questions from the kind of standardized instrument being tested in the assessment of oral health literacy. Another participant said it appears that for each of the items included in the assessment described by Mazor, one can examine the difference in difficulty of each of the messages as well as the performance of indi- vidual participants. There are informed providers (i.e., the messages) and activated patients (i.e., participants). One should be interested not only in the performance of the participants—the test takers—but also the per- formance of the items that represent the context. Mazor said she agreed completely. Looking at how difficult the items are—the items within a clip and the individual items associated with each clip—is important. It is planned that during the final year of the project work can focus on modifications to messages that allow one to test whether it is easier to give a message in one way versus another. That would, hopefully, lead to recommendations to providers as well as public health communication personnel about how better to construct messages. One participant said that the study under discussion is measuring both the understanding of the stimulus and the understanding of the question. But is the study trying to match these in terms of level of dif- ficulty? Mazor said that each of the demonstrations will have a number of restatements associated with it. One could conceivably write easy ones or hard ones. One does not want everything to be too easy because that would not allow discrimination of levels of health literacy. The level of difficulty is really a function of both the original statement and the item associated with that statement. Another participant said that one of the differences between oral com- munication and print communication is that the printed material can be taken home while the oral communication exists in the interaction only. Additionally in terms of the immediacy of measurement, there is also

OCR for page 29
0 MeASUReS oF HeALTH LITeRACY the delay factor. Has thinking been given to exploring not only what is understood in the office, but what is understood once the patient returns home? Mazor said that one of the reasons print materials are valuable is because one can take them and review them later. Patients want materi- als to take home. This underscores the fact that attention must be paid to print material. In the study of oral health literacy, there are measures of cognitive function as well as memory.