Future Directions for the National Healthcare Quality and Disparities Reports (2010)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

6 Improving Presentation of Information The NHQR and NHDR can be forward-looking documents that not only present historical trend data but also convey a story of the potential for progress and the health benefits to the nation of closing quality and disparity gaps. To serve as catalysts for improvement, the Future Directions committee envisions that the reports will extrapolate rates of change to indicate when gaps between current and recommended care might be closed and will present benchmarks on best-in-class performance. The committee makes sug- gestions for organizing report content to tell a more complete quality improvement story, realize greater integration between quality improvement and disparities elimination, improve takeaway messages and data displays, and achieve a better match between the AHRQ products and potential audiences. The NHQR and NHDR provide an enormous amount of data—principally presented in graphs—about how the United States is performing on various measures of health care and how performance has bettered or worsened over time. Although these data are useful, the NHQR and NHDR have potential beyond reporting on historical trends; the reports can also illuminate realistic levels of performance for all to strive toward and provide information on how long it will take to close gaps between current and recommended care at the current pace of improvement. In this chapter, the committee expands on its vision that future versions of the NHQR and NHDR should tell a more complete story of how to move toward achieving a high-quality, high-value health care system. To make the information presented in the NHQR and NHDR more forward-looking and action-oriented, the committee recommends that AHRQ make greater use of benchmarking and suggests improvements to data displays and the general organization of the NHQR and NHDR. Helping audiences for the NHQR and NHDR better understand gaps in the quality of U.S. health care—whether between actual performance and receiving the recommended standard of care, or between population groups or geographic regions—and better understand the benefits of closing those gaps would provide audiences with stronger evidence and rationales for improving quality. The chapter begins by reviewing the Future Directions committee’s suggestions for how AHRQ’s lineup of products could better serve current and expanded audiences. The committee underscores the importance of integrating disparities reduction into quality improvement by enhancing the relationship between the structure of the two national healthcare reports. Finally, suggestions are made on improving data displays and the statistical quality of quality reporting. 111 

112 NATIONAL HEALTHCARE QUALITY AND DISPARITIES REPORTS MATCHING PRODUCTS TO AUDIENCE NEEDS At present, the national healthcare reports and related products are consulted by a variety of stakeholders, many of which have different interest areas (e.g., heart disease, rural health, racial disparities, delivery settings) and dif- ferent levels of sophistication for data interpretation and analysis. The 2001 IOM report Envisioning the National Health Care Quality Report stated that the NHQR was not to be a “single static report, but rather a collection of annual reports tailored to the needs and interests of particular constituencies” (IOM, 2001, p. 6). The committee believes that AHRQ needs to expand and refine its quality reporting product line to provide products and data that are useful and understandable for a variety of audiences. Therefore, the committee recommends the following: Recommendation 6: AHRQ should ensure that the content and presentation of its national health- care reports and related products (print and online) become more actionable, advance recognition of equity as a quality of care issue, and more closely match the needs of users by: • incorporating priority areas, goals, benchmarks, and links to promising practices; • redesigning print and online versions of the NHQR and NHDR to be more integrated by recognizing disparities in the NHQR and quality benchmarks in the NHDR; • taking advantage of online capability to build customized fact sheets and mini-reports; and • enhancing access to the data sources for the reports. The committee’s suggested products, along with their potential audiences, are reviewed in Table 6-1. Refining the Organization of the NHQR and NHDR Integrating efforts to improve quality with efforts to eliminate disparities increases opportunities to positively affect change. Presenting the same organizational framework and measures in both reports reinforces users’ understanding of the relationship between overall health care quality and the depth of health care disparities. But currently, the two reports are not well linked beyond presenting the same measures. Changing the Highlights Section of the Reports The committee proposes that AHRQ present the same Highlights section in both the NHQR and NHDR to underscore the relationship between health care quality and equity. The text of the Highlights section should be developed so that the section can be published as a stand-alone document that could be the subject of dissemina- tion events targeted to relevant stakeholder audiences. The document could: • Spotlight areas with the greatest potential for quality improvement impact and provide detail on what the value of closing quality gaps would be to population health and equity. • Feature progress on priority areas and toward any established national goals. • Discuss evidence-based policies and best practices that may enhance quality improvement or factors that hinder progress as informed by data within the body of the report. • Emphasize takeaway messages directed to different audiences (e.g., policy makers, health care providers, and the public) on what they can do to improve health care quality on prioritized topics and measures. • Include a summary of state performance and the state of disparities. The committee believes that a summary of state performance should be part of the Highlights section of the reports and would be of interest to legislators and policy makers at the state and national levels. A one- to two-page summary of state performance, perhaps in a scorecard fashion, should be included, and AHRQ could compile this from the information it already provides in the State Snapshots (e.g., ratings from very strong to very weak on overall health care quality, preventive measures, acute care measures, chronic measures, hospital care measures, cancer care measures). Currently, the State Snapshots are not available until several months after the reports have 

IMPROVING PRESENTATION OF INFORMATION 113 TABLE 6-1  Tailoring Products to Meet the Needs of Multiple Audiences Product Potential Audiences Recommended Content Shared “Highlights” Section Policy makers, media, public, Features progress on the national priorities areas and measurement [redefined product to be used in foundations and other funders areas with the greatest potential for quality improvement impact on both the NHQR and NHDR and of research, national quality population health, value and equity; evidence-based policies/best that can be disseminated as a organizations practices that will enhance or hinder progress; actions that stakeholder stand-alone product] groups can take; and what is needed to make progress toward national goals. Includes a summary of progress by states. Includes a summary on state of disparities. NHQR Quality, advocacy, and standards Information on a set of measures organized by the expanded quality [refinement of existing product] setting organizations; health care framework to address: effectiveness, safety, timeliness, patient- providers, plans, payers, and centeredness, access, efficiency, care coordination, and capabilities of purchasers at the national and health systems infrastructure. state level; research community Includes access, a topic previously addressed only in the NHDR. Details that disparities exist (beyond the current displays on geographic variation or age) by including a separate chapter or summary on socioeconomic, racial, and ethnic disparities; and acknowledges in messages when socioeconomic and racial disparities exist for individual measures. NHDR Quality, advocacy, and standards Maintains parallelism with the NHQR by applying the expanded [refinement of existing product] setting organizations; health care quality framework to its organization and presentation of measures. providers, plans, payers, and Includes benchmarks, not just comparisons among populations. purchasers at the national and state level; research community Provides more in-depth coverage of priority populations. Fact Sheets and Mini-Reports Advocacy groups, strategic Includes short story of key facts and potential actions related to certain [expanded products] partners for dissemination, disease-specific or priority population topics in the NHQR and NHDR. media, public State Snapshots State government; health care Provides expanded measure set sortable by core measures, Healthcare [refinement of existing product] providers, plans, payers, and Effectiveness Data and Information Set (HEDIS) measures, state purchasers at the state and local rankings, and comparisons between states with similar population level characteristics, not simply in neighboring geographic regions. Adds to context by including best performance attained (for all states and for peers), variation within state (e.g., by geography, providers, payers, race/ethnicity/language, and the availability and type of insurance). Includes access measures. Online Data Access Advocacy groups; stakeholders Has capability to collect text and data by topic to yield a customized [expansion of existing in quality improvement, media, report. approaches] and public Links to other helpful data sources and intervention information. • Customizing reports via the Researchers (for access to Provides access to full datasets for user manipulation, and links to NHQRDRnet primary data for additional other sites that provide expanded metrics on health care data (e.g., national, state, and local level • Data repository of primary CMS), and more local and organizational level data. analyses) datasets Has tools to show users how to mirror AHRQ’s analytic approaches. Guide to Using the NHQR and All potential user audiences, but Explains how to access and utilize available data. NHDR particularly researchers Gives examples of how different stakeholder groups can apply the [new product] knowledge to action. 

114 NATIONAL HEALTHCARE QUALITY AND DISPARITIES REPORTS been released; the committee urges AHRQ to include this information in the Highlights section even if the detailed Snapshots are not posted online at the same time. Additionally, a summary on the state of disparities should be included. Although the Highlights document proposed by the committee would be longer than the current Highlights section, sharing the same Highlights section should streamline AHRQ staff efforts. Organizing the NHQR and NHDR by the Framework Components The framework for health care quality and disparities measurement (see Chapter 3) is both a tool to assess whether balance is achieved in the selection of quality measures and a way to organize the reports. Table 6-2 suggests chapters (or sections) for future iterations of the NHQR and NHDR. To increase parallelism across both national healthcare reports, access to care, a topic currently addressed only in the NHDR, should also be included in future NHQRs. After carefully considering whether efficiency and health systems infrastructure should be featured in both reports, the committee concludes, as discussed in Chapter 3, that efficiency measures of overuse and underuse are of interest for populations included in the NHDR and infrastructure is also applicable to equity concerns. Given the limited nature of measures at this time, however, the same efficiency and infrastructure mea- sures may not always be available to include in both reports. Incorporating Equity into the NHQR Based on interviews with current and potential users, the committee finds that, to some extent, the NHQR and NHDR have different audiences. There is one school of thought that improving health care performance overall will ameliorate the problem of disparities in health care; this view tends to neglect the reality that disparities in health care usually persist even as overall performance levels improve. The committee believes that closing equity gaps is one of the most important factors in raising overall health care quality (Chin and Chien, 2006; Clarke et al., 2004). For that reason, the NHQR should incorporate the concept of equity by including an additional section focusing on disparities elimination. Quantifying the impact of disparities on overall quality performance may be one way to define the connection between health care quality and disparities in the Highlights section of the reports. Furthermore, the commentary on each measure within the NHQR could reflect the degree to which disparities remain or are growing even as quality improves so that conclusions on the state of quality are not misleading. An HIV/AIDS measure reported TABLE 6-2  Sections Recommended for Future National Healthcare Reports NHQR NHDR Highlightsa New focus New focus Effectiveness ✓ ✓ Patient safety ✓ ✓ Timeliness ✓ ✓ Patient-centeredness ✓ ✓ Efficiency ✓ New Access New ✓ Care coordination New New Health systems infrastructure New New Priority populationsb New (women, children, elderly) ✓ Disparities summaryb New State performance summaryb New New ✓ Currently in the report. a Not a framework component, but a section currently in the NHDR. b Not a framework component. 

IMPROVING PRESENTATION OF INFORMATION 115 in the 2008 NHQR provides a concrete example of a situation where the nation as a whole is performing well but data in the NHQR mask disparities shown in the NHDR (AHRQ, 2009d, p. 65, 2009e, p. 63). Presenting Data on Priority Populations The NHDR is required by the 1999 federal law under which it was established to report on “prevailing dis- parities in health care delivery as it relates to racial factors and socioeconomic factors in priority populations.”  Priority populations were defined in the authorizing legislation with respect to the agency’s full portfolio of activi- ties (research, evaluation, and demonstration projects): low-income groups, minority groups, women, children, the elderly, and individuals with special health care needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care. AHRQ’s overall activities are also to address inner-city and rural areas. The fourth chapter of the NHDR, “Priority Populations,” includes limited supplemental measures specific to each priority population. AHRQ has presented data on priority populations in the NHDR by offering: (1) summaries of the findings presented earlier in the report on access and on the core measures the NHDR shares with the NHQR (e.g., Tables 4.2 and 4.3 in the 2008 NHDR), and (2) occasional additional measures of particular interest for specific popula- tions (e.g., hospital admissions for uncontrolled diabetes for American Indian and Alaska Native populations). The committee encourages more comprehensive treatment of the priority populations both within the reports and through other vehicles (e.g., alternate year treatment of priority populations in the reports, spinoff mini-reports with additional detail, customization of data via NHQRDRnet). The national reports should convey key measures that address top health concerns of the priority populations if they are not already part of the AHRQ set of core measures; inclusion would depend on passing the same rigorous evaluation process for measures outlined in Chapter 4. Given the limitations in the length of a print version of the NHDR, other vehicles can provide additional opportunities for more in-depth treatment. Specialized products for audiences interested in specific priority popu- lations may garner more attention than solely expanding the priority population sections within the NHQRs and NHDRs. These derivative products could be published over time, perhaps in conjunction with partners who have a particular interest in care related to a topic or population. While “women,” “children,” and the “elderly” are priority populations, they do not belong solely in the NHDR. At a minimum, the committee believes a summary of findings for these populations should be available in the NHQR. Moving to the NHQR much of this material, which is now in the NHDR, would open up space in the NHDR. The committee expects further inclusion of children’s quality measures in the future as a result of the findings from AHRQ’s National Advisory Council Subcommittee on Quality Measures for Children in Medicaid and Children’s Health Insurance Programs, and the ongoing AHRQ- and CMS-funded IOM study of Pediatric Health and Health Care Quality Measures (AHRQ, 2009c; IOM, 2010). The full number of metrics and the vari- ous analyses that might be performed will likely exceed the capacity of the print NHQRs and NHDRs; as noted earlier, more detailed treatment could be accomplished through a special topic report, alternating in-depth sections in the NHQR or NHDR, and/or an ability to customize reports through Web-based capabilities. Bridging the NHQR and NHDR An inherent problem in having two separate reports is that data on a subject (e.g., cancer, heart disease, a priority population) are split between the NHQR and NHDR. Moreover, in the NHDR, data on a subject are often hard to find because information is dispersed between different sections (see Box 6-1). This fragmentation of information will continue to exist, but adding an index at the end of each book would help users find information more readily. The committee also notes that introductory page(s) for the same topic in the NHQR and NHDR tend to cover the same types of information but are laid out differently; this requires unnecessary effort on the part of AHRQ staff and leads to confusion by readers. For example, the pages on effectiveness of cancer screening in the NHQR  Healthcare Research and Quality Act of 1999, Public Law 106-129, 106th Cong., 1st sess. (November 19, 1999). 

116 NATIONAL HEALTHCARE QUALITY AND DISPARITIES REPORTS BOX 6-1 How Do I Find Disease-Specific Information in the NHDR? When examining a health topic or specific population in the NHDR, the information is often difficult to find. For exam­ ple, if one aims to look at colorectal cancer, one is unable to access all of the information about colorectal cancer in one place.a The report is organized by sections, and all but one (Access to Health Care) contain some information on colorectal cancer. (See the following sections in the 2008 NHDR: Quality of Health Care, pp. 39-43, 46, 122, 127; and Priority Populations, pp. 177-181, 185, 188-189, 199, 201, 203, 207, and 211 [AHRQ 2009d]). These pages include infor- mation on screening, mortality, and differences by race and socioeconomic status. Additional data on colorectal cancer are available in the NHQR. Where should consumers, policy makers, or health services researchers go to access sought-after information? Should one search in the Quality of Health Care section or the Priority Population section of the NHDR? Without an accom­panying index, one must sift through the entire report to find all disease-specific information. All of the information on a single topic could be linked through an online search tool within the existing NHQRDRnet (which now only supports data tables but not text), other search tools, and linkages between sections in the NHQR and NHDR on the main report site. Even with linkages, the multiple areas in which a single topic appears within the NHQR and NHDR may not be intui- tive to readers. a Colorectal cancer and breast cancer are reported in alternate years in the NHDR. (AHRQ, 2009e, pp. 32-33) and NHDR (AHRQ, 2009d, p. 39) contain similar yet not identical information. The committee finds it logical to convey the same information in both locations. Expanding and Sustaining Interest Through Derivative Products AHRQ provides online access to the national healthcare reports and State Snapshots, to a few report-related fact sheets, and to an online data query system (NHQRDRnet); the Future Directions committee suggests changes to each of these products (see Table 6-1, presented earlier in this chapter). Fact Sheets and Mini-Reports AHRQ has previously developed three fact sheets to supplement the NHQR and NHDR. The fact sheets have addressed the subjects of children and adolescents (AHRQ, 2005b, 2008a, 2009b), women’s health (2005c), and rural health (2005a). These fact sheets are not easily accessible on the national healthcare report-related websites; instead they are listed on AHRQ’s Measuring Quality website. Concise fact sheets are a way to expand AHRQ’s reach of NHQR and NHDR findings and are useful for reaching new audiences. Timing the release of fact sheets to specific events (e.g., heart disease or breast cancer awareness months) could help sustain interest in the national healthcare reports throughout the year. Currently, Internet traffic to the NHQR and NHDR tends to decrease about two months after the report release date. Periodic releases of fact sheets could direct Internet traffic to the reports. There may be times when a derivative product elaborating on a specific topic (e.g., a mini-report) could provide information beyond what can be contained in a fact sheet. The committee believes that such mini-reports could provide expanded treatment of priority populations. Priority population-specific derivative products would allow fuller exploration, for example, of the particular health care concerns of a priority population (e.g., children, women) or the diversity of health care experiences of different granular ethnicity groups within a race category (e.g., the Asian American population, for instance, is made up of persons of Japanese, Korean, and Cambodian heritage, among other granular ethnicity groups). 

IMPROVING PRESENTATION OF INFORMATION 117 State Snapshots In 2006, using data collected for the NHQR, AHRQ created the Web-based State Snapshots to fulfill the needs for state-level information of members of Congress, state officials, health care providers, and purchasers. As noted by previous IOM guidance, “analyses such as state-by-state comparisons on health care are familiar and meaningful to members of Congress, other policy makers, and consumers” (IOM, 2002, p. 5). The commit- tee finds AHRQ’s Web-based State Snapshots to be a valuable addition to the NHQR and NHDR and recognizes recent improvements to the State Snapshots website. Nevertheless, the committee urges further development. For example, the State Snapshots do not show any data on access measures, and the committee believes these data are important to have at the state-level. Health care report cards provide information about the quality of care by geographic regions, health plans, hospitals and other institutions, and even individual practitioners (Epstein, 1995). Report cards use various systems of scoring and passing judgment on quality, whether the end result is to grade national health performance, rank a state’s health care quality against all others, compare head-to-head the quality of care delivered in cities across the country, or to develop a list of best value hospitals (Brooke et al., 2008; Chernew and Scanlon, 1998; Davies et al., 2002; Hibbard and Jewett, 1997; Romano et al., 1999). A 2006 qualitative study conducted by AcademyHealth indicated that users of the State Snapshots suggested a rank ordering of states so that states could compare their performance against all others (Martinez-Vidal and Brodt, 2006). Currently, in the State Snapshots, each state is ranked on 18 selected measures. The committee’s view is that state-by-state ratings should be more clearly available so that states know what the best attained level of quality performance is, and then they could contact and learn from states with the best rates on specific quality measures. Additionally, it would be useful if state data and rank- ings were easily sortable for high-profile sets of metrics such as AHRQ core or HEDIS (Healthcare Effectiveness Data and Information Set) measures, given the almost 200 measures that AHRQ tracks for states. AHRQ displays average regional performance on measures in the State Snapshots, but state audiences have indicated that adjoining states are not always peers. AHRQ has provided a graphical “dial” to show states how they fit on a spectrum of contextual factors (i.e., demographics, health status, etc.), but states have noted that they would like flexibility to be able to identify a different coterie of peer states (for example, states that have the same degree of contextual factors). Then, for example, a state policy maker could assess state performance against states that have a comparable extent of persons below the poverty level. NHQRDRnet Users of the NHQR and NHDR suggest that their needs for information tend to be topic specific and episodic; most users of the national healthcare reports are unlikely to read the reports from cover to cover. Additionally, the reports are viewed and downloaded online more than they are used in hard copy. Thus, improving the ability of users to find needed information online is an important aim. In 2008, AHRQ added an online interactive tool called NHQRDRnet that can be used to query and search the databases behind the NHQR and NHDR by content areas (quality, access, patient safety, priority populations), clinical conditions, care type or settings, and dimensions of access (e.g., insurance coverage, usual source of care, utilization). The committee applauds AHRQ’s intent to facilitate searching for content in the national healthcare reports but finds navigating the NHQRDRnet website difficult. The committee also observes that it takes fewer steps to gain similar information from the more straightforward and easier to use Appendix D of the NHQR and NHDR. AHRQ recently commissioned a usability survey that queried current users of the national healthcare reports about their experience with and impressions of the NHQR and NHDR and related Web content (Social & Scientific Systems and UserWorks, 2009). Comments from survey participants regarding ease of using the website and clar- ity of information echoed the committee’s findings (e.g., difficulty finding the reports using a basic Web search, organization of report, and Web content not matching user expectations). The major change the Future Directions committee suggests for the NHQRDRnet is the development of a tool or sorting function that would allow users to customize their own reports. Now, one can search for all data  Personal communication, Farah Englert, Agency for Healthcare Research and Quality, November 12, 2009. 

118 NATIONAL HEALTHCARE QUALITY AND DISPARITIES REPORTS related to a topic—for example, cancer—and all data files since 2002 are displayed for download; the search does not yield a fact sheet or summarization of the current content on the subject of interest. At a minimum, links to relevant text of the current year’s NHQR and NHDR would enhance the site’s usability and ability to tell a more comprehensive story. Additionally, one- or two-page fact sheets or more in-depth mini-reports on topics (whether individual clinical conditions, framework components, or something more specific, such as quality and disparities issues of specific interest to Hispanic persons) would be useful. AHRQ’s partnerships with other stakeholders would be assisted by having prepackaged collections of information in its NHQRDRnet index. Web-based products, in addition to the NHQRDRnet, can be configured to make it easier to guide readers to other AHRQ or non-AHRQ resources that may help with quality improvement. For example, future online ver- sions of the NHQRs and NHDRs could link to interventions highlighted in the AHRQ Health Care Innovations Exchange or other related resources (e.g., CMS, entities utilizing measures for which data at the national level are still aspirational). This linking capacity should also be available through the Web-based version of the NHQR and NHDR without the reader having to go through the NHQRDRnet. Online Access to Data Used in the NHQR and NHDR The committee discussed the extent to which the NHQR and NHDR should provide data for geographic areas below the state level. Various stakeholders have noted that the national healthcare reports contain information that is too “high level” for making decisions at the local or practice level. Consequently, the reports may be of less use to some health care providers, local policy makers, or some researchers than if the performance data were strati- fied to show performance at more local or organizational levels and provided in a timelier manner. State-based data are a unit of analysis that policy makers as well as the public can easily relate to and use for comparative purposes. Given the interest in substate variation (e.g., the Dartmouth Atlas, the University of Wisconsin/Robert Wood Johnson Foundation county-by-county health rankings), these data would be useful to develop over time (The Dartmouth Institute for Health Policy & Clinical Practice, 2010; RWJF and the University of Wisconsin, 2010). The NHQR and NHDR could also include linkages to other HHS data resources on community health status indicators (HHS, 2010). The data included in the NHQR and NHDR may be reported nearly a year or more after they have been submitted to AHRQ because of the processes involved with compiling data sources, cleaning the data, analyz- ing it, reviewing the reports at the departmental level, and submitting the work for production. For entities that are evaluating performance in real time (daily, weekly, monthly), such data may have limited use. Still, there are groups that do not have day-to-day access to performance measurement data and would benefit from the wider availability of nationally collected data at a more local level (Kerr et al., 2004). For example, AHRQ has made available county-level data on the number of Hispanic Medicare beneficiaries with diabetes who did not have an eye exam so that one area’s community aging agencies could focus intervention efforts (Moy, 2009). In deciding whether to recommend that AHRQ provide more locally based data, the committee balanced the usefulness of local data, its timeliness, its reliability, and the additional workload for AHRQ staff. AHRQ staff indicated that it is possible to drill down to at least the larger Metropolitan Statistical Areas for about half of the State Snapshot measures, but that smaller Metropolitan Statistical Areas and counties would be more dif- ficult because, for instance, some datasets are likely to require special permissions to present the data in these ways. The committee encourages AHRQ to explore the feasibility and value of drilling down for at least some high-impact measures. When summarized in the reports, the Highlights section, or the proposed guide to using the NHQR and NHDR (discussed below), more localized data can inform readers about variation within states. Such detail could be presented in the State Snapshots to show substate variation, particularly when it is readily available in the datasets AHRQ already uses, and perhaps as a derivative product similar to the Atlas of Mortality  Personal communication, Ernest Moy, Agency for Healthcare Research and Quality, November 18, 2009. AHRQ staff estimate that data for large Metropolitan Statistical Areas (MSAs) could be provided for more than 50 percent of the State Snapshots measures. All CMS Compare systems could support MSA-level estimates. The Behavioral Risk Factor Surveillance System (BRFSS) could be used for MSA-level analyses for the top 150 MSAs. Other data sources, such as vital statistics data and hospital data from Healthcare Cost and Utilization Project (HCUP) states, would require special permissions in order to analyze and publish data at the MSA-level. 

IMPROVING PRESENTATION OF INFORMATION 119 (Pickle et al., 1996), depending on the availability of data for coverage of the United States. Links could be made to the HHS Community Health Status indicators site if it eventually included health care-related metrics and not just health status. Individuals wanting to work with primary data are often not satisfied with the data available through the national healthcare reports’ website. AHRQ provides Excel files with the data points reflected in its graphs and text, but it does not provide access to the original datasets. Although AHRQ does not have in-house all of the databases it uses in the NHQR and NHDR, most of the data AHRQ uses are from federally sponsored datasets. The committee believes that data access could be expanded so that researchers can download the full dataset to manipulate it as needed. This is consistent with the efforts of data.gov, a website currently under development that will house all federal executive branch datasets, to “increase public access to high value, machine readable datasets generated by the Executive Branch of the Federal Government.” AHRQ is among the agencies contributing data, as are other federal agencies whose data AHRQ acquires (e.g., the Centers for Disease Control and Prevention, CMS). Because the website is still under construction, the committee is unable to discern which of AHRQ’s datasets will be made available. Nonetheless, the committee feels that making available datasets that support findings in the NHQR and NHDR would be a service to providers and health services researchers. The committee encourages AHRQ and its partners to provide access to the data in a timely fashion, even prior to its publication in the NHQR and NHDR, to allow those with the capacity for analysis to use it for their own needs. Such data access was previously recommended to AHRQ by the IOM in the 2001 report Envisioning the National Health Care Quality Report. Further, AHRQ should provide tools for analysts who want to replicate AHRQ’s methods to produce comparative data for their locale or population cohort. Until such tools are available, analytic methods will need to be clearly specified in methodology descriptions. Proposed Development of a Guide to Using the NHQR and NHDR Given the diversity of resources that AHRQ now offers and the potential for greater direct data access, the committee suggests that AHRQ develop a guide to using the NHQR and NHDR. As envisioned by the commit- tee, this technical assistance product would review the resources that the print NHQR and NHDR and websites have to offer and, more importantly, would provide examples of how different stakeholder groups can apply the knowledge to action (e.g., Hispanic elders diabetes project). The guide to using the NHQR and NHDR would go beyond telling someone how to navigate a website. Instead, it would tells users how to access the data resources, provide tools for manipulating data for analyses, explain the methods used by AHRQ in its analyses, and offer suggestions for meaningful analyses. This guide, when it becomes available, should be referenced in the Highlights section of the NHQR and NHDR. Dissemination Strategies The committee proposes communicating the findings of the NHQR and NHDR to diverse audiences through a series of new products and modifications to existing documents. The goal of expanded dissemination efforts should be to raise awareness, visibility, and use of the reports. Between 2003 and 2008, AHRQ distributed approximately 24,000 print copies of the NHQR and NHDR. The annual release of the NHQR and NHDR should be more widely publicized in advance, and momentum from the release of the reports should not be permitted to dwindle. The committee sought input on dissemination and media strategies for the NHQR and NHDR, as well as sample fact sheets, from Ketchum, a public relations and communications firm.  Ketchum suggested ways to repackage the wealth of information contained in the NHQR and NHDR throughout the year so that findings can be made continually relevant. AHRQ could, for example, produce suc- cinct derivative materials that convey targeted messages (e.g., mini-reports and fact sheets), and link distribution  The data.gov website is accessible at http://www.data.gov/ (accessed December 12, 2009).  Personal communication, Farah Englert, Agency for Healthcare Research and Quality, November 16, 2009.  Copies of the sample media and communication plan and sample fact sheets developed by Ketchum were provided directly to AHRQ staff and archived in the IOM public access file for the Future Directions project. 

120 NATIONAL HEALTHCARE QUALITY AND DISPARITIES REPORTS and media outreach to appropriate audiences (e.g., advocacy groups for specific clinical conditions or population groups). In addition to relying on traditional media outreach (e.g., participating in roundtables, telebriefings, radio media tours, outreach around editorial calendars), AHRQ could take advantage of wide-reaching and increasingly common Web-based tools (e.g., blogs, advanced search engine options, inbound linking programs, social media such as Facebook and Twitter). In Chapter 5, the committee emphasizes that a patient’s language need is relevant to health care quality and disparities. Communicating information about quality and disparities in languages other than English is one way to reach diverse audiences. The committee acknowledges the potential resource burdens of document translation and does not intend for AHRQ to translate the entire NHQR and NHDR. When derivative products are specifically geared to audiences that may have limited English proficiency, AHRQ may wish to consider translation. HHS’ Office of Minority Health Resource Center provides technical assistance in communications and outreach to other HHS agencies (HHS, 2009b). As AHRQ expands dissemination of the reports and their derivative products to diverse stakeholder audiences, the Health Resource Center may provide valuable capacity-building support. TELLING A STORY IN THE NHQR AND NHDR The committee recommends that the NHQR and NHDR tell a clear and compelling story about the impact of making progress—or of not making progress. The ways in which information is presented and summarized in the reports and related products can enhance or impede users’ understanding of the messages the reports are meant to convey. For that reason, the committee believes that AHRQ should move the reports from their current chartbook format to make them less a catalog of data and more of a comprehensive story that conveys key mes- sages through text, graphs, and displays. The committee believes that doing this effectively requires enhancing the presentation of takeaway messages on the state of quality and disparities, focusing attention on closing gaps in performance, including benchmarks to allow comparisons with high-quality performance, identifying ways to affect change, and providing information that contributes to the development of the national health care data infrastructure (Box 6-2). Enhancing the Presentation of Takeaway Messages In the Healthcare Research and Quality Act of 1999, Congress directed AHRQ to submit “an annual report on national trends in the quality of health care,” and AHRQ has interpreted this as needing to present “assessments of change over time” (Moy et al., 2005). Although documenting the past performance of the U.S. health care system is important and historical data certainly play a role in forming a comprehensive picture of health care quality and disparities, users of the national healthcare reports have indicated that the performance of past years (especially more than 5 years ago) is not necessarily helpful for assessing where and how quality improvements can be made today (Lansky, 2009; Martinez-Vidal and Brodt, 2006). The committee believes it would be more useful for AHRQ to interpret national trends as a way to inform the future, using available historical data to inform readers of the likelihood of closing gaps in health care quality at the current pace. Forward-looking messages regarding national trends for the future could be determined using the following central pieces of information: • the nation’s current level of performance (expressed using means and standard errors); • how the nation has achieved the current level of performance (expressed by the historical annual rate of change and standard error of the estimated change); and • how far the nation has to go to close the performance gap between current practice and the recommended standard of care (goal or the benchmark)—the number of years to achieving the desired performance level based on the historical annual rate of change and corresponding interval estimate.  In 2004, AHRQ was advised to use a chartbook format for future iterations of the NHQR (Gold and Nyman, 2004).  Healthcare Research and Quality Act of 1999, Public Law 106-129 § 902(g) and § 913(b)(2), 106th Cong., 1st sess. (November 19, 1999). 

IMPROVING PRESENTATION OF INFORMATION 121 BOX 6-2 Key Elements of Telling a Story in the NHQR and NHDR Enhancing Takeaway Messages that Address Closing the Performance Gap • “At the current rate of change, it will take ‘X’ years to close the gap between current practice and the recom- mended standard of care (goal level or the benchmark).” • The net health benefit of closing the gap (including clinical preventable burden and cost-effectiveness) is quantified. • Areas on which to focus attention so as to more effectively improve quality are specified. Identifying Ways to Effect Change in the Health Care System • Highlight the impact of evidence-based policies that can help drive change. • Provide data analyses. • Include vignettes or links to innovative practices that have resulted in higher performance. Presenting Benchmarks and Other Data • benchmark of best-in-class performance; • between and within-state variation, when available; • variation by sociodemographic variables (e.g., race, ethnicity, language need, socioeconomic status, and insur- ance status); • data presented by accountable units, whenever feasible (e.g., types of payers, delivery sites); • displays with visual clarity and embedded explanations of the essential finding(s); and • meaningful summarizations. Contributing to the National Health Care Data Infrastructure • illustrating developmental* and emerging measures even when only subnational data are available; and • highlighting when data are unavailable and when greater efforts are needed for national collection. * Developmental refers to measures that are currently partially developed but not yet well tested or validated, or measures that have been validated but still lack sufficient national data on which to report. Using this strategy, AHRQ could transform its wealth of available trend data into an informative direction for the future. Possible templates for presenting rates of change and years to closing quality and disparity gaps are offered in Appendix H. As previously described in Chapter 4, the impact of closing the gap would be determined as part of the mea- sure selection/ranking process, and data gleaned (e.g., reduction in clinically preventable burden, increase in net health benefit, and cost-effectiveness) in determining the relative ranking of measures are useful and should be presented for each measure in the reports. Additionally, the benefit to the country—if, for example, all states were performing at the level of the highest one—would also be key information. Presenting Benchmarks and Other Data To better convey key messages, data displays should present benchmarks. The committee believes benchmark- ing is a key tool for continuous quality improvement. Thus, it is expected that benchmarks will change over time depending on the frequency of obtaining updated data from the sources for the national healthcare reports. Goals, on the other hand, tend to be fixed for a longer period and set by an advisory body or at the direction of some entity such as the Secretary of HHS. (See Chapter 2 for committee definitions of goals, benchmarks, and targets.) In the context of the national healthcare reports and AHRQ’s role, the Future Directions committee emphasizes the use of benchmarks rather than goals because the committee believes the presentation of performance data, but not the setting of national goals, is within AHRQ’s purview. Benchmarks reflect empirical facts. On the other hand, the committee believes that the setting of goals for health care quality improvement (e.g., for priority areas and/or measures) requires the direction of the Secretary of HHS. 

122 NATIONAL HEALTHCARE QUALITY AND DISPARITIES REPORTS Goals or fixed targets for measures can complement benchmarks and could be set at various levels of attain- ment. For example, they may be aspirational—“All patients shall receive.” Goals might be set at a lower level if a finding from the measure selection assessment shows that there is little gain in health benefit beyond 85 percent of the target population receiving a service. Or a goal might be set for all states to achieve the rate of the best performing state. Data illuminating who is delivering care and where care is delivered are necessary to identify opportuni- ties for system change; these accountable units may be states, types of payers (e.g., Medicare, Medicaid, private insurance), or delivery systems. The committee encourages the development and presentation of these data in the reports and State Snapshots. This topic is addressed more fully in Chapter 2. Identifying Ways to Affect Improvements in the Health Care System Although the reports by themselves do not affect change, they can link to entities that have improved quality and reduced or eliminated disparities. For policy makers and those engaged in measurement and improvement, having the reports illustrate actual, effective quality improvement interventions alongside comparative data would be useful. As previously discussed, AHRQ’s NHQRDRnet site links to AHRQ’s Health Innovation Exchange, and this type of connection should also be included in the online version of the reports through embedded hyperlinks. Additionally, AHRQ should consider qualitatively highlighting “islands of excellence” (whether health systems, hospitals, or geographic regions) that consistently deliver recommended care that is less costly, more efficient, and produces better outcomes (Fisher et al., 2008). Such better performing communities or entities can be showcased in textboxes and sidebars. Currently, AHRQ links State Snapshots to other measure report cards in specific states and should continue such nonfederal linkages. In addition to the Health Innovation Exchange, AHRQ might link with sources such as the Robert Wood Johnson Foundation’s (RWJF’s) Finding Answers: Disparities Research for Change program (www.SolvingDisparities.org), The Commonwealth Fund’s “Why Not the Best?” quality improvement resource (http://whynotthebest.org), and the Institute for Healthcare Improvement’s (IHI) website (www.ihi.org). These sources, among others, offer multiple strategies for hospitals, providers, and other actors to improve the quality of health care. The links should be accompanied by an expressed caveat that the links are intended to highlight known best or promising practices, and that their inclusion should not be construed as an endorsement of the program or entity by AHRQ. USING BENCHMARKS TO SHOW ACHIEVEMENT Benchmarks are one method of comparing data in order to improve the efficiency and the quality of health care (Deming, 1994). In Chapter 2, the committee defined a benchmark as the quantifiable highest level of performance achieved to date. (Some additional definitions of benchmarking are shown in Table 6-3.) Presenting performance TABLE 6-3  Definitions of Benchmarking from Various Sources Source Definition of Benchmarking Vaziri, 1992 A continuous process comparing an organization’s performance against that of the best in the industry considering critical consumer needs and determining what should be improved. Kleine, 1994 An excellent tool to use in order to identify a performance goal for improvement, identify partners who have accomplished these goals, and identify applicable practices to incorporate into a redesign effort. Cook, 1995 A kind of performance improvement process by identifying, understanding, and adopting outstanding practices from within the same organization or from other businesses. Camp, 1998 The continuous process of measuring products, services, and practices against the toughest competitors or those companies recognized as industry leaders. SOURCES: Camp, 1998; Cook, 1995; Kleine, 1994; Vaziri, 1992. 

IMPROVING PRESENTATION OF INFORMATION 123 data in the context of benchmarks stimulates debate around policy priorities, promotes transparency, fosters accountability, indicates what needs to be done, and supplies concrete milestones for evaluation and identification of areas to improve (Gawande et al., 2009; van Herten and Gunning-Schepers, 2000a,b). Benchmarks identify “demonstrably attainable,” superior performance and encourage others to emulate the practices by which this is achieved (Kiefe et al., 1998, p. 443). The original idea of using benchmarks in Continu- ous Quality Improvement and Total Quality Management (CQI/TQM) was that organizations could learn from the processes of an organization with better outcomes and adapt those processes, as appropriate, to their own circumstances (Dattakumar and Jagadeesh, 2003; McKeon, 1996). Benchmarking is not a static process; ideally, the level of best performance will continually evolve as positive progress is made, and the benchmark will move accordingly. At each successive stage—or in each publication year of the NHQR or NHDR—a different entity has the potential to take the role of “best-in-class,” which may engender a “race to the top” (Weissman et al., 1999). The committee proposes approaches to benchmarking that AHRQ could incorporate into the NHQR, NHDR, and related products. The benchmarking approaches proposed by the committee do not require AHRQ to develop targets that must be attained by a specific endpoint (as has been done for Healthy People 2010); rather these strate- gies use benchmarks to highlight standards of care that are reported in data available to AHRQ. The Current Use of Benchmarks in the National Healthcare Reports The NHQR and NHDR were initially envisioned as a means to provide policy makers with snapshots of qual- ity and disparities over time and to allow “providers and payers” to “assess their performance relative to national benchmarks” (Moy et al., 2005, p. 377). The hope was that government agencies, communities, and providers would turn to the NHQR and NHDR to compare their own health care data against national progress. Until recently, AHRQ used only an implicit benchmark—namely, the need to strive for better-than-average performance. Displays in the reports imply that states with rates below average performance should aim to achieve performance rates better than average. In a 2006 review of AHRQ’s presentation of state data, state policy makers indicated that presenting performance relative to the national average was misleading: while a state may have been doing better than average on a given measure, if the average was low compared to the recommended standard of care, the level of performance could be taken out of context to indicate that the state need not focus quality improvement efforts in that area (Martinez-Vidal and Brodt, 2006). For a limited number of measures in the 2008 NHQR and NHDR, AHRQ reports targets established for Healthy People 2010. Partially because Healthy People focuses on measuring health improvement rather than health care improvement, these targets are not available for all measures presented in the NHQR and NHDR. The Healthy People targets are not tied to actual performance achieved by providers and health care organizations, and most targets are consequently viewed as aspirational. According to the committee’s definition, a benchmark should be demonstrated as being attained by some defined entity, not just as being aspirational. For this reason, Healthy People targets tend not to be the ideal source of benchmarks for the national healthcare reports. While the inclusion of these targets may be useful and warranted as one point of information, they should be presented in conjunction with more realistic benchmarks. Presenting Best-in-Class Benchmarks One of the most common and easily understood methods of benchmarking is to provide comparisons rela- tive to top performing nations, states, geographic regions, or health care entities. A key issue in benchmarking is whose performance is being measured and to which audiences the benchmark is relevant. In health care quality improvement, best practices can occur at various levels of the health care system, including at the individual phy- sician level (Kiefe et al., 2001); at the service provision level, such as in intensive care units (Zimmerman et al.,  The HealthyPeople 2010 targets are, in almost all cases, higher than the currently achieved national progress or even the best performing state. For some measures presented in the NHQR and NHDR, however, performance is at or above the Healthy People target. For example, the composite measure for children ages 19-35 months who received all recommended vaccines includes the Healthy People target of 80 percent attainment. The national average for this measure was at 80.6 percent, achieving this target. 

124 NATIONAL HEALTHCARE QUALITY AND DISPARITIES REPORTS 2003); at the health care system level; or at the state level (Reintjes et al., 2007). The committee also explored establishing benchmarks at discrete levels of the health care system (e.g., top decile of hospitals), as well as at the state level. Defining a benchmark can depend on the “class” from which the measure is derived. For example, a benchmark might provide information on the best performance rate among states, the best performance rate among hospitals, the best performance rate among large hospitals, or the best performance rate for care received by Hispanics in any state. Although it is technically true that AHRQ could choose any “class” from which it would designate a “best- in-class” benchmark, the committee finds that in the context of the national healthcare reports, where much of the analysis is done at the state level, setting benchmarks by state may appeal to a number of relevant audiences and may be most feasible given data availability. State-level data are generally available to AHRQ, and thus state-level benchmarking units can be determined for many, although not all, measures in the NHQR and NHDR. This approach could satisfy the needs of congressional and state policy makers, principal audiences to which the reports are geared. A 2004 AHRQ publication A Resource Guide for State Action was designed to help states assess the quality of care in their states and develop strategies to address gaps in quality. The Resource Guide advised that the “rate for the top State or top tier of States” may be “assumed to be a feasible goal for States to achieve” (Coffey et al., 2004). Figure 6-1 shows that it is possible to display a best-in-class benchmark (a state in this instance) along with the national performance average and the Healthy People target. The committee does not intend that the style, format, and layout of this figure be adopted by AHRQ; rather, the committee presents this figure to show the relationship of a benchmark relative to the type of performance data that are in the domain of the NHQR or NHDR. From the per- spective of the NHQR, which tends to provide state-based data as well as national average performance, the highest performing state, Oregon, provides a benchmark that could be applied across both reports. 100% Healthy People 90% 2010 target: 90% 80% Oregon: 74.3% National benchmark: 70% 74.3% Non- High Hispanic, income: White: 60% White: 61.9% 60.9% National 59.9% average: DC: 57.3% 50% 52% Poor: 45.1% 40% Black: 36.8% Hispanic: 30% 33.2% 20% FIGURE 6-1  Oregon’s performance rate for pneumococcal vaccination sets a national benchmark for other states to strive to achieve. No race, Hispanic ethnicity, or income group on a national level achieves the vaccination rate of Oregon. NOTE: Percentage of individuals age 65+ who ever received a pneumococcal vaccination, 2006. SOURCE: Adapted from data in the AHRQ State Snapshots. Figure 6-1 R01677 editable vectors 

IMPROVING PRESENTATION OF INFORMATION 125 The committee recognizes that some measures and their corresponding data sources may be amenable to choosing a different benchmarking “class” than a state. A measure that uses only HEDIS data may, for instance, lend itself to analyzing data by health plans. Thus, AHRQ could present a best performing plan as the benchmark. Similarly, AHRQ might decide that the hospitals comprising the HCUP datasets constitute a comparable set of observations and could present a best performing hospital as the benchmark. Denoting a best-in-class benchmark is as important for measures in the NHDR as it is for measures in the NHQR, and the committee concludes that for each measure, the benchmark used in the NHDR should mirror the benchmark used in the NHQR. The goal of quality improvement efforts should not be to strive just for the Hispanic population to receive care at the rate of the non-Hispanic population. Rather, quality improvement efforts should aim to improve the quality of care for all populations. In the case of the NHDR, different disparity populations would be compared against the quality benchmark in addition to being compared against the best performing popu- lation. For example, AHRQ may establish a state-based benchmark for a specific measure of lipid control and use this same benchmark in both the NHQR and NHDR. The committee recognizes that reporting in the NHDR which state has the best rate on lipid control by specific populations would be useful (e.g., reporting that X state has the highest performance level for Hispanics and Y state has the highest performance level for African Americans), but such data are not always available. Adopting a separate benchmark based on the best performing population group within a “class” can prove difficult as there are multiple population groups studied in the NHDR and detailed data are not always available or sample sizes may be too small to stratify population data by hospital, health plan, or even state. Ideally, data would be available for sociodemographic descriptors within whichever class a benchmark was being set; when they are not, this leads to looking to an alternate solution for presenting a benchmark in the NHDR. The committee advises AHRQ that the benchmark can be the best performing state or can come from the class of units compared in the measure’s data source. When the data are available, the committee encourages AHRQ to present multiple population-specific benchmarks (i.e., a benchmark that is uniform with the NHQR as well as other benchmarks that are population specific). When multiple achievement levels are available, alterna- tives to presenting the data graphically may be needed (e.g., listing in textboxes). The committee encourages the analysis of performance data by accountable units (e.g., states, health plans, hospitals). When it is feasible for AHRQ to analyze data for a measure by multiple accountable units, there is the possibility for multiple benchmarks of attained performance for one specific measure. Presenting multiple benchmarks might add clutter to graphs, so AHRQ may choose to present the multiple achievement levels in a sidebar text box. The Future Directions committee believes benchmarks provide a means to supply concrete milestones for comparison and evaluation. For comparative purposes, having a uniform benchmarking unit such as a state may be useful, although other classes (e.g., plans, hospitals) may be informative for entities implementing programs to improve quality and eliminate disparities. Thus, the committee recommends: Recommendation 7: To the extent that the data are available, the reporting of each measure in the NHQR and NHDR measure set should include routinely updated benchmarks that represent the best known level of performance that has been attained. Data Limitations in Benchmarking As discussed above, AHRQ could present data on a high-performing entity for which data are available (e.g., the best performing health plans based on data from the National Committee for Quality Assurance). This approach, however, may require particular attention to issues of statistical reliability. The population distribution from which a benchmark is derived must be considered carefully so that entities are not evaluated against a population that is not well-matched to their particular case-mix, geography, or other relevant factors (Linderman et al., 2006). When the population of analysis includes high-performing entities that have a small number of cases, the analysis must be corrected to account for the small-numbers problem (Normand et al., 2007). There are techniques—including the Achievable Benchmarks of Care method, which uses a Bayesian estimator to reduce the impact of entities with a small number of eligible patients—that AHRQ could use to adjust for the small denominator problem (i.e., if a 

126 NATIONAL HEALTHCARE QUALITY AND DISPARITIES REPORTS plan had only one qualifying patient, then the performance of that plan could be either 0 percent or 100 percent) (Weissman et al., 1999). As an additional consideration, data on state performance may be unavailable for all measures. Although the State Snapshot website does not include state data for 26 of AHRQ’s 46 core measures, the committee finds it feasible for AHRQ to obtain state data for some of these (e.g., access measures, measures from Centers for Disease Control and Prevention data).10 Furthermore, for measures in which data on the best performing state are available, not all states may have reported on the measure or been included in analysis (e.g., the Healthcare Cost and Utilization Project). Therefore, the best performing state may actually be the “reported best performing state.” AHRQ may consider recognizing this in either introductory text or in a footnote. For many measures of health care quality, even the highest performing state, population, or provider does not deliver the level of care recommended in guidelines. Benchmarking within a field of low performers may result in further underperformance because low performance is seen as normal (Reinertsen and Schellekens, 2005). AHRQ should take this into consideration when determining the class from which to derive a benchmark and should ensure the benchmark represents a desirable level of performance. REFINING THE PRESENTATION OF DATA The success of the NHQR and NHDR in reaching various audiences and spurring action depends on the presentation of information. The products developed by AHRQ have the potential to tell a more complete quality improvement story, provided the products are accessible, engaging, and informative. Improving the Presentation of Graphic Displays Graphic displays in the NHQR and NHDR document historical trends, present geographic variation using maps, and stratify measures by demographic characteristics. Communicating information through the simultane- ous presentation of words, numbers, and, in some cases, pictures, requires that the displays be as effective as possible (Tufte, 1983). Therefore, visual design problems can undermine the usefulness of data being presented (Few, 2006). To assess the quality of graphic displays in the two reports, the committee commissioned input from Howard Wainer, an expert on data display.11 The suggestions presented to AHRQ are only one way of enhancing data displays, but they represent well-regarded, theory-based practice. Documenting Historical Trend Data As noted earlier, the committee finds there should be less reliance in the NHQR and NHDR on trended data displays unless the trends inform future activities rather than solely document the past. Currently, most trend data take up substantial space in the documents without being particularly informative other than to reinforce a repeated message—that the pace of change is slow. These trend graphs are often visually cluttered with overlapping lines and many numbers over-written on a graph (see Box 6-3). Captioning and Labeling Captions for each display should be informative and focused. A good graph can be made even stronger by having an informative and interpretive caption or figure heading. Captions that explicitly relay the principal point of the display have benefits: the reader can discover the point of the display more easily and less helpful displays are eliminated. Additionally, strong graphical displays avoid legends whenever possible because legends require 10 Because the State Snapshots were initially developed to supplement measures in only the NHQR, access measures are not included in the State Snapshots; in accordance with the committee’s recommendation to integrate access in the quality portfolio of measures, it is important for AHRQ to include access measures in the State Snapshots. 11 Howard Wainer’s paper, “Commentaries on the 2008 National Healthcare Quality Report, the 2008 National Healthcare Disparities Report and State Snapshots,” was provided directly to AHRQ staff and archived in the IOM public access file for the Future Directions project. 

IMPROVING PRESENTATION OF INFORMATION 127 BOX 6-3 A Suggested Approach to Improving Data Displays Figure A (both before and after) presents data on patients with tuberculosis who completed treatment, but the original Figure A (below left) is visually cluttered: the graph contains a multitude of data points, a y-axis that is not descriptive, a caption that does not convey findings, and labels that are far from the data points. The reader is less readily able to discern which age groups performed best, which performed worst, and if any age groups had improved in the percentage of patients that received the recommended care. To more clearly convey information, Figure A (below right) was revised so that the reader can more readily gain a sense of the component data without having to append the “visual noise of numerical values.” These improvements include: • e liminating the numerical value from each data point (the numerical values should be archived online if they are not in the text). Data for the terminal year should be provided in text if not inserted into the graph; • less compression of the y-axis makes the graph more readable; and • adjusting the scales on the x- and y-axis to reflect the data distribution and provide maximum acuity. Additionally, instead of a heading that defines the measure specifications, the heading was changed to an informative caption that conveys the graph’s key finding: “Although rates of completion of tuberculosis treatment have been increas- ing overall, adults are 10% less likely than children to complete treatment.” When AHRQ determines the scales of the x- and y-axes, the purpose of the graph should be taken into consideration. For example, in the “after” figure below, the compressed y-axis scale may exaggerate differences between age groups. However, the compression allows readers to more easily determine the best and worst performing groups. AHRQ should weigh these considerations and consider the absolute level of performance when choosing axes scales. BEFORE AFTER Figure A. Patients with tuberculosis who com- Figure A. Although rates of completion of tu- pleted a curative course of treatment within 1 berculosis treatment have been increasing year of initiation of treatment, by age group, overall, adults are 10% less likely than chil- 1998-2004. dren to complete treatment. Percent with Completed Treatment 90 age 0-17 85 age 65 and over ALL AGES age 45-64 80 age 18-44 75 1998 1999 2000 2001 2002 2003 2004 SOURCE for the “Before” Graph: AHRQ, 2009e, p. 83. Box 6-3, “after” figure Box 6-3, “first” figure R01677 R01677 editable vectors uneditable bitmapped image 

128 NATIONAL HEALTHCARE QUALITY AND DISPARITIES REPORTS the viewer to learn the legend and apply it to the display. This requires two moments of perception and makes the viewer read the display rather than see it (Green, 1998). See Boxes 6-4 and 6-5 for examples. Scales The choice of x- and y-axis scales can influence the readability and interpretability of a graph. The x- and y-axis should place observed differences on a scale that acknowledges the range of possible clinically important differences. In a series of experiments conducted at AT&T Bell Laboratories, Cleveland and colleagues determined that scales should be chosen so that the data fill as much of the scale-line rectangle as possible (Cleveland, 1994a,b). A separate issue that must be considered is the choice of the ratio of the x- and y-axis scales. Altering the ratio BOX 6-4 A Suggested Approach to Improving the Labeling of Graphic Data Figure B (both before and after) presents data on rates of pressure sores among nursing home residents. The original figure (below left) could better convey its key finding (that although rates of pressure sores have been declining, Black residents still have higher rates compared to all other racial groups) with modifications: • T he display should have a caption that interprets the data as opposed to simply naming the measure. • The graphic elements should be labeled directly (i.e., legends should be used as infrequently as possible). Terminal year data should be included in text if AHRQ decides not to include it in the graph. BEFORE AFTER Figure B. High-risk, long-stay nursing home Figure B. Although rates of pressure sores residents with pressure sores, by race/ among high-risk, long-stay nursing home res- ethnicity, 1999-2006. idents have decreased in recent years, Black residents still have higher rates compared to all other racial groups. 20 18 Black Nursing home residents with pressure sores (%) 16 AIAN 14 Hispanic TOTAL 12 White API 10 National 8 bench- mark: 7.5% 6 4 2 0 1999 2000 2001 2002 2003 2004 2005 2006 SOURCE for the “Before” Graph: AHRQ, 2009d, p. 84. Box 6-4, “after” figure Box 6-4, “first” figure R01677 R01677 editable vectors uneditable bitmapped image 

IMPROVING PRESENTATION OF INFORMATION 129 BOX 6-5 An Example of a Complex Data Display An informative heading or caption should explain what constitutes better performance. The measure captions cur- rently used as headings in the NHQR and NHDR do not always indicate whether better performance is associated with a positive percentage change or a negative percentage change. As shown in Figure C, below left, without reading the supporting text for this figure, a reader might not readily grasp that being on a transplant waiting list for a dialysis patient is a positive thing and that a high percentage is desirable. BEFORE AFTER Figure C. Dialysis patients under age 70 who Figure C. Although standards of care recom- were registered on a waiting list for transplan- mend that dialysis patients (under age 70) be tation, by age group, 1999-2004. registered for transplantation, older patients are less likely than younger patients to be registered. 50 Dialysis patients registered for transplantation (%) 45 40 age 0-19 35 30 age 20-39 25 Healthy People target: 25% 20 age 40-59 15 ALL AGES 10 age 60-69 5 0 1998 1999 2000 2001 2002 2003 2004 SOURCE for the “Before” Graph: AHRQ, 2009e, p. 50. Box 6-5, “after” figure Box 6-5, “first” figure R01677 R01677 editable vectors uneditable bitmapped of the scale can modify a person’s image perception of the data (Cooper et al., 2003; Schriger and Cooper, 2001). The committee encourages AHRQ to choose an aspect ratio that appropriately conveys the data. Alternative Data Displays In addition to displaying trend data in graphs, AHRQ might consider utilizing alternate visual displays. For example, Figure 6-2 below, which was created by the CDC, succinctly presents information to readers, including readers who may not be data experts. The figure could be further improved by specifying whether the symbols represent absolute numbers of infected people or a rate ratio. In creating the figure, CDC likely meant for the figure to represent a rate ratio; however, readers may draw the conclusion that seven times more African Americans are infected than Whites (an absolute count). The display could be made clearer by including 100 small symbols for each group, and coloring in 7 for the African American population, 2.5 for the Hispanic population, and 1 for the White population. The committee recognizes that there are benefits to readers in using a small number of graphic formats with the same type of display from page-to-page, so that readers do not have to learn to interpret a new type of graph, 

130 NATIONAL HEALTHCARE QUALITY AND DISPARITIES REPORTS HIV Infections Rate African Americans Hispanics Whites African Americans experience new HIV infections at SEVEN TIMES the rate of Whites, and Hispanics experience new HIV infections at TWO AND A HALF TIMES the rate of Whites. FIGURE 6-2  Example of an alternate visual display. SOURCE: HHS, 2009a. Figure 6-2 R01677 but finds that some diversity of presentation can enhance a report. Alternative displays might be particularly useful for the Highlights section and fact sheets. redrawn with editable text The legends on the maps in the NHQR and NHDR are often uninformative as they are simply above aver- age, average, below average, and no data. For ease of comprehension, the labels might, at a minimum, contain numeric values (averages and ranges). Additionally, ordering by performance level achieved makes a coherent visual impact and suggests an implicit underlying structure. For example, gradations of a single color would better show performance levels on maps (see Figure 6-3 below). A sequence of “increasing darkness” of a single color can assist the reader in identifying increasing or decreasing rates, as utilized by Pickle and colleagues over five gradients (Pickle et al., 1996). Additionally, colors should be chosen to avoid common color vision deficiencies and so that no single color visually dominates (Pickle et al., 1996). In the NHQR and NHDR maps, the color black represents better performance and AHRQ’s use of two other colors (green and blue) does not have the visual impact of a single color gradient. Examples of such visual displays abound, and the committee believes that AHRQ may benefit from additional professional consultation on how to better present its data. Enhancing the Supporting Text for Data Displays The text supporting a data display should convey information gleaned from data analysis, such as analysis not captured in the figure and implications of significant findings. Currently, supporting text for displays in the NHQR and NHDR describes what the graph depicts. The text refrains from providing additional analyses and provides minimal direction on methods that could be used to improve quality or eliminate disparities. Refining the Presentation of Summarized Information Summary and composite measures are useful tools for conveying information about complex constructs, such as the multiple elements of appropriate care for a stage of life (e.g., end-of-life care) or a condition that is 

IMPROVING PRESENTATION OF INFORMATION 131 NYC Age-specific rate per 100,000 population >53.0 – 63.8 >47.8 – 53.0 >42.3 – 47.8 >37.4 – 42.3 28.7 – 37.4 FIGURE 6-3  Illustration of a gradient shading scheme. SOURCE: Pickle et al., 1996. inadequately portrayed through a single measure (e.g., diabetes). To be consistent with AHRQ’s use of the terms composite and summary measures, this report definesFigure composite 6-3 measures as the bundling of two or more mea- sures that look at different aspects of care for a specific clinical condition (AHRQ, 2008b). 12 As an example, the composite measure on diabetes care isredrawn the percentagefrrom source of adults age 40 and over with diabetes who received all to replace low-resolutioneye three recommended services (hemoglobin A1c measurement, dilated bitmap examination, foot examination). Summary measures bundle a number of conceptually similar specific measures of health care services or outcomes across multiple conditions or health care settings in order to present a single metric for a given aspect of health care delivery (e.g., combining performance rates for all prevention measures). AHRQ’s State Snapshots present such summary measures to report the performance of single and combined states on measures for differ- ent types of care (i.e., preventive, acute treatment, chronic care) and settings of care (i.e., home health, hospital, nursing home, ambulatory) (AHRQ, 2009a). Similarly, AHRQ summarizes measures in the Highlights section by core measure totals, types of settings, and types of clinical measures (including some clinical conditions across composite and individual measures). The committee’s purpose is not to recommend specific composite or summary measures for inclusion in the national healthcare reports; rather, the committee considers desirable properties that AHRQ may consider when evaluating the way in which such measures are reported. A principal consideration in the use of a composite or summary measure is the quality of the individual measures being inputted and the relationship of these measures to one another (Murray et al., 2000). The weight of the measures that comprise the composite or summary measure may need to be considered. AHRQ does not use differential weights in its composite and summary measures; rather, it weighs every component measure equally. Implicit in choosing weights are subjective judgments about the relative clinical significance and prioritization of the component measures. AHRQ should clearly denote that composite and summary measures use equal weights and provide the denominators for each component measure (in an appendix, for instance) so that users of the data can perform their own analyses using differential weights, if they so choose (Martinez-Vidal and Brodt, 2006). 12 Ten out of the 12 reported composite measures in the reports involve the bundling of process measures, while the other 2 involve outcome measures for surgical procedures. 

132 NATIONAL HEALTHCARE QUALITY AND DISPARITIES REPORTS Presenting the Methodology of Summary and Composite Measures Several standard pieces of information should accompany any composite or summary measure. While such information need not be displayed in the main body of a report, it should appear at least as an appendix, including: • the methodological considerations taken into account when creating a composite or summary measure (e.g., how the measure is weighted); • a description of the individual constituent measures that make up the composite or summary measure, their data source, and the distribution (e.g., means, standard deviations, ranges, floor and ceiling effects); • a summary description of the psychometric properties of the composite measure, including how the com- ponent measures relate to each other (i.e., the pair-wise correlation coefficients of the individual quality measures or a coefficient alpha); and • the standard error of the composite measure, in addition to the estimated composite measure. The general methodology for the composite measures presented in the 2008 NHQR is discussed in the print report (AHRQ, 2009e, p. 20), and some measure specifications for composites included in the NHQR and NHDR are provided via online appendixes.13 However, the appendixes do not contain all of the information outlined above. For instance, data on the individual constituent measures for the reported composite measures are some- times unavailable or not easily accessible. Likewise, some methodological information is provided online for the summary measures used in the State Snapshots,14 but much of the above information is also missing for those measures. For example, the first figure presented in the 2008 NHQR (AHRQ, 2009e, p. 3) pools trend data from quality measures to quantify the overall change in quality for the health care system according to the measures AHRQ has chosen to profile. The median annual rate of change from baseline to most recent data year is reported as 1.4 percent. The NHQR does not, however, report the distribution of the underlying rates of change across measures, including the distribution and variability of the underlying rates. While it is important to know how many indica- tors are getting better and how many are getting worse, standard errors and correlations in rates of change are essential to identifying which measures tend to improve or worsen together. The committee recognizes the benefits of using composite measures and summarization techniques, and AHRQ staff should continue to identify measurement areas that can benefit from such presentation. However, the commit- tee finds that AHRQ needs to be more transparent in its methods. Methodological information may be presented in the print and online reports, although such detail may be more appropriate for appendixes where researchers who need such facts can obtain them. Enhancing the Summary Dashboards of the State Snapshots Dashboards are a valuable tool for efficiently and effectively communicating summarized information (Few, 2006). AHRQ utilizes this technique to provide a picture of how a state is performing relative to other states on “overall health care quality” and for 12 topics across types of care (i.e., prevention, acute, and chronic), settings of care, and clinical conditions.15 Despite the intended purpose of simplistically conveying information, the state dashboards may confuse users. For instance, Montana appears to be doing worse today than in the baseline year, although performance may or may not be better than in the past. The Montana dashboard does not say the arrows on the meters are reflecting relative performance, nor does it have a statement such as, “Montana’s overall perfor- 13 The 2008 measure specifications are accessible at http://www.ahrq.gov/qual/qrdr08/measurespec/ (accessed January 15, 2010). 14 Accessibleby visiting the Methods section of the State Snapshots at http://statesnapshots.ahrq.gov/snaps08/Methods.jsp?menuId=58&state= #stateSummary (accessed January 15, 2010). 15 See the Montana dashboard at http://statesnapshots.ahrq.gov/snaps08/dashboard.jsp?menuId=4&state=MT&level=0 (accessed December 8, 2009). 

IMPROVING PRESENTATION OF INFORMATION 133 mance is better in the most recent data year than its baseline performance, but other states have improved more, so its overall performance ranks lower than previously.” In the State Snapshots, Arkansas is positively rated for having a low disparity rate. 16 This rating, however, may not reflect better outcomes. The low disparity rate is principally because the performance metrics of Arkansas’s White population are lower than the corresponding data for the White population in other states. Meanwhile, the quality data attributed to Arkansas’s Black or African American population are in line with the corresponding national measures. Thus, lower quality metrics associated with both White and Black individuals in Arkansas results in a smaller difference between the two populations (and thus a smaller disparity). 17 Statistical Quality of Data Reporting Given the volume and numerous sources of reported measures, there are challenges in providing clear and useful information to readers. However, clearly stating the analytic methodology for the reports and making this methodology more readily available is important for the researchers, as they may seek to manipulate the data for their own purposes, or look to replicate such measurement reporting. Providing such methodological information also enhances the transparency of the NHQR and NHDR. For three sections of the NHQR or NHDR, the committee assessed (1) measurement properties and defini- tions of quality indicators, (2) the description and use of analytical adjustments, (3) methods of summarization, (4) selection and use of benchmarks, and (5) use of prediction rules. (See Appendix H for additional information.) The committee’s review indicated that, when possible, AHRQ should make available online the following supple- mentary information to inform the research community: • Data quality. Information regarding who collected the data, the reliability and validity of collected data, limitations of the data, and the extent of missing data should be reported. While this information may be difficult to gather, the quality of the NHQR and NHDR hinge on the quality of the data. A standard template could be constructed and populated, and when information cannot be determined, at a minimum, this fact could be stated. • Description and use of analytical adjustments. Key features of analytical adjustment are required for read- ers to understand and correctly interpret findings. These features include a clear definition of the outcome (including the units of measurement); the observed covariates and definitions used in adjustment; justifica- tion for adjustment and how the adjustment was made; the sample sizes or weights used in the analysis; the reference population used; and how well the statistical model performed (fit) for adjustment. • Summary measures. The choice and definition of methods of summarization should be made explicit. For example, if the summary measure is a change in performance from one time period to the next, the time periods need to be stated; the estimate should be defined (regression-based coefficient or difference in means); and the statistical significance or other metric for displaying uncertainty in the estimate should be provided. • Prediction rules. In some instances, prediction inferences for when a particular goal will be achieved are made. In such instances, the statistical model used for the prediction should be stated, its fit assessed rela- tive to reasonable competitor models, and the statistical uncertainty surrounding the prediction should be reported. One prediction would be the number of years to reach a particular benchmark at the current rate of change. 16 See Arkansas: Focus on Disparities at http://statesnapshots.ahrq.gov/snaps08/disparities.jsp?menuId=47&state=AR&level=83 (accessed December 20, 2009). 17 Personal communication, William Golden, University of Arkansas for Medical Sciences and Arkansas Medicaid, Department of Human Services, December 8, 2009. 

134 NATIONAL HEALTHCARE QUALITY AND DISPARITIES REPORTS Conclusions on Data Presentation Taking advantage of the full power of data displays and concise summarization will be critical for AHRQ to continue to streamline a vast amount of information. To strengthen data presentation in the reports, the committee recommends: Recommendation 8: AHRQ should engage experts in communications and in presentation of statisti- cal and graphical information to ensure that more actionable messages are clearly communicated to intended audiences, summarization methods and the use of graphics are meaningful and easily understood, and statistical methods are available for researchers using data. SUMMARY The data presented in the NHQR, NHDR, and their related products need to provide clear and coherent mes- sages about the state of health care and the level of quality that has been achieved. The reports should strive to promote actionability by relaying realistic benchmarks and leading users to resources that illuminate methods of quality improvement and disparities elimination. As discussed, AHRQ can explore various dissemination strategies to ensure the messages are effectively conveyed to relevant audiences. By employing the messaging and presenta- tion strategies discussed in this chapter, the NHQR and NHDR may be more valuable to a wider spectrum of users while still presenting data and methods useful to researchers in the field. REFERENCES AHRQ (Agency for Healthcare Research and Quality). 2005a. Fact sheet: Health care disparities in rural areas: Selected findings from the 2004 National Healthcare Disparities Report. Rockville, MD: Agency for Healthcare Research and Quality. ———. 2005b. Fact sheet: Selected findings on child and adolescent health care from the 2004 National Healthcare Quality/Disparities Re- ports. Rockville, MD: Agency for Healthcare Research and Quality. ———. 2005c. Fact sheet: Women’s health care in the United States: Selected findings from the 2004 National Healthcare Quality and Dispari- ties Reports. Rockville, MD: Agency for Healthcare Research and Quality. ———. 2008a. Fact sheet: Selected findings on child and adolescent health care from the 2007 National Healthcare Quality and Disparities Reports. Rockville, MD: Agency for Healthcare Research and Quality. ———. 2008b. National Healthcare Quality Report, 2007. Rockville, MD: Agency for Healthcare Research and Quality. ———. 2009a. 2008 State Snapshots. http://statesnapshots.ahrq.gov/snaps08/index.jsp (accessed July 22, 2009). ———. 2009b. Fact sheet: Disparities in children’s health care quality: Selected examples from the National Healthcare Quality and Dispari- ties Reports, 2008. Rockville, MD: Agency for Healthcare Research and Quality. ———. 2009c. Introductory remarks and charge to the subcommittee: Slide 9. Agency for Healthcare Research and Quality. Presentation to the AHRQ NAC Subcommittee on Children’s Healthcare Quality Measures for Medicaid and CHIP Programs, July 22, 2009. Rockville, MD. ———. 2009d. National Healthcare Disparities Report, 2008. Rockville, MD: Agency for Healthcare Research and Quality. ———. 2009e. National Healthcare Quality Report, 2008. Rockville, MD: Agency for Healthcare Research and Quality. Brooke, B. S., B. A. Perler, F. Dominici, M. A. Makary, and P. J. Provonost. 2008. Reduction of in-hospital mortality among California hospitals meeting Leapfrog evidence-based standards for abdominal aortic aneurysm repair. Journal of Vascular Surgery 47(6):1155-1164. Camp, R. 1998. Benchmarking: The search for industry best practices that lead to superior performance. Milwaukee, WI: American Society for Quality Press. Chernew, M., and D. P. Scanlon. 1998. Health plan report cards and insurance choice. Inquiry 35(1):9-22. Chin, M. H., and A. T. Chien. 2006. Reducing racial and ethnic disparities in health care: An integral part of quality improvement scholarship. Quality and Safety in Health Care 15(2):79-80. Clarke, C. L., J. Reed, D. Wainwright, S. McClelland, V. Swallow, J. Harden, G. Walton, and A. Walsh. 2004. The discipline of improvement: Something old, something new? Journal of Nursing Management 12(2):85-96. Cleveland, W. S. 1994a. The elements of graphing data. Summit, NJ: Hobart Press. ———. 1994b. Visualizing data. Summit, NJ: Hobart Press. Coffey, R. M., T. L. Matthews, and K. McDermott. 2004. Diabetes care quality improvement: A resource guide for state action. Rockville, MD: Agency for Healthcare Research and Quality. Cook, S. 1995. Practical benchmarking: A manager’s guide to creating a competitive advantage. London: Kogan Page. Cooper, R. J., D. L. Schriger, R. C. Wallace, V. J. Mikulich, and M. S. Wilkes. 2003. The quantity and quality of scientific graphs in pharmaceuti- cal advertisements. Journal of General Internal Medicine 18(4):294-297. 

IMPROVING PRESENTATION OF INFORMATION 135 The Dartmouth Institute for Health Policy & Clinical Practice. 2010. The Dartmouth Atlas of Health Care. http://www.dartmouthatlas.org/ (ac- cessed February 23, 2010). Dattakumar, R., and R. Jagadeesh. 2003. A review of literature on benchmarking. Benchmarking: An International Journal 10(3):176-209. Davies, H. T., A. E. Washington, and A. B. Bindman. 2002. Health care report cards: Implications for vulnerable patient groups and the organi- zations providing them care. Journal of Health Politics, Policy and Law 27(3):379-400. Deming, W. E. 1994. The new economics for industry, education, government—2nd Edition. Cambridge, MA: Massachusetts Institute of Tech- nology Center for Advanced Engineering Study. Epstein, A. 1995. Performance reports on quality—prototypes, problems, and prospects. New England Journal of Medicine 333(1):57-61. Few, S. 2006. Information dashboard design: The effective visual communication of data. Sebastopol, CA: O’Reilly Media. Fisher, E. S., D. C. Goodman, and A. Chandra. 2008. Regional and racial variation in health care among Medicare beneficiaries: A brief report of the Dartmouth Atlas Project. Princeton, NJ: Robert Wood Johnson Foundation. Gawande, A., D. Berwick, E. Fisher, and M. McClellan. 2009. 10 steps to better health care. The New York Times, August 13, 2009, A27. Gold, M., and R. Nyman. 2004. Evaluation of the development process of the National Healthcare Quality Report. Washington, DC: Math- ematica Policy Research, Inc. Green, M. 1998. Toward a perceptual science of multidimensional data visualization: Bertin and beyond. http://graphics.stanford.edu/courses/ cs448b-06-winter/papers/Green_Towards.pdf (accessed May 13, 2010). HHS (U.S. Department of Health and Human Services). 2009a. Health disparities: A case for closing the gap. http://www.healthreform.gov/ reports/healthdisparities/disparities_final.pdf (accessed February 18, 2010). ———. 2009b. Office of Minority Health Resource Center. http://raceandhealth.hhs.gov/templates/browse.aspx?lvl=1&lvlID=8 (accessed February 4, 2010). ______. 2010. Community health status indicators project. http://www.communityhealth.hhs.gov (accessed March 26, 2010). Hibbard, J., and J. Jewett. 1997. Will quality report cards help consumers? Health Affairs 16(3):218-228. IOM (Institute of Medicine). 2001. Envisioning the National Healthcare Quality Report. Washington, DC: National Academy Press. ———. 2002. Guidance for the National Healthcare Disparities Report. Washington, DC: The National Academies Press. ———. 2010. Pediatric health and health care quality measures. http://www.iom.edu/Activities/Quality/PediatricQualityMeasures.aspx (ac- cessed January 4, 2010). Kerr, E. A., E. A. McGlynn, J. Adams, J. Keesey, and S. M. Asch. 2004. Profiling the quality of care in twelve communities: Results from the CQI study. Health Affairs 23(3):247-256. Kiefe, C. I., N. W. Weissman, J. J. Allison, R. Farmer, M. Weaver, and O. D. Williams. 1998. Identifying achievable benchmarks of care: Con- cepts and methodology. International Journal for Quality in Health Care 10(5):443-447. Kiefe, C. I., J. J. Allison, O. D. Williams, S. D. Person, M. T. Weaver, and N. W. Weissman. 2001. Improving quality improvement using achievable benchmarks for physician feedback: A randomized controlled trial. Journal of the American Medical Association 285(22):2871-2879. Kleine, B. 1994. Benchmarking for continuous performance improvement: Tactics for success. Total Quality Environmental Management 3(3):283-295. Lansky, D. 2009. Priorities for quality improvement and ways to leverage collected information: An employer perspective. Pacific Business Group on Health. Presentation to the IOM Committee on Future Directions for the National Healthcare Quality and Disparities Reports, March 12, 2009. Newport Beach, CA. PowerPoint Presentation. Linderman, K., R. G. Schroeder, and A. S. Choo. 2006. Six Sigma: The role of goals in improvement teams. Journal of Operations Manage- ment 24(6):779-790. Martinez-Vidal, E., and A. Brodt. 2006. State recommendations for revisions to the AHRQ Snapshots. Washington, DC: AcademyHealth. McKeon, T. 1996. Benchmarks and performance indicators: Two tools for evaluating organizational results and continuous quality improvement efforts. Journal of Nursing Care Quality 10(3):12-17. Moy, E. 2009. Lessons learned in developing NHQR and NHDR. Agency for Healthcare Research and Quality. Presentation to the IOM Com- mittee on Future Directions for the National Healthcare Quality and Disparities Reports, February 9, 2009. Washington, DC. PowerPoint Presentation. Moy, E., E. Dayton, and C. M. Clancy. 2005. Compiling the evidence: The National Healthcare Disparities Reports. Health Affairs 24(2): 376-387. Murray, C. J., J. A. Salomon, and C. Mathers. 2000. A critical examination of summary measures of population health. Bulletin of the World Health Organization 78(8):981-994. Normand, S. L., R. E. Wolf, J. Z. Ayanian, and B. J. McNeil. 2007. Assessing the accuracy of hospital clinical performance measures. Medical Decision Making 27(1):9-20. Pickle, L. W., M. Mungiole, G. K. Jones, and A. A. White. 1996. Atlas of United States mortality. Hyattsville, MD: National Center for Health Statistics. Reinertsen, J., and W. Schellekens. 2005. 10 powerful ideas for improving patient care: What every healthcare executive should know. Chicago, IL: Health Administrative Press. Reintjes, R., M. Thelen, R. Reiche, and Á. Csohán. 2007. Benchmarking national surveillance systems: A new tool for the comparison of com- municable disease surveillance and control in Europe. European Journal of Public Health 17(4):375-380. Romano, P. S., J. A. Rainwater, and D. Antonius. 1999. Grading the graders: How hospitals in California and New York perceive and interpret their report cards. Medical Care 37(3):295-305. 

136 NATIONAL HEALTHCARE QUALITY AND DISPARITIES REPORTS RWJF (Robert Wood Johnson Foundation) and the University of Wisconsin Population Health Institute. 2010. County health ratings. http:// www.countyhealthrankings.org/ (accessed February 23, 2010). Schriger, D. L., and R. J. Cooper. 2001. Achieving graphical excellence: Suggestions and methods for creating high-quality visual displays of experimental data. Annals of Emerging Medicine 37(1):75-87. Social & Scientific Systems, and UserWorks. 2009. Final report: Usability evaluation of the Agency for Healthcare Research and Quality (AHRQ) QRDR web sites. Silver Spring, MD: Social & Scientific Systems, Inc. and UserWorks, Inc. Tufte, E. R. 1983. The visual display of quantitative information. Cheshire, CT: Graphics Press. van Herten, L. M., and L. J. Gunning-Schepers. 2000a. Targets as a tool in health policy. Part II: Guidelines for application. Health Policy 53(1):13-23. ———. 2000b. Targets as a tool in health policy: Lessons learned. Health Policy 53(1):1-11. Vaziri, K. 1992. Using competitive benchmarking to set goals. Quality Progress 25(10):81-85. Weissman, N. W., J. J. Allison, C. I. Kiefe, R. Farmer, M. T. Weaver, O. D. Williams, I. G. Child, J. H. Pemberton, K. C. Brown, and S. Baker. 1999. Achievable benchmarks of care: The ABCs of benchmarking. Journal of Evaluation in Clinical Practice 5(3):269-281. Zimmerman, J. E., C. Alzola, and K. T. Von Rueden. 2003. The use of benchmarking to identify top performing critical care units: A preliminary assessment of their policies and practices. Journal of Critical Care 18(2):76-86.