National Academies Press: OpenBook
« Previous: 1 Introduction and Context
Suggested Citation:"2 Criteria for Selecting Measures." Institute of Medicine. 2013. Toward Quality Measures for Population Health and the Leading Health Indicators. Washington, DC: The National Academies Press. doi: 10.17226/18339.
×

2

Criteria for Selecting Measures

The committee was asked to describe methods for selecting quality measures for the Leading Health Indicators (LHIs) and for the work of improving the public’s health more broadly. In this chapter, the committee offers guidance for selecting measures to assess and improve quality in the multisectoral health system, and especially among health care and public health participants.

To make progress toward long, healthy lives for all, pertinent stakeholders need to create healthy communities and address the underlying factors that influence health outcomes and disparities in those outcomes among different subgroups. Measurement is an essential ingredient of such efforts. The committee believes that a summary measure of population health is needed along with or as a part of a set of measures of quality. Such a measure is needed for every community and must be consistently defined in order for public health officials and members of the public to understand the quality of their public health system and to compare it with national and other appropriate benchmarks of quality (e.g., summary scores measuring system quality for peer counties or states and for the nation), and to support mutual accountability among partners and contributors in the multisectoral health system. The committee reviewed the 2011 Institute of Medicine (IOM) report For the Public’s Health: The Role of Measurement in Action and Accountability and endorses its rationale and recommendation for the national adoption of a summary measure of population health equivalent to health-adjusted life years (HALYs) or health-adjusted life expectancy (HALE) (IOM, 2011a).

It is incumbent on decision makers and professionals working in quality improvement to consistently answer the question “Quality for whom?” Furthermore, once this question is answered, the public health agencies and the health care delivery organizations need policies to trans-

Suggested Citation:"2 Criteria for Selecting Measures." Institute of Medicine. 2013. Toward Quality Measures for Population Health and the Leading Health Indicators. Washington, DC: The National Academies Press. doi: 10.17226/18339.
×

late into action the broad goal of addressing disparities in health status. It is not enough to monitor and report disparities; measurement can also serve as a tool to ascertain whether inputs (e.g., resources and capacity) and processes (e.g., policies, programs, and services) are succeeding in moving health improvement efforts toward the desired health outcomes and greater equity. Equity across all groups by age, socioeconomic status, race, gender, and sexual orientation can be measured as ranges (i.e., differences among groups), and using such metrics as the Gini coefficient, the index of dissimilarity, the slope index of inequality, and the index of disparity developed for Healthy People 2010 (Pearcy and Keppel, 2002). It is possible to publish health-adjusted life expectancy data by quintiles of income, sex, race, and ethnicity (see, for example, the evidence reviewed by Clarke et al. [2010]). One strategy for ensuring that equity is a guiding principle from the beginning is to apply a consistent and scientifically justifiable approach to race/ethnicity and socio-economic classification in the process of population measurement. Also, it is not enough to look at descriptive data; rather, it is important that system inputs be used to drive improvement in achieving equitable outcomes. Asada and colleagues (2013) have proposed a potentially useful analytic approach to measuring disparities, using functional limitation data (i.e., activities of daily living) from the 2009 American Community Survey to develop disparity profiles by states, showing whether disparities were associated primarily with race and ethnicity, socioeconomic factors, or both. Because the social determinants chapter in Healthy People 2020 was under development during the deliberations leading to selection of the LHIs, those factors are minimally reflected in the LHIs. Although it is beyond this report’s scope to discuss the social determinants of health in depth, measuring the inequities (in inputs) that lead to disparities in outcome provides essential information about systemic challenges or obstacles at the level of government policy and institutional practices (see, for example, the IOM [2003c] report Unequal Treatment: Confronting Racial and Ethnic Disparities in Health Care).

The committee attempted to apply a coherence-creating, unifying, standardized quality measurement approach to the LHIs, but the LHIs were not designed primarily from a quality perspective. The final selection of LHIs for Healthy People 2020 was made by HHS with input from the IOM (2011b)1 and the Secretary’s Advisory Committee on Health

___________________

1 The IOM’s 2011 report Leading Health Indicators for Healthy People 2020: Letter Report does provide that report’s authoring committee’s rationale for selecting candidate LHIs for HHS (IOM, 2011b).

Suggested Citation:"2 Criteria for Selecting Measures." Institute of Medicine. 2013. Toward Quality Measures for Population Health and the Leading Health Indicators. Washington, DC: The National Academies Press. doi: 10.17226/18339.
×

Promotion and Disease Prevention (Honore et al., 2011). The present IOM committee recognizes that HHS and its advisers had to reconcile many considerations in a complex process that included considering the burden of disease; managing mandates, resources, current priorities, and capabilities; weighing and balancing stakeholder interests and expectations; and making important tradeoffs. This array of factors and forces explains both the very large and heterogeneous set of Healthy People 2020 objectives (approximately 1,200 that cover many but not necessarily all top burden of disease priorities) and the heterogeneity or dissimilarity of the LHIs (population vs. clinical/individual, disease specific vs. risk behavior, related to one outcome/endpoint vs. pertinent to a dozen outcomes/endpoints), which presented a challenge to the IOM committee in its attempt to identify a coherent set of quality measures.

There were also some constraints on the HHS advisory committee’s work. For example, one requirement was that the LHIs be drawn from the Healthy People 2020 objectives. As a result of the many considerations and constraints made in selecting them, the resulting set of 26 LHIs (see Box 1-1 for the complete list of LHIs) is highly heterogeneous: some LHIs relate to outcomes, while others relate to processes. Four of the LHIs—fatal injuries, homicides, infant deaths, and suicides—are themselves outcome measures. Another four are intermediate outcome measures (preterm births, adolescents with major depressive episodes, adults with controlled hypertension, and adult diabetics with poor glucose control) and 12 LHIs could be classified as measures of the effectiveness of an intervention.

When the LHIs are organized according to the logic model above, most measures fit in the categories of Healthy Conditions and Health Outcomes. Few LHIs fit in the category of Interventions (one example is receiving primary care services), and none fits under Resources and Capacity. Despite the lack of LHIs in this last category, measures in this area are an important contributor to a system’s success in moving toward desired outcomes. For example, research exploring the relationship between public health funding and outcomes is still in its early stages, but Mays and Smith (2011) have provided suggestive evidence that public health funding is correlated with better health outcomes. Some aspects of Resources and Capacity were discussed in a previous IOM report (2012), and reviewing that report’s discussion of the “foundational capabilities” of public health agencies shows a great deal of congruence with the six drivers of quality improvement in public health.

The committee did not set out to replicate the many causal networks that could be developed to show how various system inputs lead to cer-

Suggested Citation:"2 Criteria for Selecting Measures." Institute of Medicine. 2013. Toward Quality Measures for Population Health and the Leading Health Indicators. Washington, DC: The National Academies Press. doi: 10.17226/18339.
×

tain intermediate outcomes that result in ultimate population health outcomes. This has been done by others for various chronic diseases, including the Centers for Disease Control and Prevention (CDC)/National Institutes of Health (NIH) Prevention Impacts Simulation Model (PRISM) described in Homer and colleagues (2010) and also the work of Jones and colleagues (2006), and Wolfson and Rowe (2001), and although many of the models available have specific purposes and limitations, they illustrate the complexity of causal factors and the interactions among them. They also suggest important methods for structuring and understanding the relationship of specific interventions to specific health outcomes, and they help quantify components and indicate which interventions might be best in a specific place to achieve a desired set of outcomes. A previous IOM committee called for modeling to help inform decision makers about interventions that are likely to have the greatest impact (IOM, 2011a). That committee recommended “that the Department of Health and Human Services (HHS) coordinate the development and evaluation and advance the use of predictive and system-based simulation models to understand the health and consequences of underlying determinants of health. HHS should also use modeling to assess intended and unintended outcomes associated with policy, funding, investment, and resource options” (IOM, 2011a, p. 103 [Recommendation 6]). Such work would also help identify, develop, and refine measures of quality. In Chapter 3 (see Table 3-1), the committee offers sample measures along the trajectory of the various LHIs. For example, the update of childhood vaccines is an intermediate outcome, whose ultimate outcomes are the rates of morbidity and mortality from certain infectious diseases.

GENERAL CONSIDERATIONS FOR SELECTING AND PRESENTING MEASURES OF QUALITY

The committee provides some preliminary considerations for selecting measures of quality related to the LHIs and for population health more broadly. Also, as previously noted, the quality characteristics do not apply directly to the LHIs or to measures of quality. Measures cannot in and of themselves be thought of as effective, population-centered, or transparent. However, several of the characteristics are embedded in the criteria for measure selection as described below. Measures of quality may measure outcomes or intermediate outcomes, may reflect on effi-

Suggested Citation:"2 Criteria for Selecting Measures." Institute of Medicine. 2013. Toward Quality Measures for Population Health and the Leading Health Indicators. Washington, DC: The National Academies Press. doi: 10.17226/18339.
×

ciency or on cost-effectiveness2 and will ideally be associated with evidence-based interventions (e.g., programs and policies).

In its deliberations, the committee agreed that although measures of quality may be presented as long lists or catalogues (similar to the National Quality Forum–endorsed system of almost 700 measures, or as an extension of the 1,200 Healthy People 2020 objectives themselves), a set or portfolio of measures is likely to be more useful to—and thus, more used by—practitioners involved in quality improvement, and also more informative to communities and other audiences wishing to understand how the system is performing. There are at least two such comprehensive, but parsimonious, measure portfolios currently in use: America’s Health Rankings (AHR), which uses 44 measures, and the County Health Rankings (CHR), which uses 30 measures in 5 domains (Remington and Booske, 2011). The AHR has been published since 1990 (United Health Foundation, 2012) and represents an effort to quantify and analyze the status and changes in health in the U.S. population from year to year. The information used is from publicly accessible data, collected mostly by the federal government. A major data source is Behavioral Risk Factor Surveillance System (BRFSS) of the Centers for Disease Control and Prevention (CDC), but it has limited usefulness at the local level. AHR reports the rankings by state and includes sections on health disparities and how the United States compares to other countries. More recently, CHR began rankings by county within each state. CHR is based at the University of Wisconsin Population Health Institute, which has ranked Wisconsin’s counties since 2003. CHR uses data from publicly available local and federal government sources. Other pertinent measurement activities include the CDC Community Health Status Indicators, which provides a portfolio of 200 measures, with a set for each of the 3,141 U.S. counties, and allows comparisons between peer counties to inform quality improvement efforts and the sharing of best practices. The 2011 IOM report For the Public’s Health: The Role of Measurement in Action and Accountability provides additional descriptions of indicator efforts. In the realm of health care delivery there also are multiple ongoing efforts, ranging from the National Committee on Quality Assurance HEDIS (Healthcare Effectiveness Data and Information Set) measures, to the hundreds of additional measures from many other organizations provided as part of the National Quality Forum’s Quality Positioning System, its measure endorsement process. Finally, the Agency for

___________________

2 The “extent to which the results are achieved at a lower cost compared with alternatives” (World Bank, 2007, p. 65).

Suggested Citation:"2 Criteria for Selecting Measures." Institute of Medicine. 2013. Toward Quality Measures for Population Health and the Leading Health Indicators. Washington, DC: The National Academies Press. doi: 10.17226/18339.
×

Healthcare Research and Quality (ARHQ) National Quality Measures Clearinghouse contains a large collection of measures (more than 2,000)—the vast majority of measures belong to the health care delivery domain and small number are listed under the population health domain.3

CRITERIA FOR MEASURE SELECTION

In comments made at the committee’s first meeting, the Assistant Secretary for Health suggested the committee provide methodological guidance that could be used in selecting measures of quality. In response, the committee reviewed a number of existing resources for sample criteria (see Table 2-1 below and Table B-1 in Appendix B for a description of criteria across several different sources). The committee’s review included the National Quality Forum’s criteria for evaluating measures, the Institute for Healthcare Improvement’s key measurement principles that apply to the Triple Aim, the HHS Secretary’s Advisory Committee on Health Promotion and Disease Prevention Objectives for 2020 operational criteria for selection of LHIs, the 2003 IOM report Priority Areas for National Action: Transforming Health Care Quality and the 2010 IOM report Future Directions for the National Healthcare Quality and Disparities Reports. The AHRQ National Quality Measures Clearinghouse (not in the table) provides the following criteria for measures included in their database: (1) the measure is cited in peer-reviewed/National Library of Medicine journal, (2) the measure has documented evidence of reliability and validity, and (3) the measure has been developed, adopted, adapted, or endorsed by an appropriate organization.4

With the exception of the criteria for selecting LHIs, the sets of criteria listed were developed for measures largely pertinent to health care delivery, and the committee did not feel that any existing set of measures was sufficiently specific for selecting measures of quality for the multisectoral health system. Because none of the sets of criteria listed is primarily oriented toward population health, they miss some of the nuances that are important in population health interventions. For example, data availability at the state and local level is a critical issue for population health measures, but not for clinical care measures, where data are intrinsically part of the medical record.

___________________

3 The population health domain includes dimensions such as population health state, social determinants of health, and environment.

4 See http://www.qualitymeasures.ahrq.gov/about/inclusion-criteria.aspx (accessed May 31, 2013).

Suggested Citation:"2 Criteria for Selecting Measures." Institute of Medicine. 2013. Toward Quality Measures for Population Health and the Leading Health Indicators. Washington, DC: The National Academies Press. doi: 10.17226/18339.
×

Topically similar criteria in these sets were grouped into five categories: (1) impact or importance of the condition or outcome to be measured; (2) improvability, or the extent of the gap between current practice and evidence-based best practice and the likelihood that the gap can be closed; (3) scientific soundness of the measure, including validity and reliability; (4) geographic, temporal, and population coverage to ensure that the measure has sufficient granularity to be useful in monitoring actions to improve health at different geographic levels in important population subgroups; and (5) data availability to ensure that data are readily available in a form useful for quality and performance measurement (see Table 2-1 and Appendix B). The first two categories—impact and improvability—refer to characteristics of condition(s), outcome(s), and associated interventions that a measure would address, while the final three categories—scientific soundness, coverage, and data availability— address characteristics of the measures themselves. Thus, the committee developed a set of criteria that embody most characteristics of these existing sets, while using words and providing definitions that indicate characteristics that are more relevant for population health, and emphasizing certain criteria that are particularly important for population health, such as coverage at the state and local level, where many population-based interventions are implemented. The committee also recognizes the potential usefulness of a stepwise approach to applying

TABLE 2-1 Quick Comparison of Published Criteria for Measure Selection (Detailed Table Provided in Appendix B)

Published Criteria
Category
   of Criteria
 
NQF, 2012b
HHS SAC,
2011
 
IOM, 2010
 
IOM, 2003b
Impact
(importance)
 
X
 
X
 
X
 
X
Improvability Scientifically X unclear X X
sound measure X n/a X n/a
Geographic, temporal, and population coverage n/a X X unclear
Data availability X n/a X n/a

NOTES: X indicates that the published criteria included one or more items in the category listed in the first column; n/a = not applicable.

Suggested Citation:"2 Criteria for Selecting Measures." Institute of Medicine. 2013. Toward Quality Measures for Population Health and the Leading Health Indicators. Washington, DC: The National Academies Press. doi: 10.17226/18339.
×

these criteria, in which the target of potential measures is first evaluated for importance and improvability (seen through a population health lens), followed by an evaluation of the measures characteristics (see IOM, 2010, pp. 69-74).

Finding 2-1: The committee finds that partners in the multisectoral health system currently use a vast and complex array of measures of quality in a manner that seems uncoordinated.

This finding refers to measures of public health relevance, thus excluding measures that are largely pertinent to clinical care. Developing a more coordinated approach would include paying attention to the three purposes of measurement: assessment, improvement, and accountability. A previous IOM committee provided a measurement framework for accountability that acknowledged the different forces at work when accountability is contractually required compared to when accountability is informal or “soft”—what that committee termed “compact accountability” (IOM, 2011a). A challenge identified by the previous committee also persists, in the proliferation of metrics, with limited effort to coordinate, consolidate, and organize in a manner that increases coherence and reduces overlap and duplication.

RECOMMENDATION 2-1: The committee recommends that the Department of Health and Human Services and its partners in population health improvement (e.g., public health agencies, health care organizations, and communities) adopt a portfolio of measures of the quality of the multisectoral health system. The portfolio of measures should

a. include summary scores reflecting population-level health outcomes and healthy conditions.

b. balance parsimony with sufficient breadth.

c. inform assessment, improvement, and accountability of the multisectoral health system.

When it refers to summary scores, the committee envisions

  • a “healthy outcomes” summary score that is a composite of (the quantitative values for) all outcome measures selected in a port-
Suggested Citation:"2 Criteria for Selecting Measures." Institute of Medicine. 2013. Toward Quality Measures for Population Health and the Leading Health Indicators. Washington, DC: The National Academies Press. doi: 10.17226/18339.
×

folio (the summary measures of population health, such as HALE, could serve as such a score, but others could be developed to reflect several different health outcomes of interest); and

  • a “healthy conditions” summary score (e.g., such as the community well-being indicator described in IOM [2012]) that is a composite of all intermediate outcomes (or determinants of health) measures selected in a portfolio.

These scores and values could be made publicly available through regular reports and other channels of communication.

A parsimonious (or manageable) portfolio will not be exhaustive or exhausting, it will instead consist of a small number of the most important things, up to, say, four or five per LHI topic. The portfolio will also reflect relevant different areas of the logic model across measures and within each topic, and the measures will not be redundant or overlapping. Measurement has three somewhat overlapping purposes. Measures are needed for the assessment of overall quality in the multisectoral health system, beginning with a focus on governmental public health and health care (where the areas of responsibility and accountability are more clearly delineated) and moving on to the roles and responsibilities of other stakeholders. Measures are also necessary to inform quality improvement and to demonstrate accountability for population health improvement. Measures of quality will inform clinicians and health care organizations, public health agencies, and others in their community health improvement process work at the state and local levels. Measures of quality will also assist members of the public and their elected officials to hold public health officials and other stakeholders, involved in communitywide programs and efforts to improve health, accountable for the quality and effectiveness of their actions (IOM, 2006).

RECOMMENDATION 2-2: The committee recommends that the Department of Health and Human Services and other relevant organizations adopt the following set of criteria for selecting and prioritizing measures of quality for use in population health improvement, including the Leading Health Indicators:

Suggested Citation:"2 Criteria for Selecting Measures." Institute of Medicine. 2013. Toward Quality Measures for Population Health and the Leading Health Indicators. Washington, DC: The National Academies Press. doi: 10.17226/18339.
×

Criteria for conditions or outcomes to be measured

a. Reflective of a high preventable burden5

b. Actionable at the appropriate level for intervention

Criteria for the measures

c. Timely

d. Usable for assessing various populations

e. Understandable

f. Methodologically rigorous

g. Accepted and harmonized

Judging the criterion “reflective of a high preventable burden” requires estimates of the frequency of the condition or disease entity and the effectiveness of the interventions. This means that the measures need to refer to interventions and outcomes related to health conditions that account for considerable morbidity and mortality and that are also preventable. A standard measure is needed to determine what is a large burden and what is not, but context matters as well. As an example, the preventable burden of infant mortality is likely relatively small since infant mortality, however catastrophic, is still relatively low. It is likely on the LHI list for several reasons: these are catastrophic events, rates in some groups are much higher than in others, it is an indicator of many issues in the clinical care system and in society as a whole, and it is a measure used in international comparisons. Assessing the evidence of effectiveness may be done by reviewing published findings of the Guide to Community Preventive Services, the Cochrane Collaboration, and similar entities. The following characteristics of interventions are important to consider: the feasibility of implementing the intervention (particularly the feasibility in different settings); the potential for improvement (e.g., for a specific health condition, an intervention already delivered at a very high level has less potential for improvement than an equally effective intervention delivered at a lower level); and the extent to which the intervention has externalities (i.e., benefits other conditions or has non-health benefits). The corollary to the “effectiveness of the intervention” condi-

___________________

5 The concept of high preventable burden has two components: high burden and existence of effective interventions. This concept (burden x effectiveness), refers to burden as the absolute burden, not relative burden. In other words, a condition like phenylketonuria (PKU) has a high preventable burden if one thinks of the denominator as all people with PKU, but it is a low absolute preventable burden if one uses the entire population.

Suggested Citation:"2 Criteria for Selecting Measures." Institute of Medicine. 2013. Toward Quality Measures for Population Health and the Leading Health Indicators. Washington, DC: The National Academies Press. doi: 10.17226/18339.
×

tion is a case where the magnitude of a problem is such that action is necessary though there is inadequate information on effective interventions (see IOM, 2011a). For example, evidence may be limited to lessons learned or preliminary information on best practices, or there may be multiple potential interacting interventions and insufficient evidence to help pinpoint the most effective. Also, the context of interventions is a critical factor to consider. There are different considerations for measures of relevance to the local level compared to those with relevance to the national level. At the local level, interventions informed by limited evidence, including perhaps emerging best practices or the equivalent, may be tested, so measures of quality are needed. In the absence of evidence-based recommendations from, for example, the Guide to Community Preventive Services, interventions or combinations of interventions must have an associated rigorous evaluation.

Although the committee concluded that the nine characteristics of quality in public health do not directly align with measures of quality, it recognizes that a measure that reflects a high preventable burden enables organizations to assess whether their interventions are effective, efficient, equitable (because vulnerable populations may bear a higher burden of, for example, infant mortality) and risk reducing.

The criterion “actionable at the appropriate level of intervention” reflects whether a measure provides sufficient information about a problem to help identify a way to address it, and whether there are effective programs and policies that can be adopted by relevant stakeholders (e.g., local jurisdictions). Finding an effective programs may be done both by referring to systematic reviews, such as those provided by the Guide to Community Preventive Services and the Cochrane Collaboration, or by looking to other rigorous efforts to identify best practices, such as the work of the Public Health Law Research program in establishing the Law Atlas, which could in the future help identify associations between changes in policy and health outcomes.6

The criterion “timely” refers both to data for a measure (1) being collected frequently enough to make it possible to track changes in the measure that reflect actions intended to affect the outcome or condition and (2) being made available quickly enough (e.g., within 6 months of collection) to be acted on.

“Usable for assessing various populations” means that data are available and can be used to assess different populations (e.g., defined by de-

___________________

6 See http://lawatlas.org/about.

Suggested Citation:"2 Criteria for Selecting Measures." Institute of Medicine. 2013. Toward Quality Measures for Population Health and the Leading Health Indicators. Washington, DC: The National Academies Press. doi: 10.17226/18339.
×

mographics or defined by living in a certain zip code) and at different levels (national, state, local).

  1. At the total population level and also, to allow levels of equity to be observed, at the subpopulation level (including disparities in every measure).
  2. At national, state, and local levels, depending on where policy, programmatic, system, or clinical action is needed. National data include those collected in such systems as the BRFSS, vital records, the American Community Survey, and the Small Area Health Insurance Estimates, while state and local data include those used in the Community Health Status Indicators, drawn from electronic health records, and from other sources used by America’s Health Rankings and the County Health Rankings.
  3. At levels applicable to the public health system and to the clinical care system where possible.

There are several data-related challenges related to “usable” criterion. There are considerable data limitations at state and local levels and there is a great need for investment in better data infrastructure (an issue that also relates to one of the six drivers of public health quality—metrics and information technology). For example, data needed for all the clinical parameters (e.g., controlled blood pressure, hemoglobin A1c) are unavailable at the local level. Even assuming universal electronic health records (and attaining this was not within reach at the time this report was written), there are many individuals who will remain outside the clinical care system and thus for whom there are no data. For example, uninsured individuals receiving emergency department care may have their blood pressure captured, but without a data generation process similar to that in the National Health and Nutrition Examination Survey, it may not be possible to find out blood pressure levels in a community. Virtually no county has adequate data from BRFSS, and even in the case of large counties, the county-level data are of little value in dealing with the many sub-county issues.

“Understandable” means that no great level of sophistication is required from decision makers, including public health and health care practitioners, policy makers, and the public to understand the criteria. Furthermore, do the measures have face validity? Is the selection process transparent to users and other audiences?

The criterion “methodologically rigorous” refers to the measures having suitable methodological and quantitative characteristics, such as

Suggested Citation:"2 Criteria for Selecting Measures." Institute of Medicine. 2013. Toward Quality Measures for Population Health and the Leading Health Indicators. Washington, DC: The National Academies Press. doi: 10.17226/18339.
×

sensitivity, specificity, reliability, validity, and consistency over time and being managed by an established, regularly updated process. The committee believes that a broader criterion is needed because common methodologic criteria such as validity and reliability do not cover issues such as representativeness and consistency. For example, blood pressure control may be a good measure for a clinical care system (i.e., valid and reliable), but a poor measure if the goal is control in the population (which deals with those with poor care, poor adherence, with population-level issues such as physical activity and sodium restriction to reduce population blood pressure levels).

Finally, “accepted and harmonized” refers to measures such as those endorsed by the NQF (primarily focused on health care), or that are in standard use (e.g., used by America’s Health Rankings and the County Health Rankings). Difficulties arise, and harmonization is needed, in cases where there are many commonly used measures for the same phenomenon, such as binge drinking and defining an appropriate “norm” in consumption level, or conditions that are not well (or easily) measured, such as major depression in adolescents.

ENDORSING QUALITY MEASURES FOR THE FIELD

In its review of available measures, the committee was unable to find many measures of quality that reside outside the clinical sector. At its information-gathering meeting, the committee learned from participants that the universe of quality measures seems to include very few such measures compared to metrics that aggregate individual-level data related to specific disease states or clinical interventions (Jarris and Stange, 2012).

The report Priorities for the 2011 National Quality Strategy (National Priorities Partnership, 2011) contains a table that details priorities, goals, and sample measures organized according to the HHS Three-Part Aim. The section of the table with the heading Population Health is subdivided into three columns: clinical preventive services, healthy lifestyle behaviors, and community health index. The last item refers to truly population health measures (consistent with the Jacobson and Teutsch [2012] definition of “total population health”), and the two sources cited are the County Health Rankings and the AHRQ Prevention Quality Indicators. The former is a portfolio of measures that includes several population-based measures, including health behaviors (e.g., motor vehicle crash death rate and adult obesity), social and economic factors (e.g., high school graduation rates and percentage of children in poverty), and

Suggested Citation:"2 Criteria for Selecting Measures." Institute of Medicine. 2013. Toward Quality Measures for Population Health and the Leading Health Indicators. Washington, DC: The National Academies Press. doi: 10.17226/18339.
×

of the physical environment (e.g., drinking water safety and fast food restaurants). The AHRQ Prevention Quality Indicators is a set of individual and composite measures derived from entirely clinical data and designed to assess the effectiveness of a local community’s ambulatory health care delivery system.7 This situation concerning population-based measures suggests that key national efforts to describe health priorities and measures have been predominantly clinical in orientation, as has much of the national effort on quality and prevention, and more attention is warranted for measures related to population-based preventive interventions.

The first step in finding ways to measure quality that are relevant to many different system actors, beginning with public health agencies and health care organizations, could be to find a shared quality language that both health care and public health partners understand (and that can ultimately be understood by stakeholders outside the health sector). For example, the committee began the search for quality measures for the LHIs by reviewing the measures relevant to the LHIs that have been endorsed by the NQF. Where appropriate NQF-endorsed measures do exist, not only are they useful for improving the quality of care, but they can also further progress on the LHIs. This is a linkage that is sometimes not recognized, i.e., that a pediatrician working in a HEDIS-compliant practice setting is not merely working to improve care, but may also be contributing to improving health outcomes at a level far beyond the individual patient.

The advantage offered by having an endorsing entity with an accepted endorsing process is that organizations that want to use measures of quality in their work can simply look to the endorsed sets and select appropriate measures. An endorsement process is one of the first steps in improving measurement and is a prerequisite to the use of standardized measures and measure sets. In order to ensure progress in the formal adoption of population-based measures of quality, an entity charged with endorsing population health measures will need the following characteristics:

  • Be nongovernmental, to ensure independence;
  • Have the appropriate high-level leadership, organization, and expertise to enable review and endorsement measures of quality intended for population health improvement (e.g., measures of

___________________

7 See http:​//​www.​qualityindicators.​ahrq​.gov/​Downloads/​Modules/​PQI/​V44/​CompositeUser_​Technical_​Specification_​PQI%20V4.4.pdf (accessed June 26, 2013).

Suggested Citation:"2 Criteria for Selecting Measures." Institute of Medicine. 2013. Toward Quality Measures for Population Health and the Leading Health Indicators. Washington, DC: The National Academies Press. doi: 10.17226/18339.
×

the social and environmental determinants of health), not just measures of clinical care;

  • Have processes designed and resourced to evaluate and endorse measures of quality for population health improvement (a process separate and distinct from any existing process for endorsing measures of clinical care); and
  • Include expert panel membership and staff support to identify and assess measures of quality for the multisectoral health system, with consideration of data sources, methodology, and other issues that span sectors and disciplines.

Measuring the quality of population-based nonclinical interventions, such as policies, presents more challenges than measuring the quality of individual-based clinical interventions. Clinical interventions can demonstrate improved outcomes in the short term, while population-level action (e.g., clean air laws or universal preschool) may take a generation to bear fruit. For example, evaluating new population health measures will require expertise in interacting with city councils, with planners and land use experts, with educators, and with community organizations. The report For the Public’s Health: The Role of Measurement in Action and Accountability (IOM, 2011a) asserted that improving population health will require entirely new kinds of measures and data. For example, the authoring committee wrote that

there currently is no coordinated, standard set of true measures of a community’s health—not aggregated information about the health of individuals residing in a community, but rather measures of green space, availability of healthy foods, land use and zoning practices that are supportive of health, safety, social capital, and social cohesion, among many other determinants of health. (IOM, 2011a, p. 5)

There currently is no organization that endorses measures of quality to be used for population health (i.e., measures for the multisectoral health system). However, NQF endorses measures of quality for the health care delivery system, and it or a similar entity, appropriately constituted, could perform the same role for the universe of measures that go beyond the health care delivery system.

Given the conclusions of IOM committees and other groups that health care is only responsible for a modest proportion of the factors that influence population health, the committee calls for changes in the approach to measurement of quality.

Suggested Citation:"2 Criteria for Selecting Measures." Institute of Medicine. 2013. Toward Quality Measures for Population Health and the Leading Health Indicators. Washington, DC: The National Academies Press. doi: 10.17226/18339.
×

RECOMMENDATION 2-3: The committee recommends that the Department of Health and Human Services ensure the implementation of a systematic approach to develop and manage a portfolio of measures of quality for the multisectoral health system. HHS also should establish or designate a nongovernmental and appropriately equipped entity to endorse measures of quality.

An entity endorsing measures of quality for the multisectoral health system would need to be guided by a strong research infrastructure, elements of which were described in a recommendation in an earlier report (IOM, 2012) that called for a research agenda and funding to support the public health research and evaluation infrastructure. Also, an endorsing entity would not have to be, and ideally will not be, an organization that develops measures; although the two skillsets overlap somewhat, the roles and purposes are very different. With regard to the role of HHS in systematizing the approach to measures of quality, the committee has learned about two evolving HHS activities in the area of quality measurement. These are the Centers for Medicare & Medicaid Services (CMS) Quality Measures Task Force and the HHS Interagency Measurement Policy Council. The CMS Task Force is charged with “developing recommendations on CMS measure implementation with the goal of aligning and prioritizing measures across programs and avoidance of duplication or conflict among developing and implemented measures” and one of its goals is to coordinate “measure implementation, development and measurement policies” with other agencies in HHS (Goodrich, 2012). The Measurement Policy Council, which was established in 2012 as a subgroup of the HHS National Quality Strategy effort, is focused on policies for measure development, implementation, and alignment across HHS. The council includes AHRQ, CMS, the Office of the National Coordinator for Health Information Technology, the Substance Abuse and Mental Health Services Administration, the HHS Office of the Assistant Secretary for Planning and Evaluation, the Health Resources and Services Administration, CDC, the Office of Minority Health, the Food and Drug Administration, and others. Its initial focus is on “alignment and prioritization of measures in six major areas: hypertension, smoking cessation, depression, health care acquired conditions, patient experience, and care coordination” (Goodrich, 2012). It appears that the HHS-wide council, like the CMS Task Force, is largely oriented toward clinical care. Moreover, if the objective is to improve the health of the population by creating healthy conditions, coordination and measures are needed

Suggested Citation:"2 Criteria for Selecting Measures." Institute of Medicine. 2013. Toward Quality Measures for Population Health and the Leading Health Indicators. Washington, DC: The National Academies Press. doi: 10.17226/18339.
×

that involve sectors of government beyond HHS, for example, the array of executive branch leaders (including the Departments of Education, Housing, and Transportation) participating in the National Prevention, Health Promotion, and Public Health Council.

Data and information are needed to identify and develop measures of quality, and to support the development and management of the portfolio of measures described in Recommendation 2-3. Strengthening public health agency capacity in this area will also spur progress in the priority area for quality improvement in public health “Metrics and Information Technology.” To this end, the committee endorses both Recommendations 1 and 2 from the 2011 IOM report For the Public’s Health: The Role of Measurement in Action and Accountability, and Recommendation 6 from the 2012 IOM report For the Public’s Health: Investing in a Healthier Future. The former called for strengthening the population health information infrastructure, and for integrating, aligning, and standardizing health data and health outcome measurement at all geographic levels.8 The latter called for a research infrastructure to establish the value of public health and prevention strategies, mechanisms for their effective implementation, health and economic outcomes derived, and the comparative effectiveness and impact of those strategies.9

___________________

8 Recommendation 1: The committee recommends that: (1) The Secretary of Health and Human Services transform the mission of the National Center for Health Statistics to provide leadership to a renewed population health information system through enhanced coordination, new capacities, and better integration of the determinants of health. (2) The National Prevention, Health Promotion, and Public Health Council include in its annual report to Congress on its national prevention and health-promotion strategy an update on the progress of the National Center for Health Statistics transformation. Recommendation 2: The committee recommends that the Department of Health and Human Services support and implement the following to integrate, align, and standardize health data and health-outcome measurement at all geographic levels:

a. A core, standardized set of indicators that can be used to assess the health of communities.

b. A core, standardized set of health-outcome indicators for national, state, and local use.

c. A summary measure of population health that can be used to estimate and track health-adjusted life expectancy for the United States.

9 Recommendation 6: The committee recommends that Congress direct the Department of Health and Human Services to develop a robust research infrastructure for establishing the effectiveness and value of public health and prevention strategies, mechanisms for effective implementation of these strategies, the health and economic outcomes derived from this investment, and the comparative effectiveness and impact of this investment. The infrastructure should include

  • A dedicated stream of funding for research and evaluation.
  • A national research agenda.
Suggested Citation:"2 Criteria for Selecting Measures." Institute of Medicine. 2013. Toward Quality Measures for Population Health and the Leading Health Indicators. Washington, DC: The National Academies Press. doi: 10.17226/18339.
×

RECOMMENDATION 2-4: The Department of Health and Human Services should develop, implement, and support data collection, analysis, and dissemination mechanisms and infrastructure for the portfolio of quality measures so they are usable for health assessment and improvement at the national, state, and local levels.

The committee hopes that the implementation of all IOM recommendations in this area will contribute to future data systems that address current limitations. The ability to develop good quality measures requires ensuring that timely data are available at national, state and, in particular, local levels, and that these data can be stratified for vulnerable sub-populations to assess changes in health disparities during improvement efforts.

___________________

  • Development of data systems and measures to capture research-quality information on key elements of public health delivery, including program implementation costs.
  • Development and validation of methods for comparing the benefits and costs of alternative strategies to improve population health.
Suggested Citation:"2 Criteria for Selecting Measures." Institute of Medicine. 2013. Toward Quality Measures for Population Health and the Leading Health Indicators. Washington, DC: The National Academies Press. doi: 10.17226/18339.
×
Page 27
Suggested Citation:"2 Criteria for Selecting Measures." Institute of Medicine. 2013. Toward Quality Measures for Population Health and the Leading Health Indicators. Washington, DC: The National Academies Press. doi: 10.17226/18339.
×
Page 28
Suggested Citation:"2 Criteria for Selecting Measures." Institute of Medicine. 2013. Toward Quality Measures for Population Health and the Leading Health Indicators. Washington, DC: The National Academies Press. doi: 10.17226/18339.
×
Page 29
Suggested Citation:"2 Criteria for Selecting Measures." Institute of Medicine. 2013. Toward Quality Measures for Population Health and the Leading Health Indicators. Washington, DC: The National Academies Press. doi: 10.17226/18339.
×
Page 30
Suggested Citation:"2 Criteria for Selecting Measures." Institute of Medicine. 2013. Toward Quality Measures for Population Health and the Leading Health Indicators. Washington, DC: The National Academies Press. doi: 10.17226/18339.
×
Page 31
Suggested Citation:"2 Criteria for Selecting Measures." Institute of Medicine. 2013. Toward Quality Measures for Population Health and the Leading Health Indicators. Washington, DC: The National Academies Press. doi: 10.17226/18339.
×
Page 32
Suggested Citation:"2 Criteria for Selecting Measures." Institute of Medicine. 2013. Toward Quality Measures for Population Health and the Leading Health Indicators. Washington, DC: The National Academies Press. doi: 10.17226/18339.
×
Page 33
Suggested Citation:"2 Criteria for Selecting Measures." Institute of Medicine. 2013. Toward Quality Measures for Population Health and the Leading Health Indicators. Washington, DC: The National Academies Press. doi: 10.17226/18339.
×
Page 34
Suggested Citation:"2 Criteria for Selecting Measures." Institute of Medicine. 2013. Toward Quality Measures for Population Health and the Leading Health Indicators. Washington, DC: The National Academies Press. doi: 10.17226/18339.
×
Page 35
Suggested Citation:"2 Criteria for Selecting Measures." Institute of Medicine. 2013. Toward Quality Measures for Population Health and the Leading Health Indicators. Washington, DC: The National Academies Press. doi: 10.17226/18339.
×
Page 36
Suggested Citation:"2 Criteria for Selecting Measures." Institute of Medicine. 2013. Toward Quality Measures for Population Health and the Leading Health Indicators. Washington, DC: The National Academies Press. doi: 10.17226/18339.
×
Page 37
Suggested Citation:"2 Criteria for Selecting Measures." Institute of Medicine. 2013. Toward Quality Measures for Population Health and the Leading Health Indicators. Washington, DC: The National Academies Press. doi: 10.17226/18339.
×
Page 38
Suggested Citation:"2 Criteria for Selecting Measures." Institute of Medicine. 2013. Toward Quality Measures for Population Health and the Leading Health Indicators. Washington, DC: The National Academies Press. doi: 10.17226/18339.
×
Page 39
Suggested Citation:"2 Criteria for Selecting Measures." Institute of Medicine. 2013. Toward Quality Measures for Population Health and the Leading Health Indicators. Washington, DC: The National Academies Press. doi: 10.17226/18339.
×
Page 40
Suggested Citation:"2 Criteria for Selecting Measures." Institute of Medicine. 2013. Toward Quality Measures for Population Health and the Leading Health Indicators. Washington, DC: The National Academies Press. doi: 10.17226/18339.
×
Page 41
Suggested Citation:"2 Criteria for Selecting Measures." Institute of Medicine. 2013. Toward Quality Measures for Population Health and the Leading Health Indicators. Washington, DC: The National Academies Press. doi: 10.17226/18339.
×
Page 42
Suggested Citation:"2 Criteria for Selecting Measures." Institute of Medicine. 2013. Toward Quality Measures for Population Health and the Leading Health Indicators. Washington, DC: The National Academies Press. doi: 10.17226/18339.
×
Page 43
Suggested Citation:"2 Criteria for Selecting Measures." Institute of Medicine. 2013. Toward Quality Measures for Population Health and the Leading Health Indicators. Washington, DC: The National Academies Press. doi: 10.17226/18339.
×
Page 44
Next: 3 Measures of Quality »
Toward Quality Measures for Population Health and the Leading Health Indicators Get This Book
×
Buy Paperback | $42.00 Buy Ebook | $33.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The Institute of Medicine (IOM) Committee on Quality Measures for the Healthy People Leading Health Indicators was charged by the Office of the Assistant Secretary for Health to identify measures of quality for the 12 Leading Health Indicator (LHI) topics and 26 Leading Health Indicators in Healthy People 2020 (HP2020), the current version of the Department of Health and Human Services (HHS) 10-year agenda for improving the nation's health.

The scope of work for this project is to use the nine aims for improvement of quality in public health (population-centered, equitable, proactive, health promoting, risk reducing, vigilant, transparent, effective, and efficient) as a framework to identify quality measures for the Healthy People Leading Health Indicators (LHIs). The committee reviewed existing literature on the 12 LHI topics and the 26 Leading Health Indicators. Quality measures for the LHIs that are aligned with the nine aims for improvement of quality in public health will be identified. When appropriate, alignments with the six Priority Areas for Improvement of Quality in Public Health will be noted in the Committee's report. Toward Quality Measures for Population Health and the Leading Health Indicators also address data reporting and analytical capacities that must be available to capture the measures and for demonstrating the value of the measures to improving population health.

Toward Quality Measures for Population Health and the Leading Health Indicators provides recommendations for how the measures can be used across sectors of the public health and health care systems. The six priority areas (also known as drivers) are population health metrics and information technology; evidence-based practices, research, and evaluation; systems thinking; sustainability and stewardship; policy; and workforce and education.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!