4
Measurement and Accountability

Accountability refers to “the principle that individuals, organizations and the community are responsible for their actions and may be required to explain them to others” (Benjamin et al., 2006). The notion of accountability has several meanings that span the fields of accounting, law, ethics, management theory and practice, and governance. Models of accountability include regulatory, legal, accreditation or certification (sometimes quasiregulatory), and pay-for-performance models, or their public health equivalent, eligibility for funding based on past performance. This chapter does not endeavor to examine all dimensions of accountability but rather focuses on the role of indicators in holding to account all stakeholders that contribute to the conditions for health in a community. Other dimensions of accountability will be examined in the committee’s later reports on public health law and funding. Two important examples are the intertwined topics of political will and governance (and governing bodies).1 As noted in this chapter, the governance (and related regulatory and funding) mechanisms that pertain to the work of local public health agencies are among the stronger and more concrete levers for holding agencies accountable.

This chapter examines performance2 indicators and how they can be implemented both at the level of governmental public health and in the con-

1

The National Public Health Performance Standards Program defines governing body as “the individual, board, council, commission, or other body with legal authority over the primary governmental public health agency” (HHS and CDC, 2008).

2

Performance refers to the interventions—policies, programs, and processes—implemented with the intent of improving population health; it represents one of the steps along the inputs-to-outputs or inputs-to-outcomes logic model presented in Chapter 2.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 111
4 Measurement and Accountability Accountability refers to “the principle that individuals, organizations and the community are responsible for their actions and may be required to explain them to others” (Benjamin et al., 2006). The notion of account- ability has several meanings that span the fields of accounting, law, ethics, management theory and practice, and governance. Models of accountability include regulatory, legal, accreditation or certification (sometimes quasiregu- latory), and pay-for-performance models, or their public health equivalent, eligibility for funding based on past performance. This chapter does not endeavor to examine all dimensions of accountability but rather focuses on the role of indicators in holding to account all stakeholders that contribute to the conditions for health in a community. Other dimensions of account- ability will be examined in the committee’s later reports on public health law and funding. Two important examples are the intertwined topics of political will and governance (and governing bodies).1 As noted in this chapter, the governance (and related regulatory and funding) mechanisms that pertain to the work of local public health agencies are among the stronger and more concrete levers for holding agencies accountable. This chapter examines performance2 indicators and how they can be implemented both at the level of governmental public health and in the con- 1 The National Public Health Performance Standards Program defines governing body as “the individual, board, council, commission, or other body with legal authority over the pri- mary governmental public health agency” (HHS and CDC, 2008). 2 Performance refers to the interventions—policies, programs, and processes—implemented with the intent of improving population health; it represents one of the steps along the inputs- to-outputs or inputs-to-outcomes logic model presented in Chapter 2. 111

OCR for page 111
112 FOR THE PUBLIC’S HEALTH: MEASUREMENT tributions of other health-system stakeholders. (As defined in Chapter 1, the health system comprises public health, clinical care, and other stakeholders that acknowledge their current and potential contributions to a community’s health.) Governmental public health is not the only actor in the system that is ac- countable for or involved in creating the conditions for health. Clinical care providers are de facto stewards of a community’s health and are mandated or otherwise charged with health-related duties. Others, such as employ- ers and businesses, may not currently see themselves as contributing to or detracting from a community’s health and well-being (see Chapter 1), but their recognition of their roles and their ability to contribute to health could be facilitated. They often face regulatory pressures, such as rules regarding environmental waste and pollution and zoning limitations. Others, such as community-based organizations, may be seasoned contributors to health, but there are no measurement frameworks for accountability for their work. Those roles are discussed in greater detail later in this chapter. The measurement of performance and the demonstration of account- ability and quality in clinical care have a long history, with a major national movement punctuated by milestone Institute of Medicine (IOM) reports on the subject of quality, federal quality initiatives (such as those undertaken by the Centers for Medicaid and Medicare Services), and the creation of such bodies as the National Committee for Quality Assurance (and its Healthcare Effectiveness Data and Information Set quality measures) and the National Quality Forum and its efforts to set national priorities and endorse standards for and conduct outreach and education on performance improvement in clinical care. Accountability (in the broad sense of demonstrating results and effec- tiveness to the public) is a somewhat more recent focus in the public health community, and this is in part due to the complex array of factors that contribute to population health and the challenging nature of communi- cating about them. As described in Chapter 1, one challenge is that health outcomes (such as disease and death) have multiple interconnected causal pathways, and the science required to elucidate them is far from advanced in many cases. In addition, public health agencies, although broadly charged with ensuring the public’s health, have direct or clearly traceable responsibil- ity for only a small proportion of those pathways. The simple logic model introduced in Chapter 2 and reprised in Figure 4-1 suggests that a straightforward measurement framework for account- ability would link all inputs (resources, capacities, processes, interventions, and policies) with outputs (intermediate and more distal health outcomes). However, there are many obstacles to such a framework, and these are discussed below. It is important to note that accountability is closely linked with needs assessment, planning, and priority-setting—activities identified at the beginning of the process.

OCR for page 111
113 MEASUREMENT AND ACCOUNTABILITY In the context of the social and environmental determinants of health Processes/ Needs Intermediate Health Planning and Resources Capacities interventions/ assessment outcomes outcomes prioritization policies Across different geographic levels and including public health agencies and stakeholders Equity/disparities across population sub-groups FIGURE 4-1 From inputs to outputs logic model. NOTE: The thickness of some arrows denotes the present report’s focus on those Figure 2-2 interactions. The committee recognizes that detailed performance objectives may be identified and measurements conducted at each step of the continuum depicted in Figure 4-1 by all stakeholders in the system. For example, there may be specific objectives for public health agencies or for hospitals that assess their community’s health needs, objectives for the process of planning (such as the number of partners engaged in planning and collaborative plan- ning activities undertaken), objectives that monitor resource use, and so on. In this chapter, however, the committee focuses on the “macro” or broader accountability for the entire continuum of community-health improvement (from needs assessment to the most distal outcomes) and not on any detail of the “micro” accountabilities that could be examined and described for each step in the process. In the process of community-health improvement, after a community’s health needs are assessed, priorities are identified, and plans are made and implemented, performance measurement is needed to hold implementers (the full spectrum of system stakeholders in addition to public health agen- cies) accountable and to spur continuous quality improvement to increase the effectiveness, efficiency, and equity of actions taken to improve popula- tion health. Performance measurement is the main way to monitor account- ability in the health system. A FRAMEWORK FOR ACCOUNTABILITY The measurement framework for accountability discussed in this chap- ter applies to the delivery of funded public health programs by public health agencies; the role of public health agencies in mobilizing the overall public health system; and the roles, contributions, and performance of health- system partners (other governmental agencies, private-sector stakeholders, and communities). Assessing and measuring accountability at any level (local, state, or

OCR for page 111
114 FOR THE PUBLIC’S HEALTH: MEASUREMENT national) and holding organizations accountable require the following four elements: · n identified body with a clear charge to accomplish particular A steps toward health goals. · nsuring that the body has the capacity to undertake the required E activities. · easuring what is accomplished against the identified body’s clear M charge. · he availability of tools to assess and improve effectiveness and T quality (such as a feedback loop as part of a learning system, incen- tives, and technical assistance). Those who influence population health can be held accountable through two models: · ontract model: When an oversight party has direct control over C implementers (for example, through statute or funding), standard direct methods of accountability can be used, with the caveat that accountability indicators are used to measure execution of agreed- on strategy. Holding implementers accountable in this context may involve regular programmatic progress reports, such evaluations as program reviews, and other tools typically used in the work of continuous quality improvement. · utual accountability3 (or compact) model: When no oversight M party has financial or other direct authority over those who are implementing, stakeholders must assume both an oversight role and an implementation role. Involved parties agree on overall priorities and strategies and then on actions and measures of actions that each organization will undertake. The group—which may take the form of a coalition, alliance, board, or other structure—holds individual organizations accountable for performance through public report- ing and other agreed-on mechanisms, such as incentives for future leadership roles and funding. Compact refers loosely both to the social compact and to the coalitions or other structures formed in many communities, agreements entered into, and other creative and innovative mechanisms used around the country to bring varied stakeholders together to assess health (or other community needs), devise strategies for improving it, and evaluate performance in 3Mutual accountability is used in international-development circles (for example, OECD, 2009; World Bank, 2008). OECD (2009) defines it as “a process by which partners hold one another responsible for the commitments that they have voluntarily made to each other.”

OCR for page 111
115 MEASUREMENT AND ACCOUNTABILITY implementing the strategies. Frameworks and measures that link interventions to outcomes can help facilitate all this. For example, in contract accountability, funds might be given to a stakeholder by local government or by a foundation to create an anti- tobacco media campaign or by a health department to a community-based organization for provision of smoking-cessation services. In mutual ac- countability, an agreement might be drafted by an antismoking coalition to advocate jointly for a specific legislative strategy, such as tobacco taxes. In short, the type of accountability depends not on the category of entity but on what it is accountable for doing. Those in the contract model can be held accountable by the individuals, agencies, or organizations that hold authority—legislators, a chief executive, boards of health, public health agencies, a philanthropic organization, taxpayers, and so on. Those who entered into an agreement in the mutual accountability or compact model will be accountable to whomever they have entered into agreement with, possibly including an array of system stakeholders. This framework for accountability works whether or not the oversight party has financial, administrative, or other control over the implementing party, but the specifics must be operationalized differently in the two set- tings. Regardless of setting, accountability depends on good measurement and links to the standard sets of outcome measures and measures of com- munity health in Chapter 3, Recommendation 2. For an overview of the framework for measurement in accountability that the committee discusses in this chapter, see Figure 4-2. An important element of the framework is that health-system partners need to align and coordinate their efforts con- sistently to ensure the greatest impact and achieve population health goals. Challenges for Measurement and Accountability There are many challenges to implementing a measurement framework for accountability. For many of the determinants of health, no specific entity or body is charged with improving a given determinant and made accountable for it. For example, food deserts may be one factor contribut- ing to poor nutritional status and obesity of some Americans (Franco et al., 2009; USDA, 2010), but the authority (or responsibility) for addressing this problem is unclear and widely distributed among various public-sector and private-sector entities (as examples of the former, local government planning and zoning policies, and tax incentives for businesses; and as an example of the latter, supermarket-chain decisions about the location of new stores). A simple, quantifiable outcome-based measure would be ideal (for example, easy to communicate and easy to understand) for evaluating the performance of public health agencies and other stakeholders in the health

OCR for page 111
116 FOR THE PUBLIC’S HEALTH: MEASUREMENT BOX 4-1 Evidence Base and Public Health Research Performance measurement is especially important when evidence to elu- cidate the pathways from system inputs (such as resources and capacities) to system outputs and outcomes is unavailable. To affect intermediate and ultimate health outcomes (for example, decreases in obesity and in all distal outcomes for which obesity is a primary risk factor, such as diabetes and CVD morbidity and mortality), public health agencies and other stakeholders must use evidence- based population-level interventions whenever possible and translate them into metrics for accountability. Gaps in knowledge should stimulate research and evaluations (Glasgow, 2010). Individual and population-based strategies (that is, the medical model and the ecologic model) differ considerably, as discussed in Chapter 1. For example, administering a vaccine to a patient is an individual-based intervention; ensuring optimal levels of immunization in a community or nation is a population-based strategy. In many areas, population-based strategies are not well developed and have less precise effect sizes. The nation’s public health research enterprise is producing an expanding body of evidence concerning population-level (public health) interventions, programs, and policies that are efficacious and cost-effective in reducing health risks for specific populations at risk (for example, preventive interventions documented in the Guide to Community Preventive Services, 2010). Many of the interventions have been evaluated for their effects on individual- level health outcomes or outcomes observed in small and controlled population groups. As a result, relatively little evidence suggests what scale of implementation must be achieved to produce a sustained effect on population health at the level of an entire community, region, or state. For example, some evidence suggests the vaccination coverage that needs to be achieved to provide optimal protection against vaccine-preventable diseases, but this type of evidence is lacking for many other types of public health programs, policies, and interventions, such as those which target obesity-prevention programming and food-safety inspection. Similarly, there is relatively little evidence to suggest how to achieve the scale and quality of implementation needed to affect health on a population system. However, holding the agencies and organizations accountable for specific health outcomes—such as reduced rates of cardiovascular disease (CVD), diabetes, or obesity—or specific modifiable determinants of health, such as smoking prevalence in the community, is not possible for several reasons: · here is a naturally shifting baseline of diseases and other condi- T tions (both up and down) for all health outcomes in a commu- nity, regardless of whether a public health intervention has been implemented.

OCR for page 111
117 MEASUREMENT AND ACCOUNTABILITY level. Implementation research at the level of the public health delivery system is needed to produce such evidence, including how best to divide and coordinate implementation responsibilities among available public-sector and private-sector stakeholders and specifically what roles government public health agencies should play vis-à-vis other stakeholders in the health system; what levels of human, monetary, institutional, technologic, and information resources are required for successful implementation; and what complementary mix of services, programs, and activities must be available (for example, enhanced HIV screening will have minimal public health effect if access to treatment is not simultaneously ensured). There are also large evidence gaps concerning the crosscutting public health practices that are required to facilitate decision-making and to ensure successful implementation of interventions. Those practices include community health as- sessment, epidemiologic investigation, community health planning, policy devel- opment, communication, workforce development, evaluation and monitoring, and quality improvement. For the public health interventions, programs, and policies that currently are supported by strong evidence, a measurement system is needed to assess adoption, reach, and implementation fidelity at state and local levels. The system would facilitate research to address current gaps in evidence and would support accountability mechanisms. It could also allow detection of practice variation among public health agencies, communities, and states and identify outcomes (health and economic consequences) that result from practice variation, thus allowing targeted improvement in efficiency and effectiveness. Correspondingly, such a system could be used to support public reporting and benchmarking, ac- creditation and quality-improvement applications, performance-based contracting, and pay-for-performance applications. There is a lack of precision with which public health workers can say they have achieved outcomes in a given community. The circumstances and programs that maximize health are many, and their relationships with one another are not always well understood. Dynamic models to which information can be added (as described in Chapter 3) can also create an evidence base from which to link salu- tary processes that lead to better intermediate outcomes that in turn increase the health-adjusted life expectancy of a population. · here is a lack of precision of the effect size for known interventions— T even for those considered best practices—partly because of other underlying conditions in a community (the determinants of health from the individual level of genes to the broadest environmental factors) (see Box 4-1 on evidence-based research). · here may be stakeholders (such as private-sector entities) that are T not part of a framework for accountability whose actions (both supportive of and detrimental to health) can substantially influence the success of interventions. · here is a lack of knowledge about effective interventions for many T health challenges that are identified as priorities. This may require

OCR for page 111
118 FOR THE PUBLIC’S HEALTH: MEASUREMENT innovation and implementation of “promising practices” whose efficacy is uncertain (see Box 4-1). · t may take many years or even decades for the results of public I health interventions to materialize. Such long timelines may not meet the needs of policy-makers and the public, who need to see and use intermediate measures that demonstrate progress. Measuring health outcomes is important and helps the system (at all levels) to know where it stands; owing to the factors listed above, however, distal health outcomes (such as death and diseases) are not useful in the context of accountability. Because of those factors, there is confusion and inconsistency regarding how to implement a framework for accountability in the nation’s health system (again, defined as the multiple partners work- ing to improve population health). The lack of such a framework and the lack of consistency (for example, in what is measured) can confuse policy- makers and the public and erode their confidence in system performance. Transparency in measuring performance and in demonstration of account- ability to the public and to policy-makers is a critical underpinning of any population health effort. The committee concludes that a framework for accountability is needed that includes · greement among implementing agencies, stakeholders, and those A holding them accountable on specific plans of action for targeting health priorities. · olding of implementing agencies or stakeholders accountable for H execution of the agreed-on plans (strategies, interventions, policies, and processes). · easurement of execution and outcomes of the agreed-on plans M and agreement on revisions to a plan of action. A model of accountability is needed that works both when there are areas with established (for example, evidence-based) best practices (as in the case of tobacco prevention) and when there is a less well-developed evidence base (as in the case of obesity prevention). The framework that the committee proposes applies in both situations because accountability measures assess the execution of agreed-on strategies. In settings where there are best practices based on evidence, accountability is primarily fidel- ity to established models of effective interventions. In settings where there are no clear best practices, accountability is based primarily on efficient and effective management of agreed-on innovative interventions (or programs or processes), including the placing of a higher premium on evaluation and modification as new information becomes available.

OCR for page 111
119 MEASUREMENT AND ACCOUNTABILITY In the larger context of accountability, there are other unique challenges and issues in establishing accountability for population health, including how to align missions of diverse organizations or stakeholders in pursuit of shared population health goals at the environmental level; the need for strategic agreement established through a spectrum of mechanisms, such as law, financial and other types of incentives, and voluntary agreements; addressing challenges inherent in collective action (for example, free riders, or interested parties that benefit but do not contribute); and the presence of internal accountabilities (organization missions) and external account- abilities (contracts, and legal, financial, or social compacts or pressures) throughout the system. Role of Measurement in Accountability Measurement has unique and powerful roles to play in an accountability system, especially when other legal and financial drivers of accountability are weak or absent. Measurement can elucidate shared responsibilities for population outcomes and reveal the levels of effort and achievement needed to reach shared objectives. Measurement can be used in tandem with and is also a vehicle for legal mechanisms (such as contract compliance and liabil- ity) and incentive and financing mechanisms (such as pay-for-performance, eligibility, and resource allocation). Measurement comes into play both at the beginning of the account- ability process (for example, to inform a community and help it to decide where resources should be directed) and at the end after coordination and development of a strategy and its execution (for example, to measure outcomes of processes of a health department or business). Measurement provides a basis of alignment of efforts among health-system stakehold- ers. Efforts may be strategies required by law, or agreed on in contracts or agreements. Measuring and reporting on process indicators can help to strengthen accountability pathways. Having indicators available for those who take part in the accountability process may demonstrate the need for greater involvement of all stakeholders in the health system. Indicators could help to illustrate the lack of collective action in the current system in which many of those who can and do affect health (both favorably and adversely) are not part of a formal or organized system. These measures would make the contributions (or lack thereof) of various stakeholders observable and help to spur collective action. The committee’s next report will address the legal mechanisms that can assist in the alignment of strategies. Figure 4-2 depicts a framework for the measurement dimension of ac- countability that draws on the work of the IOM Committee on Quality of Health Care in America (see Berwick, 2002; IOM, 2001). In that context, a framework was provided to demonstrate the changes needed in the US

OCR for page 111
Nonmodifiable factors in the environment 120 Community Health Measures Health Outcome Measures Coordination and Assessment by Gov PH (needs assessment, Level D: prioritization, Environment*** agreement) Coordination and Assessment by Gov PH (evaluation, CQI, measure accountability) Resources & Activities & Accountability Capacities Processes* Measures** Gov Public Health Other Gov Community Media Level C: Organizations Academia Medical Care Figure 4-2 broadside Employers/Business Level B: Microsystem (programs, interventions, evidence base on specific areas of population health) Level A: Individuals FIGURE 4-2 A framework for measurement in accountability. NOTE: Gov = Government; PH = Public Health; CQI = Continuous Quality Improvement. * Activities and processes are influenced by agreed-on strategies (strategies agreed on by those being held accountable and those holding other parties accountable through contracts or compact agreements). ** Accountability measures assess how well the agreed-on strategies are executed and this may also be thought of as strategy execution measures. ***Stakeholder activities both influence the environment and work within it to shape outcomes. SOURCE: Adapted from Berwick, 2002; IOM, 2001.

OCR for page 111
121 MEASUREMENT AND ACCOUNTABILITY medical care delivery system, and four levels were described: Level A, the experience of patients and communities; Level B, the microsystem of care (for example, provider practices); Level C, organizations (for example, managed-care organizations); and Level D, the environment shaped by policy, payment, regulation, and accreditation. The present committee believes that that framework holds relevance for its own examination of measurement in the context of accountability and has adapted it for its own purposes. The cycle begins after a needs assessment has been done, priorities set, and a plan agreed on. Level A in the committee’s adaptation of the framework includes persons (whose aggregated health information constitutes health-outcome measures) and neighborhoods. Level B refers to microsystems, which in the context of population health4 are programs, policies, and interventions that may be thought to refer to the points of contact or interactions among community groups, local businesses, others in the neighborhood, and their local public health agencies and allied entities. An example of microsystems is an in- teraction among a health department, a local medical care provider, com- munity coalition, or local business concerning a particular health outcome. Specifically, a health department could assist a food retailer in facilitating healthful customer choices or could support a local business in develop- ing a workplace prevention and wellness program. Often in public health, such microsystems need to align and integrate across organizations; for example, the local cancer-control program should feed into the statewide cancer-control program, which feeds into the national program. Level C consists of organizations described as actors in the public health system in the 2003 IOM report The Future of the Public’s Health in the 21st Century (IOM, 2003a) and as components of the health system. The organizations include the local public health agency, hospitals and other clinical care enti- ties, community organizations, schools, businesses, religious congregations, and many others that perform roles that influence health outcomes. Level D refers to the environment, which includes a variety of social, physical (both naturally occurring and constructed), and economic factors and is shaped in part by social realities, large-scale policies (and political will), and economic arrangements (Syme and Ritterman, 2009). Figure 4-2 also depicts account- ability pathways for all levels but focuses on Level C—the organizations that perform functions that affect health outcomes. CONTEXT AND HISTORY Performance measurement and reporting are not new ideas in public health; agencies have had to report on their performance to federal or 4 As is sometimes pointed out, the patient in public health practice is the community.

OCR for page 111
134 FOR THE PUBLIC’S HEALTH: MEASUREMENT population health is to function, the committee believes that a continuing focused effort in an existing organization (such as the PHAB, but with a broader mandate) or a new accountability organization may be needed to develop accountability measures and to track and report commitments by public health agencies and other stakeholders. To help federal agencies, pub- lic health funders, and communities, the accountability organization would involve them in the development of accountability measures and reporting requirements. The accountability organization would need the capacity to understand the underlying logic model that links the actions taken by public health agencies and stakeholders with intermediate and health outcomes so that it can help to identify the critical processes, resources, and capabilities of each stakeholder that are central to the intervention strategy. Such an organization · ould assist in the measurement and reporting of performance of C nongovernment public health stakeholders that are accountable for upholding mutual accountability “compacts” formed with others in improving community health outcomes. · ould validate and serve as a repository of accountability indica- C tors and serve as a facilitator of process integrity and objectivity on behalf of funders, taxpayers, and communities. · ould need to be constituted appropriately to incorporate neces- W sary expertise and demonstrate needed independence. TYPES AND EXAMPLES OF NEEDED ACCOUNTABILITY MEASURES There are various ways to measure accountability of stakeholders in the health system. For clinical care services delivered in public health department clinics, it may be most reasonable to consider measurement strategies that are used in the clinical care delivery sector. Such strategies (Healthcare Effectiveness Data and Information Set [HEDIS] measures for instance) are likely to grow in importance in the wake of major changes to the clinical care delivery system, and the clinical services provided in public health may also change as a result of a decrease, due to the ACA, in the number of uninsured people who need the immunization, family-planning, or communicable-disease services offered by clinics that public health agen- cies operate. With regard to funding of public health agencies, a local system of public health accounts is needed to enable management to understand how well resources are aligned with interventions and outcomes. As noted in Chapter 1, measurement of financial resources and their effects on services and outcomes is inadequate. Public health agencies do not collect data in a

OCR for page 111
135 MEASUREMENT AND ACCOUNTABILITY standardized way to link decisions on how resources are spent at the local level (in the health department) with the population health outcomes that they are designed to improve. And, as discussed in this chapter and else- where in this report, the linking of inputs and outcomes is not a simple or straightforward process, for a variety of reasons (see Box 4-4 for a discus- sion of so-called scorecards). Information is needed not merely by funding stream or categorical program; rather it is needed as a type of accounting— what resources go toward what health outcomes, and what is the effect? The committee began to discuss this topic and expects to gather additional information in preparing its third report. For many stakeholders in the health system, developing accountability measures has been challenging for several reasons. Many in the business sector or non-health-related parts of the nonprofit sector have not always seen themselves as stakeholders in health. There may be data gaps or dif- ficulties in gathering needed data, and evidence available to guide the selec- tion of measures may be sparse. Such measurement systems as HEDIS may serve as a partial model, but there are considerable differences, compared with clinical settings (described in more detail earlier in this chapter), in the ability to link cause and effect in population health and in how account- ability is traced both in the government public health infrastructure (which is supported by taxpayers and accountable to them and to elected officials) and among the many stakeholders in the system (where accountabilities are much less clearly defined and certainly more difficult to monitor and evalu- ate). However, the committee believes that the concerted efforts of national public health leaders, with support from public health systems and services research and input from communities, can move the field toward develop- ing and implementing good performance measures that can be adopted by implementers and those holding them accountable. Criteria for selecting such measures may include face validity (meaningfulness, relevance, and understandability), feasibility (availability or collectability of data), meth- odologic soundness (validity and reliability), and fairness (to the stakehold- ers whose performance they will evaluate). The set of performance measures used may differ, depending on a com- munity’s identified needs and priorities, on the mix of stakeholders, and on the expectations of funders. The Ten Essential Public Health Services (see Box 4-2) may also serve as a tool for identifying measures to assess. See Table 4-1 for some examples of possible measures of performance for agreed-on strategies. Over the last few decades, efforts to measure and report quality and performance related to health have increased. Spurred by such employer initiatives as the Leapfrog programs for reporting hospital-service quality, such government initiatives as Medicare quality measures for ambulatory care and hospitals, and health care quality organization standards of the

OCR for page 111
136 FOR THE PUBLIC’S HEALTH: MEASUREMENT BOX 4-4 Improving the Next Generation of “Scorecards” Several sets of indicators have been developed or have been made available to the public in the last 2 decades. Prominent examples include the State of the USA (SUSA) health measures (developed as part of a national, federally driven key-indicators effort), the Community Health Status Indicators effort (supported by HHS agencies and several nonprofit organizations), and the County Health Rank- ings (developed by academic researchers with foundation support) (Community Health Status Indicators, 2009; County Health Rankings, 2009; SUSA, 2010). Some indicator sets are sometimes called scorecards, and the committee believes that it is important to address this terminology. Although these indicator sets (discussed in Chapter 2) attempt to provide snapshots of health by county or by state (in the case of SUSA) and can speak volumes about the needs and chal- lenges in a given community, the snapshots generally are not intended to and do not inform about the performance of the public health agencies in the communities and about the resources available to them. These sets also provide only limited information about the underlying determinants of health (most commonly, data on educational attainment and income). It is the committee’s understanding that true scorecards (which are available in many fields, including health) are intended to convey information about performance (such as the quality of services provided) either for internal quality-improvement purposes or for external communication, but some of the indicator sets currently called scorecards are in fact unable to provide the type of information a true scorecard would give. The committee believes that policy-makers and the public cannot draw useful inferences about public health agency competence or capabilities from these so-called scorecards (in their most common current formats). Examples of true scorecards may be found in the clinical National Committee on Quality Assurance, the National Quality Forum (NQF), and others, the medical care delivery system has begun to report performance and quality-improvement activities linking process to health outcomes. What the present committee recommends should not be consid- ered in isolation from the efforts of those groups, and existing efforts should be incorporated when possible and when they are pertinent to population health. For example, the Centers for Disease Control and Prevention Na- tional Public Health Performance Standards program is designed to measure public health practices at the state and local levels and provides the tool Mobilizing for Action through Planning and Partnerships, which evaluates the capacity of local public health systems to conduct the Ten Essential Public Health Services. In the clinical care setting, the NQF uses continuous quality improvement as part of its vision, which is a facet of a measurement framework for accountability recommended by the committee. The national

OCR for page 111
137 MEASUREMENT AND ACCOUNTABILITY context, but, as discussed in this chapter and elsewhere in this report, the clinical frameworks do not translate easily to public health practice. The committee believes that it is crucial that future indicator sets described as community scorecards look not only at important distal health outcomes and determinants (largely the measures of community health discussed in Chapters 2 and 3) but also at their interrelationships with upstream underlying processes and policies, which may differ at the local, state, and national levels. Given that health outcomes are the products of a chain of proximal and distal influences, the interrelationships among health and its determinants should serve as an organizing framework for future measurement of health outcomes. Rather than presenting a “flat-file” list of health indicators, for every distal health outcome of importance, future efforts would map out the causal web of determinants that af- fect outcomes and the sequence of upstream activities that enable communities to alter the determinants. The systems-modeling activities described and called for in Chapter 3 could transform this kind of performance measurement by leading to a next generation of health indicators that measure performance along complex and nonlinear causal pathways and at the national, state, and local levels. It would require research and development to identify the most important health outcomes at the distal end of the pathway, the activities that are effective at each stage of influence, and the best metrics for each indicator. To accomplish that, changes in survey administration and data collection and analysis will be necessary. As dis- cussed in Chapter 3, there may be value in individual health data from electronic health records that could be collectively analyzed with appropriate privacy and security safeguards (such as aggregation) to complement the understanding of community or population health. Without such understanding, the implementation of policies and other strategies that affect social, environmental, and behavioral determinants cannot be measured, monitored, and improved. accreditation effort, as discussed earlier, will also be a useful tool as it moves forward in its development. Implementation of the Measurement Framework for Accountability Chapter 1 discusses the causes, and causes of causes, that lead to the untoward health outcomes of infant mortality and cardiovascular disease. As is true for the vast majority of conditions or illnesses, neither of those outcomes is amenable to improvements that are influenced solely by public health agencies. A variety of stakeholders are necessary to alter the micro and macro societal conditions in which infants die and people suffer heart attacks and congestive heart failure. Beyond the most macro level—the deeply embedded socioeconomic realities that characterize the nation— employers, community organizations, clinical care providers, schools, busi-

OCR for page 111
138 FOR THE PUBLIC’S HEALTH: MEASUREMENT TABLE 4-1 Examples of Measures of Common Agreed-on Strategies Sample Measure Stakeholder Number of employers who have voluntarily Business, nonprofits adopted and complied with smoke-free workplace policies Number of (nonchain) restaurants voluntarily Business (retail) posting or complying with requirements for disclosure of nutritional information School adherence to nutrition guidelines, Schools including removal of some vending-machine products Planning and zoning decisions consistent with Planning department local needs Small-business compliance with smoking bans Business (something intermediate to) high school Schools, community-services agencies graduation rates Percentage of community housing that is Planning department, local affordable (give parameters) government, developers Percentage of community housing that Police, planning, local government, is safe and livable (give parameters) community groups, faith-based organizations Percentage of poor children (specify percentage Public health agency, social services, of federal poverty level) who receive early- nonprofit organizations, childhood interventions (from public health including advocacy groups and and other social-service agencies) philanthropic organizations Percentage of medical-insurance plans that Clinical care implement health-literacy education; Schools percentage of medical-insurance plans or medical care providers that adopt health- literacy strategies and implement steps to increase cultural competence of their staff; measures of health literacy in adolescents Percentage of employers that provide wellness Business, employers services to employees Percentage of employers who adopt policies Business, employers supportive of breastfeeding mothers (including dedicated, acceptable space and time to pump) Percentage of baby-friendly (that is, breastfeeding- Clinical care supportive) hospitals (specific parameters have been described elsewhere) nesses, and many others can undertake strategies that address one or both of those outcomes (and many others). For example, in a community that has unacceptable infant mortality, the local public health agency might serve as the convener of stakeholders, alerting other community organizations to the problem’s root causes by pre- senting evidence of associations between different types of changeable risk

OCR for page 111
139 MEASUREMENT AND ACCOUNTABILITY factors and infant outcomes. In doing so, it might identify weaknesses in the local programs, services, and interventions available to prevent unplanned pregnancies and poor birth outcomes. Those convened might also engage a broader circle of participants who are in a position to influence environ- mental changes. Those convened could then consider, plan, and deploy an array of strategies. After reaching consensus on the top (most effective, evidence-based, and locally appropriate) strategies to be undertaken, the coalition could develop agreements with various stakeholders who would all commit to playing a concrete role in improving the outcome of concern. For example, the local public health or social services agencies would com- mit to better links to clinical care providers who are working with newly pregnant women to ensure that at-risk women receive case management and other essential services. The public health agency and clinical care providers in the community might develop agreements to ensure that no pregnant woman misses prenatal care services because of insurance status or difficulty in accessing a provider. Local businesses that cater to women and families could join in a mass-media and social marketing campaign on the importance of prenatal care. Schools could initiate or intensify efforts to educate adolescents about family planning and refer them to clinical services (as part of a broader effort to delay sexual activity and improve awareness and behaviors). A broader coalition might work to alter the community environment more substantially. For example, local employers and businesses learning of the relationship of secondhand smoke to poor infant outcomes might commit to initiating or enhancing smoke-free environments to diminish exposure of pregnant women and alter the behaviors of other members of the community. Schools boards, learning of the relationship between gradu- ation rates and infant mortality, might be persuaded to redouble efforts to increase graduation. Town planners, alerted to an association between early sexual activity and lack of recreational outlets, might agree to work to de- sign programs and build facilities to serve adolescents. Food retailers, made more aware of the relationship between nutrition and birth outcomes, might commit to developing food and menu labeling as part of a communication effort. The process of involving those many organizations could include developing a coalition that could acquire formal nonprofit status (501(c)3) and apply for funding from relevant private and public-sector funders. Implementing a measurement framework for accountability could include agreements and contracts (in cases in which funding is provided) and a variety of tools for communicating with the public on the status and progress of the community’s joint efforts (for example, through newsletters, news releases, and monthly, quarterly, or annual reports). Evidence-based indicators could be selected to help the community to hold accountable all stakeholders who have agreed to contribute to the initiative in some

OCR for page 111
140 FOR THE PUBLIC’S HEALTH: MEASUREMENT manner. Indicators could include an array of process measures, such as the percentage of newly pregnant women receiving social support (the Women, Infants, and Children federal food and nutrition program and food stamps) who are referred for additional services, a measure of the level of tracking and follow-up of women who do not access needed services, indicators of clinicians’ attempts to initiate smoking cessation in pregnant women, mea- sures of mass-media and other communication outputs, and the percentage of businesses that adopt smoke-free policies to decrease exposure of preg- nant women. As the work progresses, indicators that can be used to hold stakeholders accountable will become more refined, interventions will also be fine-tuned, efforts to collect data will begin to produce results, and the public will have regularly updated information about progress in addressing one of the community’s top health needs. The committee believes that the task of identifying performance mea- sures can be simplified under the proposed framework because, for the most part, performance measures need to be measures of execution of strategies or of immediate outcomes of execution. CONCLUDING OBSERVATIONS In this chapter, the committee has outlined the three components of a measurement system for accountability that can be applied to the entire health system, from public health agencies to a vast array of stakeholders, and made a recommendation describing standard approaches and measures for implementing the framework in the context of both contract account- ability and mutual or compact accountability. Accountability requires measurements that track resources to outcomes; in general, these measures are not yet developed. Accountability is primarily for processes required by funders or agreed to by those in mutual account- ability arrangements—processes over which organizations have control, rather than health outcomes for which public health is often only one of many contributors and determinants and therefore cannot be held directly accountable. Simple measures of accountability (clear lines from inputs to outputs) based on quantified improvements in health outcomes, although desirable, are not possible, so the three-part, more complex framework of account- ability measurement and continuous quality improvement presented in the report is needed. This measurement strategy can be operationalized across the spectrum of degree of certainty about best practices and so avoids “ac- countability paralysis” from lack of precise science. The strategy can be operationalized in both contract and compact operating environments and for all stakeholders, and the principles embodied are applicable at national,

OCR for page 111
141 MEASUREMENT AND ACCOUNTABILITY state, and local levels. Priority steps to develop and implement the measure- ment framework should be taken. REFERENCES Alameda County Public Health Department. 2008. Life and Death from Unnatural Causes: Health and Social Inequity in Alameda County. Oakland, CA: Alameda County Public Health Department. Baicker, K., D. Cutler, and Z. Song. 2010. Workplace wellness programs can generate savings. Health Affairs 29(2):304-311. Bates, L. J., B. A. Lafrancois, and R. E. Santerre. 2010. An empirical study of the con- solidation of local public health services in connecticut. Public Choice DOI 10.1007/ s11127-010-9606-9. Benjamin, G., M. Fallon, P. E. Jarris, and P. Libbey. 2006 (September). Final Recommenda- tions for a Voluntary National Accreditation Program for State and Local Public Health Departments. Alexandria, VA: PHAB. Berwick, D. M. 2002. A user’s manual for the IOM’s ‘quality chasm’ report. Health Affairs 21(3):80-90. Bourgeois, F., W. W. Simmons, K. Olson, J. S. Brownstein, and K. D. Mandi. 2008. Evalua- tion of influenza prevention in the workplace using a personally controlled health record: Randomized controlled trial. Journal of Medical Internet Research 10(1):e5. Bovaird, T. 2008. Emergent strategic management and planning mechanisms in complex adaptive systems: The case of the UK best value initiative. Public Management Review 10(3):319-340. CDC (Centers for Disease Control and Prevention). 2010. Health Impact Assessment: Fact Sheet. Atlanta, GA: CDC. Communities Count. 2008. Summary of Social and Health Indicators: Communities Count 2008: Executive summary. In Communities Count 2008: Social and health indicators across King County. Seattle, WA: Communities County. Pp. iv-xi. Community Health Status Indicators. 2009. About the Community Health Status Indicators Project. http://www.communityhealth.hhs.gov/AboutTheProject.aspx?GeogCD=&Peer Strat=&state=&county= (January 6, 2010). County Health Rankings. 2009. About the Community Health Status Indicators Project. http://www.countyhealthrankings.org/latest-news/how-washington-county-residents- stay-healthy (January 6, 2010). Department of Population Health Sciences. 2008. Wisconsin County Health Rankings—2008. Madison: University of Wisconsin School of Medicine and Public Health. EPA (Environmental Protection Agency). 2010. HUD-DOT-EPA Interagency Partnership for Sustainable Communities. http://www.epa.gov/smartgrowth/partnership/index.html (September 8, 2010). Franco, M., A.V. Diez-Roux, J. A. Nettleton, M. Lazo, F. Brancati, B. Caballero, T. Glass, and L. V. Moore. 2009. Availability of healthy foods and dietary patterns: The multi-ethnic study of atherosclerosis. American Journal of Clinical Nutrition 89(3):897-904. Glasgow, R. E. 2010. HMC research translation: Speculations about making it real and going to scale. American Journal of Health Behavior 34(6):833-840. Goetzel, R. Z., and R. J. Ozminkowski. 2008. The health and cost benefits of work site health- promotion programs. Annual Review of Public Health 29:303-323. Guide to Community Preventive Services. 2010. What Is the Community Guide? http://www. thecommunityguide.org/index.html (August 16, 2010). The Healthy Development Measurement Tool. 2006. Case Studies: Hope SF-Redevelopment of San Francisco’s Public Housing. http://www.thehdmt.org/case_studies.php (September 5, 2010).

OCR for page 111
142 FOR THE PUBLIC’S HEALTH: MEASUREMENT Heinen, L., and H. Darling. 2009. Addressing obesity in the workplace: The role of employers. Milbank Quarterly 87(1):101-122. HHS (Department of Health and Human Services). 2010. Affordable Care Act: Laying the Foundation for Prevention. http://www.healthreform.gov/about/index.html (June 22, 2010). HHS and CDC. 2008. Local Public Health Governance Performance Assessment: Model Standards. Washington, DC: HHS and CDC. Houle, B., and M. Siegel. 2009. Smoker-free workplace policies: Developing a model of public health consequences of workplace policies barring employment to smokers. Tobacco Control 18(1):64-69. IOM (Institute of Medicine). 1988. The Future of Public Health. Washington, DC: National Academy Press. IOM. 1997. Improving Health in the Community: A Role for Performance Monitoring. Wash- ington, DC: National Academy Press. IOM. 2001. Crossing the Quality Chasm: A New Health System for the 21st Century. Wash- ington, DC: National Academy Press. IOM. 2003a. The Future of the Public’s Health in the 21st Century. Washington, DC: The National Academies Press. IOM. 2003b. The Governmental Public Health Infrastructure. In The Future of the Public’s Health. Washington, DC: The National Academies Press. Pp. 96-177. Keall, M., M. G. Baker, P. Howden-Chapman, M. Cunning, and D. Ormandy. 2010. Assessing housing quality and its impact on health, safety and sustainability. Journal of Epidemiol- ogy and Community Health 64(9):765-71. Koivusalo, M. 2010. The state of health in all policies (HiAP) in the European Union: Potential and pitfalls. Journal of Epidemiology and Community Health 64:500-503. Krieger, J., and D. L. Higgins. 2002. Housing and health: Time again for public health action. American Journal of Public Health 92(5):758-768. Lemstra, M., and C. Neudorf. 2008. Health Disparity in Saskatoon: Analysis to Intervention. http://www.uphn.ca/doc/public/HealthDisaparitiesinSaskatoonExecutiveSummary.pdf (January 2010). MacDonald, J. M., R. J. Strokes, D.A. Cohen, A. Kofner, and G. K. Ridgeway. 2010 The ef- fect of light rail transit on body mass index and physical activity. American Journal of Preventive Medicine 39(2):105-112. Michigan Department of Community Health. 2008a. Increasing Access to Healthy Foods: Michigan’s New Property Tax Incentive for Retail Food Establishments (Public Act 231 of 2008). Lansing: Michigan Department of Community Health. Michigan Department of Community Health. 2008b. Reducing infant mortality in Michigan: Lessons from the field. Lansing: Michigan Department of Community Health. NACCHO (National Association of County and City Health Officials). 2009a. Integrating performance improvement processes: MAPP, National Public Health Performance Stan- dards, and accreditation. Washington, DC: NACCHO. NACCHO. 2009b. Mobilizing for Action through Planning and Partnerships and the National Public Health Performance Standards Program: What is the relationship? Washington, DC: NACCHO. National Center for Appropriate Technology. 2010. Smart growth resource: HUD, DOT and EPA announce interagency partnership for sustainable communities. http://www.smart- growth.org/misc/default.asp?art=83&res=1024 (September 2, 2010). NPLAN (National Policy and Legal Analysis Network). 2009. Model Menue Labeling Ordi- nance. Oakland, CA: National Policy and Legal Analysis Network. NYC (New York City) Department of City Planning. 2010. Fresh Food Stores: Overview. http://home2.nyc.gov/html/dcp/html/fresh/index.shtml (September 20, 2010).

OCR for page 111
143 MEASUREMENT AND ACCOUNTABILITY OECD (Organisation for European Economic Co-operation). 2009. Mututal Accountablity. Paris, France: OECD. Office of Health Assessment and Epidemiology. 2010. Life Expectancy in Los Angeles County: How Long Do We Live and Why? A Cities and Communities Health Report. Los Angeles, CA: County of Los Angeles Department of Public Health. Okie, S. 2007. The employer as health coach. New England Journal of Medicine 357(15): 1465-1469. Ozminkowski, R. J., D. Ling, R. Z. Goetzel, J. A. Bruno, K. R. Rutter, F. Isaac, and S. Wang. 2002. Long-term impact of Johnson & Johnson’s Health & Wellness Program on health care utilization and expenditures. Journal of Occupational and Environmental Medicine 44(1):21-29. PHAB (Public Health Accreditation Board). 2009. Proposed state standards and measures, adopted by the PHAB board of directors, July 16, 2009 for PHAB Beta Test. Alexandria, VA: PHAB. PHAB. 2010. E-Newsletter: Public Health Accreditation Board. http://archive.constantcontact. com/fs030/1102084465533/archive/1103695445388.html (September 8, 2010). Public Health Functions Steering Committee. 1994. The Public Health Workforce: An Agenda for the 21st Century. Full Report of the Public Health Functions Project. Washington, DC: HHS. RWJF (Robert Wood Johnson Foundation) and W.K. Kellogg Foundation. 2006. Turning Point Performance Management Collaborative. http://www.turningpointprogram.org/ (September 2, 2010). Seattle and King County Public Health Department. 2010. King County Community Health Indicators. http://www.kingcounty.gov/healthservices/health/data/chi.aspx (January 6, 2010). Simon, P. A., P. Leslie, G. Run, G. Z. Jin, R. Reporter, A. Aguirre, and J. E. Fielding. 2005. Impact of restaurant hygiene grade cards on foodborne-disease hospitalizations in Los Angeles County. Journal of Environmental Health 46(7):32-36. Simon, P., C. J. Jarosz, T. Kuo, and J. E. Fielding. 2008. Menu Labeling as a Potential Strategy for Combating the Obesity Epidemic: A Health Impact Assessment (HIA). Los Angeles, CA: Los Angeles County Department of Public Health. Summers, C., L. Cohen, A. Havusha, F. Sliger, and T. Farley. 2009. Take Care New York 2012: A Policy for a Healthier New York City. New York, NY: New York City Department of Health and Mental Hygiene. SUSA (State of the USA). 2010. The State of the USA. http://www.stateoftheusa.org (September 1, 2010). Syme, S. L., and M. L. Ritterman. 2009. The importance of community development for health and well-being. Community Development Investment Review 5(3):1-13. Teisman, G. 2008. Complexity and management of improvement programmes: An evolution- ary approach. Public Management Review 10(3):341-359. University of Wisconsin Population Health Institute. 2010. Mobilizing Action Toward Commu- nity Health (MATCH): Population health metrics, solid partnerships, and real incentives. http://uwphi.pophealth.wisc.edu/pha/match.htm (June 10, 2010). USDA (United States Department of Agriculture). 2010. About the Food Environment Atlas. http://ers.usda.gov/FoodAtlas/about.htm (September 8, 2010). White House Task Force on Childhood Obesity. 2010. Solving the Problem of Childhood Obesity Within a Generation. Washington, DC: Executive Office of the President of the United States. World Bank. 2008. Mutual Accountability. New York, NY: United Nations Development Group.

OCR for page 111