. "4 Measurement and Accountability." For the Public's Health: The Role of Measurement in Action and Accountability. Washington, DC: The National Academies Press, 2011.
The following HTML text is provided to enhance online
readability. Many aspects of typography translate only awkwardly to HTML.
Please use the page image
as the authoritative form to ensure accuracy.
For the Public’s Health: The Role of Measurement in Action and Accountability
4
Measurement and Accountability
Accountability refers to “the principle that individuals, organizations and the community are responsible for their actions and may be required to explain them to others” (Benjamin et al., 2006). The notion of accountability has several meanings that span the fields of accounting, law, ethics, management theory and practice, and governance. Models of accountability include regulatory, legal, accreditation or certification (sometimes quasiregulatory), and pay-for-performance models, or their public health equivalent, eligibility for funding based on past performance. This chapter does not endeavor to examine all dimensions of accountability but rather focuses on the role of indicators in holding to account all stakeholders that contribute to the conditions for health in a community. Other dimensions of accountability will be examined in the committee’s later reports on public health law and funding. Two important examples are the intertwined topics of political will and governance (and governing bodies).1 As noted in this chapter, the governance (and related regulatory and funding) mechanisms that pertain to the work of local public health agencies are among the stronger and more concrete levers for holding agencies accountable.
This chapter examines performance2 indicators and how they can be implemented both at the level of governmental public health and in the con-
1
The National Public Health Performance Standards Program defines governing body as “the individual, board, council, commission, or other body with legal authority over the primary governmental public health agency” (HHS and CDC, 2008).
2
Performance refers to the interventions—policies, programs, and processes—implemented with the intent of improving population health; it represents one of the steps along the inputs-to-outputs or inputs-to-outcomes logic model presented in Chapter 2.
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole. Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 111
For the Public’s Health: The Role of Measurement in Action and Accountability
4
Measurement and Accountability
Accountability refers to “the principle that individuals, organizations and the community are responsible for their actions and may be required to explain them to others” (Benjamin et al., 2006). The notion of accountability has several meanings that span the fields of accounting, law, ethics, management theory and practice, and governance. Models of accountability include regulatory, legal, accreditation or certification (sometimes quasiregulatory), and pay-for-performance models, or their public health equivalent, eligibility for funding based on past performance. This chapter does not endeavor to examine all dimensions of accountability but rather focuses on the role of indicators in holding to account all stakeholders that contribute to the conditions for health in a community. Other dimensions of accountability will be examined in the committee’s later reports on public health law and funding. Two important examples are the intertwined topics of political will and governance (and governing bodies).1 As noted in this chapter, the governance (and related regulatory and funding) mechanisms that pertain to the work of local public health agencies are among the stronger and more concrete levers for holding agencies accountable.
This chapter examines performance2 indicators and how they can be implemented both at the level of governmental public health and in the con-
1
The National Public Health Performance Standards Program defines governing body as “the individual, board, council, commission, or other body with legal authority over the primary governmental public health agency” (HHS and CDC, 2008).
2
Performance refers to the interventions—policies, programs, and processes—implemented with the intent of improving population health; it represents one of the steps along the inputs-to-outputs or inputs-to-outcomes logic model presented in Chapter 2.
OCR for page 112
For the Public’s Health: The Role of Measurement in Action and Accountability
tributions of other health-system stakeholders. (As defined in Chapter 1, the health system comprises public health, clinical care, and other stakeholders that acknowledge their current and potential contributions to a community’s health.)
Governmental public health is not the only actor in the system that is accountable for or involved in creating the conditions for health. Clinical care providers are de facto stewards of a community’s health and are mandated or otherwise charged with health-related duties. Others, such as employers and businesses, may not currently see themselves as contributing to or detracting from a community’s health and well-being (see Chapter 1), but their recognition of their roles and their ability to contribute to health could be facilitated. They often face regulatory pressures, such as rules regarding environmental waste and pollution and zoning limitations. Others, such as community-based organizations, may be seasoned contributors to health, but there are no measurement frameworks for accountability for their work. Those roles are discussed in greater detail later in this chapter.
The measurement of performance and the demonstration of accountability and quality in clinical care have a long history, with a major national movement punctuated by milestone Institute of Medicine (IOM) reports on the subject of quality, federal quality initiatives (such as those undertaken by the Centers for Medicaid and Medicare Services), and the creation of such bodies as the National Committee for Quality Assurance (and its Healthcare Effectiveness Data and Information Set quality measures) and the National Quality Forum and its efforts to set national priorities and endorse standards for and conduct outreach and education on performance improvement in clinical care.
Accountability (in the broad sense of demonstrating results and effectiveness to the public) is a somewhat more recent focus in the public health community, and this is in part due to the complex array of factors that contribute to population health and the challenging nature of communicating about them. As described in Chapter 1, one challenge is that health outcomes (such as disease and death) have multiple interconnected causal pathways, and the science required to elucidate them is far from advanced in many cases. In addition, public health agencies, although broadly charged with ensuring the public’s health, have direct or clearly traceable responsibility for only a small proportion of those pathways.
The simple logic model introduced in Chapter 2 and reprised in Figure 4-1 suggests that a straightforward measurement framework for accountability would link all inputs (resources, capacities, processes, interventions, and policies) with outputs (intermediate and more distal health outcomes). However, there are many obstacles to such a framework, and these are discussed below. It is important to note that accountability is closely linked with needs assessment, planning, and priority-setting—activities identified at the beginning of the process.
OCR for page 113
For the Public’s Health: The Role of Measurement in Action and Accountability
FIGURE 4-1 From inputs to outputs logic model.
NOTE: The thickness of some arrows denotes the present report’s focus on those interactions.
The committee recognizes that detailed performance objectives may be identified and measurements conducted at each step of the continuum depicted in Figure 4-1 by all stakeholders in the system. For example, there may be specific objectives for public health agencies or for hospitals that assess their community’s health needs, objectives for the process of planning (such as the number of partners engaged in planning and collaborative planning activities undertaken), objectives that monitor resource use, and so on. In this chapter, however, the committee focuses on the “macro” or broader accountability for the entire continuum of community-health improvement (from needs assessment to the most distal outcomes) and not on any detail of the “micro” accountabilities that could be examined and described for each step in the process.
In the process of community-health improvement, after a community’s health needs are assessed, priorities are identified, and plans are made and implemented, performance measurement is needed to hold implementers (the full spectrum of system stakeholders in addition to public health agencies) accountable and to spur continuous quality improvement to increase the effectiveness, efficiency, and equity of actions taken to improve population health. Performance measurement is the main way to monitor accountability in the health system.
A FRAMEWORK FOR ACCOUNTABILITY
The measurement framework for accountability discussed in this chapter applies to the delivery of funded public health programs by public health agencies; the role of public health agencies in mobilizing the overall public health system; and the roles, contributions, and performance of health-system partners (other governmental agencies, private-sector stakeholders, and communities).
Assessing and measuring accountability at any level (local, state, or
OCR for page 114
For the Public’s Health: The Role of Measurement in Action and Accountability
national) and holding organizations accountable require the following four elements:
An identified body with a clear charge to accomplish particular steps toward health goals.
Ensuring that the body has the capacity to undertake the required activities.
Measuring what is accomplished against the identified body’s clear charge.
The availability of tools to assess and improve effectiveness and quality (such as a feedback loop as part of a learning system, incentives, and technical assistance).
Those who influence population health can be held accountable through two models:
Contract model: When an oversight party has direct control over implementers (for example, through statute or funding), standard direct methods of accountability can be used, with the caveat that accountability indicators are used to measure execution of agreed-on strategy. Holding implementers accountable in this context may involve regular programmatic progress reports, such evaluations as program reviews, and other tools typically used in the work of continuous quality improvement.
Mutual accountability3 (or compact) model: When no oversight party has financial or other direct authority over those who are implementing, stakeholders must assume both an oversight role and an implementation role. Involved parties agree on overall priorities and strategies and then on actions and measures of actions that each organization will undertake. The group—which may take the form of a coalition, alliance, board, or other structure—holds individual organizations accountable for performance through public reporting and other agreed-on mechanisms, such as incentives for future leadership roles and funding. Compact refers loosely both to the social compact and to the coalitions or other structures formed in many communities, agreements entered into, and other creative and innovative mechanisms used around the country to bring varied stakeholders together to assess health (or other community needs), devise strategies for improving it, and evaluate performance in
3
Mutual accountability is used in international-development circles (for example, OECD, 2009; World Bank, 2008). OECD (2009) defines it as “a process by which partners hold one another responsible for the commitments that they have voluntarily made to each other.”
OCR for page 115
For the Public’s Health: The Role of Measurement in Action and Accountability
implementing the strategies. Frameworks and measures that link interventions to outcomes can help facilitate all this.
For example, in contract accountability, funds might be given to a stakeholder by local government or by a foundation to create an anti-tobacco media campaign or by a health department to a community-based organization for provision of smoking-cessation services. In mutual accountability, an agreement might be drafted by an antismoking coalition to advocate jointly for a specific legislative strategy, such as tobacco taxes. In short, the type of accountability depends not on the category of entity but on what it is accountable for doing. Those in the contract model can be held accountable by the individuals, agencies, or organizations that hold authority—legislators, a chief executive, boards of health, public health agencies, a philanthropic organization, taxpayers, and so on. Those who entered into an agreement in the mutual accountability or compact model will be accountable to whomever they have entered into agreement with, possibly including an array of system stakeholders.
This framework for accountability works whether or not the oversight party has financial, administrative, or other control over the implementing party, but the specifics must be operationalized differently in the two settings. Regardless of setting, accountability depends on good measurement and links to the standard sets of outcome measures and measures of community health in Chapter 3, Recommendation 2. For an overview of the framework for measurement in accountability that the committee discusses in this chapter, see Figure 4-2. An important element of the framework is that health-system partners need to align and coordinate their efforts consistently to ensure the greatest impact and achieve population health goals.
Challenges for Measurement and Accountability
There are many challenges to implementing a measurement framework for accountability. For many of the determinants of health, no specific entity or body is charged with improving a given determinant and made accountable for it. For example, food deserts may be one factor contributing to poor nutritional status and obesity of some Americans (Franco et al., 2009; USDA, 2010), but the authority (or responsibility) for addressing this problem is unclear and widely distributed among various public-sector and private-sector entities (as examples of the former, local government planning and zoning policies, and tax incentives for businesses; and as an example of the latter, supermarket-chain decisions about the location of new stores).
A simple, quantifiable outcome-based measure would be ideal (for example, easy to communicate and easy to understand) for evaluating the performance of public health agencies and other stakeholders in the health
OCR for page 116
For the Public’s Health: The Role of Measurement in Action and Accountability
BOX 4-1
Evidence Base and Public Health Research
Performance measurement is especially important when evidence to elucidate the pathways from system inputs (such as resources and capacities) to system outputs and outcomes is unavailable. To affect intermediate and ultimate health outcomes (for example, decreases in obesity and in all distal outcomes for which obesity is a primary risk factor, such as diabetes and CVD morbidity and mortality), public health agencies and other stakeholders must use evidence-based population-level interventions whenever possible and translate them into metrics for accountability. Gaps in knowledge should stimulate research and evaluations (Glasgow, 2010).
Individual and population-based strategies (that is, the medical model and the ecologic model) differ considerably, as discussed in Chapter 1. For example, administering a vaccine to a patient is an individual-based intervention; ensuring optimal levels of immunization in a community or nation is a population-based strategy. In many areas, population-based strategies are not well developed and have less precise effect sizes. The nation’s public health research enterprise is producing an expanding body of evidence concerning population-level (public health) interventions, programs, and policies that are efficacious and cost-effective in reducing health risks for specific populations at risk (for example, preventive interventions documented in the Guide to Community Preventive Services, 2010).
Many of the interventions have been evaluated for their effects on individual-level health outcomes or outcomes observed in small and controlled population groups. As a result, relatively little evidence suggests what scale of implementation must be achieved to produce a sustained effect on population health at the level of an entire community, region, or state. For example, some evidence suggests the vaccination coverage that needs to be achieved to provide optimal protection against vaccine-preventable diseases, but this type of evidence is lacking for many other types of public health programs, policies, and interventions, such as those which target obesity-prevention programming and food-safety inspection.
Similarly, there is relatively little evidence to suggest how to achieve the scale and quality of implementation needed to affect health on a population
system. However, holding the agencies and organizations accountable for specific health outcomes—such as reduced rates of cardiovascular disease (CVD), diabetes, or obesity—or specific modifiable determinants of health, such as smoking prevalence in the community, is not possible for several reasons:
There is a naturally shifting baseline of diseases and other conditions (both up and down) for all health outcomes in a community, regardless of whether a public health intervention has been implemented.
OCR for page 117
For the Public’s Health: The Role of Measurement in Action and Accountability
level. Implementation research at the level of the public health delivery system is needed to produce such evidence, including how best to divide and coordinate implementation responsibilities among available public-sector and private-sector stakeholders and specifically what roles government public health agencies should play vis-à-vis other stakeholders in the health system; what levels of human, monetary, institutional, technologic, and information resources are required for successful implementation; and what complementary mix of services, programs, and activities must be available (for example, enhanced HIV screening will have minimal public health effect if access to treatment is not simultaneously ensured).
There are also large evidence gaps concerning the crosscutting public health practices that are required to facilitate decision-making and to ensure successful implementation of interventions. Those practices include community health assessment, epidemiologic investigation, community health planning, policy development, communication, workforce development, evaluation and monitoring, and quality improvement.
For the public health interventions, programs, and policies that currently are supported by strong evidence, a measurement system is needed to assess adoption, reach, and implementation fidelity at state and local levels. The system would facilitate research to address current gaps in evidence and would support accountability mechanisms. It could also allow detection of practice variation among public health agencies, communities, and states and identify outcomes (health and economic consequences) that result from practice variation, thus allowing targeted improvement in efficiency and effectiveness. Correspondingly, such a system could be used to support public reporting and benchmarking, accreditation and quality-improvement applications, performance-based contracting, and pay-for-performance applications.
There is a lack of precision with which public health workers can say they have achieved outcomes in a given community. The circumstances and programs that maximize health are many, and their relationships with one another are not always well understood. Dynamic models to which information can be added (as described in Chapter 3) can also create an evidence base from which to link salutary processes that lead to better intermediate outcomes that in turn increase the health-adjusted life expectancy of a population.
There is a lack of precision of the effect size for known interventions—even for those considered best practices—partly because of other underlying conditions in a community (the determinants of health from the individual level of genes to the broadest environmental factors) (see Box 4-1 on evidence-based research).
There may be stakeholders (such as private-sector entities) that are not part of a framework for accountability whose actions (both supportive of and detrimental to health) can substantially influence the success of interventions.
There is a lack of knowledge about effective interventions for many health challenges that are identified as priorities. This may require
OCR for page 118
For the Public’s Health: The Role of Measurement in Action and Accountability
innovation and implementation of “promising practices” whose efficacy is uncertain (see Box 4-1).
It may take many years or even decades for the results of public health interventions to materialize. Such long timelines may not meet the needs of policy-makers and the public, who need to see and use intermediate measures that demonstrate progress.
Measuring health outcomes is important and helps the system (at all levels) to know where it stands; owing to the factors listed above, however, distal health outcomes (such as death and diseases) are not useful in the context of accountability. Because of those factors, there is confusion and inconsistency regarding how to implement a framework for accountability in the nation’s health system (again, defined as the multiple partners working to improve population health). The lack of such a framework and the lack of consistency (for example, in what is measured) can confuse policymakers and the public and erode their confidence in system performance. Transparency in measuring performance and in demonstration of accountability to the public and to policy-makers is a critical underpinning of any population health effort.
The committee concludes that a framework for accountability is needed that includes
Agreement among implementing agencies, stakeholders, and those holding them accountable on specific plans of action for targeting health priorities.
Holding of implementing agencies or stakeholders accountable for execution of the agreed-on plans (strategies, interventions, policies, and processes).
Measurement of execution and outcomes of the agreed-on plans and agreement on revisions to a plan of action.
A model of accountability is needed that works both when there are areas with established (for example, evidence-based) best practices (as in the case of tobacco prevention) and when there is a less well-developed evidence base (as in the case of obesity prevention). The framework that the committee proposes applies in both situations because accountability measures assess the execution of agreed-on strategies. In settings where there are best practices based on evidence, accountability is primarily fidelity to established models of effective interventions. In settings where there are no clear best practices, accountability is based primarily on efficient and effective management of agreed-on innovative interventions (or programs or processes), including the placing of a higher premium on evaluation and modification as new information becomes available.
OCR for page 119
For the Public’s Health: The Role of Measurement in Action and Accountability
In the larger context of accountability, there are other unique challenges and issues in establishing accountability for population health, including how to align missions of diverse organizations or stakeholders in pursuit of shared population health goals at the environmental level; the need for strategic agreement established through a spectrum of mechanisms, such as law, financial and other types of incentives, and voluntary agreements; addressing challenges inherent in collective action (for example, free riders, or interested parties that benefit but do not contribute); and the presence of internal accountabilities (organization missions) and external accountabilities (contracts, and legal, financial, or social compacts or pressures) throughout the system.
Role of Measurement in Accountability
Measurement has unique and powerful roles to play in an accountability system, especially when other legal and financial drivers of accountability are weak or absent. Measurement can elucidate shared responsibilities for population outcomes and reveal the levels of effort and achievement needed to reach shared objectives. Measurement can be used in tandem with and is also a vehicle for legal mechanisms (such as contract compliance and liability) and incentive and financing mechanisms (such as pay-for-performance, eligibility, and resource allocation).
Measurement comes into play both at the beginning of the accountability process (for example, to inform a community and help it to decide where resources should be directed) and at the end after coordination and development of a strategy and its execution (for example, to measure outcomes of processes of a health department or business). Measurement provides a basis of alignment of efforts among health-system stakeholders. Efforts may be strategies required by law, or agreed on in contracts or agreements. Measuring and reporting on process indicators can help to strengthen accountability pathways. Having indicators available for those who take part in the accountability process may demonstrate the need for greater involvement of all stakeholders in the health system. Indicators could help to illustrate the lack of collective action in the current system in which many of those who can and do affect health (both favorably and adversely) are not part of a formal or organized system. These measures would make the contributions (or lack thereof) of various stakeholders observable and help to spur collective action. The committee’s next report will address the legal mechanisms that can assist in the alignment of strategies.
Figure 4-2 depicts a framework for the measurement dimension of accountability that draws on the work of the IOM Committee on Quality of Health Care in America (see Berwick, 2002; IOM, 2001). In that context, a framework was provided to demonstrate the changes needed in the US
OCR for page 120
For the Public’s Health: The Role of Measurement in Action and Accountability
FIGURE 4-2 A framework for measurement in accountability.
NOTE: Gov = Government; PH = Public Health; CQI = Continuous Quality Improvement.
* Activities and processes are influenced by agreed-on strategies (strategies agreed on by those being held accountable and those holding other parties accountable through contracts or compact agreements).
** Accountability measures assess how well the agreed-on strategies are executed and this may also be thought of as strategy execution measures.
***Stakeholder activities both influence the environment and work within it to shape outcomes.
SOURCE: Adapted from Berwick, 2002; IOM, 2001.
OCR for page 121
For the Public’s Health: The Role of Measurement in Action and Accountability
medical care delivery system, and four levels were described: Level A, the experience of patients and communities; Level B, the microsystem of care (for example, provider practices); Level C, organizations (for example, managed-care organizations); and Level D, the environment shaped by policy, payment, regulation, and accreditation.
The present committee believes that that framework holds relevance for its own examination of measurement in the context of accountability and has adapted it for its own purposes. The cycle begins after a needs assessment has been done, priorities set, and a plan agreed on. Level A in the committee’s adaptation of the framework includes persons (whose aggregated health information constitutes health-outcome measures) and neighborhoods. Level B refers to microsystems, which in the context of population health4 are programs, policies, and interventions that may be thought to refer to the points of contact or interactions among community groups, local businesses, others in the neighborhood, and their local public health agencies and allied entities. An example of microsystems is an interaction among a health department, a local medical care provider, community coalition, or local business concerning a particular health outcome. Specifically, a health department could assist a food retailer in facilitating healthful customer choices or could support a local business in developing a workplace prevention and wellness program. Often in public health, such microsystems need to align and integrate across organizations; for example, the local cancer-control program should feed into the statewide cancer-control program, which feeds into the national program. Level C consists of organizations described as actors in the public health system in the 2003 IOM report The Future of the Public’s Health in the 21st Century (IOM, 2003a) and as components of the health system. The organizations include the local public health agency, hospitals and other clinical care entities, community organizations, schools, businesses, religious congregations, and many others that perform roles that influence health outcomes. Level D refers to the environment, which includes a variety of social, physical (both naturally occurring and constructed), and economic factors and is shaped in part by social realities, large-scale policies (and political will), and economic arrangements (Syme and Ritterman, 2009). Figure 4-2 also depicts accountability pathways for all levels but focuses on Level C—the organizations that perform functions that affect health outcomes.
CONTEXT AND HISTORY
Performance measurement and reporting are not new ideas in public health; agencies have had to report on their performance to federal or
4
As is sometimes pointed out, the patient in public health practice is the community.
OCR for page 134
For the Public’s Health: The Role of Measurement in Action and Accountability
population health is to function, the committee believes that a continuing focused effort in an existing organization (such as the PHAB, but with a broader mandate) or a new accountability organization may be needed to develop accountability measures and to track and report commitments by public health agencies and other stakeholders. To help federal agencies, public health funders, and communities, the accountability organization would involve them in the development of accountability measures and reporting requirements. The accountability organization would need the capacity to understand the underlying logic model that links the actions taken by public health agencies and stakeholders with intermediate and health outcomes so that it can help to identify the critical processes, resources, and capabilities of each stakeholder that are central to the intervention strategy. Such an organization
Could assist in the measurement and reporting of performance of nongovernment public health stakeholders that are accountable for upholding mutual accountability “compacts” formed with others in improving community health outcomes.
Could validate and serve as a repository of accountability indicators and serve as a facilitator of process integrity and objectivity on behalf of funders, taxpayers, and communities.
Would need to be constituted appropriately to incorporate necessary expertise and demonstrate needed independence.
TYPES AND EXAMPLES OF NEEDED ACCOUNTABILITY MEASURES
There are various ways to measure accountability of stakeholders in the health system. For clinical care services delivered in public health department clinics, it may be most reasonable to consider measurement strategies that are used in the clinical care delivery sector. Such strategies (Healthcare Effectiveness Data and Information Set [HEDIS] measures for instance) are likely to grow in importance in the wake of major changes to the clinical care delivery system, and the clinical services provided in public health may also change as a result of a decrease, due to the ACA, in the number of uninsured people who need the immunization, family-planning, or communicable-disease services offered by clinics that public health agencies operate.
With regard to funding of public health agencies, a local system of public health accounts is needed to enable management to understand how well resources are aligned with interventions and outcomes. As noted in Chapter 1, measurement of financial resources and their effects on services and outcomes is inadequate. Public health agencies do not collect data in a
OCR for page 135
For the Public’s Health: The Role of Measurement in Action and Accountability
standardized way to link decisions on how resources are spent at the local level (in the health department) with the population health outcomes that they are designed to improve. And, as discussed in this chapter and elsewhere in this report, the linking of inputs and outcomes is not a simple or straightforward process, for a variety of reasons (see Box 4-4 for a discussion of so-called scorecards). Information is needed not merely by funding stream or categorical program; rather it is needed as a type of accounting—what resources go toward what health outcomes, and what is the effect? The committee began to discuss this topic and expects to gather additional information in preparing its third report.
For many stakeholders in the health system, developing accountability measures has been challenging for several reasons. Many in the business sector or non-health-related parts of the nonprofit sector have not always seen themselves as stakeholders in health. There may be data gaps or difficulties in gathering needed data, and evidence available to guide the selection of measures may be sparse. Such measurement systems as HEDIS may serve as a partial model, but there are considerable differences, compared with clinical settings (described in more detail earlier in this chapter), in the ability to link cause and effect in population health and in how accountability is traced both in the government public health infrastructure (which is supported by taxpayers and accountable to them and to elected officials) and among the many stakeholders in the system (where accountabilities are much less clearly defined and certainly more difficult to monitor and evaluate). However, the committee believes that the concerted efforts of national public health leaders, with support from public health systems and services research and input from communities, can move the field toward developing and implementing good performance measures that can be adopted by implementers and those holding them accountable. Criteria for selecting such measures may include face validity (meaningfulness, relevance, and understandability), feasibility (availability or collectability of data), methodologic soundness (validity and reliability), and fairness (to the stakeholders whose performance they will evaluate).
The set of performance measures used may differ, depending on a community’s identified needs and priorities, on the mix of stakeholders, and on the expectations of funders. The Ten Essential Public Health Services (see Box 4-2) may also serve as a tool for identifying measures to assess. See Table 4-1 for some examples of possible measures of performance for agreed-on strategies.
Over the last few decades, efforts to measure and report quality and performance related to health have increased. Spurred by such employer initiatives as the Leapfrog programs for reporting hospital-service quality, such government initiatives as Medicare quality measures for ambulatory care and hospitals, and health care quality organization standards of the
OCR for page 136
For the Public’s Health: The Role of Measurement in Action and Accountability
BOX 4-4
Improving the Next Generation of “Scorecards”
Several sets of indicators have been developed or have been made available to the public in the last 2 decades. Prominent examples include the State of the USA (SUSA) health measures (developed as part of a national, federally driven key-indicators effort), the Community Health Status Indicators effort (supported by HHS agencies and several nonprofit organizations), and the County Health Rankings (developed by academic researchers with foundation support) (Community Health Status Indicators, 2009; County Health Rankings, 2009; SUSA, 2010).
Some indicator sets are sometimes called scorecards, and the committee believes that it is important to address this terminology. Although these indicator sets (discussed in Chapter 2) attempt to provide snapshots of health by county or by state (in the case of SUSA) and can speak volumes about the needs and challenges in a given community, the snapshots generally are not intended to and do not inform about the performance of the public health agencies in the communities and about the resources available to them. These sets also provide only limited information about the underlying determinants of health (most commonly, data on educational attainment and income). It is the committee’s understanding that true scorecards (which are available in many fields, including health) are intended to convey information about performance (such as the quality of services provided) either for internal quality-improvement purposes or for external communication, but some of the indicator sets currently called scorecards are in fact unable to provide the type of information a true scorecard would give. The committee believes that policy-makers and the public cannot draw useful inferences about public health agency competence or capabilities from these so-called scorecards (in their most common current formats). Examples of true scorecards may be found in the clinical
National Committee on Quality Assurance, the National Quality Forum (NQF), and others, the medical care delivery system has begun to report performance and quality-improvement activities linking process to health outcomes. What the present committee recommends should not be considered in isolation from the efforts of those groups, and existing efforts should be incorporated when possible and when they are pertinent to population health. For example, the Centers for Disease Control and Prevention National Public Health Performance Standards program is designed to measure public health practices at the state and local levels and provides the tool Mobilizing for Action through Planning and Partnerships, which evaluates the capacity of local public health systems to conduct the Ten Essential Public Health Services. In the clinical care setting, the NQF uses continuous quality improvement as part of its vision, which is a facet of a measurement framework for accountability recommended by the committee. The national
OCR for page 137
For the Public’s Health: The Role of Measurement in Action and Accountability
context, but, as discussed in this chapter and elsewhere in this report, the clinical frameworks do not translate easily to public health practice.
The committee believes that it is crucial that future indicator sets described as community scorecards look not only at important distal health outcomes and determinants (largely the measures of community health discussed in Chapters 2 and 3) but also at their interrelationships with upstream underlying processes and policies, which may differ at the local, state, and national levels. Given that health outcomes are the products of a chain of proximal and distal influences, the interrelationships among health and its determinants should serve as an organizing framework for future measurement of health outcomes. Rather than presenting a “flat-file” list of health indicators, for every distal health outcome of importance, future efforts would map out the causal web of determinants that affect outcomes and the sequence of upstream activities that enable communities to alter the determinants. The systems-modeling activities described and called for in Chapter 3 could transform this kind of performance measurement by leading to a next generation of health indicators that measure performance along complex and nonlinear causal pathways and at the national, state, and local levels. It would require research and development to identify the most important health outcomes at the distal end of the pathway, the activities that are effective at each stage of influence, and the best metrics for each indicator. To accomplish that, changes in survey administration and data collection and analysis will be necessary. As discussed in Chapter 3, there may be value in individual health data from electronic health records that could be collectively analyzed with appropriate privacy and security safeguards (such as aggregation) to complement the understanding of community or population health. Without such understanding, the implementation of policies and other strategies that affect social, environmental, and behavioral determinants cannot be measured, monitored, and improved.
accreditation effort, as discussed earlier, will also be a useful tool as it moves forward in its development.
Implementation of the Measurement Framework for Accountability
Chapter 1 discusses the causes, and causes of causes, that lead to the untoward health outcomes of infant mortality and cardiovascular disease. As is true for the vast majority of conditions or illnesses, neither of those outcomes is amenable to improvements that are influenced solely by public health agencies. A variety of stakeholders are necessary to alter the micro and macro societal conditions in which infants die and people suffer heart attacks and congestive heart failure. Beyond the most macro level—the deeply embedded socioeconomic realities that characterize the nation—employers, community organizations, clinical care providers, schools, busi-
OCR for page 138
For the Public’s Health: The Role of Measurement in Action and Accountability
TABLE 4-1 Examples of Measures of Common Agreed-on Strategies
Sample Measure
Stakeholder
Number of employers who have voluntarily adopted and complied with smoke-free workplace policies
Business, nonprofits
Number of (nonchain) restaurants voluntarily posting or complying with requirements for disclosure of nutritional information
Business (retail)
School adherence to nutrition guidelines, including removal of some vending-machine products
Schools
Planning and zoning decisions consistent with local needs
Planning department
Small-business compliance with smoking bans (something intermediate to) high school graduation rates
Business
Schools, community-services agencies
Percentage of community housing that is affordable (give parameters)
Planning department, local government, developers
Percentage of community housing that is safe and livable (give parameters)
Police, planning, local government, community groups, faith-based organizations
Percentage of poor children (specify percentage of federal poverty level) who receive early-childhood interventions (from public health and other social-service agencies)
Public health agency, social services, nonprofit organizations, including advocacy groups and philanthropic organizations
Percentage of medical-insurance plans that implement health-literacy education; percentage of medical-insurance plans or medical care providers that adopt health-literacy strategies and implement steps to increase cultural competence of their staff; measures of health literacy in adolescents
Clinical care
Schools
Percentage of employers that provide wellness services to employees
Business, employers
Percentage of employers who adopt policies supportive of breastfeeding mothers (including dedicated, acceptable space and time to pump)
Business, employers
Percentage of baby-friendly (that is, breastfeeding-supportive) hospitals (specific parameters have been described elsewhere)
Clinical care
nesses, and many others can undertake strategies that address one or both of those outcomes (and many others).
For example, in a community that has unacceptable infant mortality, the local public health agency might serve as the convener of stakeholders, alerting other community organizations to the problem’s root causes by presenting evidence of associations between different types of changeable risk
OCR for page 139
For the Public’s Health: The Role of Measurement in Action and Accountability
factors and infant outcomes. In doing so, it might identify weaknesses in the local programs, services, and interventions available to prevent unplanned pregnancies and poor birth outcomes. Those convened might also engage a broader circle of participants who are in a position to influence environmental changes. Those convened could then consider, plan, and deploy an array of strategies. After reaching consensus on the top (most effective, evidence-based, and locally appropriate) strategies to be undertaken, the coalition could develop agreements with various stakeholders who would all commit to playing a concrete role in improving the outcome of concern. For example, the local public health or social services agencies would commit to better links to clinical care providers who are working with newly pregnant women to ensure that at-risk women receive case management and other essential services. The public health agency and clinical care providers in the community might develop agreements to ensure that no pregnant woman misses prenatal care services because of insurance status or difficulty in accessing a provider. Local businesses that cater to women and families could join in a mass-media and social marketing campaign on the importance of prenatal care. Schools could initiate or intensify efforts to educate adolescents about family planning and refer them to clinical services (as part of a broader effort to delay sexual activity and improve awareness and behaviors).
A broader coalition might work to alter the community environment more substantially. For example, local employers and businesses learning of the relationship of secondhand smoke to poor infant outcomes might commit to initiating or enhancing smoke-free environments to diminish exposure of pregnant women and alter the behaviors of other members of the community. Schools boards, learning of the relationship between graduation rates and infant mortality, might be persuaded to redouble efforts to increase graduation. Town planners, alerted to an association between early sexual activity and lack of recreational outlets, might agree to work to design programs and build facilities to serve adolescents. Food retailers, made more aware of the relationship between nutrition and birth outcomes, might commit to developing food and menu labeling as part of a communication effort. The process of involving those many organizations could include developing a coalition that could acquire formal nonprofit status (501(c)3) and apply for funding from relevant private and public-sector funders.
Implementing a measurement framework for accountability could include agreements and contracts (in cases in which funding is provided) and a variety of tools for communicating with the public on the status and progress of the community’s joint efforts (for example, through newsletters, news releases, and monthly, quarterly, or annual reports). Evidence-based indicators could be selected to help the community to hold accountable all stakeholders who have agreed to contribute to the initiative in some
OCR for page 140
For the Public’s Health: The Role of Measurement in Action and Accountability
manner. Indicators could include an array of process measures, such as the percentage of newly pregnant women receiving social support (the Women, Infants, and Children federal food and nutrition program and food stamps) who are referred for additional services, a measure of the level of tracking and follow-up of women who do not access needed services, indicators of clinicians’ attempts to initiate smoking cessation in pregnant women, measures of mass-media and other communication outputs, and the percentage of businesses that adopt smoke-free policies to decrease exposure of pregnant women. As the work progresses, indicators that can be used to hold stakeholders accountable will become more refined, interventions will also be fine-tuned, efforts to collect data will begin to produce results, and the public will have regularly updated information about progress in addressing one of the community’s top health needs.
The committee believes that the task of identifying performance measures can be simplified under the proposed framework because, for the most part, performance measures need to be measures of execution of strategies or of immediate outcomes of execution.
CONCLUDING OBSERVATIONS
In this chapter, the committee has outlined the three components of a measurement system for accountability that can be applied to the entire health system, from public health agencies to a vast array of stakeholders, and made a recommendation describing standard approaches and measures for implementing the framework in the context of both contract accountability and mutual or compact accountability.
Accountability requires measurements that track resources to outcomes; in general, these measures are not yet developed. Accountability is primarily for processes required by funders or agreed to by those in mutual accountability arrangements—processes over which organizations have control, rather than health outcomes for which public health is often only one of many contributors and determinants and therefore cannot be held directly accountable.
Simple measures of accountability (clear lines from inputs to outputs) based on quantified improvements in health outcomes, although desirable, are not possible, so the three-part, more complex framework of accountability measurement and continuous quality improvement presented in the report is needed. This measurement strategy can be operationalized across the spectrum of degree of certainty about best practices and so avoids “accountability paralysis” from lack of precise science. The strategy can be operationalized in both contract and compact operating environments and for all stakeholders, and the principles embodied are applicable at national,
OCR for page 141
For the Public’s Health: The Role of Measurement in Action and Accountability
state, and local levels. Priority steps to develop and implement the measurement framework should be taken.
REFERENCES
Alameda County Public Health Department. 2008. Life and Death from Unnatural Causes: Health and Social Inequity in Alameda County. Oakland, CA: Alameda County Public Health Department.
Baicker, K., D. Cutler, and Z. Song. 2010. Workplace wellness programs can generate savings. Health Affairs 29(2):304-311.
Bates, L. J., B. A. Lafrancois, and R. E. Santerre. 2010. An empirical study of the consolidation of local public health services in connecticut. Public Choice DOI 10.1007/s11127-010-9606-9.
Benjamin, G., M. Fallon, P. E. Jarris, and P. Libbey. 2006 (September). Final Recommendations for a Voluntary National Accreditation Program for State and Local Public Health Departments. Alexandria, VA: PHAB.
Berwick, D. M. 2002. A user’s manual for the IOM’s ‘quality chasm’ report. Health Affairs 21(3):80-90.
Bourgeois, F., W. W. Simmons, K. Olson, J. S. Brownstein, and K. D. Mandi. 2008. Evaluation of influenza prevention in the workplace using a personally controlled health record: Randomized controlled trial. Journal of Medical Internet Research 10(1):e5.
Bovaird, T. 2008. Emergent strategic management and planning mechanisms in complex adaptive systems: The case of the UK best value initiative. Public Management Review 10(3):319-340.
CDC (Centers for Disease Control and Prevention). 2010. Health Impact Assessment: Fact Sheet. Atlanta, GA: CDC.
Communities Count. 2008. Summary of Social and Health Indicators: Communities Count 2008: Executive summary. In Communities Count 2008: Social and health indicators across King County. Seattle, WA: Communities County. Pp. iv-xi.
Community Health Status Indicators. 2009. About the Community Health Status Indicators Project. http://www.communityhealth.hhs.gov/AboutTheProject.aspx?GeogCD=&PeerStrat=&state=&county= (January 6, 2010).
County Health Rankings. 2009. About the Community Health Status Indicators Project. http://www.countyhealthrankings.org/latest-news/how-washington-county-residents-stay-healthy (January 6, 2010).
Department of Population Health Sciences. 2008. Wisconsin County Health Rankings—2008. Madison: University of Wisconsin School of Medicine and Public Health.
EPA (Environmental Protection Agency). 2010. HUD-DOT-EPA Interagency Partnership for Sustainable Communities. http://www.epa.gov/smartgrowth/partnership/index.html (September 8, 2010).
Franco, M., A.V. Diez-Roux, J. A. Nettleton, M. Lazo, F. Brancati, B. Caballero, T. Glass, and L. V. Moore. 2009. Availability of healthy foods and dietary patterns: The multi-ethnic study of atherosclerosis. American Journal of Clinical Nutrition 89(3):897-904.
Glasgow, R. E. 2010. HMC research translation: Speculations about making it real and going to scale. American Journal of Health Behavior 34(6):833-840.
Goetzel, R. Z., and R. J. Ozminkowski. 2008. The health and cost benefits of work site health-promotion programs. Annual Review of Public Health 29:303-323.
Guide to Community Preventive Services. 2010. What Is the Community Guide? http://www.thecommunityguide.org/index.html (August 16, 2010).
The Healthy Development Measurement Tool. 2006. Case Studies: Hope SF-Redevelopment of San Francisco’s Public Housing. http://www.thehdmt.org/case_studies.php (September 5, 2010).
OCR for page 142
For the Public’s Health: The Role of Measurement in Action and Accountability
Heinen, L., and H. Darling. 2009. Addressing obesity in the workplace: The role of employers. Milbank Quarterly 87(1):101-122.
HHS (Department of Health and Human Services). 2010. Affordable Care Act: Laying the Foundation for Prevention. http://www.healthreform.gov/about/index.html (June 22, 2010).
HHS and CDC. 2008. Local Public Health Governance Performance Assessment: Model Standards. Washington, DC: HHS and CDC.
Houle, B., and M. Siegel. 2009. Smoker-free workplace policies: Developing a model of public health consequences of workplace policies barring employment to smokers. Tobacco Control 18(1):64-69.
IOM (Institute of Medicine). 1988. The Future of Public Health. Washington, DC: National Academy Press.
IOM. 1997. Improving Health in the Community: A Role for Performance Monitoring. Washington, DC: National Academy Press.
IOM. 2001. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academy Press.
IOM. 2003a. The Future of the Public’s Health in the 21st Century. Washington, DC: The National Academies Press.
IOM. 2003b. The Governmental Public Health Infrastructure. In The Future of the Public’s Health. Washington, DC: The National Academies Press. Pp. 96-177.
Keall, M., M. G. Baker, P. Howden-Chapman, M. Cunning, and D. Ormandy. 2010. Assessing housing quality and its impact on health, safety and sustainability. Journal of Epidemiology and Community Health 64(9):765-71.
Koivusalo, M. 2010. The state of health in all policies (HiAP) in the European Union: Potential and pitfalls. Journal of Epidemiology and Community Health 64:500-503.
Krieger, J., and D. L. Higgins. 2002. Housing and health: Time again for public health action. American Journal of Public Health 92(5):758-768.
Lemstra, M., and C. Neudorf. 2008. Health Disparity in Saskatoon: Analysis to Intervention. http://www.uphn.ca/doc/public/HealthDisaparitiesinSaskatoonExecutiveSummary.pdf (January 2010).
MacDonald, J. M., R. J. Strokes, D.A. Cohen, A. Kofner, and G. K. Ridgeway. 2010 The effect of light rail transit on body mass index and physical activity. American Journal of Preventive Medicine 39(2):105-112.
Michigan Department of Community Health. 2008a. Increasing Access to Healthy Foods: Michigan’s New Property Tax Incentive for Retail Food Establishments (Public Act 231 of 2008). Lansing: Michigan Department of Community Health.
Michigan Department of Community Health. 2008b. Reducing infant mortality in Michigan: Lessons from the field. Lansing: Michigan Department of Community Health.
NACCHO (National Association of County and City Health Officials). 2009a. Integrating performance improvement processes: MAPP, National Public Health Performance Standards, and accreditation. Washington, DC: NACCHO.
NACCHO. 2009b. Mobilizing for Action through Planning and Partnerships and the National Public Health Performance Standards Program: What is the relationship? Washington, DC: NACCHO.
National Center for Appropriate Technology. 2010. Smart growth resource: HUD, DOT and EPA announce interagency partnership for sustainable communities. http://www.smartgrowth.org/misc/default.asp?art=83&res=1024 (September 2, 2010).
NPLAN (National Policy and Legal Analysis Network). 2009. Model Menue Labeling Ordinance. Oakland, CA: National Policy and Legal Analysis Network.
NYC (New York City) Department of City Planning. 2010. Fresh Food Stores: Overview. http://home2.nyc.gov/html/dcp/html/fresh/index.shtml (September 20, 2010).
OCR for page 143
For the Public’s Health: The Role of Measurement in Action and Accountability
OECD (Organisation for European Economic Co-operation). 2009. Mututal Accountablity. Paris, France: OECD.
Office of Health Assessment and Epidemiology. 2010. Life Expectancy in Los Angeles County: How Long Do We Live and Why? A Cities and Communities Health Report. Los Angeles, CA: County of Los Angeles Department of Public Health.
Okie, S. 2007. The employer as health coach. New England Journal of Medicine 357(15): 1465-1469.
Ozminkowski, R. J., D. Ling, R. Z. Goetzel, J. A. Bruno, K. R. Rutter, F. Isaac, and S. Wang. 2002. Long-term impact of Johnson & Johnson’s Health & Wellness Program on health care utilization and expenditures. Journal of Occupational and Environmental Medicine 44(1):21-29.
PHAB (Public Health Accreditation Board). 2009. Proposed state standards and measures, adopted by the PHAB board of directors, July 16, 2009 for PHAB Beta Test. Alexandria, VA: PHAB.
PHAB. 2010. E-Newsletter: Public Health Accreditation Board. http://archive.constantcontact.com/fs030/1102084465533/archive/1103695445388.html (September 8, 2010).
Public Health Functions Steering Committee. 1994. The Public Health Workforce: An Agenda for the 21st Century. Full Report of the Public Health Functions Project. Washington, DC: HHS.
RWJF (Robert Wood Johnson Foundation) and W.K. Kellogg Foundation. 2006. Turning Point Performance Management Collaborative. http://www.turningpointprogram.org/ (September 2, 2010).
Seattle and King County Public Health Department. 2010. King County Community Health Indicators. http://www.kingcounty.gov/healthservices/health/data/chi.aspx (January 6, 2010).
Simon, P. A., P. Leslie, G. Run, G. Z. Jin, R. Reporter, A. Aguirre, and J. E. Fielding. 2005. Impact of restaurant hygiene grade cards on foodborne-disease hospitalizations in Los Angeles County. Journal of Environmental Health 46(7):32-36.
Simon, P., C. J. Jarosz, T. Kuo, and J. E. Fielding. 2008. Menu Labeling as a Potential Strategy for Combating the Obesity Epidemic: A Health Impact Assessment (HIA). Los Angeles, CA: Los Angeles County Department of Public Health.
Summers, C., L. Cohen, A. Havusha, F. Sliger, and T. Farley. 2009. Take Care New York 2012: A Policy for a Healthier New York City. New York, NY: New York City Department of Health and Mental Hygiene.
SUSA (State of the USA). 2010. The State of the USA. http://www.stateoftheusa.org (September 1, 2010).
Syme, S. L., and M. L. Ritterman. 2009. The importance of community development for health and well-being. Community Development Investment Review 5(3):1-13.
Teisman, G. 2008. Complexity and management of improvement programmes: An evolutionary approach. Public Management Review 10(3):341-359.
University of Wisconsin Population Health Institute. 2010. Mobilizing Action Toward Community Health (MATCH): Population health metrics, solid partnerships, and real incentives. http://uwphi.pophealth.wisc.edu/pha/match.htm (June 10, 2010).
USDA (United States Department of Agriculture). 2010. About the Food Environment Atlas. http://ers.usda.gov/FoodAtlas/about.htm (September 8, 2010).
White House Task Force on Childhood Obesity. 2010. Solving the Problem of Childhood Obesity Within a Generation. Washington, DC: Executive Office of the President of the United States.
World Bank. 2008. Mutual Accountability. New York, NY: United Nations Development Group.
OCR for page 144
For the Public’s Health: The Role of Measurement in Action and Accountability
This page intentionally left blank.
