National Academies Press: OpenBook
« Previous: SUMMARY
Suggested Citation:"1: INTRODUCTION AND FRAMEWORK." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×

1
Introduction and Framework

Background

At the request of the U.S. Department of Health and Human Services (DHHS), the panel is assessing the state of the art for performance measures for public health programs. The goal is to recommend a set of measures for use by local, state, and federal officials to assist in evaluating progress toward public health goals. Such measures would be an integral part of Performance Partnership Grant (PPG) arrangements between a state and the federal government. Under a negotiated PPG agreement, as currently proposed, DHHS and each state would agree on a set of objectives and performance measures for individual federal-state grants; see box for definitions. These agreements will identify specific outcomes, processes, and capacity objectives for a period of 3–5 years. The PPG program will offer states increased control over funding priorities and program management of certain categorical programs in return for accepting increased accountability for results.

Performance Measure: a quantitative indicator that can be used to track progress toward an objective, i.e., to detect change over time and difference in change across programs

Objective: a specific level of measurable attainment between two points in time

Suggested Citation:"1: INTRODUCTION AND FRAMEWORK." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×

In addition to the interest of DHHS and some members of Congress in applying performance measures to public health programs, several other factors seem to account for the growing interest in performance measurement systems. Such systems promise improved documentation of the achievements of public and private agencies and also serve to identify areas needing improvement. In fact, many people in public health believe that funding cannot be expected to increase or even be maintained at current levels without better documentation of the return on program investments. Performance measurement can also complement and extend on-going public health monitoring efforts, including Healthy Cities, Healthy Communities, and Healthy People 2000 and 2010, as well as state benchmarking activities and state efforts to develop systems to monitor managed care providers. Finally, the Government Performance and Results Act (GPRA) requires the federal government to measure the performance of all federal programs. As a result, some administrators of health programs at the federal, state, and local levels are concerned that the implementation process for the GPRA may become dominated by concerns of fiscal management unless good public health measures are available to evaluate program effectiveness.

As articulated by DHHS, the PPG concept envisions that DHHS, in consultation with states, public health professionals, private organizations, public agencies, and citizens, will develop a broad menu of performance measures that can be used in one or more of the following ten areas: chronic diseases; sexually transmitted diseases (STDs), human immunodeficiency virus (HIV) infection, and tuberculosis; mental health; immunization; substance abuse; and three sub-categories of prevention of special interest to DHHS—sexual assault, disabilities, and emergency medical services. Each state-federal PPG agreement would specify performance measures as a basis for monitoring the program objectives selected for that state. DHHS has proposed that a small set of measures be designated as "core" measures of health problems or opportunities of national importance. Although these core measures would be monitored in each state, not all states would necessarily be required to address them as program priorities, since not every problem of national importance is a problem or priority in every state. A key element of the PPG concept is that progress toward the programmatic objectives is monitored through data regularly reported by the state.

To assist DHHS in developing the plan for the PPG program, the panel was charged with the following tasks: (1) to identify measurable objectives that states and other interested parties might want to achieve through PPG agreements that can be monitored at the state and national level, either now or with small modifications to existing data systems; (2) to identify measures relevant to PPG agreements that cannot be assessed but that are important to states and the federal government and therefore require further development; and (3) to recommend improvements to state and federal surveys and data systems to facilitate future collection of information for both existing and developmental measures. Task 1 is the subject of this report; tasks 2 and 3 will be the subject of a subsequent

Suggested Citation:"1: INTRODUCTION AND FRAMEWORK." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×

report. In carrying out tasks 2 and 3, the panel will pay careful attention to data adequacy and assess the need for additional data sources and program effectiveness research to supplement those that are currently available.

Candidate PPG measures that were assessed by the panel came from attendees at four regional meetings—more than 1,500 people—sponsored by DHHS during late 1995 and early 1996 and from professional associations. More than 3,200 candidate PPG measures were proposed to the panel.

Study Framework

The panel views its role as a technical one, to identify and assess measures that states can use to evaluate their progress toward important health objectives and to recommend actions to improve the utility of such measures. This report provides an assessment of measures that could be used over the next 3–5 years by states and the federal government to monitor progress in meeting agreed-upon health objectives. The report does not attempt to review all of the program options and policies to be considered in structuring PPG agreements between DHHS and states: such issues as funding levels, matching requirements, hold-harmless funding provisions, allocation of resources decisions, financial incentives, and the like are not covered. The panel's goal is to provide technically sound methods for assessing progress in meeting public health objectives and to provide states and others with practical and useful tools to advance their public health objectives.

The panel's framework for assessing potential performance measures is simple: a public health program operating at the state level, with a certain size and structure (capacity), uses the resources provided by a federal funding program (process) to improve the health of the population it serves. The panel assumes that the effectiveness of a state program in using resources can most appropriately be evaluated by assessing the degree to which desired changes in health outcomes are achieved, together with a judgment of the degree to which those changes can be attributed to a program. When a firm causal link between the resources and processes used and the health outcome sought has not been established, as is often the case, or when the program resources are a small part of all the resources that contribute to the outcome, the panel believes that performance assessment must necessarily depend on a combination of health outcome, process, and program capacity measures. Furthermore, the panel suggests that performance measures be understood and adopted as the product of an evolutionary process, to be revised as additional empirical evidence is obtained and better methods of data collection are implemented.

Suggested Citation:"1: INTRODUCTION AND FRAMEWORK." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×

Definitions

Public Health

In considering performance measures for public health programs, the panel was mindful of the concept of "core" public health functions developed by the Institute of Medicine (IOM) that is now widely accepted within the public health community. These core functions are assessment, policy development, and assurance (Institute of Medicine, 1988); see box. The IOM report also states that public health programs should include both disease prevention and health promotion, with "health" encompassing physical, mental, and environmental health.

The ten specific areas that the panel was asked by DHHS to examine with regard to performance measures are a subset of the full range of public health concerns. Many critical responsibilities of state and local public health agencies, such as maternal and child health, injury prevention, and environmental health, are not covered in this report, but the guidelines for assessing performance measures presented here can be applied to these other areas.

It is important to note that state public health departments are not always the designated recipients of federal funds. In the areas of substance abuse and mental health, for example, the grantee may well be the state department of human services. In many states, public health responsibilities are distributed among local districts. A PPG agreement in any given state, therefore, will need to clearly identify lines of responsibility and assure that the performance goals are reasonable given the organizational structure and resources available.

Public Health Assessment: "the regular and systematic collection, assembly, analysis and dissemination of information on the health of the community. This information includes statistics on health status, community health needs, and epidemiologic and other studies of health problems."

Public Health Policy Development: "the development of comprehensive public health policies by promoting use of the scientific knowledge base in decision-making about public health and by leading in developing public health policy."

Public Health Assurance: "assures that services necessary to achieve agreed upon goals are provided, either by encouraging actions by other entities (private or public sector), by requiring such actions through regulation, or by providing services directly" (Institute of Medicine, 1988:7–8)

Suggested Citation:"1: INTRODUCTION AND FRAMEWORK." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×

Health Outcome: Change (or lack of change) in the health of a defined population related to an intervention, characterized in the following ways:

health status outcome:

change (or lack of) in physical or mental status

social functioning:

change (or lack of) in the ability of an individual to function in society

consumer satisfaction:

response of an individual to services received from health provider or program

Risk Status (intermediate outcome): Change (or lack of) in the risk demonstrated or assumed to be associated with health status.

Process: What is done to, for, with, or by defined individuals or group as part of the delivery of services, such as performing a test or procedure or offering an educational service.

Capacity: The ability to provide specific services, such as clinical screening and disease surveillance, made possible by the maintenance of the basic infrastructure of the public health system, as well as by specific program resources.

Outcomes, Risk Status, Process, and Capacity

Health outcome, risk status, process, and capacity measures are all needed at different times and in different situations to intelligently monitor both changes in the health status of defined populations and the performance of all public and private agencies in working toward specified health goals; see box for definitions adopted by the panel for this report. In some cases, actual health outcomes are impractical to measure as indicators of program performance because too much time is required between intervention and outcome or because many confounding factors affect the ultimate health outcome. In such cases, the panel recommends using an "intermediate" measure, risk status, for which there is general consensus that the result being measured is related to the health outcome.1

Meaningful analysis of performance requires determining whether desired health outcomes are achieved, whether specific agency commitments are carried out, and whether the agency has the capacity to conduct all the necessary processes. Outcomes are fundamental, and any process or capacity measure used to

1  

Although many of the "risk status" outcome measures in this report might otherwise be considered "process" measures, classifying such measures as "intermediate" outcomes is more appropriate in view of the short-term nature of the proposed performance agreements.

Suggested Citation:"1: INTRODUCTION AND FRAMEWORK." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×

assess performance should be widely accepted as closely related to them. For example, if a state's PPG goal is to reduce its mortality rate from breast cancer, it can reduce the risk of such an adverse health outcome by increasing the number of mammograms it provides to women aged 50 and over. However, there are also a series of process activities (e.g., health education programs, requirements that private insurers include coverage of, say, mammography, surgical and nonsurgical treatment, and postoperative follow-up care) and capacity indicators (e.g., number of trained staff and facilities offering mammography screening) that are believed to be related to the level of mortality from breast cancer and can be monitored over time. A detailed set of such measures could provide some understanding of what particular service mechanisms are present and may affect the trend in the outcomes of interest. The capacity of public agencies is important for any comprehensive and accurate assessment of program performance. Infrastructure activities, such as the maintenance of various public health data and surveillance systems, are as important as monitoring drinking water quality and conducting restaurant inspections in promoting the public health. The panel notes, in fact, that DHHS supported a major study of public health infrastructure, which is expected to provide infrastructure capacity measures for use in the PPG process (Lewin-VHI, Inc., 1997).

Assessment Guidelines

In considering how to assess the appropriateness of individual measures for tracking the performance of state public health agencies under the PPG process, the panel reviewed materials developed by DHHS, state partners, and other professional groups (see Annotated Bibliography). The panel established guidelines for the assessment of proposed measures:

  • 1.  

    Measures should be aimed at a specific objective and be result oriented. PPG measures must clearly specify a desired public health result, including identifying the population affected and the time frame involved. Process and capacity measures should clearly specify the health outcome, or long-term objective, to which they are thought to be related.

  • 2.  

    Measures should be meaningful and understandable. Performance measures must be seen as important to both the general public and policy makers at all levels of government and they should be stated in nontechnical terms.

  • 3.  

    Data should be adequate to support the measure. Adequate data on the populations of interest must be available for the use of measures and have the following characteristics:

    • Data to track any objective must meet reasonable statistical standards for accuracy and completeness;
Suggested Citation:"1: INTRODUCTION AND FRAMEWORK." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×
  • Data to track any objective must be available in a timely fashion, at appropriate periodicity, and at reasonable cost; and
  • Data applied to a specific measure must be collected using similar methods and with a common definition throughout the population of interest. Comparisons of a measure across states are valid only if the definition and collection methodology are consistent across states.

4. Measures should be valid, reliable, and responsive. Measures should, as much as possible, capture the essence of what they purport to measure (i.e., be unbiased and valid for their intended purpose), be reproducible (i.e., reliable), and be able to detect movement toward a desired objective (i.e., be responsive).

That a measure can be valid for one purpose but not for another is an important factor in performance measurement. For example, a state's infant mortality rate is usually considered a valid measure in assessing the actual change in a state's rate of infant death from one period to another, but changes in that rate may not be a valid measure of the performance of an individual public health agency: the agency may have no control over many factors that can affect infant mortality, such as changing socioeconomic conditions or the demographic characteristics of the population. Performance measures must also be reliable: have a high likelihood of yielding the same results on repeated trials and, therefore, low levels of random error in measurement. Similarly, performance measures should be known to be responsive to change at the level of change that one would like to detect.

In an ideal world, each performance measure would fully satisfy all four guidelines; unfortunately, not many available health outcome measures can do so. For example, many factors not under a state agency's control can affect health outcomes, compromising the validity of measures of program effect. Consequently, the panel recommends that health outcome measures be used in conjunction with process and capacity measures to derive appropriately conservative inferences about the performance of a state agency. This approach will provide public officials and consumers with an opportunity to examine steps taken by agencies to achieve specific health outcomes and to better understand whether changes in the magnitude or direction of particular strategies should be considered. A combination of health outcome, process, and capacity measures should be used to identify what additional research is needed to establish more precisely the relations among program interventions and outcomes.

It is important that agencies that engage in performance monitoring specify the assumed relationship between any process or capacity measure proposed and the particular health outcome to which it is believed to be related and document, with empirical evidence and professional judgment, the assumed relationship. If states elect to implement new, experimental approaches to realize PPG objectives,

Suggested Citation:"1: INTRODUCTION AND FRAMEWORK." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×

they must collect the data necessary to document the effectiveness of those interventions.

One of the constraints of the PPG process, as currently formulated, is that the performance objectives must be judged capable of realization within 5 years. Yet many important public health objectives, such as lowered incidence of cancer and HIV infection, cannot be achieved over this short time period. However, it would be unwise to divert resources from those objectives simply because demonstrable results cannot be expected in the 5-year period. The panel recommends that DHHS and the states consider negotiating some items in their performance agreements that allow for longer term goals if relevant risk behaviors and process data can be used to measure progress toward the desired health outcomes.

Suggested Citation:"1: INTRODUCTION AND FRAMEWORK." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×
Page 5
Suggested Citation:"1: INTRODUCTION AND FRAMEWORK." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×
Page 6
Suggested Citation:"1: INTRODUCTION AND FRAMEWORK." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×
Page 7
Suggested Citation:"1: INTRODUCTION AND FRAMEWORK." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×
Page 8
Suggested Citation:"1: INTRODUCTION AND FRAMEWORK." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×
Page 9
Suggested Citation:"1: INTRODUCTION AND FRAMEWORK." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×
Page 10
Suggested Citation:"1: INTRODUCTION AND FRAMEWORK." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×
Page 11
Suggested Citation:"1: INTRODUCTION AND FRAMEWORK." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×
Page 12
Next: 2: CURRENT DATA SOURCES THAT CAN SUPPORT PPG MEASURES »
Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health Get This Book
×
Buy Paperback | $44.00
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The U.S. Department of Health and Human Services is carrying out an ambitious new program to assure that funds for public health programs are spent as effectively as possible. Under the new program, every state will develop a set of performance objectives to measure its progress in terms of outcomes, processes, and capacity. In the first phase of the program, states are to propose such measures to be achieved over three to five years.

This book examines the technical issues involved in the development of performance measures in 10 areas: mental health, substance abuse, HIV infection, sexually transmitted diseases, tuberculosis, immunization, chronic diseases, disability, rape, and emergency medical services. From more than 3,200 candidate measures proposed by researchers, policymakers, and public health professionals, the panel proposes more than 50 potential outcome measures. The book details the advantages and limitations of potential measures as well as the data sources that can support them. This volume will be an invaluable resource to everyone involved in public health.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!