data to make such adjustments in those cases in which the relationship between particular variables and outcomes has empirically been established. Another problem with making comparisons of outcomes across states is that comparable data are often not available. Similarly, accurately comparing the progress made by different states in realizing their process and capacity objectives can be extremely difficult if states choose different process and capacity measures or set different levels of accomplishment (i.e., performance objectives). Consequently, using cross-state comparisons of "performance" as the analytic basis for determining financial rewards or penalties for participating agencies may be very problematic.

Consequently, the panel concludes that performance monitoring must make use of process and capacity measures to complement available measures of outcomes. Whenever process and capacity measures are used in performance agreements, the panel recommends that the relationships between them and desired health outcomes be explicitly related to professional standards, published clinical guidelines, or other references in the professional literature. Of course, process and capacity measures selected by a state for its performance agreements should possess the same statistical attributes as outcome measures: namely, they should be valid, reliable, and responsive. Although this "multimeasure" approach will not provide public officials or consumers with conclusive evidence of the effectiveness of particular interventions, it will allow interested parties to examine actions taken by agencies to realize their objectives and suggest whether changes in the magnitude or direction of their efforts should be considered.

Certain public health outcomes of interest to the public, program administrators, and elected officials cannot be measured in the short term because of inadequate empirical knowledge, incomplete data, or insufficient time to observe change. Yet such short-term considerations should not inhibit states and localities from implementing optimal long-run strategies for addressing public health concerns. For example, a long-term perspective is needed to measure changes in behavior, such as, smoking, for which an evaluation of the outcome would require a 20-year perspective. Moreover, short-term monitoring of performance associated with specific federally funded programs does not provide an appropriate basis for assessing the full set of responsibilities of state and local health, mental health, and substance abuse agencies. Clearly, the individual diseases and health conditions that the panel studied for this report are only a subset of those diseases and conditions that are of concern to public health agencies around the country.

Over the long term, the panel believes that it would be preferable to monitor the progress made by public health agencies in a more generic and less disease-specific approach. Until that is done, monitoring performance associated with federal funding of a particular program will be complicated considerably by the fact that funding support for programs in health agencies often comes from multiple sources. The federal mental health block grant, for example, represents

The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement