Appendix D Analysis of Comments on Draft Report

In September and October of 1996 nearly 3,000 copies of the draft report, ''Assessment of Performance Measures in Public Health," were distributed for public comment both by the Department of Health and Human Services (DHHS) and the National Research Council (NRC). They were sent to a wide range of individuals and institutions, including various state government health agencies and professional associations. Recipients were invited to send comments or suggestions on the draft report, by mail, fax, or electronic mail. A total of 110 organizations and individuals supplied the panel with comments on the draft report, which are listed in the second section of this appendix.

The panel benefited greatly from the thoughtful and constructive comments on the draft report and wishes to thank each of the people who took the time to prepare comments. As can be seen from the list, the majority of the respondents were from state health agencies, representing mental health, alcohol and substance abuse, emergency medical services, family services, and preventive health. Other comments came from groups representing special populations, e.g., children and Native Americans of all ages.

Each comment was logged in and coded to enable the panel to review them efficiently. Comments ranged from brief to extensive, with many offering helpful suggestions for improving the report in various subject areas. The vast majority of the respondents praised the panel for providing a valuable framework for considering performance measures in public health, substance abuse, and mental health. Many commented on the care and thoughtfulness that was evident in the draft report, which are discussed in the next section.

Substantive issues raised by the respondents fell into six broad categories.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 137
Appendix D Analysis of Comments on Draft Report In September and October of 1996 nearly 3,000 copies of the draft report, ''Assessment of Performance Measures in Public Health," were distributed for public comment both by the Department of Health and Human Services (DHHS) and the National Research Council (NRC). They were sent to a wide range of individuals and institutions, including various state government health agencies and professional associations. Recipients were invited to send comments or suggestions on the draft report, by mail, fax, or electronic mail. A total of 110 organizations and individuals supplied the panel with comments on the draft report, which are listed in the second section of this appendix. The panel benefited greatly from the thoughtful and constructive comments on the draft report and wishes to thank each of the people who took the time to prepare comments. As can be seen from the list, the majority of the respondents were from state health agencies, representing mental health, alcohol and substance abuse, emergency medical services, family services, and preventive health. Other comments came from groups representing special populations, e.g., children and Native Americans of all ages. Each comment was logged in and coded to enable the panel to review them efficiently. Comments ranged from brief to extensive, with many offering helpful suggestions for improving the report in various subject areas. The vast majority of the respondents praised the panel for providing a valuable framework for considering performance measures in public health, substance abuse, and mental health. Many commented on the care and thoughtfulness that was evident in the draft report, which are discussed in the next section. Substantive issues raised by the respondents fell into six broad categories.

OCR for page 137
Substantive Issues Addition of Subpopulations in PPG Agreements. Several organizations urged the panel to take into consideration various subpopulations of interest for the measures contained in the draft report, e.g., children and adolescents, ethnic and racial minorities, and persons with multiple health conditions: The failure to demonstrate the importance of ethnicity as variables throughout the health outcomes measures needs to be visited by the panel. I specifically request that services to Native Americans be included in the performance measures. There are no dual diagnosis measures proposed and there should be at least some process or capacity measures suggested. Measure smoking among 18–24 years of age in addition to all adults 18+. Percentage of school children who eat five or more servings of fruits and vegetables daily. The comments convinced the panel that additional measures for children and youth were needed in a number of health areas covered in the report, as this group is at high-risk in virtually all states. As explained in the report, however, other populations of special interest to state health agencies can vary greatly across states; therefore, states should be encouraged to specify their own subpopulations of interest and focus their PPG efforts accordingly. The panel expects that specific priority populations will be a central element of performance agreements between states and DHHS. Modifications to Draft Measures. Several organizations urged the panel to take into consideration modifications to the measures contained in the draft report. The majority of such comments asked the panel to consider making particular measures more specific; other comments asked for more standardization of measures across the health areas addressed in the report. Examples of the first type of comment included: The outcome measures for Substance Abuse consistently refer to "alcohol and drug abuse." It is preferable to use the terminology "abuse of alcohol and other drugs."The EMS process measure "Percentage of trauma patients going to trauma centers'' needs elaboration and revision. "Trauma centers" need some definition since not every hospital that may describe itself as a trauma center meets criteria. Change the proposed measure of "percentage of children with blood lead greater than 15 micrograms per deciliter" to "the percentage of children under six years of age with blood lead of 10 micrograms per deciliter or greater." Many of the suggestions for modifying the specific wording of measures

OCR for page 137
contained in the draft report were accepted by the panel. For example, "intravenous drug" was changed to "injection drug" and "communicable" disease was changed to ''vaccine-preventable" disease. Other wording changes were made to add greater clarity or specificity to the measure descriptions and to make them consistent (whenever possible) across the health areas considered by the panel. In addition, limitations of both the measures and the cited data sources were explicitly acknowledged. Additions to PPG Measures. Several organizations urged the panel to consider additional PPG measures. In reviewing these suggestions, the panel paid careful attention to whether a proposed measure was supported by a viable data source for state-level PPG purposes, as well as whether the measure could satisfy the panel's selection guidelines: (1) be aimed at specific objectives and results oriented; (2) be meaningful and understandable; (3) be supported by adequate data; and (4) be valid, reliable, and responsive. Unfortunately, there were more than three times as many measures suggested for which there is no data source than suggestions for which a data source was specified. Examples of measures without a data source included: Percentage of merchants selling tobacco products to minors (under 18). An outcome should be developed to assure that primary care providers either receive supplemental training in mental health services or use standardized screening tools for assessing the mental health status of primary care patients. Percentage of adults, aged 35–44 who have never lost a permanent tooth due to dental caries or periodontal disease. Examples of suggestions for measures with a data source in at least one state included: Percentage of children with serious emotional disorders enrolled in school who are progressing academically and socially. Rate of survival from out of hospital cardiac arrest. Percentage of peers stating disapproval of marijuana use. The panel accepted several of the suggestions. The report includes additional measures of outcome and risk reduction measures in several of the health areas examined by the panel, e.g., mental health, substance abuse, and STDs, HIV, and tuberculosis. The panel did not include some other well developed outcome measures either because they fell outside the scope of the panel's activities (such as dental health) or because they fell into the category of process or capacity measures, which are not offered as an all-inclusive listing but only provided as examples of many that states may want to use.

OCR for page 137
Revisions to PPG Measure Classification. A few organizations expressed disagreement with the panel's classification of measures into outcome, process, and capacity: I think the listing of outcomes on page 5 for chronic diseases includes a number of things (percentage of women receiving pap smears, for instance) which are actually processes; I would list them as such. Despite the designation of 47 measures as outcome measures, many of them are process measures. The draft report incorrectly classifies community changes as processes rather than outcomes. Most of the document appears to measure individual change. In the field of prevention we may address organizational practices, community development, and changes in attitude. In response to these kinds of comments, the panel provides additional clarification about the definitions used; see Chapter 1. Criticism of Draft Outcome Measures. A number of reviewers expressed concern that particular outcome measures suggested that their agency would be held accountable for health outcomes that are affected by multiple factors, many of which are outside their immediate programmatic control. In particular, a number of substance abuse and mental health agencies expressed disagreement over the panel's use of population-based measures to monitor their performance: We are very concerned that the majority of proposed substance abuse indicators involve population-based data. By contrast, we are very supportive of those measures which are focused on treated populations. We are concerned that only three of the eight proposed Substance Abuse Outcome Measures address the outcomes of substance abuse clients. The remaining five outcome measures address issues of substance abuse within broad populations that are, for the most part, not recipients of services funded through our Administration. We recommend that the Council more specifically identify potential confounding variables in measuring outcomes and guidelines for risk adjusting for them. Otherwise, the proposed outcome measures are likely to reach false conclusions about program effectiveness. The measures chosen tend to reflect the public health perspective. They emphasize goals for the general population rather than for the seriously and persistently mentally ill. In several cases the panel was persuaded that a measure contained in the draft report was not the most appropriate measure for PPG purposes. In some cases, suggested outcome measures were substituted for ones contained in the draft report (e.g., the EMS measure concerning cardiac arrest survival was deemed

OCR for page 137
more appropriate that the one concerning central nervous system injuries); in other cases, measures were revised. Although the panel recognizes that the traditional perspective of most administrators of substance abuse and mental health agencies is to ensure adequate and appropriate treatment for their clients—in contrast to the traditional public health perspective, which assumes responsibility for an entire at-risk population—the panel concludes that some population-based measures are appropriate for performance agreements. The panel recognizes, however, that in many of the health areas covered in this report, such measures cannot be affected, in the short run, solely by the actions taken by a given state agency. But, when combined with related process and capacity measures that are under the direct control of a state agency, such measures can provide useful insights regarding the state's progress in meeting important goals. Over the long run (5–10 years), state agencies should be able to demonstrate their impact on improving the functioning of their target populations, including those at risk of suffering from substance abuse and mental health problems. Data Availability and Comparability Issues. Several organizations urged the panel to take into consideration various data issues. Several people observed that the measures contained in the draft report were not consistent with similar measures in Healthy People 2000 or other indicator systems (e.g., HEDIS). The panel has attempted to make the measures contained the revised report identical to those in other indicator systems whenever possible. However, there are two reasons for having measures in this report worded differently from similar measures in other indicator systems: (1) performance measures should not contain explicit numerical goals, although performance agreements between states and DHHS would be expected to contain specific targets; and (2) the measures parallel the language used in the major surveys used to support the measure, since the data for those surveys, in effect, define the measure. In reviewing comments on data, the panel made a distinction between issues concerning data availability, data comparability, and other broad data issues including cost considerations and validity of data sources. Data availability concerns included: The data resources listed to measure vaccination for high risk groups will not be able to measure vaccination rates for children 2–5 years, adolescents, and high-risk non-elderly adults without substantial increase in the sample size and cost. There is currently no data available on the number of children and adolescents who receive mental health services and live in noncustodial living situations. It may also be problematic for states to collect and report the data which you request if the data source falls outside the control of the State Mental Health Authority or the State Substance Abuse Authority.

OCR for page 137
Targeting and tracking individuals with "mental illness" will be difficult to aggregate, and certainly will not be uniform within the state or nation. Each state has different parameters for tracking, recording mental health client data. All states will need to develop similar mechanisms to capture the information needed, otherwise the information obtained will not be useful for nationwide or statewide application or planning. The Behavioral Risk Factor Surveillance System (BRFSS) will offer limited comparability across states. The measure descriptions now specify more completely the exact populations that can be supported by each of the listed data sources. More importantly, the report underscores the point that not every state is expected or required to adopt the potential measures. The panel's assumption is that if a state does not have the data system available for a measure, that measure would, by definition, not be part of its performance agreement with DHHS. In addition, some states may have their own systems that are better than those available in other states. In such cases, the state would be expected to use those data instead of data from the source(s) listed in the report. Although many state administrators raised a concern about data availability for one or more of the draft outcome measures, the panel does not intend that all of the measures would be expected of every state. If the data needed to support a given measure are only available for a limited number of states, that performance measure could be used only for those states. That measure could be used to examine the progress made in a particular state, quite apart from any state-to-state comparisons. Organizations and Individuals that Provided Comments Advocacy, Inc. Alabama Department of Mental Health and Mental Retardation Alaska Department of Health and Social Services American Social Health Association American Public Health Association Anishnabek Community and Family Services Arizona Department of Economic Security Arizona Department of Health Services, Division of Behavioral Health Services Arkansas Department of Health Association of State and Territorial Chronic Disease Program Directors Association of State and Territorial Dental Directors Association of State and Territorial Disability Prevention Programs Association of State and Territorial Health Officials Association of Trauma Surgeons

OCR for page 137
Atlanta Project Norma K. Bowyer California Department of Health Services, Health and Welfare Agency California Mental Health Planning Council Center for Research in Ambulatory Health Care Administration Centers for Disease Control and Prevention Coalition for American Trauma Care Community Family Planning Council, United Way of New York City Community Health Care Association of New York State Connecticut Department of Mental Health and Addiction Services Connecticut Department of Public Health Jean R. Cox County of Los Angeles, Department of Health Services, Office of AIDS Programs and Policy Shirley Datz-Johnson Davis County Courthouse, Utah Delaware Department of Services for Children, Youth and Their Families Delaware Health and Social Services, Division of Alcoholism, Drug Abuse and Mental Health Department of Health and Human Services, Office of the Secretary Department of Health and Human Services, Office of Minority Health East Coast Prevention Consortium Georgia Department of Human Resources Hawaii Department of Health, Emergency Medical Services Systems Branch Hawaii Department of Health, Alcohol and Drug Abuse Division Illinois Department of Mental Health and Developmental Disabilities Illinois Department of Public Health Indiana State Department of Health Inter-Tribal Council of Michigan, Inc. Iowa Department of Public Health Cabinet for Health Services, Commonwealth of Kentucky Legal Action Center Samuel Lin, M.D., Ph.D. Maryland Department of Health and Mental Hygiene, Alcohol and Drug Abuse Administration Massachusetts Department of Public Health and Mental Health Massachusetts Department of Public Health, Executive Office of Health and Human Services Michigan Department of Community Health Michigan Community Public Health Agency Minnesota Department of Health Minnesota Department of Human Services Mississippi Department of Health

OCR for page 137
Mississippi Department of Mental Health Missouri Department of Health Missouri Department of Mental Health Morrow & Morrow National Alliance of Sexual Assault Coalitions National Association of County & City Health Officials National Association of State Alcohol and Drug Abuse Directors, Inc. National Association of State Emergency Medical Services Directors National Association of State Mental Health Program Directors National Center for Health Statistics National Coalition Against Sexual Assault Nebraska Department of Health New Jersey Department of Health and Senior Services New Jersey Department of Human Services, Division of Mental Health and Hospitals New Jersey Office of Emergency Medical Services New Mexico Department of Health New York State Department of Health New York State Office of Alcoholism and Substance Abuse Services New York State Office of Mental Health North Carolina Department of Human Resources North Carolina Department of Environment, Health and Natural Resources North Dakota Department of Health Ohio Department of Health Oklahoma Department of Mental Health and Substance Abuse Services Pennsylvania Department of Health Project Rehab L. James L. Rivers Max Schneier Science and Epidemiology Committee Society for Public Health Education, Inc. State Block Grant Coordinators State EMS Directors Association State of South Carolina State Rape Prevention Program Directors Substance Abuse and Mental Health Services Administration Tennessee Department of Health Texas Department of Health Texas Department of Mental Health and Mental Retardation United South and Eastern Tribes, Inc. University of Alabama at Birmingham, School of Medicine State of Utah Utah Department of Health

OCR for page 137
Vermont Office of Alcohol and Drug Abuse Programs Virginia Department of Health Virginia Department of Mental Health, Mental Retardation and Substance Abuse Services Virginia Mental Health Planning Council Washington Department of Social and Health Services Wisconsin Department of Health and Family Services Wisconsin Department of Health and Social Services.