Appendix D Analysis of Comments on Draft Report
In September and October of 1996 nearly 3,000 copies of the draft report, ''Assessment of Performance Measures in Public Health," were distributed for public comment both by the Department of Health and Human Services (DHHS) and the National Research Council (NRC). They were sent to a wide range of individuals and institutions, including various state government health agencies and professional associations. Recipients were invited to send comments or suggestions on the draft report, by mail, fax, or electronic mail. A total of 110 organizations and individuals supplied the panel with comments on the draft report, which are listed in the second section of this appendix.
The panel benefited greatly from the thoughtful and constructive comments on the draft report and wishes to thank each of the people who took the time to prepare comments. As can be seen from the list, the majority of the respondents were from state health agencies, representing mental health, alcohol and substance abuse, emergency medical services, family services, and preventive health. Other comments came from groups representing special populations, e.g., children and Native Americans of all ages.
Each comment was logged in and coded to enable the panel to review them efficiently. Comments ranged from brief to extensive, with many offering helpful suggestions for improving the report in various subject areas. The vast majority of the respondents praised the panel for providing a valuable framework for considering performance measures in public health, substance abuse, and mental health. Many commented on the care and thoughtfulness that was evident in the draft report, which are discussed in the next section.
Substantive issues raised by the respondents fell into six broad categories.
Substantive Issues
Addition of Subpopulations in PPG Agreements. Several organizations urged the panel to take into consideration various subpopulations of interest for the measures contained in the draft report, e.g., children and adolescents, ethnic and racial minorities, and persons with multiple health conditions:
The failure to demonstrate the importance of ethnicity as variables throughout the health outcomes measures needs to be visited by the panel. I specifically request that services to Native Americans be included in the performance measures. There are no dual diagnosis measures proposed and there should be at least some process or capacity measures suggested. Measure smoking among 18–24 years of age in addition to all adults 18+. Percentage of school children who eat five or more servings of fruits and vegetables daily.
The comments convinced the panel that additional measures for children and youth were needed in a number of health areas covered in the report, as this group is at high-risk in virtually all states. As explained in the report, however, other populations of special interest to state health agencies can vary greatly across states; therefore, states should be encouraged to specify their own subpopulations of interest and focus their PPG efforts accordingly. The panel expects that specific priority populations will be a central element of performance agreements between states and DHHS.
Modifications to Draft Measures. Several organizations urged the panel to take into consideration modifications to the measures contained in the draft report. The majority of such comments asked the panel to consider making particular measures more specific; other comments asked for more standardization of measures across the health areas addressed in the report. Examples of the first type of comment included:
The outcome measures for Substance Abuse consistently refer to "alcohol and drug abuse." It is preferable to use the terminology "abuse of alcohol and other drugs."The EMS process measure "Percentage of trauma patients going to trauma centers'' needs elaboration and revision. "Trauma centers" need some definition since not every hospital that may describe itself as a trauma center meets criteria. Change the proposed measure of "percentage of children with blood lead greater than 15 micrograms per deciliter" to "the percentage of children under six years of age with blood lead of 10 micrograms per deciliter or greater."
Many of the suggestions for modifying the specific wording of measures
contained in the draft report were accepted by the panel. For example, "intravenous drug" was changed to "injection drug" and "communicable" disease was changed to ''vaccine-preventable" disease. Other wording changes were made to add greater clarity or specificity to the measure descriptions and to make them consistent (whenever possible) across the health areas considered by the panel. In addition, limitations of both the measures and the cited data sources were explicitly acknowledged.
Additions to PPG Measures. Several organizations urged the panel to consider additional PPG measures. In reviewing these suggestions, the panel paid careful attention to whether a proposed measure was supported by a viable data source for state-level PPG purposes, as well as whether the measure could satisfy the panel's selection guidelines: (1) be aimed at specific objectives and results oriented; (2) be meaningful and understandable; (3) be supported by adequate data; and (4) be valid, reliable, and responsive. Unfortunately, there were more than three times as many measures suggested for which there is no data source than suggestions for which a data source was specified. Examples of measures without a data source included:
Percentage of merchants selling tobacco products to minors (under 18). An outcome should be developed to assure that primary care providers either receive supplemental training in mental health services or use standardized screening tools for assessing the mental health status of primary care patients. Percentage of adults, aged 35–44 who have never lost a permanent tooth due to dental caries or periodontal disease.
Examples of suggestions for measures with a data source in at least one state included:
Percentage of children with serious emotional disorders enrolled in school who are progressing academically and socially. Rate of survival from out of hospital cardiac arrest. Percentage of peers stating disapproval of marijuana use.
The panel accepted several of the suggestions. The report includes additional measures of outcome and risk reduction measures in several of the health areas examined by the panel, e.g., mental health, substance abuse, and STDs, HIV, and tuberculosis. The panel did not include some other well developed outcome measures either because they fell outside the scope of the panel's activities (such as dental health) or because they fell into the category of process or capacity measures, which are not offered as an all-inclusive listing but only provided as examples of many that states may want to use.
Revisions to PPG Measure Classification. A few organizations expressed disagreement with the panel's classification of measures into outcome, process, and capacity:
I think the listing of outcomes on page 5 for chronic diseases includes a number of things (percentage of women receiving pap smears, for instance) which are actually processes; I would list them as such. Despite the designation of 47 measures as outcome measures, many of them are process measures. The draft report incorrectly classifies community changes as processes rather than outcomes. Most of the document appears to measure individual change. In the field of prevention we may address organizational practices, community development, and changes in attitude.
In response to these kinds of comments, the panel provides additional clarification about the definitions used; see Chapter 1.
Criticism of Draft Outcome Measures. A number of reviewers expressed concern that particular outcome measures suggested that their agency would be held accountable for health outcomes that are affected by multiple factors, many of which are outside their immediate programmatic control. In particular, a number of substance abuse and mental health agencies expressed disagreement over the panel's use of population-based measures to monitor their performance:
We are very concerned that the majority of proposed substance abuse indicators involve population-based data. By contrast, we are very supportive of those measures which are focused on treated populations. We are concerned that only three of the eight proposed Substance Abuse Outcome Measures address the outcomes of substance abuse clients. The remaining five outcome measures address issues of substance abuse within broad populations that are, for the most part, not recipients of services funded through our Administration. We recommend that the Council more specifically identify potential confounding variables in measuring outcomes and guidelines for risk adjusting for them. Otherwise, the proposed outcome measures are likely to reach false conclusions about program effectiveness. The measures chosen tend to reflect the public health perspective. They emphasize goals for the general population rather than for the seriously and persistently mentally ill.
In several cases the panel was persuaded that a measure contained in the draft report was not the most appropriate measure for PPG purposes. In some cases, suggested outcome measures were substituted for ones contained in the draft report (e.g., the EMS measure concerning cardiac arrest survival was deemed
more appropriate that the one concerning central nervous system injuries); in other cases, measures were revised.
Although the panel recognizes that the traditional perspective of most administrators of substance abuse and mental health agencies is to ensure adequate and appropriate treatment for their clients—in contrast to the traditional public health perspective, which assumes responsibility for an entire at-risk population—the panel concludes that some population-based measures are appropriate for performance agreements. The panel recognizes, however, that in many of the health areas covered in this report, such measures cannot be affected, in the short run, solely by the actions taken by a given state agency. But, when combined with related process and capacity measures that are under the direct control of a state agency, such measures can provide useful insights regarding the state's progress in meeting important goals. Over the long run (5–10 years), state agencies should be able to demonstrate their impact on improving the functioning of their target populations, including those at risk of suffering from substance abuse and mental health problems.
Data Availability and Comparability Issues. Several organizations urged the panel to take into consideration various data issues. Several people observed that the measures contained in the draft report were not consistent with similar measures in Healthy People 2000 or other indicator systems (e.g., HEDIS). The panel has attempted to make the measures contained the revised report identical to those in other indicator systems whenever possible. However, there are two reasons for having measures in this report worded differently from similar measures in other indicator systems: (1) performance measures should not contain explicit numerical goals, although performance agreements between states and DHHS would be expected to contain specific targets; and (2) the measures parallel the language used in the major surveys used to support the measure, since the data for those surveys, in effect, define the measure.
In reviewing comments on data, the panel made a distinction between issues concerning data availability, data comparability, and other broad data issues including cost considerations and validity of data sources. Data availability concerns included:
The data resources listed to measure vaccination for high risk groups will not be able to measure vaccination rates for children 2–5 years, adolescents, and high-risk non-elderly adults without substantial increase in the sample size and cost. There is currently no data available on the number of children and adolescents who receive mental health services and live in noncustodial living situations. It may also be problematic for states to collect and report the data which you request if the data source falls outside the control of the State Mental Health Authority or the State Substance Abuse Authority.
Targeting and tracking individuals with "mental illness" will be difficult to aggregate, and certainly will not be uniform within the state or nation. Each state has different parameters for tracking, recording mental health client data. All states will need to develop similar mechanisms to capture the information needed, otherwise the information obtained will not be useful for nationwide or statewide application or planning. The Behavioral Risk Factor Surveillance System (BRFSS) will offer limited comparability across states.
The measure descriptions now specify more completely the exact populations that can be supported by each of the listed data sources. More importantly, the report underscores the point that not every state is expected or required to adopt the potential measures. The panel's assumption is that if a state does not have the data system available for a measure, that measure would, by definition, not be part of its performance agreement with DHHS. In addition, some states may have their own systems that are better than those available in other states. In such cases, the state would be expected to use those data instead of data from the source(s) listed in the report.
Although many state administrators raised a concern about data availability for one or more of the draft outcome measures, the panel does not intend that all of the measures would be expected of every state. If the data needed to support a given measure are only available for a limited number of states, that performance measure could be used only for those states. That measure could be used to examine the progress made in a particular state, quite apart from any state-to-state comparisons.
Organizations and Individuals that Provided Comments
Advocacy, Inc.
Alabama Department of Mental Health and Mental Retardation
Alaska Department of Health and Social Services
American Social Health Association
American Public Health Association
Anishnabek Community and Family Services
Arizona Department of Economic Security
Arizona Department of Health Services, Division of Behavioral Health Services
Arkansas Department of Health
Association of State and Territorial Chronic Disease Program Directors
Association of State and Territorial Dental Directors
Association of State and Territorial Disability Prevention Programs
Association of State and Territorial Health Officials
Association of Trauma Surgeons
Atlanta Project
Norma K. Bowyer
California Department of Health Services, Health and Welfare Agency
California Mental Health Planning Council
Center for Research in Ambulatory Health Care Administration
Centers for Disease Control and Prevention
Coalition for American Trauma Care
Community Family Planning Council, United Way of New York City
Community Health Care Association of New York State
Connecticut Department of Mental Health and Addiction Services
Connecticut Department of Public Health
Jean R. Cox
County of Los Angeles, Department of Health Services, Office of AIDS Programs and Policy
Shirley Datz-Johnson
Davis County Courthouse, Utah
Delaware Department of Services for Children, Youth and Their Families
Delaware Health and Social Services, Division of Alcoholism, Drug Abuse and Mental Health
Department of Health and Human Services, Office of the Secretary
Department of Health and Human Services, Office of Minority Health
East Coast Prevention Consortium
Georgia Department of Human Resources
Hawaii Department of Health, Emergency Medical Services Systems Branch
Hawaii Department of Health, Alcohol and Drug Abuse Division
Illinois Department of Mental Health and Developmental Disabilities
Illinois Department of Public Health
Indiana State Department of Health
Inter-Tribal Council of Michigan, Inc.
Iowa Department of Public Health
Cabinet for Health Services, Commonwealth of Kentucky
Legal Action Center
Samuel Lin, M.D., Ph.D.
Maryland Department of Health and Mental Hygiene, Alcohol and Drug Abuse Administration
Massachusetts Department of Public Health and Mental Health
Massachusetts Department of Public Health, Executive Office of Health and Human Services
Michigan Department of Community Health
Michigan Community Public Health Agency
Minnesota Department of Health
Minnesota Department of Human Services
Mississippi Department of Health
Mississippi Department of Mental Health
Missouri Department of Health
Missouri Department of Mental Health
Morrow & Morrow
National Alliance of Sexual Assault Coalitions
National Association of County & City Health Officials
National Association of State Alcohol and Drug Abuse Directors, Inc.
National Association of State Emergency Medical Services Directors
National Association of State Mental Health Program Directors
National Center for Health Statistics
National Coalition Against Sexual Assault
Nebraska Department of Health
New Jersey Department of Health and Senior Services
New Jersey Department of Human Services, Division of Mental Health and Hospitals
New Jersey Office of Emergency Medical Services
New Mexico Department of Health
New York State Department of Health
New York State Office of Alcoholism and Substance Abuse Services
New York State Office of Mental Health
North Carolina Department of Human Resources
North Carolina Department of Environment, Health and Natural Resources
North Dakota Department of Health
Ohio Department of Health
Oklahoma Department of Mental Health and Substance Abuse Services
Pennsylvania Department of Health
Project Rehab
L. James L. Rivers
Max Schneier
Science and Epidemiology Committee
Society for Public Health Education, Inc.
State Block Grant Coordinators
State EMS Directors Association
State of South Carolina
State Rape Prevention Program Directors
Substance Abuse and Mental Health Services Administration
Tennessee Department of Health
Texas Department of Health
Texas Department of Mental Health and Mental Retardation
United South and Eastern Tribes, Inc.
University of Alabama at Birmingham, School of Medicine
State of Utah
Utah Department of Health
Vermont Office of Alcohol and Drug Abuse Programs
Virginia Department of Health
Virginia Department of Mental Health, Mental Retardation and Substance Abuse Services
Virginia Mental Health Planning Council
Washington Department of Social and Health Services
Wisconsin Department of Health and Family Services
Wisconsin Department of Health and Social Services.