National Academies Press: OpenBook

Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health (1997)

Chapter: APPENDIX D: ANALYSIS OF COMMENTS ON DRAFT REPORT

« Previous: APPENDIX C: POTENTIAL HEALTH OUTCOME AND RISK STATUS MEASURES
Suggested Citation:"APPENDIX D: ANALYSIS OF COMMENTS ON DRAFT REPORT." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×

Appendix D Analysis of Comments on Draft Report

In September and October of 1996 nearly 3,000 copies of the draft report, ''Assessment of Performance Measures in Public Health," were distributed for public comment both by the Department of Health and Human Services (DHHS) and the National Research Council (NRC). They were sent to a wide range of individuals and institutions, including various state government health agencies and professional associations. Recipients were invited to send comments or suggestions on the draft report, by mail, fax, or electronic mail. A total of 110 organizations and individuals supplied the panel with comments on the draft report, which are listed in the second section of this appendix.

The panel benefited greatly from the thoughtful and constructive comments on the draft report and wishes to thank each of the people who took the time to prepare comments. As can be seen from the list, the majority of the respondents were from state health agencies, representing mental health, alcohol and substance abuse, emergency medical services, family services, and preventive health. Other comments came from groups representing special populations, e.g., children and Native Americans of all ages.

Each comment was logged in and coded to enable the panel to review them efficiently. Comments ranged from brief to extensive, with many offering helpful suggestions for improving the report in various subject areas. The vast majority of the respondents praised the panel for providing a valuable framework for considering performance measures in public health, substance abuse, and mental health. Many commented on the care and thoughtfulness that was evident in the draft report, which are discussed in the next section.

Substantive issues raised by the respondents fell into six broad categories.

Suggested Citation:"APPENDIX D: ANALYSIS OF COMMENTS ON DRAFT REPORT." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×

Substantive Issues

Addition of Subpopulations in PPG Agreements. Several organizations urged the panel to take into consideration various subpopulations of interest for the measures contained in the draft report, e.g., children and adolescents, ethnic and racial minorities, and persons with multiple health conditions:

The failure to demonstrate the importance of ethnicity as variables throughout the health outcomes measures needs to be visited by the panel. I specifically request that services to Native Americans be included in the performance measures. There are no dual diagnosis measures proposed and there should be at least some process or capacity measures suggested. Measure smoking among 18–24 years of age in addition to all adults 18+. Percentage of school children who eat five or more servings of fruits and vegetables daily.

The comments convinced the panel that additional measures for children and youth were needed in a number of health areas covered in the report, as this group is at high-risk in virtually all states. As explained in the report, however, other populations of special interest to state health agencies can vary greatly across states; therefore, states should be encouraged to specify their own subpopulations of interest and focus their PPG efforts accordingly. The panel expects that specific priority populations will be a central element of performance agreements between states and DHHS.

Modifications to Draft Measures. Several organizations urged the panel to take into consideration modifications to the measures contained in the draft report. The majority of such comments asked the panel to consider making particular measures more specific; other comments asked for more standardization of measures across the health areas addressed in the report. Examples of the first type of comment included:

The outcome measures for Substance Abuse consistently refer to "alcohol and drug abuse." It is preferable to use the terminology "abuse of alcohol and other drugs."The EMS process measure "Percentage of trauma patients going to trauma centers'' needs elaboration and revision. "Trauma centers" need some definition since not every hospital that may describe itself as a trauma center meets criteria. Change the proposed measure of "percentage of children with blood lead greater than 15 micrograms per deciliter" to "the percentage of children under six years of age with blood lead of 10 micrograms per deciliter or greater."

Many of the suggestions for modifying the specific wording of measures

Suggested Citation:"APPENDIX D: ANALYSIS OF COMMENTS ON DRAFT REPORT." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×

contained in the draft report were accepted by the panel. For example, "intravenous drug" was changed to "injection drug" and "communicable" disease was changed to ''vaccine-preventable" disease. Other wording changes were made to add greater clarity or specificity to the measure descriptions and to make them consistent (whenever possible) across the health areas considered by the panel. In addition, limitations of both the measures and the cited data sources were explicitly acknowledged.

Additions to PPG Measures. Several organizations urged the panel to consider additional PPG measures. In reviewing these suggestions, the panel paid careful attention to whether a proposed measure was supported by a viable data source for state-level PPG purposes, as well as whether the measure could satisfy the panel's selection guidelines: (1) be aimed at specific objectives and results oriented; (2) be meaningful and understandable; (3) be supported by adequate data; and (4) be valid, reliable, and responsive. Unfortunately, there were more than three times as many measures suggested for which there is no data source than suggestions for which a data source was specified. Examples of measures without a data source included:

Percentage of merchants selling tobacco products to minors (under 18). An outcome should be developed to assure that primary care providers either receive supplemental training in mental health services or use standardized screening tools for assessing the mental health status of primary care patients. Percentage of adults, aged 35–44 who have never lost a permanent tooth due to dental caries or periodontal disease.

Examples of suggestions for measures with a data source in at least one state included:

Percentage of children with serious emotional disorders enrolled in school who are progressing academically and socially. Rate of survival from out of hospital cardiac arrest. Percentage of peers stating disapproval of marijuana use.

The panel accepted several of the suggestions. The report includes additional measures of outcome and risk reduction measures in several of the health areas examined by the panel, e.g., mental health, substance abuse, and STDs, HIV, and tuberculosis. The panel did not include some other well developed outcome measures either because they fell outside the scope of the panel's activities (such as dental health) or because they fell into the category of process or capacity measures, which are not offered as an all-inclusive listing but only provided as examples of many that states may want to use.

Suggested Citation:"APPENDIX D: ANALYSIS OF COMMENTS ON DRAFT REPORT." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×

Revisions to PPG Measure Classification. A few organizations expressed disagreement with the panel's classification of measures into outcome, process, and capacity:

I think the listing of outcomes on page 5 for chronic diseases includes a number of things (percentage of women receiving pap smears, for instance) which are actually processes; I would list them as such. Despite the designation of 47 measures as outcome measures, many of them are process measures. The draft report incorrectly classifies community changes as processes rather than outcomes. Most of the document appears to measure individual change. In the field of prevention we may address organizational practices, community development, and changes in attitude.

In response to these kinds of comments, the panel provides additional clarification about the definitions used; see Chapter 1.

Criticism of Draft Outcome Measures. A number of reviewers expressed concern that particular outcome measures suggested that their agency would be held accountable for health outcomes that are affected by multiple factors, many of which are outside their immediate programmatic control. In particular, a number of substance abuse and mental health agencies expressed disagreement over the panel's use of population-based measures to monitor their performance:

We are very concerned that the majority of proposed substance abuse indicators involve population-based data. By contrast, we are very supportive of those measures which are focused on treated populations. We are concerned that only three of the eight proposed Substance Abuse Outcome Measures address the outcomes of substance abuse clients. The remaining five outcome measures address issues of substance abuse within broad populations that are, for the most part, not recipients of services funded through our Administration. We recommend that the Council more specifically identify potential confounding variables in measuring outcomes and guidelines for risk adjusting for them. Otherwise, the proposed outcome measures are likely to reach false conclusions about program effectiveness. The measures chosen tend to reflect the public health perspective. They emphasize goals for the general population rather than for the seriously and persistently mentally ill.

In several cases the panel was persuaded that a measure contained in the draft report was not the most appropriate measure for PPG purposes. In some cases, suggested outcome measures were substituted for ones contained in the draft report (e.g., the EMS measure concerning cardiac arrest survival was deemed

Suggested Citation:"APPENDIX D: ANALYSIS OF COMMENTS ON DRAFT REPORT." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×

more appropriate that the one concerning central nervous system injuries); in other cases, measures were revised.

Although the panel recognizes that the traditional perspective of most administrators of substance abuse and mental health agencies is to ensure adequate and appropriate treatment for their clients—in contrast to the traditional public health perspective, which assumes responsibility for an entire at-risk population—the panel concludes that some population-based measures are appropriate for performance agreements. The panel recognizes, however, that in many of the health areas covered in this report, such measures cannot be affected, in the short run, solely by the actions taken by a given state agency. But, when combined with related process and capacity measures that are under the direct control of a state agency, such measures can provide useful insights regarding the state's progress in meeting important goals. Over the long run (5–10 years), state agencies should be able to demonstrate their impact on improving the functioning of their target populations, including those at risk of suffering from substance abuse and mental health problems.

Data Availability and Comparability Issues. Several organizations urged the panel to take into consideration various data issues. Several people observed that the measures contained in the draft report were not consistent with similar measures in Healthy People 2000 or other indicator systems (e.g., HEDIS). The panel has attempted to make the measures contained the revised report identical to those in other indicator systems whenever possible. However, there are two reasons for having measures in this report worded differently from similar measures in other indicator systems: (1) performance measures should not contain explicit numerical goals, although performance agreements between states and DHHS would be expected to contain specific targets; and (2) the measures parallel the language used in the major surveys used to support the measure, since the data for those surveys, in effect, define the measure.

In reviewing comments on data, the panel made a distinction between issues concerning data availability, data comparability, and other broad data issues including cost considerations and validity of data sources. Data availability concerns included:

The data resources listed to measure vaccination for high risk groups will not be able to measure vaccination rates for children 2–5 years, adolescents, and high-risk non-elderly adults without substantial increase in the sample size and cost. There is currently no data available on the number of children and adolescents who receive mental health services and live in noncustodial living situations. It may also be problematic for states to collect and report the data which you request if the data source falls outside the control of the State Mental Health Authority or the State Substance Abuse Authority.

Suggested Citation:"APPENDIX D: ANALYSIS OF COMMENTS ON DRAFT REPORT." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×

Targeting and tracking individuals with "mental illness" will be difficult to aggregate, and certainly will not be uniform within the state or nation. Each state has different parameters for tracking, recording mental health client data. All states will need to develop similar mechanisms to capture the information needed, otherwise the information obtained will not be useful for nationwide or statewide application or planning. The Behavioral Risk Factor Surveillance System (BRFSS) will offer limited comparability across states.

The measure descriptions now specify more completely the exact populations that can be supported by each of the listed data sources. More importantly, the report underscores the point that not every state is expected or required to adopt the potential measures. The panel's assumption is that if a state does not have the data system available for a measure, that measure would, by definition, not be part of its performance agreement with DHHS. In addition, some states may have their own systems that are better than those available in other states. In such cases, the state would be expected to use those data instead of data from the source(s) listed in the report.

Although many state administrators raised a concern about data availability for one or more of the draft outcome measures, the panel does not intend that all of the measures would be expected of every state. If the data needed to support a given measure are only available for a limited number of states, that performance measure could be used only for those states. That measure could be used to examine the progress made in a particular state, quite apart from any state-to-state comparisons.

Organizations and Individuals that Provided Comments

Advocacy, Inc.

Alabama Department of Mental Health and Mental Retardation

Alaska Department of Health and Social Services

American Social Health Association

American Public Health Association

Anishnabek Community and Family Services

Arizona Department of Economic Security

Arizona Department of Health Services, Division of Behavioral Health Services

Arkansas Department of Health

Association of State and Territorial Chronic Disease Program Directors

Association of State and Territorial Dental Directors

Association of State and Territorial Disability Prevention Programs

Association of State and Territorial Health Officials

Association of Trauma Surgeons

Suggested Citation:"APPENDIX D: ANALYSIS OF COMMENTS ON DRAFT REPORT." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×

Atlanta Project

Norma K. Bowyer

California Department of Health Services, Health and Welfare Agency

California Mental Health Planning Council

Center for Research in Ambulatory Health Care Administration

Centers for Disease Control and Prevention

Coalition for American Trauma Care

Community Family Planning Council, United Way of New York City

Community Health Care Association of New York State

Connecticut Department of Mental Health and Addiction Services

Connecticut Department of Public Health

Jean R. Cox

County of Los Angeles, Department of Health Services, Office of AIDS Programs and Policy

Shirley Datz-Johnson

Davis County Courthouse, Utah

Delaware Department of Services for Children, Youth and Their Families

Delaware Health and Social Services, Division of Alcoholism, Drug Abuse and Mental Health

Department of Health and Human Services, Office of the Secretary

Department of Health and Human Services, Office of Minority Health

East Coast Prevention Consortium

Georgia Department of Human Resources

Hawaii Department of Health, Emergency Medical Services Systems Branch

Hawaii Department of Health, Alcohol and Drug Abuse Division

Illinois Department of Mental Health and Developmental Disabilities

Illinois Department of Public Health

Indiana State Department of Health

Inter-Tribal Council of Michigan, Inc.

Iowa Department of Public Health

Cabinet for Health Services, Commonwealth of Kentucky

Legal Action Center

Samuel Lin, M.D., Ph.D.

Maryland Department of Health and Mental Hygiene, Alcohol and Drug Abuse Administration

Massachusetts Department of Public Health and Mental Health

Massachusetts Department of Public Health, Executive Office of Health and Human Services

Michigan Department of Community Health

Michigan Community Public Health Agency

Minnesota Department of Health

Minnesota Department of Human Services

Mississippi Department of Health

Suggested Citation:"APPENDIX D: ANALYSIS OF COMMENTS ON DRAFT REPORT." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×

Mississippi Department of Mental Health

Missouri Department of Health

Missouri Department of Mental Health

Morrow & Morrow

National Alliance of Sexual Assault Coalitions

National Association of County & City Health Officials

National Association of State Alcohol and Drug Abuse Directors, Inc.

National Association of State Emergency Medical Services Directors

National Association of State Mental Health Program Directors

National Center for Health Statistics

National Coalition Against Sexual Assault

Nebraska Department of Health

New Jersey Department of Health and Senior Services

New Jersey Department of Human Services, Division of Mental Health and Hospitals

New Jersey Office of Emergency Medical Services

New Mexico Department of Health

New York State Department of Health

New York State Office of Alcoholism and Substance Abuse Services

New York State Office of Mental Health

North Carolina Department of Human Resources

North Carolina Department of Environment, Health and Natural Resources

North Dakota Department of Health

Ohio Department of Health

Oklahoma Department of Mental Health and Substance Abuse Services

Pennsylvania Department of Health

Project Rehab

L. James L. Rivers

Max Schneier

Science and Epidemiology Committee

Society for Public Health Education, Inc.

State Block Grant Coordinators

State EMS Directors Association

State of South Carolina

State Rape Prevention Program Directors

Substance Abuse and Mental Health Services Administration

Tennessee Department of Health

Texas Department of Health

Texas Department of Mental Health and Mental Retardation

United South and Eastern Tribes, Inc.

University of Alabama at Birmingham, School of Medicine

State of Utah

Utah Department of Health

Suggested Citation:"APPENDIX D: ANALYSIS OF COMMENTS ON DRAFT REPORT." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×

Vermont Office of Alcohol and Drug Abuse Programs

Virginia Department of Health

Virginia Department of Mental Health, Mental Retardation and Substance Abuse Services

Virginia Mental Health Planning Council

Washington Department of Social and Health Services

Wisconsin Department of Health and Family Services

Wisconsin Department of Health and Social Services.

Suggested Citation:"APPENDIX D: ANALYSIS OF COMMENTS ON DRAFT REPORT." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×
Page 137
Suggested Citation:"APPENDIX D: ANALYSIS OF COMMENTS ON DRAFT REPORT." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×
Page 138
Suggested Citation:"APPENDIX D: ANALYSIS OF COMMENTS ON DRAFT REPORT." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×
Page 139
Suggested Citation:"APPENDIX D: ANALYSIS OF COMMENTS ON DRAFT REPORT." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×
Page 140
Suggested Citation:"APPENDIX D: ANALYSIS OF COMMENTS ON DRAFT REPORT." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×
Page 141
Suggested Citation:"APPENDIX D: ANALYSIS OF COMMENTS ON DRAFT REPORT." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×
Page 142
Suggested Citation:"APPENDIX D: ANALYSIS OF COMMENTS ON DRAFT REPORT." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×
Page 143
Suggested Citation:"APPENDIX D: ANALYSIS OF COMMENTS ON DRAFT REPORT." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×
Page 144
Suggested Citation:"APPENDIX D: ANALYSIS OF COMMENTS ON DRAFT REPORT." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×
Page 145
Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health Get This Book
×
Buy Paperback | $44.00
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The U.S. Department of Health and Human Services is carrying out an ambitious new program to assure that funds for public health programs are spent as effectively as possible. Under the new program, every state will develop a set of performance objectives to measure its progress in terms of outcomes, processes, and capacity. In the first phase of the program, states are to propose such measures to be achieved over three to five years.

This book examines the technical issues involved in the development of performance measures in 10 areas: mental health, substance abuse, HIV infection, sexually transmitted diseases, tuberculosis, immunization, chronic diseases, disability, rape, and emergency medical services. From more than 3,200 candidate measures proposed by researchers, policymakers, and public health professionals, the panel proposes more than 50 potential outcome measures. The book details the advantages and limitations of potential measures as well as the data sources that can support them. This volume will be an invaluable resource to everyone involved in public health.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!