National Academies Press: OpenBook

Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health (1997)

Chapter: 2: CURRENT DATA SOURCES THAT CAN SUPPORT PPG MEASURES

« Previous: 1: INTRODUCTION AND FRAMEWORK
Suggested Citation:"2: CURRENT DATA SOURCES THAT CAN SUPPORT PPG MEASURES." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×

2
Current Data Sources that Can Support PPG Measures

When a measure is proposed for use in PPG monitoring, appropriate data must be available to support its use. Unfortunately, few data sources are ideal for this purpose.

Although many types of data that have some applicability to monitoring the health of state populations are collected and assembled across the country, few come from concerted efforts to monitor the effects of public health interventions. In an ideal situation, data would be collected from the specific population of interest (or a representative sample); within the relevant time frame; using valid, reliable, and responsive measures. However, collecting and assembling data are expensive, and expanding data collection efforts can reduce the resources available for programs. As a result, the PPG process will often have to rely on data collected for one purpose or for generalized purposes to address another purpose, and states and the federal government must understand limitations of the applicability of the data. For example, the Centers for Disease Control and Prevention (CDC) supports the National Notifiable Diseases Surveillance System (NNDSS), which is designed to monitor, on a weekly basis, the occurrence of a set of diseases important to public health (including diphtheria, hepatitis A and B, STDs, tuberculosis). The NNDSS receives reports from all 50 states, five U.S. territories, New York City, and Washington, D.C., but it was designed primarily to identify outbreaks of specific diseases for rapid public health intervention, not to calculate precise incidence or prevalence rates across the country. Not all cases of the diseases receive medical care (which is how reporting usually originates), not all conditions are accurately diagnosed, not all diagnosed conditions are reported, and the completeness of reporting varies among participants.

Suggested Citation:"2: CURRENT DATA SOURCES THAT CAN SUPPORT PPG MEASURES." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×

NNDSS data might provide a gross measure of a state's changing rate of a reported disease, but not be appropriate (except for a few diseases) for comparing small changes in rates of incidence.

The lack of appropriate data may be the most important factor limiting effective monitoring of public health performance. States and the federal government may need to use data collected for other purposes and to rely on data that are not entirely comparable across states. Understanding the limits of such data is important if performance monitoring is to be effective.

One use of public health performance measures might be to examine and compare the effects of public health interventions among states. In that case, individual states and the federal government will want to compare the outcomes of similar (and different) interventions in different settings. For this purpose, comparable health outcomes data are needed from all states. However, states have had little incentive to standardize their data collection efforts with those of other states. A notable exception has been the development of the vital statistics system, a cooperative state-federal administrative data system that contains considerable health information. Data collection efforts at the national level (sponsored by the federal government or organizations with national and multistate agendas) are usually in a better position to collect health-related data using comparable definitions, questions, and methods across many or all states. Others, such as the Behavioral Risk Factor Surveillance System (BRFSS) and the Healthcare Cost and Utilization Project, use similar questions and definitions but differ in methods. Largely because of budget constraints, however, national data collection efforts such as the National Health Interview Survey (NHIS) usually have as their objective the provision of national population estimates of health, and they have not yet had the sample size or sample design required to make state-level estimates.

Two surveys designed to generate state-level estimates are the National Immunization Survey (NIS) and the BRFSS. The NIS is a random-digit-dial telephone survey of households with small children, using samples drawn from all 50 states, Washington, D.C., and 27 metropolitan areas. The survey yields state and regional estimates of immunization completeness for children aged 19–35 months. This federally run survey uses comparable data collection methods across all states and regions, and comparisons of rates of immunization can reasonably be made among states.

BRFSS is a state survey designed to assess the prevalence of health-related behavioral risk factors associated with the leading causes of premature death and disability. It is a random-digit-dial survey of samples that can be generalized to state populations. While the CDC provides overall support and technical oversight for the BRFSS, individual states administer the survey and have the opportunity to add their own questions. As a result, sampling design and data collection methods may vary from state to state. Consequently, BRFSS data should be used cautiously when making comparisons among states. For example, if states

Suggested Citation:"2: CURRENT DATA SOURCES THAT CAN SUPPORT PPG MEASURES." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×

have significantly different BRFSS response rates, users of the data should consider how nonresponse bias may have affected estimates of differences among the state rates being compared.

Other differences in data collection methods that affect data comparability are mode of collection (e.g., substance abuse rates ascertained from mail surveys or computer-aided interviews may be more accurate than those from telephone or in-person interviews) and use of proxy respondents (e.g., rates of breast self-examination may be more accurate if proxy responses are not allowed). Lack of complete comparability does not preclude using different data sources when making comparisons among states or populations, but the limits to comparability need to be considered when drawing conclusions about observed differences. Data comparability is also an issue, of course, when examining changes over time within a state. The effects of changes as well as state-to-state differences in data collection and analysis methods should always be of concern to data users.

In spite of its limitations for making comparisons among states, BRFSS (as well as a variety of other state-operated population surveys) may be a more convenient model than other federally directed surveys for assessing a state's progress toward meeting some PPG performance goals. States have considerable flexibility to add their own questions to this ongoing survey, and new questions do not require the same level of review required by law and regulation of many national surveys. In addition, because the sampling procedures and survey mechanism are established and ongoing, adding questions is relatively inexpensive.

Some data sources provide state-level data for some, but not all, states. For example, with CDC support, 49 states and Washington, D.C., have or are planning statewide tumor registries that capture incidence rates for most cancers. Other national efforts to collect health-related data from all states are often incomplete because they rely on voluntary state submission of the data. In these cases, such as with the National Facility Register of Substance Abuse Treatment and Prevention Programs, state data are often effectively not available because of long lag times in their submission.

Some regional, national, and other population-specific sources may be useful in PPG monitoring even if they do not provide state-level data. If a state adopts a PPG goal of increasing influenza vaccination rates among the elderly residents of a major metropolitan area, for example, it would be necessary to have data to measure progress toward that goal. The NHIS includes samples of specific large metropolitan areas in the United States and could be a source of data for such a measure. Data from a range of metropolitan areas surveyed through the NHIS might help to distinguish changes attributable to state interventions from changes that reflect national trends. National data that might serve as useful comparisons for monitoring changes in measures of the health status of state populations include the NHIS, the Medical Expenditure Panel Survey, Monitoring the Future, the National Hospital Discharge Survey, and the Healthcare Cost and Utilization Project. These national-level data may also be useful in distinguishing changes in

Suggested Citation:"2: CURRENT DATA SOURCES THAT CAN SUPPORT PPG MEASURES." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×

rates specific to a state from those resulting from more general changes in the national environment. However, users need to consider comparability issues for these sources, too. Data may be collected at state or substate levels with different methods than those used to collect the national-level data. As noted above, for example, because each state's BRFSS survey is conducted independently and because response rates vary by state, BRFSS data cannot confidently be aggregated across all 50 states to obtain national-level estimates.

Another source of data that may be of potential use in PPG monitoring is the Drug Abuse Warning Network (DAWN). Although DAWN does not provide state-level rates, it does provide estimates of the number of drug-related visits to hospital emergency departments in 21 metropolitan areas of the country and of the number of drug-related deaths in 40 metropolitan areas. If a state targets reduction in emergency room admissions due to alcohol and other drug abuse as one of its performance objectives, monitoring that rate within a major metropolitan area may be an appropriate and practical measure of performance, especially if data are not available from elsewhere in the state. The state should consider how the metropolitan population represents the population of interest and whether any confounding factors might influence the data, such as availability and use of hospitals run by the Indian Health Service or the Department of Veterans Affairs, which are not included in the DAWN surveillance system.

Other state data sources may be useful in PPG monitoring, such as the hospital discharge data system maintained in many states (e.g., the CHARS data system in the state of Washington). Some states conduct their own population surveys to assess health status and insurance coverage. Trauma registries are maintained by many states, and Medicaid claims files are available in various forms in many states. Of course, if these data are to be used for PPG purposes, comparisons across states will be valid only if the relevant data are collected comparably and cover comparable populations and the inferences are not extended beyond specific subpopulations (e.g., Medicaid patients).

Frequency of collection and turnaround time are important considerations when assessing the utility of any data source in PPG monitoring. If performance measures are designed to detect changes in 3–5 years, at least two data collection points must be in the time frame of interest, and the data must be available for analysis within a reasonable time after collection. Many potential data sources for PPG monitoring may not be useful because slow turnaround times make them inaccessible in the required time frames or because policy decisions or budget constraints delay or halt continuing data collection.

Appendix B summarizes currently available data sources for health-related data that may be useful in PPG monitoring. The table indicates whether the source provides data at the national or state level and whether it provides data for all or some states, as well as the general type of data and the frequency of collection in each case.

As partnerships between the states and federal government are established

Suggested Citation:"2: CURRENT DATA SOURCES THAT CAN SUPPORT PPG MEASURES." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×

through the PPG mechanism, interest in improving data sources for monitoring PPGs will probably increase. This state-federal collaboration (as well as state-local collaboration) offers great potential for identifying and finding methods to collect the most useful data for PPG monitoring. Identifying methods to improve data resources for PPG monitoring will be a major component of the second phase of the panel's work.

Suggested Citation:"2: CURRENT DATA SOURCES THAT CAN SUPPORT PPG MEASURES." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×
Page 13
Suggested Citation:"2: CURRENT DATA SOURCES THAT CAN SUPPORT PPG MEASURES." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×
Page 14
Suggested Citation:"2: CURRENT DATA SOURCES THAT CAN SUPPORT PPG MEASURES." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×
Page 15
Suggested Citation:"2: CURRENT DATA SOURCES THAT CAN SUPPORT PPG MEASURES." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×
Page 16
Suggested Citation:"2: CURRENT DATA SOURCES THAT CAN SUPPORT PPG MEASURES." National Research Council. 1997. Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. Washington, DC: The National Academies Press. doi: 10.17226/5806.
×
Page 17
Next: 3: POTENTIAL PPG MEASURES FOR 1997-2002 »
Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health Get This Book
×
Buy Paperback | $44.00
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The U.S. Department of Health and Human Services is carrying out an ambitious new program to assure that funds for public health programs are spent as effectively as possible. Under the new program, every state will develop a set of performance objectives to measure its progress in terms of outcomes, processes, and capacity. In the first phase of the program, states are to propose such measures to be achieved over three to five years.

This book examines the technical issues involved in the development of performance measures in 10 areas: mental health, substance abuse, HIV infection, sexually transmitted diseases, tuberculosis, immunization, chronic diseases, disability, rape, and emergency medical services. From more than 3,200 candidate measures proposed by researchers, policymakers, and public health professionals, the panel proposes more than 50 potential outcome measures. The book details the advantages and limitations of potential measures as well as the data sources that can support them. This volume will be an invaluable resource to everyone involved in public health.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!