2
Data Needs and Studies Planned
HISTORICAL OVERVIEW OF THE DATA NEEDS
Ron Manderscheid (National Association of County Behavioral Health & Developmental Disability Directors and Johns Hopkins University) discussed the evolution of federal definitions for adults with serious mental illness over the past 60 years. He noted that the definitional work has kept abreast of the evolution of mental health services research. Definitions have changed as various versions of the DSM of the American Psychiatric Association (APA) were developed and as understanding of functional impairment has grown. He suggested that the definitions have had a major impact on the perception of services and services research. Right now, for example, the Murphy-Johnson bill in the U.S. House of Representatives and the Cassidy-Murphy bill in the U.S. Senate use the term “adults with serious mental illness,” which is the definition developed by SAMHSA in the late 1990s. In addition to services and research applications, the definitions have policy and legal implications.
Manderscheid noted key concepts that are important in discussing estimation: prevalence, the total number of cases for a defined period of time; incidence, the number of new cases for a defined period of time; treated prevalence, the number of cases under care in specialty mental health settings; and community prevalence, the total number of cases in the community, including those under care.
From the 1950s to the 1970s, the field relied on treated prevalence and defined persons with mental illness by their diagnoses. There were no national epidemiological surveys, so prevalence rates of various diag-
noses were based on the number of patients treated. The first national epidemiological survey, launched in the early 1980s, was the Epidemiological Catchment Area (ECA)1 Project that Darrel Regier led and will be talking about later in the workshop. The ECA set new benchmarks that gave the field the ability to collect data in the community using lay interviewers. The ECA provided the capacity to produce national community prevalence estimates with adults.
During the 1980s, additional work was conducted to categorize adults with serious and persistent mental illness, which was defined by five different disorders (schizophrenia, bipolar disorder, depression, anxiety, and personality disorders), a Global Assessment of Functioning (GAF) score of 50 or less, and duration of 1 year. This was termed the “diagnosis-disability-duration” definition. Calculating prevalence in this way resulted in estimates of serious and persistent mental illness of about 2.8 percent of adults.
In the 1990s, the legislation that created SAMHSA in 1992, Public Law 102321, the Alcohol Drug Abuse Mental Health Administration Reorganization Act, required that SAMHSA develop a definition of adults with “serious mental illness” and that this definition be operationalized and applied to the Community Mental Health Services Block Grant Program. Manderscheid explained that the development of the new definition started with diagnosis and disability and eliminated duration. It included any disorder and a GAF score of 60 or less. Using this definition, he said, the prevalence of serious mental illness was about 5.8 percent of the adult population. The definition was tested with the ECA data that were collected in the early 1980s, and the definition was applied to the first iteration of the National Comorbidity Survey. The prevalence rates were consistent over time.
In the 2000s, SAMHSA used a proxy measure that was a variant of the Kessler Six Items Scale (K6), plus the World Health Organization Disability Assessment Schedule (WHODAS). When standardized on a global assessment scale score of 50 or less, this proxy measure produced a prevalence rate of about 4 percent of the adult population.
Manderscheid said that the definitions were mainly used for policy applications and then later for legal applications. He emphasized that the development and application of the definition of serious mental illness is not just for intellectual interest; it also has tremendous implications for the funding that Congress appropriates for this population.
________________
1 For details, see Regier, D.A., Myers, J.K., Kramer, M., Robins, L.N., Blazer, D.G., Hough, R.L., Eaton, W.W., and Locke, B.Z. (1984). The NIMH Epidemiologic Catchment Area Program: Historical context, major objectives, and study population characteristics. Archives of General Psychiatry, 41, 934-941.
Looking to the future, Manderscheid suggested that the era of electronic health records may change how prevalence data are collected, with increased reliance on samples constructed from these records. Having the right variables measured in the same way in electronic health records would be critical, and much developmental work would be needed to develop nationally comparable systems that could communicate with each other. He pointed out that there has been congressional interest in providing funds to the behavioral health world to adopt electronic health records.
Manderscheid also remarked that the field is in the midst of another transition from focusing on problems using deficit-based measures (i.e., diagnoses and functional impairments) to strength-based measures with concepts and measures being developed by consumers, peer programs, and researchers. One example of this is the development of measures of well-being and of dimensions of well-being—physical, mental, and social. Manderscheid concluded by noting three recent papers and reports that address these topics.2
Manderscheid’s presentation was followed by a discussion that focused primarily on points he made related to electronic health records and the capacity of this data source to estimate prevalence. Hortensia Amaro (University of Southern California) asked for Manderscheid’s thoughts on how well estimates based on electronic health records would reflect population-level estimates, considering that some populations are uninsured, underinsured, or do not use health services as frequently as others. Manderscheid replied that his assumption is that a universal system of electronic health records would be developed. A system that is not universal would indeed have inherent biases, underrepresenting immigrants who are not citizens, people who are in jail, those not currently enrolled in a health insurance program, and others. He pointed out that some of those same biases exist in the estimates based on current national surveys. For example, when people began to abandon landline telephones in favor of cell phones, the biases introduced into telephone surveys, which at the time were relying primarily on samples of landlines, had to be corrected. In his opinion, there might be a shift to new ways
________________
2 Institute of Medicine. (2015). Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press.
Manderscheid, R.W., Ryff, C.D., Freeman, E.J., McKnight-Eily, L.R., Dhingra, S., and Strine. T.W. (2010). Evolving definitions of mental illness and wellness. Preventing Chronic Disease, 7(1). Available: http://www.cdc.gov/pcd/issues/2010/jan/09_0124.htm [November 2015].
Schulte, P.A., Guerin, R.J., Schill, A.L., Bhattacharya, A., Cunningham, T.R., Pandalai, S.P., Eggerth, D., and Stephenson, C.M. (2015). Considerations for incorporating well-being in public policy for workers and workplaces. American Journal of Public Health, 105(8), e31-e44.
of collecting data, including more investment into the development of electronic health records.
Commenting further on the population considerations, Dean Kilpatrick (Medical University of South Carolina) added that, even if electronic health records were universal, the data would only reflect situations when a person decides to visit a physician because he or she has a problem. Furthermore, the nature of the health care system is such that health records reflect encounters to address problems and not a more general understanding of the patient. Kilpatrick also agreed with Amaro that people would be missing from the system if they do not have access to health care. Manderscheid noted that as part of the development of electronic health records there are parallel efforts to develop personal health records, which are electronic records that belong to individuals and centralize all of their health information. Electronic health records and personal health records could be brought together in a systematic way to develop a new system. He emphasized his agreement with Kilpatrick that electronic health records are not currently ready for this type of use but said that these ideas need to be put on the table for planning for the future.
Along the same lines, Theo Vos (University of Washington) added that another subset of the population that would be missing from electronic health records are people with unrecognized disease who have not sought care, which is particularly applicable for mental disorders. In addition, he said, in health records one would be relying on the different ways that clinicians determine diagnoses. Comparability would be lost in terms of being able to control the inclusion and exclusion criteria, as well as the case definitions. Manderscheid agreed that comparable definitions and structures are needed in electronic health records.
Nora Cate Schaeffer (University of Wisconsin) asked about informed consent issues for research using electronic health records and about how the need for covariates—which are available in population studies, but typically not in electronic health records—could be addressed. Manderscheid replied that there is a need for incorporating a systematic plan into the design of any system for electronic health records, not only obtaining permission for the possible use of the data in research, but also advance directives for the assignment of medical power of attorney and permission for sharing private information. In terms of covariates, Manderscheid said that it would also be important to decide early on the basic structure and scope of an electronic health records system, and whether it should include only service use data or additional covariates to enable research.
Robert Krueger (University of Minnesota) commented that the usability of electronic health records as a basis for prevalence data would be
affected by how clinicians on the front lines actually use diagnostic systems. For example, in mental health settings, clinicians often use “not otherwise specified” diagnoses on encounter forms: that is, they often do not use the diagnostic system in the way it was intended. This is important to keep in mind when considering the use of electronic health records for prevalence data.
In a similar vein, Graham Kalton (Westat) commented that the survey research concept of reliability is important to consider when discussing the use of any administrative records. Administrative data are often collected by a variety of people who apply definitions in different ways, which would affect the quality of estimates derived from those data.
Robert Gibbons (University of Chicago) remarked that it is very easy to dismiss the usefulness of electronic health records, but this process can change from what is now a passive process to a more active process. However, it is important to think about two distinct issues. First, the measurement process could be greatly improved and made more comparable to the data collection involved in large-scale surveys. Second, the population coverage bias inherent in electronic health records is more difficult to address.
Mark Olfson (Columbia University) agreed with the concerns about reliability of the data because of the differences among the raters who would be entering information into electronic health records systems. He suggested that one way forward may be to begin integrating the routine collection of brief self-report measures into electronic health records.
Darrel Regier (Uniformed Services University) added that one of the things the APA revision team for the DSM-5 has been trying to do with the DSM-5 cross-cutting measures and the WHODAS disability measure is to eventually include self-report measures in electronic health records. However, this inclusion can happen only if there is an electronic platform for collecting these types of cross-cutting and disability measures. An electronic platform would inform clinicians and physicians and guide them through a more rigorous diagnostic process of examining symptom profiles of patients and following those symptom profiles over time for outcome measures. He said that some of the thinking that has gone into the understanding of diagnoses can apply to how researchers and others approach electronic health records and diagnosis in clinical settings in general.
David Cella (Northwestern University) said that the conclusion seems to be that there should be some investment in the capture of standardized information in electronic health records that could be used as the basis for policy decisions and funding allocation. Clinicians may continue to use “not otherwise specified” or not provide any documentation at all,
or they may choose to assume a more active role in the development of a standardized diagnostic approach.
Vos pointed out that, as part of the Global Burden of Disease study, the researchers started analyzing large volumes of U.S. medical data from private health insurers, Medicare, and Medicaid. They found that the data on chronic, persistent illness from these records are very comparable to survey data. However, for chronic episodic or shorter duration conditions, it is more difficult to decipher whether a diagnosis, seen at one point in a record, was still present or not over the course of a year. The other issue they encountered was that medical records often do not provide information on severity, or severity may not be defined in a standard way. He said that, in his view, electronic health records are a wonderful source to work with, but they will never do away with the need to collect survey data as a complement.
A NEW NATIONAL INSTITUTE OF MENTAL HEALTH INITIATIVE
Lisa Colpe (National Institute of Mental Health) discussed a new initiative to field a nationally representative, in-person, household survey to assess mental disorders and their correlates among youth and adults: NIMH began conversations with SAMHSA last year about a possible follow-up study to the National Survey of Drug Use and Health (NSDUH). Colpe said that the two agencies have a history of collaboration and that NIMH also supported the expansion of SAMHSA’s Mental Health Surveillance Study in order to produce disorder-based estimates.
NIMH’s current plans are to collect data from an age-stratified sample of 13,500 people: one-third between 13 and 17 years old, one-third between 18 and 30 years old, and one-third over 30 years old. This strategy will allow for an oversample of people in the younger age group. Another goal is to follow people over time, so the sample will be designed to result in a sufficient number of people who are eligible for the types of follow-up studies that are planned.
Colpe said that NIMH plans to use a 5-minute household screener followed by a 65-minute personal interview that is administered by computer-assisted personal interview (CAPI) and audio computer-assisted self-interview (ACASI). This data collection method follows the procedure used in the NSDUH. For a subset of the sample, NIMH plans to administer a telephone follow-up clinical interview at 2-4 weeks post-interview for clinical validation and calibration of the disorder modules in the survey. The current plan is to use the SCID for the follow-up, along with a psychotic symptom scale.
The first step will be to conduct a field test with 1,500 respondents to test self-administered versions of scales that have generally been inter-
viewer administered in the past. Although the study has cross-sectional aspects for producing prevalence rates, it will also be used as a platform for follow-up longitudinal studies with subsets of respondents, such as those who score above a threshold on the psychotic symptom scale. Other populations may be identified either for longitudinal follow-up to track whether a disorder develops over time or to participate in a more complete evaluation. All respondents will be asked for consent to be contacted again in the future, which will offer flexibility for future studies.
The planned survey module topics include comprehensive demographics; mental disorders (including substance use and personality measures); suicidality (including past and recent suicidal behavior, access to firearms); psychotic-like experiences (including diagnostic history, family history); traumatic experiences (including childhood adversities, exposures to violence, disasters, life-threatening events), research domain criteria (RDoC) dimensional measures,3 NIMH common data elements,4 and chronic health conditions (including head injury, health behaviors).
In addition to questions about disorders, the survey will include a relatively large module covering health and mental health service use, frequency, and how effective respondents find those services. Another module pertains to lifetime as well as past year homelessness in persons with mental illness, which will allow NIMH to examine the number of persons who are chronically homeless or who are rotating in and out of homelessness. NIMH also plans to collect data on people who have served in the military and the types of exposures during their service. Colpe added that some modules will be rotated or administered to a subset of the full sample.
With regard to specific disorders, Colpe provided a list that NIMH hopes to include in the study but noted that there may be others. For adults, these include depression, mania, posttraumatic stress disorder, panic disorder, social phobia, agoraphobia, generalized anxiety disorder, eating disorders, obsessive-compulsive disorder, attention deficit hyperactivity disorder, and substance use. For children, the same list of disorders is planned with the addition of specific phobia, oppositional defiant disorder, conduct disorder, and separation anxiety disorder.
Colpe explained that the project will become part of the existing contract SAMHSA has to conduct the NSDUH study. The deliverables will be reports based on the survey data and clinical calibration; datasets for public and restricted use, which will be added to the NIMH data repositories; and the survey instrument modules and documentation as a resource for researchers.
________________
3 See https://www.nimh.nih.gov/research-priorities/rdoc/index.shtml [December 2015].
4 See https://www.phenxtoolkit.org/index.php [December 2015].
Colpe closed by laying out the project timeline:
- 2015 to 2016: consulting with the field, instrument programming, interviewer training, and field materials development;
- 2016 to 2017: carrying out the pilot test, integrating findings, and adjusting the main survey;
- 2017 to 2018: conducting the main cross-sectional survey, launching a planned clinical reappraisal study (first follow-up study by telephone), and launching the second follow-up study; and
- 2019 and subsequent years: producing the national findings report, manuscript drafts, and the public-use data file.
Olfson asked Colpe whether the NIMH team has given any thought to oversampling low income people, ethnic or racial minorities, or other groups of policy interest. Colpe replied that right now NIMH only plans to oversample by age groups. Kalton asked if Colpe could explain further the reason for the way the age stratification is designed and whether this will be the most efficient design. Colpe explained that budget limitations drove their decision to make sure they had an adequate sample for youth and “transition age” people. She said that the team acknowledges that it will take more screening of households to achieve these specifications.
Kalton also asked for clarification on whether the NIMH sample will be drawn from the NSDUH sample. Russell clarified that the NIMH study will be using retired segments from the NSDUH sample, but the study will not include households that participated in the NSDUH. He added that the sample is about 2-3 years old, and because of that some updating of the records will be necessary.
Kilpatrick asked Colpe about the purpose of the 5-minute household screener. He also asked whether they will interview all household members or a randomly selected individual within each household. Colpe replied that the purpose of the screener is to find out who is in the household, after which 0, 1, or 2 members will be selected for the interview.
Kilpatrick also asked whether the decisions to administer the full modules will be based only on the responses to the screening questions. Colpe responded that most of the disorder modules will be asked on the basis of the answers to stem symptom questions. Kilpatrick expressed concern that the choice of stem questions can have a large effect on the estimates if they are restricted to only certain symptoms and exclude people with the disorder who have other symptoms. He suggested considering a planned missing data design, which would involve administering full modules to predetermined subsets of the sample.
Regarding the screening questions for psychotic symptoms, Krueger asked whether there are plans to conduct a clinical reappraisal. Colpe
replied that the team plans to use the PQ-16, which has been identified as an NIMH common data element (see above) and will use a psychotic symptom scale for a clinical reevaluation. The team plans on dividing the responses into score bins—low, medium, and high—and they will examine a proportion of the cases in each bin in order to get a sense of what these scores mean in the general population and in light of the clinical evaluations.
Regier asked Colpe what measures will be used for personality disorders and for RDoCs. Colpe answered that they will be using some of the RDoC recommended measures. For personality disorders, a screen will be used, one of the personality inventory scales in the DSM-5, and some items from the other DSM-5 cross-cutting measures. For clinical follow-up they will use the International Classification of Diseases (ICD-10) International Personality Disorders Examination.
Stephen Blumberg (National Center for Health Statistics) asked about plans for using proxy respondents for some individuals with disorders who may be either unwilling or unable to participate in the survey. Colpe said that they do not anticipate a routine use of proxy responses, but they will track why people could not respond for themselves. In the case of the youth interviews, there will be some modules that the researchers would prefer to have the parents answer, such as those on insurance and income.
Kalton asked whether there are any concerns related to the potentially wide variation in the amount of time it will take respondents to complete the interviews. Colpe replied that they do anticipate this to vary greatly by respondent, and they have a cap to make sure it is not too long. They also planned the instrument layout so that the most important questions are at the beginning.
Regier noted that some disorders of high interest in the context of disability issues, such as schizophrenia and autism spectrum disorder, are not included in the new NIMH survey. He said that he would like to see attention given to these severe mental disorders by either clinical follow-up, subsampling, or screening, to obtain clinically valid diagnoses for these disorder areas. Regier went on to say that he has been concerned for years about the autism surveys of the Centers for Disease Control and Prevention (CDC) because the prevalence rates produced based on those data collections do not match what clinicians report. He said that the new NIMH initiative is a fantastic opportunity to take advantage of some of the more current knowledge about psychopathology and about using a more dimensional approach to psychopathology. However, he emphasized that not including the two areas of schizophrenia and autism spectrum disorder would be a missed opportunity to address the problem of not having better data on those disorders.
Colpe explained that there are a couple of ways they are exploring
psychotic symptoms. The survey will be asking whether participants have a diagnosis of autism or schizophrenia. In addition, a psychotic symptom scale will be used in the clinical evaluation to determine where a person might manifest symptoms in terms of the spectrum of psychosis. For autism, the best the survey may be able to do is to identify that group and include them in one of the follow-up clinical components. She said that the survey will also include questions about family history for those disorders. Regier pointed out that one of the problems with autism spectrum disorder, in particular, is the incentive for families to say that they have a child on the autism spectrum, because this enables them to receive a range of services that would otherwise not be available to them. He thinks that this incentive may explain in part the currently observed higher autism prevalence rates.
Olfson agreed with Regier’s concerns about the lack of data on both autism and schizophrenia; he noted that the reasons for the deficit in that area are real and very difficult to get around. In the case of schizophrenia, estimates obtained from surveys will not reflect the overall population rate unless people in various institutional settings are included, in addition to the household population. In other words, beyond the measurement challenges and the need for clinical judgment, there are also difficult sampling and statistical issues to consider.
Olfson asked Vos how credible community-based estimates of those disorders are obtained in the Global Burden of Disease study. Vos said that relying on household surveys is likely to grossly underestimate a number of mental and substance use disorders, including psychotic disorders, drug use disorders, and some of the childhood disorders like autism and Asperger’s. For those conditions Vos said that the Global Burden of Disease study tends to rely on different data collection methods, and the researchers pay special attention to selection bias issues. For the drug use disorders, they use indirect estimates that combine survey data, mental health records, needle exchange program data, and judiciary data. For psychotic disorders, they rely on studies that explicitly sample from mental health services records. Individuals with these disorders are often not included in surveys because they are either homeless or institutionalized, and they are also much more likely to be nonresponders.
Schaeffer asked whether estimates based on some other sampling frame, such as a frame based on clinic or community services records, have been considered. Colpe replied that NIMH has not considered this yet, and she said she would like to get the workshop participants’ thoughts about it. Along the same lines, Manderscheid asked whether Colpe is considering integrating a sampling frame based on the public mental health system in order to include people served by these systems. Colpe said that the National Comorbidity Survey Replication used a
school frame for adolescents. In other programs, NIMH is working with states with the 5 percent set-aside for coordinated specialty care for first-episode psychosis. If the states develop systems to allow them to store that type of data, NIMH might be able to use those programs as a frame.
Regier reminded the group that the ECA did sample the institutionalized population of nursing homes, prisons, and long-stay psychiatric hospitals. In terms of prevalence rates for schizophrenia, they found that far more people with schizophrenia are being served in primary care than are being served in the institutions, although the most severely ill may be in institutions.
Still on the subject of sampling frames, Kilpatrick raised a point about the low level of long-term inpatient mental health care that is available, which results in a disproportionate number of people with mental illness being in prisons and jails before they are incarcerated or adjudicated. He also asked whether as part of identifying primary sampling units it would be possible to identify institutions such as jails, prisons, and long-term care facilities. Colpe replied that they are able to identify homeless shelters and similar facilities and include people living in those in the sampling.
Vos remarked that integrating these approaches within a household survey may not work. If the sampling units are small, including people in prisons or long-term institutions that happen to fall into the sample will not help. He suggested that for these kinds of disorders, separate endeavors may be needed where all available sources of information are consulted in a geographic area, in order to develop a sampling frame. These sources of information could include school-based services, disability services for autism spectrum disorders, mental health services, primary care facilities, institutions for psychotic disorders, and various service providers for drug use disorders.
Kilpatrick commented that it is important to think about whether excluding a population that is not typically included in household surveys would have a meaningful impact on prevalence estimates. For example, even if the rate of a disorder is much higher among the chronically homeless than the general population, the proportion of homeless relative to the total population might be so small that not including them would not affect the prevalence estimates. Furthermore, it is also important to consider how the data will be used beyond the purposes of prevalence estimates. For example, if knowing the number of individuals with serious mental disorders is necessary to plan for services, then knowing the concentration of risk in specific places may be more important than the prevalence rate for the nation.
Regier reminded the group of some of the advantages of the ECA study, which used the five areas of Baltimore, Durham, New Haven, Los
Angeles, and St. Louis as the catchment areas. In each catchment area, the researchers identified all the prisons, nursing homes, and hospitals that served the population of interest, so they knew they were capturing the service delivery system for each of those five catchment areas. The researchers sampled the institutions with the goal of combining those data with the data from the general population survey: with this approach, the study was able to address such disorders as schizophrenia and bipolar disorder. For some surveys in the future, the catchment area approach, in addition to a national survey, might be the strategy that enables researchers to link prevalence estimates with information about service use. Regier added that a disadvantage of some national surveys is that they do not include data on incidence or short-term acute disorders. The ECA collected data on 1-month prevalence and 1-year incidents. On the basis of that information, the researchers were able to identify information about disorder onset, offset, and duration.
Connie Citro (Committee on National Statistics) noted from Colpe’s presentation that the NIMH study is going to collect information about military service and homelessness. She asked if the researchers had also considered asking questions about involvement with the criminal justice system, such as being on parole or having been incarcerated. Colpe responded that the survey will include such items as ever having been in jail and having been in jail the past year.
James Jackson (University of Michigan) pointed out that in the National Survey of American Life the researchers devised a methodology of estimating whether there were people in the household who had some attachment to the household but had a different living arrangement, such as being institutionalized or homeless, at the time of the interview. He said that those survey data are available for analysis.
Vos commented that this does not address the issue of the differential response rate across groups of people with different disorders. Typically, people with drug use disorders, psychotic symptoms, or autism, which are often associated with intellectual disability, are much more likely to be nonresponders. Even if the overall response rate for the survey is high, the majority of the people in the sample with specific disorders could be missing.
Potter mentioned that there are a couple of new data sources that could be useful in producing estimates of the institutional population. First, all nursing homes are required to submit extensive assessment data to the Centers for Medicare & Medicaid Services for all of their patients, regardless of payers. Data are submitted when a person is admitted, at 90 days, and when there is a change. The second data source is a new Medicare payment mechanism for collecting claims data for people who use inpatient psychiatric facilities.
Manderscheid added that it does not appear that the NIMH survey is designed in a way that can address the legal requirements that SAMHSA has for producing estimates of adults with serious mental illness. He urged NIMH to give some consideration to strength-based measures, which is a growing area of interest. The Healthy People 2020 initiative has made major investments in this area and will continue to do that over time.5 There are opportunities to build some synergies on these topics. Colpe responded that the new NIMH survey is not expected to replace SAMHSA’s existing procedures for estimating serious mental illness. For example, the NIMH study will produce national estimates, not the state-level estimates that are required of SAMHSA for administering the block grant programs. With respect to strength-based measures, Colpe said that her presentation did not include all of the measures planned for the survey, such as the Strength and Difficulties Questionnaire, which is planned to be used for youth. She added that there are other measures in the study that are not necessarily disorder based.
________________
5 See http://www.cdc.gov/nchs/healthy_people/hp2000.htm [December 2015].
