6
Key Themes and Possible Next Steps
In considering the measures and data collection approaches discussed, Ron Manderscheid (National Association of County Behavioral Health & Developmental Disability Directors and Johns Hopkins University) remarked that, in the end, the purpose of the data collection will have to drive the methods selected, and SAMHSA would have to decide whether the primary purpose is epidemiological research, clinical assessment, policy development, or something else. When there is legislation with funding attached, congressional intent takes priority in terms of defining the parameters for how to carry out a data collection. If, as part of a congressional hearing, the question of how many adults are in the United States with serious mental illness is raised, there has to be an answer to that question, and there have to be data that can back up that answer.
Manderscheid pointed out that there are significant opportunities for synchrony in this work that could lead to substantial progress. Federal agencies tend to work in isolation as do many researchers. The challenge is to overcome that isolation and create synchrony. Kathleen Merikangas (National Institute of Mental Health) agreed that there are a large number of similar data collections in the United States, all using different methods, and she underscored the benefits of coordination.
Manderscheid noted that some of the relevant concepts that have not been discussed as part of the workshop include resiliency, recovery, wellness, and well-being. He reiterated that these are major themes in the world of mental health and that they are also becoming more and more aligned with funding and policy initiatives. He said that it would be use-
ful if the workshop and the larger study contributed to the discussion in this area.
Darrell Regier (Uniformed Services University) said that some of the current legislation is critical of the emphasis that SAMHSA has placed on recovery, and that some argue that the agency has not paid enough attention to severe mental illness and to finding ways of helping individuals with severe mental illness into treatment. It is also important to note that the patient perspective is gathering greater influence in the United States. It is not clear that this is the case in government and areas of research such as the Global Burden of Disease, which has a focus on pathology and impairment, as opposed to on strengths. Nonetheless, it is important to begin to focus on strengths and resiliency, which will help modify the field’s understanding of the treatment or disability implications of individuals with these illnesses. Regier noted that it will be very interesting to see if, in the future, some of the national surveys will start adding measures of the concepts of recovery and resilience.
Regier also commented that as part of the development of the DSM-5 there was much debate about whether to use a resilience, strength-based approach. However, the decision was that more research and evidence of the importance of these concepts is needed. At that point it did not appear that there were sufficient data to be able to develop a scale.
Returning to the possible goal of being able to produce an estimate of prevalence of severe mental disorders in the United States, Regier suggested that a computerized adaptive testing (CAT)-type approach that combined several measures might produce a better assessment of prevalence rates than any single measure. Such an approach might include (1) disorder measures; (2) specificity measured with a scale such as the Composite International Diagnostic Interview; (3) distress measured with the K6, K8, or K10; and (4) a measure of severity. He added that if the goal is to cover 13 domains, and not just depression and anxiety, the only way to do so is by adaptive testing or a sequence similar to the one used in the DSM-5 field trials, with cross-cutting measures at Level 1 and Level 2, followed by severity measures. Going forward, it will be important for the national surveys to find ways of using the new approaches and technologies being developed.
James Jackson (University of Michigan) added that CAT-type approaches that essentially adapt tests to a particular individual appear to fit the way in which serious mental illness manifests itself, which is highly individualized; they also fit with the ideas driving developments in the area of precision medicine and individualized approaches. Robert Gibbons (University of Chicago) commented that the Educational Testing Service (ETS) has been using adaptive testing for 30 years in one form or another. However, the ETS CAT approach is based on unidimensional
item response theory, which might work well for the measurement of mathematical ability but is limited when applied to multidimensional constructs such as depression.
Fred Conrad (University of Michigan) commented, first, that he does not think that it would be wise for SAMHSA to substitute data from electronic health records for self-reported data. He said that it seemed clear based on the discussions that electronic health records are not designed for producing population estimates, and if they could be used for that purpose, it would be in the distant future. Second, he suggested, if SAMHSA fields its own survey, it would have to be a mixed mode survey, because it would be best to collect the data offering mode choices that are convenient for respondents and fit with their preferred modes of communication. This approach would help address the concern regarding the high nonresponse rates among people who suffer from mental illness. Third, Conrad noted that population coverage appears to also be a concern. There are people who are not going to be included in sampling frames using traditional methods because they are homeless or institutionalized. Although only a very small proportion of all the homeless are believed to be chronically homeless, if the chronically homeless are much more likely to suffer from mental illness than the rest of the population, then the risk of coverage error is quite high. Methods like respondent-driven sampling or other techniques focused on hard-to-reach populations may be more useful than traditional sampling. Neil Russell (SAMHSA) agreed with Conrad that electronic health records are not yet well developed enough to be considered for use of this type and that the coverage error question also deserves further attention.
Nora Cate Schaeffer (University of Wisconsin) noted that SAMHSA has the challenge of preparing for the next round of the National Survey on Drug Use and Health (NSDUH), as well as the opportunity to think about what might be possible after the next survey and further in the future. Even if electronic health records cannot be used now, such an approach might be something that is important to begin preparing for now. She said that this is a field that is in constant transition, and the technology is not very nimble, which means that introducing testing of new approaches early might be useful, where possible.
Stephen Blumberg (National Center for Health Statistics) added that although electronic health records may not work as a sampling frame or as primary data collection, they may work well as sources of secondary data. For example, it may be possible to collect the information that is needed about serious mental illness in a survey, and then, with permission, link to electronic health records to see if there is a related diagnosis in the medical record. The electronic health records could also be used to check whether people who did not meet clinical significance for a disorder in a survey might have a diagnosis in their medical records.
Schaeffer, moving to a different topic, remarked that there was little discussion of group comparisons for the different measures. One of the promises of IRT in the 1980s was the possibility of estimating different parameters for different socioeconomic groups. As the United States becomes more diverse, linguistically, culturally, and in other ways, it seems that computer-adapted testing could be really useful in making less biased comparisons across groups.
Jonaki Bose (SAMHSA) noted that one of the main questions for SAMHSA is whether it is possible to identify impairment associated with specific mental disorders. Based on the discussion, it appears that the answer is that this cannot be done through data collection. One may be able to do so in retrospective analyses of the Global Assessment of Functioning, World Health Organization Disability Assessment Scale, or data on days of disability, but it does not appear to be feasible to incorporate a more direct measurement approach into the data collection process. She said that this conclusion was useful to learn.
Bose added that both of the points that Manderscheid made early on and that Schaeffer made about looking forward to future applications of electronic health records were helpful. Administrative records inherently have a lot of problems for estimation purposes, and SAMHSA is well aware of these. But it will be important for the agency to have a voice in the potential development of a system of electronic health records for the purposes of estimating prevalence of mental illness disorders, and there is value in looking 15 years out into the future.
She also noted that the discussions have given SAMHSA cutting-edge ideas to consider that may be useful. For example, she had not thought of the possibility of applying computerized adaptive testing for specific disorder-level measurement, even though much of her work has been in the education field, where this technique is frequently used.
On the issue of coverage bias, Bose said that it is something that is discussed by the SAMHSA team all the time. The NSDUH cannot be everything for everyone, she noted. Another important take-home message from the discussions is to continue to pay close attention to what populations are not included and continue to investigate alternate data sources. She added that another reality is the need to balance priorities in funding for the present survey administration and development for the future.
Bringing the discussion back to some of SAMHSA’s original questions, Benjamin Druss (Emory University) asked the workshop participants to comment on the goals of producing state-level estimates, and, in particular, whether there are meaningful state-level differences in rates of either symptomology or functioning. Bose said that even though
SAMHSA’s estimates of serious mental illness are model based, they do see that some states have higher rates than others.
Bose also asked the participants to comment on the practice of using cutoff points for severity at the 95th percentile, and whether these are metrics that stand independent of the distribution. Manderscheid replied that, 25 years ago, they anchored the cutoff at 5.8 percent. They tested this between states using modeling and did not find significant statistical variations. States still use this today, and if there is a change in their population, it would appear in the estimate. However, it is not clear whether this is still valid today.
Druss encouraged the participants to also weigh in on the ideal frequency for a potential survey, keeping in mind that one of the parameters specified by SAMHSA was to collect data no less frequently than every 5 years. Given that SAMHAS’ flagship survey is the NSDUH, Druss asked that comments address the frequency needed for both mental illness and substance use data.
Theo Vos (University of Washington) suggested that larger changes would probably be seen over time in substance use disorders than mental health, so more frequent survey administration would be needed on substance use. He thinks every 5 years on either topic would be acceptable. For the Global Burden of Disease study, some countries provide data every 10 years, and they are happy even with that interval. He also added that some data on substance use are available through monitoring of overdose deaths. In his view, collecting data on either substance use or mental health with surveys that are based on a traditional design is challenging, due to the factors that have been discussed throughout the workshop.
