5
Key Themes and Possible Next Steps
This chapter summarizes the workshop’s two discussion sessions, which followed the morning and afternoon presentations. The workshop chair, Kathleen Merikangas, noted several themes that had emerged from the talks on measurement. One of these is that a focus on the two domains, disorder and impairment, is really critical. However, the practice of collecting data on individual disorders with specific diagnostic criteria, which has characterized epidemiological research of mental disorders in the past couple of decades, is unlikely to help advance the goals that SAMHSA has laid out. A different approach suggested by Ian Hickie was that in community studies researchers could start with asking about impairment, on the basis of a definition of impairment that is orthogonal to symptoms. Once impairment is ascertained, information could be collected about the major classes or sources of those impairments. This approach would be a major shift from the current tradition that builds symptoms into impairment and infers causality, based on the DSM framework, which is also difficult to apply to general population studies. The shift would also mean focusing on the rates of impaired children, rather than rates of disorders, such as depression, anxiety, and attention deficit hyperactivity disorder (ADHD). The question of course, is related to implementation: What is the best way to do a survey that focuses on impairment?
Another theme, Merikangas noted, is the importance of informants. From the perspective of survey design and implementation, an immediate concern is the cost associated with obtaining data from as many as three
informants (e.g., parent, child, teacher). The question is whether there are practical ways of reducing costs. There was also a lot of discussion of the instruments that could be used for measuring impairment. The emphasis was on differences between different developmental stages and how developmental norms, such as school readiness, are relevant. More work is needed to determine which instruments to use and on the optimal use of informants.
Merikangas said that the limitations of cross-sectional data were also discussed. Thinking in terms of 12-month impairment rates is very difficult. The lack of longitudinal studies also limits the understanding of the effectiveness of the medical care and services that are provided. Other countries do longitudinal studies, and it is a shame that this has not been done in the United States, on at least subsamples.
Another topic participants noted is the need to involve many different voices. Merikangas referred to the presentation by Peter Szatmari, about how stakeholders can be involved from the early stages in the development of a data collection to assure that the measures will work in a population study. Some of the groups to think about include epidemiologists, policy researchers, service providers, and even representatives of the public. This early inclusion of a wide range of stakeholders would be another big shift in thinking in the area of mental health data collections.
Jonaki Bose noted that there were two threads in the discussion. One is related to addressing SAMHSA’s immediate need to provide state-level estimates of serious emotional disturbance in children, based on a specific definition. For example, if it is possible to develop a one-time model, assume relative stability over time, and identify some predictive variables that each state is likely to have, it might serve SAMHSA’s needs well for the purposes of resource allocation. The other thread is a broader discussion of how to collect data that provide a more complete picture of child serious emotional disturbance beyond what is required by the legislation, and it might be helpful to distinguish between these two areas of discussion.
Dean Kilpatrick recalled that someone once said that mental illness was anything that was significantly disturbing to oneself or anyone else and brought a person to the attention of mental health professionals. This is a tongue-in-cheek definition, but it suggests that there are two things that are important to measure for any diagnosis. One is what might be considered distress, which could include any behaviors or problems that are disturbing to oneself. The other is what is disturbing to other people, which tends to be better captured by impairment. Impairment is indeed really important, but distress is important as well, and including it might be a good way of screening. In other words, if one is distressed, that information could be followed up with questions about why.
Susanna Visser said she wondered whether asking about impairment first would underestimate prevalence rates. She said that if children are appropriately treated, they might not show up in impairment assessments. For example, approximately 80 percent of the children with ADHD are medicated for the disorder. Furthermore, a comparison of the rates of agreement between children being treated for ADHD under Medicaid and children who meet diagnostic criteria showed that only one-third of the children who are taking medication for the disorder meet the diagnostic criteria.
In addition, Visser noted, most instruments do not provide instructions on whether to report symptoms and impairment on or off medications, and this is typically not accounted for in epidemiological studies. When she and her colleagues asked parents whether they were reporting about their children when on or off medication, half of the respondents said that they were reporting off medication or they were reporting based on a mix of what is observed on and off medication. Those who said that they were thinking primarily of the child on medication were asked whether they could report about symptoms and impairment off medication, and the researchers found that not everyone could. To some extent, ability to report off medication impairment depends on the disorder and type of medication that is used. For some externalizing disorders, the effects of taking the medication are more readily noticeable than for some of the internalizing disorders. Moreover, some parents cannot report about children’s symptoms off medication because the children are on the medication all of the time.
Nora Cate Schaeffer (University of Wisconsin–Madison) asked whether impairment that is actually caused by medication is a concern and whether there is a way to also factor this in. Glorisa Canino replied that it depends on the definition of impairment and the measure used. Most impairment instruments do not ask about impairment due to symptoms or due to medications, instead focusing on the functioning of the child. She added that in her presentation she was advocating for an impairment measure that is not tied to symptoms, following the definition of the International Classification of Functioning for Children and Youth (ICF-CY). However, if the goal is to simply produce state-level prevalence estimates, it might be possible to use a shorter scale, such as the Strengths and Difficulties Questionnaire (SDQ), even if it does not meet all of the criteria in the definition in the legislation.
Canino also pointed out that the situation in which a large percentage of children with a disorder, such as ADHD, are being treated with medication is primarily characteristic of the United States, particularly in the white population. This is not the case in other countries, or among minority populations in the United States. In other countries, perhaps
one in three or one in four children with serious emotional disturbance might be treated.
Hortensia Amaro (University of Southern California) commented that the emphasis on impairment is important from a policy perspective because it focuses on earlier interventions. This perspective could improve outcomes before children are at a stage when they have a positive diagnosis.
Ian Hickie offered a cautionary comment about the data available from administrative records, based on his experience with ADHD and depression data in Australia. Typically, the rates of treatment in these datasets are very high, with 80-90 percent of the people with a diagnosis receiving medication. It is likely that this is due to a diagnosis being assigned after a decision is made about treatment. These rates are very different from the rates that are obtained from population surveys, which Canino said are likely to show that 20-30 percent of the children with serious emotional disturbance are receiving treatment. Surveys also show that it is often children with the most severe impairment who are not getting treatment.
Another challenge pointed out by Hickie is that although many of the interventions improve function, they do not completely eliminate symptoms or impairment. In the case of ADHD, symptoms tend to greatly improve with treatment, but educational and social difficulties persist. Consequently, continuous measurement is useful, and a focus on impairment is important, he said. The symptom measure may have gone way down on the ADHD scale, but the impairment in the area of education may still be quite considerable. There are many developmental areas for which prescribing medication is not sufficient, and investments into educational and social supports are necessary in order to eliminate impairments.
Neil Russell said that in the context of SAMHSA’s goals, the point made by Visser was very important, and the issue of how treatment affects a researcher’s ability to measure prevalence should be investigated further. It would be a shame if the definition and measurement would lead to a large proportion of children being missed because they were being treated and asymptomatic or functioning better. From a policy perspective, the goal would be to distinguish between children who are impaired and children who are being treated but had been impaired prior to treatment. It is not clear how big of a methodological issue this is, but it may have a large impact on resource allocation. A Venn diagram would be particularly useful, showing how symptoms, function, and treatment map onto each other.
Merikangas said that this overlap issue is the reason that the National Comorbidity Survey Replication Adolescent Supplement (NCS-A) is
focused on obtaining information about problems over the course of a lifetime. If the focus is limited to the present, a child might be on medication, or something else in the child’s life might be affecting the immediate perception. This approach can also provide some information about the history of the problem, albeit the historical aspects of the data tend to be less reliable.
Benjamin Druss commented that several of the speakers mentioned that the prevalence rates are going to be different depending on what definitions are used. There was no discussion of whether workshop participants believe that the rates vary substantially over time, and if so, how quickly they are likely to change and why. This is one of the clinical and policy questions that needs to inform the methodological discussion, particularly about whether these data collections need to be carried out annually.
Merikangas replied that this question can be addressed separately for diagnosis and impairment. The term “surveillance” is sometimes used (based on infectious disease models) when the goal might be to monitor whether there are rapid changes. Using the same framework, sometimes a reporter calls and wants to know whether there is “an epidemic of depression.” However, there is no evidence that the rates of many of the mental health disorders change substantially over time. The ups and downs are slight, she said.
The other question is whether impairment levels change as a function of more treatment, and there are good data to indicate that this is the case. This finding means that to the extent there is an interest in disorders, it may be possible to estimate classes of disorders more broadly than has been done in surveys, such as the NCS-A, which included operationalizations of all of the diagnostic criteria, questions about symptoms, and duration. Merikangas added that Alan Zaslavsky’s research seems to indicate the same thing, which is that there are differences among the schools, but the differences are not necessarily attributable to the disorders. The impairment is absorbing much broader contextual influences. Merikangas added that she would want to move away from doing prevalence surveys in which the instruments are developed to tap specifically into the DSM and focus instead on getting an index of the disorder, severity, and impairment duration.
Michael Davern (NORC at the University of Chicago) noted that SAMHSA emphasized a need to produce prevalence estimates, and that there are some covariates that they would also like to have. He asked whether his understanding is correct and whether focusing on impairment can help address that mandate. Russell replied that SAMHSA does have a legislative mandate and a definition, but if there is a way to obtain these numbers using an approach that starts with impairment, then that
would be useful to know. Canino added that previous discussions implied that a proxy of the DMS disorders might also be acceptable.
Merikangas said that she did not interpret the legislative mandate as a requirement to produce prevalence rates of each DSM disorder. Unless the focus is shifted from disorders to children, some children will be counted more than once, for example, if they have both depression and ADHD. Collecting information about a child’s impairment before asking about the causes eliminates this problem and simplifies the process.
Furthermore, if the data about impairment and disorders are collected more efficiently, it would be possible to allocate more resources on a better understanding of regional differences or oversampling of some demographic groups in some years. This more in-depth data could lead to more informed decisions about treatment.
Bose asked for clarification on the assumption that prevalence rates might not change quickly enough to warrant annual data collections. Merikangas replied that if the question was the rate of impairment and finding out the cause of the impairment in each state, then perhaps annual surveys could be done. The question is whether it would be possible to move away from collecting comprehensive information on the full spectrum of DSM disorders. In other words, how much effort is worth investing into each of these, when research indicates that the rates remain rather stable for some of these conditions over years. Zaslavsky’s approach appears to be based on similar reasoning, because he did not use the prevalence estimates from the NCS-A: he just combined all of the disorders to determine whether there is any disorder present or not. Bose agreed that even a one-time study that uses good predictors for modeling serious emotional disturbance would be an improvement over what is available now.
Kilpatrick agreed with the views that changes should not be expected from one year to another for the mental disorders discussed. Over time, more effective screenings and treatments should make a difference, as they did for more traditional health conditions, such as high blood pressure, but the rates should not be expected to change annually. However, even if SAMHSA concludes that annual data collections are not needed, it does not mean that a large, comprehensive baseline study is not justified. There are many relevant topics in the areas of both child and adult mental health that have not been measured or have not been measured well, such as exposure to potentially traumatic events and other stressors.
Szatmari also agreed that perhaps annual cross-sectional data collections are not needed, but he argued that there is a great need for longitudinal data and that following children for 6 months or 1 year would be very useful, because although the rates do not change, the children contributing to those rates change. He also noted that discussion and agreement
are needed on what disorders need to be measured. In other words, what is needed in order to understand a child and what can be excluded. The basket could include important topics that have not been discussed, such as substance use, asthma, and epilepsy.
Hickie commented that one unintended consequence of the focus on prevalence is that once it becomes clear that these conditions do not change very much, the result is less frequent surveys because no differences are being observed. Cross-sectional impairment rates might also be stable, but the duration and the consequences of impairment could change dramatically as a result of better services. The duration of the impairment matters a lot, particularly in childhood, and, from a policy perspective, reducing that duration is very important.
Hickie said that one problem with perceptions about mental health is that diseases are not cured, but the reality is that interventions do change consequences dramatically. If the duration of impairments is shortened for children, their likelihood of success in education and their long-term success in employment increases substantially. However, one limitation associated with the impairment scales now available is that it is not known how they predict future functioning, in the crucial transition from school to higher education and employment. This further supports the need for frequent measurement, which can capture these types of issues and better reflect the benefits of services.
Graham Kalton reminded the participants that SAMHSA has a relatively straightforward mandate. While the data collection may be complicated to implement, the requirements are fairly clear. The discussions about how one could expand on the mandate are interesting and could contribute to the development of a comprehensive research agenda, but they also have cost implications, so the priority should be to consider what would satisfy the requirement to measure serious emotional disturbance and produce state-level estimates at some regular intervals. Replacing annual data collections with the longitudinal follow-ups would be more expensive, even if the number of surveys is the same, because longitudinal surveys have a distinct set of challenges.
Kalton added that one design option that may be worth considering and has not been discussed is to collect annual data on smaller samples than what is needed to produce the estimates and then average them over several years, as is done, for example, in the American Community Survey and the National Crime Victimization Survey. The advantage of this approach is that it does not alternate between a large data collection in one year and no data collection in other years. The operations and costs are spread out. Jeremy Aldworth noted that the approach of aggregating small chunks of sample over the years was used in the Mental Health Surveillance Study. An advantage of this method is that it is possible to
make modifications over the course of the years as the data come in and new information becomes available.
Hickie said that the traditional methods tend to involve face-to-face interviewing, which is very expensive and also has many limitations. However, new technologies are more cost effective and could enable more frequent data collections. Kalton replied that those in the survey business have been working on developing methods that would enable better use of these new technologies, such as online data collections, but the question is whether there is a way to do these with a good representative sample, rather than with people who sign up and are essentially “professional survey takers.” Hickie said that starting with a representative sample was important, but there are various ways for integrating the two methods, one of which was described in the presentation by Michael Kogan.
Merikangas asked Canino to share her thoughts on the discussions that were centered around impairment throughout the workshop and the possibility of not collecting full DSM diagnoses every year, while still being able to provide estimates that meet the criteria of child serious emotional disturbance. Canino said that based on the mandate, what needs to be measured is DSM disorders and impairment. Although the issues surrounding the measurement of DSM disorders have been generally resolved over the years, the measurement of impairment remains a challenge. Impairment is important for determining treatment and for predicting service use, and it is amazing that so much effort has been invested into developing a classification system for disorders and measures of disorders and so little effort has been invested into discussing what is most relevant for the families that are affected, which is the functioning of their children. The ultimate goal with any treatment is to see an improvement in a child’s functioning.
Canino recalled the perspective of child psychiatrist Michael Rutter on the difference between impairment in children and impairment in adults. Rutter said that if a child has a disorder and as a consequence of that the child drops out of school, he (or she) might not be able to get a job and might have difficulties for the rest of his life, even if he gets better or no longer has the disorder. In other words, the consequences of the childhood disorder are much more severe and are much longer lasting than those for adults. Because of that, it is crucial to get the measurement right.
When the Columbia Impairment Scale was developed, the youth version of the International Classification of Functioning did not exist. Canino, who worked with Hector Bird (Columbia University) on its development, noted that the definition used for youth was much simpler than any for adults, limited to a decrease in function. It was understood that functioning was a continuum that went from high functioning to low functioning and impairment. It was also understood that impairment
was context related. However, the conceptualization of impairment in the ICF-CY is not as clear: it refers to a decrease or deterioration in the functioning of a child that is expressed as a consequence of a health condition but is separate from the health condition. The ICF-CY also acknowledges that the disability can be manifested both at an individual level and at a societal level. Furthermore, it takes into consideration the developmental level of the child, which is extremely important because being impaired at the age of 3, when the child is in a family context and the impairments are related to that context, is different from being impaired as an adolescent. The instrument that is used or developed has to be one that considers all of these aspects. It is also important to note that this definition takes into consideration both mental and physical disorders to understand the construct of impairment for an individual.
Canino noted that the ICF-CY also discusses substantial impairment. This is important because for the purposes of resource allocation, the definition has to be limited in scope. Hickie argued that the opposite should be happening, with more focus on providing services to children who are not severely impaired yet, which would be ideal. Perhaps that is realistic in Australia, Canino said, but in the United States the discussion has not focused on that concept as yet.
Bose commented that if data are collected on both impairment and disorder, and there are people who have the beginning of impairments and perhaps no diagnosis of disorder, it would be possible to also conduct analyses focused on those groups, regardless of what the definition is for funding purposes. Canino agreed that there would be great advantages to being able to conduct those types of analyses. She added that she concurs with the perspective that impairment comes first and that it is not always a consequence of a disorder.
The instrument that comes closest in Canino’s view to meeting the current goals of measuring impairment is the World Health Organization Disability Assessment Schedule (WHO-DAS) for children. Canino said that the child WHO-DAS still requires psychometric validation, but it is working in Nigeria, which is an indication that it could work in the United States. She added that it is important to conduct the analyses for the United States, and SAMHSA could commission such work. The first step would be to analyze the data already available on adolescents. If that looks promising, the next step would be to conduct research to determine how it works with younger children, using the instrument that has been adapted from the adult WHO-DAS. Research would also be useful to develop severity scores or cutoff scores for various instruments and to reduce the length of the instruments.
Visser said that the issue of differential psychometrics as a function of ethnicity and race would also need to be the subject of research. At the
moment, it does not appear that race and ethnicity could be added to a model, and that is a limitation. For example, there is a large percentage of Puerto Ricans in New York and a large percentage of Mexican Americans in Texas, and those two groups have different health-seeking behaviors, as well as different rates of reporting symptoms and impairment. If state is added to the model, these differences could become a problem. The goal would be to find an impairment measure that has strong psychometrics properties within racial and ethnic subgroups because there will be no other way to account for these types of differences if a modeling technique is used.
Merikangas added that another research project that SAMHSA will need to undertake is to evaluate the role of informants by developmental level. She asked Canino to clarify how one would measure a decrease in functioning. Is this an area where longitudinal data would be most useful? Canino replied that parents can be asked how their children’s functioning compares with that in the previous year.
Davern said that some of the speakers made a convincing case from a research perspective that prioritizing impairment would be better. If the goal is to use these estimates for funding allocations, then one result of focusing on the prevalence of impairment rather than the prevalence of disorder could mean that states that are successful at reducing impairment would get less funding than they would need on the basis of the prevalence of disorders over time. This potential result could also be an issue for a modeling approach, depending on whether state is included in the model.
Visser noted that the states are very experienced with these types of challenges and have creative ways of addressing them. Many of the contracts they fund are performance based. Based on the experience in other areas, such as infant mortality, as the focus shifts to the final few percent, it is recognized that those are more difficult to address and more funding is needed.
Kalton said that, in general, it may be premature to think about specific methods, such as modeling. It is important to first think through the overall design. One of the first questions is whether it would be feasible to have a good instrument that could be administered to children on an annual basis, possibly on the internet, with the purpose of measuring the relative levels of serious emotional disturbance across the states. If this is possible without the need for follow-up clinical interviews, it would be the most important parameter to establish first. Including clinical interviews is expensive, even if there is modeling involved, and it would be difficult to do on a reasonable scale. Perhaps what would be useful is an instrument that could be administered regularly and would only be
subjected to some validation work occasionally and then modified on the basis of that validation work, as needed.
Kalton added that there are a number of other smaller issues that deserve further discussion. One is that the current definition seems to include a criterion of sufficient duration and a reference to the past year. It is not clear how this would fit with the available scales, which seem to have varying reference periods. Another issue is the role of comorbidity, which came up in several presentations, but it is not yet completely clear as to how comorbidity and impairment interrelate and what the consequences of such interrelationships might be.
In addition, Kalton said, it will be important to get a good handle on the implications of the child’s age and what is feasible to do with different age groups. In particular, what can be done with those under the age of 6, and what can be done with children in the 0-3 age group? Furthermore, how important is it to have data from three different informants—the child, the parent, and the teacher—and would it be possible to develop an approach that does not require that? Some of these activities would be enormously costly and some might be mutually exclusive, such as combining an internet data collection with a complicated design that requires multiple informants.
When grappling with these ideas, Kalton said, perhaps the fitness-for-use perspective that was mentioned in the presentations would be useful to keep in mind. Perhaps the goal is not perfection, but to identify a measure that is reasonable and acceptable and charts the variation across states so that funding allocations can be based on it. Canino mentioned that there is no consensus on where to place impairment cut points. Perhaps this does not matter if the goal is to allocate a fixed amount of funding across states. But if the overall amount of funding available depends on the rates, then that is a different scenario.
Canino addressed Kalton’s point about the number of informants by saying that for pragmatic purposes, such as the data collection that SAMHSA may be undertaking, reasonable decisions can be made on the basis of the age of the child. For example, for children under 12, she would only do parent interviews. For children 13 and older, she would collect data from the child only. For specific purposes—for example, if a good measure of ADHD is needed—she would include the child and the teacher. This approach would not meet the requirement for obtaining research funding from the National Institute of Mental Health (NIMH) because NIMH reviewers would expect all three informants to be included, but it would be a practical solution for SAMHSA.
Following up on Kalton’s question about whether the “gold standard” clinical interview would always need to be included, Bose asked the participants for their views on a hypothetical design that would be
based on a one-time cross-sectional study that included the Composite International Diagnostic Interview (CIDI, which is administered by lay interviewers), and also included potential predictor variables, such as the full SDQ, a short scale that Goodman has tested for his purposes, or some other set of predictors. What would be the flaws of a model for which the outcome would be serious emotional disturbance based on the CIDI score for the age group that the CIDI covers and the predictor variables? Would a design along these lines pass the laugh test? Also, what sample size would be required? If data were available for 10,000 children who completed the CIDI, would it be possible to develop a predictive model? And what sample size would be sufficient for calibration? Typically, it appears that very small sample sizes, such as 10 percent or less, are used for calibration, but would it be adequate to administer a clinical version to 40 cases for each disorder?
Merikangas said that a similar approach was used as part of the NCS-A in about four sites. The test included oversamples of children with certain disorders that were more difficult to measure, such as mania. A lot of work was done very inexpensively on a small sample, which was later used to calibrate the broader instrument for larger samples. A recent Canadian study also used a similar approach, but the data from the study were not yet available for this workshop.
Steven Heeringa noted that Ronald Kessler and his colleagues have used this calibration approach for the CIDI-SC mental health diagnoses in the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS).1 The samples might be 400 or 500, and there might be 40 positive screens for each diagnosis. That particular calibration, represented by the calibration from Y to Y* in his model, will have an extreme amount of variability. Heeringa said that, in his understanding, the way some of the other studies have used this approach was to determine what criteria in the Y variable should be used to produce the most reliable estimate, that is, as a projection of true value, rather than to develop an adjustment model. In the case of the study described by Aldworth, Heeringa commented that the standard errors of the coefficients in the logit model and, therefore, the variance of any predictions from that model are going to be a function of the strength of the relationships, as well as the relative
______________
1Kessler, R.C., Santiago, P.N., Colpe, L.J., Dempsey, C.L., First, M.B., Heeringa, S.G., Stein, M.B., Fullerton, C.S., Gruber, M.J., Naifeh, J.A., Nock, M.K., Sampson, N.A., Schoenbaum, M., Zaslavsky, A.M., and Ursano, R.J. (2013). Clinical reappraisal of the Composite International Diagnostic Interview Screening Scales (CIDI-SC) in the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). International Journal of Methods in Psychiatric Research, 22(4), 303-321.
sample sizes in Step 1 and Step 2 (see Chapter 4). Adjusting 10,000 cases on a model fitted to 40 cases might be difficult.
The Aging Demographics and Memory Study (ADAMS) described by Heeringa included dementia assessments that cost approximately $1,000 per person for the sample members that were in the Y measurement. The ratio was 857 assessments and about 8,000 to 9,000 cases, and there was no guarantee that it would produce a model that functioned as well as it did. But the researchers had very good initial information to be able to stratify the Step 2 sample and improve the efficiency of estimating the model. Because of the amount of information that was available for the stratification, the precision of the estimates of prevalence was high even from direct estimation, and it was much better than would have been possible on the basis of a naïve sample of a similar size. For the next ADAMS data collection, the researchers are not going to include the assessments because they have concluded that they have enough information on that particular step. Heeringa and his colleagues were able to refine the screening items to obtain these predictions of dementia through that step.
Kilpatrick commented that if the ultimate purpose is to establish the relative numbers of cases per state for the allocation of funding, then even if the lack of clinical calibration resulted in elevated prevalence rates—and it is not clear that it would—the relative burden per state would presumably still be correct. He also wondered whether more of the state- or substate-level data available from the U.S. Census Bureau could be used for modeling.
Schaffer noted that there was a lot of discussion of gold standards, and perhaps interviewer effects deserve more attention. When diagnostic-style interviews are included, there is probably only a small number of trained interviewers, and it is likely that there is a high interviewer variance. It is worth thinking about how to take this into account even if there are not enough interviewers to estimate it. At the minimum, attention should be paid to how the cases are assigned so that a few great interviewers do not end up dominating an entire dataset.
Visser described one of her ongoing studies with colleagues as a context for some cost considerations. Their Project to Learn about ADHD and Youth Mental Health (PLAY-MH) study is a multistage design that includes a minimum of 5,000 children at each study site. The sites receive $250,000 each year for 2 years to do screenings of the entire school district and then pull out a sample. On the basis of simulations, the researchers determined that they needed 650 diagnostic interviews to be able to estimate prevalence of 2 percent or greater. Getting those 650 interviews has not been easy, even with incentives for the families, the schools, and even the teachers who are conducting the screenings ($4 per screen). If SAMHSA is considering some school-based data collections, in a few sites
or even nationally, a partnership with the Centers for Disease Control and Prevention might be useful to leverage mutual resources.
Russell asked for clarification on the idea of starting the measurement with impairment. How would the disorders be ascertained in this case and how does one ensure that the definition of child serious emotional disturbance is being met? At the minimum, SAMHSA would need to be able to say that the process resulted in a fairly decent proxy measure. Merikangas replied that the main difference would be the different order, but some disorder information would still be collected. The exact data that would go into the basket and the process would need to be decided. Hickie agreed that the disorders would not be skipped and clarified that according to current methods, one has to have the disorder in order for an impairment to be obtained. The idea would be to separate the two and allow for impairment information to emerge, independent of the disorder. It would still be possible to obtain the prevalence rate, but impairment data would not be limited to those with a disorder.
Kalton reminded the group about the discussion related to the role of treatment and that it will be important to capture the disorder information even if a child is successfully being treated and the impairment has been reduced or eliminated. Hickie agreed that the disorder questions should not be skipped, regardless of the level of impairment. Visser cautioned that the issue of how exactly the disorder information will be captured deserves very careful consideration, especially if the intent is to do model-based estimation.
