3
Measurement Challenges for Population Surveys
Peter Szatmari (University of Toronto) focused his presentation on current issues in psychiatric epidemiology of serious emotional disturbances in children, primarily in the areas of behavioral and developmental disorders and the measurement of impairment among children between the ages of 2 and 6. He also discussed his experiences with the Ontario Child Health Study (OCHS) to illustrate some of the challenges.
Szatmari said that there is widespread agreement in the literature on the prevalence estimates of many disorders, with or without impairment, and this is especially true in the area of disruptive behavior disorders—oppositional defiant disorder, attention deficit hyperactivity disorder (ADHD), conduct disorder—across developmental stages. A just-published meta-analysis found that there is relatively good agreement on the prevalence of mental disorders at around 13 percent.1 The analysis also found that the prevalence rates would be approximately 17 percent higher if the criterion of impairment were not included. The availability of this research raises the question of whether another epidemiological study of child and adolescent psychiatric disorders is necessary. Szatmari pointed out that there are many unanswered questions that are relevant to mental health and addiction policies. Although SAMHSA’s definition
______________
1Polanczyk, G.V., Salum, G.A., Sugaya, L.S., Caye, A., and Rohde, L.A. (2015). Annual research review: A meta-analysis of the worldwide prevalence of mental disorders in children and adolescents. Journal of Child Psychology and Psychiatry, 56(3), 345-365.
of child serious emotional disturbance is limited to substantial impairment, Szatmari argued that, from a policy perspective, focusing on any disorder and any impairment, regardless of severity, is more meaningful. One would not want to limit treatment to only severe cancer or severe diabetes, and any mental disorder that causes impairment similarly requires treatment.
To better understand child serious emotional disturbance, more longitudinal data are needed on onset, chronicity, and course of disorder and impairment. For example, there are virtually no data available on the extent to which disorder and impairment follow similar or different pathways and how they are linked. Further work is also needed to link the available data to administrative databases. Linking epidemiological data to administrative databases, as discussed by Benjamin Druss (see Chapter 2), is likely to be very useful in informing mental health policies and guiding resource allocation for assessment and treatment. Finally, research is needed to enable a more nuanced understanding of modifiable risk factors for serious emotional disturbance in children.
Canada has had several initiatives to measure child serious emotional disturbance and impairment. One such initiative was the National Longitudinal Study of Children and Youth, which is now inactive, but included multiple waves, and eight cycles. The survey was focused on symptoms and impairment, rather than disorders. Because the study was funded by the government, confidentiality restrictions made it very difficult for researchers to use the data. A current initiative is the Early Development Instrument (EDI), which is turning out to be very useful. The instrument, which is completed by teachers for each child in the last year of kindergarten, measures “school readiness” on the basis of motor, language, social, and emotional-behavioral milestones. The data have been collected in most Canadian provinces, and they are now beginning to be used in many other countries around the world, with the support of the World Health Organization and UNICEF.
Szatmari discussed his experience with the 1987 OCHS and the current, 2014–2015, sequel to that study (OCHS-S). The study, which has a nested design, includes 25,000 children, in 13,500 households, in 180 neighborhoods across Canada. Up to three children are sampled in each family, which provides useful information about the familiality of disorders and a better understanding of what families face.
One of the main goals of the OCHS-S is to document the prevalence of mental disorders among children between the ages of 4 and 16. The team also wants to document child mental health needs and assess health system response by linking data on children with serious emotional disturbance to administrative databases, such as those for education and social services. The researchers also want to better understand the influence of
poverty and income inequality on child serious mental disturbance and identify modifiable environmental influences, within a nested design.
In order to measure disorder, in 1983 the OCHS used a tool called the Survey Diagnostic Instrument. This was updated for the OCHS-S to make it congruent with the DSM IV. The disorders measured are ADHD, conduct disorder, oppositional defiant disorder, major depressive disorder, specific phobia, social phobia, separation anxiety disorder, generalized anxiety disorder, and substance use disorders. A randomly selected subset of the sampled children received a structured psychiatric interview, the Mini International Neuropsychiatric Interview for Children and Adolescents (MINI KID). The diagnostic instrument was completed by at least two informants for each child: parents, for all the children (4-16); the children themselves for those aged 12-16; and teachers for those aged 4-12.
Impairment is measured in multiple ways in the OCHS. The survey includes an independent measure of impairment, similar to the Strengths and Difficulties Questionnaire (SDQ) (see Chapter 2). The OCHS also includes the Brief Impairment Scale (BIS), and the MINI KID, which has impairment tied to each disorder. Impairment is defined across four domains: family, peers, school, and community.
The OCHS-S data will shed light on how the prevalence of mental disorders has changed since 1983, when it was 18 percent. Of great interest is the overlap between chronic medical conditions and mental disorders. In 1983, among children with chronic medical conditions and a functional limitation from the medical condition, rates of disorder were around 40 percent. This rate was 14 percent among children with no medical condition. Comparing families living above and below the poverty line, in 1983 children living in poverty experienced more mental health difficulties, and only 17 percent of those children with a disorder were using mental health services. Since then, more funding has been devoted to child health initiatives, and there are more programs available, so the new data are expected to provide important information for research and policy.
Szatmari said that the OCHS involves a large team across Canada, and the current study went through a very thorough process to collect stakeholders’ input on the design. In addition to a comprehensive review of the literature, the team spent a lot of time talking to government policy makers about the mental health issues of concern to them and what questions they wanted answered. The team also conducted focus groups with clinicians and service providers, getting their input on the same issues. In hindsight, not enough time was spent on patient engagement, talking with children and families. The patient engagement literature is very impressive and reveals the important role patients can play in designing a study, formulating research questions, talking about measurement tools, and even in the interpretation of the results. Going forward, this area
deserves a lot of attention in mental health and epidemiological surveys, Szatmari said.
Szatmari also discussed his views on measuring impairment. He noted that it is important to distinguish impairment, distress, impact, and the burden associated with a disorder, as well as how these concepts differ from the quality of life. All of these concepts have slightly different nuances. Robert Goodman has argued that the measurement structure of impairment is stable, and it is very much defined by context: family, school, community.
It is also important to consider the relationship between symptoms and impairment, which was also discussed by Glorisa Canino (see Chapter 2). Cross-sectional studies indicate that impairment and symptoms overlap but are not identical. Unfortunately, not enough data exist from longitudinal studies. Some treatment studies provide experimental information on impairment and symptoms, but most focus on symptoms, and significant improvement in symptoms does not necessarily signify significant improvement in impairment. The research suggests that they are two separate constructs.
As Canino also described, Szatmari noted, there are two theoretical frameworks for measuring impairment. According to the DSM framework, symptoms cause impairment. In contrast, the framework of the International Classification of Functioning for Children and Youth (ICF-CY) is much more complex, and there is a bi-directional relationship between impairment, symptoms, and environment.2 Szatmari argued that it is impossible to make causal inferences when it comes to symptoms and impairment, and it is best to be agnostic on this topic. Therefore, it is important to have an independent global measure of impairment. While the Children’s Global Assessment Scale (CGAS) had a crucial role in the development of the field, it is an example of a measure that is not independent because it is “contaminated” by symptoms. Although this is a very important topic, the field overall has invested substantially more resources into the measurement of symptoms than the measurement of impairment.
Szatmari said that a very complicated measurement issue is that of comorbidity. If there are two disorders present, it is difficult to determine whether the second disorder is associated with any impairment in addition to the impairment that is associated with the first disorder. Another measurement issue, as was discussed earlier, is related to the role of informants. There are differences in informant reports not only for symptoms
______________
2See Cramm, H., Aiken, A.B., and Stewart, D. (2012). Perspectives on the International Classification of Functioning, Disability, and Health: Child and Youth Version (ICF-CY) and occupational therapy practice. Physical and Occupational Therapy in Pediatrics, 32(4), 388-403.
but also for impairment. The Bella study, which measures mental health in children, adolescents, and young adults in Germany, found differences between parent and child reports, and research based on the original OCHS found differences between parent and teacher reports. Another issue is that if not all disorders are measured in a study, the result can be data on impairment but no data on disorder.
Szatmari said that he agreed with Canino that the ICF-CY framework is more nuanced. It allows for variation by sex/gender, culture, context (school, home, community), and so on. As Canino noted, functioning is a continuous concept, and cutoffs that determine impairment have not yet been validated. There is virtually no work in this area establishing what is and what is not impairment and how mild impairment differs from substantial impairment.
One important question that has been raised is whether measures of impairment need to be disorder specific. Szatmari argued that the impairment associated with ADHD can be different from the impairment associated with anxiety or mood disorders.
A related question is whether measures of impairment need to be developmentally specific and whether measures exist that are relevant at different developmental stages. Focusing on the very challenging issue of measuring impairment in the 2- to 6-year-old age group, Szatmari pointed out that symptoms or disorders have a very different impact on family, peers, school, and community at this developmental stage than they do for older age groups. For example, for the younger age group, caregiver burden is a very important measure of impairment, and the Parent Stress Index is one example of a scale available to capture that. There are also several other scales that are very good at measuring social relationships for infants.
Another concept that is very important to measure for the 2–6 age group is the acquisition of developmental competence in motor skills, language, and communication. Szatmari said that the Vineland Adaptive Behavior Scale is the best for this purpose. Two other very useful instruments are the Adaptive Behavior Scale and the Ages and Stages Questionnaire. The question is whether developmental assessments are the same as measuring impairment, and researchers in the infancy field would say that they measure the same concept, but there may be others who do not agree.
The EDI, discussed earlier, is a measure that captures school readiness, which is also very relevant for children between 2 and 6 and can be considered an aspect of impairment. A final measurement challenge noted by Szatmari as relevant for children younger than 6 is that, at this developmental stage, disruptive behavior disorders, autism spectrum
disorders, and impairment are very difficult to disentangle. Because of that, it is best to measure all three.
In summary, Szatmari emphasized that impairment is important to measure because it drives the need for treatment. However, the measurement of impairment is complex, multidimensional, and influenced by multiple factors, not just disorder. The use of multiple informants is also necessary, and the role of teachers as informants would deserve further consideration. Furthermore, attempting to apply one construct of impairment to all child and adolescent developmental stages is an oversimplification. Ultimately, as Canino had noted earlier, the ICF framework is more useful when thinking about impairment than the DSM because it is more nuanced.
Ideally, Szatmari said, SAMHSA would count children with any diagnoses (not just specific diagnoses) and any impairment. This concept of serious emotional disturbance would be really useful for the next round of epidemiological studies. Including teachers as informants is very important, particularly when it comes to measuring impairment in addition to symptoms, because teachers in primary school know the children very well. Moreover, at least in the Canadian experience, collecting teacher data was not particularly challenging. More work is needed to develop instruments appropriate for each developmental stage, but having a global measure of impairment that is independent of symptoms and disorder makes a lot of sense: otherwise, the causal relationships are too entangled.
Ian Hickie (University of Sydney) began his presentation by noting that he serves on the Australian Mental Health Commission and recently participated in a meeting at the Australian Academy of Health and Medical Sciences. The meeting included people from national health agencies and welfare and social science agencies, researchers, and others. At the meeting, the researchers had one view of what is needed, while those involved in policy making argued that the research never answers the questions that are most relevant to public policy. This is similar to SAMHSA’s predicament, he said. The discussion needs to focus on the fundamental question of the purpose of specific programs and what kinds of measurement and instruments will deliver the kinds of information that are of use to those who are responsible for those programs.
It is important to note that there are some differences between surveillance activities and national health surveys, Hickie said. Surveillance methods are primarily applicable when the goal is to accurately measure changes in incidence or populations, or in regional variations, and the
depth of the data is not a priority. Examples might be suicide prevention programs, or alcohol and drug use programs, if the intent is to enable a rapid public response. By contrast, the current focus of most national health surveys is resource allocation, and the Australian government clearly states that it is interested in return on investment. In certain ways, the Australian system is similar to the U.S. system, where taxes are collected by the federal government and returned to the states on the basis of resource allocation considerations. Because there is constant competition for these resources among health and social service agencies, key issues are the return on investment and the strategies of reducing impairment over time.
Referring to the prior discussions about instrumentation challenges, Hickie commented on the issue raised by Szatmari about the relationship between symptoms and impairment. In the DSM tradition, symptoms are asked first, followed by impairment. However, nearly all of the longitudinal studies in Australia show that the impairment has its onset earlier than a diagnosable disorder. Consequently, Hickie argued, instruments should ask about impairment first. In other words, the first question should be whether a child is impaired, and if so, what is the cause or what is the nature of the impairment. This is the opposite of the current common practice of asking whether the child has any symptoms, and then asking whether he or she has an impairment. The approach of asking about impairment first would pick up on a different set of problems than asking first about a symptom, and it better reflects the real world. The issue of the age of onset of the impairment further underscores that symptoms and impairment have different trajectories and likely respond to different sets of interventions.
Hickie noted that there are also positive aspects to the DSM system: for example, the early diagnostic interview schedules were very useful to Australian researchers in the 1990s, when several national surveys were introduced, many based on methods developed in the United States. These early efforts were very useful because the then-prevalent public perception in Australia was that mental disorders in children were rare and that substantial resource investments were not needed. New national data revealed the true magnitude of the problem and led to discussions of the concept of an unmet need. The related notion of a “met un-need” also emerged around the same time to describe services provided to those who may not really need them, according to a concept of optimization.
Hickie discussed research based on work done by the U.K. Govern-
ment Office for Science, on the topic of the mental wealth of nations.3 This research shows that early intervention is key. Although it seems obvious to want to provide the most services to those who are most severely impaired and have an obvious need for care, it is not clear whether that is the best return on investment in these situations. As Szatmari had discussed with regard to cancer care, in the past 30 years there has been increasing emphasis on early intervention: early diagnosis and treatment have reduced the progression of illness and the morbidity associated with cancer. In contrast, the concept of early intervention is not widely accepted in the field of child development, Hickie said, but to build the most mental capital, early intervention strategies are needed. And when looking for early evidence of deviations, they will be in impairment, not symptoms.
One challenge with using cross-sectional surveys in the area of mental health is that there is a high variability in testing on any particular day, especially for symptoms. Another challenge is that the data do not contain enough information about long-term implications. Furthermore, understanding prevalence is not enough. As Szatmari noted, surveys from across the world all show the same prevalence rates, and the surveys also consistently show that when impairment is factored in, the rates are lower. Getting the estimates right is easier for adolescents than younger children, but if the age of onset is obtained, those data can inform resource allocation and interventions.
Hickie described two initiatives in Australia focused on early intervention and risk factors. One is a population-based mental health promotion framework, centered around four topics: (1) individual brain development; (2) the family and social context of development; (3) the educational and experiential aspects; and (4) autonomy, social connection, and physical activity. The other initiative is centered around effective prevention and early and continuing intervention for mental health problems. It focuses on (1) early universal prevention (reducing risks, trauma, alcohol, drugs, inappropriate technology use) and reducing harms (suicide and self-harm); (2) early intervention for anxiety and depression; (3) continuing care supported by technology; and (4) building new service delivery platforms.
As in Canada, Australia also began placing a lot of emphasis on tracking early development, particularly on school readiness. Recently, the age of preschool screening was lowered from 4 to 3, and there is a desire to attend to problems early. For example, in the autism area, there is a lot of
______________
3Beddington, J., Cooper, C.L., Field, J., Goswami, U., Hupper, F.A., Jenkins, R., Jones, H.S., Kirkwood, T.B.L., Sahakian, B.J., and Thomas, S.M. (2008). The mental wealth of nations. Nature, 455, 1057-1060.
focus on the period between the ages of 2 and 3. These initiatives have been very controversial because some have argued that they are driven by the pharmaceutical industry, but Hickie emphasized that the initiatives are driven by data from the early development indices.
Australia now also has an assessment, the National Assessment Program–Literacy and Numeracy (NAPLAN), which looks at educational assessment after entry to school, in school years 3, 5, 7, and 9. The plan is to use these assessments for research beyond just population mapping, including individual-level emotional tracking.
Hickie also noted that there has been interest in metrics such as disability-adjusted life years, which is a measure of overall disease burden, expressed as the number of years lost due to disability in the population. This measure can be helpful for understanding the persistence of emotional difficulties and the impacts on educational and social development over time.
Another development in Australia is the transition to a National Disability Insurance Scheme, which is trying to provide assistance to parents on the basis of individual need rather than diagnostic categories. An important component of this scheme is the magnitude of impairment and the amount of support that may be needed on an ongoing basis. The national surveys conducted since the 1990s have provided continuous data on service utilization, and young people continue to have the lowest rates of service utilization, despite funding invested into changing this. As Druss also noted in the previous session, the survey data on this topic are very useful for public policy because they can shed light on whether the programs are working. As Kathleen Merikangas also pointed out earlier, comorbidity is a major issue, which means that understanding the use of physical health services is also important.
Australian government surveys have the same limitations as surveys elsewhere, Hickie said, with not enough emphasis being placed on whether they are designed in a way that enables linkages to other data sources. There is increasing interest in this, and the goal should be to be able to link surveys not only to other sources of health data, but also to other information, such as education and welfare databases. So far, privacy concerns have limited the availability of longitudinal databases. To the extent that longitudinal data are available, these have generally been conducted by research agencies, rather than the government. The Brisbane Longitudinal Study of Adolescent Twins is one example of such research, which has provided useful data on the persistence of impairment.
Australia also faces challenges similar to other countries for telephone surveys, which are becoming increasingly difficult to field as more households abandon landlines. In-person interview visits are also challenging to conduct in Australia, Hickie said. However, online technologies turn
out to be a very good way to collect in-depth information on health-related topics, and Australia has been experimenting with online sampling of young people as part of a large investment by the government in a cooperative for research on young people. One advantage of online data collection is that respondents are more likely to report certain things, such as suicidality and substance use, online than in a telephone interview. Online data collections can also be more flexible in accommodating modules of different questions administered to subsets of the sample. Tracking over time is also possible in online surveys.
Hickie said that a lot of emphasis has been placed in the field on standardizing instruments, but this focus assumes that data collections will continue to be based on the same types of approaches as they have been for the past couple of decades. Although the sampling challenges are substantial, especially if the goal is to produce nationally representative data, technological developments represent an opportunity for government agencies to rethink their approaches.
In conclusion, Hickie reemphasized the importance of the implications of age of onset and of obtaining accurate impairment data. He believes that beginning with impairment and then finding out the cause of the impairment would be a better approach to collecting these data than the traditional way of starting with disorder and then asking about impairment.
ADVANTAGES AND DISADVANTAGES OF THE STRENGTHS AND DIFFICULTIES QUESTIONNAIRE
Robert Goodman (Kings College, London) joined the workshop briefly by telephone to comment on the Strengths and Difficulties Questionnaire (SDQ). He noted that as one of the developers of the SDQ, he has a vested interest in it, and therefore a potential conflict. He said that the group should discuss the relative merits of the SDQ without his participation, but that he would briefly describe the potential usefulness of the instrument and answer any questions. He also noted, going forward, electronic use of the SDQ will require a license fee, while the paper versions will continue to be available free of charge.
Simply put, Goodman said, the SDQ works surprisingly well in studies of the general population, considering the complexity of the concepts measured. The instrument, which includes 25 items on psychological attributes, is relatively brief and can be used for children as young as 3. Despite its simplicity, the SDQ has good mathematical properties. For example, at least in the United Kingdom, the mean SDQ difficulties score can be used to predict the prevalence rate of mental disorders as judged by a much longer, independent diagnostic assessment.
The SDQ also includes an impact supplement and measures impairment with reasonable psychometric properties. In addition, what is more relevant to the public and policy makers, one can determine from the SDQ what proportion of parents who are reporting on their children believe that their child has a definite or severe level of difficulty.
Kathleen Merikangas asked Goodman to comment on how the SDQ could tap into meeting SAMHSA’s goals of measuring both the DSM-equivalent diagnosis and impairment. Goodman replied that there is uncertainty about whether this would work in the United States. In the United Kingdom, there is a strong relationship between the mean SDQ total difficulty score in a particular population and the prevalence of DSM diagnoses in that population. For use in other countries, it would be important to start with a validation study, as one would with any other measures. There are datasets in the United States that could be used to do this testing, and if it turns out that it works, this would be a very powerful way statistically to look at the prevalence of DSM disorders in a particular sample. In terms of the SDQ impairment measure, which is simply asking respondents to judge how much the difficulty is interfering in everyday life or causing distress, it has psychometric properties that are as good as any.
Heather Ringeisen asked Goodman about whether he has done any research on the possibility of shortening the SDQ to a handful of essential items, for example, five or six items that would be well suited for use in time-sensitive national surveys. For example, the National Health Interview Survey includes a five-item version of the SDQ, and an additional impact item. Goodman replied that he has done this type of research and that he is a minimalist by nature, but in his view the short version of the instrument does not work well, either for one specific disorder or for predicting the presence of any mental disorder. Goodman added that he wishes that it would work, but the short version does not have enough sensitivity or specificity. However, he said, it would be possible to scale back the questions to just one—asking parents in a broader way whether they think their child has a problem or not. That approach would potentially be useful. However, between the one question and the current length of the SDQ, no other length really makes sense.
Susanna Visser noted that the Centers for Disease Control and Prevention has data from community-based studies that involved parents and teachers completing the full SDQ. The SDQ scores were used to identify children who would be invited for a follow-up diagnostic evaluation as part of a multistage design. The current sample includes approximately 1,000 families, and new families are being added. These data could be used to carry out a validation study on a U.S. population.
Canino asked Goodman to clarify why he used the word “impact”
instead of “impairment.” Goodman said that the DSM criteria specified both distress and impairment, and it seemed useful to link the two constructs together to retain the distress component. The impact portion of the instrument first asks whether the person has any difficulties related to concentration, emotion, behavior, or relationships. If there are difficulties, several follow-up questions are asked, including whether the difficulties cause distress and whether the difficulties interfere with everyday life in the key areas of family, learning, leisure, and peer relationships. The impact score is derived on the basis of the follow-up questions about distress and interference with everyday life.
