6

Assembling the Pieces and Integrating Elements

Throughout the workshop, participants shared a myriad of experiences, evidence, and practice in multiple domains related to mental health and violence prevention. In the last panel of the workshop, speakers and participants communicated knowledge and best practices on inexpensive and more nimble program evaluation. They also discussed final thoughts and raised additional questions on advancing the science and practice.

EVALUATION OF PROGRAMS FOR VIOLENCE PREVENTION AND MENTAL HEALTH PROMOTION¹

A number of traditional and alternative evaluation designs can be used to assess the efficacy of violence prevention programs, by addressing important questions such as could a program work under optimal conditions? Does a program work under realistic conditions? And how does a program work, and can it be improved? Hendricks Brown described a 2009 National Research Council and Institute of Medicine report that laid out these three stages of evaluation, starting with efficacy and effectiveness studies and leading to implementation and dissemination studies (NRC and IOM, 2009). The former two address whether a program achieves the desired outcomes, and the latter deal with adoption, sustainability, and scaling up, and require different evaluation models.

___________________

¹ This section summarizes information presented by Hendricks Brown, Northwestern University.

Issues of evaluation are value-laden, he remarked, not just from a scientific perspective, but also a community perspective. Of note, there is a history of abuse in research, particularly in communities of color. There are also issues of cost, some of which disproportionately burden nongovernmental organizations that work with people of color. He further explained that these issues need to be confronted upfront because they are relevant to whether an evaluation can be carried out. He cautioned, though, that if a program is to be considered evidence based or evidence informed, it should have information to that end, at least under some set of circumstances. At a minimum, certain indicators could be measured, such as program fidelity and participation. Furthermore, Brown said the gold standard for evaluation is the randomized controlled trial (RCT) because it is designed to eliminate as many biases as possible. He noted that while sound evaluation evidence can be obtained from an RCT, sometimes it is unethical or impractical to conduct one.

Because violence is a relatively low-frequency event, large studies are needed to observe the effect sizes that indicate program success. He described the example of looking at completed suicide in youth. Because the youth suicide rate is small, it is estimated that a study would require at least 1 million person-years of risk, which would mean following 1 million youth for 1 year or 100,000 youth for 10 years; it would require intense labor input to follow up with so many individuals. Such large studies are expensive. Thus, he considered other options to enrolling large numbers of people in trials. The first is to look at an intermediary outcome; for example, assessing suicide attempts versus completed suicide, because attempts occur at a higher frequency than completed suicides. The second is to combine data across trials and synthesize findings. A third approach is to use a less expensive evaluation design, such as using administrative records to screen and identify individuals over a long period of time. The Positive Parenting Program, or Triple P, a family support program to prevent behavioral and emotional problems in children, used such an approach. In the United States, the National Death Index and the National Violent Death Reporting System are other sources of data.

Additionally, he described a method he called “roll-out design,” sometimes referred to as “dynamic wait list design” and “stepped wedge trial design,” in which groups are randomized to treatment and control groups as the program is rolled out. It has the added benefit of allowing an examination of the implementation strategy. He gave an example of one such program, which used a suicide gatekeeper program called Question, Persuade, and Refer (QPR) in a school district. The program was rolled out to randomly assigned schools, with new schools added each quarter. The outcome, school referrals for suicide, was measured before and after the schools were enrolled. By using a rolling method of enrollment, the

first year and the first three quarters of the second year always maintained a control group (i.e., un-enrolled) for comparison. One of the advantages of this methodology is that, when the whole community has agreed to the intervention, there is still a process for incorporating everyone. Because all the schools received the intervention, there was no delay or associated costs. At the same time, there was a benefit to the schools that received the intervention first because often school districts are eager to implement a program sooner rather than later. However, those who receive the intervention later have the advantage of more efficient implementation. Finally, an important benefit of roll-out design is that evaluation is built into implementation—meaning that cost issues around evaluation may be averted and that accountability is naturally integrated with the process.

Once a program has been deemed effective, the final step in evaluation is making a program work. Brown noted that there are two areas for this step. On the research side, implementation science gathers generalized knowledge of program design; while on the practice side, quality improvement is an ongoing local evaluation. He shared the RE-AIM perspective for ensuring program success:

• Reach: the percentage of the community that receives the program
• Effectiveness: does the program have benefit?
• Adoption: bring into host organizations and service-delivery systems
• Implementation with fidelity
• Maintenance

Brown further noted that these measurable elements determine whether the implementation will be successful.

In the discussion following the presentation, participants queried Brown regarding issues of sustainability, such as funding evaluations and ensuring continuity and consistency as personnel change at program sites. Participants discussed requiring a measure of evaluation in grant proposals and the importance of building partnerships with the entire community. They also considered alternate methods of gathering data for evaluation, such as practice-based evidence and ongoing data collection in mobile health (mHealth) programs, that could integrate with traditional methods.

REFLECTIONS AND THE WAY FORWARD

The workshop closed by synthesizing the discussions over the 2 days, with participants offering reactions and thoughts on the topics presented. They spoke from multiple perspectives, reflecting the diversity of opinions and practices present at the workshop.

Mental Health Services²

Colleen Barry recalled Thomas Insel’s remarks that untreated mental illness and alcohol and substance use disorders are associated with violence, and there is a role for mental health services and policies. She noted that there is also a role for understanding evidence; from a program perspective, dissemination is a critical piece in seeing translation of evidence into practice. Despite several rigorous studies demonstrating evidence of certain practices, there has been very little uptake of these programs. She also said that, in regard to evaluation, there is a role for examining unintended consequences of policies.

One way to implement and disseminate services and policies would be to create a financial incentive, such as an appropriately applied pay-for-performance model. She gave the example of an accountable care organization in Massachusetts with 64 different performance measures. Because only one measure was related to the population of individuals with mental health and substance use disorders, there was little change in regard to that system of care. She closed by noting that sustainability is critical, and financing and insurance changes can be instrumental as well.

Mental Health and Justice³

Sheldon Greenberg reflected on the knowledge gap between the mental health field and the justice system. He noted that while mental illness plays a significant role in justice, it is not on the radar of the politically driven system. Within all disciplines of the mental health field, the end goal is the same: better quality of service and support for people with mental illness. So what, he queried, is the formula for achieving this goal, and how can practitioners, researchers, and advocates develop it? He envisioned this formula to be cross-cutting, with different stakeholders having the ability to refine it for their own practices.

He echoed Barry’s comments about disseminating research, also noting that along the translation pathway, barriers exist that inhibit frontline professionals from accessing research around what works. Other types of research are of interest as well. When a patient first comes in contact with the system, what happens to the information that is initially provided? How does that impact interventions? Research on fear was also of interest to Greenberg, particularly people’s fear of what will occur after the initial contact that might inhibit them from being honest. Knowing more about this

___________________

² This section summarizes information presented by Colleen Barry, Johns Hopkins Bloomberg School of Public Health.

³ This section summarizes information presented Sheldon Greenberg, Johns Hopkins University School of Education.

fear could assist in developing more trust with professionals and countering misinformation in the media and the general public.

He closed by asserting that cross-disciplinary collaboration for policy, coupled with mandates for providing better education and training in all fields, could build an institutional culture of understanding across all disciplines. In parallel to this work in professional fields, people with mental illness and their families, he argued, should be better equipped to engage the system at any point of contact.

Culture and Construction of Mental Health⁴

Janis Jenkins reiterated her earlier comment that empirical research demonstrates the central place of culture in nearly every aspect of mental illness. Thus, the ecological model of risk and protective factors could be enhanced by integrating culture across different domains, such as the individual and his or her relationship to community and society. Understanding the role of culture will require a deeper, broader understanding and could include research, such as ethnography of people and their encounters with violence as both victims and perpetrators. In particular, more information is needed about adolescents and their view of the cultural legitimacy of their frustration, anger, and violence, as well as ethnographic research on the culture of law enforcement and cultural assumptions, toward the goal of more reciprocal engagement.

For effective community intervention programs, Jenkins emphasized a need to account for the cultural aspects of the relationship between mental illness and violence, particularly in incorporating the perspective and expertise of people with mental illness, and to acknowledge the limitations of psychopharmacology as treatment. She also considered the importance of the school setting in teaching what constitutes a culture of violence. Finally, she closed by challenging the culture of scarcity as an excuse for failing to adequately support mental health programs.

Global Perspectives of Mental Health⁵

Dévora Kestel challenged the notion that further research on mental illness is needed, stating that she wished it were possible to implement even one-third of what is known. Instead, research on services that are intervention- and action-oriented is more important for those who want

___________________

⁴ This section summarizes information presented by Janis Jenkins, University of California, San Diego.

⁵ This section summarizes information presented by Dévora Kestel, Pan American Health Organization.

to change current practice. She remarked that, on the implementation side, she would like to see more linkage of evidence-based practices and practice-based evidence, as well as increased dissemination of those practices that have “worked enough.”

She questioned whether deinstitutionalization had worked, as the idea of “putting people away” has not changed in the United States or elsewhere. People with mental illness are not receiving the community-based care they need, but instead still end up in institutions, whether psychiatric hospitals or prisons or others. She argued that this is a priority policy direction that needs greater uptake in all mental health disciplines. A comprehensive network of community-based mental health services, she opined, is the best way to prevent violence.

Influence of Violence on Mental Health⁶

James Mercy emphasized the importance of addressing the intersection of suicide and interpersonal violence, noting that one is not more important than the other, because several factors related to each overlap. He also pointed out that while suicide might result in greater mortality, the morbidity related to interpersonal violence should not be overlooked. Evidence is emerging that indicates the long-term effects of interpersonal violence, including chronic disease. It also has a strong impact on mental health: Exposure to violence in childhood is responsible for 30 percent of adult psychopathology (Kessler et al., 2010). Thus, he noted, it is important to look not only at the influence of mental health on violence but also at the influence of violence on mental health.

Mercy remarked that there are effective treatments to mitigate the effects of exposure to violence, but it is an area that needs more research into implementation and dissemination, particularly for scaling up in low- and middle-income countries. He also raised the question of whether these treatments could be considered primary prevention, since they might in the long run reduce interpersonal and self-directed violence.

Further research is needed, he concluded, in gathering better data linking mental health and the means of perpetrating violence and in what works in preventing violence related to mental illness, such as physician counseling or background checks.

___________________

⁶ This section summarizes information presented by James Mercy, Centers for Disease Control and Prevention.

Brain, Behavior, and Targeted Interventions⁷

James Blair remarked that while another speaker had mentioned that population-based interventions achieve greater “bang for the buck,” he believed there was an important role for individual-based interventions, as well. He cited an example of a school-based anti-bullying program in the United Kingdom that saw a reduction in rates of bullying, except in one group. The individuals in this group, he noted, are better suited for specific interventions to change underlying behavior; although screening tools are imperfect, they are somewhat useful for identifying these individuals.

Importantly, do current interventions actually work for these individuals? he queried. While some individuals do see a reduction in aggression with certain interventions, such as cognitive behavioral therapy (CBT), the neurobiological mechanisms are not clear. Blair commented that research to illuminate this mechanism more clearly, coupled with the overt behavioral change, would be important support in calling for additional resource allocation for these interventions. At the same time, there are individuals who do not respond as well to these interventions, so developing new interventions for them is an important priority.

Discussion

To close the workshop, participants shared their perspectives on the discussions over the 2 days and presented their thoughts on research, program implementation, and policy. Important questions were raised on:

• How to create constructive and ongoing collaboration, particularly among those with opposing political aims, that works toward a shared goal.
• How to mainstream some of the nontraditional approaches raised, such as therapeutic jurisprudence.
• How to end programs that do not work, expand those that do, and allocate resources for them.
• How to operationalize cross-cultural situational analysis, such as the role of science in policy, attitudes of policymakers, and incentivization of native providers, in developing countries.
• How to better adapt program evaluation designs that are truly fit for the purpose.
• How to create public buy-in to invest in dissemination of successful programs and program evaluation.

___________________

⁷ This section summarizes information presented by James Blair, National Institute of Mental Health.

• How to harmonize data and data systems so that researchers who work from different angles can be more aware of evidence-based practices and instruments and access common data elements.

Participants shared additional lessons or approaches learned from their own experiences:

• Individuals at the highest risk of violence are also at the highest risk of re-offending. One participant remarked that the best intervention for this group might be to reach them at a young age; in particular, addressing the earliest stages of behavior through a better understanding of the precursors of both mental health and violence.
• Similarly, another participant noted that trauma early in life leads to sequela through the life span. Preventing that trauma could lead to a different developmental arc for those individuals.
• At the same time, treatments for posttraumatic stress disorder (PTSD) and other illnesses, such as attention deficit hyperactivity disorder (ADHD), do have measurable effects in the brain, so even those in the highest-risk groups for violence can be treated, even later in life. In particular, if treatment prevents reoccurrence of violence, it could be considered a form of prevention.
• A few participants discussed the potential for addressing the fear response and reactivity. There are several treatments, such as CBT, that reduce threat sensitivity and responsiveness in people with PTSD. These have implications for similar interventions in people with high levels of aggression.

REFERENCES