As Murry emphasized in her keynote address, the selection, adaptation, and implementation of evidence-based programs work best when they emerge from a strong partnership among researchers, program developers, and communities. The first panel of the workshop examined five different approaches to improving practice as exemplars of how such partnerships have played out in specific communities. From these case studies, the presenters drew broader lessons about how communities and program developers can best implement and adapt programs to specific contexts. They also touched upon some of the issues discussed in subsequent panels, including the sustainability of programs and the nature of the relationships among stakeholders.
THE SECOND DECADE PROJECT
In public health, communities tend to focus on specific issues, observed Patrick O’Carroll, health administrator for Region X of the U.S. Public Health Service. One city might focus on bullying, another on obesity, another on active living and good eating, and another on drug abuse. Yet, in all communities, young people in the second decade of life establish many health-promoting or health-damaging behaviors that enormously influence their long-term health status. At the same time, public health administrators have excellent guidance on a wide range of issues, including the ones that dominate local communities. The underlying challenge is to integrate all of this guidance into a framework that is comprehensive, simple, and easy to use.
The Second Decade project, which was launched by the leadership of the U.S. Department of Health and Human Services in Region X, targets community leaders who are champions for adolescent well-being in their communities.1 These leaders could be mayors, city council members, agency heads, school board members, parents, leaders of community- or faith-based organizations, or many other individuals. The key is that they are in a position to assemble a broad coalition in a community that can be engaged to improve adolescent health and well-being.
To characterize these leaders and their needs, the project conducted more than 30 interviews validating the problem and asking for input. These interviews revealed that what community leaders need is guidance that is simple and action oriented. This guidance must incorporate broad multi-impact measures, the interviews showed, rather than a long list of separate programs.
The findings of the interviews served as a guide for communities to
___________________
1 For more information on the project, see http://sites.nationalacademies.org/cs/groups/dbassesite/documents/webpage/dbasse_172943.pdf [May 2017].
develop an architecture for prevention, O’Carroll said, with coalitions serving as the foundation. The project then adds cross-cutting approaches in four areas—the built environment, school-based health centers, parent engagement, and health and safety zones—along with issue-specific measures in such areas as smoking, teen pregnancy, nutrition, activity, relationships, violence, drug and alcohol use, and auto safety. For example, a particular group might work on an anti-smoking campaign through excise taxes, cessation services, and tobacco-free campuses. At the same time, cross-cutting approaches, such as parent engagement, a school-based health center, and changes in the environment, could contribute to the campaign while simultaneously affecting other issues. The overall concept is to develop “a broad community plan to develop a place . . . that is going to be healthy for adolescents.”
The final evaluation draft of the community guide was completed in the summer of 2015, and five pilot test sites were selected from candidate communities across the country. An evaluation is currently under way in these sites to assess the usefulness of the guide in, first, assembling and establishing a broad and inclusive coalition, and, second, developing an appropriate, actionable plan that incorporates proven multi-impact measures. “Is it at the level that makes sense to [communities]?” asked O’Carroll. “Can they produce a plan within a year that has those multi-impact measures?”
BRIGHT STAR COMMUNITY OUTREACH
Since the beginning of 2012, more than 2,100 people have been murdered in Chicago. “Who does the post-trauma counseling for those families, whether the victim’s family or the perpetrator’s family?” asked Christopher Harris, pastor of Bright Star Church in Chicago and founder of the Bright Star Community Outreach project. “In most cases, nobody.”
In urban communities, African Americans, Hispanics, and other community members tend not to go to counseling. They may not know or trust counselors, they may not be able to afford it, and they do not want to be labeled. But trauma “is not just a black or brown problem,” said Harris. “It’s a human problem.”
Through the Bronzeville Dream Center, Bright Star Community Outreach is developing five core competencies to bring effective, sustainable change to Chicago: counseling, mentoring, parenting, workforce development, and advocacy.2 Bright Star Community Outreach also promotes what Harris called the four C’s: concentration, communication, collaboration, and compassion. Concentration focuses attention on particular communi-
___________________
2 More information about the program is available at http://brightstarcommunityoutreach.com/bronzeville-dream-center [May 2017].
ties and the services that need to be delivered. Communication ensures that the community understands what the project wants to do not for them but with them. Collaboration brings all of the partners to the table to develop an authentic community effort with a collective impact. Compassion unites stakeholders so that they can sit at the same table and address common problems.
In urban communities, churches are bedrock institutions, Harris said. Bright Star Community Outreach is therefore identifying, training, and certifying faith leaders to do post-trauma counseling. More than 40 faith leaders have committed to training in a model developed at a post-trauma counseling center in Israel known as NATAL. The Bronzeville Dream Center has also used the Communities That Care model to develop a framework from risk and protective factors assessed in the community. Multiple workgroups are addressing training, care coordination, and network development.
Researchers need to bring their heads, their hands, and their hearts to the table to work on these issues, Harris concluded. They need to see the families who need counseling as their families, he said, and they need to include the people whom they want to help in the development of these programs. Community leaders in Bronzeville trusted Bright Star Community Outreach as a partner. “The reason our community outreach had so much buy-in is because they trusted us to lead the effort,” said Harris. The community also expected the organization to act as a broker between researchers and the community, which required learning information and speaking a language it had never spoken before. “People don’t care how much you know until they know how much you care,” Harris said. “That’s why we’re having the success that we’re having.”
PROMOTING SCHOOL-COMMUNITY-UNIVERSITY PARTNERSHIPS TO ENHANCE RESILIENCE (PROSPER)
As indicated by its name, partnerships also are at the heart of the PROmoting School-community-university Partnerships to Enhance Resilience (PROSPER) project, said Richard Spoth, the F. Wendell Miller senior prevention scientist and director of the Partnerships in Prevention Science Institute at Iowa State University.3 The partnership creates a network of individual sites, including community teams designed to implement and sustain programs in the community, a prevention coordinator team to link communities to the extension system and provide technical assistance, and a state coordinator/management team to coordinate technical assistance and provide guidance and ongoing support. “The quintessence of PROSPER,” said Spoth, “is effective, sustainable partnerships.”
___________________
3 For more information about the program, see http://helpingkidsprosper.org [May 2017].
The first goal of the project is to sustain evidence-based programs with high quality to prevent substance misuse and other problem behaviors among youth. The objectives of the community teams include planning and coordinating family programs, including recruitment and monitoring for quality; working with schools to coordinate a school program, also encompassing monitoring for quality; and generating resources for ongoing programming.
The second goal is to build and maintain well-functioning and productive teams. In this regard, the objectives of the community teams include holding regular, effective meetings, maintaining an active membership, building connections with school and community organizations, engaging in strategic communication throughout the community to promote awareness of the project’s efforts, and recognizing and rewarding supporters and contributors.
Spoth reported that community teams have achieved high recruitment rates for family program participation, compared to traditional approaches, and that all programs have been implemented with high levels of quality. Social network analyses have indicated reductions in negative peer influences, with additional positive effects for strengthening family relationships, parenting, and youth skill outcomes. Youth have scored significantly lower on a range of problem behavior outcomes, including substance misuse and conduct problems (Spoth et al., 2013a, 2015). In addition, implementation of the family program is more cost efficient through a PROSPER partnership, and PROSPER is cost effective and cost beneficial overall. As an example of PROSPER’s effectiveness, Spoth et al. (2017) found that for every 100 young people who likely will misuse prescription drugs long term in non-PROSPER communities, 20 to 26 fewer would be expected to do so in PROSPER communities.
PROSPER fosters linkages with existing infrastructures; the organization of sustainable community teams; proactive, sustainability-oriented technical assistance focusing on benchmarking and web-based support; and capacity-building focused on critical tasks, including recruitment, implementation quality, and sustainability. It has educated and trained partnership members about the importance of quality monitoring and related strategies. It also has assessed benchmarked progress across all phases, with special attention to core components. As a result, the program has had very high, long-term adherence ratings (Spoth et al., 2011).
PROSPER has a wide variety of tools that help partners meet objectives. For example, technical assistance from prevention coordinators, including guidance with the application of a Web-based resource tracker, has helped project communities raise an average of $23,000 per academic year in in-kind and cash contributions (Spoth and Greenberg, 2011).
Spoth drew several overarching lessons from his experiences with PROSPER. Due to the degree of complex systems change required, considerable resources must be devoted in formative stages to ensure that system change barriers are addressed quickly, he said. Assessing readiness, adoption support, implementation capacity-building, and well-functioning implementation staff are key factors for success. With effective systems-level supports for program implementation, program-level adaptations compromising quality are less of an issue, he said.
He also described several possible courses of action to engage in what he called meta capacity-building (Spoth and Greenberg, 2011; Spoth et al., 2013b). Networked prevention systems could be strengthened by building on existing infrastructures, learning from existing implementation systems research, and linking with health care reform efforts such as the Community Benefit program. The prevention workforce could be strengthened by building out currently available training/certification systems and by organizing a network of university-supported trainers. And, he suggested, sustainable funding mechanisms could be expanded through private-public partnerships linked with integrated preventive health homes and through the use of prevention and wellness funds to support networked communities.4
THRIVENYC
In 2015, the leaders of New York City came together to unveil the mental health plan ThriveNYC.5 Gary Belkin, executive deputy commissioner of mental hygiene in the New York City Department of Health and Mental Hygiene, described the plan as ambitious in scope, rigorous in how it defined the problem, expansive in addressing how mental health issues affect all individuals and institutions in society as a broad public health challenge, and comprehensive in calling on all sectors and all citizens to address the issue.
Based on six broad principles—changing the culture, acting early, closing treatment gaps, partnering with communities, using data better, and positioning government to lead—ThriveNYC consists of more than 50 new initiatives funded at close to $1 billion. It is training a quarter of a million New Yorkers in mental health first aid. It is bringing together 29 hospital systems, which collectively are responsible for 80 percent of the live births in New York City, to achieve universal screening and connection to care for pre- and postpartum women with depression. A faith weekend united more than 1,000 houses of worship on the theme of mental health.
___________________
4 For more information on integrated preventive health homes, see http://www.integration.samhsa.gov/integrated-care-models/health-homes [May 2017].
5 For more information on ThriveNYC, see http://www1.nyc.gov/nyc-resources/thrivenyc.page [May 2017].
A multiagency effort aims to spread socioemotional learning skills to all public prekindergarten and early day care provider staff, who collectively reach 100,000 children each year. A new Mental Health Services Corps is putting 400 clinicians in high-need parts of the city to bring best practices into those settings and to move the system in more innovative directions. “All told, these initiatives will touch, if we do it well, on the order of a million New Yorkers,” Belkin said.
ThriveNYC poses a major implementation and evaluation challenge, Belkin observed. Meeting that challenge means “allowing ourselves to embrace the variation, to embrace the fragmentation . . . to purposefully diversify ownership of these best practices.” As the components of the project go to scale, necessary learning happens only in the context of variation, driving many opportunities for hypothesis testing, Belkin observed. In this way, continuous quality improvement is a far better fit to the science of implementation and can be hardwired into the project. Indeed, one of the ThriveNYC initiatives is establishment of a Mental Health Innovation Laboratory in the New York City Department of Health and Mental Hygiene to serve as a source of improvement and implementation technical assistance to people who want to innovate. Similarly, the Early Years Collaborative is applying learning collaborative models to whole neighborhoods to improve early childhood outcomes. “The hypothesis testers are the communities, and we’re building their capacity to do that,” said Belkin. “If they aren’t partners, or leaders of testing in real time iteratively, then we can’t scale.”
Partnering with universities has been frustrating, he noted. Faculty members and their leaders tend to value, and be rewarded to value, a narrow set of practices and ideas regarding how knowledge is generated—“controlled science,” as Belkin described it—that are remote to the problems and questions posed by scaled implementation, or “improvement science.” ThriveNYC is focused on immediate capacity-building and testing at the local level, which differs from the typical academic model. “We need to reward academics differently,” he said. “We need to design grants differently, so they’re capacity builders, so the people using these programs become smart implementers.”
Belkin also advocated diffusing ownership, even if it means that practices sometimes will divert from the ideal. In addition, best practices need to be packaged in such a way that they can be taken up by diffusion engines to reach the scale needed, he observed, stating, “We need to be bold in task shifting, in having nonspecialists do the steps in these programs that nonspecialists can do.” Belkin argued that more research needs to focus on interventions that do not need to go through, or only go through, the behavioral health system. That system and those experts do not have the reach required to meet needs, he said. As a result, ThriveNYC invests in interventions that rest on the larger health system, schools, workplaces, nonhealth
human services, social networks, and other systems, with behavioral health partners holding important referral, coaching, and enabling roles. A public health approach to mental health needs to be fit into larger policy agendas and enlist other partners and diffusion pathways, he said.
IMPLEMENTING EVIDENCE-BASED MODELS IN THE CHILD WELFARE AND JUVENILE JUSTICE SETTINGS
Since 2006, New York City has had a policy of investing in evidence-based interventions in its child welfare and juvenile justice systems. “What we were doing was not working,” said Gladys Carrión, commissioner of the New York City Administration for Children’s Services. “We wanted to improve outcomes.”
One result of this policy has been major declines in foster care placements and juvenile justice placements. “We’ve done that through heavily investing in prevention,” said Carrión. The focus has been on reducing the rate of maltreatment, reducing out-of-home placements, and improving well-being. The city spends more than $80 million on evidence-based models annually, and the amount has been growing every year. It serves thousands of families in a variety of systems. Running such a large and complex system “is not for the faint of heart,” she remarked.
The general approach taken has been to embed evidence-based models in the many community-based organizations that have city contracts and case management responsibility. “Communities know their families and their children best,” said Carrión. Focus groups and data mining have revealed who is coming into the child welfare and juvenile justice systems, why they are entering those systems, and what issues they have. Implementation science guides the work of both the city and community-based organizations. “If you speak to any of our partners, they’re very well versed,” said Carrión. “They’re able to understand what these models are, what they do, and how implementation and fidelity issues are addressed.”
Carrión observed that good working relationships with program developers and implementers have made it possible to make adjustments to reflect differing contexts and needs. “For the most part, we have been very successful in addressing what the challenges are in implementation, identifying what is an adjustment and what is an adaptation, and working to develop those adaptations that we need,” she said. For instance, evaluations associated with the use of evidence-based models revealed an increase in the number of very young children coming into the child welfare system. Identifying this trend allowed responses to be developed that would address the issue. Research and data are needed to know which are the best models for a given system and circumstance, she noted. Long-term relationships with program developers then help ensure that models are working and effective.
The drivers of implementation are competency, leadership, and organization, according to Carrión. Support from the “very top” is critical, as is collaboration, given the many challenges. The city does assessments of programs and looks at outcomes, but it does not have the capacity to do rigorous and continuous evaluations of programs. In addition, she said, the city has largely failed to document its work in such a way that lessons can be shared. The changing demographics of the city, variety of languages and cultures, and fiscal constraints all complicate determinations of how programs are working. Questions posed by Carrión include the following: How do fidelity measures and scoring correlate with outcomes? How can a common understanding of fidelity be created to assess the impact of multiple evidence-based models? How can models be adjusted to serve diverse communities?
“Despite those challenges, we have seen tremendous improvements in our outcomes, the reduction of repeat maltreatment, the reduction of children coming into our system, the shortening of length of stays, and improvement in well-being,” Carrión concluded—all in a system that does more than 55,000 investigations of abuse and neglect each year.
DISCUSSION
Aligning Diverse Programs
The challenge posed by aligning diverse programs was one focus of the question-and-answer period. Spoth, for example, called attention to the need for integrated systems change. “If you effectively integrate systems, you adopt a larger perspective,” he said, noting national attention has been focused on the reform of health care systems, which offers an opportunity to take a broader perspective that encompasses many systems.
Spoth also pointed to the availability of good guidance on working with developers and implementers across programs. In addition, he said that his programs, because of proactive technical assistance work, have had less of an issue maintaining fidelity. A strong support system enables communities to maintain fidelity rather than changing programs in ways that make them less effective or less amenable to evaluation.
Belkin emphasized the importance of aligning policy and programs. In New York City, the establishment of mental health counseling by the mayor brought together “a couple dozen city agencies,” he said. “That changes conversations and ways of looking at things.” People in public health spend a lot of time looking at interventions, but they spend less time thinking about policy, he said.
Working with Communities to Evaluate Programs
Another topic was the need for communities to understand that programs must be evaluated. “That’s the only time you get accountability,” said Harris. Community-based and faith-based organizations may not appreciate that they cannot ask for dollars without providing data. Yet evaluators also need to avoid destroying the morale of program leaders by insisting that a program is misguided, Harris observed. “From a communications perspective, tell them that there’s a way for them to do it better. And then enable them to do better, educate them to do better, and empower and release them to do better,” Harris suggested. Communities also may evaluate programs in a different way than would an evaluator. “That’s where you need to build the bridge to bring those two together,” he noted.
Belkin observed that a more nuanced and pluralistic understanding of efficacy and knowledge may be needed. He suggested asking what sorts of questions need data from controlled studies, and what sorts of questions need data from iterative improvements.
Coping with Change
The panelists also spoke to the issue of changes in the community that affect their programs, such as new leaders in schools, government, and community organizations. As Spoth said, “It has been the bane of our existence.” The turnover of personnel is less rapid in a rural community than an urban center, but it nevertheless occurs. Technical assistance and other support systems can orient new people and sustain relationships, he said.
Harris pointed to the need for a bottom-up rather than a top-down approach to change. People with needs in a community will always be there. New leaders therefore must learn quickly what needs exist. “Speaking truth to power is the most important thing,” he stated. This often requires that researchers and program developers speak out in ways that community leaders can understand. He commented, “If you look at many of the brochures that come out from many of our institutions, grandmama can’t read that. But if you put it in words that grandmama can articulate, then it changes the whole trajectory of how the communication is set forth.”
Change is relentless and inherent, Belkin said. But steps can be taken to foster different habits of community engagement and planning, regardless of personnel turnover. In this way, improvement can become continuous rather than episodic.
