Despite the scale-up successes reported at the workshop and the recent expansion of family-focused prevention programs into new settings, the potential of many evidence-based interventions is often not fully realized, several speakers pointed out. “Frequently when these interventions are implemented, they are not implemented with quality, or that quality is not sustained over time, which is tantamount to having limited impact,” Spoth said. He and other participants at the workshop pointed out numerous challenges that need to be addressed when scaling up programs. These challenges include
- Lack of demand for the programs,
- Insufficient organizational capacity,
- Lack of sustainable funding, and
- Factors other than evidence from research that influence decision making around whether or not to implement a particular program.
Scaling up can also be hampered by an over-reliance on program developers who do not have the expertise or time to scale-up and disseminate their programs, and rigid adherence to the programs which may need to be adapted to specific populations or organizations. McCannon identified additional factors that may impede efforts to successfully scale-up a program, including attempting to scale-up too quickly and a lack of explicit goals and objectives for what needs to be accomplished within a specific timeframe. Each of these scale-up challenges is explored in more detail below, with strategies for meeting those challenges appearing in the next chapter.
Several participants noted that a major impediment to scale-up of programs is a lack of demand for them by organizations and the communities they aim to help. “We have the medicine, but the patients don’t come,” Terje Ogden said. He noted that 3.5 percent of children in Norway have a conduct disorder, but only 0.4 percent receive treatment for it (Skogen and Torvik, 2013). The lack of demand for effective family-focused programs can be caused by inadequate marketing and referrals, as well as a lack of a clearinghouse for interventions shown to work.
There is no clear hub for communities to find evidence-based programs and know what it will take to implement them well, said Lauren Supplee, Director of Family Strengthening at the Administration on Children and Families. Kathy Stack of the U.S. Office of Management and Budget pointed out that clearinghouses such as the What Works Clearinghouse provided by the Institute of Education Sciences1 could be more robust so they show not only what works but what is needed as far as implementation and technical assistance.
MaryBeth Musumeci from Kaiser Commission on Medicaid and the Uninsured noted that Medicaid enables states to offer a wide array of services aimed at improving children’s mental, emotional, and behavioral health, and is being expanded in several states with the implementation of the Patient Protection and Affordable Care Act, but a number of children who are eligible for those services are not enrolled in them. “States need to think creatively about how to take up some of these new options,” she said.
Insufficient demand for programs or their lack of adoption by organizations, providers, and policy makers can also be caused by a lack of data showing what interventions are needed for specific communities and would be feasible for them to adopt. Often a wide array of programs are available, and it is difficult for states and communities to choose the one best suited to their needs without data that assess those needs, Brian Bumbarger noted. Supplee pointed out that states and communities vary widely in their capacity to collect useful administrative data or access national survey data at a meaningful level so as to assess the need for certain programs. Data on the rates of child abuse or delinquency in specific communities is needed, for example, when considering whether to adopt programs aimed at lowering these rates. Supplee noted that some federally funded programs require states to provide such data before they can receive funding for them. Clarece Holden added that Congress will not fund Substance Abuse and Mental Health Services Administration and other agencies’ prevention
________________
1Details about the clearinghouse may be found at http://ies.ed.gov/ncee/wwc (accessed July 28, 2014).
programs unless data showing the need for the programs, their effectiveness, and their cost are provided. “So much of building the evidence could be done more cheaply if we can improve access to administrative data and create much more efficient processes for matching individual data across systems,” Stack said.
However, as Bumbarger pointed out, there is currently no data infrastructure to help communities understand their needs or to continuously monitor progress and quality improvement. He noted that a prevention planning framework called Communities That Care helps communities collect local data so they can identify and prioritize specific risk and protective factors they might want to target and match those to suitable interventions. But these types of frameworks are only being applied in a few states and communities and need to be scaled up to serve a wider arena, Supplee said. Bumbarger emphasized there is a national need for community-level infrastructure for epidemiological surveillance on an ongoing basis, which was delineated in a recent publication by the Society for Prevention Research (Mrazek et al., 2004).
Once a program has been chosen it has to be marketed so the constituents it targets use the program. The principles of a dissemination support system espoused by Kreuter and Bernhardt (2008) and detailed at the workshop by Supplee are that once users review programs for their feasibility in their communities, they engage experts in design and marketing who can package the program so it is both workable and appealing. Then users should have dissemination agents spread the adoption of the model program by talking to people about why it might work for them. Supplee also suggested there be feedback loops so communities can feed their information back to the developers to help them improve their program designs and efficacy so it reaches a greater population, thereby building demand for them.
Frequently program developers are asked to disseminate to a wider constituency, but they do not have the expertise or capacity to do so on a large scale. They lack the time, workforce, and knowledge to package and support scale-up of their models, Supplee noted. “Developers cannot and should not do it all. We really need to think carefully about a public health infrastructure to support scale-up much like private industry,” she emphasized. Bumbarger added, “It’s a herculean task and we cannot continue to put all of this responsibility and burden for scale-up at the feet of the developers. It’s just not realistic.”
INSUFFICIENT ORGANIZATIONAL CAPACITY
Supplee noted that evidence is mounting that the capacity of the servicing agency is key to programs being successful. “A lot of the models
are only as good as the communities and referral systems they are placed in,” she said. For example, one study that randomized community-based mental health programs for youth to either intense technical assistance around organizational capacity or control conditions found that outcomes for youth in the programs receiving technical assistance were significantly better (Glisson et al., 2013). Building organizational capacity and ensuring staff are trained in the areas that match the services sought are variables that deserve our attention, she said.
Carolyn Webster-Stratton added that a major barrier to the successful implementation of The Incredible Years® program was inadequate agency readiness, including a lack of short- or long-term goals clearly mapped out, failure to select motivated clinicians with the expertise or training to deliver the program, and inadequate recruitment and engagement with families, including improper handling of the logistics such as day care for the children while the parents are in groups, providing meals, and scheduling of the programs.
Many speakers noted that a lack of sustainable funding is a major impediment to scaling up interventions. “Prevention gets done by community organizations and nonprofits and mom and pop organizations, and it is all funded by this patchwork quilt of yard and bake sales, car washes, and temporary grants. Sustainability is really a big challenge to making that leap from knowing what works to having a population-level public health impact,” Bumbarger said. Supplee pointed out that grants are often for only 5 years, but it takes almost that long for many organizations to hit their stride, as far as implementation is concerned, and reach a steady state. At that point their funding runs out. Webster-Stratton agreed and pointed out that organizations often are given early grant or foundation funding, “But when the funding ends, the program dies because the funding is not sustained.”
Supplee added that often funding is stymied by reimbursement restrictions, such that some but not all of the services a specific program provides will be reimbursed. An example Webster-Stratton gave is that individual therapy may be reimbursable but not group therapy. Kimberly Hoagwood added, “These clinics are in deep trouble in many cases because they are not able to capture the funds for which they are billing.” In addition there can be conflict over what the organization wants to provide and what federal and state governments wish to fund, which can complicate implementation, especially since many organizations rely on multiple sources of funding. “One funding stream wants them to focus on birth outcomes, the other funding stream wants to focus on infant health outcomes or maternal health
outcomes and they are feeling very pulled. There is a triangle between the developer, the program administrator, and the funder,” Supplee said.
Inadequate funds to provide for clinician training or consultation by accredited mentors of clinicians is also impeding effective implementation, Webster-Stratton emphasized. Funds are also inadequate for providing the full scale of services or frequency of services, delineated in original programs, or the add-on components designed for specific populations, such as families on welfare. Thomas Dishion pointed out that funding and sustainability has been challenging for the Family Check-Up program in all the settings in which it has been applied. “We have a couple agencies within a mile of us who do not have the money for even initial training,” he said. Most of the staff are receiving low salaries for attending to large caseloads and have no time for clinical supervision. Supervisors often are not paid to review monitoring videotapes or clinical processes, and are continually stressed by having to find funding. This fosters burnout and high staff turnover, Dishion noted.
Funding can also disappear when administrative or political regimes change or when there are state or nationwide budget crises, several speakers noted. Politically motivated compromises on funding can also hamper the scale-up of programs, David Olds noted. “There is a fundamental question about to what degree evidence-based programs can be built on compromise,” he said.
Another major frustration cited by participants is that funding is siloed in various agencies, requiring a cobbling together of funds. Ron Prinz pointed out that the Triple P-Positive Parenting Program can be funded by and applied in a number of different sectors, including health care systems, educational settings, and the juvenile justice system. “You don’t want it bottled up or controlled by one sector or agency,” he said. But as Hoagwood noted there is no single state or federal agency focused on prevention in children, which has a small piece of many governmental mental health agencies. Supplee added, “We need to think about how we build an implementation infrastructure that is useful across disciplines and contexts. It is very siloed right now. How can we maximize our investments by building capacity across contexts?”
A lack of funding can also be attributable to a lack of priority for child prevention programs combined with limited resources. One study found that states invest in two to six times more programs and services for adults than for children (NRI, 2012), and the current fiscal crisis has dampened funding overall. Hoagwood noted that three-quarters of states are facing severe budget crises and are responding by reducing community mental health services and grants (Lutterman and NASMHPD Research Institute, 2012).
Joe McCannon, consultant to The Bill & Melinda Gates Foundation, emphasized that at the same time it can be counterproductive to obsess
about payment as the sole lever that will fuel implementation. “Don’t just look at payment, but also look at recognition, collaboration, transparency, and regulation on a rolling basis and go to where the opportunity lies. You need to take every opportunity that’s presented to you and budget for flexibility,” he said.
FACTORS INFLUENCING DECISION MAKING
Scientific evidence is just one tool policy makers use when deciding what programs to scale-up and fund. Supplee noted that, besides evidence from research, decision makers use administrative data, anecdote, politics, and other types of information in making their decisions. In the book Using Evidence: How Research Can Inform Public Services, Nutley and colleagues (2007) discuss different ways evidence informs social policy and note that the direct use of findings from research (i.e., this one study led to a specific policy) when making policies is rare. More frequently there is indirect use of evidence, such as that gleaned from a body of research including impact, epidemiological, or developmental studies but not one specific study. Nutley and colleagues also note that evidence is sometimes used in a political way such that the policy maker cherry picks findings to support a decision, Supplee said. Stack added that a lot of policy and decision making “gets made based on gut intuition. Every new administration comes in with their ideas on what is going to work,” she said.
Emotional as well as logistical factors also influence decision making, both Supplee and McCannon pointed out. For example, an educational administrator may choose a textbook because it has a more appealing cover, even though the evidence base behind another textbook is stronger. McCannon emphasized that an initiative that tries to root itself purely in evidence and reason may fail because it ignores the emotional factors that can enter into decision making that can be more compelling. He gave an example of an initiative to help the homeless that relied on a success story of a homeless man who overcame several obstacles. This story countered policy makers’ resistance to fund a program for the homeless, who they tended to view as hopeless cases. “Tying back to these stories and emotions is crucial because it is where we get our energy to instill change,” he said.
But there are also rational reasons for not leaning too heavily on evidence from research when making policies. Even well-conducted studies can offer limited evidence because it has not been replicated or validated in a real-world setting, Supplee noted. “We hear from states and communities, ‘Is this program going to work for my population?’ and we say ‘We’re not sure,’” she said.
Replication of findings can be difficult to do because programs often evolve by adapting to local circumstances, so a rigid replication study in
which the same exact program as the original is tested may not be appropriate. “The odds are great that if you’re going to do a randomized controlled trial of a whole program, then you are going to find no impact because it does not allow you to look at the variations and find the things that work,” said Stack. In addition, “There is little empirical knowledge on core components, so it is hard for us to say what can be changed and adapted to specific populations,” Supplee said. Factors that moderate impacts are often not known and can pose problems when scaling an intervention in different populations in different communities, she said.
Several participants noted that programs have to adapt and evolve to meet the unique needs of program implementers and a major barrier can be rigid requirements that do not allow the flexibility to mold programs to specific populations and settings. Mr. McCannon, in his presentation on lessons from other fields and sectors on effective spread and scale-up, noted that one lesson learned from efforts to broadly implement practices to improve health care quality is that “Adaptation is what you see in initiatives that are really thriving” (McCannon and Perla, 2009). They are able to agree on the core science and not doing anything that corrupts it, but they make sure they allow for adaptation in every setting, in every clinic, in every environment in which a program is implemented.”
Often developers and implementers feel the need to control and maintain fidelity to interventions when they are applied more broadly, but McCannon said, “I think you will find that when you are really successful in a large-scale change initiative, you have lost control of it.” He added that when he visited a successful program in Sweden and asked how they were able to expand so effectively they responded, “We’re jazz players—we have a core theme, which is the intervention or the practice, but we improvise constantly.” McCannon added that “Contamination may not be such a bad thing—you want people to be cross-pollinating and learning from each other.”
Sometimes programs are scaled up too quickly before they have been adequately adapted and optimized for specific delivery settings, and before capacity has been built in those settings. Olds noted, “I was convinced by leaders in one state to develop the Nurse–Family Partnership (NFP) there overnight, which I’ve regretted because the level of local ownership and commitment to developing the program well was never built. The result was that the quality of program implementation in that particular state really suffers.” Bumbarger added, “Often as soon as a program demonstrates some effectiveness there is a desire among policy makers, and in some cases practitioners and providers, to get that program out into the
field before it has been optimized, which is not the most efficient way to scale-up.”
McCannon suggested what he called “the rule of 5X to 10X, which means that if 10 organizations were involved in the first phase of scale-up, it is appropriate to expand to no more than between 50 and 100 organizations in the next phase. The 5X to 10X rule seems to be a good indicator for those initiatives that do especially well,” he said. He added that it was critical that scale-up have specific goals delineated with deadlines for specific benchmarks to be accomplished. A major pitfall is when “you can’t say explicitly what you are trying to accomplish and the date by which you are trying to accomplish it,” he said. Olds noted that one reason the NFP has been so successful is because it has clearly elucidated its goals, methods, and objectives.
Glisson, C., A. Hemmelgarn, P. Green, and N. J. Williams. 2013. Randomized trial of the Availability, Responsiveness and Continuity (ARC) organizational intervention for improving youth outcomes in community mental health programs. Journal of the American Academy of Child and Adolescent Psychiatry 52(5):493-500.
Kreuter, M. W., and J. M. Bernhardt. 2008. Reframing the dissemination challenge: A marketing and distribution perspective. Commentary. American Journal of Public Health 99(12):2123-2127.
Lutterman, T., and NASMHPD Research Institute. 2012. Impact of the state fiscal crisis on state mental health systems, Winter 2011-2012 update. Preliminary results based on 41 states reporting. http://media.wix.com/ugd/186708_c2fd199b2a9f4d04818b889b93c3a884.pdf (accessed June 6, 2014).
McCannon, C. J., and R. J. Perla. 2009. Learning networks for sustainable, large-scale improvement. Joint Commission Journal on Quality and Patient Safety 35(5):286-291.
Mrazek, P., A. Biglan, and J. D. Hawkins. 2004. Community-monitoring systems: Tracking and improving the well-being of America’s children and adolescents. http://www.preventionresearch.org/CMSbook.pdf (accessed June 4, 2014).
NRI (National Research Institute). 2012. Analysis of data from NRI State Profile System, 2012. http://www.nri-inc.org/#!data-sets/czyr (accessed September 9, 2014).
Nutley, S. M., I. Walter, and H. T. O. Davies. 2007. Using evidence: How research can inform public services. Bristol, UK: The Policy Press.
Skogen, J. C., and F. A. Torvic. 2013. Conduct disorders among children and youth in Norway: Estimated prevalence and the use of interventions. Report 2013:4. The Norwegian National Institute of Public Health.
