5

Lessons Learned: Evaluating Community- and Hospital-Based Models and Initiatives

The workshop’s third set of presentations examined some of the lessons learned from programs directed at curbing violence. Daniel Webster, professor of health and policy management at the Johns Hopkins Bloomberg School of Public Health, director of the Johns Hopkins Center for Gun Policy and Research, and deputy director for research for the Johns Hopkins Center for the Prevention of Youth Violence, first described the influence of Baltimore’s Safe Streets Program. Jeffrey Butts, director of the Research and Evaluation Center at John Jay College of Criminal Justice, then reviewed his research team’s evaluation of New York’s Cure Violence program, and Thea James, vice president of mission and associate chief medical officer at the Boston Medical Center, discussed the implementation and expansion of a hospital-based violence intervention advocacy program. Following the three presentations, Webster moderated an open discussion among the workshop participants. Highlights and main points made by individual speakers are in Box 5-1.

LESSONS LEARNED FROM BALTIMORE’S SAFE STREETS PROGRAM¹

When Webster began working in Baltimore nearly 25 years ago, the city already had a long history with gun violence. In 2006, after Webster

__________________

¹ This section is the rapporteurs’ synopsis of the presentations made by Daniel Webster, professor of health and policy management at the Johns Hopkins Bloomberg School of Pub-

and his colleagues returned from a visit to Chicago to learn more about Cure Violence (then known as Ceasefire), the city decided to try the program, and he has been involved in evaluating this program since then.

On a staggered schedule, from July 2007 to November 2008, the city started the Safe Streets Baltimore program in five neighborhoods, with two still in operation. One of these neighborhoods, known as Cherry Hill, is geographically isolated from the rest of the city, noted Webster. More recently, the city introduced the program in a west Baltimore neighbor-

__________________

lic Health, director of the Johns Hopkins Center for Gun Policy and Research, and deputy director for research for the Johns Hopkins Center for the Prevention of Youth Violence. The statements have not been endorsed or verified by the National Academies of Sciences, Engineering, and Medicine.

hood, Mondawmin, which is where the uprising began after Freddie Gray’s death; in the neighboring Sandtown area; and in the northwest Baltimore neighborhood called Lower Park Heights.

To assess how the Safe Streets program affected attitudes, Webster and his colleagues developed and validated a tool that uses common scenarios that lead to shootings based on research and information obtained from Baltimore’s youth (see Table 5-1). He noted that the neighborhoods had similar risk factors that could be easily assessed in a short on-street survey, and that after controlling for individual-level risk facts, there was a “fairly powerful deterrent effect” with regard to changing attitudes about the acceptability of using guns to resolve conflicts in a variety of circumstances (Milam et al., 2016).

The influence of the program on homicides and shootings was mixed. By May 2012, two of the neighborhoods experienced a drop in homicides, 21 percent in McElderry Park and 34 percent in Cherry Hill, but in one neighborhood, Madison East End, where the program had been running for only 18 months, the homicide rate rose by 200 percent. That is a relatively rare outcome, but that is what they found. Webster noted that non-fatal shootings fell by 41 percent in Madison East End and 34 percent in Elwood Park. In some but not all neighborhoods, there was a spillover effect in that violence decreased in surrounding neighborhoods. One interesting finding, said Webster, was that outreach workers of the type Roberto Rodríguez of Stand Up to Violence described in Chapter 4 were

TABLE 5-1 Attitudinal Question on the Legitimacy of Using Guns to Settle Disputes

SOURCE: Webster presentation, June 16, 2016.

having the same effect at reducing gun violence as was a focused deterrence approach used by law enforcement specifically targeting offenders.

His team’s most recent analysis, through December 2015, found that when aggregated across neighborhoods and duration, there was no net change in homicides associated with the program, but there was a 27 percent reduction in non-fatal shootings. The most impressive results, said Webster, occurred in the geographically isolated Cherry Hill neighborhood, where there was a 44 percent reduction in homicides and a 41 percent drop in non-fatal shootings over a 7-year span starting in November 2008 even though the resources allocated to that neighborhood were reduced. Two other neighborhoods, Lower Park Heights and McElderry Park, also saw a reduction in homicides, 38 percent and 23 percent, respectively, though the program’s inability to reduce gun violence in the Mondawmin neighborhood is of great concern, said Webster. Nonetheless, he believes this program has been the most effective action Baltimore has taken to combat gun violence, even though it has only been introduced in a handful of communities. “If you look at all of the investment in a variety of different law enforcement approaches, including focused deterrence, Safe Streets has done better,” said Webster. In fact, the focused deterrence model has so far not been effective in Baltimore and has been associated with more shootings.

In talking about evidence-based programs, whether Cure Violence or a focused deterrence approach, Webster believes that implementation matters, as does who is involved in carrying out an intervention. “When I think about what might explain some of the varied effects that we are seeing with this program, I mention neighborhood as being important in and of itself,” said Webster. “I also think there is this aspect that is very hard to quantify in a scientific way, but it is getting the right people who can do this most challenging work.”

EVALUATING THE CURE VIOLENCE PROGRAM

Before beginning their outside evaluation of the Cure Violence program, Jeffrey Butts and his colleagues collaborated with Gary Slutkin and his team at the Cure Violence headquarters in Chicago to develop a fair description of the program’s model in order to lay the groundwork for a controlled study to measure the program’s effectiveness. It was during this process that Butts noticed that Cure Violence’s approach to attitudinal changes is to work on multiple angles simultaneously, rather than just targeting the people involved in violence and forcing them to stop.

Given the importance of norm change, Butt’s team decided to measure that as well as collecting administrative data about violence in the test communities. Using and adapting some of Webster’s measures, crews

of young people with iPads went into the neighborhoods and conducted respondent-driven sampling. Although this survey technique has drawbacks with regard to the validity of the sample, said Butts, it is a good way to get a quick sample of the traditionally hard-to-reach and hard-to-recruit population of 18- to 30-year-old males. He explained that this was not a follow-up longitudinal panel study in which his team would go back and find the same people over time. “What we are doing is measuring the prevalent social attitudes among 18- to 30-year-old men in these neighborhoods and seeing if the trajectory of those norms about violence moves in a neighborhood that has Cure Violence versus one that does not,” said Butts.

One surprise, he said, was that 40 to 50 percent of the young men surveyed in some of these neighborhoods reported they had been shot at, which could mean they were standing on a corner and someone fired a gun, and that some 20 percent had been stabbed. Unsurprising was the lack of confidence among these young men in public institutions and their ability to help the neighborhood in response to violence. Police and public officials received low marks while fire and ambulance crews received more positive reviews.

When it came to measuring violence, Butts said there is no such thing as data on the incidence of violence, but rather data on proxies of violence, such as injuries reported by hospitals. In New York City, there is a dataset available containing diagnostic codes for everyone treated in the emergency department of every hospital in the city, and that is one of the sources of outcomes data that his team has been following over time. That dataset, though, does not include all crime because in many of these neighborhoods, crime-related injuries are treated at home. Another incomplete proxy is reported crime by mid-block coordinates for shots fired, assaults, robberies with a weapon, and homicides. Butts and his team are now examining the datasets, which cover 20 years, to determine if there are changes in incidents of violence that can be associated with changes in community norms.

As a short aside, Butts challenged the research community to stop arguing about whether focusing on hot spots or hot people is the better approach. “Obviously, you have to do both,” said Butts, using influenza as an analogy. “We would not say we would combat a flu epidemic just by finding the people with the flu. We would also try to teach people about how to avoid the flu.” He added that while supporters of both views are good at “cherry picking” data to defend their particular approach, the reality is that solving the problem of community violence will require all of the tools at society’s disposal. The key is developing the evidence to show that a given intervention is truly correlated with the cause of violence in a way that is reliable and replicable.

Butts cautioned that the demand for specific experimental designs as the only acceptable evidence for public policy excludes any model that is not delivered at the individual level. He added that no organization has the resources to do that type of study. If experimental design is treated as the only evidence that can be used to evaluate program effectiveness, then the only studies done to shape policy that focus on changing behavior will focus on the individual rather than community level. “That can work and is definitely part of the toolbox, but unless we focus on the structural and social context of all of those individual behaviors, we will not really solve this problem. We have to accept evidence that is not just experimental,” said Butts.

A HOSPITAL-BASED APPROACH TO VIOLENCE INTERVENTION²

For Thea James, the emergency department is an ideal place to capture a young person’s attention and begin to break the cycle of community violence. In the United States, more than 700,000 young people ages 12 to 24 are seen in emergency departments for non-fatal injuries sustained from assaults. Moreover, said James, up to 40 percent of young people under age 24 who are admitted for violent injuries are readmitted for subsequent violence-related injuries, and half of these youth who return are homicide victims. Violence, she added, is the leading cause of death for African American males ages 15 to 34, the second leading cause of death for young Latinos, and the fifth leading cause of death among white males in the same age group.

As a reminder to the workshop attendees, James defined the social determinants of health as the economic and social conditions that influence individual and group differences in health status as well as opportunities for employment, to get a good education, and in general to thrive (see Figure 5-1). She noted that if the cycle of violence is not broken when treating young people, who often have their own children, what happens to these young individuals is likely to happen to their children, too. In fact, she said, when she and her colleagues created Boston Medical Center’s Violence Intervention Advocacy Program (VIAP),³ which is part of the National Network of Hospital-Based Violence Intervention Programs,⁴

__________________

² This section is the rapporteurs’ synopsis of the presentation made by Thea James, vice president of mission and associate chief medical officer at Boston Medical Center, and the statements have not been endorsed or verified by the National Academies of Sciences, Engineering, and Medicine.

³ For more information, see http://www.bphc.org/whatwedo/violence-prevention/violence-intervention-advocate-program/Pages/default.aspx (accessed August 21, 2016).

⁴ For more information, see http://nnhvip.org (accessed August 21, 2016).

they created it to provide services for both the injured individuals and their families.

Helping individuals to leverage their own personal strengths and assets can affect population health and even prevention when dealing with a young person who has children, said James. Doing so can be a form of treatment, and it can break cycles of violence, she said. “I think many people have many innate talents and strengths,” said James. “They just need an opportunity to make those things come to fruition.” She noted that many of the young individuals who have received the hospital-based intervention have told her that without that intervention, the cycle of violence would have continued and they would either be dead or in jail.

When James and her colleagues created VIAP, one of its main tenets was that they would do everything possible to see every young person who came to the Boston Medical Center emergency department who had been shot or stabbed. “There was no cherry picking involved,” said James. “It took a while in the beginning to get the program to the point

where everyone understood that the goal was to connect with every single person who come through the door. We did not have a precedent for how this works.”

When developing the program, James started with the formula that parents use to help their children grow up to have opportunities for a good life. Parents make sure that they mentor their children and are there when their children need them. Parents give their children shelter, food, and opportunities for education. Parents teach their children how to navigate life, how to hold down a job, and how to be responsible. “I thought, that is what we will do. Our staff will be mentors, and we will partner with community-based organizations to provide the other specific things that people need,” said James. She also accepted that because VIAP’s clients are young people, these individuals would have some missteps, but just as a good parent will do, she and her team would give them the support they need to get through these challenges.

The program she and her colleague developed uses a trauma-informed approach and assumes these young individuals have experienced traumas that have shaped the way they think, their expectations, and their perspectives. Operating from that viewpoint allows staff to see many of the behaviors of these young people as manifestations of trauma rather than stubbornness at not wanting to engage in physical therapy or being surly or uncooperative when they act like they do not care about getting help. “I think any of us who had been in any way assaulted would probably react the same way,” said James.

VIAP was started with five goals:

1. Provide opportunities for progress and “real” change that would put the individual on a different track in life.
2. Prevent retaliation.
3. Serve as positive role models and promote positive alternatives to violence.
4. Strengthen family networks.
5. Contribute to safer, healthier communities.

James and her team learned early that it was important to reach out to other staff to get them on board with what they were doing and gain allies among hospital staff, particularly given that the program involves staff from a wide range of departments. Outreach to other hospital staff is not something that they did in the beginning, and it caused some tension with other staff who were concerned that their roles in the hospital were being usurped. Once her team had formed alliances throughout the hospital, the program ran smoothly. Another early approach James and her colleagues used was to plot a four-stage path that these young individuals would

take through the program. Stage 1 starts at intake and involves building trust and creating a relationship with an individual. There is daily contact through this stage and staff work with the individual to create a safety plan and an assessment of that individual’s needs, and to complete any relevant documents. In the second stage, the program connects the individual with needed services, including transitional assistance, victim’s assistance, housing, medical follow-up, primary care, and mental health counseling. Stage 3 helps the individual with personal development by connecting them with the educational system, job readiness and training programs, job opportunities, and continuing mental health services. The final stage aims to help the individual be independent and maintain the gains they have made so far while nurturing further growth and development. These stages are not set in stone, but rather serve as a means of organizing tiers of service that meet these young individuals where they are in their lives.

When the program was launched, James hired two young people from the community to be what she calls advocates and some of her colleagues call intervention specialists. These two young men worked 7 days per week from 4 p.m. to midnight, and over the course of those 8 hours they would go from room to room getting to know who these young people were, gaining an understanding of how they ended up being shot or stabbed, and developing a list of what their needs would be going forward and which partners the program would have to involve to meet those needs. These interactions, said James, gave her and her colleagues the opportunity to establish what resources the program would need to succeed, develop, and move forward. One thing they learned in this first year was that the tattoos on these young individuals were sources of information about the lives they were leading. One tattoo said, for example, “Born to be hated. Dying to be loved.” Another said, “I am a product of my adversity.” A third read, “Death is nothing, but to live defeated is to die every day.” What these tattoos said to her, James recounted, was how hopeless these young men are, that from their perspective the trajectories of their lives were bleak and therefore they had nothing to lose. “It gave us some idea about what they would need from us from the perspective of mentorship and partnership,” said James.

To James, the most important element of VIAP is having one caring adult these young individuals can depend on and trust. “That is the person who gets them on the other side of the bridge,” said James. Based on their experiences in the first year, she and her colleagues have transitioned the program from being purely peer based and hospital based to a full case management model with community partnerships. The program now has three advocates serving as case managers as well as a family support coordinator, a data research manager, and an employment

readiness specialist. Emergency medicine residents can now choose the program as an elective rotation, and the program also has social work interns. The program has two City street workers from Boston Centers for Youth and Families who James explained are essentially on duty in the neighborhoods 24 hours per day, 365 days per year. Every time someone gets injured or fatally wounded, these individuals receive cell phone notification texts. They go to the hospital and the bedsides of these young men and women, some of whom they have formed relationships with in the community. These street workers have proven invaluable to the families of those involved, the hospital staff, and the public safety staff. “They will often mitigate chaos in the hospital because when someone gets injured, many people from the community convene on the hospital,” said James. Having the street workers at the injured individual’s bedside almost immediately is invaluable, she added, because their trusted status enables them to defuse the call for retaliation.

James and her colleagues also work in tandem with a behavioral health team called the Community Violence Response Team based in the hospital’s trauma department. Every person on the hospital’s “protected list,” which restricts visitors to protect an injured person’s safety, is assigned to one of VIAP’s advocates and to a member of the Community Violence Response Team. In addition, individuals can also be assigned to a family support coordinator, creating a team of three individuals looking out for each patient. This team works in tandem with victims of violence. Participation in the program is voluntary, said James, and individuals who turn down the offer to join when they are in the hospital can always join the program later.

Today, the program has a van provided by the hospital to transport program participants and move them into their new apartments, which they can live in rent free for 1 year if they have secured a job and are actively engaged in the mental health portion of the program. The program provides the participants with clothing so they make a good impression when interviewing for jobs and they get help obtaining their driver’s licenses.

As a final note, James explained that program staff members serve as mentors to the advocates and street workers, as well as to their families. “We send them to many different types of training and we help them get into college.” She recounted stories of one advocate who had been performing poorly in school who is now consistently an honor roll student in college and planning to go to graduate school. Another young man who had not graduated high school when he became an advocate recently invited James to come to the graduation ceremony for his M.B.A. degree.

DISCUSSION

Daniel Webster started the discussion by asking James if VIAP clients undergo formal screening for PTSD. She replied that the Community Violence Response Team does the PTSD screen. Webster also asked James to elaborate further on the trauma-informed approach her program takes. James said her staff conducts simulations in a simulation laboratory for nurses so they understand how a patient’s behaviors can be manifestations of the trauma that patient has suffered. She noted as an example that after the Boston Marathon bombing, nurses reported that some patients had repeated outbursts, but nobody saw this as a negative behavior; they understood the role of trauma in the patients’ actions.

Pamela Russo of the Robert Wood Johnson Foundation (RWJF) asked Webster to comment on why he thought some neighborhoods that implemented Safe Streets initiatives had better results than others. Webster replied that the neighborhoods in which the program worked best were those in which the outreach workers saw the role as a life mission. In contrast, neighborhoods that did not have good outcomes were staffed by outreach workers who saw their job as merely a means to a paycheck. “I believe that this is a program model that is highly dependent on these credible messengers building trust. Yes, they all get the same training that I am sure is helpful, but I think people bring different things to this job,” said Webster. Another factor, he added, is the differences in the neighborhoods themselves. The two areas in Baltimore where the program has been least successful, he said, are the most challenged in terms of community organization and participation and the number of abandoned buildings, for example.

He also noted that this work is incredibly difficult and experience leads him to believe that the program has underinvested in its outreach workers. In fact, he added, the thing that he is least proud of with the Safe Streets program is that the salary for doing this dangerous job did not increase from 2007 to 2015. Referring back to DeVone Boggan’s comments about respect (see Chapter 4), Webster emphasized that it is important to respect the workers that make or break these programs.

The real value of these models, said Butts, is the fact that you can have people from a neighborhood with the background and experience to be credible messengers and who have the ability to tap into the information available in the community about conflicts that are about to erupt. What that means, he said, is that programs are hiring people with challenges of their own, and as a result, the trauma-informed approach that James described should also be used when hiring, supervising, and supporting the workers as much as the participants.

Butts also commented that the challenge he and his colleagues are facing when trying to evaluate programs such as Cure Violence and

get a picture of what different programs emphasize is that just because they all have the same branding does not mean they are all doing the same thing. “There are variations among sites that happen because of the personalities of the people running the program or the number of people hired and their background,” said Butts. He claimed there is “an epidemic of confirmation bias among these programs,” and admitted he has his own biases about what makes these programs strong. What is needed, said Butts, is the time and resources to invest in enough research and fidelity measures to know which exact components and behaviors of the programs lead to the best outcomes. “We are far from understanding the huge assortment of variables that go into that mix,” said Butts.

Antonia Clemente with the Healing Center, a domestic violence and gender justice organization, commented that none of the presentations mentioned violence against women and children and asked why that had not been part of the workshop’s conversation. She also asked if Cure Violence and these other programs address the connections between gun violence and domestic violence. Webster acknowledged the importance of the issues Clemente raised and said that the Cure Violence Health Model was created to address gang violence more than other types of violence. That being said, he noted that Cure Violence clearly recognizes that partner violence is part of the social norms that accept violence in these communities and therefore is part of what the program tries to address. Domestic violence is just not the primary focus of this particular program. He noted that many of the concepts being discussed at this workshop are related to women’s safety and intimate partner violence. Betty Lee Davis, a clinical social worker, added that other workshops have addressed partner violence.

Charles Branas of the University of Pennsylvania commented that while Cure Violence has spread to many cities around the country, he wondered how in the future it would be possible to pair this type of program with a university or scientific evaluation team early in the process. Butts replied that RWJF did that when it attached his evaluation team to the program early in its development. He added that RWJF has expressed an interest in knitting together evaluation and programmatic partners to have a common conversation. Webster added that the CDC’s youth violence support center is the other major funder of research in this area.

Butts said the hospital component can be an incredibly powerful part of a violence intervention program if done well, but the political and economic power of the health sector can easily overwhelm the neighborhood base of what a Cure Violence program is supposed to be. “You do not want to professionalize it too much, but somehow there has to be a linking mechanism that allows the neighborhood-oriented credible messengers to feel comfortable interacting with the health industry,” Butts

explained. The best example of this that he has seen is in New Orleans, Louisiana, where there is an effective, respectful collaboration among law enforcement, the hospital, and residents in the neighborhood. The most important piece is that the police understand they should not interfere with or dilute the power of the neighborhood component, which requires cooperation and communication with the leadership of the police department. In contrast, he has heard stories in which the police try to undermine programs by walking up to workers onsite after a shooting, which destroys the trust the community has in these workers.

George Isham of HealthPartners wondered in relation to scaling and consistency if there is a need to bring in collaborators from business schools who could share what they know about creating consistent leadership and scaling interventions so that they are more reproducible across geographies and organizations. Webster thought that was an important point and said he is co-chair of a recently formed Safe Streets Baltimore advisory committee that includes people with just that expertise to help them think about program management. He noted the importance of working collaboratively with other experts to think about how to scale effective initiatives to address violence on a national scale.

An unidentified workshop participant asked the panelists to comment on the difficulty of paying the on-the-ground workers a salary commensurate with the difficulty and danger associated with their jobs in the community. Both Webster and Butts agreed this is a significant problem and both expressed dismay that these workers not only have a hard time getting paid what they deserve, but because these programs are often funded through city contracts, there can be gaps and delays in paying salaries. They also agreed this is an issue that will have to be addressed if these programs are to be sustainable, particularly given that the most effective workers, the ones who play a major role in the success of these programs, do not treat these as 9-to-5 professions. “We have to recognize just how challenging it is to do [these jobs] and support the workers. They are saving lives and we need to value them,” said Webster.

Dorothy Indyk, a retired preventive medicine specialist, remarked that she is disappointed and distressed that the lessons learned over 30 years in the HIV field about the importance of integrating prevention and treatment are being relearned by this field. “No matter which conference I go to, I find the same problem. We are creating new siloes,” said Indyk. She noted that while CDC has expanded its programs dramatically, it is still silo-based in terms of how it defines prevention. “We have to define prevention holistically across the life cycle. We are still doing acute-level interventions,” she added.

An unidentified workshop participant asked if any of the evaluations of these programs have been able to identify how structural differences

among different communities affect the ability of these interventions to be successful. Webster responded that research has so far made some general observations. It should be no surprise that if you are going to the places where the problem of gun violence is the worst, that these organizations will be the most stressed, have the least resources, and so on, said Webster. Another variable appears to be how overtaxed the organizations running these programs are and how truly committed they are to the community. He said he has seen the highest resourced organizations not do well when the commitment piece is missing.

Butts remarked that knowledge development in this area is restricted or impeded by the issue of data outcomes. He recounted a conversation he had with a physician who said he could say exactly how many cases of hepatitis B there were in his community from year to year. Butts replied to this physician that he does not have a state legislature deciding what hepatitis B is, nor is hepatitis B a different disease in every state. The definitions of violent crimes, the reporting rates, and the way that violent crimes are addressed differ across jurisdictions and communities. “It is very hard to know who has been violent or who will be violent, and that squishiness makes it hard to implement solid programs and evaluate them,” said Butts.