The September workshop included a session with four speakers outlining aspects of existing resilience interventions that might serve as models for the Department of Homeland Security’s (DHS’s) resilience initiative. Speakers from military and non-military programs were invited to present their experience with various resilience interventions. Speakers were also asked to discuss measuring program effectiveness and evidence supporting various types of interventions. Lt. Col. Daniel Johnston from the U.S. Army provided an overview of the Comprehensive Soldier Fitness (CSF) Program. The director of the Military Operational Medicine Research Program in the U.S. Army Medical Research Materiel Command, Col. Carl Castro, discussed his perspective on various initiatives within the military. Dr. Randall Beaton from the University of Washington in the Schools of Nursing and Public Health summarized issues he identified as key lessons from resilience programs in first-responder populations. Dr. George Everly from Johns Hopkins Bloomberg School of Public Health and the Resiliency Science Institutes presented an overview of his work on resilience.
While the majority of the presentations focused on military, first-responder, and law enforcement personnel, aspects of the interventions could be applied broadly to DHS employees. After the presentations, speakers participated in a panel discussion that addressed questions for workshop attendees. Planning committee member Dr. Joseph Barbera moderated the panel discussion. Themes that emerged from individual presentations and the panel discussion can be found in Box 5-1.
Themes from Individual Speakers on Resilience Programs
- Role of leadership
- Relationship between physical and mental well-being in resilience
- Linkages between resilience and family/social support
- Addressing organizational cultures within program design and implementation
- Role of evidence and performance measurement in developing and improving interventions
The issue of resilience under stressful conditions is particularly relevant for those serving in the military. Lt. Col. Daniel Johnston stated that the U.S. Army’s Comprehensive Soldier Fitness Program was born in response to the prevalence of post-traumatic stress disorder (PTSD) and an increase in suicides among army personnel. The program also offers the possibility of improved performance. Johnston indicated that these increases have been attributed to dramatic changes in the operational tempo of the military in a post-9/11 world. Prior to 9/11 a member of the armed services could anticipate being deployed once or twice during a career. Other stressors such as moves to new duty stations were fairly predictable. Since 2001, the cycle has become condensed with multiple deployments and more frequent changes in duty stations. Johnston noted that it is important to know that suicide victims are equally divided among those who have been in combat and those who have not.
While in theater, soldiers are often exposed to traumatic experiences. As a result, the amount of psychological and physical problems within the military population has increased. Johnston cited a study on the prevalence of PTSD, depression, alcohol use, and drug use among veterans that was carried out by the Department of Veterans Affairs (VA) from 2003 to 2006. The study found that the combined rate of mental health disorders among veterans from Afghanistan was about 6 percent. After the conflict in Iraq started, this rate rose to 37 percent (Seal et al., 2009).
The CSF has defined itself as “a structured, long-term assessment and development program to build the resilience and enhance the
performance of every soldier, family member, and defense agency civilian.” The CSF model incorporates five domains of fitness: (1) physical, (2) family, (3) social, (4) emotional, and (5) spiritual. The CSF model uses four program components or pillars to measure fitness domains and to train individuals and groups:
- The Global Assessment Tool (GAT) is used to assess individual soldiers and drive the development of interventions. The GAT is an online assessment tool and currently includes four domains of fitness: family, social, spiritual, and emotional.
- The Comprehensive Resilience (Self-Development) Modules are online training units that are not linked to performance on the GAT. These modules are designed to increase overall resilience. However, their effect on individual GAT scores is not known at the present. The training is currently not tailored to the individual’s performance levels. The modules will be voluntary for family members and defense agency civilians.
- The Master Resilience Trainers (MRTs) is a training program designed to teach resilience and performance optimization skills to unit personnel and their families. The MRT courses are taught at the University of Pennsylvania, the Army’s training program at Victory University, and at various sites around the country. The training is held by the Mobile Training Teams at specific CSF-PREP (Performance Resilience Enhancement Program) sites at 11 installations around the country. The PREP sites specifically focus on skill sets and training in addition to the core MRT training that involves mental performance enhancement.
- The Institutional Military Resilience Training is taught in Training and Doctrine Command schools, the Noncommissioned Officer Education System, and the Officer Education System. The training is progressive, sequential, and targeted to the unit deployment cycle. It has shown to be valuable and well accepted in the Basic Officer Leader Course.
The U.S. Army is concerned about maladaptive and undesirable coping mechanisms and responses from soldiers caused by stress and the rigors of combat. Examples of maladaptive behaviors include drug and alcohol use, smoking, harming practices such as cutting, domestic violence, inappropriate and unhealthy eating habits, risk-seeking behaviors such as reckless driving, and suicidal intentions. The CSF is
focused on teaching soldiers positive, desirable, and mature adaptive responses and behaviors, as well as enhancing mental performance. Johnston asserted that the CSF model is similar to the military’s physical training (PT) program. Soldiers are regularly assessed by taking a PT test. As with PT, the CSF program elements not only make soldiers more “hardy” or physically resilient, but also enhance their performance in the “heat of the moment.” The CSF is not intended to be a single course, event, or requirement, or a “screen” for any physical or psychological disease or dysfunction, including suicide. The assessment is part of a long-term process. The program is focused on resilience and performance enhancement, and the elements taught are more expansive than interventions that just focus on doing something after an individual has a negative psychological, physical, social, or professional outcome. The focus of the GAT, the Comprehensive Resilience Modules, the Master Resiliency Trainers, and the Institutional Military Resilience Training is to promote long-term resilience and enhance performance.
The Global Assessment Tool
The GAT is the largest undertaking of the program and is the backbone of the assessment. The information gathered by the GAT is used to structure meaningful reports around psychological fitness in specific areas for individual soldiers. The survey is designed to collect key pieces of information in a reasonable amount of time. The survey has 105 questions and takes approximately 15 minutes to complete. Soldiers are required to participate. The survey is voluntary for military family members and civilians employees. Once an individual has taken the survey he or she receives his or her score in each of the nonphysical domains along with some standard language about the results.
Confidentiality of the data is essential. Individual scores are kept confidential and are not reported to the military command or leadership. Soldiers need to know that the GAT is for their assessment and is not shared with their commanders. The minute soldiers feel that this information will be reported to their commanders or the leadership, they will stop giving honest and accurate responses.
The psychological fitness score includes measures of family, social, emotional, and spiritual fitness. Johnston pointed out that the term spirituality should not be confused with religiosity. He indicated that it is an important distinction and that the military is not trying to teach religion. Instead, spirituality is focused on the value of believing in
something greater, which could be the unit, the Army, or the family at home. This belief helps give soldiers a purpose. On a scale of 1 to 5 with 5 being the highest and 1 the lowest, the mean psychological fitness score is around 3.8. The distribution is tight, and the distribution between males and females is fairly equivalent.
Working closely with the Consortium for Health and Military Performance at the Uniformed Services University of the Health Sciences, Lt. Col. Johnston is developing the physical fitness component of the GAT in order to provide a meaningful online physical assessment. Smartphone applications are also under development. Questions in the GAT cover the following areas:
- Healthy habits: Nutrition, sleep, dietary supplements, hydration, caffeine, tobacco, alcohol
- Physical performance: Score on unit physical fitness test, flexibility, exercise frequency of individual
- Physical build: Body fat calculation based on waist, height, weight
Johnston also pointed out that the data indicate that across the board, regardless of a soldier’s GAT score, everyone experiences relatively the same amount of traumatic events. After deployment, people that have very low GAT scores have a significant increase in reported symptoms in their post-deployment health assessment.
Master Resilience Trainers
The MRT is a 10-day course intended to teach skills that increase positive adaptive psychological fitness, and it includes a foundation in how to deal with conflict, communication problems, and so on. After the training, individuals serve in several capacities both as a role model and unit trainer of the core skill sets taught in the MRT. They are expected to personally use the skills they have been taught, engage in discussions about work objectives and progress, teach these skills to others, act as the commander’s advisor regarding issues related to total fitness and resilience training, and know the referral options for professional counseling including behavioral health providers, chaplains, and other appropriate resources.
As part of the training, soldiers receive a skill identifier. A skill identifier in the Army means they have an additional duty description.
Therefore when they go back to their units they are able to use their MRT-required training and instruction and set of activities with their unit. The master resilience trainers are soldiers in the E6 and E7 levels and generally are between 26 and 35 years old. It is this group that is leading and mentoring younger soldiers, 18-24 years of age. The participation of the senior enlisted soldiers is critical because they frequently interact with the younger soldiers who are more likely to experience problems.
Johnston described the MRT course he participated in at the University of Pennsylvania. In that course there were approximately 200 students. Many students were initially skeptical. However, after the course the participants were excited about the training and looked forward to using it with their units. The training focused on understanding that resilience is about using critical thinking, gaining knowledge, and practicing skills in order to overcome challenges, act mature, and bounce back. The training explored how to deal with negative patterns of thought and how to develop a set of tools to deal with hardship, thus enabling the trainees to be better able to bounce back from adversity. The training enforced the concept that most challenges are temporary, not permanent; local, not global; and can be changed by one’s own effort.
Comprehensive Resilience (Self-Development) Modules
Comprehensive Resilience Modules are online video modules developed by experts in various fields. Many of the video modules are very similar to the materials taught in the MRT courses. These are currently not linked to the GAT, and the program is currently developing its future strategy so that it will be linked with evidence-based resilience training and the assessment tool to evaluate effectiveness. The modules are also available to all family members in the Defense Eligibility Enrollment Reporting System. The family modules are similar to the military ones but are generally geared for the family members for self-assessment and are worded slightly differently. To date, more than 900,000 individuals have voluntarily gone through the online modules.
Institutional Military Resilience Training
The Institutional Military Resilience Training (MRT) program has trainers go into the training and school environment and work with soldiers during their educational process. This also provides an opportunity to reach leaders through these schools. It is essential that leadership is part of the process and engaged with the program from the four-star level down.
Lesson Learned from the Comprehensive Solider Fitness Program
Johnston discussed examples of research that examined the importance of psychological fitness. Studies of psychological fitness show that it affects soldiers’ ability to complete training as well as their risk of developing problems such as PTSD. Soldiers whose GAT scores are in the bottom 25th percentile have a 2- to 10-times higher likelihood of negative outcomes, are less likely to complete training, and are more likely to have other types of undesirable behaviors such as drug use and suicide. Individuals with high psychological fitness are more likely to complete the training and less likely to develop PTSD after a traumatic event. Johnston pointed out that even at relatively low levels of combat there are significantly higher levels of anxiety in people with low psychological fitness.
The data indicate that emotional fitness is an important measure. People who score very low in emotional fitness have almost twice as many primary care visits as an individual with a high score. This finding is important when looking at the utilization of medical resources and has implications beyond the military. Soldiers with low emotional fitness have significantly higher rates of reporting three or four symptoms of PTSD. Emotional fitness also appears to influence psychosomatic symptoms associated with memory, balance, ringing in the ears, and dizziness. Lower rates of emotional fitness are also related to higher rates of depressive symptoms.
Ongoing performance measurement is a part of the program. A review of a unit over the course of a year during which the MRT program took place, reveals several interesting findings. Of those reporting suicide gestures, attempts, or ideations, 78 percent had not been trained. At Fort Leonard Wood, Missouri, the CSF program looked at individuals who had to leave the Army because of medical reasons. People that
received weekly CSF training ended up staying in the Army and had a much lower chance of being separated out of the Army, even for general medical reasons. Army units that had Master Resiliency Trainers experienced improvement in their GAT scores over time.
In a study with Navy recruits, various psychological measures such as depression and stress were assessed. Recruits with high scores were put through the normal basic Navy training. Recruits who scored very low were separated into control and study groups. The control group went through the normal training. The study group had resilience training in addition to the normal training. The researchers observed a large difference in scores for the control group compared to the study group, with more individuals in the control group being separated from military service for psychological reasons (Williams, 2004).
One of the emerging areas of research that have resulted from the Army’s partnerships with different universities and researchers is the work around post-adversity growth. Much of the Army’s programs have historically focused on individuals who are experiencing problems and how to help them. However, it is also important to consider what positive outcomes can result from adversity. Johnston asserts that it is possible to help various groups such as disabled veterans to productively deal with adversity and experience growth under a new set of circumstances. Although this is outside the scope of the CSF at the moment, it is a possible new area to explore.
Col. Carl Castro is the director of the Military Operational Medicine Research Program in the U.S. Army Medical Research Materiel Command at Fort Detrick, Maryland. He works on a broad research portfolio that includes psychological health and resilience. This research includes about $40 million in funding for resilience efforts that are not focused on treatment or recovery. There is also funding for family resilience research within the Family Transitions in Well-Being Program.
Castro quoted a statement by the Army vice chief of staff Gen. Peter W. Chiarelli in 2009:
We have a force that is much more resilient than I ever thought it was going to be, but it is much more stressed.
The challenge facing the Army today is the overall well-being of the force.
Castro stated that the core of this quote is true and that through self-assessments and objective measures the force is very resilient. The data in Johnston’s presentation illustrate the strong coping skills of the force before deployment. Although they are not optimal, they are good and show where there is room for improvement.
Castro commented that the Army and the Department of Defense (DOD) are threat-based organizations. As a result, the mission of his group is to develop effective medical countermeasures against combat and operational stressors to maximize warrior health, performance, and fitness. There are four focus areas:
- injury prevention and reduction,
- psychological health and resilience,
- physiological health, and
- environmental health and protection.
Importance of Long-Term Thinking
Resilience research began in child psychiatry and child psychology and is now being applied to adults. Castro asserts that resilience cannot be a priority just for adults confronted with disasters or potential traumas. It has to be a priority for the nation, which starts when children are young, not when they join the military at 18. As an example, the national school lunch program was launched during World War II because the military found that a large percentage of the men recruited into the military were malnourished. The school lunch program was a long-term effort to build a strong population in order to ensure a strong military in the future. The nation is faced with a similar long-term problem with resilience today. This issue requires creative, strategic, and long-term policy making.
There are many ways to think about resilience. Although disorders such as PTSD or events such as suicides are the logical conclusions along a resilience continuum, resilience is not just about them. There are broader outcomes to consider. Less resilient individuals are more likely to engage in self-harm or self-destructive behaviors. For example, 67 percent of soldiers who attempted suicide in 2007 self-reported using alcohol or drugs during the event (U.S. Army, 2008). Many people who
have no intention of hurting themselves engage in self-destructive or risk-taking behaviors such as smoking, unprotected sex, binge drinking, drunk driving, reckless driving, and overeating. The question becomes: How do people learn to self-regulate and stop these behaviors?
Much of the research on resilience is more in the basic science area. Research in general shows that people 18 to 24 years old are susceptible to peer pressure and more likely to engage in risky behaviors. A lot of work is directed at this age group because it is the most susceptible group in the military as well as in colleges. Something happens around age 25 that fundamentally changes the outlook. Castro believes that a lot of work is left to do on this topic at the basic science level.
Comprehensive Solider Fitness Model
Castro added a disclaimer that he was one of the researchers to develop the GAT discussed in Johnston’s presentation. Although he is no longer working on the CSF program he believes it is the most comprehensive resilience development effort in the DOD.
There were no best practices at the time the CSF was developed, and as a result the CSF was rolled out without being validated. To address this issue the rollout included in-depth ongoing program assessment. The program was designed to be recursive. There was initially a lot of criticism about the evidence base of the program, which may have had some validity 3 years ago but does not now because of the built-in assessment.
Castro stated that he questions some of the assumptions within the CSF program model. In particular, he questioned the sequence of events and how they are processed to be perceived as traumatic or adverse. He asserts that the more significant flaw is that this implies an individual either has growth or is traumatized by the event when both are possible at the same time. Research in the United States, Israel, and Norway indicates that individuals can have growth and decrements at the same time. To illustrate this point he said that some soldiers with combat-related PTSD report an improved perspective on life as a result of their combat experiences. This indicates that there is a post-growth curve even when there are problems. Although the CSF model includes a post-adversity growth line it is not necessarily an accurate picture.
The concept that resilience is not an all-or-nothing phenomenon is important particularly in training service members. The important message to individuals is that resilience is not going to be all good or
bad, and that if they are not enjoying their life, they should seek help. There are telltale signs of when someone needs to seek help. For instance, the first sign is often when a soldier’s partner, whether a spouse, boyfriend, or girlfriend, tells the individual he or she needs help.
Castro does not agree with the view some have that resilience is a state that people either have or do not have. How resilience is defined will determine how it is researched scientifically—as a trait, state, or process. The fields of child psychiatry and psychology view resilience as a process. Castro supports this approach. He noted that his group funds a broad range of research that looks at all three approaches.
The Army has adopted a universal prevention approach and does not focus just on at-risk groups. The whole idea of the CSF is to tell people what they need to know when they need to know it, so the training modules are targeted to the deployment cycles. The family trainings are organized the same way. Some of the modules are lifetime skills, and others are uniquely applicable to the current deployment.
Physical Health and Recovery
Resilience is the ability to bounce back and recover, and there is a connection between physical health and resilience. Individuals that are physically in shape, generally feel better mentally, psychologically, and emotionally. They are also better able to recover from mental health and physical health challenges. Although the term the GAT uses is psychological resilience, Castro prefers emotional resilience. Psychological resilience was chosen for political and cultural reasons.
The Army operationalized resilience in terms of determining when an individual is fit to return to duty following an event such as a combat death in his or her unit. Establishing return-to-duty standards poses scientific challenges that transcend the specific type of trauma. The essential requirements of military occupations must be matched to the soldier’s ability and to the extent of recovery to injury effects. In order to accurately, safely, and quickly return soldiers to military occupations post-injury, the military must first identify when a soldier returns to “normal,” both physically and psychologically. To achieve this goal, the Army is developing new tools, including mathematical models and sensors for physiological status, to assist with return-to-duty assessments.
Suicide Care Continuum
The Army has a model for the suicide continuum of care. The concept to this approach is that the solution depends on where the individual is on the continuum. The CSF program sits in the prevention, education, and training part of the continuum.
Castro voiced some concerns about the suicide program. He believes that the Shoulder-to-Shoulder Program places blame for the suicide on the junior leader and the buddy for the suicide. He asserts that some of that responsibility should be on the individual. The individual needs to have some responsibility for his or her own mental health. Psychological autopsies in the Army show that less than half the soldiers who complete suicide are loners. Individuals that commit suicide are often described by their peers as strange or oddball and as not fitting in. Castro believes that it is possible to target individuals more effectively if it is acknowledged that they have responsibility for their own mental health and physical health.
The military performs early screenings, risk assessments, treatment, recovery, and post-intervention care. It is important to have effective treatments available for those identified as being at risk for suicide. The assessment of recovery is difficult and largely based on judgment and experience.
Castro had several closing thoughts. The first is that all resilience programs should be evidenced based. He is suspicious of programs or research that are “evidence-informed” or “open trial,” noting that open trial means there was no control group and the design was underpowered. He urged workshop participants to be wary of these and similar terms. Additionally, Castro believes that it is time to start discussing resilience within a broader national framework and not just in terms of specific populations. This can be achieved by synchronizing well-conceptualized research strategies and continuum-of-care models as well as coordinating efforts across the entire psychological health and resilience domain. He stated it is critically important to the success of future programs that continuous program and process improvement is included in the design.
Randal Beaton is a research professor emeritus at the University of Washington in the Schools of Nursing and Public Health and an emergency
medical technician (EMT). Beaton noted that the combination of a Ph.D. and an EMT is a rare one and has enhanced his ability to work in the field with first responders. Some of Beaton’s programs and work have been developed through the support of the National Institute of Occupational Safety and Health (NIOSH), the Federal Emergency Management Agency (FEMA), the Centers for Disease Control and Prevention (CDC), and the Health Resources and Services Administration (HRSA). In particular, the resources from NIOSH and FEMA have assisted in developing and implementing workforce resilience programs for first responders, primarily firefighters, and paramedics. Beaton has also conducted some research with volunteer rescue workers and emergency dispatchers.
Beaton began by defining resilience in terms of first responders. Resilience is the capacity of first responders and first-responder organizations to continue to function both physically and psychologically despite various exposures. His presentation focused on six primary lessons from first-responder programs and the implications of these lessons for long-term resilience programs for DHS.
Lesson 1: Resilient First Responders
Firefighters are exposed to a host of different and potentially traumatizing incidents. The role of firefighters has expanded to include handling hazardous materials and providing emergency medical services, and firefighting is almost a sideline for many firefighters. In urban settings, 80 percent of calls are now related to emergency medical services.
Although there are the obvious stress exposures such as fires and disasters, it is important not to forget about exposures to routine stressors. Beaton noted that firefighters are also exposed to stressors from within the organization such as suboptimal leaders. A program participant talked about the stress related to the tyranny of a poorly designed standard operating procedure (SOP) that arbitrarily takes away latitude and flexibility. There are also the nonwork exposures. Beaton commented that in his research he found that a risk factor for PTSD in the fire service was having financial problems at home (Corneil et al., 1999).
Given the nature and extent of exposures in their line of duty, Beaton asserts that the vast majority of first responders appear to be resilient. He illustrated his assertion through various individual studies with different groups looking at PTSD and self-reported ratings of the effects of
traumatic events. The studies looked at different groups including multiple cohorts of military veterans, British ambulance drivers, 9/11 rescue workers, and Canadian firefighters. In all of these studies, 15-20 percent of the participants met the criteria for PTSD. This is higher than would be seen in the average population, which is closer to 2 percent. However, more than 80 percent of the participants in the first-responder study did not meet the criteria for PTSD (Corneil et al., 1999).
Beaton performed a cluster analysis of self-reported symptoms in 2,000 firefighters and paramedics in Washington State. The preliminary factor analysis identified symptom factors such as apprehension, head and face symptoms, anger, and gastrointestinal symptoms, to name a few. Beaton’s hypothesis was that the respondents would fall into different groups or clusters of emotional responders, somatic responders, or resilient responders. Instead he found that about 83 percent of the firefighters looked the same as a convenience sample of age-matched healthy males from the community. However, about 8 percent of firefighters and paramedics had stress in almost every stress domain. In conclusion, Beaton considered most firefighters resilient (Beaton et al., 1995).
DHS could take away several potential lessons from Beaton’s work with first responders. One possible approach is to target programs to the populations they are intended to help. For example, employee assistance programs must be targeted to DHS or its components. The employee assistance programs counselors must understand the work group they are counseling.
It is also possible to use online resources and trainings to extend the reach of the program to a broader audience. Beaton discussed an online training course on disaster behavioral health that he helped to developed with staff at the Northwest Center for Public Health Preparedness. Thus far a total of 850 people have completed this online training.1 Such online trainings can be offered to an entire workforce, accessed anonymously.
Lesson 2: Targets Are Not Always Obvious
Beaton discussed the results of a battery of surveys taken by approximately 2,000 firefighters and paramedics in Washington State in 1989. The surveys had a 50 percent response rate. Fifty-seven sources of occupational stressors were included in the survey, from which 14 stress factors emerged. The number one factor was sleep disturbance. Four items were included in the sleep disturbance scale, and they accounted for about a quarter of all the variance of the test scores. Wages, benefits, and reduction in the workforce were the number two occupational stressor, management or labor conflict with an immediate supervisor was third, and concerns about personal safety were fourth. Family financial strain was seventh, and past critical incident stress was eighth on the scale (Beaton and Murphy, 1993).
Management/labor conflict and sleep disturbance ranked the highest if considered separately for firefighters and paramedics and the factors associated with job dissatisfaction. Beaton noted there was a small percentage of respondents reporting discrimination; however, if it was reported, then it was strongly associated with job dissatisfaction. Communicating news of tragedy to surviving kin or friends was also associated with job dissatisfaction (Beaton and Murphy, 1993).
This information illustrates that it is not always clear where the real problems lie. Beaton asserts that unless the real problems are understood it is not possible to develop an effective program. He noted that there are several ways to do an effective needs assessment. In particular he mentioned anonymous surveys, focus groups, and “town hall” style meetings as possible options. Each option has both pros and cons. For example, different types of information are gathered in an anonymous survey compared to a focus group. Given that some of the DHS workforce is unionized, it might be possible to attend a union meeting.
Lesson 3: Understanding the Culture
Beaton asserted that it is vitally important to recognize the primacy of organizational culture when developing and implementing a workforce resilience program. To build a culturally appropriate program it is necessary to understand the culture. Beaton discussed the DHS
organizational chart as a road map of potentially very different cultures with different needs and programs tailored to meet those needs. By understanding the cultures of the various organizations, DHS is more likely to have buy-in and support from the leadership at every level and every stage in the process.
Lesson 4: Importance of Leadership
Beaton commented that in general there are two types of leaders. There are those who are task motivated and love lists and checking them off, and there are those who are more focused on relationships and can better communicate with people. Although both types of leaders are necessary, it is important to recognize that they prosper in very different types of situations. In one intervention with a fire department in Washington State, one of the leadership training components was focused around the concept of “leader match.” The leaders were asked to self-assess their style and then think of situations that matched their leadership style. They were then shown videos about different administration and management approaches in delivering bad news such as a poor performance rating. They reviewed and practiced the scenarios. They were also taught some stress management skills. Department outcomes were followed up after 3 and 9 months (Beaton et al., 2001).
Interestingly, this study documented temporary reductions in gastrointestinal symptoms in frontline supervisors. There were also temporary decreases in frontline firefighters’ recollections of past critical incidents. This is particularly interesting because the training was aimed at supervisors and not frontline firefighters. Subordinates’ rating of superiors who took the training improved at 3 months, but then returned to the pre-intervention level at 9 months, indicating that one-time trainings have short-term but not long-lasting benefits.
Beaton noted similar short-term gains in a NIOSH Leadership Intervention Demonstration Project, which included 24 hours of training that covered stress management, leader match, team building, and conflict resolution. Although there were some borderline reductions in absenteeism in the organization after the intervention, absenteeism rates returned to pre-intervention levels in the 1-year follow-up period. Beaton stated that the lesson here is that if the goal is long-term resilience and sustainability, then a more programmatic approach is needed (Beaton et al., 2003).
Implications of Leadership Training for DHS
Leaders at all levels are the backbone of the organization. Training leaders to be more effective and resilient potentially provides multiplicative benefits. Effective leaders are more resilient and can improve the morale and performance of their subordinates and enhance their subordinates’ resilience.
Lesson 5: Physical Fitness Matters
A wellness/fitness program funded by FEMA included multiple assessments and interventions including a baseline physical assessment, physician exams, and testing such as lipid panels. Every shift at every station had people trained to be peer fitness trainers, who helped with motivation and event injuries. FEMA funded workout areas in each station that included strength and aerobic equipment. Additionally, the chief of the department required that everyone exercise at every shift and document it. The training also included flexibility and stamina components. It was a multipronged policy-driven program with support from the leadership.
There was an overall decrease in on-the-job injuries during the intervention period. Beaton noted that the fitness program benefited out-of-shape firefighters the most. After 10 or 20 years of being a firefighter, about one-third of firefighters end up being in poor shape. The gap between the fit and the out-of-shape firefighters narrowed at the 1-year assessment (Lewis et al., 2005).
The study included a burnout scale that had been developed previously (Murphy et al., 1994) and assessed burnout symptoms such as fatigue, headaches, and insomnia. Looking at the mean burnout scores across all age groups, it was found that sedentary firefighters in every age group had the highest burnout scores. Sedentary firefighters in the 20-30 year age group actually had as many burnout symptoms as sedentary firefighters in the 50-60 year age group (Beaton and Vares, 2011).
Firefighters who infrequently exercised (one to two times a week) did not appear very different from the group exercising three to seven times a week. This suggests that firefighters do not have to exercise 50 minutes, 5 times a week, to get real benefits. Just 30 minutes of aerobic exercise a couple of times a week made a real difference in terms of burnout symptoms. The key points are:
- Exercise is associated with improved mental health in all age groups.
- Fitness programs increase the fitness of those least fit.
- Improved fitness may decrease on-the-job injury and absenteeism.
Lesson 6: Training Matters
Beaton commented that improved training of job-specific tasks enhanced mental health and resilience (Beaton and Johnson, 2002). He cited a study of canine handlers involved in the 9/11 rescue-and-recovery effort that found that canine handlers who had more training were less likely to suffer from PTSD (Alvarez and Hunt, 2005). Increased training increased confidence of responders. Beaton noted that many organizations that engage in disaster response drills and exercises are building resilience in their workforces (Beaton et al., 2004).
George Everly is an associate professor of psychiatry at Johns Hopkins Bloomberg School of Public Health and the executive director of Resiliency Science Institutes at the University of Maryland, Baltimore County, training centers. In 1992, Everly was invited to train physicians and psychologists in Kuwait to treat PTSD.
Importance of Culture
Everly echoed Beaton’s comments about the importance of understanding culture in program design. He added that culture is a particularly important variable in the treatment of trauma. In his experience in Kuwait, he found that they could not approach treatment or rehabilitation from a pathology perspective and instead adopted an approach based upon resilience. As a result, the mental health clinics in Kuwait were the first geared toward fostering human resilience in the wake of some significant traumatic event.
The approach Everly’s group has taken is to build resilience by creating a resilient culture through leadership. This differs from working
with individuals to foster their individual resilience, and it offers a slightly different perspective.
Based on his experience in business school, Everly focused on fostering leadership as a means of building a resilient organizational culture. Fostering leadership through training is based on the idea that it is possible to make a better manager and a better leader. The goal of traditional leadership training is to foster followership in order to enhance compliance. Everly used the term resilient leadership in a slightly different way. He sees leadership as a mechanism and not an endpoint. Leadership is a means of building a resilient organizational culture. It is a distinct variance from traditional leadership programs.
Everly noted that Malcolm Gladwell discusses the “Law of the Few” in his book The Tipping Point (Gladwell, 2000). This concept states that certain key people are conduits and gatekeepers. If these people are trained and shift their attitudes to building organizational resilience, it is possible to ultimately shift the culture of the entire organization.
Everly was invited to work with the Chinese government to develop a leadership course as part of the preparations for the Shanghai Trade Meetings and the Beijing Olympics. The idea of resilient leadership was to use leadership as a mechanism not only to enhance compliance or followership but also to enhance resilience of individuals through enhancing the culture. In developing the course, Everly did not focus on what successful leaders do; instead he looked at leaders held in high regard by those that followed them when things did not go well. Everly’s group included materials from the U.S. Civil War, the Crimean War, and World War II.
Additionally, Everly and his collaborators looked at various information sources on leaders such as The Art of War by Sun Tzu, The Guide to the Righteous Protector by Desiderius Erasmus, Machiavelli’s The Prince, and historians’ ratings of U.S. presidents’ crisis leadership. Everly et al. (2010) also used empirically based causal modeling research, as well as interviews and surveys with elite military and law enforcement personnel. They empirically evaluated nine core leadership factors and from them were able to identify the four most important. Everly noted that these factors not only predicted crisis leadership success but also rankings and overall leadership.
Everly commented that there is tension between evidenced-based research and qualitative research. Statistics are great, Everly said, but he still wants to talk to people. Additionally, although he is interested in data and theory, he is more interested in how they are applied. Everly
and his colleagues, while developing a resilient leadership training program, used an application framework based on Bandura’s self-efficacy investigations in social learning (Bandura, 1997). They ultimately piloted a training course for the Hong Kong government. More than 600 people registered for the course the first year, 550 the second year, and 600 the third year. A shortened version of the same resilient leadership course was used at the FBI National Academy in Quantico for 3 years.
Resistance, Resilience, Recovery
Everly and his colleagues at Johns Hopkins developed an overarching framework within which resilience could be better applied. Historically, resilience was defined as the ability to withstand or adapt to a rebound from extreme challenges or adversity. Everly’s group at Johns Hopkins developed a resistance, resilience, and recovery model as an outcome-driven continuum of care wherein adaptation and rebound were segregated (Kaminsky et al., 2007). Everly and colleagues developed the term resistant to describe the person who has developed a sort of psychological immunity and is not deterred by adversity. Resilience is then the term used to describe rebounding from adversity’s perturbations. In order to end the continuum and make it as comprehensive as possible, the model includes recovery through treatment and rehabilitation. Everly’s group argues that there are three elements to the overall construct of resilience.
Where Does Human Resilience Reside?
The Hopkins model emphasizes the importance of cognitions or the cognitive primacy model. In the words of Hans Selye, “It is not what happens to you that matters; it is how you take it.” In 300 BC Epictetus stated, “Men are disturbed, not by things, but the views which they take of them.” Everly focused on the question “Does the workplace make people sick, or is it something else?” From there his group developed a very linear model where stressors lead to an acute stress arousal that creates psychophysiologic symptoms and then general illness. Everly looked at physical illness at 1 year. Everly commented that it is possible to test each of these phases of the model. The analysis measured stressors on the job such as job control, workload, qualifications, and affiliation.
His group also used other standard measures from the business organizational behavior literature.
Using linear models, Everly commented that it is important to note that the direct effect of job control, workload qualifications, and affiliation did not have a very powerful predictive effect on illness.
Linear regression assumes linearity, so Everly’s group later analyzed the data in a simple structural equation model that assumed either random or elliptical relationships. The group expanded its research from physical illness and focused more on job satisfaction, desire to leave the job, turnover intention, and/or performance on the job. Again, there are no significant direct lines in this model, which suggests that there are mediating variables.
The next challenge for the researchers was to identify the mediating variables. It goes back to Selye and “It is not what happens to you that matters; it is how you take it.”
Some discussions split traumatic stress and burnout into two issues, which would underscore the pathogenic nature of simple, slow erosion burnout.
When Everly’s group reanalyzed the data, it found a direct line from stress arousal to the desire to leave the job. The group was able to identify the cognitive factors that appear to have the greatest pathogenic quality. Everly referred to this as the negative reiteration factor that led to reduced personal accomplishments and perceptions of poor performance, which leads to poorer performance on the job. Everly commented that it is important to develop an intervention that affects the pathogenic core of any debilitating phenomenon.
Everly’s interventions are focused on what most effectively mitigates the reiterating negative cognitions. He noted that to understand this component it is necessary to move to qualitative analysis. Everly quoted Henry Murray: “There is nothing so powerful as a well-phrased question,” and David McClelland: “The purpose of psychological testing is to ask a question that uncovers the essence of the person.”
To identify resilient people Everly’s group developed a questionnaire and fielded it with various groups, including deep undercover federal agents, members of SWAT teams, and Navy SEALs. The survey questions were simple, such as “What is the key to being immune to stress?” The response was positive attitude, training, and a healthy lifestyle. “What is the key to bouncing back from excessive stress?” The response was having a positive attitude, an outlet, a hobby, and a support network that included leadership. “If most people have a weakness that
makes them vulnerable to excessive stress, what is it?” The responses were a lack of perspective, tenacity, and preparation, as well as having a negative attitude. “What is the key to motivating people to help them be successful?” The responses included leading by example, training, and experience.
In focus groups with Navy SEALs the themes of the discussions included attitude, interpersonal support, and training as critical factors. Everly also noted that the SEALs’ belief that they were part of a greater mission was important. Based upon this research, Everly asserts that there are six qualities of resilient people. They are
- optimism or faith,
- social support,
- decisiveness (attempting to control only those things over which they have control) and responsibility,
- perseverance and tenacity, and
Everly noted that in meta-analyses on human resilience, social support explains the greatest variance of all the other variables. Decisiveness was defined as the attempt to control only those things over which you actually have control and then to take responsibility for them. Self-control may be the ability of a sniper to control his or her breathing to reduce anxiety and reduce fine motor discontrol—perhaps similar physiological interventions can be taught, as well as other interventions about overcoming impulsiveness and similar issues.
How Is a Resilient Culture Best Cultivated?
Leadership is a mechanism to engender a culture of resilience. When Everly and his colleagues looked at the role of poor leadership in direct effects models they found burnout, job dissatisfaction, poor performance, and turnover intentions were related to the stressors of role conflict, role ambiguity, suboptimal leadership, and role overload. Role conflict and role overload contributed minimally to the direct effects models and were mediated through the leadership. Leadership should be trained because it dictates organizational culture.
Resilient leadership is defined as behaviors that help others withstand crisis and adapt to or rebound from adversity. The goal of the
Resiliency Sciences Institute training program is to teach leaders and managers to be not only resilient, but also resilient leaders. The covenants of resilient leadership are strength and honor, which is consistent with military research that describes resilient leadership as authentic leadership. Honor in authentic leaders is exhibited by being confident and optimistic and possessing a high moral character and ethical reasoning. Strength in leadership provides purpose, motivation, and the ability to be decisive in highly stressful conditions. Leaders with these characteristics are the most likely to create loyalty, obedience, admiration, and respect. Authentic leaders are effective and make their followers feel safe. Part of this safety is founded on predictability. Resilient leaders ease fear and provide hope for those who follow; safety is based in trust, and trust in honor and integrity. The good news is that all of these factors are behavioral and can be taught.
A Pedagogical Framework for Leaders to Create a “Culture of Resilience”
Albert Bandura asserted that people guide their lives by their beliefs of personal efficacy. Bandura goes on to say that such beliefs influence
- the courses of action people choose to pursue,
- how much effort they put forth in given endeavors,
- how long they will persevere in the face of obstacles and failures,
- their resilience to adversity,
- whether their thought patterns are self-hindering or self-aiding,
- how much stress and depression they experience in coping with taxing environmental demands, and
- the level of accomplishments they realize.
To build self-efficacy, Everly asserts the following principles:
- Resilience by doing—Allow people to have success. Using successive approximation, allow people to develop an increasing sense of self-efficacy realized via their own agency.
- Vicarious resilience—Assign people to successful work groups or projects so they can experience a sense of shared success.
- Resilience via interpersonal support (encouragement, coaching, and mentoring)—Find positive people to provide support. Use a buddy system. Create surveillance systems and safety nets.
- Physiologic self-regulation—Teach people to mitigate stress arousal.
Everly noted that organizations such as fraternities attempt to engender self-efficacy by saying that if you are there then you are special, and therefore they expect special things from you. People often live up to those expectations. If people are treated like victims, then they will behave like victims. If people are treated like survivors, then they will behave like survivors. If people are treated like heroes, then they will act like heroes.
Everly-Strouse Leadership Scorecard
Everly and Douglas Strouse developed a leadership scorecard to determine if leaders truly exhibit resilient leadership. The quick set of questions asks:
- Are you optimistic?
- Are you decisive?
- Do you show integrity?
- Do you communicate openly?
When leaders are given this short survey, they usually score fours and fives. The second time they take the test, they are asked to provide at least two concrete observable behaviors demonstrating the resilient behavior to others. In general, scores drop down to the twos and threes. Almost everyone believes they are being self-efficacious and are resilient and good leaders, but it has to be demonstrated.
The Resiliency Sciences Institute training program is housed at the University of Maryland, Baltimore County. The certification in resilient leadership teaches three components of resilient leadership: the resilient leadership characteristics, the resilient moment communications model, and how to develop “psychological body armor.”
Planning committee member and panel moderator Joseph Barbera noted that several presentations discussed correlations in the data and potentially implied a direct cause-and-effect relationship. He asked speakers to comment on the development of an evidence base for this work. Beaton noted that the fitness and burnout symptoms were
correlations, and that a potential way to test the direction of influence would be to show that improved fitness decreases burnout rates in response to an intervention over a period of time. There are some nonsignificant changes on burnout scales, but many of the associations are frustrating because they are multifactorial. Barbera noted that information can be interpreted in many ways.
Castro noted that he is currently engaged in several group randomized trials. Group randomized trials are extremely difficult, expensive, and time consuming, which is why they are rarely done. Castro added that it is possible to have meaningful outcomes with small effects. There have been studies of many of the principles presented by the various speakers. In general, all of the presentations were hitting on the same principles and the same intervention strategies.
Castro asserted that agencies with multiple offices present a perfect opportunity to do group randomized trials. Although these studies are easy to design, hard to do, expensive, and time consuming, they can be done. The benefit of then having that data is that effectiveness can be shown. It is also possible to look at competing approaches so effectiveness does not become personality driven. The reality is that interventions and their potential effectiveness are often about politics and salesmanship. Programs should not be about who is the better salesman; they should be about the evidence.
Core Program Components
Planning committee chair James Peake noted that DHS is developing a common strategic core for resilience programs and asked the speakers what program components they felt were required in order to have an effective resilience program, and how those components could be scaled up overtime. Castro responded that a chapter he authored with Dr. Amy Adler in Resilience and Mental Health: Challenges Across the Lifespan focused on this issue (Castro and Alder, 2011). The chapter lays out the core and fundamental principles of building a resilience training program and also lays out how to ensure that it is both scalable and trainable. He added that the ability to scale the program and train others to implement them is critical when working with large populations. After all, it is not reasonable to have a program that only a small number of people can participate in if the population is large. That was one of the requirements for the CSF.
Beaton noted that with all the various cultures and subcultures within DHS it would be helpful to have core features that could be tailored to the different groups in order to ensure that they are appropriate and suitable. For example, firefighters and medics have a lot in common, but it is important to recognize that they have significant differences as well.
Participant Engagement and Trust
Planning committee member Karen Sexton noted that soldiers appear to trust the GAT survey and are participating without fear of retribution. How has the military been able to assure participants that it is anonymous? She added that in nursing they have not been successful in convincing nurses that surveys are anonymous, and as a result the surveys do not accurately capture all of the issues. Castro replied that it would be more accurate to think of the survey as confidential rather than anonymous. After all, the survey responses are linked to soldiers’ medical records. Instead it is made clear that the information is not shared with the leadership. Additionally, in his experience Castro has found that soldiers and Marines are very forthcoming with information. When he has asked them directly about their concerns, he has found their input to be thoughtful, reflective, and well informed. They have clear opinions about mission success. Beaton commented that in his work with the fire service, it took years of working with the fire service, the unions, and the state council before they trusted the researchers enough to even allow the surveys to be administered.
Castro commented that there is a perception that soldiers will not honestly respond to the survey. To address this concern his group compared self-reported information against the military records. The study looked at PT scores, awards, sick visits, DUIs, and so on. Castro found that the self-reported information was generally very accurate. Soldiers share information about socially undesirable behaviors such as drug and alcohol use in the survey where there is no incentive for them to do so. Beaton also added that much of his data is self-reported survey information. He has found that there are other measures that can be employed to see if respondents are being honest in their responses or if they are simply responding in a way that is perceived as socially desirable. He also found anomalies when he compared the number of incidents reported by firefighters and their service records. He found that firefighters were overreporting critical incidents. He added that it is difficult to precisely know what influences subjective responses.
Family Outreach and Engagement
Planning committee member David Sundwall commented that based on the presentations, social support systems—which in a large part is the family—are a critical core element of resilience programs. DHS has indicated that there are legal and logistical constraints that make it impossible to actively reach out to employees’ families. Sundwall wanted to know if other sectors such as police and fire departments, as well as private-sector organizations, have the same issue. Beaton added that he found that in the fire service a significant factor for PTSD was marital status. Married firefighters had half the rate of PTSD compared to their single counterparts in the fire service.
Barbera noted that it is important to consider family stressors, personal stressors, and job stressors, and how they relate to each other. He noted that improving stressors at work could inadvertently cause issues in one of the other areas. For example, in his experience he was able to cope with stressful shifts in the emergency department but found stressors difficult when he was home. Behavior that was effective at work did not work in other areas. Beaton agreed and stated that behavior that can increase efficiency and effectiveness in one area can be very ineffective and dysfunctional in another.
Program Design and Evidence Building
A workshop participate asked Castro to discuss the development of the CSF program and the ongoing program evaluation and evidence development supporting it. Castro noted at the time the Army started working on the program design there was very little evidence base in this field. However, there was a sense of urgency that something should be done in this field, and if they waited for the perfect evidence, then it would not happen. To get the program approved the design had to include ongoing program improvements. As a result, evaluation is a core program component. While it would have been great to have all the training modules in place for the initial rollout, that was not practical. The original GAT included about 100 questions, and participants received their score and a tailored narrative. Various training modules are gradually being added. In looking at the outcome data, CSF wanted to see how it related to suicides, promotions, and data, they were already collecting. Although the program has been criticized, as it matures and builds more evidence those criticisms will be less of an issue.
Kathryn Brinsfield from DHS asked panelists to comment on how to change aspects of the culture to improve resilience but retain other components of the culture that support the mission. Beaton responded that it is possible to change aspects of a culture while leaving other components intact. Given that within DHS, commitment to the mission is such a strong and important aspect of the culture, it could be argued that improving the effectiveness of leadership will increase the probability of the mission being accomplished. Changing organizational cultures is an inherently slow process. For instance, an appreciation for the psychological factors within the fire service has changed overtime. Everly added that if a culture is toxic, then it needs to change. Beginning with the leadership is a cost-effective and efficient way to do that. It is also important to identify and include the informal leaders into the process. Informal leaders have peer credibility and are the cultural gatekeepers and conduits. For example, if you wanted to change the culture in a hospital, then you would start with the nurses, not the physicians.
Barbera voiced his concern that high-stress situations change the culture in negative ways overtime. His experience in emergency departments that suffer from chronic stress and pressure shows that the staff that cares the most burns out. When they leave, the more resilient and perhaps toxic people are de facto leaders because of seniority and can possibly poison the workplace. Is it possible to recognize this possibility and address it before it reaches a crisis level? Castro responded that it is important to define the attributes that are desirable in the culture in order to effectively foster them. In his opinion every culture has aspects that are unhealthy and need to be changed. The military is constantly working on improving its culture. It is a conscious decision to actively change the culture. Although there is still a great deal of work to do, the military is making progress. Castro agreed with Beaton that culture change is not an overnight process and requires ongoing reinforcement and support. He used the example of sexual harassment in the military. The armed forces continue to do stand-down trainings and have required topic-specific annual training. There is constant assessment and evaluation. Change also happens through attrition as people leave. Castro added that real change takes a lot of planning up front. He believes that a good place to start is with the junior
leaders, such as the squad and team leaders in the Army. As they rise through the ranks they bring long-term change with them.
Everly asserts that it is also important to change the organizational values and to publicize those values broadly through the cultural conduits. Once individuals buy in, it is necessary to make it clear that behavior aberrant or at variance from those values is not rewarded. He also noted that peer groups, particularly in young adults, are often able to do things in a culture that policy cannot.
Alvarez, J., and M. Hunt. 2005. Risk and resilience in canine search and rescue handlers after 9/11. Journal of Traumatic Stress 18:487-505.
Bandura, A. 1997. Self-efficacy: The exercise of control. New York: Freeman.
Beaton, R., and C. Johnson. 2002. Evaluation of domestic preparedness training for first responders. Prehospital and Disaster Medicine 17:119-125.
Beaton, R., and S. Murphy. 1993. Sources of occupational stress among fire fighters/EMTs and fire fighters/paramedics and correlations with job-related outcomes. Prehospital and Disaster Medicine 8:140-150.
Beaton, R., and R. Vares. 2011. Resiliency training for Everett Fire Department firefighters and paramedics. Series of resiliency trainings offered on site. Everett Fire Training Center, Everett, WA. December 26-29.
Beaton, R., S. Murphy, K. Pike, and M. Jarrett. 1995. Stress symptom factors in firefighters and paramedics. In S. Sauter and S. Murphy (Eds.), Organizational risk factors for job stress. Washington, DC: APA Press.
Beaton, R., S. Infield, T. Ollis, and G. Bond. 2001. Outcomes of a leadership intervention for a metropolitan fire department. Psychological Reports 88:1049-1066.
Beaton, R., S. Murphy, L. C. Johnson, M. Salazar, and W. Corneil. 2003. Objective and subjective outcomes of a leadership intervention for fire fighters. Oral paper accepted for presentation at Work, Stress and Health Conference. Toronto, Canada.
Beaton, R., M. Oberle, J. Wicklund, A. Stevermer, and D. Owens. 2004. Evaluation of the Washington State National Pharmaceutical Stockpile Dispensing Exercise Part II—Dispensary site worker findings. Journal of Public Health Management and Practice 10:77-85.
Castro, C., and A. Adler. 2011. Military mental health training: Building resilience. In S. M. Southwick (Ed.), Resilience and mental health: Challenges across the lifespan. Cambridge, UK: Cambridge University Press.
Corneil, W., R. Beaton, S. Murphy, C. Johnson, and K. Pike. 1999. Exposure to traumatic incidents and prevalence of posttraumatic stress symptomatology
in urban fire fighters in two countries. Journal of Occupational Health Psychology 4:131-141.
Everly, G. S., Jr., D. A. Srouse, and G. S. Everly, III. 2010. The secrets of resilient leadership: When failure is not an option. Six essential characteristics for leading in adversity. DiaMedica Publishing, New York.
Gladwell, M. 2000. The tipping point. New York: Little Brown.
Kaminsky, M., O. L. McCabe, A. M. Langlieb, G. S. Everly. 2007. An evidence-informed model of human resistance, resilience, and recovery: The Johns Hopkins’ outcome-driven paradigm for disaster mental health services. Brief Treatment and Crisis Intervention 7(1):1-11.
Lewis, E., R. Beaton, J. Davis, B. Surina, and K. Harmon. 2005. Tacoma Fire Department wellness-fitness outcomes. In Compendium of Best Practices— Wellness-Fitness Program. Fairfax, VA: International Association of Fire Chiefs.
Murphy, S., R. Beaton, K. Pike, and K. Cain. 1994. Firefighters & paramedics: Years of service, job aspirations and burnout. American Association of Occupational Health Nursing Journal 42:1-7.
Seal, K. H., T. J. Metzler, K. S. Gima, D. Bertenthal, S. Maguen, and C. R. Marmar. 2009. Trends and risk factors for mental health diagnoses among Iraq and Afghanistan veterans using Department of Veterans Affairs health care, 2002-2008. American Journal of Public Health 99(9):1651-1658.
U.S. Army. 2008. Army suicide event reporting: Calendar year 2007. http://www.peaceispatriotic.org/articles/2007armySuicideEventReport.pdf (accessed February 17, 2012).
Williams, R. A., B. K. Hagerty, et al. 2004. Biopsychosocial effects of the bootstrap intervention in Navy recruits. Military Medicine 169:814-820.