Performance management is an evolving iterative process that continuously assesses performance needs and expectations. Components of such a quality measurement and reporting system might include translating quality-of-care measurement concepts into performance-measure specifications; pilot testing the performance-measure specifications to determine their validity, reliability, feasibility, and cost; ensuring use of the performance measures and their submission to a performance-measure repository; and analyzing and displaying the performance measures in a format or formats suitable for the intended users and audiences (IOM, 2006). As outlined in Box 4-1, the National Quality Forum (NQF) recommends that government and private sector health care providers use common performance measures for both clinical measures and quality indicators (Kizer, 2000, 2001). A high-performing posttraumatic stress disorder (PTSD) management system will adopt commonly accepted and used performance measures whenever they are available, and will coordinate the use of these measures with other systems with which they interact (for example, DoD might coordinate with VA).
The collection and appropriate use of patient-specific data are essential to managing performance in any health care system. As detailed in the committee’s phase 1 report, easily administered self-report measures of PTSD symptoms are available and widely used, such as the Primary Care PTSD screen and the PTSD Checklist (PCL). Other psychosocial, symptom severity, and functional assessment tools include the Clinician-Administered PTSD Scale, the Mississippi Scale, the Connor-Davidson Resilience Scale, and the Global Assessment of Function.
BOX 4-1
National Quality Forum (NQF)
NQF is a nonprofit, public-private organization, established in 1999, that has nearly 400 member organizations. It is a voluntary consensus standards body with a mission to improve the quality of American health care by setting national priorities and goals for performance improvement, endorsing national consensus standards for measuring and publicly reporting on performance, and promoting the attainment of national goals through education and outreach programs. NQF does not develop health care performance standards but instead endorses standards that have been developed by other entities after they have been carefully reviewed against established criteria and successfully complete a rigorous consensus process (Kizer, 2000, 2001).
NQF meets the requirements of a voluntary consensus standards body, defined by the attributes of openness (i.e., broadly inclusive of interested stakeholders), balance of interest, due process, consensus, and an appeals process. The Office of Management and Budget Circular A-119 defines consensus as “general agreement, but not necessarily unanimity, and includes a process for attempting to resolve objections by interested parties.” Voluntary consensus standards are defined as technical standards such as specifications of materials, performance, design or operation; test methods; sampling procedures; and related management systems practices that are developed or adopted by voluntary consensus standards bodies. They have legal standing.
The National Technology and Transfer Advancement Act of 1995 and Circular A-119 explicitly direct federal agencies to use voluntary consensus standards, in lieu of developing their own standards, unless the consensus standards would be inconsistent with applicable law or otherwise impractical. All 18 federal agencies (including DoD Health Affairs and Veterans Health Administration) involved in providing, paying for, or regulating health care have been represented in NQF and several have been represented on its Board of Directors. DoD policy also explicitly encourages its agencies to adopt and use nongovernmental voluntary consensus standards (DoD, 2011b).
This chapter describes current and planned performance management efforts at DoD and VA. It focuses primarily on the tools that both departments use to measure and manage performance.
DEPARTMENT OF DEFENSE
This section describes the current and planned efforts by DoD to manage performance for PTSD care and identifies the challenges that the department faces in doing so. In particular, the committee focuses on the efforts of DoD and the service branches to measure PTSD treatment outcomes.
In 2009, DoD adopted a strategic performance measurement framework called the Quadruple Aim, which incorporates the three dimensions of improved quality of care—population health, a positive patient experience, and cost—and adds a fourth dimension, increased readiness (Dinneen, 2011). The framework does not define specific performance goals for the department or what metrics will be used to measure performance.
Several reports have documented that DoD does not have adequate systems in place to manage performance and to improve the quality of mental health care for service members (DoD Task Force on Mental Health, 2007; IOM, 2010, 2013). In two recent reports to Congress, DoD stated that it could improve the efficiency and effectiveness of its health care system through
- establishing system accountability, continuous innovation, access to appropriate care, information continuity, and provision of well-managed and coordinated care (DoD, 2013a);
- setting detailed and specific goals and tasks related to the performance of the system; and
- creating the Defense Health Agency (DHA) (DoD, 2013b).
To that end, DoD has mandated that by September 2014 all patient-centered medical home clinics in military treatment facilities (MTFs) will use a standard performance dashboard for the top five chronic illnesses, including anxiety and trauma-related disorders, to monitor and improve performance.
Tools
A major barrier that DoD must overcome to improve PTSD management is the lack of systematic collection, analysis, and dissemination of metrics to assess the quality of PTSD care. Metrics measure program effectiveness, quality of care, program awareness, and availability and acceptance of mental health services.
Executive Order 13625 (August 31, 2012) called for DoD to
review all of its existing mental health and substance abuse prevention, education, and outreach programs across the military services and the Defense Health Program to identify the key program areas that produce the greatest impact on quality and outcomes, and rank programs within each of these program areas using metrics that assess their effectiveness. By the end of Fiscal Year 2014, existing program resources shall be realigned to ensure that highly ranked programs are implemented across all of the military services and less effective programs are replaced.
No specific DoD policies or procedures stipulate the use of measurement-based care for PTSD. In an effort to assess mental health outcomes across DoD, the assistant secretary of defense for health affairs released guidance on clinical outcomes for mental health in MTFs (Woodson, 2013). The guidance calls for the service branches to document clinical outcomes at all points of mental health care by using standardized measures, specifically those of the Army’s Behavioral Health Data Portal (BHDP) (see Box 4-2).
Behavioral Health Data Portal
The goal of the BHDP “is to effectively engage providers to use metrics in their daily care to systemically improve the quality of care across DoD” (U.S. Army, 2013). At each visit, soldiers complete a self-report assessment by using a netbook while waiting to see a mental health care provider at an appointment. For PTSD, the patient completes measures of both military and nonmilitary trauma at initial evaluation and at every follow-up visit. The results are available to the clinician, in real time, to inform clinical decision-making. The BHDP can also track which therapies are provided
BOX 4-2
Key Functional Elements of the Behavioral Health Data Portal
- rapid check-in capability for beneficiaries using barcode scans of identification cards,
- sorting and filtering of provider and clinic patient lists,
- tracking ability for the patient care team,
- real-time graphing of clinical outcomes and symptom presentations for provider dashboards,
- reporting of readiness data from ePROFILE and eMEB,
- deployment history reports,
- warrior transition unit status and case management,
- standardized intake documentation template based on intake note structure,
- patient satisfaction data collection,
- identification and tracking of risk levels assigned by the provider,
- integration of deployment health assessment data,
- initial aggregate reporting capability, and
- the ability to create and publish different surveys for specific clinic processes.
SOURCE: U.S. Army, 2013.
and when they are delivered. Data can be aggregated at many levels to compare patients, providers, clinics, and MTFs.
As of December 2013, the BHDP was operational in specialized mental health programs at all 57 Army MTFs with an estimated 30,000–40,000 entries per month (LTC Millard Brown, U.S. Army, personal communication, December 19, 2013) and the Air Force was beginning to pilot the portal at three sites. The Navy currently uses its Psychological Health Pathways, which is similar to the BHDP, at some bases, but eventually this system will be replaced by the BHDP. The BHDP has the potential to serve DoD as a universal method of collecting routine metrics for mental health care, including for PTSD, in an electronic format to improve its quality and effectiveness. At present, no evaluation of the data is available, so its usefulness cannot yet be determined.
The Navy Bureau of Medicine and Surgery has begun quarterly assessments of compliance with the VA/DoD clinical practice guideline for PTSD in all specialty mental health clinics. No additional information, however, on how the assessments are conducted or on rates of compliance was provided (U.S. Navy, 2013).
Electronic Health Records
DoD has a universal electronic health record, but no attempt has been made to use it as a mechanism for assessing treatment outcomes in the aggregate, and purchased care providers outside DoD cannot access it. The 2012 report to Congress from the DoD/VA Interagency Program Office stated that both departments continue to work on developing a single integrated electronic health record and the Virtual Lifetime Electronic Record (VLER). The VLER Health Exchange program manages the electronic exchange of clinically relevant health information between the departments and other government and private-sector health-exchange partners. The departments have implemented the VLER Health Exchange pilot at four joint locations, partnering with private-sector health information exchange organizations and the VA at another seven locations (DoD/VA Interagency Program Office, 2011).
Challenges and Limitations
Tracking outcome measures is fundamental in ensuring quality throughout the care continuum, from prevention through treatment. DoD Instruction 6490.05 (DoD, 2011a) “Maintenance of Psychological Health in Military Operations,” directs medical and line leaders to evaluate the effectiveness of their prevention programs empirically and to collect and analyze data on the stressors and stress reactions experienced by service
members. A recent IOM report (2014) noted that DoD had numerous resilience and prevention programs that lacked an evidence base for their effectiveness or metrics to assess their impact. That report called on DoD to “dedicate funding, staffing, and logistical support for data analysis and evaluation to support performance monitoring of programs for accountability and continuous improvement.”
At its site visits, the committee found minimal or no use of outcome data to improve performance of DoD PTSD programs or services regardless of the care setting. Most PTSD programs, including specialized PTSD programs, that the committee visited did not collect and analyze outcome data. In the few cases where data were collected, personnel were almost universally not available to enter and analyze them or to evaluate and disseminate results. Resources also were not available for conducting follow-up assessments of former program participants to determine long-term outcomes. Although providers and patients often spoke in glowing terms about these programs, their effectiveness in the long and short terms remains virtually unknown. DoD leaders informed the committee that standard outcome metrics are not communicated between DoD and VA to facilitate continuity of PTSD care.
DEPARTMENT OF VETERANS AFFAIRS
Unlike DoD, in which each service branch is essentially an autonomous organization, VA is a more unified health care system (described in greater detail in Chapter 3). VA has been found to perform as well as, and in many cases exceeds, the performance of other systems on measures of prevention, management of chronic diseases, and treating acute conditions (IOM and NAE, 2013).
The VA 2011–2015 strategic plan (VA, 2011) highlights the need for performance measures for assessing progress in meeting its major initiative of improving mental health care for veterans. Four of the five performance measures in the plan address PTSD, but there does not appear to be any documentation as to whether those targets had been met:
- Provide 96% of patients with a mental health evaluation within 14 days of their first (index) mental health encounter by the end of 2012. Strategic target: 96%.
- Screen 97% of eligible patients at required intervals for PTSD by the end of 2012.
- Increase the percentage of Operation Enduring Freedom (OEF) and Operation Iraqi Freedom (OIF) veterans who have a primary diagnosis of PTSD and who receive a minimum of eight psychotherapy
-
sessions within a 14-week period. FY 2011 target: 15%; strategic target: 60%.
- Increase the percentage of eligible OEF and OIF PTSD patients who are evaluated at required intervals for symptoms. FY 2011 target: to be determined; strategic target: to be determined.
A 2009 study by the RAND Corporation and the Altarum Institute found that the mental health care offered in VA was as good as or better than that available from private insurers or Medicare and Medicaid on the basis of nine measures of quality, such as the use of medications and treatment engagement. But the report also found that the quality of care varied among the veterans integrated service networks (VISNs) and that treatment for substance use disorders and care for veterans with diagnoses of PTSD or the other four diagnoses of interest was inadequate on the basis of VA’s own performance guidelines (Watkins et al., 2011). For example, only 20% of veterans who had PTSD and should have received an evidence-based treatment did so. .
The ability of VA to implement a population-based approach to PTSD care for veterans is somewhat restricted by the current eligibility regulations for enrollment into VA health care. One exception to this lack of a population-based approach is the ability of veterans who served in a theater of combat after 1998 and left active duty after January 2003 to receive health care services from the VA for 5 years after their service.
Tools
The current VA performance management system does not allow clinicians to adequately track a patient’s PTSD treatments, other than medications, or any patient outcomes in the electronic health record, so it is difficult to determine whether the psychotherapy or pharmacotherapy being used is effective and safe. To address this tracking issue, VA has developed and is implementing electronic health record documentation templates for cognitive processing therapy and prolonged exposure therapy to identify when those therapies are used. System-wide implementation of the templates was to begin in November 2013 (Office of Mental Health Operations, 2013), but as of January 2014 it was still not operational because of information technology issues (Kathleen Lysell, VA Office of Mental Health Services, personal communication, January 29, 2014). The process measures do not appear to be tied to tracking short-term and long-term patient outcomes, only whether a specific therapy was given. Although the committee believes that this data collection effort is a good step, it notes that unless this measure and the one for pharmacotherapy are tied to con
tinuous tracking of individual patient outcomes, they will not necessarily result in improved delivery of effective care.
Challenges and Limitations
The VA Northeast Program Evaluation Center (NEPEC) is responsible for conducting evaluations of VA PTSD clinical programs and specialized services. It compiles the annual The Long Journey Home, an internal report that provides detailed data on each VA specialized outpatient PTSD program (SOPP) and specialized intensive PTSD program (SIPP), presented in the aggregate, by VISN, and by individual facility. Information includes demographics (sex, race, era of service, work status, and so on), service (type of specialized treatment, prior psychiatric treatment, and prior specialized treatment), number of veterans served, number of visits, staffing, workload, direct costs and cost efficiency by program type, treatments offered in house or referred out, comorbid diagnoses, and outcome measures, such as changes in PTSD Checklist (PCL) scores if they are available (VA, 2012).
The most compelling evidence of the lost opportunity to use data from The Long Journey Home to improve program performance can be found in the lack of demonstrated improvement in specialized programs over the years. Patients in the SIPPs are assessed at program entry and 4 months after treatment completion by using the PCL, the Mississippi short-form, and the NEPEC scale for PTSD (VA, 2012); similar data are not provided for veterans in SOPPs, although PCL scores at intake are provided. Rates of patient follow-up in the SIPPs range from 0% to 100% (average, 54%) (VA, 2012). There are other data gaps in the information collected in The Long Journey Home: veterans who receive PTSD care in other venues, such as mental health clinics, are not included; data on pharmaceutical use are given only for prazosin and benzodiazapines; and the number of veterans who receive care in the medical facility versus those who are referred out for care is not reported.
Many of the SIPPs demonstrated little or no improvement over the course of years. At several of the committee’s VA site visits, clinical staff stated that although they were aware of the Long Journey Home reports, they had not seen them and thus did not use them. The Long Journey Home is an example of where routine data for at least one type of PTSD program are being collected but are not used to improve the quality of care or national standardization of these intensive, expensive programs.
SUMMARY
A high-performing PTSD management system requires well-defined performance measures and feedback mechanisms to ensure that both fa-
vorable and unfavorable activities, processes, and outcomes are recognized (such as identifying best practices), addressed, and used to improve care. The results of periodically administered measures need to be made available to clinicians to inform patient care decisions and to leaders to keep them apprised of how patients and clinicians are doing and where improvements might be made.
DoD lacks a mechanism for the systematic collection, analysis, and dissemination of data for assessing the quality of PTSD care. Metrics of program effectiveness, quality of care, program awareness, and availability and acceptance of PTSD services are needed. There are no specific DoD policies or procedures that stipulate the use of measurement-based care for PTSD and no consistent use of standardized outcome measures, before, during, or after treatment. Although the BHDP might improve the collection of patient data, it is currently being used only by the Army and no data on its effectiveness are currently available.
VA does not track the PTSD treatments a patient receives, other than medications, or any treatment outcomes in the electronic health record. This lack of performance measures makes it difficult to determine whether the psychotherapies or pharmacotherapies being used are effective and safe for treating PTSD and any comorbidities. The exceptions to the lack of data collection in the VA are the SOPPs and SIPPs, where PTSD symptoms are measured at intake but treatment outcome measures are collected only for the SIPPs at 4 months after veterans leave the programs. For several of the SIPPs, the difference in veterans’ PTSD symptoms prior to and after treatment is not substantial, and it was not clear whether or how those outcome data are used to improve the programs. Furthermore, fewer than one-third of veterans who have PTSD are treated in those specialized programs; outcome data for those treated in other VA settings are not available.
This chapter underscores the need to improve performance management in both DoD and VA. Performance metrics that can be used to track patient symptoms and outcomes are available and could be readily implemented in both departments.
REFERENCES
Dinneen, M. 2011. Military health system overview. Presentation to the Committee on the Assessment of Ongoing Efforts in the Treatment of PTSD. Washington, DC: DoD Office of Strategic Management. February 28, 2011.
DoD (Department of Defense). 2011a. DoDI 6490.05—Maintenance of psychological health in military operation. Washington, DC.
DoD. 2011b. DoDI 4120.24—Defense Standardization Program. Washington, DC. July 13, 2011.
DoD. 2013a. Response to Congress: First submission under section 731 of the National Defense Authorization Act for Fiscal Year 2013. Washington, DC.
DoD. 2013b. Response to congressional defense committees: Supplemental submission under page 177 of House Report 113-102 on National Defense Authorization Act for Fiscal Year 2014. Reference ID: 1-9843BDD.
DoD Task Force on Mental Health. 2007. An achievable vision: Report of the Department of Defense task force on mental health. Falls Church, VA: Defense Health Board.
DoD/VA (Department of Veterans Affairs) Interagency Program Office. 2011. Annual report to Congress 2011. Washington, DC.
IOM (Institute of Medicine). 2006. Improving the quality of health care for mental and substance-use conditions: Quality chasm series. Washington, DC: The National Academies Press.
IOM. 2010. Provision of mental health counseling services under TRICARE. Washington, DC: The National Academies Press.
IOM. 2013. Returning home from Iraq and Afghanistan: Assessment of readjustment needs of veterans, service members, and their families. Washington, DC: The National Academies Press.
IOM. 2014. Preventing psychological disorders in service members and their families: An assessment of programs. Washington, DC: The National Academies Press.
IOM and NAE (National Academy of Engineering). 2013. Bringing a systems approach to health. Washington, DC: Institute of Medicine and National Academy of Engineering.
Kizer, K. 2000. The national quality forum enters the game. International Journal for Quality Health Care 12(2):85-87.
Kizer, K. 2001. Establishing health care performance standards in an era of consumerism. JAMA 286(10):1213-1217.
OMHO (Office of Mental Health Operations). 2013. VA OMHO response to data request from the Committee on the Assessment of Ongoing Efforts in the Treatment of PTSD. Washington, DC. November 20, 2013.
U.S. Army. 2013. Information paper: Army behavioral health service line (BHSL). Washington, DC. October 10, 2013.
U.S. Navy. 2013. Navy and Marines response to data request from the Committee on the Assessment of Ongoing Efforts in the Treatment of PTSD. Washington, DC. November 18, 2013.
VA (Department of Veterans Affairs). 2011. Strategic plan refresh FY2011–2015. Washington, DC.
VA. 2012. The Long Journey Home XXI: Treatment of posttraumatic stress disorder (PTSD). West Haven, CT: Northeast Program Evaluation Center.
Watkins, K. E., H. A. Pincus, B. Smith, S. M. Paddock, T. E. Mannle, Jr., A. Woodroffe, J. Solomon, M. E. Sorbero, C. M. Farmer, K. A. Hepner, D. M. Adamson, L. Forrest, and C. Call. 2011. Veterans Health Administration Mental Health Program Evaluation: Capstone Report. Santa Monica, CA: RAND Corporation and Altarum Institute.
Woodson, J. 2013. Military treatment facility mental health clinical outcomes guidance. DoD Office of the Assistant Secretary of Defense for Health Affairs. Washington, DC. September 9, 2013.
