National Academies Press: OpenBook

Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop—in Brief (2020)

Chapter: Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop - in Brief

Suggested Citation:"Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26034.
×

Proceedings of a Workshop


IN BRIEF

December 2020

Achieving Excellence in Sepsis Diagnosis

Proceedings of a Workshop—in Brief

On August 27, 2020, the Board on Health Care Services of the National Academies of Sciences, Engineering, and Medicine hosted a virtual workshop titled Achieving Excellence in Sepsis Diagnosis.1 The workshop featured invited speakers and discussions to examine the current scientific landscape and research opportunities for improving sepsis diagnosis within the U.S. health care system. This workshop was the first in a series of three workshops on diagnostic excellence funded by the Gordon and Betty Moore Foundation.2 This Proceedings of a Workshop—in Brief highlights presentations and discussions that occurred at the workshop.3

BACKGROUND

Harvey Fineberg from the Gordon and Betty Moore Foundation described the concept of diagnostic excellence, noting that it requires more than preventing diagnostic errors. He stated that diagnostic excellence should span the entire patient experience of the diagnostic process and consider factors such as the appropriateness and efficiency of diagnostic testing as well as workforce, health care infrastructure, and financial costs. “We are referring to the way the health system, strategies, approaches, and sequences of examination all contribute to understanding of the patient’s condition in a way that is safe, timely, efficient, and accurate,” said Fineberg. Infection—including sepsis—is one of three clinical areas4 the Moore Foundation has prioritized in its diagnostic excellence initiative because errors, delays, and inefficiencies in the diagnosis of these conditions are responsible for a disproportionate share of serious patient harm and preventable death (Newman-Toker et al., 2019).

THE VISION FOR DIAGNOSTIC EXCELLENCE IN SEPSIS

J. Michael Gaziano from the Million Veteran Program and Harvard Medical School described diagnostic accuracy across three levels: biological framework, health system structure and data, and patient experience. He said that health care is traditionally organized around individual organ systems, but noted that this framework is overly simplistic because complex and interconnected biological processes can contribute to patients’ health problems. Gaziano highlighted the critical role of data to improve diagnosis, suggesting that new strategies enabling clinicians to interpret and act on patient data will be key for ensuring high-quality diagnosis.

__________________

1 The workshop agenda and presentations are available at https://www.nationalacademies.org/event/08-27-2020/achieving-excellence-in-sepsis-diagnosis-a-workshop (accessed October 14, 2020).

2 More information about the workshop series is available at https://www.nationalacademies.org/our-work/advancing-diagnostic-excellence-a-workshop-series (accessed October 14, 2020).

3 This Proceedings of a Workshop—in Brief is not intended to provide a comprehensive summary of information shared during the workshop. The information summarized here reflects the knowledge and opinions of individual workshop participants and should not be seen as a consensus of the workshop participants, the planning committee, or the National Academies of Sciences, Engineering, and Medicine.

4 The other clinical areas include acute cardiovascular events and cancers.


Image

Suggested Citation:"Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26034.
×

Abraham Verghese from Stanford University said that diagnostic excellence should begin with the simplest form of diagnostic examination, “one that is evident to our eyes and our senses,” before moving to more sophisticated diagnostic techniques. He emphasized the importance of direct patient observation and physical examination, noting that clinicians’ instincts may be clouded by complex tools and technologies. Verghese also emphasized the importance of patient engagement in the diagnostic process. He noted that diagnosis, particularly for infection, is often rife with uncertainty, and that this uncertainty can be unsettling for patients who seek a definitive explanation for their illness: “The diagnosis is important in more than just the classification of disease—it is important to the patient in the embodied narrative of their particular experience of illness.”

Helen Haskell from Mothers Against Medical Error and Consumers Advancing Patient Safety also highlighted the importance of focusing on patients’ and family members’ roles in the diagnostic process. She shared the story of her young son’s death in 2000 due to complications from sepsis and internal bleeding following elective surgery. Haskell said that although she was at her son’s bedside and observed his condition, her lack of medical knowledge and inexperience navigating the hospital system limited her ability to advocate effectively for his care. She emphasized the scope of the health threat posed by sepsis in hospital settings, noting that sepsis is often the cause of adverse events and deterioration in hospitalized patients. Failure to identify sepsis continues to occur despite heightened awareness and other advancements such as rapid response teams. Haskell attributed this failure to the fragmented and silo-like nature of the health care system. She noted that in conversations with individuals who have experienced diagnostic errors, a common theme of their experiences is the failure of clinicians to “listen deeply” to patients. Haskell emphasized that patients and their families can provide critical contributions to the diagnostic process, but they need to be equipped with the tools and knowledge to recognize the symptoms of sepsis, and clinicians need to practice respectful and empathic communication with patients and their families.

Andrew Bindman from the University of California, San Francisco,5 noted that the health system’s response to the coronavirus disease 2019 (COVID-19) pandemic could inform future efforts to achieve diagnostic excellence. Fineberg agreed, noting that the pandemic has increased the urgency and public awareness of the importance of accurate and timely diagnosis, especially for an infectious disease with significant public health implications. He also noted that the COVID-19 pandemic has raised public awareness about the trajectory of disease course and the importance of monitoring for symptoms that may suggest worsening illness, which may also be relevant for public awareness of sepsis. Gaziano added that COVID-19 has also illustrated the need to better understand heterogeneity in disease presentation among patients. Haskell added that patient experiences should inform research and care, and noted that many patients with “long-haul” COVID-19 are dismissed without taking into account their symptoms and input.

CHALLENGES AND OPPORTUNITIES IN SEPSIS DIAGNOSIS

Manu Shankar-Hari from Guy’s and St. Thomas’ NHS Foundation Trust and King’s College London provided an overview of sepsis and sepsis diagnosis. He described sepsis as a medical condition that occurs when infection triggers a dysregulated systemic immune response that can rapidly lead to tissue damage, organ failure, and even death. He highlighted the complexity and heterogeneity of sepsis, noting that most infections are uncomplicated and it is difficult to identify the subset of patients who will go on to develop sepsis and face life-threatening illness. The concept of sepsis has evolved over time, with a set of consensus definitions first published in 1992 (Bone et al., 1992). These definitions were recently revised, with sepsis defined as “life-threatening organ dysfunction caused by a dysregulated host response to an infection” (Singer et al., 2016) and with septic shock defined as a “subset of sepsis in which underlying circulatory, cellular, and metabolic abnormalities are associated with a greater risk of mortality than sepsis alone” (Shankar-Hari et al., 2016). Shankar-Hari noted that there are no gold standard tests for sepsis or its clinical features, which adds to the difficulty of obtaining an accurate diagnosis. He also noted that, while sepsis is often thought to follow a bacterial infection, it can also occur with viral infections, as has been evident with COVID-19 (Laing et al., 2020).

Kristina Rudd from the University of Pittsburgh reviewed epidemiological trends in sepsis, noting that in 2017 there were an estimated 48.9 million cases of sepsis and 11 million deaths (Rudd et al., 2020). In 2017, sepsis accounted for approximately 20 percent of all deaths worldwide (Rudd et al., 2020). While incidence varies among countries, Rudd noted that countries with lower-quality health care generally have a higher incidence of sepsis.

In the United States, approximately 1.1 million patients are diagnosed with sepsis each year, and approximately 200,000 individuals die as a result (Rhee et al., 2017). Susceptibility to sepsis varies by demographic characteristics and health status—risk factors include advanced age, frailty (Fernando et al., 2019), comorbidities (Beck et al., 2016),

__________________

5 As of November 16, 2020, Andrew Bindman is affiliated with Kaiser Permanente.

Suggested Citation:"Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26034.
×

and residence in medically underserved areas (Goodwin et al., 2016). Rudd also noted that there are meaningful racial and ethnic disparities in sepsis. For example, Black Americans have a higher risk of sepsis than White Americans (Kempker et al., 2018).

Emanuel Rivers from Henry Ford Hospital emphasized the importance of early sepsis diagnosis and intervention to improve patient outcomes. He noted that as recently as the 1990s there were no standard procedures for early sepsis care. Over time, the paradigm for sepsis care has shifted, with a growing understanding that sepsis occurs in settings outside the intensive care unit (ICU), including the emergency department (ED) and general inpatient settings. Rivers quoted Peter Safar, stating “critical care is not a location, it is a process. It takes place not only in the ICU but everywhere” (Safar, 1974). He said this shift toward identifying sepsis and delivering critical care throughout the hospital changed the landscape of sepsis care, facilitating earlier diagnosis and treatment.

Rivers noted that sepsis mortality has consistently declined over the past two decades, and attributed this success in part to lessons learned from a seminal trial followed by several influential clinical trials that emphasized the importance of early diagnosis and treatment (Mouncey et al., 2015; Rivers et al., 2001; The ARISE Investigators and the ANZICS Clinical Trials Group, 2014; The ProCESS Investigators, 2014). He noted that policy changes have also contributed to lower mortality. For example, the New York State Department of Health found a 4 percent mortality reduction and decreased length of hospitalization following implementation of a statewide policy to improve the early recognition and treatment of severe sepsis and septic shock (Levy et al., 2018; Seymour et al., 2017).

LEVERAGING NOVEL BIOMARKERS AND MOLECULAR DIAGNOSTICS

Steven Opal from Brown University said that quickly and accurately identifying pathogens is a key component of effective sepsis care because it enables the treatment to be tailored to the patient’s infection. He noted that the time required to identify pathogens via traditional laboratory techniques is often too long for patients who are in septic shock. In recent years, however, hospitals have successfully deployed novel technologies such as mass spectroscopy for rapid detection. Opal noted that another challenge in tailoring therapies is the need to identify resistance or susceptibility of bacterial infections to antimicrobial agents. Opal suggested that using targeted therapies is important because it improves patient outcomes and avoids the overuse of antibiotics, which can contribute to antimicrobial resistance (AMR). He noted that AMR is expected to become a public health crisis in the coming decades, with economic costs projected to grow to $100 trillion worldwide within 30 years (Review on Antimicrobial Resistance, 2014).

Nuala Meyer from the University of Pennsylvania discussed strategies for characterizing the aberrant immune response in sepsis, noting that although the current sepsis definition describes the condition as a dysregulated host response to infection (Singer et al., 2016), the field has yet to fully characterize a well-regulated immune response to infection. Meyer said that dysregulated inflammation has historically been identified as a hallmark of systemic immune dysregulation. However, patients’ responses to infection are often heterogeneous and immune system tolerance for inflammation may vary. She noted that criteria for systemic inflammatory response syndrome (SIRS) are often used in sepsis diagnosis, but these criteria are neither specific to infection nor adequately sensitive—they may fail to capture 12 percent of patients with septic shock (Kaukonen et al., 2015). Meyer said that the inflammatory process in response to sepsis is biologically complex, with pro-inflammatory and anti-inflammatory cascades occurring simultaneously (Angus and van der Poll, 2013). Therefore, single biomarkers are unlikely to capture the full extent of a patient’s immune dysregulation. She suggested that future diagnostic strategies should integrate data from multiple biomarkers to fully characterize dysregulation in sepsis and improve the quality of prognostic assessments.

“Biomarkers have been called the perfect answers for sepsis, but we don’t actually know what the questions are,” said John Marshall from the University of Toronto. He noted that more than 200 unique biomarkers have been proposed for sepsis, yet few are routinely used in clinical practice. Marshall described three uses for biomarkers of organ dysfunction: prognostication (prediction of an individual’s risk for organ dysfunction), diagnosis (early detection of organ dysfunction), and monitoring (identifying response to therapy and resolution of organ dysfunction). Marshall emphasized that identifying and applying biomarkers of organ dysfunction in sepsis are complex because organ dysfunction is an outcome of sepsis, adding that biological mechanisms of organ dysfunction are still poorly understood. Thus, although biomarkers of organ dysfunction hold promise for earlier intervention and improved patient outcomes, their application in clinical practice is premature.

Timothy Sweeney from Inflammatix, Inc., described his experience developing novel diagnostic tools for sepsis. He explained that because sepsis is characterized by both infection and life-threatening organ dysfunction, there is a need for novel diagnostics to identify either the presence of an infection or the severity of the condition (Gunsolus et al., 2019). Sweeney noted that even when there is clear clinical evidence of infection it is often difficult to identify infectious pathogens. Novel technologies such as genetic sequencing can improve identification, but pathogens often

Suggested Citation:"Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26034.
×

remain unidentified because they are not easily accessible. A study of 2,300 patients with pneumonia found that even after a full suite of microbiological testing, researchers failed to identify a causative pathogen in 62 percent of cases (Jain et al., 2015). Sweeney suggested that researchers could develop artificial intelligence (AI) tools to identify sepsis from electronic health records (EHRs). However, this effort is complicated by the lack of a gold standard for sepsis diagnosis that can be used to train algorithms. Sweeney stressed the importance of considering timing and workflow when developing novel diagnostic strategies, because even perfect diagnostic tools are not useful in clinical practice if they fail to deliver actionable results within an appropriate timeframe. Sweeney stressed that diagnostic test developers need to look beyond the quality of the test itself to ensure that it can be appropriately integrated within clinical workflow to improve patient outcomes.

USING MACHINE LEARNING AND ARTIFICIAL INTELLIGENCE TOOLS

Christopher Seymour from the University of Pittsburgh School of Medicine described available data sources and strategies for improving data quality when applying machine learning (ML) and AI strategies to sepsis diagnosis. He reviewed and assessed four sources of data: clinicians, structured data from EHRs (i.e., data entered in predefined fields), unstructured data from EHRs (i.e., data entered in free-text form, such as clinical notes), and emerging data types (e.g., protein markers and microbiomes). Each of these four data types has limitations in quality, including problems with reliability and validity. For example, it is difficult to directly obtain clinicians’ assessments, and there may be differences in their classifications of disease. Structured data fields from EHRs might be more widely available and present lower data curation costs, but they may lack reliability and validity and often fail to capture information salient for the diagnosis of sepsis (e.g., time-sensitive indicators may not be recorded with sufficient frequency). Unstructured data from EHRs capture detailed information about patients’ health, but require the application of natural language processing technologies to be adapted for ML purposes. Emerging data types are often poorly validated and may not be cost effective, given the need to generate large volumes of data to support ML algorithm development. Because of these limitations, Seymour concluded that there is no optimal single data source, and any data source used for ML will require some compromise in reliability, validity, cost, or availability. He recommended that processes for algorithm development be explicit about data limitations, and that data types should be prioritized based on the context in which the algorithm will be implemented (e.g., clinical care, research, surveillance, or quality improvement).

Shamim Nemati from the University of California, San Diego, described his work developing an ML model to enable real-time sepsis prediction for patients in the ICU. Nemati and colleagues (2018) developed an algorithm by retrospectively applying sepsis diagnostic criteria to a cohort of 30,000 patients. A primary goal of the project was to improve clinicians’ ability to interpret the result of the algorithm by identifying the primary factors contributing to each patient’s risk score. The algorithm was then adapted for real-time application in a format that could be implemented across different health care systems. Nemati said the developers collaborated with ICU clinicians to ensure that the algorithm was easy to adopt in clinical practice and provided actionable information to clinicians using appropriate clinical decision-support interventions. Nemati emphasized the importance of providing sepsis prediction at the right time (i.e., prior to clinical recognition of sepsis and the initiation of treatment). He said that the utility of an algorithm goes beyond its technical performance and classification capabilities—it is also important to consider clinical workflow and interpretability to ensure that novel tools can be effectively integrated in clinical practice.

Raj Ratwani from MedStar Health described the importance of considering human factors in ML and AI, defining human factors as “a multidisciplinary science that focuses on understanding human capabilities and designing technology systems and processes to meet these capabilities.” He defined the “AI reality gap” as the disparity between the design of ML and AI tools and the ways they are implemented in clinical practice. Ratwani emphasized the importance of developing clinical decision-support tools via user-centered design, focusing on end users’ needs and incorporating their feedback in product design, development, implementation, and long-term use (Filice and Ratwani, 2020). He said that it is also critical to consider clinical workflow in algorithm development to ensure that actionable information is provided to the appropriate clinicians while avoiding information overload.

Suchi Saria from Johns Hopkins University discussed the importance of strategies for implementing ML and AI in sepsis diagnosis and described her team’s effort implementing a Targeted Real-time Early Warning System (TREWS), a tool for the early detection of sepsis across the ED and inpatient care units at the Johns Hopkins Health System (Henry et al., 2015). She suggested that interdisciplinary teams—including expertise in human factors, quality improvement, and engineering—should be used to optimize the implementation of algorithms in clinical practice. Saria agreed with Ratwani that tying ML into workflow to provide added value for users is a key strategy for real-world deployment. She suggested that interpretability, accuracy, and timeliness are key criteria for promoting the adoption of clinical alerts, noting that implementations that have lacked these criteria have had limited success (Giannini et al., 2019). Improving accuracy and timeliness require improving data quality for algorithm development (Saria and Henry, 2020) and newer

Suggested Citation:"Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26034.
×

learning techniques that are less susceptible to bias and false alerting (Subbaswamy and Saria, 2020). Reinforcing other comments on the utility of diagnostic tools and algorithms, Saria added that ML and AI tools are only useful when they are trusted and effectively implemented by clinicians.

SYSTEM IMPROVEMENT EFFORTS

Hallie Prescott from the University of Michigan and Veterans Affairs Ann Arbor Healthcare System described the role of continuous quality improvement (CQI) collaboratives in ensuring excellence in sepsis diagnosis and the importance of embracing diagnostic uncertainty in clinical decision making. She described her work with the Michigan Hospital Medicine Safety Consortium,6 a CQI initiative focused on sepsis that has been implemented across 50 hospitals. Prescott noted that the initiative focuses on collecting high-quality data, with professional EHR data abstractors embedded at each member institution. These data are then centrally analyzed and used to provide feedback on quality. She said that the CQI project has been successful for improving diagnosis and promoting appropriate treatment, including a reduction in overtreatment of infections. Prescott said that the consortium’s trustworthy and high-quality, data-driven approach have been the keys to its success; these features have built confidence in the project’s metrics and have driven behavior change to improve patient care.

Prescott also emphasized the importance of embracing diagnostic uncertainty, noting that that the diagnosis of sepsis is often subjective and varies among clinicians (Rhee et al., 2016). Diagnosis is traditionally viewed as a binary construct—sepsis or not sepsis. Prescott recommended that clinicians use a revised, risk-based nonbinary paradigm that considers both the likelihood of infection and the severity of a patient’s illness to drive treatment decision making (Prescott and Iwashyna, 2019) (see Figure 1). She noted that this framework allows clinicians to more effectively tailor treatment to individual patients.

Vincent Liu from Kaiser Permanente described health system strategies to improve sepsis diagnosis. He noted that most research on sepsis focuses on the “golden hours” of treatment, 3–6 hours after a patient presents to the health care system. However, there may be opportunities for earlier intervention and even primary prevention in the days prior to hospitalization, before a patient’s infection progresses. He described a study that found that nearly half of the patients treated for sepsis were seen by a clinician in the week prior to hospitalization (Liu et al., 2018). He suggested that novel biomarkers and predictive analytics could be developed for use in community-based settings of care to identify patients at high risk for developing sepsis and thus facilitate prevention or intervention prior to hospitalization.

Liu also suggested leveraging embedded research approaches within health systems and ensuring data interoperability for a learning system for sepsis diagnosis and care. He noted that millions of data elements are collected each day but remain in silos, preventing valuable analyses. He suggested that improvements in EHR interoperability have set the stage for the implementation of technical standards to improve sepsis diagnosis and treatment, but there is still progress to be made in strengthening data governance and security, protecting patient privacy, and ensuring broader implementation and uptake. Uniting data elements and allowing information to flow seamlessly across applications can allow health care systems to improve patient care by learning “dynamically from every patient that comes in the door.”

Jeremy Kahn from the University of Pittsburgh discussed policy approaches to improving sepsis diagnosis. He first described the four main policy levers that can be used for quality improvement in general, including market incentives, resource allocation, regulation, and financial incentives. He noted that these levers have been underutilized in

Image
FIGURE 1 An alternate risk-based approach to the diagnosis of sepsis.
SOURCES: Hallie Prescott presentation, August 27, 2020; adapted from Prescott and Iwashyna, 2019.

__________________

6 For more information, see https://www.mi-hms.org (accessed November 10, 2020).

Suggested Citation:"Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26034.
×

policies to improve sepsis diagnosis, although not without reason. In particular, the difficulty of measuring diagnostic quality is a key barrier to addressing diagnostic excellence via health care policy. Kahn stressed that better characterization and measurement of diagnostic quality is a necessary foundation for incentivizing health systems and clinicians to improve diagnosis. Two primary components of diagnostic quality include efficiency and accuracy, but he noted that these elements can be in conflict with each other, which adds to the challenge of assessing and rewarding diagnostic quality. He added that, while implementing new policies to improve sepsis diagnosis, researchers and policy makers should consider the potential for unintended consequences and take steps to prevent or mitigate negative effects. He also noted that it is important to collect data on many aspects of the diagnostic process, including delayed diagnoses and overuse of diagnostic testing and evaluation.

Kahn went on to propose two shorter-term policy solutions for improving diagnostic quality: adopting holistic policies that address the full clinical process from diagnosis through resolution, and rewarding team-based care, including performance incentives for nonphysician clinicians. As an example of a holistic policy applied to sepsis diagnosis, he described New York State’s sepsis regulations, which include protocols for the recognition and treatment of sepsis, staff education, reporting of both process and outcome data, and communication with patients’ families (New York State, 2015, 2018). Kahn said that this policy, implemented in 2013, has been associated with a dramatic reduction in sepsis mortality in part because it included clinician education as a core component (Kahn et al., 2019).

Workshop presenters also discussed how systems-level interventions can mitigate existing disparities in sepsis diagnosis. Kahn emphasized that health care equity should be considered in the development and implementation of any new tools or policies for improving sepsis diagnosis. Prescott agreed, noting that the improvements in sepsis mortality achieved through the New York State regulations have been disproportionate across the population, with less improvement seen in lower-resourced hospitals. “Just setting a benchmark is not sufficient. We need to set it up so every hospital is able to improve,” Prescott said. Liu emphasized the importance of providing information to empower patients, families, and communities—through groups such as the Sepsis Alliance and the Rory Staunton Foundation—to help them recognize and act on early signs of sepsis. Prescott also advocated for improving patient–clinician communication by encouraging clinicians to be transparent and honest about the challenges of diagnostic uncertainty, and by clearly describing symptoms that may be a sign of sepsis requiring urgent medical care.

A VISION FOR THE FUTURE OF SEPSIS DIAGNOSIS

In a final session, workshop participants reflected on their visions for diagnostic excellence in sepsis. Daniel Yang from the Gordon and Betty Moore Foundation noted that the workshop had highlighted various tensions in sepsis diagnosis, particularly the tension between accuracy and timeliness. He added that diagnostic excellence in sepsis requires patient centeredness and cost effectiveness in addition to accuracy and timeliness in diagnosis. “I think the only way to define [diagnostic excellence] is by capturing all these elements,” while also ensuring that focusing on one area does not have unintended consequences elsewhere, said Yang. He added that diagnostic excellence should be framed as “a longitudinal journey as opposed to just a single decision at a single point in time.” For example, he advocated for processes and structures to safely send patients home from the ED, so if they develop signs and symptoms of sepsis, there are mechanisms to seek clinician input and additional care as needed.

Building on Yang’s comments, Laura Evans from the University of Washington also highlighted competing priorities in sepsis diagnosis. She suggested that clinicians should be more explicit about the trade-offs required in diagnosis and be honest with patients and families about the goals of care to better align treatment with patients’ values. Evans noted that success in diagnosis requires moving beyond discipline-based silos to integrate novel strategies that use biomarkers, ML, CQI, and molecular diagnostics. She recommended that funding be designed to support this integration and collaboration.

“I would love to imagine that 5 years from now, no rehab hospital in the country has patients with missing limbs because of sepsis,” said Christine Goeschel from MedStar Health. She emphasized the importance of building a culture in health care that acknowledges diagnostic errors and learns from them, as well as the importance of realigning health systems to focus on patients and their needs. Goeschel offered four “Cs” to improve care: context, culture, communication, and cost. She noted that context can affect the success of interventions to improve diagnosis and called for a culture of transparency and clinician support. She added that effective communication is needed among all parties, including clinicians and patients, clinicians and administrators, and clinicians and researchers. Goeschel emphasized the urgent need to improve diagnosis in sepsis, stating, “everyone needs to start because many people died today of sepsis … and we cannot forget that.”

Foster Gesten, a health care consultant, echoed Evans’s suggestions for teamwork and cross-disciplinary collaboration. Gesten suggested that success in diagnostic excellence should be defined by improvements in patient and population outcomes. He added that new diagnostic tools should be assessed based on their ability to inform treat-

Suggested Citation:"Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26034.
×

ment decision making and whether they meaningfully reduce diagnostic error and patient harm. Gesten said researchers and clinicians should prioritize vulnerable populations and ensure that novel tools for improving sepsis diagnosis and treatment are useful in diverse settings of care. He suggested looking beyond mortality outcomes to consider outcomes that more fully capture diagnostic excellence, including the overuse of diagnostic testing and treatment, missing diagnoses, and delays in diagnosis.

Michael Howell from Google Health considered the role of early recognition and treatment of infection as a potential way of eliminating sepsis. However, he said he was circumspect about the potential of current AI technologies. “There is fundamental ambiguity baked into sepsis and sepsis diagnoses,” he noted, and “AI is not going to help with that.” He stated that ML cannot solve fundamental problems such as missing data. Instead, Howell suggested that ML and AI could be applied in the short term to identify disease patterns and phenotypes, noting that these technologies excel at identifying patterns that may be obscure to human observers. He also suggested that current technologies could be applied to promote EHR interoperability and data sharing.

Considering how resources should be invested to improve diagnosis in sepsis, Goeschel suggested simultaneously targeting diverse areas of intervention, including technology, policy, and education. She suggested developing education campaigns to teach patients and families to recognize the signs and symptoms of sepsis to promote earlier intervention. Gesten agreed about the importance of sepsis awareness and patient activation, calling for a better system for delivering discharge instructions to patients and tools (e.g., telemedicine and digital health) that enable patients to contribute health data for diagnosis from home. Derek Angus from the University of Pittsburgh said that sepsis has traditionally been viewed as the purview of clinicians in the critical care setting, but there may be valuable opportunities to engage and integrate a community health perspective that emphasizes primary and secondary prevention at home and in outpatient settings. Individual participants made further suggestions for achieving excellence in sepsis diagnosis (see Box 1).

Bindman concluded the workshop by recommending that the field take a multimodal approach, investing simultaneously in system-level improvements and technologies such as biomarkers and ML. He also emphasized the importance of patient-centered diagnosis, and of incorporating patients and families as key members of the diagnostic team.

Suggested Citation:"Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26034.
×

REFERENCES

Angus, D. C., and T. van der Poll. 2013. Severe sepsis and septic shock. The New England Journal of Medicine 369(9):840–851. doi: 10.1056/NEJMra1208623.

Beck, M. K., A. B. Jensen, A. B. Nielsen, A. Perner, P. L. Moseley, and S. Brunak. 2016. Diagnosis trajectories of prior multi-morbidity predict sepsis mortality. Scientific Reports 6(1):36624. doi: 10.1038/srep36624.

Bone, R. C., R. A. Balk, F. B. Cerra, R. P. Dellinger, A. M. Fein, W. A. Knaus, R. M. H. Schein, and W. J. Sibbald. 1992. Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis. The American College of Chest Physicians/Society of Critical Care Medicine Consensus Conference Committee. Chest 101(6):1644–1655. doi: 10.1378/chest.101.6.1644.

Filice, R. W., and R. M. Ratwani. 2020. The case for user-centered artificial intelligence in radiology. Radiology: Artificial Intelligence 2(3):e190095. doi: 10.1148/ryai.2020190095.

Fernando, S. M., D. I. McIsaac, B. Rochwerg, S. M. Bagshaw, J. Muscedere, L. Munshi, N. D. Ferguson, A. J. E. Seely, D. J. Cook, C. Dave, P. Tanuseputro, and K. Kyeremanteng. 2019. Frailty and invasive mechanical ventilation: Association with outcomes, extubation failure, and tracheostomy. Intensive Care Medicine 45(12):1742–1752. doi: 10.1007/s00134-019-05795-8.

Giannini, H. M., J. C. Ginestra, C. Chivers, M. Draugelis, A. Hanish, W. D. Schweickert, B. D. Fuchs, L. Meadows, M. Lynch, P. J. Donnelly, K. Pavan, N. O. Fishman, C. W. Hanson, 3rd, and C. A. Umscheid. 2019. A machine learning algorithm to predict severe sepsis and septic shock: Development, implementation, and impact on clinical practice. Critical Care Medicine 47(11):1485–1492. doi: 10.1097/CCM.0000000000003891.

Goodwin, A. J., N. R. Nadig, J. T. McElligott, K. N. Simpson, and D. W. Ford. 2016. Where you live matters: The impact of place of residence on severe sepsis incidence and mortality. Chest 150(4):829–836. doi: 10.1016/j.chest.2016.07.004.

Gunsolus, I. L., T. E. Sweeney, O. Liesenfeld, and N. A. Ledeboer. 2019. Diagnosing and managing sepsis by probing the host response to infection: Advances, opportunities, and challenges. Journal of Clinical Microbiology 57(7):e00425-19. doi: 10.1128/JCM.00425-19.

Henry, K. E., D. N. Hager, P. J. Pronovost, and S. Saria. 2015. A targeted real-time early warning score (TREWScore) for septic shock. Science Translational Medicine 7(299):299ra122. doi: 10.1126/scitranslmed.aab3719.

Jain, S., W. H. Self, R. G. Wunderink, S. Fakhran, R. Balk, A. M. Bramley, C. Reed, C. G. Grijalva, E. J. Anderson, D. M. Courtney, J. D. Chappell, C. Qi, E. M. Hart, F. Carroll, C. Trabue, H. K. Donnelly, D. J. Williams, Y. Zhu, S. R. Arnold, K. Ampofo, G. W. Waterer, M. Levine, S. Lindstrom, J. M. Winchell, J. M. Katz, D. Erdman, E. Schneider, L. A. Hicks, J. A. McCullers, A. T. Pavia, K. M. Edwards, and L. Finelli. 2015. Community-acquired pneumonia requiring hospitalization among U.S. Adults. The New England Journal of Medicine 373(5):415–427. doi: 10.1056/nejmoa1500245.

Kahn, J. M., B. S. Davis, J. G. Yabes, C.-C. H. Chang, D. H. Chong, T. B. Hershey, G. R. Martsolf, and D. C. Angus. 2019. Association between state-mandated protocolized sepsis care and in-hospital mortality among adults with sepsis. JAMA 322(3):240–250. doi: 10.1001/jama.2019.9021.

Kaukonen, K.-M., M. Bailey, D. Pilcher, D. J. Cooper, and R. Bellomo. 2015. Systemic inflammatory response syndrome criteria in defining severe sepsis. The New England Journal of Medicine 372(17):1629–1638. doi: 10.1056/NEJMoa1415236.

Kempker, J. A., M. R. Kramer, L. A. Waller, and G. S. Martin. 2018. Risk factors for septicemia deaths and disparities in a longitudinal U.S. cohort. Open Forum Infectious Diseases 5(12):ofy305. doi: 10.1093/ofid/ofy305.

Laing, A. G., A. Lorenc, I. del Molino del Barrio, A. Das, M. Fish, L. Monin, M. Muñoz-Ruiz, D. R. McKenzie, T. S. Hayday, I. Francos-Quijorna, S. Kamdar, M. Joseph, D. Davies, R. Davis, A. Jennings, I. Zlatareva, P. Vantourout, Y. Wu, V. Sofra, F. Cano, M. Greco, E. Theodoridis, J. Freedman, S. Gee, J. N. E. Chan, S. Ryan, E. Bugallo-Blanco, P. Peterson, K. Kisand, L. Haljasmägi, L. Chadli, P. Moingeon, L. Martinez, B. Merrick, K. Bisnauthsing, K. Brooks, M. A. A. Ibrahim, J. Mason, F. Lopez Gomez, K. Babalola, S. Abdul-Jawad, J. Cason, C. Mant, J. Seow, C. Graham, K. J. Doores, F. Di Rosa, J. Edgeworth, M. Shankar-Hari, and A. C. Hayday. 2020. A dynamic COVID-19 immune signature includes associations with poor prognosis. Nature Medicine 26(10):1623–1635. doi: 10.1038/s41591-020-1038-6.

Levy, M. M., F. C. Gesten, G. S. Phillips, K. M. Terry, C. W. Seymour, H. C. Prescott, M. Friedrich, T. J. Iwashyna, T. Osborn, and S. Lemeshow. 2018. Mortality changes associated with mandated public reporting for sepsis. The results of the New York state initiative. American Journal of Respiratory and Critical Care Medicine 198(11):1406–1412. doi: 10.1164/rccm.201712-2545OC.

Liu, V. X., G. J. Escobar, R. Chaudhary, and H. C. Prescott. 2018. Healthcare utilization and infection in the week prior to sepsis hospitalization. Critical Care Medicine 46(4):513–516. doi: 10.1097/CCM.0000000000002960.

Suggested Citation:"Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26034.
×

Mouncey, P. R., T. M. Osborn, G. S. Power, D. A. Harrison, M. Z. Sadique, R. D. Grieve, R. Jahan, S. E. Harvey, D. Bell, J. F. Bion, T. J. Coats, M. Singer, J. D. Young, and K. M. Rowan. 2015. Trial of early, goal-directed resuscitation for septic shock. The New England Journal of Medicine 372(14):1301–1311. doi: 10.1056/NEJMoa1500896.

Nemati, S., A. Holder, F. Razmi, M. D. Stanley, G. D. Clifford, and T. G. Buchman. 2018. An interpretable machine learning model for accurate prediction of sepsis in the ICU. Critical Care Medicine 46(4):547–553. doi: 10.1097/ccm.0000000000002936.

Newman-Toker, D. E., A. C. Schaffer, C. W. Yu-Moe, N. Nassery, A. S. Saber Tehrani, G. D. Clemens, Z. Wang, Y. Zhu, M. Fanai, and D. Siegal. 2019. Serious misdiagnosis-related harms in malpractice claims: The “Big Three”—vascular events, infections, and cancers. Diagnosis 6(3):227–240. doi: 10.1515/dx-2019-0019.

New York State. 2015. Volume C (Title 10)/Chapter V–Medical Facilities/Subchapter A–Medical Facilities—Miniumum Standards/Article 1–General/Part 404–Integrated Outpatient Services/Title: Section 404.2–Legal Base (effective date January 1, 2015). https://regs.health.ny.gov/content/section-4042-legal-base (accessed October 21, 2020).

New York State. 2018. Volume C (Title 10)/Chapter V–Medical Facilities/Subchapter A–Medical Facilities—Miniumum Standards/Article 1–General/Part 404–Integrated Outpatient Services/Title: Section 405.4–Medical Staff. https://regs.health.ny.gov/content/section-4054-medical-staff (accessed October 21, 2020).

Prescott, H. C., and T. J. Iwashyna. 2019. Improving sepsis treatment by embracing diagnostic uncertainty. Annals of the American Thoracic Society 16(4):426–429. doi: 10.1513/AnnalsATS.201809-646PS.

Review on Antimicrobial Resistance. 2014. Antimicrobial resistance: Tackling a crisis for the health and wealth of nations. London, UK: HM Government. https://amr-review.org/sites/default/files/AMR%20Review%20Paper%20-%20Tackling%20a%20crisis%20for%20the%20health%20and%20wealth%20of%20nations_1.pdf (accessed December 1, 2020).

Rhee, C., S. S. Kadri, R. L. Danner, A. F. Suffredini, A. F. Massaro, B. T. Kitch, G. Lee, and M. Klompas. 2016. Diagnosing sepsis is subjective and highly variable: A survey of intensivists using case vignettes. Critical Care 20:89. doi: 10.1186/s13054-016-1266-9.

Rhee, C., R. Dantes, L. Epstein, D. J. Murphy, C. W. Seymour, T. J. Iwashyna, S. S. Kadri, D. C. Angus, R. L. Danner, A. E. Fiore, J. A. Jernigan, G. S. Martin, E. Septimus, D. K. Warren, A. Karcz, C. Chan, J. T. Menchaca, R. Wang, S. Gruber, M. Klompas, and CDC Prevention Epicenter Program. 2017. Incidence and trends of sepsis in U.S. hospitals using clinical vs claims data, 2009–2014. JAMA 318(13):1241–1249.

Rivers, E., B. Nguyen, S. Havstad, J. Ressler, A. Muzzin, B. Knoblich, E. Peterson, and M. Tomlanovich. 2001. Early goal-directed therapy in the treatment of severe sepsis and septic shock. The New England Journal of Medicine 345(19):1368–1377. doi: 10.1056/nejmoa010307.

Rudd, K. E., S. C. Johnson, K. M. Agesa, K. A. Shackelford, D. Tsoi, D. R. Kievlan, D. V. Colombara, K. S. Ikuta, N. Kissoon, S. Finfer, C. Fleischmann-Struzek, F. R. Machado, K. K. Reinhart, K. Rowan, C. W. Seymour, R. S. Watson, T. E. West, F. Marinho, S. I. Hay, R. Lozano, A. D. Lopez, D. C. Angus, C. J. L. Murray, and M. Naghavi. 2020. Global, regional, and national sepsis incidence and mortality, 1990–2017: Analysis for the global burden of disease study. The Lancet 395(10219):200–211. doi: 10.1016/s0140-6736(19)32989-7.

Safar, P. 1974. Critical care medicine—quo vadis? Critical Care Medicine 2(1):1–5.

Saria, S., and K. E. Henry. 2020. Too many definitions of sepsis: Can machine learning leverage the electronic health record to increase accuracy and bring consensus? Critical Care Medicine 48(2):137–141. doi: 10.1097/ccm.0000000000004144.

Seymour, C. W., F. Gesten, H. C. Prescott, M. E. Friedrich, T. J. Iwashyna, G. S. Phillips, S. Lemeshow, T. Osborn, K. M. Terry, and M. M. Levy. 2017. Time to treatment and mortality during mandated emergency care for sepsis. The New England Journal of Medicine 376(23):2235–2244. doi: 10.1056/NEJMoa1703058.

Shankar-Hari, M., G. S. Phillips, M. L. Levy, C. W. Seymour, V. X. Liu, C. S. Deutschman, D. C. Angus, G. D. Rubenfeld, M. Singer, and the Sepsis Definitions Task. 2016. Developing a new definition and assessing new clinical criteria for septic shock: For the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). JAMA 315(8):775–787. doi: 10.1001/jama.2016.0289.

Singer, M., C. S. Deutschman, C. W. Seymour, M. Shankar-Hari, D. Annane, M. Bauer, R. Bellomo, G. R. Bernard, J.-D. Chiche, C. M. Coopersmith, R. S. Hotchkiss, M. M. Levy, J. C. Marshall, G. S. Martin, S. M. Opal, G. D. Rubenfeld, T. van der Poll, J.-L. Vincent, and D. C. Angus. 2016. The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). JAMA 315(8):801–810. doi: 10.1001/jama.2016.0287.

Subbaswamy, A., and S. Saria. 2020. From development to deployment: Dataset shift, causality, and shift-stable models in health AI. Biostatistics 21(2):345–352. doi: 10.1093/biostatistics/kxz041.

The ARISE Investigators and the ANZICS Clinical Trials Group. 2014. Goal-directed resuscitation for patients with early septic shock. The New England Journal of Medicine 371(16):1496–1506. doi: 10.1056/NEJMoa1404380.

The ProCESS Investigators. 2014. A randomized trial of protocol-based care for early septic shock. The New England Journal of Medicine 370(18):1683–1693. doi: 10.1056/NEJMoa1401602.

Suggested Citation:"Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26034.
×

DISCLAIMER: This Proceedings of a Workshop—in Brief was prepared by Megan Snair, Emily Zevon, and Cyndi Trang as a factual summary of what occurred at the workshop. The statements made are those of the rapporteurs or individual workshop participants and do not necessarily represent the views of all workshop participants; the planning committee; or the National Academies of Sciences, Engineering, and Medicine.

*The National Academies of Sciences, Engineering, and Medicine’s planning committees are solely responsible for organizing the workshop, identifying topics, and choosing speakers. The responsibility for the published Proceedings of a Workshop—in Brief rests with the rapporteurs and the institution. The planning committee comprises Andrew Bindman (Chair), University of California, San Francisco; Derek Angus, University of Pittsburgh; Pascale Carayon, University of Wisconsin–Madison; Gari Clifford, Emory University; Laura Evans, University of Washington; Kathryn McDonald, Johns Hopkins University; Lyle Moldawer, University of Florida; and Saul Weingart, Tufts University.

REVIEWERS: To ensure that it meets institutional standards for quality and objectivity, this Proceedings of a Workshop—in Brief was reviewed by Vincent Liu, Kaiser Permanente, and Christine Goeschel, MedStar Health. Lauren Shern, National Academies of Sciences, Engineering, and Medicine, served as the review coordinator.

STAFF: Emily Zevon, Erin Balogh, Cyndi Trang, Anesia Wilks, and Sharyl Nass, Board on Health Care Services, Health and Medicine Division, National Academies of Sciences, Engineering, and Medicine.

SPONSOR: This workshop was supported by the Gordon and Betty Moore Foundation.

For additional information regarding the workshop, visit https://www.nationalacademies.org/event/08-27-2020/achieving-excellence-in-sepsis-diagnosis-a-workshop.

Suggested citation: National Academies of Sciences, Engineering, and Medicine. 2020. Achieving excellence in sepsis diagnosis: Proceedings of a workshop—in brief. Washington, DC: The National Academies Press. https://doi.org/10.17226/26034.

Health and Medicine Division

Image

Copyright 2020 by the National Academy of Sciences. All rights reserved.

Suggested Citation:"Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26034.
×
Page 1
Suggested Citation:"Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26034.
×
Page 2
Suggested Citation:"Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26034.
×
Page 3
Suggested Citation:"Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26034.
×
Page 4
Suggested Citation:"Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26034.
×
Page 5
Suggested Citation:"Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26034.
×
Page 6
Suggested Citation:"Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26034.
×
Page 7
Suggested Citation:"Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26034.
×
Page 8
Suggested Citation:"Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26034.
×
Page 9
Suggested Citation:"Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2020. Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26034.
×
Page 10
Achieving Excellence in Sepsis Diagnosis: Proceedings of a Workshop—in Brief Get This Book
×
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

On August 27, 2020, the Board on Health Care Services of the National Academies of Sciences, Engineering, and Medicine hosted a workshop titled Achieving Excellence in Sepsis Diagnosis. The workshop featured invited speakers and discussions to examine the current scientific landscape and research opportunities for improving sepsis diagnosis within the U.S. health care system. This publication highlights presentations and discussions that occurred at the workshop.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!