technologies such as deep learning and machine learning are riding atop the utmost peak of inflated expectations for emerging technologies, as noted by the Gartner Hype Cycle, which tracks relative maturity stages for emerging technologies (Chen and Asch, 2017; Panetta, 2017) (see Figure 4-1). Without an appreciation for both the capabilities and limitations of AI technology in medicine, we will predictably crash into a “trough of disillusionment.” The greatest risk of all may be a backlash that impedes real progress toward using AI tools to improve human lives.

Over the past decade, several factors have led to increasing interest in and escalating hype of AI. There have been legitimate discontinuous leaps in computational capacity, electronic data availability (e.g., ImageNet [Russakovsky et al., 2015] and digitization of medical records), and perception capability (e.g., image recognition [Krizhevsky et al., 2017]). Just as algorithms can now automatically name the breed of a dog in a photo and generate a caption of a “dog catching a frisbee” (Vinyals et al., 2017), we are seeing automated recognition of malignant skin lesions (Esteva et al., 2017) and pathology specimens (Ehteshami et al., 2017). Such functionality is incredible but can easily lead one to mistakenly assume that the computer “knows” what skin cancer is and that a surgical excision is being considered. It is expected that an intelligent human who can recognize an object

in a photo can also naturally understand and explain the context of what they are seeing, but the narrow, applied AI algorithms atop the current hype cycle have no such general comprehension. Instead, these algorithms are each designed to complete specific tasks, such as answering well-formed multiple-choice questions.

With Moore’s law of exponential growth in computing power, the question arises whether it is reasonable to expect that machines will soon possess greater computational power than human brains (Saracco, 2018). This comparison may not even make sense with the fundamentally different architectures of computer processors and biological brains, because computers already can exceed human brains by measures of pure storage and speed (Fischetti, 2011). Does this mean that humans are headed toward a technological singularity (Shanahan, 2015; Vinge, 1993) that will spawn fully autonomous AI systems that continually self-improve beyond the confines of human control? Roy Amara, co-founder of Institute for the Future, reminds us that “we tend to overestimate the effect of a technology in the short run and underestimate the effect in the long run” (Ridley, 2017). Among other reasons however, intelligence is not simply a function of computing power. Increasing computing speed and storage makes a better calculator, but not a better thinker. For the near future at least, this leaves us with fundamental design and concept issues in (general) AI research that have remained unresolved for decades (e.g., common sense, framing, abstract reasoning, creativity; Brooks, 2017).

Explicit advertising hyperbole may be one of the most direct triggers for unintended consequences of hype. While such promotion is important to drive interest and motivate progress, it can become counterproductive in excess. Hyperbolic marketing of AI systems that will “outthink cancer” (Brown, 2017) can ultimately set the field back when confronted by the hard realities in attempting to deliver changes in actual patient lives (Ross and Swetlitz, 2017). Modern advances do reflect important progress in AI software and data, but can shortsightedly discount the “hardware” of a health care delivery system (people, policies, and processes) needed to actually execute care. Limited AI systems can fail to provide insights to clinicians beyond what they already knew, undercutting many hopes for early warning systems and screening asymptomatic patients for rare diseases (Butterfield, 2018). Ongoing research has a tendency to promote the latest technology as a cure-all (Marcus, 2018), even if there is a “regression to regression” where well-worn methods backed by a good data source can be as, or more, useful than “advanced” AI methods in many applications (Razavian et al., 2015).

A combination of technical and subject domain expertise is needed to recognize the credible potential of AI systems and avoid the backlash that will come from overselling them. Yet, there is no need for pessimism if our benchmark is improving on the current state of human health. Algorithms and AI systems cannot provide “guarantees of fairness, equitability, or even veracity” (Beam and Kohane, 2018),

but no humans can either. The “Superhuman Human Fallacy” (Kohane, 2017) is to dismiss computerized systems (or humans) that do not achieve an unrealizable standard of perfection or improve on the best performing human. For example, accidents attributed to self-driving cars receive outsized media attention even though they occur far less frequently than accidents attributed to human-driven cars (Felton, 2018). Yet, the potential outsized impact of automated technologies reasonably makes us demand a higher standard of reliability (Stewart, 2019) even if the necessary degree is unclear and may even cost more lives in opportunity cost while awaiting perfection (Kalra and Groves, 2017). In health care, it is possible to determine where even imperfect AI clinical augmentation can improve care and reduce practice variation. For example, gaps exist now where humans commonly misjudge the accuracy of screening tests for rare diagnoses (Manrai et al., 2014), grossly overestimate patient life expectancy (Christakis and Lamont, 2000; Glare et al., 2003), and deliver care of widely varied intensity in the last 6 months of life (Barnato et al., 2007; Dartmouth Atlas Project, 2018). There is no need to overhype the potential of AI in medicine when there is ample opportunity (as reviewed in Chapter 3) to address existing issues with undesirable variability, crippling costs, and impaired access to quality care (DOJ and FTC, 2015).

To find opportunities for automated predictive systems, stakeholders should consider where important decisions hinge upon humans making predictions with a clear outcome (Bates et al., 2014; Kleinberg et al., 2016). Though human intuition is powerful, it is inevitably variable without a support system. One could identify scarce interventions that are known to be valuable and use AI tools to assist in identifying patients most likely to benefit. For example, an intensive outpatient care team need not attend to everyone, but can be targeted to only those patients that AI systems predict are at high risk of morbidity (Zulman et al., 2017). In addition, there are numerous opportunities to deploy AI workflow support to assist humans to rapidly answer or complete repetitive information tasks (e.g., documentation, scheduling, and other back-office administration).

HOW COULD IMPROPER AI HURT PATIENTS AND THE HEALTH SYSTEM?

The evolution of AI techniques applied to medical-use cases parallels better processing power and cheaper storage capabilities (Deo, 2015) and the exponential increase in health data generated from scientific and clinical systems (e.g., electronic health records [EHRs], picture archiving and communication systems, and -omics) if not directly from patients (e.g., mobile sensors and social media interactions). Most of the conceptual foundations for AI are not new, but the combined advances can finally now translate theoretical models into usable technologies.

This will mark a fundamental change in the expectations for the next generation of physicians (Silver et al., 2018). Though there is much upside in the potential for the use of AI systems to improve health and health care, like all technologies, implementation does not come without certain risks. This section outlines some ways in which AI in health care may cause harm in unintended ways.

Correlation or Causation? Prediction Versus Action?

Poorly constructed or interpreted models from observational data can harm patients. Incredible advances in learning algorithms are now toppling world-class professional humans in games such as chess, go (Silver et al., 2018), poker (Brown and Sandholm, 2018), and even complex real-time strategy games (AlphaStar Team, 2019). The key distinction is that these can be reliably simulated with clear outcomes of success and failure. Such simulations allow algorithms to generate a virtually unlimited amount of data and experiments. In contrast, accurate simulations of novel medical care with predictable outcomes may well be impossible, meaning medical data collection requires high-cost, high-stakes experiments on real people. In addition, high-fidelity, reliably measured outcomes are not always achievable, because AI systems are constrained to learning from available observational health data.

The implementation of EHRs and other health information systems has provided scientists with rich longitudinal, multidimensional, and detailed records about an individual’s health data. However, these data are noisy and biased because they are produced for different purposes in the process of documenting care. Health care data scientists must be careful to apply the right types of modeling approaches based on the characteristics and limitations of the underlying data.

Correlation can be sufficient for diagnosing problems and predicting outcomes in certain cases. In most scenarios, however, patients and clinicians are not interested in just predicting outcomes given “usual care” or following a “natural history.” Often, the whole point of paying attention to health data is to intervene to change the expected outcomes.

Predictive models already help decision makers assess patient risk. However, methods that primarily learn associations between inputs and outputs can be unreliable, if not overtly dangerous when used for driving medical decisions (Schulam and Saria, 2017). There are three common reasons why this is the case. First, performance of association-based models tends to be susceptible to even minor deviations between the development and the implementation datasets. The learned associations may memorize dataset-specific patterns that do not generalize as the tool is moved to new environments where these patterns no longer hold (Subbaswamy et al., 2019). A common example of this phenomenon is shifts in provider practice with the introduction of new medical evidence,

technology, and epidemiology. If a tool heavily relies on a practice pattern to be predictive, as practice changes, the tool is no longer valid (Schulam and Saria, 2017). Second, such algorithms cannot correct for biases due to feedback loops that are introduced when learning continuously over time (Schulam and Saria, 2017). In particular, if the implementation of an AI system changes patient exposures, interventions, and outcomes (often as intended), it can cause data shifts or changes in the distribution of the data that degrade performance. Finally, it may be tempting to treat the proposed predictors as factors one can manipulate to change outcomes, but these are often misleading.

Consider, for instance, the finding discussed by Caruana et al. (2015) regarding risk of death among those who develop pneumonia. Their goal was to build a model that predicts risk of death for a hospitalized individual with pneumonia so that those at high risk could be treated and those at low risk could be safely sent home. The model applying supervised learning counterintuitively learned that patients who have asthma and pneumonia are less likely to die than patients who only have asthma. They traced the result back to an existing policy that patients who have asthma and pneumonia should be directly admitted to the intensive care unit, therefore receiving more aggressive treatment that in turn improved their prognosis (Cabitza et al., 2017). The health care system and research team noted this confounded finding, but had such a model been deployed to assess risk, then sicker patients might have been triaged to a lower level of care, putting them at greater risk. In this example, the association-based algorithm learned risk conditioned on the triage policy in the development dataset that persisted in the implementation environment. However, as providers begin to rely on these types of tools, practice patterns deviate (a phenomenon called practice policy shift) from those observed in the development data. This shift hurts the validity and reliability of the tool (Brown and Sandholm, 2018).

In another example, researchers observed that the time a lab value is measured can often be more predictive than the value itself (Agniel et al., 2018). For instance, the fact that a hospital test was done at 2:00 a.m. was more predictive of patient outcomes than the actual results of the test, because the implied emergency that prompted the test was at an unusual time. Similarly, a mortality prediction model may learn that patients visited by the chaplain have an increased risk of death (Chen and Altman, 2014; Choi et al., 2015).

Finally, a prostate screening test can be determined to be “protective” of near-term mortality, not because the actual test does anything, but because patients who receive that screening test are those who are already fairly healthy and have a longer life expectancy (Agniel et al., 2018). A model based on associations may very well be learning about the way local clinical operations run but not generalize well when moving across hospitals or units with different practice

patterns (Schulam and Saria, 2017). More broadly, both humans and predictive models can fail to generalize from training to implementation environments because of many different types of dataset shift—shift in dataset characteristics over time, in practice pattern, or across populations—posing a threat to model reliability and the safety of downstream decisions made in practice (Subbaswamy and Saria, 2018). Recent works have proposed that proactive learning techniques are less susceptible to dataset shifts (Schulam and Saria, 2017; Subbaswamy et al., 2019). These algorithms proactively correct for likely shifts in data.

In addition to learning a model once, an alternative approach is to update models over time so that they continuously adapt to local and recent data. Such adaptive algorithms offer constant vigilance and monitoring for changing behavior. However, this may exacerbate disparities when only well-resourced institutions can deploy the expertise to do so in an environment. In addition, regulation and law, as reviewed in Chapter 7, faces significant challenges in addressing approval and certification for continuously evolving systems.

Rule-based systems are explicitly authored by human knowledge engineers, encoding their understanding of an application domain into a computing inference engine. These are generally more explicit and interpretable in their intent, making these easier to audit for safety and reliability. On the other hand, they take less advantage of relationships that can be automatically inferred through data-driven models and therefore are often less accurate. Integrating domain-knowledge within learning-based frameworks, and combining these with methods for measuring and proactively eliminating bias, provides a promising path forward (Subbaswamy and Saria, 2018). Much of the literature on predictive modeling is based on black box models that memorize associations. Increases in model complexity can reduce both the interpretability and ability of the user to respond to predictions in practical ways (Obermeyer and Emanuel, 2016). As a result, these models are susceptible to unreliability, leading to harmful suggestions. Evaluating for reliability and actionability are key in developing models that have the potential to affect health outcomes. These issues are at the core of the tension between “black box” and “interpretable” model algorithms that afford end users some explanation for why certain predictions are favored.

Training reliable models depends on training datasets to be representative of the population where the model will be applied. Learning from real-world data—where insights can be drawn from patients similar to a given index patient—has the benefit of leading to inferences that are more relevant, but it is important to characterize populations where there are inadequate data to support robust conclusions. For example, a tool may show acceptable performance on average across individuals captured within a dataset but may perform poorly for specific subpopulations because the algorithm has not had enough data to learn from.

In genetic testing, minority groups can be disproportionately adversely affected when recommendations are made based on data that do not adequately represent them (Manrai et al., 2016). Test-time auditing tools that can identify individuals for whom the model predictions are likely to be unreliable can reduce the likelihood of incorrect decision making due to model bias (Schulam and Saria, 2017).

Amplification or Exacerbation?

AI systems will generally make people more efficient at what they are already doing, whether that is good or bad. Bias is not inherently undesirable, because the whole point of learning from (clinical) practices is that there is an underlying assumption that human experts are making nonrandom decisions biased toward achieving desirable effects. Machine learning relying on observational data will generally have an amplifying effect on our existing behavior, regardless of whether that behavior is beneficial or only exacerbates existing societal biases. For instance, Google Photos, an app that uses machine learning technology to organize images, incorrectly identified people with darker skin tones as “gorillas,” an animal that has historically been used as a racial slur (Lee, 2018). Another study found that machine translation systems were biased against women due to the way in which women were described in the data used to train the system (Prates et al., 2018). In another example, Amazon developed a hiring algorithm based on its prior hiring practices, which recapitulated existing biases against women (Dastin, 2018). Although some of these algorithms were revised or discontinued, the underlying issues will continue to be significant problems, requiring constant vigilance, as well as algorithm surveillance and maintenance to detect and address them (see Chapter 6). The need for continuous assessment about the ongoing safety of systems is discussed in Chapter 7, including a call for significant changes in regulatory compliance. Societal biases reflected in health care data may be amplified as automated systems drive more decisions, as further addressed in Chapters 5 and 6.

AI Systems Transparency

Transparency is a key theme underlying deeper issues related to privacy and consent or notification for patient data use, and to potential concerns on the part of patients and clinicians around being subject to algorithmically driven decisions. Consistent progress will only be feasible if health care consumers and health care systems are mutually recognized as trusted data partners.

As discussed in detail in Chapter 7, there are tensions that exist between the desire for robust data aggregation to facilitate the development and validation of novel AI models and the need to protect consumer privacy as well as demonstrate respect

for consumer preferences through informed consent or notification procedures. However, lack of transparency about data use and privacy practices runs the risk of fostering a situation that lacks clear consent when patient data are used in ways that patients do not understand, realize, or accept. Current consent practices for the use of EHRs and claims data are generally based on models focused on the Health Insurance Portability and Accountability Act (HIPAA) privacy rules, and some argue that HIPAA needs updating (Cohen and Mello, 2018). The progressive integration of other sources of patient-related data (e.g., genetic information, social determinants of health) and the facilitated access to highly granular and multidimensional data are changing the protections provided by the traditional mechanisms. For instance, with more data available, reidentification becomes easier to perform (Cohen and Mello, 2019). As discussed further in Chapter 7, regulations need to be updated and consent processes will need to be more informative of those added risks. Educating patients about the value of having their data used to help advance science and care, but also being explicit about the potential risks of data misuse or unintended negative effects is crucial.

In addition to issues related to data use transparency, peer and community review of publications that describe AI tools, with dissemination of code and source data, is necessary to support scientific reproducibility and validation. The risks of “stealth research” (Ioannidis, 2015), where claims regarding important, high-stakes medical advancements are made outside of the peer-reviewed scientific literature, are too great. While there will be claims of commercial concerns for proprietary intellectual property and even controversial concerns over “research parasites” (Longo and Drazen, 2016), some minimal level of transparency must be expected. Before clinical acceptance of systems can be expected, peer-reviewed publication of model performance and sources of training data should be expected just as much as population descriptions in randomized controlled trials. This is necessary to clarify the representativeness of any models and what populations to which they can reasonably be expected to apply.

For review of AI model development and validation, different models of accountability can be considered, such as the development of review agencies for automated AI and other systems in medicine. If not through existing structures such as the U.S. Food and Drug Administration (FDA) or Clinical Laboratory Improvement Amendments (CLIA), these can be modeled after the National Transportation Safety Board. In the latter case, such an agency has no direct enforcement authority, but in the event of any adverse event that could harm people, full disclosure of all data and information to the review board is required to ensure that the community can learn from mistakes. Refer to Chapter 7 for additional reading related to current and necessary policies and regulations in the use of AI systems in health care.

known as a “digital fingerprint,” could be generated and stored by the device as soon as an image is created. The hash would then be used to determine if the image had been altered in any way, because any modification would result in a new hash. This would require an update to hospital information technology (IT) infrastructure, which has historically been very difficult. However, a set of standards similar to ones for laboratories such as CLIA could be established to ensure that the medical imaging pipeline is secure.

Algorithmic defenses against adversarial attacks are a very active area of research within the broader machine learning community (Qiu et al., 2019; Yuan et al., 2018). As of yet, there are no defenses that have proven to be 100 percent effective, and new defenses are often broken almost as quickly as they are proposed. However, there have been successful defenses in specific domains or on specific datasets. On the handwritten digit dataset known as MNIST, several approaches have proven to be robust to adversarial attacks while retaining high levels of predictive accuracy (Kannan et al., 2018; Madry et al., 2017). It remains to be seen if some specific property of medical imaging (such as low levels of pose variance or restricted color spectrum) could be leveraged to improve robustness to adversarial attacks, but this is likely a fruitful direction for research in this area.

Both types of defenses, infrastructural and algorithmic, highlight the need for interdisciplinary teams of computer scientists, health care workers, and consumer representatives at every stage of design and implementation of these systems. Because these AI systems represent a new type of IT infrastructure, they must be treated as such and continually probed for possible security vulnerabilities. This will necessarily require deep collaborations between health care IT experts, computer scientists, the traditional health care workforce, and those the algorithm is designed to affect.

HOW COULD AI RESHAPE MEDICINE AND HEALTH IN UNINTENDED WAYS?

The examples in this chapter largely revolve around clinical cases and risks, but the implications reach far beyond to all of the application domains explored in Chapter 3. Public health, consumer health, and population health and/or risk management applications and risks are all foreseeable. Operational and administrative cases may be more viable early target areas with much more forgiving risk profiles for unintended harm, without high-stakes medical decisions depending on them. Even then, automated AI systems will have far-reaching implications for patient populations, health systems, and the workforce in terms of the efficiency and equity of delivering against the unmet and unlimited demands for health care.

changes tend to be harder on small (usually under-represented) groups who are already on the margins, amplifying existing inequity. Those who can adapt well to different economies and structures are likely those who already have better resources, education, and socioeconomic stability. Advances in health care AI technologies may well create more jobs (e.g., software development, health care system analysts) than are eliminated (e.g., data entry, scribing, scheduling), but those in the jobs that are easy to automate are unlikely to be the ones able to easily adopt the skills needed for the new jobs created. The fallout from a growing mismatch between employer skill set demand and employee training are reflected with only one in four employees feeling that they are getting training to adapt to an AI tech world (Giacomelli and Shukla, 2018).

Although the above example is on the individual worker level, even at the system level, we are likely to see increasing disparities. Well-resourced academic medical centers may be in a position to build and deploy adaptive learning AI systems, whereas smaller health care systems that care for the majority of the population are unlikely to have the resources to assemble the on-site expertise and data infrastructure needed for more than out-of-the-box systems that are subject to all of the modeling risks previously discussed.

While it is important to measure total and average improvement in human outcomes, it is equally important to also measure equitable distribution of such

benefits (e.g., Gini index [Gastwirth, 1972]). By analogy to a “food apartheid,” if we only optimize for production of total calories per acre (Haspel, 2015), all can get fatter with more empty calories, but the poor are less likely to access actual nutrition (Brones, 2018). If high-tech health care is only available and used by those already plugged in socioeconomically, such advances may inadvertently reinforce a “health care apartheid” (Thadaney and Verghese, 2019).

New Roles in an AI Future

Prior advances in technology and automation have resulted in transitions of jobs from agricultural to manufacturing to service. Where will medical workers go when even service jobs are automated? Most of the near-term changes discussed are largely applied AI in terms of data analytics for prediction, decision making, logistics, and pattern recognition. These remain unlikely to displace many human skills such as complex reasoning, judgment, analogy-based learning, abstract problem solving, physical interactions, empathy, communication, counseling, and implicit observation. There will thus likely be a shift in health care toward jobs that require direct physical (human) interaction, which are not so easily automated. The advent of the AI era will even require the creation of new job roles (Wilson et al., 2017), including

• Trainers: Teaching AI systems how to perform will require deliberate effort to evaluate and stress test them. AI systems can automate tasks and find patterns in data, but still require humans to provide meaning, purpose, and direction.
• Explainers: Advancing AI algorithms often have a “black box” nature, making suggestions without clear explanations, requiring humans versed in both the technical and application domains to explain how such algorithms can be trusted to drive practical decisions.
• Sustainers: The intelligence needs of human endeavors will continually evolve, preventing the advent of “completed” AI systems. Humans must continue to maintain, interpret, and monitor the behavior and unintended consequences of AI systems.

Deskilling of the Health Care Workforce

Even if health-related jobs are not replaced by AI, deskilling (“skill rot”) is a risk of over-reliance on computer-based systems (Cabitza et al.,2017). While clinicians may not be totally displaced, the fear is that they may lose “core competencies” considered vital to medical practice. In light of the rapid advancements of AI capabilities in reading X-rays (Beam and Kohane, 2016; Gulshan et al., 2016), will radiologists of the future be able to perform this task without the aid of

a computer? The very notion of a core competency is an evolving one that professionals will need to adapt as technology changes roles (Jha and Topol, 2016).

As the skills needed in imaging-based specialties change rapidly, radiologists and pathologists “must be willing to be displaced to avoid being replaced” (Jha and Topol, 2016). Jha and Topol (2016) articulate a future in which physicians in these specialties no longer operate as image readers as they currently do, but have evolved to be “information specialists” that manage complex AI systems and integrate the various pieces of information they might provide. Indeed, they argue that radiology and pathology will be affected by AI in such a similar manner that these specialties might be merged under the unified banner of information specialists, to more accurately reflect the skills needed by these physicians in the AI-enabled future. While this may be extreme due to the significantly different clinical information required in the two disciplines, it highlights that this era of health care is likely to be substantially disrupted and transformed.

Need for Education and Workforce Development

Advancing technologies in health care can bring substantial societal benefits, but will require significant training or retraining of the workforce for roles that emphasize where humans and machines have different strengths. The Industrial Revolution illustrated the paradox of overall technological advance and productivity growth, which first passed through a period of stagnated wages, reduced share to laborers, expanding poverty, and harsh living conditions (Mokyr, 1990). An overall beneficial shift only occurred after mass schooling and other investments in human capital to expand skills of the workforce. Such adjustments are impeded if the educational system is not able to provide the newly relevant skills.

A graceful transition into the AI era of health care that minimizes the unintended consequences of displacement will require deliberate redesign of training programs. This ranges from support for a core basis of primary education in science, technology, engineering, and math literacy in the broader population to continuing professional education in the face of a changing environment. Any professional’s job changes over time as technology and systems evolve. While complete replacement of health-related jobs by AI computer systems is unlikely, a lack of adaptation will result in a growing skill set mismatch, decreases in efficiency, and increasing cost of care delivery. In the face of the escalating complexity in medicine and computerization of data, medical training institutions already acknowledge that emphasizing rote memorization and repetition of information is suboptimal in an information age, requiring large-scale transformation. Health care workers in the AI future will need to learn how to use and interact with information systems, with foundational

education in information retrieval and synthesis, statistics and evidence-based medicine appraisal, and interpretation of predictive models in terms of diagnostic performance measures. Institutional organizations (e.g., the National Institutes of Health, health care systems, professional organizations, universities, and medical schools) should shift focus from skills that are easily replaced by AI automation to specific education and workforce development programs for work in the AI future with emphasis in science, technology, engineering, and medicine and data science skills and human skills that are hard to replace with a computer. Along with the retraining required to effectively integrate AI with existing roles, new roles will be created as well (e.g., trainers, explainers, sustainers), creating the need to develop and implement training programs to address these roles.

Moravec’s paradox notes that “it is comparatively easy to make computers exhibit adult level performance on intelligence tests or playing checkers, and difficult or impossible to give them the skills of a one-year-old when it comes to perception and mobility” (Moravec, 2018). Respectively, clinicians will need to be selected for, and emphasize training in, more distinctly “human” skills of counseling, physical examination, communication, management, and coordination.

AI System Augmentation of Human Tasks

Anxieties over the potential for automated AI systems to replace jobs rests in a false dichotomy. Humans and machines can excel in distinct ways that the other cannot, meaning that the two combined can accomplish what neither could do alone. In one example of a deep learning algorithm versus an expert pathologist identifying metastatic breast cancer, the high accuracy of the algorithm was impressive enough, but more compelling was that combining the algorithm with the human expert outperformed both (Wang et al., 2016).

HOW WILL AI TRANSFORM PATIENT, PROVIDER, AND COMPUTER INTERACTIONS?

The progressive digitization of U.S. medicine underwent a massive shift in just the last decade with the rapid adoption of EHRs spurred by the Health Information Technology for Economic and Clinical Health (HITECH) Act of 2009 (HHS, 2017). This transformation creates much of the digital infrastructure that will make AI in medicine possible, but the pace of change was so rapid that we may not have yet achieved the maturity to effectively benefit from new technology without compromising core values of the profession. Advancing AI systems will depend on massive data streams for their power, but even relatively basic billing processes, quality reporting, and business analytics that current EHRs support is

burning out a generation of clinical professionals because of increased electronic workflow requirements (Downing et al., 2018; Hill et al., 2013; Verghese, 2018).

As AI medical guidance systems driven by automated sensors increasingly direct medical care, there is concern that a result will be greater separation of patients from clinicians by digital intermediaries (Gawande, 2018). The future may see patients asking for advice and receiving direction from automated chatbots (Miner et al., 2016, 2017) while doctors and patients attentively analyze and recommend treatments for “iPatient” avatars that represent the data of their patients but are not the physical human beings (Verghese, 2008).

WHAT WILL HAPPEN TO ACCEPTANCE, TRUST, AND LIABILITY IN A HUMAN AND MACHINE AI FUTURE?

Information retrieval systems will increase democratization of medical knowledge, likely to the point where fully automated systems, chatbots, or intelligent agents are able to triage and dispense information and give health advice to patients (Olson, 2018). Less clear is how this disrupts conventions of who and what to trust. Widespread distribution of information comes with a respective risk of circulating misinformation in digital filter bubbles and echo chambers (Cashin-Garbutt, 2017).

Who is sued when something goes wrong, but all there is to point at is a faceless automation backed by a nebulous bureaucracy? Regulatory and guidance frameworks (see Chapter 7) must adapt, or leave us in an ethically ambiguous space (Victory, 2018).

HOW WILL HEALTH CARE PROVIDER ROLES BE CONCEPTUALIZED?

The classical ideal of a clinician evokes an image of a professional laying his or her stethoscope on patients for skillful examination, fulfilling a bonding and healing role. The gap between this image and reality may only widen further with advancing AI technology in medicine. The patient’s ability to tell his or her story to a live person could change in a world with voice-recognition software and AI chatbots. This may actually allow patients to be more honest in their medical interactions (Borzykowski, 2016), but could diminish one of the most effective therapeutic interventions, that of simply feeling that you are being listened to by an attentive and empathetic human being. In an AI-enabled world, the role of clinician will likely move progressively toward manager, coordinator, and counselor, challenging the classical perception of what their role is and what should be counted among one’s core competencies. Digitization of medicine is

intended to improve care delivery, particularly at the population level, but these benefits may not be felt on the frontlines of care. Instead, it can turn clinical professionals into data entry clerks, feeding data-hungry machines (optimized for billing incentives rather than clinical care). This may escalate as AI tools need even more data, amid a policy climate imposing ever more documentation requirements to evaluate and monitor metrics of health care quality.

The transition to more IT solutions, computerized data collection, and algorithmic feedback should ultimately improve the consistency of patient care quality and efficiency. However, will the measurable gains necessarily outweigh the loss of harder-to-quantify human qualities of medicine? Will it lead to different types of medical errors when health care relies on technology-driven test interpretations and care recommendations instead of human clinical assessment, interpretation, and management? These are provocative questions, but acknowledging that these are public concerns and addressing them are important from a societal perspective.

More optimistically, perhaps such advancing AI technologies can instead enhance human relationships. Multiple companies are exploring remote and automating approaches to “auto-scribe” for clinical encounters (Cashin-Garbutt, 2017), allowing patient interactions to focus on direct care instead of note-taking and data entry. Though such promise is tantalizing, it is also important to be aware of the unintended consequences or overt actions of bad actors who could exploit such passive monitoring, intruding on confidential physician–patient conversations that could make either party unwilling to discuss important issues. Health care AI developments may be better suited in the near term to back-office administrative tasks (e.g., coding, prior authorization, supply chain management, and scheduling). Rather than developing patches like scribes for mundane administrative tasks, a holistic system redesign may be needed to reorient incentives and eliminate the need for low-value tasks altogether. Otherwise, AI systems may just efficiently automate low-value tasks, further entrenching those tasks in the culture, rather than facilitating their elimination.

WHY SHOULD THIS TIME BE ANY DIFFERENT?

A special article in the New England Journal of Medicine proclaimed that

2017; see Figure 4-1) where we effectively use all information and data sources to improve our collective health. To that end are the following considerations:

1. Beware of marketing hype, but recognize real opportunities. There is no need to over-hype the potential of AI in health care when there is ample opportunity (as reviewed in Chapter 3) to address existing issues from undesirable variability, to crippling costs, to impaired access to quality care.
2. Seek out robust evaluations of model performance, utility, vulnerabilities, and bias. Developers must carefully probe models for unreliable behavior due to shifts in population, practice patterns, or other characteristics that do not generalize from the development to the deployment environment. Even within a contained deployment environment, it is important to measure robustness of machine learning approaches relative to shifts in the real-world, data-generating processes and sustain efforts to address the underlying human practices and culture from which the algorithms are learning.
3. Respective effort should be deliberately allocated to identify, mitigate, and correct biases in decision-making tools. Computers/algorithms are effective at learning statistical structure, patterns, organization, and rules in complex data sources, but they do not offer meaning, purpose, or a sense of justice or fairness. Recognize that algorithms trained on biased datasets will likely just amplify those biases (Rajkomar et al., 2018; Zou and Schiebinger, 2018).
4. Demand transparency in data collection and algorithm evaluation processes. The trade-offs between innovation and safety and between progress and regulation are complex, but transparency should be demanded along the way, as more thoroughly explored in Chapter 7.
5. Develop AI systems with adversaries (bad actors) in mind. Take inspiration from the cybersecurity industry with arms races between “white hats” versus “black hats.” Deep collaborations between “white hat” health care IT experts, computer scientists, and the traditional health care workforce are needed to sniff out system vulnerabilities and fortify them before the “black hat” bad actors identify and exploit vulnerabilities in live systems (Symantec, 2019).
6. Prioritize education reform and workforce development. A graceful transition into the AI era of medicine that minimizes displacement will require deliberate redesign of training programs and workforce development toward roles that emphasize where humans have different strengths than computers.
7. Identify synergy rather than replacement. Humans and machines can excel in distinct ways that the other cannot, meaning that the two combined can accomplish what neither could do alone. Rather than replacement, consider applications where there is limited access to a scarce resource (e.g., clinical expertise) that AI systems can relieve.

8. Use AI systems to engage, rather than stifle, uniquely human abilities. AI-based automation of mundane administrative tasks and efficient health-related operations can improve system efficiency to give more time for human patients and clinicians to do what they are better at (e.g., relationship building, information elicitation, counseling, and management). As explored in Chapter 6, avoid systems that disrupt human workflows.
9. Use automated systems to reach patients where existing health systems do not. Even as there is unique value in an in-person clinician–patient interaction, more than 90 percent of a patient’s life will not be in a hospital or doctor’s office. Automated systems and remote care frameworks (e.g., telemedicine and self-monitoring) can attend to, guide, and build patient relationships to monitor chronic health issues, meeting many who were previously not engaged at all.

REFERENCES

S. Demirci, N. Navab, S. Watanabe, S. Seno, Y. Takenaka, H. Matsuda, H. Ahmady Phoulady, V. Kovalev, A. Kalinovsky, V. Liauchuk, G. Bueno, M. M. Fernandez-Carrobles, I. Serrano, O. Deniz, D. Racoceanu, and R. Venancio. 2017. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA 318:2199–2210.

Esteva, A., B. Kuprel, R. A. Novoa, J. Ko, S. M. Sweter, H. M. Blau, and S. Thrun. 2017. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542:115–118.

Fabbri, D., M. E. Frisse, and B. Malin. 2017. The need for better data breach statistics. JAMA Internal Medicine 177(11):1696.

Felton, R. 2018. The problem isn’t media coverage of semi-autonomous car crashes. Jalopnik. https://jalopnik.com/the-problem-isn-t-media-coverage-of-semi-autonomous-car-1826048294 (accessed November 12, 2019).

Finlayson, S. G., H. W. Chung, I. S. Kohane, and A. L. Beam. 2018. Adversarial attacks against medical deep learning systems. arXiv.org. https://arxiv.org/abs/1804.05296 (accessed November 12, 2019).

Finlayson, S. G., J. D. Bowers, J. Ito, J. L. Zittrain, A. L. Beam, and I. S. Kohane. 2019. Adversarial attacks on medical machine learning. Science 363:1287–1289.

Fischetti, M. 2011. Computers versus brains. Scientific American. https://www.scientificamerican.com/article/computers-vs-brains (accessed November 12, 2019).

Frey, C., and M. Osborne. 2013. The future of employment: How susceptible are jobs to computerisation? Working Paper. Oxford Martin Programme on Technology and Employment.

Gastwirth, J. L. 1972. The estimation of the Lorenz curve and Gini index. Review of Economics and Statistics 54:306–316.

Gawande, A. 2018. Why doctors hate their computers. The NewYorker. https://www.newyorker.com/magazine/2018/11/12/why-doctors-hate-their-computers (accessed November 12, 2019).

Giacomelli, G., and P. Shukla. 2018. How AI can combat labor displacement. InformationWeek. https://www.informationweek.com/big-data/how-ai-can-combat-labor-displacement-/a/d-id/1331997 (accessed November 12, 2019).

Gillon, R. 1994. Medical ethics: Four principles plus attention to scope. BMJ 309:184–188.

Glare, P., K. Virik, M. Jones, M. Hudson, S. Eychmuller, J. Simes, and N. Christakis. 2003. A systematic review of physicians’ survival predictions in terminally ill cancer patients. BMJ 327:195–198.

Gulshan, V., L. Peng, M. Coram, M. C. Stumpe, D. Wu, A. Narayanaswamy, S. Venugopalan, K. Widner, T. Madams, J. Cuadros, R. Kim, R. Raman, P. C. Nelson, J. L. Mega, and D. R. Webster. 2016. Development and validation of a deep

learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 316:2402–2410.

Haspel, T. 2015. In defense of corn, the world’s most important food crop. The Washington Post. https://www.washingtonpost.com/lifestyle/food/in-defense-of-corn-the-worlds-most-important-food-crop/2015/07/12/78d86530-25a8-11e5-b77f-eb13a215f593_story.html (accessed November 12, 2019).

Herrendorf, B., R. Rogerson, and A. Valentinyi. 2009. Two perspectives on preferences and structural transformation. Working Paper 15416, National Bureau of Economic Research. https://doi.org/10.3386/w15416.

HHS (U.S. Department of Health and Human Services). 2017. HITECH Act enforcement interim final rule. https://www.hhs.gov/hipaa/for-professionals/special-topics/hitech-act-enforcement-interim-final-rule/index.html (accessed December 26, 2018).

Hill, R. G., Jr., L. M. Sears, and S. W. Melanson. 2013. 4000 Clicks: A productivity analysis of electronic medical records in a community hospital ED. American Journal of Emergency Medicine 31:1591–1594.

Horne, F. 2016. If we don’t change direction soon, we’ll end up where we’re going. Healthcare Management Forum 29:59–62.

Ioannidis, J. P. A. 2015. Stealth research. JAMA 313:663.

IOM (Institute of Medicine). 2013. Best care at lower cost: The path to continuously learning health care in America. Washington, DC: The National Academies Press. https://doi.org/10.17226/13444.

Jha, S., and E. J. Topol. 2016. Adapting to artificial intelligence: Radiologists and pathologists as information specialists. JAMA 316:2353–2354.

Jiang, J. X., and G. Bai. 2019. Types of information comprised in breaches of protected health information. Annals of Internal Medicine. https://doi.org/10.7326/M19-1759 (accessed November 12, 2019).

Kalra, N., and D. G. Groves. 2017. The enemy of good: Estimating the cost of waiting for nearly perfect autonomous vehicles. Santa Monica, CA: RAND Corporation. https://www.rand.org/pubs/research_reports/RR2150.html (accessed November 12, 2019).

Kannan, H., A. Kurakin, and I. Goodfellow. 2018. Adversarial logit pairing. arXiv.org. https://arxiv.org/abs/1803.06373 (accessed November 12, 2019).

Kleinberg, J., J. Ludwig, and S. Mullainathan. 2016. A guide to solving social problems with machine learning. Harvard Business Review. https://hbr.org/2016/12/a-guide-to-solving-social-problems-with-machine-learning (accessed November 12, 2019).

Kohane, I. 2017. What my 90-year-old mom taught me about the future of AI in health care. CommonHealth. https://www.wbur.org/commonhealth/2017/06/16/managing-mom-weight-algorithm.

Suggested citation for Chapter 4: Chen, J., A. Beam, S. Saria, and E. A. Mendonça. 2020. Potential trade-offs and unintended consequences of artificial intelligence. In Artificial intelligence in health care: The hope, the hype, the promise, the peril. Washington, DC: National Academy of Medicine.