levels of automation and augmentation (see Chapters 3, 4, and 5), and regulatory oversight (see Chapters 4, 5, 6, and 7).

This publication synthesizes the major literature to date, in both the academic and general press, to create a reference document for health care AI model developers, clinical teams, patients, “fRamilies,” and regulators and policy makers to:

1. identify the current and near-term uses of AI within and outside the traditional health care systems (see Chapters 2 and 3);
2. highlight the challenges and limitations (see Chapter 4) and the best practices for development, adoption, and maintenance of AI tools (see Chapters 5 and 6);
3. understand the legal and regulatory landscape (see Chapter 7);
4. ensure equity, inclusion, and a human rights lens for this work; and
5. outline priorities for the field.

The authors of the eight chapters are experts convened by the National Academy of Medicine’s Digital Health Learning Collaborative to explore the field of AI and its applications in health and health care, consider approaches for addressing existing challenges, and identify future directions and opportunities.

This final chapter synthesizes the challenges and priorities of the previous chapters, highlights current best practices, and identifies key priorities for the field.

SUMMARY OF CHALLENGES AND KEY PRIORITIES

This section summarizes the key findings and priorities of the prior chapters without providing the underlying evidence or more detailed background. Please refer to the referenced chapters for details.

Promote Data Access, Standardization, and Reporting of Data Quality, While Minimizing Data Bias

It is widely accepted that the successful development of an AI system requires high-quality, population-representative, and diverse data (Shrott, 2017; Sun et al., 2017). Figure 8-1 outlines a standardized pathway for the collection and integration of multiple data sources into a common data model, which efficiently feeds the transformation to a feature space for AI algorithm training. However, some of the standardization tools and data quality assessments and methodologies for curating the data do not yet exist. Interoperability is critical at all layers, including across the multivendor electronic health record and ancillary

components of a health care system, between different health care systems, and with consumer health applications. We cannot disregard the fact that there are varying data requirements for the training of AI and for the downstream use of AI. Some initiatives do exist and are driving the health care community in the direction of interoperability and data standardization, but they have yet to see widespread use (HL7, 2018; Indiana Health Information Exchange, 2019; NITRD et al., 2019; OHDSI, 2019).

Methods to assess data validity and reproducibility are often ad hoc. Ultimately, for AI models to be trusted, the semantics and provenance of the data used to derive them must be fully transparent, unambiguously communicated, and available, for validation at least, to an independent vetting agent. This is a distinct element of transparency, and the conflation of data transparency with algorithmic transparency complicates the AI ecosystem’s discourse. We suggest a clear separation of these topics. One example of a principles declaration that promotes data robustness and quality is the FAIR (findability, accessibility, interoperability, and reusability) Principles (Wilkinson et al., 2016).

These principles, put forth by molecular biology and bioinformatics researchers, are not easily formalized or implemented. However, for health care AI to mature, a similar set of principles should be developed and widely adopted.

The health care community should continue to advocate for policy, regulatory, and legislative mechanisms that improve the ease of data aggregation. These would include (but are not limited to) a national patient health care identifier and mechanisms to responsibly bring together data from multiple sources. The debate should focus on the thoughtful and responsible ability of large-scale health care data resources to serve as a public good and the implications of that ability. Discussions around wider and more representative data access should be carefully balanced by stronger outreach, education, and consensus building with the public and patients in order to address where and how their data can be reused for AI research, data monetization, and other secondary uses; which entities can reuse their data; and what safeguards need to be in place. In a recent commentary, Glenn Cohen and Michelle Mello propose that “it is timely to reexamine the adequacy of the Health Insurance Portability and Accountability Act (HIPAA), the nation’s most important legal safeguard against unauthorized disclosure and use of health information. Is HIPAA up to the task of protecting health information in the 21st century?” (Cohen and Mello, 2018).

When entities bring data sources together, they face ethical, business, legislative, and technical hurdles. There is a need for novel solutions that allow for robust data aggregation while promoting transparency and respecting patient privacy and preferences.

AI to benefit the common good, at the very least its design and deployment should avoid harms to fundamental human values. International human rights provide a robust and global formulation of those values” (Latonero, 2018).

For objective governance, a new neutral agency or a committee within an existing governmental or nongovernmental entity, supported by a range of stakeholders, could own and manage the review of health care AI products and services while protecting developers’ intellectual property rights. One example of this type of solution is the New Model for Industry-Academic Partnerships, which developed a framework for academic access to industry (Facebook) data sources: The group with full access to the data is separate from the group doing the publishing, but both are academic, independent, and trusted. The group with full access executes the analytics and verifies the data, understands the underlying policies and issues, and delivers the analysis to a separate group who publishes the results but does not have open access to the data (Social Science Research Council, 2019). To ensure partisan neutrality, the project is funded by ideologically diverse supporters, including the Laura and John Arnold Foundation, the Democracy Fund, the William and Flora Hewlett Foundation, the John S. and James L. Knight Foundation, the Charles Koch Foundation, the Omidyar Network, and the Alfred P. Sloan Foundation. Research projects use this framework when researchers use Facebook social media data for election impact analysis, and Facebook provides the data required for the research but does not have the right to review or approve the research findings prior to publication.

Perhaps the best way to ensure that equity and inclusion are foundational components of a thriving health care system is to add them as a dimension to the quadruple aim, expanding it to a Quintuple Aim for health and health care: better health, improved care experience, clinician well-being, lower cost, and health equity throughout (see Figure 8-2).

Promote a Spectrum of Transparency-Based Trust, Based on Considerations of Accuracy, Risk, and Liability

A key challenge to the acceptance and widespread use of AI is the tension between data and algorithmic transparency, accuracy, perceived risk, and tort liability. One of the priorities identified in this publication is the need to present each health care AI tool along with the spectrum of transparency related to the potential harms and context of its use. Evaluating and addressing appropriate transparency in each subdomain of data, algorithms, and performance, and systematically reporting it must be a priority. In addition, health system leaders must understand the return on investment and the risks and benefits of adoption,

including the risks of adverse events post-implementation; and informatics implementers must understand the culture and workflows where AI tools will be used so the algorithms can be adjusted to reflect their needs. All stakeholders should prioritize equity and inclusion, requiring transparency on how AI tools are monitored and updated. Many of these are shared, not siloed, responsibilities.

In all cases, the transparency of the underlying data used for AI model generation should be endorsed. While granular, patient-level data should not be publicly shared, publishing information on the data sources from which they were aggregated; how the data were transformed; data quality issues; inclusion and exclusion criteria that were applied to generate the cohort; summary statistics of demographics; and relevant data features in each source should be conventional practice. This information could be a supporting document and would tremendously improve the current understanding of and trust in AI tools.

The need for algorithmic transparency is largely dependent on the use context. For applications that have immediate clinical impact on patient quality of life or health outcomes, the baseline requirement for transparency is high. However, the level of transparency could be different depending on the (1) known precision accuracy of the AI; (2) clarity of recommended actions to end users; (3) risk to the patient or target; and (4) legal liability. For example, if an AI tool has high-precision accuracy and low risk, provides clear recommendations to the

end user, and is unlikely to impose legal liability on the institution, manufacturer, or end user, then the need for complete algorithmic transparency is likely to be lower. See Figure 8-3 for additional details on the relationships of transparency and these axes within different conceptual domains.

Focus of Near-Term Health Care AI: Augmented Intelligence Versus Full Automation

Although some AI applications for health care business operations are likely to be poised for full automation, most of the near-term dialogue around AI in health care should focus on promoting, developing, and evaluating tools that support human cognition rather than replacing it. Popular culture and marketing have overloaded the term “AI” to the point where it means replacing human labor, and as a result, other terms have emerged to distinguish AI that is used to support human cognition. Augmented intelligence refers to the latter, which is the term the authors of this chapter endorse.

Mehran Sahami, a Stanford University computer science faculty member who formerly worked at Google as a senior research scientist said, “Technology is not neutral. The choices that get made in building technology then have social ramifications” (Singer, 2018).

Health care professionals have requirements for continuing education as part of their scope of practice; we suggest that new continuing education AI curricula be developed and delivered. Some important topics that should be covered are how to (1) assess the need, validity, and applicability of AI algorithms in clinical care; (2) understand algorithmic performance and the impact on downstream clinical use; (3) navigate medical liability and the ways in which AI tools may impact individual and institutional liability and medical error; (4) advocate for standardization and appropriate transparency for a given use case; (5) discuss emerging AI technologies, their use, and their dependence on patient data with patients and “fRamilies” and the patient–clinician relationship; (6) ensure the Quintuple Aim of equity and inclusion when measuring impact; and (7) know when and how to bring in AI experts for consults. As the field evolves, the nature and emphasis of these topics will change, necessitating periodic review and updating.

Professional health education should incorporate how to critically evaluate the utility and risk of these AI tools in clinical practice. Curricula should provide an understanding of how AI tools are developed, the criteria and considerations for the use of AI tools, how best to engage and use such tools while prioritizing patient needs, and when human oversight is needed.

For health care system leadership and AI implementers, it is important to have training on the importance and lenses of the multiple disciplines that must be brought together to evaluate, deploy, and maintain AI in health care.

Current clinical training programs bear the weight of growing scientific knowledge within a static time window of training. We recognize the impracticality of each clinician or team being an expert on all things health care–AI related. Instead, we propose that each team has a basic and relevant understanding as described and adds an AI consult when and where needed. Such consults could be done virtually, supporting the team effort and group decision making, and costing less than if they were done onsite. Regional or content-expert AI consults could be leveraged across many health care systems. One example of such regional consults is the National Institutes of Health–funded Undiagnosed Diseases Network (UDN), which seeks “to improve and accelerate diagnosis of rare and undiagnosed conditions (NIH, 2019). The UDN uses both basic and clinical research to improve the level of diagnosis and uncover the underlying disease mechanisms associated with these conditions.” National (or global) efforts like this can support the building and deployment of responsible AI solutions for health care.

The April 2019 collaborative effort by UnitedHealthcare and the American Medical Association to create nearly two dozen International Classification of Diseases, Tenth Revision, codes to better incorporate social determinants of health into health care delivery is a laudable and responsible step in the right direction (Commins, 2019).

AI should be considered where scale is important and resources are insufficient for current needs. Some of these environments include complex patients with multiple comorbid conditions, such as chronic disease sufferers and the elderly, or low-resource settings. For innovative telehealth—disaster relief and rural areas—when resources are limited and access difficult, triaging or auto-allocating resources can be powered by AI solutions. Current mobile technology allows for critical imaging at the local site, and the U.S. Department of Veterans Affairs has operationalized a robust telehealth program that serves its very diverse population (VA, 2016).

We strongly suggest that a robust and mature underlying information technology governance strategy be in place within health care delivery systems prior to embarking on substantial AI deployment and integration. The needs for on- or offsite hardware infrastructure, change management, inclusive stakeholder engagement, and safety monitoring all require substantial established resources. Systems that do not possess these infrastructure components should develop them before significant AI deployment.

Balancing Regulation and Legislation for Health Care Innovation

The regulatory and legislative considerations for AI use in consumer and professional health care domains are documented in Chapter 7. AI applications have great potential to improve patient health but could also pose significant risks, such as inappropriate patient risk assessment, treatment recommendations, privacy breaches, and other harms (Evans and Whicher, 2018). Overall, the field is advancing rapidly, with a constant evolution of access to data, aggregation of data, new developments in AI methods, and expansions of how and where AI is added to patient health and health care delivery. Regulators should remain flexible, but the potential for lagging legislation remains an issue.

In alignment with recent congressional and U.S. Food and Drug Administration developments and guidance, we suggest a graduated approach to the regulation of AI based on the level of patient risk, the level of AI autonomy, and how static or dynamic certain AI tools are likely to be. To the extent that machine learning–based models continuously learn from new data, regulators should adopt postmarket surveillance mechanisms to ensure continuing (and ideally, improving)

high-quality performance. Liability accrued within the deployment of various contexts of AI will continue to be a developing area as regulators, courts, and the insurance industry weigh in. Understanding regulation and liability is essential to evaluating risks and benefits.

The linkages between innovation, safety, progress, and regulation are complex. Regulators should engage in collaborative efforts with stakeholders and experts to continuously evaluate deployed clinical AI for effectiveness and safety based on real-world data. Throughout that process, transparency can help deliver well-vetted solutions. To enable both AI development and oversight, governmental agencies should invest in infrastructure that promotes wider data collection and access to data resources for building AI solutions, within a framework of equity and data protection (see Figure 8-4).

The Global Conundrum

The United States and many other nations prioritize human rights values and are appropriately measured and thoughtful in supporting data collection, AI development, and AI deployment. Other nations, with China and Russia being prime examples, have different priorities. The current AI arms race in all fields, including and beyond health care, creates a complex and, some argue, untenable geopolitical state of affairs (Apps, 2019). Others point out that it is not an AI arms race because interdependencies and interconnections among nations are needed to support research and innovation. Regardless, Kai Fu Lee outlines China’s competitive edge in AI in his 2018 book AI Superpowers: China, Silicon Valley, and the New World Order (Lee, 2018). Putin has also outlined a national

AI strategy. And in February 2019, the White House issued an Executive Order on Maintaining American Leadership in Artificial Intelligence (White House, 2019). The downstream implications of this AI arms race in health care raise questions and conundrums this publication does not cover. We acknowledge they are countless and should be investigated.

CONCLUSIONS

AI is poised to make transformative and disruptive advances in health care and could improve the lives of patients, “fRamilies” and health care professionals. However, we cannot start with an AI hammer in our hands and view every problem as the proverbial nail. When balancing the need for thoughtful, inclusive health care AI that plans for and actively manages and reduces potential unintended consequences while not yielding to marketing hype, we should be guided by the adage “haste makes waste” (Sample, 2019). The wisest guidance for AI is to start with real problems in health care that need solving, explore the best solutions for the problem by engaging relevant stakeholders, frontline users, and patients and their “fRamilies”—including AI and non-AI options—and implement and scale the ones that meet the Quintuple Aim.

In 21 Lessons for the 21st Century, Yuval Noah Harari writes, “Humans were always far better at inventing tools than using them wisely” (Harari, 2018, p. 7).

It is up to us, the stakeholders, experts, and users of these technologies, to ensure that they are used in an equitable and appropriate fashion to uphold the human values that inspired their creation—that is, better health and wellness for all.

REFERENCES

Suggested citation for Chapter 8: Matheny, M., S. Thadaney Israni, D. Whicher, and M. Ahmed. 2020. Artificial intelligence in health care: Hope not hype, promise not peril. In Artificial intelligence in health care: The hope, the hype, the promise, the peril. Washington, DC: National Academy of Medicine.