notifications to consumers after a breach of personal health record information, including information that may be collected to train, validate, or use health care AI systems (16 C.F.R. Part 318). The FTC enforces this rule.

• State tort law: When one individual or entity injures another, tort law may allow the injured individual to recover damages. Injury could result from the use of health care AI systems, including when the behavior of developers, providers, hospitals, or other health care actors falls below the standard of care. State law determines the applicable standard of care and when tort liability will exist.

We summarize each of these categories of regulatory and legal oversight by application in Table 7-1, referencing the applicable laws and regulations for different types of AI systems. Liability refers to the legal imposition of responsibility for injury through the state tort law system.

TABLE 7-1 | Typical Applicability of Various Laws and Regulations to U.S. Health Care Artificial Intelligence Systems

Clinical Urgency	Type of health care Al system	Description	Common Rule	HIPAA/Office of Civil Rights	FTC	FDCA	Liability
	Research	Al intended to assist human subjects research	X	X	X
	Operations	Al that is used to enhance clinical operations, such as patient management, scheduling, and physician documentation		X	X
	General health and wellness	Al that is used by consumers for entertainment		X	X
	Clinical: mobile engagement; health and wellness; medical device data systems	Al that is used by consumers for entertainment Al in certain categories for which FDA has announced that it does not intend to enforce FDCA requirements		X	X	X	X
	Direct-to-consumer	Al that is marketed directly to consumers and constitutes a medical device		X	X	X	X
	Clinical: informing clinical management	Al that assists physicians by informing, enhancing, aggregating, and verifying information		X	X	X	X
	Clinical: driving clinical management	Al that assists physicians by giving treatment, diagnosis, or screening advice, while relying on physician interpretation of said advice to direct patient care		X	X	X	X
	Clinical: treating or diagnosing	Autonomous Al that provides treatment or diagnoses, screens, disease without physician interpretation		X	X	X	X

AI tools should at least extend to accuracy and relevancy of data inputs and model outputs, marketing of AI systems for specific clinical indications, and transparency or auditability of clinical AI performance.

Medical Device Regulation

Some AI systems, particularly those algorithms that will perform or assist with clinical tasks related to diagnosis, interpretation, or treatment, may be classified as medical devices and fall under applicable FDA regulations. Other AI systems may instead be classified as “services” or as “products,” but not medical devices (see Box 7-1). FDA’s traditional regulatory processes for medical devices include establishment registration and listing plus premarket submissions for review and approval or clearance by FDA’s Center for Devices and Radiological Health Office of Device Evaluation or Office of In Vitro Diagnostics and Radiological Health. In the United States, the Medical Device Amendments of 1976 (P.L. 94-295) to the FDCA (21 U.S.C. § 360c) established a risk-based framework for the regulation of medical devices. The law established a three-tiered risk classification system based on the risk posed to patients should the device fail to perform as intended. The FDCA (21 U.S.C. § 360j) definition of a medical device is summarized in Box 7-1.

reasonable assurance of safety and effectiveness. Even for this highest risk class of devices, the evidence FDA requires for premarket approval has long been flexible, varying according to the characteristics of the device, its conditions of use, the existence and adequacy of warnings and other restrictions, and other factors. There is generally more flexibility in the amount of clinical evidence needed for medical devices than for drugs and biological products, because they are subject to different statutory criteria and the mechanism of action and modes of failure are generally more predictable and better characterized for devices than for drugs and biological products.

Additionally, the design process for a medical device is more often an iterative process based largely on rational design and non-clinical testing rather than clinical studies. However, this last aspect is not, in general, true for clinical AI systems. The machine learning process is itself a kind of observational research study. In some cases—particularly for medium- and high-risk clinical AIs—the design process may depend on lessons learned as such tools are deployed or on intermediate results that inform ways to improve efficacy (FDA, 2019b). The Clinical Decision Support Coalition and other organizations have recently opined that many types of clinical AI tools should not be regulated or that the industry should instead self-regulate in all application areas that FDA chooses not to enforce on the basis of their review of risks to the public health. Notably, the principles and risk-based classification processes have recently been updated to address requirements for software as a medical device (SaMD) products (see FDA, 2017c § 6.0, p. 11; IMDRF, N12 § 5.1).

It is worth noting the distinction between CDS software tools, including clinical AIs, that replace the health professional’s role in making a determination for the patient (i.e., automation) and those that simply provide information to the professional, who can then take it into account and independently evaluate it (i.e., assistance). The former may be deemed by FDA to be a medical device and subject to medical device regulations. Under the 21st Century Cures Act, if a CDS product has multiple functions, where one is excluded from the definition of a medical device and another is not, FDA can assess the safety and effectiveness to determine whether the product should be considered a medical device (21st Century Cures Act § 3060). Also, FDA can still regulate the product as a medical device if it finds that the software “would be reasonably likely to have serious adverse health consequences” or meets the criteria for a Class III medical device. Clinical AI systems that are deemed to be medical devices will generally require either De Novo or premarket approval submissions (FDA, 2018a). In some instances, where a valid pre-1976 predicate exists, a traditional 510(k) submission may be appropriate.

Note, too, that the 21st Century Cures Act’s statutory language, while already in force, is subject to implementing regulations to be developed by FDA over

time and leaves considerable ambiguity that subjects developers of clinical AI systems to FDA enforcement discretion. For example, uncertainty remains when software is being used in “supporting or providing recommendations,” or when it “enables a health care professional to independently review the basis for [its] recommendations.” FDA has issued some draft guidance (FDA, 2017b), and more guidance will undoubtedly be forthcoming. But ambiguity will likely be present nonetheless, as will the possibility of enforcement discretion.

Oversight of safety and effectiveness does not just come from regulators, whether domestic or international. In particular, diagnostic testing that is provided by laboratories and other enterprises as services is subject to oversight provided by the Clinical Laboratory Improvements Act of 1988 (CLIA, P.L. 100-578) and the Patient Safety and Quality Improvement Act of 2005 (P.L. 109-41). Certain clinical AI tools that are services rather than products may be appropriate to regulate under CLIA. It is possible that some clinical AIs—especially ones that have aspects similar to diagnostics classified as laboratory-developed tests (LDTs), developed and performed in university-based health facilities or other provider organizations—may be deployed strictly as services for patients in the care of those institutions and not marketed commercially.

FDA’s Digital Health Initiative

FDA has expressed interest in actively promoting innovation in the digital health space. FDA’s proposed Digital Health Software Precertification (Pre-Cert) Program aims to (1) substantially reduce regulatory burdens for most suppliers and operators of clinical AI systems and (2) improve the health system’s rates of responsiveness to emerging unmet health needs, including personalized medicine (FDA, 2018c).

The 21st Century CuresAct and FDA documents reflect an increasing realization that data from real-world operations are necessary for oversight. Health care information technology (IT) systems are so complex and the conditions under which clinical AI systems will operate so diverse that development, validation, and postmarket surveillance must depend on utilizing real-world data and not just clinical trials data or static, curated repositories of historical data. “Real-world data (RWD) are data relating to patient health status and/or the delivery of health care routinely collected from a variety of sources,” including electronic health record (EHR) systems (FDA, 2019c). “Real-world evidence (RWE) is the clinical evidence regarding the usage, and potential benefits or risks, of a medical product derived from analysis of RWD” (FDA, 2019c). All of these are subject to applicable HIPAA and other privacy protections such that RWD and RWE must be rigorously de-identified (El Emam, 2013) prior to use for the secondary

purposes of clinical AI development and productization. RWE and RWD are discussed in greater detail below.

The goal of FDA’s Pre-Cert Program (FDA, 2019a) is to establish voluntary, tailored, pragmatic, and least-burdensome regulatory oversight to assess software developer organizations of all sizes. The Pre-Cert Program simultaneously aims to establish trust that developers have adequate quality management system (QMS) processes in place and a culture of quality and organizational excellence such that those developers can develop and maintain safe, effective, high-quality SaMD products. The Pre-Cert Program leverages the transparency of organizational QMS compliance and product safety as well as quality metrics across the entire life cycle of SaMD. It uses a streamlined premarket review process and leverages postmarket monitoring to verify the continued safety, effectiveness, and quality performance of SaMD in the real world. The premarket review for a precertified organization’s SaMD product is informed by the organization’s precertification status, precertification level, and the SaMD’s risk category. With this program, FDA envisions leveraging the risk-category framework for SaMD developed by the International Medical Device Regulators Forum (IMDRF) to inform the risk category (FDA, 2017c, 2019b). The IMDRF framework describes the spectrum of software functions, some of which may not meet the definition of a device in Section 201(h) of the FDCA and others that may meet the definition of a device, but for which FDA has expressed that it does not intend to enforce compliance. For the purposes of the Pre-Cert Program, the application of FDA’s long-established risk category framework would remain consistent with the current definition of device under Section 201(h) of the FDCA and FDA’s current enforcement policies. The IMDRF framework establishes types and subtypes of SaMD products based on the state of the health care condition and the significance of the information provided by the products (IMDRF, 2014).

Most clinical AI systems are multielement “ensembles” of a plurality of predictive models with an evidence-combining “supervisor” module that establishes a collective answer or output from the ensemble-member models’ execution. Clinical AI involves prediction, classification, or other intelligence-related outputs. These are generated from data supplied as inputs to the model, from fewer than 10 to many hundreds of phenotypic input variables or—in the case of time-series or spectrum-analytic AI systems, image-processing AI systems, or AI systems that include genomics biomarkers—a large number of engineered features that are derived from very high-dimensional raw data inputs. Likewise, present-day genomics-based diagnostics typically involve dozens of input variables, for which there are regulatory policies and procedures that have been established for more than 10 years. These govern existing regulated diagnostics, such as in vitro diagnostic multivariate index assays (IVDMIAs) (FDA, 2007a).

Not all clinical AI systems will manifest hazards or have risk levels comparable to those associated with existing IVDMIA products. However, the methodology, review, and clearance criteria that have been found effective for the regulation of IVDMIAs may form a useful point of reference for the regulatory practice for clinical AI systems.

Clinical AI Systems That May Merit Different Regulatory Approaches

FDA has indicated that it will apply a risk-based assessment framework, where the risk level of different clinical AI systems will be influenced by the different types of on-label clinical indications and contexts in which they are intended to be used, plus the different situations in which their off-label usage might plausibly be anticipated, adopting the IMDRF framework (FDA, 2019b; IMDRF, 2014).

For example, a clinical AI system’s intended use might be as a screening test to determine the person’s susceptibility to, or propensity in the future to, develop a clinical condition or disease that has not yet materialized; this affords time for longitudinal observation, repeat testing, and vigilance to monitor signs and symptoms of the emergence of the disease and is accordingly lower risk. Similarly, an AI system designed to classify a condition’s stage or current severity, or to establish the prognosis or probable clinical course and rate of progression of a condition, functions essentially like a biomarker that characterizes risk and does so in a manner that is amenable to multiple repeat tests and observations over a period of time.

Such situations have low time sensitivity and a plurality of opportunities for the experienced clinicians to review, second-guess, and corroborate the recommendations of the screening clinical AI system. In IMDRF parlance, these are clinical AI systems that “inform” clinical management but do not “drive” clinical management. Indeed, the “informing care” function of some present-day clinical AI tools of this type is to automatically/reflexively order the appropriate standard-of-care confirmatory diagnostic testing and monitoring. These clinical AI systems provide additional evidence or advice (e.g., regarding the likelihood of the condition screened for and/or the cost-effectiveness of pursuing a diagnostic workup for the condition) and promote consistency, relevancy, and quality in diagnostic workups. In general, such screening or informational clinical AI systems will be classified as having low risk. As such, many clinical AI systems are outside the formal scope of medical device regulation and do not require establishment registration and listing or other regulatory filings (21st Century Cures Act § 3060).

By contrast, some classification, forecasting, and prognostic biomarker clinical AI algorithms that instead drive clinical management and/or involve clinical indications may be associated with a medium or high risk; the AI systems could

contain faults that cause harm via commissive or omissive errors, either directly or through clinicians’ actions or inaction. Perioperative, anesthesiology, critical care, obstetrics, neonatology, and oncology use-cases are examples of medium- or high-risk settings (Therapeutic Monitoring Systems, Inc., 2013). In such situations, there is great time sensitivity and there may be little or no time or opportunity to seek additional testing or perform more observations to assess the accuracy of the AI’s recommendation or action. In some instances, such as oncology and surgery, the decision making informed by the AI tool may lead to therapeutic actions that are not reversible and either close other therapeutic avenues or alter the clinical course of the illness and perhaps its responsiveness to subsequent therapy. Such AI tools would, by Section 3060 of the 21st Century Cures Act, be formally within the scope of medical device regulation and would require establishment registration, listing, and other regulatory filings—De Novo, 510(k), premarket approval, or precertification—and associated postmarket surveillance, reporting, and compliance procedures.

Explainability and Transparency from a Regulatory Perspective

AI systems are often criticized for being black boxes (Pasquale, 2016) that are very complex and difficult to explain (Burrell, 2016). Nevertheless, such systems can fundamentally be validated and understood in terms of development and performance (Kroll, 2018; Therapeutic Monitoring Systems, Inc., 2013), even if not in terms of mechanism—and even if they do not conform to preexisting clinician intuitions or conventional wisdom (Selbst and Barocas, 2018). Notably, the degree of “black box” lack of explainability that may be acceptable to regulators validating performance might differ from the amount of explainability clinicians demand, although the latter is an open empirical question. This chapter addresses explainability to clinicians and other nonregulators only to the extent that it interacts with regulatory requirements. Instead, the focus is largely on validation by regulators, which may be satisfied by some current development processes.

While the rest of this section focuses on how explainability and transparency may or may not be required for regulators to oversee safety and efficacy, regulators may also require explainability for independent reasons. For instance, regulators may require clinical AI tools to be explainable to clinicians to whose decision making they are coupled; to quality assurance officers and IT staff in a health provider organization who acquire the clinical AI and have risk-management/legal responsibility for their operation; to developers; to regulators; or to other humans. The European Union’s General Data Protection Regulation (GDPR) right to explanation rules, for instance, enacted in 2016 and effective

May 2018, applies to AI systems as well as humans and web services (Kaminski, 2019) and governs European Union citizens worldwide. Similar standards may be implemented in the United States and other jurisdictions. Such standards and regulations are important for public safety and for the benefits of clinical AI systems to be realized through appropriate acceptance and widespread use. However, the notion of explainability is not well defined. There is a lack of agreement about both what constitutes an adequate explanation of clinical AI tools, and to whom the explanation must be provided to conform to applicable right to explanation rules and thus be suitable for regulatory approval.

Current right to explanation regulations and standards fails to acknowledge that human data scientists, clinicians, regulators, courts, and the broader public have limitations in recognizing and interpreting subtle patterns in high-dimensional data. Certain types of AI systems are capable of learning—and certain AI models are capable of intelligently and reliably acting upon—patterns that humans are entirely and forever incapable of noticing or correctly interpreting (Seblst and Barocas, 2019). Correspondingly, humans, unable to grasp the patterns that AI recognizes, may be in a poor position to comprehend the explanations of AI recommendations or actions. As noted, the term “black box” is sometimes pejorative toward AI, especially neural networks, deep learning, and other fundamentally opaque models. They are contrasted to logistic regression; decision-tree; and other older-technology, static, deterministic models—all with low dimensionality but are able to show the inputs that led to the recommendation or action, with variables that are generally well known to the clinician and causally related.

If society, lawmakers, and regulatory agencies were to expect every clinical AI system to provide an explanation of its actions, it could greatly limit the capacity of clinical AI developers’ use of the best contemporary AI technologies, which markedly outperform older AI technology but are not able to provide explanations understandable to humans. Regulators do not currently require human-comprehensible explanations for AI in other industries that have potential risks of serious injury or death. For example, autonomous vehicles are not required to provide a running explanation or commentary on their roadway actions.

While requiring explainability may not always be compatible with maximizing capacity and performance, different forms of transparency are available that might enable oversight (see Figure 7-1). For instance, transparency of the initial dataset—including provenance and data-processing procedures—helps to demonstrate replicability. Transparency of algorithm or system architecture is similarly important for regulatory oversight. When AI systems are transparent not just to the regulator but more broadly, such transparency can enable independent validation and oversight by third parties and build trust with users.

advisory output that each clinical AI system performs; (2) the versioning of the AI system’s elements that performed the transaction, traceable to the data sources; and (3) the validation and software quality-assurance testing that led to the AI systems being authorized for production and subsequent use. These logs enable the examination of the inputs, outputs, and other details in case of anomalies or harms. The logs are open to the clinician-users, employers, organizations who acquire/authorize the AI system’s deployment (e.g., health provider organizations, health plans, or public health agencies), regulators, developers, and the courts. The individual release-engineered and version-controlled instances of present-day industrial-strength clinical AI systems are identifiable and rigorously auditable, based on these SDLC controlled-document artifacts, which are maintained by the developer organization that owns the intellectual property.

For this type of clinical AI system, the regulatory agencies’ traditional submissions and compliance processes for SaMD are feasible and may not need substantial alteration (e.g., FDA, 2018b). The types of evidence required by plaintiffs, defendants, counsel, and the courts may not need substantial alteration, although the manner of distributed storage, retrieval, and other aspects of provisioning such evidence will change. Moreover, the availability of such evidence will not be significantly altered by the nature of clinical AI systems, provided that developers follow QMS and CGMPs and maintain conformity, including controlled-document artifacts retention.

Some clinical AI tools will be developed using RWD. Because RWD are messy in ways that affect the quality and accuracy of the resulting inferences, as described in Chapter 6, more rigorous requirements for auditing clinical AI systems developed and validated using RWD will need to be established.

AI Performance Surveillance and Maintenance

Several architectural and procedural aspects of machine learning–based clinical AI systems will require significant changes in regulatory compliance and submissions, in terms of scale and scope. In modern AI applications using dynamic data sources—such as clinical data streams stored in EHRs, data collected via sensor-enabled wearable devices and combined with other forms of data, and other RWD—the AI models and algorithms are likely to experience drift over time or as the algorithms are deployed across institutions whose catchment areas and epidemiology differ (dataset shift, see Quiñonero-Candela et al., 2009; Subbaswamy et al., 2019). These longitudinal drifts and shifts entail expanded design control, design assurance, and evidentiary requirements, as discussed in Chapter 6 and below. Therefore, the traditional approach of assessing performance using a static, limited dataset to make assessments about the ongoing safety of a system is inadequate with regard to

clinical AI. Continuous machine learning offers one solution for dataset shift and drift by updating with new population-specific data (FDA, 2019b). Not all clinical AI systems aim to do this, and very few clinical AI systems implement continuous learning today. The goals of learning health systems and personalized medicine do create an impetus for more continuous machine learning–based AI systems, as discussed in Chapters 3 and 6 in more detail. However, current regulatory and jurisprudential methods and infrastructure are not prepared for this.

Natural Language Processing and Text Mining AI

Unstructured notes constitute another important source of RWD, and appropriate standards for the extraction, parsing, and curation of unstructured information for clinical AI systems is therefore another open area requiring regulatory oversight. Natural language processing (NLP) algorithms and text mining are important for certain kinds of clinical AI that use unstructured data such as clinical impressions and other remarks, as discussed in Chapter 5.

There will be a need for retention and curation of the unstructured source-text documents as well as the discrete labels or concept codes and values derived by NLP from those documents. Retention of all of these is necessary because NLP algorithms may change over time. The underlying lexical reference data and parameters that govern the parser’s operation may likewise change from one release to the next. Thus, release engineering regression testing and validation of successive releases of a clinical AI model that depends on unstructured text must be able to demonstrate that the NLP subsystem continues to meet its specifications and delivers to the clinical AI model inputs that are substantially equivalent to the results it delivered for the same test cases and document content in previous releases. Furthermore, there is natural variability in how different individuals speak. Unlike physiology, factors such as culture and training affect how individuals describe a phenomenon. Clinical AI systems must be robust to these variations.

Clinical Decision Support Systems

Another architectural factor to consider when regulating clinical AI systems is that traditional CDS systems have tended to be embedded in tangible medical devices or in single-site on-premises EHR systems (Evans and Whicher, 2018). The system configurations and dated-signed records of changes in such architectures are readily auditable by users, regulators, and courts. By contrast, many contemporary AI systems are deployed on cloud-based, geographically distributed, nondeterministically parallelized, spatially arbitrary computing architectures that, at any moment, are physically unidentifiable. To create and maintain a full log of each processor that

contributed in some part to the execution of a multielement ensemble model AI is possible in principle but would likely be cost-prohibitive and too cumbersome to be practical. Therefore, the limited traceability and fundamental non-recreatability and non-retestability of a patient’s or clinician’s specific execution of an AI system that may have contained a fault or that produced errors or failures—untoward, unexpected deviations from its specifications, validation testing, and hazard analysis—may pose particular problems for regulators, courts, developers, and the public. These nondeterministic, noncollocation aspects of contemporary cloud-based AI implementations contrast with traditional criteria for tracking product changes (e.g., 510(k) supplements or adverse event reporting systems).

Hazard Identification, Risk Analysis, and Reporting Recommendations for Safe Clinical AI Systems

Identifying hazards is a necessary step to support safe system design and operations. Identifying hazardous situations requires experts to carefully and thoroughly evaluate the system via one of several methods. Successful assurance of public safety rests on (1) identifying and analyzing all significant possible scenarios that could result in accidents of differing severity, and (2) devising and documenting effective means of mitigating the scenarios’ likelihood, frequency, and severity. Although hazard identification and quantitative risk assessment are important, risk management also depends on qualitative or subjective judgments (e.g., human observation, intuition, insight regarding processes and mechanisms of causation, creativity in anticipating human actions and psychology, and domain expertise). Each of these judgments introduces biases and chances of omissions. Thus, hazard identification should be a structured process.

Traditional modes of risk assessment and hazard analysis (e.g., hazard and operability study [HAZOP] or process hazard analysis [PHA]) that have been used in regulation of medical devices for decades can also be used for clinical AI systems (ISO, 2009, 2016). However, new hazard types related to geographic dispersity and the dynamic, nondeterministic execution of cloud-based clinical AI systems and machine learning mean that new risks must be evaluated and new mitigations must be devised, tested, and documented. Clinical AI systems may exhibit emergent properties that depend on the whole system as it evolves in time and are not specified or statically defined by the system’s parts or subsystems (Johnson, 2002; Louis and Nardi, 2018). This means that the safety of a clinical AI system, like an autonomous vehicle, cannot “solely be analyzed or verified by looking at, for example, a hardware architecture” or only one physical system instantiation (Bagschik et al., 2018). As a result, requirements “must be derived in a top-down development process which incorporates different views on a system at all levels” (Bagschik et al., 2018).

Number Needed to Treat (NNT) and Number Needed to Harm (NNH) are useful measures of population-level clinical utility of a therapeutic or a diagnostic (Cook and Sackett, 1995; Laupacis et al., 1988). A product (i.e., medication or medical device) or a health service (i.e., clinical intervention, procedure, or care process) that has a very high NNT value (>100) or that has a very low NNH value (<10) is unlikely to meet clinicians’ or consumers’ expectations of probable clinical benefit and improbable clinical harm.

The international CONsolidated Standards of Reporting Trials (CONSORT, 2010) and STARD (Standards for Reporting of Diagnostic Accuracy Studies; Bossuyt et al., 2003) initiatives pertain to the verification of diagnostic accuracy conforming to existing good clinical practice rules and guidelines (Steyerberg, 2010). While these initiatives are not focused on studies that aim to demonstrate diagnostic device equivalence, many of the reporting concepts involved are nonetheless relevant and applicable to clinical AI. The CONSORT guidelines aim to improve the reporting of randomized controlled trials, enabling reviewers to understand their design, conduct, analysis, and interpretation, and to assess the validity of their results (CONSORT, 2010). However, CONSORT is also applicable to observational, nonrandomized studies and AI derived from machine learning.

According to a 2007 FDA guidance document,

traditional FDA-approved medical device, many clinical AI systems will change over time with the addition of new data (FDA, 2019b), though for some models and developers the cost of adaptation may exceed the benefits. Especially for lower risk software, allowing market access on the basis of less substantial data on safety and efficacy and then monitoring carefully as the software is deployed in clinical practice may lead to smoother oversight that is still robust. However, prior efforts to rely on increased postmarket surveillance have encountered difficulty in developer compliance and agency enforcement (Woloshin et al., 2017), although the extent of this difficulty is contested (Kashoki et al., 2017).

As part of its Digital Health Innovation Action Plan, FDA is developing the Pre-Cert Program, in which certain developers can apply to be precertified based on a “robust culture of quality and organizational excellence” and commitment to monitoring real-world performance (FDA, 2019c). In the program as envisioned, precertified companies will be able to market lower risk SaMD without premarket review and will receive a streamlined premarket review for higher risk SaMD. FDA will work with developers to collect and interpret real-world information to ensure that the software remains safe and effective in the course of real-world use (FDA, 2019c), including the potential for updates and changes without further review (FDA, 2019b).

Companion Diagnostic Versus Complementary Diagnostic Clinical AIs

Diagnostics that inform the use of drugs, biologics, or therapeutic devices come in several regulatory forms. A companion diagnostic is sometimes required for drug/biologic/device approval (FDA, 2016b); the in vitro diagnostic device (IVD) and the associated therapeutic (i.e., drug, biologic, or other intervention) must be cross-labeled, and the IVD is thereafter used as a “gating” criterion for prescribing the therapeutic product. A complementary diagnostic, in contrast, merely provides additional information relevant to, or supplementary to and corroborative of, decisions guiding care of the patient in regard to the associated therapeutic product. Complementary diagnostics are not required for FDA approval of the associated therapeutic product, and need not be cross labeled. Finally, a combination product is a product composed of two or more regulated components produced and marketed as a single entity (21 C.F.R. § 3.2(e)). It is likely that many clinical AI systems whose hazard analyses indicate that they have medium or high risk could be successfully regulated as complementary diagnostic medical devices.

Two additional types of diagnostics are not regulated as commercially marketed products. An LDT is a type of IVD that is designed, manufactured, and used

within a single health services facility for the care of patients for whom named clinicians in that facility have responsibility. Diagnostic tests that are not marketed commercially beyond the therapeutics development process, clinical trials, and regulatory marketing approval are generally referred to by FDA and others as development tools. Such tests are established and overseen similarly to other development tools such as biomarkers.

Liability Under State Tort Law

State tort law also provides a source of risk and of regulatory pressure for the developers and users of clinical AI systems, as well as other AI systems that could cause injury but that are not the focus of this section. Briefly, state tort law may make the developers or users of clinical AI systems liable when patients are injured as a result of using those systems. Such liability could come in the form of malpractice liability—that is, potential lawsuits against health providers, hospitals or other health care systems, and AI system developers for performing below the standard of care (Froomkin et al., 2019). Developers could also face product liability for defects in the design or manufacturing of AI systems or for failure to adequately warn users of the risks of a particular AI system. By imposing liability for injuries caused by AI systems when those injuries could have reasonably been avoided, whether by more careful development or more careful use, tort law exerts pressure on developers.

How exactly tort law will deal with clinical AI systems remains uncertain, because court decisions are retrospective and the technology is nascent. Tort law is principally grounded in state law, and its contours are shaped by courts on a case-by-case basis. This area will continue to develop. Three factors influencing tort liability are of particular note: the interaction of FDA approval and tort liability, liability insurance, and the impact of transparency on tort liability. To be clear, this area of law is still very much developing, and this section only sketches some of the ways different aspects of health care AI systems may interact with the system of tort liability.

Interaction of FDA Approval and Tort Liability

Different regulatory pathways influence the availability of state tort lawsuits against AI developers and, indirectly, the ability of state tort law (and liability insurers reacting to that law) to create independent incentives for the safe and effective development of clinical AI systems. In general, states may not establish statutory requirements that are “different from, or in addition to” FDA requirements regulating devices (21 U.S.C. § 360k). The U.S. Supreme Court has also held that this preempts certain state tort lawsuits alleging negligent design or manufacturing. For devices, including clinical AI apps, that undergo a full

premarket approval, state tort lawsuits are generally preempted under the Supreme Court’s holding in Riegel v. Medtronic, 552 U.S. 312 (2008). Nevertheless, this preemption will not apply to most AI apps, which are likely to be cleared through the 510(k) clearance pathway rather than premarket approval. Clearance under the 510(k) pathway will generally not preempt state tort lawsuits under the reasoning of Medtronic v. Lohr, 518 U.S. 470 (1996), because rather than directly determining safety and efficacy, FDA finds the new app to be equivalent to an already approved product. It is unclear what preemptive effect De Novo classification will have on preempting state tort lawsuits, because the Supreme Court emphasized both the thoroughness of premarket review and its determination that the device is safe and effective, rather than equivalent to an approved predicate device.

State tort lawsuits alleging violations of industry-wide requirements, such as CGMP or other validation requirements, are a contestable source of state tort liability. Some courts have found that lawsuits alleging violations of state requirements that parallel industry-wide requirements are preempted by federal law and that such violations may only be addressed by FDA. Other courts disagree, and the matter is currently unsettled (Tarloff, 2011). In at least some jurisdictions, if app developers violate FDA-imposed requirements, courts may find parallel duties under state law and developers may be held liable. Nevertheless, if app developers comply with all FDA-imposed industry-wide requirements, states cannot impose additional requirements.

Liability Insurance

The possibility of liability creates another avenue for regulation through the intermediary of insurance. Developers, providers, and health systems are all likely to carry liability insurance to decrease the risk of a catastrophic tort judgment arising from potential injury. Liability insurers set rules and requirements regarding what information must be provided or what practices and procedures must be followed in order to issue a policy. Although insurers are often not considered regulators, they can exert substantial, if less visible, pressure that may shape the development and use of clinical AI systems (Ben-Shahar and Logue, 2012).

Impact of Transparency on Tort Liability

Transparency and opacity also interact with tort liability. Determining causation can already be difficult in medical tort litigation, because injuries may result from a string of different actions and it is not always obvious which action or combination of actions caused the injury. Opacity in clinical AI systems may further complicate the ability of injured patients, lawyers, or providers or health

systems to determine precisely what caused the injury. Explainable algorithms may make it easier to assess tort liability, as could transparency around data provenance, training and validation methods, and ongoing oversight. Perversely, this could create incentives for developers to avoid certain forms of transparency as a way to lessen the likelihood of downstream tort liability. On the other hand, courts—or legislatures—could mandate that due care, in either the development or use of clinical AI tools, requires some form of transparency. To take a hypothetical example, a court might one day hold that when a provider relies on an algorithmic diagnosis, that provider can only exercise due care by assessing how the algorithm was validated. Developers or other intermediaries would then need to provide sufficient information to allow that assessment.

PRIVACY, INFORMATION, AND DATA

Regulation regarding patient privacy and data sharing is also highly relevant to AI development, implementation, and use, whether clinical AI or AI used for other health care purposes (“health care AI”). The United States lacks a general data privacy regime, but HIPAA includes a Privacy Rule that limits the use and disclosure of protected health information (PHI)—essentially any individually identifiable medical information—by covered entities (i.e., almost all providers, health insurers, and health data clearinghouses) and their business associates where the business relationship involves PHI (45 C.F.R. § 160.103). Covered entities and business associates may only use or disclose information with patient authorization, if the entity receives a waiver from an institutional review board or privacy board, or for one of several exceptions (45 C.F.R. § 164.502). These listed exceptions include the use and disclosure of PHI for the purposes of payment, public health, law enforcement, or health care operations, including quality improvement efforts but not including research aimed at creating generalizable knowledge (45 C.F.R. § 164.501). For health systems that intend to use their own internal data to develop in-house AI tools (e.g., to predict readmission rates or the likelihood of complications among their own patients), the quality improvement exception will likely apply. Even when the use or disclosure of information is permitted under HIPAA, the Privacy Rule requires that covered entities take reasonable steps to limit the use or disclosure to the minimum necessary to accomplish the intended purpose. While HIPAA does create protections for patient data, its reach is limited, and health information can come from many sources that HIPAA does not regulate (Price and Cohen, 2019).

A complex set of other laws may also create requirements to protect patient data. HIPAA sets a floor for data privacy, not a ceiling. State laws may be more restrictive; for instance, some states provide stronger protections for especially

sensitive information such as HIV status or substance abuse information (e.g., N. Y. Pub. Health Law § 2783). California’s Consumer Protection Act creates general protections for consumer information, including health data. And although the European Union’s GDPR focuses on actions that directly affect the European Union, it also places limits on the processing of data about EU residents, regardless of where the EU citizen resides globally, and may therefore affect the privacy practices of non-EU entities engaged in medical AI development (Marelli and Testa, 2018). The GDPR generally requires legal and real persons to collect and process only as much personal data as necessary, obtain such data only for a listed legitimate purpose or with consent, notify individuals of the receipt of data, and engage in privacy-centered policy design. Health data are especially protected under the GDPR, and their processing is prohibited unless with explicit consent or in a number of specified exceptions, such as for health operations or scientific research.

Privacy and Patient Consent Issues in Health Care AI

With regard to discrete clinical data, unstructured textual data, imagery data, waveform and time-series data, and hybrid data used in clinical AI models, the development and deployment of AI systems have complex interactions with privacy concerns and privacy law (e.g., Loukides et al., 2010). Adequate oversight of clinical AI systems must address the nature of potential privacy concerns wherever they may arise, approaches to address those concerns, and management of the potential tension between privacy and other governance concerns for clinical AI.

Initial AI Development

Privacy concerns occur in the first instance because training health care AI depends on assembling large collections of health data about patients (Horvitz and Mulligan, 2015). Health data about individuals are typically considered sensitive. Some forms of data are particularly sensitive, such as substance abuse data or sexually transmitted disease information (Ford and Price, 2016). Other forms of data raise privacy concerns about the particular individual, such as genetic data that can reveal information about family members (Ram et al., 2018). Collecting, using, and sharing patient health data raise concerns about the privacy of the affected individuals, whether those concerns are consequentialist (e.g., the possibility of future discrimination based on health status) or not (e.g., dignitary concerns about others knowing embarrassing or personal facts) (Price and Cohen, 2019). The process of collecting and sharing may also make data more vulnerable to interception or inadvertent access by other parties.

Patient data are gathered in many different contexts, including by different providers and different insurers. This diffuse data collection occurs both laterally, as patients encounter different parts of the medical system at the same time, and longitudinally, as patients shift between different medical environments over the course of time. Identifying information provides the easiest way to reassemble different parts of patient records into more comprehensive datasets that can help drive at least some forms of health care AI (e.g., long-term predictions of efficacy or mortality). When identifying information is removed from patient data, reassembly becomes harder, especially for developers with fewer resources. On the other hand, de-identification is not proof against re-identification (Ohm, 2010). Re-identification can happen at the level of the individual (via targeted efforts) or more broadly across datasets. “Data triangulation” refers to the idea that if data gatherers can collect multiple datasets that include some overlapping information, and if some of those datasets include identifying information, then data users can merge those datasets and identify individuals in the otherwise de-identified datasets (Mello and Cohen, 2018; Terry, 2017). Under current law, covered entities are limited in how they can re-identify data, since once it is re-identified it is again governed by HIPAA restrictions, but this does not govern those that are not covered entities. In addition, data-sharing agreements often include provisions prohibiting efforts at re-identification by the data recipient (Ohmann et al., 2017).

Individual consent and authorization provide the clearest possible path toward ameliorating privacy concerns but raise their own complications. When individuals know the purposes for which their information will be used and can give meaningful informed consent to those uses, privacy concerns can be limited. For machine learning and health care AI, however, future uses may be unpredictable. The revised Common Rule does allow for the provision of broad consent for unspecified future use (45 C.F.R. § 46.116). Nevertheless, systematic differences between those willing to consent to future data use and those unwilling to consent—or unable to consent because they lack that entry into the health data system—means that relying on individual authorization can introduce bias into datasets (Spector-Bagdady, 2016). Furthermore, the more meaningful the individual opportunity to consent, the higher the procedural hurdles created for the assembly of data—and the greater the likelihood of eventual bias. The Office of the National Coordinator for Health Information Technology has developed a Model Privacy Notice to “help developers convey information about their privacy and security policies” (ONC, 2018).

Data Infrastructure

The availability of data is an underlying legal and regulatory challenge for clinical AI system development, which requires large amounts of data for training

and validation purposes. Once particular AI systems are deployed in the real world, RWD should be collected to ensure that the AI systems are performing well and, ideally, to improve that performance. However, numerous hurdles exist to the collection of sufficient data (Price, 2016). Various privacy laws, as described above, restrict the collection of identifiable information, and de-identified information can be difficult to assemble to capture either long-term effects or data across different data sources. Informed consent laws, such as the Common Rule for federally funded research or the consent requirements incorporated into the GDPR, create additional barriers to data collection. Even where privacy or informed consent rules do not actually prohibit the collection, use, or sharing of data, some health care actors may limit such actions out of an abundance of caution, creating a penumbra of data limitations. In addition, for those actors who do find ways around these requirements, criticism and outrage may arise if patients feel they are inadequately compensated for their valuable data. On an economic level, holders of data have strong incentives to keep data in proprietary siloes to derive competitive advantage, leading to more fragmentation of data from different sources. For data holders who wish to keep data proprietary for economic reasons, referencing privacy concerns can provide a publicly acceptable reason for these tactics.

At least four possibilities emerge for collection of data, with some evidence of each in current practice:

1. Large individual data holders: Some large holders of individual data possess enough data to train AI models on their own, such as health systems (e.g., Partners or Ascension), health care payers (e.g., United Healthcare or Humana), or tech/data companies (e.g., Google or Apple).
2. Data brokers and collaboration: The collaboration or collection of data from different sources is possible, but these endeavors often encounter the hurdles described above, which may introduce limitations on data sources or bias in the incorporation process.
3. Failure to collect data: In some instances, no actor may have the incentive or ability to collect and gather data. This may be a problem especially for otherwise underserved populations, whose data may be under-represented in AI development and monitoring efforts.
4. Government data infrastructure: Governmental agencies can collect health data as part of an effort to support future innovation in clinical AI (among other efforts). The Precision Medicine Initiative’s All of Us cohort is an example of such an effort (Frey et al., 2016; NIH, 2019), as is the U.S. Department of Veterans Affairs’ Million Veteran Program (Gaziano et al., 2016), although the latter has more restrictive policies for its data use (Price, 2017b).

and sustainably deployed, implemented in EHR systems, and curated over time to maintain adequate accuracy and reliability. However, clinical AI systems could potentially pose risks in terms of inappropriate treatment recommendations, privacy breaches, or other harms (Evans and Whicher, 2018), and some types of clinical AI systems will be classified by regulatory agencies as SaMDs, subject to premarket clearance or approval and other requirements that aim to protect the public’s health. Other clinical AI tools may be deemed to be LDT-type services, subject to CLIA and similar regulations. Whatever agency is involved in oversight, compliance with regulations should be mandatory rather than voluntary, given the potential for problematic incentives for system developers (Evans and Whicher, 2018). As the law and policy of health care AI systems develop over time, it is both expected and essential that multiple stakeholders—including payers, patients and families, policy makers, diagnostic manufacturers and providers, clinicians, academics, and others—remain involved in helping determine how best to ensure that such systems advance the quintuple aim and improve the health care system more generally.

• The black box nature of a clinical AI system should not disqualify a system from regulatory approval or use, but transparency, where possible, can aid in oversight and adoption and should be encouraged or potentially required. AI systems, including black box systems, should be capable of providing the users with an opportunity to examine quantitative evidence that the recommendation in the current situation is indeed the best recent historical choice, supplying de-identified, aggregated data sufficient for the user to satisfy the user’s interest in confirming that this is so, or is at least no worse and no more uncertain than decisions the user would take independently were the AI not involved.
• When possible, machine learning–based predictive models should be evaluated in an independent dataset (i.e., external validation) before they are adopted in the clinical practice. Risk assessment to determine the degree to which dataset-specific biases affect the model should be undertaken. Regulatory agencies should recommend specific statistical methods for evaluating and mitigating bias.
• To the extent that machine learning–based models continuously learn from new data, regulators should adopt postmarket surveillance mechanisms to ensure continuing (and ideally improving) high-quality performance.
• Regulators should engage in collaborative governance efforts with other stakeholders and experts throughout the health system, including data scientists, clinicians, ethicists, and others, to continuously evaluate deployed clinical AI for effectiveness and safety on the basis of RWD.

• Government actors should invest in infrastructure that enables equitable, high-quality data collection, such as technical standards and technological capability building.
• Government actors should continue and increase efforts to develop large, high-quality, voluntary health datasets for clinical AI development (among other purposes), such as the All of Us cohort within the Precision Medicine Initiative, while ensuring adequate measures to address patient notice and potential harms.

REFERENCES

Suggested citation for Chapter 7: McNair, D., and W. N. Price II. 2020. Health care artificial intelligence: law, regulation, and policy. In Artificial intelligence in health care: The hope, the hype, the promise, the peril. Washington, DC: National Academy of Medicine.

This page intentionally left blank.