National Academies Press: OpenBook

Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshop–in Brief (2021)

Chapter: Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshopin Brief

Suggested Citation:"Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshopin Brief." National Academies of Sciences, Engineering, and Medicine. 2021. Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26231.
×

Proceedings of a Workshop


IN BRIEF

July 2021

EMERGING AREAS OF SCIENCE, ENGINEERING, AND MEDICINE FOR THE COURTS

Proceedings of a Workshop—in Brief

INTRODUCTION

On February 24-25, 2021, an ad hoc planning committee under the auspices of the National Academies of Sciences, Engineering, and Medicine’s Committee on Science, Technology, and Law (CSTL)1 hosted a workshop titled Emerging Areas of Science, Engineering, and Medicine for the Courts. The workshop was organized to explore emerging issues in science, technology, and medicine that might be the basis of new chapters in a fourth edition of the Reference Manual on Scientific Evidence.2 The Reference Manual, a primary resource for federal judges on questions of science in litigation, is a joint publication of the National Academies of Sciences, Engineering, and Medicine and the Federal Judicial Center (FJC), the research and education arm of the federal judiciary. 3 The workshop was planned in collaboration with the FJC with funding from the National Science Foundation.

During opening remarks, workshop planning committee Co-chairs Kathleen McDonald O’Malley (U.S. Court of Appeals for the Federal Circuit) and Thomas D. Albright (Salk Institute for Biological Studies) described the program for the workshop, stating that, over the course of the workshop, judges would discuss how they evaluate scientific evidence in court and scientists and others would speak about emerging issues in science and technology that may come before the courts in coming years. The workshop had been convened to identify new material for existing Reference Manual chapters and emerging and other areas of science that are likely to be the focus of litigation.

Judge David S. Tatel (U.S. Court of Appeals for the District of Columbia Circuit), Co-chair of CSTL, offered framing remarks.4 The Reference Manual, he said, like CSTL, provides a crucial bridge between the legal and scientific communities—in the case of the manual by providing information on the methodology of areas of science that often present difficult issues when introduced in the form of expert testimony.

OVERVIEW OF REFERENCE MANUAL ON SCIENTIFIC EVIDENCE

“Federal judges,” Judge O’Malley said, “must play by the rules—those set down by Congress in the Federal Rules of Evidence and Procedure and those laid out by the Supreme Court.” Federal Rule of Evidence 702, she said, addresses the admissibility of expert testimony at trial.5 In 1993, almost twenty years after Rule 702 was first enacted, the Supreme Court issued a decision in Daubert v. Merrell Dow Pharmaceuticals, Inc.6 The Court instructed federal judges to take a greater “gatekeeping” role when deciding whether to allow trial testimony under Rule 702. More specifically, judges were instructed to “ensure that scientific evidence is not only relevant, but is reliable as well.”7

The Reference Manual on Scientific Evidence, O’Malley said, is one of several efforts to assist federal judges in determining the extent to which a scientific expert’s proposed testimony is grounded in acceptable scientific methods and proce-

________________

1 See https://www.nationalacademies.org/stl.

2 The third edition of the Reference Manual on Scientific Evidence is available at: https://www.nap.edu/catalog/13163/reference-manual-on-scientific-evidence-third-edition and https://www.fjc.gov/content/reference-manual-scientific-evidence-third-edition-1.

3 See https://www.fjc.gov/. The FJC developed the first two editions of the Reference Manual on Scientific Evidence. The third edition was developed in collaboration with the National Academies of Sciences, Engineering, and Medicine.

4 Judge Tatel was also a member of the workshop planning committee.

5 See https://www.uscourts.gov/sites/default/files/evidence-rules-procedure-dec2017_0.pdf, p. 15.

6 509 U.S. 579 (1993).

7 Id at 589.


Image

Suggested Citation:"Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshopin Brief." National Academies of Sciences, Engineering, and Medicine. 2021. Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26231.
×

dures. The purpose of the Reference Manual is to tap into the knowledge of the scientific community—particularly in those areas where scientific principles are regularly at play in court proceedings, or may come into play in future court proceedings. Its goal is to take knowledge gleaned from the scientific community and present it in neutral and understandable terms so that judges—who most often are not trained in fields of science8—will be able to understand relevant scientific principles to support decisions that are sound in science as well as in law.

The third edition of the Reference Manual, published ten years ago, updated chapters that had appeared in prior editions and added new chapters on neuroscience, exposure science, mental health, and forensic science. The new chapters reflected a recognition that matters in these scientific fields were being newly, or increasingly, considered (or re-considered) in courts across the country.

O’Malley suggested that, in the 10 years since the publication of the last edition of the manual, there have been enormous advances in many scientific fields and a concomitant increase in the legal issues that such technological developments pose. She said that the current workshop provided an opportunity to determine how best to revise the Reference Manual to ensure that members of the judiciary are equipped to address new and burgeoning scientific issues. O’Malley noted that an FJC survey of all federal judges, including courts of appeals, district court, magistrate, bankruptcy court, and the Court of Federal Claims judges, indicated that, most often, judges refer to the manual to obtain general background information in a scientific area, to understand a particular dispute over a technology, or to better understand the reliability of proffered scientific testimony or evidence. Judges expressed a particular interest in learning more about data science (e.g., data use, access, and related privacy concerns), computer hardware and software, eyewitness identification, artificial intelligence, risk assessment, and behavioral and biological predictors of violent behavior. The workshop, O’Malley said, would touch on many of these issues in response to these expressions of interest and additional topics identified by the workshop planning committee.

EVALUATING SCIENTIFIC EVIDENCE IN COURT

Judge Thomas D. Schroeder (U.S. District Court for the Middle District of North Carolina) moderated the workshop’s first panel.9 He opened the session by emphasizing that, under the Daubert decision and Federal Rule of Evidence 702, the judge must find by a preponderance of the evidence that the scientific methods that underlie the scientific testimony are reliable. Rather than focusing on the conclusions of the expert, the judge must instead focus on the scientific methodology that supports the expert’s testimony to ensure that those methods meet the standards of the relevant scientific profession. The role of the judge is somewhat like that of a scientific journal editor deciding whether a submitted manuscript meets scientific standards for publication—a decision that is independent of whether the journal editor agrees with the conclusions of the manuscript.

Judge Sarah S. Vance (U.S. District Court for the Eastern District of Louisiana) said that challenges to expert testimony in the form of “Daubert motions” are ubiquitous in her courtroom. In deciding such motions, she suggested that judges will benefit from an early awareness that there will be disagreement among expert witnesses. While there may be challenges to expert testimony in almost every case, she said, the most difficult cases involve challenges to the extrapolation of scientific findings. As an example, Vance described a case that involved testimony by a well-qualified epidemiologist claiming that very low levels of exposure to Benzene as a component of gasoline caused Acute Myeloid Leukemia (AML). The critical issue concerned extrapolation of published studies showing a link between high rates of exposure to Benzene and AML to the lower rates of exposure present when Benzene is one of many components of gasoline. Vance turned to the Reference Manual on Scientific Evidence to understand epidemiology research designs and practices. She then reviewed the individual studies relied upon by the expert and conducted a full evidentiary hearing

Judge Patti B. Saris (U. S. District Court for the District of Massachusetts) has seen many disputes over admissibility of expert testimony in product liability cases and patent cases. The qualifications of the expert are rarely a basis for excluding such testimony. One of Saris’ first patent cases, which involved gene cloning, required her to consider conflicting testimony by experts from Harvard, Yale, the Sorbonne, and the Max Planck Institute. Saris agreed with Vance that the most challenging cases involve the extrapolation of scientific findings to issues arising in specific legal cases. She mentioned a multidistrict litigation case in which the parties’ experts disagreed on whether the use of anti-epileptic drugs leads to an elevated risk of suicide. Understanding the science was a challenge, and well-qualified experts appeared to have legitimate disagreements. Saris used the Reference Manual to learn how scientists from different disciplines assess causal relationships. The opportunity for judges to question the parties’ experts and to view tutorials by those experts also are helpful in understanding the underlying science. Bringing the parties’ experts together to debate the issues among themselves, known informally as “hot-tubbing,” is another helpful means to clarify areas of agreement and areas of conflict.10 In jury trials, the judge must adopt procedures to assist the jurors’ understanding of the evidence, including allowing jurors to ask questions through the judge. Judges, Saris said, must

________________

8 In the context of this proceedings in brief, science is understood as encompassing testimony related to science, engineering, and medicine.

9 Judge Schroeder was a member of the workshop planning committee.

10 “‘Hot tubbing’ is the colloquial name for a process of adducing and testing expert evidence which is more formally known as concurrent expert evidence. The method has been championed in Australia.” It “is now used in other common law jurisdictions and in international arbitrations” and, on occasion, in U.S. courts. See https://civiljuryproject.law.nyu.edu/concurrent-expert-evidence-in-the-united-states-is-there-a-role-for-hot-tubbing/.

Suggested Citation:"Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshopin Brief." National Academies of Sciences, Engineering, and Medicine. 2021. Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26231.
×

be cautious in seeking information outside the presentations of the parties, noting that the internet and amicus briefs submitted by non-parties may be sources of questionable scientific assertions.

Judge Leonard P. Stark (U.S. District Court for the District of Delaware) often sees conflicts over scientific and engineering testimony in the context of patent litigation. Stark typically has 200-300 patent cases at any one time, and he said that there is rarely sufficient time to explore underlying issues to the degree that the judge would prefer. In pharmaceutical patent cases, Stark acts not only as a “gatekeeper” resolving conflicts regarding admissibility of expert testimony, but resolves the merits of the underlying dispute in a bench trial, taking on the more difficult task of determining the merits of the evidence. Conflicts among experts in patent cases often arise during the process of deciding what the specific terms in the claims of the patent actually mean. Judges rely on the parties to inform them of the issues in the case and areas of disagreements among the experts. The Reference Manual provides a neutral, reliable, and easily accessible source of information that can help judges understand presentations by the parties, Stark said. He noted that educational videos prepared by the parties are also of assistance in such cases.

Judge Barbara Hervey (Texas Court of Criminal Appeals) hears appeals arising from decisions by lower Texas criminal courts. Her appellate court encounters forensic science testimony in the context of writs of habeas corpus challenging the circumstances under which prisoners are confined. State courts have fewer resources to assist judges in understanding complex scientific issues, but Texas has developed several institutions that help inform judges’ assessments of forensic sciences. The Texas Forensic Science Commission investigates allegations of misconduct that would affect the integrity of forensic analyses conducted by a crime laboratory. The Texas Criminal Justice Integrity Unit develops training programs to educate judges, prosecutors, and defense attorneys about criminal expert testimony. Several groups are developing training videos to educate judges about specific types of forensic sciences. Hervey believes that judges need a better understanding of the current state of the forensic sciences relative to the conditions at the time of the 2009 National Academies’ forensic science study.11 They need to understand which forensic science disciplines have been validated scientifically. She suggested that this could be accomplished through a series of short guides and videos explaining the state of areas of forensic science.

DISCUSSION

In the discussion session, panelists discussed the prevalence of Daubert hearings. Stark indicated that, in almost every patent case, there is oral argument regarding the exclusion of expert testimony. In rare instances, it may be necessary to conduct more extensive pretrial hearings that include preliminary testimony by expert witnesses. The panelists agreed that understanding the reports submitted by experts is essential to making an informed assessment of scientific testimony. When questions remain, the judges on the panel differed with regard to the extent to which they independently assess the scientific studies that that experts cite in support of their testimony. Vance finds it necessary to read the underlying studies in cases involving epidemiology evidence, while Saris prefers to question the experts regarding issues that remain unclear. Judges must, of course, also be alert to attorneys’ misrepresentations of expert testimony and the underlying studies.

Schroeder mentioned that the Judicial Conference Advisory Committee on Evidence Rules is considering a rules amendment intended to control the tendency of some experts to overstate their conclusions, either in terms of their confidence in their opinions or in extending their opinions beyond reasonable limits imposed by the underlying research. Saris recalled that overstatement of conclusions was an issue that concerned a committee of the President’s Council of Advisors on Science and Technology (PCAST) on which she served.12 It was suggested that forensic pattern matching evidence, such as fingerprint identification and bullet matching, is particularly subject to such overstatement. Hervey mentioned that such overstatement occurs with bitemark evidence as well.13

Stark has found that jurors are able to deal with complex evidence in the context of patent cases. The ability of the jury to deal with complex information depends on the ability of experts and attorneys to translate the complex science in a way that is accessible to laypeople. Stark observed that, while both judges and jurors may find the complexity of the scientific evidence daunting at the outset, usually they each are able to make an informed assessment of the evidence by the end of the trial.

Albright asked how we can refocus attention upon the strength of the underlying science and areas of scientific consensus without being distracted by the disputes arising from the parties’ adversarial experts. Hervey pointed out that the

___________________

11 See National Research Council. 2009. Strengthening Forensic Science in the United States: A Path Forward. Washington, DC: The National Academies Press. https://doi.org/10.17226/12589, available at: https://www.nap.edu/catalog/12589/strengthening-forensic-science-in-the-united-states-a-pathforward#rights.

12 See Executive Office of the President, Forensic Science in Criminal Courts: Ensuring Scientific Validity of Feature-Comparison Methods, September 2016, available at: https://obamawhitehouse.archives.gov/sites/default/files/microsites/ostp/PCAST/pcast_forensic_science_report_final.pdf.

13 Many courts and scientific organizations have questioned the validity of bitemark evidence. The President’s Council of Advisors on Science and Technology, for example, found “that bitemark analysis does not meet the scientific standards for foundational validity, and is far from meeting such standards. To the contrary, available scientific evidence strongly suggests that examiners cannot consistently agree on whether an injury is a human bitemark and cannot identify the source of bitemark with reasonable accuracy.” See Executive Office of the President, Forensic Science in Criminal Courts: Ensuring Scientific Validity of Feature-Comparison Methods, September 2016, p. 87, available at: https://obamawhitehouse.archives.gov/sites/default/files/microsites/ostp/PCAST/pcast_forensic_science_report_final.pdf.

Suggested Citation:"Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshopin Brief." National Academies of Sciences, Engineering, and Medicine. 2021. Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26231.
×

Texas judicial education programs on forensic sciences focus on those areas of broad scientific consensus. This allows judges to deal with disputes over forensic evidence based on a common understanding regarding areas of scientific consensus. Vance noted that often the scientific issues that arise in court, especially in multidistrict litigation cases, are at the edge or perhaps just beyond areas of scientific consensus (e.g., in areas like epidemiology and toxicology) and that judges and juries have the obligation to consider responsible minority views within the scientific professions.

It was noted that, while judges may have authority to appoint an independent expert, this rarely happens because of opposition by attorneys and judges’ concerns about finding an expert that is neutral with regard to disputed issues. When experts are appointed by the court, the judge must require the parties to pay the costs, and this may give rise to additional disputes. The American Association for the Advancement of Science’s Court Appointed Scientific Experts (CASE) service was mentioned as a resource for identifying experts whose skills are matched to the needs of the case.14

Peer review is one of the factors that judges consider in determining the admissibility of scientific testimony. Workshop planning committee member Karen Kafadar (University of Virginia) pointed out that there are differences in quality among peer reviewed journals and asked whether too much weight is given to the fact that a study is peer reviewed. O’Malley recalled that, in developing the previous edition of the Reference Manual, Dr. Jerome Kassirer, the co-chair of the committee overseeing its development, expressed concern about the extent of deference given to peer review in light of the varied quality of scientific publications. Saris explained that judges do consider the quality of the journal, as well as the number of peer reviewed studies that are in agreement.

Returning to the issue of strengthening the use of forensic sciences, Hervey pointed out the necessity for proper funding of training programs for judges, attorneys, and court personnel. With proper funding, the courts can gain access to distinguished scientific experts and develop programs that will inform the courts about needed improvements and practices. In her experience, when judges become aware of the necessity and opportunity for improvement, they often will seek funding to implement such changes.

In response to a question from the audience, Hervey explained that the Center for Statistics and Applications in Forensic Evidence (CSAFE) works to build a statistically valid foundation for the analysis and interpretation of forensic pattern evidence, such as fingerprint identification.15 This group is preparing a series of videos for judges and attorneys on different forensic science disciplines. In response to another audience question, Vance addressed inequities that may arise when there is variability in parties’ ability to hire expert witnesses. This is a particular concern in torts cases, where individual plaintiffs may be disadvantaged, and in criminal cases, where indigent defendants may be disadvantaged. Disparity in resources is less of an issue in multidistrict litigation cases where plaintiffs may pool their resources to ensure adequate access to expertise. Educational resources such as the Reference Manual and the PCAST forensic science report16 provide assistance to disadvantaged parties by helping them better understand the scientific issues that are raised in challenges to expert testimony.

EMERGING ISSUES IN THE CLIMATE AND ENVIRONMENTAL SCIENCES

Session moderator Paul Hanle (Environmental Law Institute) introduced the session by raising questions about what a reference manual on scientific evidence should say about climate science,17 what criteria should be used for selecting material to include in the manual, and what judges need to know in order to weigh evidence in cases that involve climate science. Hanle suggested that climate science is moving fast by historical standards. He said that legal issues often turn on climate impacts and that attribution is of central importance.

Hanle noted the Sabin Center for Climate Change Law at Columbia University had counted 1,025 US climate cases by mid-2019 and 1,320 by the end of 2020—a 30 percent increase in 18 months.18 He said that many climate cases have to do with government action—meeting requirements of the National Environmental Policy Act and the Clean Air Act, for example—but that a smaller but increasing number of tort complaints have been filed alleging nuisance, negligence, or consumer fraud by fossil fuel enterprises. He noted that, while most judges have not yet presided over any climate-related cases, many judges say they expect to see cases soon as the number grows.

The panel’s first speaker, Joellen L. Russell (University of Arizona), began her presentation by speaking about carbon dioxide (CO2), the way it is measured and regulated, and how we can deal with carbon accounting. She discussed the importance of the 2007 Supreme Court decision in Massachusetts v. EPA, which held that the Clean Air Act gives the Environmental

___________________

14 See https://www.aaas.org/programs/court-appointed-scientific-experts#:~:text=CASE%20is%20a%20service%20that,to%20serve%20as%20technical%20experts.&text=CASE%20staff%20selects%20experts%20on,the%20specific%20request%20for%20assistance.

15 See https://forensicstats.org/.

16 See footnote 12.

17 “Climate science investigates the structure and dynamics of earth’s climate system. It seeks to understand how global, regional and local climates are maintained as well as the processes by which they change over time. In doing so, it employs observations and theory from a variety of domains, including meteorology, oceanography, physics,” and chemistry. See https://plato.stanford.edu/entries/climate-science/#:~:text=Climate%20science%20investigates%20the%20structure,which%20they%20change%20over%20time.&text=Some%20key%20questions%20and%20findings%20about%20anthropogenic%20climate%20change%20are%20also%20discussed.

18 See http://climatecasechart.com/climate-change-litigation/.

Suggested Citation:"Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshopin Brief." National Academies of Sciences, Engineering, and Medicine. 2021. Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26231.
×

Protection Agency the authority to regulate carbon and other greenhouse gases because such gases are pollutants.19 Russell noted that the date of the court case corresponds with an “inflection point in U.S. emissions.”

Russell stated that scientists are interested in top-down methods for estimating emissions, with a focus on what can be verified through direct measurements of atmospheric and oceanic gases, as opposed to bottom-up methods which are self-reported by emissions producers. She said that in the past there had been a great deal of measurement uncertainty in the oceans, despite the fact that “93 percent of the energy imbalance caused by this increased CO2 and other greenhouses gases is going into the oceans.” She then described the deployment of hundreds of robot floats placed in the oceans since 1999 as a “revolution in how we account for carbon,” reducing uncertainty and creating the ability to measure emissions in the oceans. Such measurements allow for an almost real-time accounting of carbon emissions and will allow for finer resolution in time and space.

The panel’s second speaker, Veerabhadran Ramanathan (Scripps Institute of Oceanography), spoke about the science of attribution, how we know which environmental changes are due to human actions, and associated public health risks. He stated that one way to judge theories and model projections is to test predictions, and that many predictions, from as far back as 100 years ago, have been confirmed by observation. Such predictions include increased humidity of the atmosphere, amplified warming in Polar Regions, and a timeline for warming due to human activities. Ramanathan also stated that if there are any flaws in the predictions, it is that the changes “are a lot more drastic than what was predicted,” including “new weather extremes” such as droughts and fires.

Ramanathan discussed the economic and public health costs of the gradual warming and resultant extreme weather events. He said that a report of the Centre for Research on the Epidemiology of Disasters and the United Nations Office for Disaster Risk Reduction estimates that, in the last 25 years, “600,000 lives have been lost globally to climate related incidents, and more than 3.5 billion people have been displaced.”20 He noted that such statistics and evidence will be used in any future discussion about climate change.

Benjamin D. Santer (Lawrence Livermore National Laboratory) identified three issues that may be relevant to climate change litigation: climate fingerprinting; event attribution; and satellite temperature data. He described climate fingerprinting as recognizing the various characteristics and signatures that different influences have on the climate system. Such signatures are easier to discern if scientists study patterns. Santer emphasized the importance of “prob[ing] beyond one number…and look[ing] at complex patterns of climate change.” He noted the importance of testable hypotheses, which can rule out non-human-driven explanations by examining data produced by satellites and other measurements.

Santer discussed event attribution as a field that “is very relevant for the courts” and one that was “kick started” by the 2003 European heat wave. Using models similar to those used by epidemiologists, climate scientists are able to compare the likelihood of an event with no human changes in greenhouse gases with the likelihood of the same event with “human caused changes of greenhouse gases.” Santer said that satellite temperature data show that measurements from 1979 to the present have increased and that these changes demonstrate the “warming of the lower atmosphere and cooling of the upper atmosphere.” These types of measurements provide further evidence of “human effects on global climate.”

Santer listed potential relevant legal issues for judges to consider with regard to climate science, including when and where the hazard is likely to occur, the scientific confidence in the projected climate change hazard, the robustness of the effect across dozens of different climate models, the reliability of the attribution, and estimations of human contributions to the change.

The panel’s final speaker, Donald J. Wuebbles (University of Illinois), spoke about estimating climate impacts and the range and uncertainty inherent in these projections and in scientific models. Complex climate models based on physics, chemistry, and biology give us the ability to represent the atmosphere and determine potential temperature changes. Using such models, scientists can generate scenarios of higher and lower fossil fuel use, and from these scenarios “determine the potential temperature changes on our planet.” Such models include ranges and uncertainty in climate projections, but “do a pretty good job of representing what is happening on the earth,” especially “at a global scale.” Wuebbles stated that the models examine a range of “future [climate] scenarios that are driven by human activities.” The models provide useful projections, such that it is clear that “by the end of the century…we are going to see a very significant impact on the U.S. economy.” Impacts are already apparent in every region and in important sectors such as health, water, agriculture, transportation, infrastructure, and energy. He noted that there are three ways for humans to react to climate change: mitigation, adaptation, or suffering. “Right now,” Wuebbles said, “we are doing some of all three,” but we need to do enough mitigation and adaptation to minimize future suffering.

DISCUSSION

In the discussion session, questions were raised about the admissibility of climate change evidence in court and about how best to write a Reference Manual chapter that it is accessible to judges who might be more skeptical of climate science’s meth-

________________

19 549 U.S. 497 (2007).

20 See Center for Research on the Epidemiology of Disasters and United Nations Office of Disaster Risk Reduction, The Human Cost of Climate Related Disasters, 1995-2015, 2015, available at: https://www.preventionweb.net/files/46796_cop21weatherdisastersreport2015.pdf.

Suggested Citation:"Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshopin Brief." National Academies of Sciences, Engineering, and Medicine. 2021. Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26231.
×

ods and conclusions. O’Malley asked whether there were well-trained scientific experts who would refute claims of climate change with theories that would be admissible under the standards of evidence. Santer said that while there may be one or two scientists willing to testify under oath that the causes of climate change are natural ones, in his opinion, those voices are not credible. Tatel asked if panelists had any suggestions for how to write a Reference Manual that is “credible to judges who are themselves perhaps climate skeptics.” It was suggested that reports from the National Academies and the Environmental Law Institute’s judicial education programs might be useful resources. Panelists and members of the workshop planning committee acknowledged the difficulty of writing a chapter on climate science and emphasized the importance of presenting arguments objectively and in a “way that is credible and respectful of the scientists and the judges.”

Other questions related to the idea of attribution, precision of measurement, and the likelihood of assigning responsibility for specific sources of pollution to specific companies or individuals. Panelists replied that the infrastructure to monitor CO2 and other pollutants has developed significantly—this has led to more precise measurements in real time, and technology is accelerating to a point where such identifications may be possible.

EMERGING ISSUES IN COMPUTER SCIENCE AND INFORMATION TECHNOLOGY

Session moderator Josh Goldfoot (U.S. Department of Justice) opened the panel by noting that computer science and information technology comprise an enormous field of study and research and that the modest goal of the panel was to draw out a few topics of direct relevance to law.

The panel’s first speaker, Jeremy Epstein (National Science Foundation), began his presentation by speaking about the ubiquity of computers. “If there is electricity,” he said, “whether in the form of an electric plug or battery, there is a computer there now (or there will be soon).” Most of these devices, he continued, are connected to the internet. He suggested that computing ubiquity means that there are advantages and risks everywhere.

Epstein framed his remarks around issues related to privacy and security; independent examination of technology; software reliability; and accessibility. He noted that privacy is a concern, even when artificial intelligence (AI) is not involved: Our phones, fitness trackers, and other devices (e.g., implanted devices like insulin pumps) all share personal locations and preferences, he said. Promises of privacy protection, he continued, are frequently not enough to provide actual privacy: In cases where data has been released without obvious identifiers, Epstein said, it has been fairly easy to reverse engineer sensitive information. Epstein said that a key issue relates to the ability of independent experts to examine thoroughly computer technology and software code. Modern hardware and software are so complex, he said, that it is effectively impossible for anyone to understand all the ways a system could fail—even with access to hardware diagrams and source code. Further, Epstein suggested, even with unlimited budgets, we do not know how to build computer systems that are secure and reliable for anything but the simplest cases.

Epstein said that modern software is rarely the product of a single company and that many organizations are unaware of where their software originates. He suggested that understanding supply chain complexities is crucial to understanding who is responsible when computers fail. In some cases, Epstein noted, software is developed and maintained by a rotating group of individuals. When this software developed in this way reaches the market, he said, it is often difficult to determine fault for software failure. Epstein said that many non-specialists with no particular education in computer science write software in the form of Excel macros or webpages. This can lead to serious consequences, such as when errors related to Excel spreadsheets resulted in a $6 billion trading loss for J.P. Morgan in 2013.21

While technology can increase accessibility, Epstein noted that it also can exacerbate the digital divide. Importantly, many of the technologies increasingly used today assume general familiarity with and access to smart phones—though this is not universally true.

The panel’s second speaker, Russ Housley (Vigil Security, LLC), discussed computer science and information technology in the context of patents, standard-setting, and standard-setting organizations.22 With regard to patenting, Housley noted that a key issue is the ability of an inventor to demonstrate that his or her invention is novel. This is particularly challenging, he said, with software, because code is often an amalgamation of existing elements. Housley observed that definitions set forward at the beginning of a patent-related hearing often set constraints on experts and, in many ways, determine outcomes. Standard-setting organizations, Housley said, require participants to disclose patents that would encumber implementation of the standard under development. Encumbered provisions will not be included in a standard unless the patent holder offers license terms that are seen as fair, equitable, and nondiscriminatory. While patent holders will want to include protections for

___________________

21 See, e.g., https://www.businessinsider.com/excel-partly-to-blame-for-trading-loss-2013-2.

22 Standard-setting addresses patent disclosures, licensing terms, transfers of patent ownership, and other issues that arise in connection with developing technical standards for consumer and other microelectronic products, associated software and components, and communications networks including the internet. Standard-setting organizations, which can be domestic or international, are dedicated to developing standards for industry. These organizations have generally adopted policies regarding the disclosure and terms of licensing of patents essential to the standards they create (so-called standard-essential patents or SEPs). In general, standard-setting organizations encourage or require member firms to disclose SEPs and license them to standards implementers under terms commonly referred to as fair, reasonable, and non-discriminatory. See National Research Council. 2013. Patent Challenges for Standard-Setting in the Global Economy: Lessons from Information and Communications Technology. Washington, DC: The National Academies Press. https://doi.org/10.17226/18510.

Suggested Citation:"Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshopin Brief." National Academies of Sciences, Engineering, and Medicine. 2021. Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26231.
×

their own patents in standards, relevant patents must be disclosed and the terms of patent licenses shared. As an example, Housley noted that, in bankruptcy proceedings for Nortel, the judge said that anyone who gets patents in the bankruptcy settlement would be bound to the terms of the disclosures that Nortel had made. Standard-setting organizations provide a venue where early drafts of documents are available, and lawyers may use this access to identify prior art that may be used as the basis for making invalidity claims against bad actors or to claim that individuals or entities are bad actors. Nonetheless, he said, while everyone wants an open and transparent standard-setting environment, such openness allows bad actors to observe developments and file patents on ideas that are, sometimes, not their own. When this happens, the actual contribution to the standard would appear to be prior art even though it is not.

Housley said that it is difficult to achieve a deep understanding of the technical issues related to standard-setting and patenting in the information technology space. It is understandable, he said, that judges depend on expert reports, but that such reports need to be written in a way that provides instruction on industry jargon, or, at least, teaches the technology in plain English.

Subbarao Kambhampati (Arizona State University) spoke about AI technology. He noted that, in the most recent version of the Reference Manual on Scientific Evidence, which was published in 2011, there is no mention of AI. Currently, he said, “it’s hard to get through a day without mentioning AI everywhere, including at the courts.” He noted that AI appears in court proceedings in the context of privacy issues, hiring discrimination issues, fraudsters impersonating others, automated devices, wrongful accusations resulting from algorithms, and injury caused by autopilot technologies.

Kambhampati suggested that what has changed in the last decade is AI’s improved ability to see, hear, and read and also to show, say, and write. These developments allow AI to “perceive” the world and to create synthetic reality. AI has been enabled and deployed widely—through the internet and cellphone technologies. Machines, he said, can look at pictures and give captions to them. They can create images of people who do not exist but who are virtually indistinguishable from images of real people. It is now possible, Kambhampati said, to manipulate videos in ways that can make anyone say anything at any point in time.

With regard to the AI issues that may arise in court, Kambhampati identified liability issues for decisions made by automatic systems, noting that many companies increasingly use automated systems to make decisions ranging from loan approvals to who to interview. If these systems are erroneous and cause injury, he asked, who is liable? Kambhampati noted that some systems have human checks while others are entirely devoid of human involvement. When errors are made, they are often due to a complex combination of general code schema (integrity constraints that govern a database) and algorithmic training data (a collection of labeled information that is used to build a machine-learning model). Kambhampati noted that bias often is introduced inadvertently through training data.

Kambhampati said that an important area of consideration relates to evidence and admissibility issues. In particular, the advent of deep fakes could have significant ramifications with regard to evidence admissibility and eyewitness testimony.23 As of now, there are some technological solutions that might be able to distinguish, for example, synthetic video from genuine video. But, he said, this is a moving target, and at some point in time, it will much harder to distinguish synthetic imagery. Kambhampati noted that we now leave digital footprints everywhere. More and more of devices are connected to the internet (increasing the probability that individuals will be spied upon and their information shared with others), and AI has the ability to analyze patterns in data collected by these devices. This data collection raises many privacy issues. Kambhampati noted that there are ways to capture digital footprints with and without consent, and it will be up to judges to determine what evidence is admissible in court.

The session’s final speaker, Alice Xiang (Sony AI), spoke about some of the tensions and collisions that occur in the application of existing legal frameworks to the algorithmic context. She focused her remarks on algorithmic bias mitigation, stating that this is an area of concern for both the policymaking community and the public at large, both in terms of the potential for algorithms to be discriminatory and in the potential that the use of algorithms may amplify existing societal disparities. As an example, Xiang described a resume recruiting filtering algorithm abandoned by Amazon because the company realized that the algorithm was biased against women. The algorithm had penalized references to women in resumes because, historically, relatively few women had gotten jobs at Amazon (at least in the training data set that contained resumes received by Amazon). The algorithm “learned” that features that are specific to women were not associated with successful applicants.

Xiang discussed the explosion of literature around algorithmic fairness, which involves the identification of a fairness metric and the equalization of that metric across different groups. The problem, she said, is that, when examining this approach from a legal standpoint, such metrics look very similar to racial balancing, having quotas, or different thresholds for different demographic groups. This would be considered suspect under equal protection doctrine. Xiang noted that there is a broader trend within jurisprudence toward what is known as anti-classification—the idea that we should prohibit classification or differential treatment on the basis of protected attributes (qualities, traits, or characteristics that, by law, cannot be discriminated against). She said that omitting protected class variables does not prevent bias. She noted, for example, that when the race variable is absent, with a name, zip code, and other obvious variables, there are weak correlations to race (a protected class variable) due to systemic societal inequalities. Xiang suggested that trying to achieve fairness by eliminating obvious variables is unlikely to be successful. If, for example, an algorithm is built that does not take gender into account, the

________________

23 Deep fakes are realistic photo, audio, video, and other forgeries generated with artificial intelligence technologies. See https://crsreports.congress.gov/product/pdf/IF/IF11333.

Suggested Citation:"Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshopin Brief." National Academies of Sciences, Engineering, and Medicine. 2021. Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26231.
×

algorithm will almost certainly be biased against women. Xiang said that it is important for judges to examine how protected class variables are used in the construction of an algorithm and not simply whether they are present or absent in training data or a selected set of features. Xiang suggested that, if we want to have effective algorithmic governance, it is important to realize that tensions will emerge at the intersection of technology and law. Addressing these tensions head-on will be important in ensuring that judges don’t simply use a naive interpretation of what might be meaningful evidence for algorithmic discrimination, but instead, take a more scientific approach to interrogating the variables used and analyzed.

DISCUSSION

In the discussion session, Xiang noted that it is very easy to accept forms of evidence that don’t address core legal questions at issue. She suggested that it is important to: (1) always return to the relevant legal question at issue; and (2) push litigants to actually answer that question, rather than relying on proxies that make it sound like they are answering the question. If someone were to say, for example, “This algorithm does not have race anywhere in the algorithm, how is it possible that it could be making a decision on the basis of race?,” the correct response would be to ask how do we know that the algorithm is not learning certain discriminatory trends based upon data that treats a particular individual differently because of that individual’s protected attribute.

Kambhampati said that perception is very much the foundation of evidence and that, given the fact that synthetic reality is possible, video or written evidence can now be called into question, introducing plausible deniability. Goldfoot identified as key questions, “What is the methodology used to gather the data used in training sets?” and “What do we know about the biases available from that data?” He asked whether there are pointers that either judges or litigators should be looking for if they are litigating this issue (or if a question is raised about whether a particular video or photo has been created by AI or is faked). Kambhampati said that currently there are distinct features that humans may not be able to see, but that an AI system can detect micro errors. He noted that this won’t always be the case. Xiang noted that some technology companies have been trying to approach the deep fake issue by building better provenance standards for images and videos that are highly salient in political contexts.

Xiang said that there is this perception that if “machine learning folks” were gathered in a room and questioned, they could identify exactly what is wrong with an algorithm. The problem, she said, is that, given the many emergent properties derived from huge data sets used to train algorithms and the relatively ambiguous instructions often given to the algorithms with regard to objectives, it would be impossible for those assembled to reach a consensus on the cause of the problem.

Workshop planning committee member Steven M. Bellovin (Columbia University) reiterated that patents have to be non-obvious and novel. With regard to computer code, he said that it is unclear what encompasses ordinary skill in the art. O’Malley noted that, unless there is something “outrageously novel” about how off-the-shelf software is used, the use will not be patent eligible; also, what is novel today will not necessarily be novel tomorrow. An audience member asked about responsibility for programming flaws. He suggested that, while programmers are accountable now, this will not work for long, because programs start writing other programs with the result that the programmer becomes more removed from the development process. He asked whether it is reasonable to expect these kinds of systems to move forward without human input, and, if so, what would that mean in terms of ethical issues? Xiang said that humans remain in the loop for important decision-making and that algorithms themselves do not necessarily have power unless we choose to have them make important decisions. She said that there is some intrinsic combination between the human and the model with AI ethics-related questions. Right now, most models are trained on data that is based on pre-existing human decisions. In that sense, she said, there are still humans you can point to as being responsible for problems in decision-making processes.

With regard to the question of “black box” algorithms, where the underlying data are either proprietary or inscrutable, Kambhampati sugggested that it is difficult to understand and characterize the relevant aggregate properties of data sets without access to underlying data, but zero knowledge proofs can be used to prove that a solution is correct absent access to the underlying data.24 Such techniques can be used to give information about data sets, while preserving the privacy of the data set.

EMERGING ISSUES IN THE BIOLOGICAL SCIENCES

Session moderator Ellen Wright Clayton (Vanderbilt University) introduced the panelists for the session on emerging issues in the biological sciences.

The panel’s first speaker, Dana Carroll (University of Utah), discussed human and agricultural genome editing. He began with a description of human genome editing, focusing on the CRISPR-Cas system—a molecular tool for making targeted changes to the genome, including by insertions of novel DNA. The CRISPR-Cas system uses the Cas-9 protein and guide

________________

24 A zero-knowledge proof (or zero-knowledge protocol) is a method by which one party (the prover) can prove to another party (the verifier) that they know a value x, without conveying any information apart from the fact that they know the value x. See https://www.altoros.com/blog/zero-knowledgeproof-improving-privacy-for-a-blockchain/.

Suggested Citation:"Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshopin Brief." National Academies of Sciences, Engineering, and Medicine. 2021. Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26231.
×

RNA to target specific sequences in a genome.25 Carroll emphasized that the system is not perfect—desired changes are not always made, and unwanted changes can occur.

CRISPR therapy can be performed on cells removed from and then placed back into the patient—ex vivo—or on cells directly in a patient’s body—in vivo. Carroll pointed to the example of Victoria Gray—the first person in the United States successfully treated for a genetic disorder (sickle cell disease) with the help of CRISPR therapy.

Carroll emphasized that there are risks and benefits to be balanced with genome editing therapies and asked what legal guidelines and professional standards should regulate them. A specific concern is rogue actors performing unregulated treatments. Carroll pointed to the 2018 case of a Chinese scientist who, to the surprise of the scientific community, modified the genome of two human embryos. This raised significant concerns because such edits can be passed to subsequent generations with potentially unknowable consequences. Carroll noted that, while in many countries, legal constraints are in place to prevent genetic manipulations of human embryos, it will be difficult to prevent such manipulations in less restrictive jurisdictions.26

Carroll noted that CRISPR technology may have major impacts on agricultural and biomedical patents. Regulating the products of biotechnology will involve making determinations about the potential benefits and hazards as well as what constitutes an invention. In the United States, the decision has generally been made to regulate products of biotechnology rather than the process by which they are made. Nonetheless, the U.S. system can diverge from a product-based regulatory approach. For instance, the insertion of foreign DNA into crop genomes is regulated, but modifications that might occur naturally (e.g., changes made to the genetic sequence of existing genes) are not.

The panel’s second speaker, Yaniv Erlich (MyHeritage and Eleven Therapeutics), discussed familial DNA searches using consumer genetic testing and open genealogical websites. Erlich said that these familial searches leverage segments of DNA that are identical in their sequence and shared between individuals who have a common relative. The frequency and length of these segments are used to identify relatives and degree of relatedness. Erlich emphasized that the capability of these genealogical databases to identify distant relatives is much greater than the capacities of forensic databases, as consumer genealogical services use 700,000 genetic markers and forensic databases only use 13-20 markers.

The ability to locate relatives is further enhanced by new tools that allow searching between databases. Consumers can download their genetic data and upload it to multiple third-party services. One such service, GEDmatch, was used by the Federal Bureau of Investigation in the identification of the Golden State Killer. Erlich discussed his own research, which showed that, given the size of genetic databases today, virtually any person of European heritage in the U.S. has a relative in a genetic database, thus making it possible to be identify them through a relative. For people of other ethnicities, the possibility of identification is less robust, but becoming more so.

Erlich concluded his remarks by emphasizing the need to balance the needs of forensics, the desire of individuals for genealogical information, and the desire for individual privacy. Erlich stated that, without policies to balance these needs, the U.S. is the first country in the world with a “free-for-all” genetic surveillance system, where any person can search for and identify relatives, or identify individuals from an anonymous DNA sample.

Steven E. Hyman (Broad Institute of MIT and Harvard University) discussed how human genes contribute to physical and mental traits, such as height and cognitive abilities, and socially ascertained traits, such as years of schooling. Hyman emphasized that most traits are complex and that genes are not fate. While physical and socially ascertained traits have genetic components, they are genetically influenced to different degrees—degree of genetic contribution within a population is known as heritability. Hyman emphasized that heritability is a property of populations, not individuals, and that high heritability does not signify immutability. For example, the disease Phenylketonuria is 100 percent heritable—caused by a mutation in a single gene—and can lead to severe intellectual disability. It is also 100 percent preventable if an affected person eats a diet free of phenylalanine.

Hyman noted that most human traits are continuous and quantitative (i.e., occur along a spectrum) and are determined by many genetic factors that, individually, have very tiny effects. Typically, genetic differences between humans occur at tens-of-millions of places in the genome. Genetic variants can give a “genetic nudge” towards (or away from) a trait or disease. Hyman gave the example of schizophrenia, where thousands of genetic variants contribute to very small increases in disease risk. If enough of these genetic “nudges” add up, with the addition of environmental effects, a person’s likelihood of developing the disease will be high. Genome wide association studies are regularly used to study how multiple common genetic variants can affect a trait. Hyman pointed to large databases like the UK Biobank, which have genotyped many people and ascertained numerous traits like common disease risk, adult height, and educational attainment. These databases allow researchers to make predictions about traits based on genetic data. Hyman emphasized that, while we can make predictions about how genetics will affect a trait at the population level, we cannot do this well at the individual level.

The panel’s final speaker, Philip Sabes (University of California, San Francisco), spoke about brain-computer interface technologies. One class of these technologies are those that read information from the human brain, e.g., devices that can

________________

25 Genome editing can be performed in both germline cells (sperm, eggs, or their precursor cells) to induce heritable genetic changes or in somatic cells (the cells of the body other than germline cells, such as skin and bones) to induce non-heritable changes.

26 In the United States, the U.S. Food and Drug Administration may not review proposals for research using human embryos, the National Institutes of Health will not consider grant proposals to edit human embryos, and federal funding cannot be used for human embryo research. State and local laws may explicitly prohibit research with embryos.

Suggested Citation:"Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshopin Brief." National Academies of Sciences, Engineering, and Medicine. 2021. Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26231.
×

translate desired movement into a computer action. Sabes noted the new technology is being developed that will reduce device size and increase the number of electrodes needed to achieve higher resolution. In addition to reading information, Sabes noted that devices can write information into the brain by electronically stimulating the brain. For example, by transmitting information into the visual cortex, it is possible to create artificial visual perceptions for people who are blind. A second class of devices involves neuromodulation, which enhance, suppress, or modulate brain activity. Such devices include deep brain stimulation to treat Parkinson’s disease and other conditions. A third class of technologies involves non-invasive brain monitoring. This category includes electroencephalography (EEG) and magnetoencephalography (MEG). While these devices are quite large currently, Sabes noted wearable devices will be developed that are capable of measuring states of arousal, attention, or mood.

Sabes concluded his remarks by discussing potential legal implications of brain-computer interface technologies. These could relate to issues of general device upkeep, how devices affect an individual’s sense of agency, and liability issues related to the lifespan of the device. Sabes also described concerns about security and privacy. Though patterns of behavior and mood are already being captured by other devices (e.g., smartphones), information from a person’s brain might be considered more sensitive. Sabes raised the question of whether companies could require employees to use brain-monitoring devices to track attention or performance. In terms of criminal liability, Sabes noted the possibility that an individual could blame criminal behavior on an implanted device. Brain-reading technologies could also be used to determine a person’s state of mind or truthfulness, raising Fifth Amendment concerns.

DISCUSSION

In the discussion session, questions were raised about the regulation of genome editing, and whether genome editing could be done covertly in a manner wherein the modifications were not detectable. Carroll suggested that this would be possible in principle, but that currently we lack the precision and efficiency to make changes that are invisible. Panelists emphasized that most traits with a genetic component are determined by numerous genetic variants, each of which can have a small effect. This makes determining the appropriate parts of the genome to modify difficult. Making edits in an attempt to change these traits could be dangerous, especially because most genes have multiple effects—both undesirable and desirable—on biology. Complex genetic traits also make predicting a trait from an individual’s genetic data difficult.

Panelists noted that the underlying genetic data used to estimate the effect of genetic variants on disease risk (and in consumer genetic databases) comes predominantly from people with European ancestry, which leads to errors when applied to non-European individuals. Panelists noted that genetic databases are increasing in numbers and diversity, raising concerns about the ease of genetic identification, discrimination, and social disruption (especially in matters of family structure and paternity). In response to a question about whether brain monitoring could aid in identifying pain, Sabes noted that mapping the specific pain-specific regions of the brain is an ongoing effort.

EMERGING AREAS IN PSYCHOLOGY, DATA, AND STATISTICAL SCIENCES

Session moderator Gary Marchant (Arizona State University) introduced the session on emerging areas in psychology, data, and statistical sciences by suggesting that, while the topics of the session may appear disparate, there are interesting overlaps and intersections. He asked the audience to consider whether these issues should be addressed in a new edition of the Reference Manual on Scientific Evidence, and if so, where?

The panel’s first speaker, Xiao-Li Meng (Harvard University), emphasized that selection bias can come from cherry picking data, which alters the strength of the evidence. He identified “7 S’(ins)” of selection bias: (1) selection of target/hypothesis (e.g., subgroup analysis); (2) selection of data (e.g., deleting “outliers” or using only “complete cases”); (3) selection of methodologies (e.g., choosing tests to pass the goodness-of-fit); (4) selective due diligence and debugging (e.g., triple checking only when the outcome seems undesirable); (5) selection of publications (e.g., only when p-value <0.05); (6) selections in reporting/summary (e.g., suppressing caveats); and (7) selections in understanding and interpretation (e.g., our preference for deterministic, “common sense” interpretation).

Meng discussed how these biases could affect the accuracy of research findings. He described how analyzing a specific sub-population after not finding a full population effect can affect the generalizability of findings and increase concerns. Meng also addressed algorithms adjusted to obtain pre-determined outcomes or desired findings; journals skewed toward only accepting significant findings; a tendency of the media to ignore caveats; and known limitations in public dissemination of results. Meng noted that humans ascribe stronger confidence to results that match expectations and “common sense.”

As an example of selection bias, Meng described estimating COVID-19 positivity rates. Individuals more likely to have COVID-19 are those more likely to be tested, which results in a sample that disproportionately excludes those without COVID-19. This results in higher positivity rates in the data. Selection biases such as these can dramatically affect data quality.

Meng stated that selection bias is inevitable but that asking the right questions can reduce its effects. He offered several questions that might be asked when a study is presented:

  • “When was this study published? How many related studies did you review? How many of these studies reached
Suggested Citation:"Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshopin Brief." National Academies of Sciences, Engineering, and Medicine. 2021. Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26231.
×

conclusions similar to the one reported in your study?

  • Who collected the data? Who cleaned the data? Were any data discarded? Why?
  • What is the comparable reference population? How was this reference population chosen? Why is it so large/ small? What happens if we change the population to…?”

Meng said that we should be mindful of selection bias and sensitive to the fact that data can be (and are) manipulated to obtain desirable findings.

The panel’s second speaker, Rebecca Doerge (Carnegie Mellon University), discussed transforming science through automation and AI. She noted that we are in an unprecedented era of science, driven by new tools that can collect and analyze massive amounts of data. Doerge said it is important to understand how tools make connections between data and massive data streams and to recognize that advancements are possible when people work inter-disciplinarily and make use of flexible workspaces and shared resources.

Doerge described a remote-controlled laboratory at Carnegie Mellon University, which uses robots and automation driven by AI technology. Every robotic AI decision is traceable and registered, and this is important to understand the AI process and track the accuracy of results. The benefit of such a lab, Doerge explained, is its 24/7 operation, which generates massive amounts of data. The robotic instrumentation is automatic and collected data is stored in the cloud. Past and current information is used to make decisions based on algorithms. By bringing together computation, data analytics, and foundational science, discovery and results are accelerated.

Doerge warned that some people may not use such technologies responsibly. She noted that more data is not always better and that algorithms can make mistakes. Humans created the first algorithms, which can be limited and biased. Using appropriate, representative populations is necessary when training models and algorithms. Doerge also described deep learning, which explores a data space and builds a model that identifies missing data. Doerge said that it is important to take time to understand the questions being asked, as well as the use and limitations of algorithms and AI, to better address issues related to transparency, accuracy, and fairness.

Daniel Kahneman (Princeton University) discussed noise, which is chance variability in individual judgments. Kahneman used images of targets to illustrate the concept of noise and to distinguish it from bias. Bullet holes scattered randomly outside the bullseye represent noise. Bullet holes scattered in a skewed way from the bullseye represent bias. Kahneman noted that both noise and bias can affect the true value of data and should be given equivalent consideration. He said that the many studies comparing human judgment and algorithms have generally found that human judgment and algorithms are either of equivalent accuracy or that algorithms perform better. This suggests the effect of noise on human judgments, that is, the variability between individuals’ decisions and within a single individual’s decisions.

Kahneman described research where insurance underwriters were asked to set a premium for the same claim. The research results suggest that noise related to the different premiums was five times as large as executives expected. Kahneman noted that noise also occurs within judicial decision-making. In a study in which judges were asked to make sentencing decisions, researchers found that judges were about as “noisy” as the underwriters were, and that if one selected two judges at random, a sentencing decision could differ by almost four years. Kahneman explained that there are three types of noise: pattern noise (e.g., when judges have personal responses to crimes and defendants); level noise (where individuals differ in the overall level of their judgments—e.g., judges varying with regard to the overall severity of sentences); and occasion noise (where equivalent events—e.g., cases—evoke different reactions on different occasions). Kahneman emphasized that there is a lot of noise in judgment and that noise is a problem everywhere judgment is applied.

The panel’s final speaker, Justice Goodwin Liu (California Supreme Court), described implicit bias as the attitudes or stereotypes that affect our understanding, actions, and decisions in an unconscious manner. Liu noted that everyone is susceptible to implicit biases, which can be positive or negative. Such biases are mental shortcuts, or schemas, and are necessary for humans to organize and categorize information—but they can have harmful effects.

Liu provided examples of how implicit biases can introduce racial biases in police stops and gendered perceptions of childcare responsibilities. Liu described studies about implicit bias in the context of employment, such as resume studies that found that individuals with white-sounding names received more positive responses than individuals with Black-sounding names, regardless of resume quality. Liu said that implicit bias in the workplace can affect hiring decisions, job assignments, performance reviews, compensation, promotions, discipline, and termination decisions.

Liu discussed findings from education research about bias: white instructors with bias can affect the educational performance of Black students, and faculty were more responsive to emails from students perceived to be white. Liu also described racial biases in pain assessment and medical treatment, as well as in professional sports. In medicine, he said, there have been many studies where pain treatment reveals racial disparities. In professional sports, one study describes how white referees call more fouls against Blacks and Black referees call more fouls against whites—the disparities are big enough to affect an appreciable number of games every year. 27 Liu also described research where, when presented with a draft legal memorandum, individuals found more errors and rated the quality of the memo lower when the author had a Black-sounding

________________

27 See Price and Wolfers, Racial Discrimination Among NBA Referees 125 Q.J. Econ. 1859 (2010).

Suggested Citation:"Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshopin Brief." National Academies of Sciences, Engineering, and Medicine. 2021. Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26231.
×

name.28 Liu noted that researchers have examined implicit bias in the context of stop and frisk policies, three strikes laws, and racial disparities in incarceration.

DISCUSSION

In the discussion session, participants asked whether noise is always a negative and whether standard criteria can be used to reduce noise. Kahneman responded that some adjustments can increase the uniformity of judgments and reduce noise, but that noise is difficult to control; only pattern noise could potentially be reduced. Expressing concerns about large differences in sentencing decisions, Kahneman noted that noise can be beneficial for evolution and learning, but not when we expect consistent decisions.

Workshop planning committee member Thomas Schroeder raised a question about whether peer review was a good type of selection bias. Meng responded that peer review improves reliability; yet peer-reviewed articles can still have errors. Further, he said, articles that do not contain results of great significance are less likely to be published by journals. This, in turn, can affect funding agency decisions about the direction of future research. Meta-analyses, where many studies are collected and reviewed, can help ameliorate selection bias. Further discussion considered the benefits and tradeoffs of pre-registration and open science approaches that ask researchers to identify their hypotheses and methods in advance.

It was suggested that research should not only examine differences between Black and white individuals, but also consider intersectionality (i.e., the interconnected nature of multiple identities such as race, gender, and religion). Liu also discussed how group decisions differ from individual decisions. He noted that, groups of appellate judge arrive at a multimember decision through a process of give and take. Panelists discussed whether civil rights laws are equipped to address implicit bias, or if new standards are necessary. Liu suggested that much anti-discrimination law is rooted in earlier paradigms that do not reflect more recent psychological and cognitive understandings.

CONCLUDING THOUGHTS

The workshop ended with the planning committee co-chairs commenting on key messages from workshop presentations and discussions. Albright noted that the workshop would serve as a source of wisdom and guidance in forthcoming efforts to update the Reference Manual on Scientific Evidence. He asked the audience to think about the discussion at the beginning of the workshop, which focused on ways in which the disciplines of science and law can work together. Decision-making, he said, is what the courts do, but as individuals, we tend to think that we understand decision-making. We have intuitions about it, but there is a science that addresses how people make decisions and how we employ different types of information. In very broad terms, this science tells us that our decisions and actions have two primary influences: reason and emotion.

To make the right choices—to survive in an uncertain world—we have evolved a rational information-based approach to decision making, Albright said. We employ evidence and well-reasoned logic to sort things out. Despite our best intentions, however, this doesn’t always work: Our emotions have an insidious influence on our choices. In civil society and in our legal system, he said, emotion is the currency of false facts and bias. All too often, emotion is what leads us to believe things that aren’t true. It adds noise to decision variables, it changes our decision criteria and, ultimately, it moves us to make the wrong choices or the right choices for the wrong reasons.

In our modern world, Albright said, science does the watching for us. Science provides evidence, objectively gained through riveted eyes, that enables us to overcome the ghosts, to recognize the phony arguments that pull at our emotions. Science is thus a natural—indeed an essential—partner to the rule of law.

The trick is knowing what the right science is, knowing what is fact, what is fiction, and what is just plain sloppy. Rule 702, Albright said, lays out a very general set of criteria for judges to use in their role as gatekeepers of scientific evidence—rules that judges can use to decide what’s valid and what’s not, to understand what the most reliable science is. The Reference Manual on Scientific Evidence has long been a fundamental and, perhaps most importantly, an impartial resource for this determination by the judiciary. As we go forward, Albright noted, the richness and complexity of scientific knowledge will continue to grow. What’s missing from Rule 702 is an intimate interactive advisory relationship between science and law. A relationship in which knowledge of how things work comes not from experts hired by parties to wage battle, but directly and impartially from reliable scientific discoveries and principles. Ultimately, he said, we must move toward a system in which—like science, engineering, and medicine—science and law are mutually informing partners in an effort to reach correct decisions.

O’Malley said that judges seek to provide parties with a fair trial, subject to the rules and limitations imposed upon them. Judges strive to understand the science at issue in a given dispute, but know that their job is not to decide which side of the scientific debate is correct. “We know,” she said, “that many decisions are not ours to make.” Where a decision is one for the jury, a judge has two main roles: (1) to give the jury the evidentiary tools they need to decide the facts, including allowing the jury to hear both sides of a scientific debate—so long as the opposing scientific views pass a minimum evidentiary threshold; and (2) to provide the jury with a correct statement of the law, which provides the jury with the lens through which those

________________

28 See Reeves, Written in Black and White: Exploring Confirmation Bias in Racialized Perceptions of Writing Skills Nextion (April 2014).

Suggested Citation:"Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshopin Brief." National Academies of Sciences, Engineering, and Medicine. 2021. Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26231.
×

facts are to be assessed.

O’Malley said that, when she first took the bench, she heard the adage that law lags science—that the scientific community was often way ahead of the law’s recognition of scientific principles. That meant that, by the time an issue had percolated through the courts and a determination of admissibility was affirmed on appeal—i.e., a determination that scientific principles had been generally accepted throughout the scientific community was confirmed, the science had often changed or moved on. The point, she said, of Daubert—the point of Federal Rule of Evidence 702—was to try to change that and allow more flexibility in the court’s ability to assess emerging science. “I doubt,” O’Malley said, “that the law will ever stop lagging behind science or that we will ever stop litigating scientific principles. But I hope that, with efforts like this workshop and the reconsideration of the Reference Manual, where scientists are engaged to help the judiciary, the gap can begin to close.”

________________

DISCLAIMER: This Proceedings of a Workshop—in Brief has been prepared by Steven Kendall, Joe S. Cecil, Jason A. Cantone, Meghan Dunn, and Aaron Wolf as a factual summary of what occurred at the meeting. The committee’s role was limited to planning the event. The statements made are those of the individual workshop participants and do not necessarily represent the views of all participants, the planning committee, the Committee on Science, Technology, and Law, the National Academies, the Federal Judicial Center, or the National Science Foundation.

REVIEWERS: To ensure that it meets institutional standards for quality and objectivity, this Proceedings of a Workshop—in Brief was reviewed by MICHAEL IMPERIALE, University of Michigan; ANDREW MAYNARD, Arizona State University; and BARBARA ROTHSTEIN, United States District Court for the Western District of Washington. MARILYN BAKER, National Academies of Sciences, Engineering, and Medicine, served as the review coordinator.

Committee on Emerging Areas of Science, Engineering, and Medicine for the Courts: Identifying Chapters for a Fourth Edition of the Reference Manual on Scientific Evidence—A Workshop:

THOMAS D. ALBRIGHT (NAS) (Co-chair) (Salk Institute for Biological Studies); KATHLEEN MCDONALD O’MALLEY (Co-chair) (U.S. District Court of Appeals for the Federal Circuit); STEVEN M. BELLOVIN (NAE) (Columbia University); KAREN KAFADAR (University of Virginia); ANDREW MAYNARD (Arizona State University); VENKATACHALAM RAMASWAMY (National Oceanic and Atmospheric Administration); THOMAS SCHROEDER (U.S. District Court, Middle District of North Carolina); DAVID S. TATEL (U.S. Court of Appeals for the District of Columbia Circuit).

National Academies of Sciences, Engineering, and Medicine Staff: STEVEN KENDALL, Program Officer; ANNE-MARIE MAZZA, Senior Director; SOPHIE BILLINGE, Senior Program Assistant.

SPONSORS: This activity was sponsored by the National Science Foundation. For additional information about the Committee on Emerging Areas of Science, Engineering, and Medicine for the Courts: Identifying Chapters for a Fourth Edition of the Reference Manual on Scientific Evidence—A Workshop, visit https://www.nationalacademies.org/event/02-24-2021/emerging-areas-of-science-engineering-and-medicine-for-the-courts-identifying-chapters-for-a-fourth-edition-of-the-reference-manualon-scientific-evidence-virtual-workshop. For additional information about the Committee on Science, Technology, and Law (CSTL), visit https://www.nationalacademies.org/cstl/committee-on-science-technology-and-law.

Suggested citation: National Academies of Sciences, Engineering, and Medicine. 2021. Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. https://doi.org/10.17226/26231.

Policy and Global Affairs

Image

Copyright 2021 by the National Academy of Sciences. All rights reserved.

Suggested Citation:"Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshopin Brief." National Academies of Sciences, Engineering, and Medicine. 2021. Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26231.
×
Page 1
Suggested Citation:"Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshopin Brief." National Academies of Sciences, Engineering, and Medicine. 2021. Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26231.
×
Page 2
Suggested Citation:"Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshopin Brief." National Academies of Sciences, Engineering, and Medicine. 2021. Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26231.
×
Page 3
Suggested Citation:"Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshopin Brief." National Academies of Sciences, Engineering, and Medicine. 2021. Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26231.
×
Page 4
Suggested Citation:"Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshopin Brief." National Academies of Sciences, Engineering, and Medicine. 2021. Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26231.
×
Page 5
Suggested Citation:"Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshopin Brief." National Academies of Sciences, Engineering, and Medicine. 2021. Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26231.
×
Page 6
Suggested Citation:"Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshopin Brief." National Academies of Sciences, Engineering, and Medicine. 2021. Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26231.
×
Page 7
Suggested Citation:"Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshopin Brief." National Academies of Sciences, Engineering, and Medicine. 2021. Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26231.
×
Page 8
Suggested Citation:"Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshopin Brief." National Academies of Sciences, Engineering, and Medicine. 2021. Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26231.
×
Page 9
Suggested Citation:"Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshopin Brief." National Academies of Sciences, Engineering, and Medicine. 2021. Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26231.
×
Page 10
Suggested Citation:"Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshopin Brief." National Academies of Sciences, Engineering, and Medicine. 2021. Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26231.
×
Page 11
Suggested Citation:"Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshopin Brief." National Academies of Sciences, Engineering, and Medicine. 2021. Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26231.
×
Page 12
Suggested Citation:"Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshopin Brief." National Academies of Sciences, Engineering, and Medicine. 2021. Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/26231.
×
Page 13
Emerging Areas of Science, Engineering, and Medicine for the Courts: Proceedings of a Workshop–in Brief Get This Book
×
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

On February 24-25, 2021, an ad hoc planning committee under the auspices of the National Academies of Sciences, Engineering, and Medicine's Committee on Science, Technology, and Law hosted a workshop titled Emerging Areas of Science, Engineering, and Medicine for the Courts. The workshop was organized to explore emerging issues in science, technology, and medicine that might be the basis of new chapters in a fourth edition of the Reference Manual on Scientific Evidence. The Reference Manual, a primary resource for federal judges on questions of science in litigation, is a joint publication of the National Academies of Sciences, Engineering, and Medicine and the Federal Judicial Center, the research and education arm of the federal judiciary.

Over the course of the workshop, judges discussed how they evaluate scientific evidence in court and scientists and others spoke about emerging issues in science and technology that may come before the courts in coming years. This publication highlights the presentation and discussion of the workshop.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!