Emerging and Readily Available
— A Framework for Addressing Ethical,
Jean-Lou Chameau, William F. Ballhaus, and Herbert S. Lin, Editors
NATIONAL RESEARCH COUNCIL AND
THE NATIONAL ACADEMIES PRESS
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001
NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance.
Support for this project was provided by the Defense Advanced Research Projects Agency under Award Number HR0011-11-C-0038. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the organizations or agencies that provided support for the project.
International Standard Book Number-13: 978-0-309-29334-1
International Standard Book Number-10: 0-309-29334-0
Library of Congress Control Number: 2013958004
This report is available from
Computer Science and Telecommunications Board
National Research Council
500 Fifth Street, NW
Washington, DC 20001
Additional copies of this report are available from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu.
Copyright 2014 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America
THE NATIONAL ACADEMIES
Advisers to the Nation on Science, Engineering, and Medicine
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences.
The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. C. D. Mote, Jr., is president of the National Academy of Engineering.
The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine.
The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. C. D. Mote, Jr., are chair and vice chair, respectively, of the National Research Council.
COMMITTEE ON ETHICAL AND SOCIETAL IMPLICATIONS OF ADVANCES IN MILITARILY SIGNIFICANT TECHNOLOGIES THAT ARE RAPIDLY CHANGING AND INCREASINGLY GLOBALLY ACCESSIBLE
WILLIAM F. BALLHAUS, The Aerospace Corporation (retired), Co-Chair
JEAN-LOU CHAMEAU, California Institute of Technology, Co-Chair
MARCUS FELDMAN, Stanford University
BRAN FERREN, Applied Minds
BARUCH FISCHHOFF, Carnegie Mellon University
MICHAEL GAZZANIGA, University of California, Santa Barbara
HANK GREELY, Stanford University
MICHAEL IMPERIALE, University of Michigan Medical School
ROBERT H. LATIFF, University of Notre Dame
JAMES MOOR, Dartmouth College
JONATHAN MORENO, University of Pennsylvania
JOEL MOSES, Massachusetts Institute of Technology
KENNETH OYE, Massachusetts Institute of Technology
ELIZABETH RINDSKOPF PARKER, University of the Pacific McGeorge School of Law
SARAH SEWALL, Harvard University
ALFRED SPECTOR, Google, Inc.
JOHN H. TILELLI, JR., Cypress International, Inc.
STEPHEN J.A. WARD, University of Oregon
HERBERT S. LIN, Study Director and Chief Scientist, Computer Science and Telecommunications Board (CSTB)
JON EISENBERG, Director, CSTB
ENITA WILLIAMS, Associate Program Officer, CSTB (through April 2013)
SHENAE BRADLEY, Senior Program Assistant, CSTB
ERIC WHITAKER, Senior Program Assistant, CSTB
RACHELLE HOLLANDER, Director, Center for Engineering, Ethics, and Society
FRAZIER BENYA, Program Officer, Center for Engineering, Ethics, and Society
JO L. HUSBANDS, Senior Program Officer, Board on Life Sciences
ANNE-MARIE MAZZA, Director, Committee on Science, Technology, and Law
COMPUTER SCIENCE AND TELECOMMUNICATIONS BOARD
ROBERT F. SPROULL, University of Massachusetts, Amherst, Chair
LUIZ ANDRÉ BARROSO, Google, Inc.
ROBERT F. BRAMMER, Brammer Technology, LLC
EDWARD FRANK, Apple, Inc.
JACK L. GOLDSMITH III, Harvard Law School
SEYMOUR E. GOODMAN, Georgia Institute of Technology
LAURA HAAS, IBM Corporation
MARK HOROWITZ, Stanford University
MICHAEL KEARNS, University of Pennsylvania
ROBERT KRAUT, Carnegie Mellon University
SUSAN LANDAU, Google, Inc.
PETER LEE, Microsoft Corp.
DAVID LIDDLE, US Venture Partners
BARBARA LISKOV, Massachusetts Institute of Technology
JOHN STANKOVIC, University of Virginia
JOHN SWAINSON, Dell, Inc.
PETER SZOLOVITS, Massachusetts Institute of Technology
ERNEST J. WILSON, University of Southern California
KATHERINE YELICK, University of California, Berkeley
JON EISENBERG, Director
LYNETTE I. MILLETT, Associate Director and Senior Program Officer
VIRGINIA BACON TALATI, Program Officer
SHENAE BRADLEY, Senior Program Assistant
RENEE HAWKINS, Financial and Administrative Manager
HERBERT S. LIN, Chief Scientist, CSTB
ENITA WILLIAMS, Associate Program Officer (through April 2013)
ERIC WHITAKER, Senior Program Assistant
For more information on CSTB, see its Web site at http://www.cstb.org, write to CSTB, National Research Council, 500 Fifth Street, NW, Washington, DC 20001, call (202) 334-2605, or e-mail the CSTB at email@example.com.
BOARD ON LIFE SCIENCES
JO HANDELSMAN, Yale University, Chair
ENRIQUETA C. BOND, Burroughs Wellcome Fund
ROGER D. CONE, Vanderbilt University Medical Center
SEAN EDDY, Howard Hughes Medical Institute
SARAH C.R. ELGIN, Washington University
DAVID R. FRANZ, Consultant, Frederick, Maryland
LOUIS J. GROSS, University of Tennessee, Knoxville
ELIZABETH HEITMAN, Vanderbilt University Medical Center
JOHN G. HILDEBRAND, University of Arizona, Tucson
RICHARD A. JOHNSON, Arnold & Porter, LLP
JUDITH KIMBLE, University of Wisconsin, Madison
CATO T. LAURENCIN, University of Connecticut
ALAN I. LESHNER, American Association for the Advancement of Science
KAREN NELSON, J. Craig Venter Institute
ROBERT M. NEREM, Georgia Institute of Technology
CAMILLE PARMESAN, University of Texas at Austin
ALISON G. POWER, Cornell University
MARGARET RILEY, University of Massachusetts
JANIS WEEKS, University of Oregon
MARY WOOLLEY, Research!America
FRAN SHARPLES, Director
SAYYEDA “AYESHA” AHMED, Senior Program Assistant
CARL-GUSTAV ANDERSON, Program Associate
BETHELHEM M. BANJAW, Financial Associate
KATHERINE BOWMAN, Senior Program Officer
INDIA HOOK-BARNARD, Program Officer
JO L. HUSBANDS, Scholar/Senior Project Director
ANGELA KOLESNIKOVA, Temporary Program Assistant
JAY LABOV, Senior Scientist/Program Director for Biology Education
KEEGAN SAWYER, Associate Program Officer
MARILEE SHELTON-DAVENPORT, Senior Program Officer
COMMITTEE ON SCIENCE, TECHNOLOGY, AND LAW
DAVID KORN, Massachusetts General Hospital, Co-Chair
RICHARD A. MESERVE, Carnegie Institution for Science, Co-Chair
BARBARA E. BIERER, Harvard Medical School
ELIZABETH H. BLACKBURN, University of California, San Francisco
JOHN BURRIS, Burroughs Wellcome Fund
CLAUDE CANIZARES, Massachusetts Institute of Technology
ARTURO CASADEVALL, Albert Einstein College of Medicine
JOE S. CECIL, Federal Judicial Center
ROCHELLE COOPER DREYFUSS, New York University School of Law
DREW ENDY, Stanford University
MARCUS FELDMAN, Stanford University
JEREMY FOGEL, Federal Judicial Center
ALICE P. GAST, Lehigh University
BENJAMIN W. HEINEMAN, JR., Harvard Law School
D. BROCK HORNBY, U.S. District Court, District of Maine
WALLACE LOH, University of Maryland, College Park
MARGARET MARSHALL (retired), Massachusetts Supreme Judicial Court
ALAN B. MORRISON, George Washington University Law School
CHERRY MURRAY, Harvard School of Engineering and Applied Sciences
ROBERTA NESS, University of Texas School of Public Health
HARRIET RABB, Rockefeller University
DAVID RELMAN, Stanford University
RICHARD REVESZ, New York University School of Law
DAVID S. TATEL, U.S. Court of Appeals for the District of Columbia Circuit
ANNE-MARIE MAZZA, Director
STEVEN KENDALL, Associate Program Officer
CENTER FOR ENGINEERING, ETHICS, AND SOCIETY ADVISORY GROUP
JOHN AHEARNE, Sigma Xi, The Scientific Research Society, Chair
ALICE AGOGINO, University of California, Berkeley
STEPHANIE J. BIRD, Ethics Consultant and Co-Editor of Science and Engineering Ethics
GLEN DAIGGER, CH2M HILL GERALD E. GALLOWAY, JR., University of Maryland, College Park
DEBORAH JOHNSON, University of Virginia
WILLIAM KELLY, American Society for Engineering Education
FELICE LEVINE, American Educational Research Association
MICHAEL LOUI, University of Illinois at Urbana-Champaign
DONNA RILEY, Smith College
CHRIS SCHAIRBAUM, Texas Instruments, Inc.
CAROLINE WHITBECK, Case Western Reserve University
WILLIAM WULF, University of Virginia
RACHELLE D. HOLLANDER, Center Director
FRAZIER BENYA, Program Officer
SIMIL RAGHAVAN, Associate Program Officer
VIVIENNE CHIN, Administrative Assistant
In 2010, the Defense Advanced Research Projects Agency (DARPA) asked the National Academies to develop and articulate a framework for policy makers, institutions, and individual researchers that would help them think through ethical, legal, and societal issues (ELSI) as they relate to research and development on emerging and readily available technologies with military relevance.1 The study was motivated in part by DARPA’s experience earlier in the previous decade with programs that encountered difficulties related to privacy concerns and the realization that a more systematic approach to ethical, legal, and societal issues was an important ingredient for success in its mission of avoiding and creating surprise through R&D. Box P.1 contains the full charge to the Committee on Ethical and Societal Implications of Advances in Militarily Significant Technologies That Are Rapidly Changing and Increasingly Globally Accessible.
Coming from the Department of Defense (DOD), this concern—stated so explicitly—is relatively new. The DOD has long required a legal review of whether weapons are in conformance with the law of armed conflict, but this requirement applies only to weapons near procurement and
1 DARPA’s original charge to the committee used the term “democratized technologies” rather than “emerging and readily available technologies.” Democratized or, equivalently, emerging and readily available technologies are those with rapid rates of progress and low barriers to entry. However, the committee believed that the term “democratized” is easily misunderstood, and this report uses the term “emerging and readily available technologies” (ERA technologies). More discussion of this topic is contained in Chapters 1 and 3.
Box P.1 The Project Statement of Task
The National Academies will develop a consensus report on the topic of ethical, legal, and societal issues relating to research on, development, and use of increasingly globally accessible and rapidly changing technologies with potential military application, such as information technologies, synthetic biology, and nanotechnology. This report will articulate a framework for policy makers, institutions, and individual researchers to think about such issues as they relate to these technologies of military relevance and to the extent feasible make recommendations for how each of these groups should approach these considerations in their research activities. A workshop to be held as early as practical in the study will be convened to obtain perspectives and foster discussion on these matters. A final report will be issued within 21 months of the project start, providing the National Research Council’s and National Academy of Engineering’s findings and recommendations.
not to R&D more generally. It is true that certain technologies—genome research, synthetic biology, and nanotechnology, for example—have in the eyes of the U.S. government warranted some degree of explicit attention to ethical, legal, and societal issues. In addition, there is a long history of academic work on ELSI concerns related to various civilian-oriented technologies. But for the most part, these technologies have been exploited for civilian purposes, and work on ethical, legal, and societal issues has been confined largely to that context.
ELSI concerns are inherently challenging, complex, and multidimensional, and their resolution often involves seeking common ground among individuals with deeply held but often unarticulated assumptions about ethics, culture, and epistemology. In some cases, finding common ground may be impossible to achieve in any reasonable time frame. Nevertheless, at the very least, ethical, legal, and societal issues are important enough to deserve serious exploration and attention, even if such common ground cannot be found, and in the committee’s view, DARPA deserves great credit for being willing to raise such issues.
How ELSI expertise and scholarship developed in the context of civilian-oriented science and technology can be applied to the military context is a central theme of this report. But the lessons offered from that expertise and scholarship will require some modification for and adaptation to the military context—that is, they cannot be adopted wholesale, given that the military context does have a number of unique attributes.
Skeptics of the Department of Defense’s attention to ELSI concerns
may well claim that any attempt to argue for uniqueness and processes different from those used for civilian-oriented research is tantamount to shoving hard issues under the table while maintaining a veneer of concern, but the committee does not share this point of view. That is, the committee recognizes the existence of real tensions between military missions (and the technology for supporting those missions) and traditional ELSI concerns. These tensions cannot be eliminated, but it is the committee’s hope that this report can help senior leadership and program managers of agencies that support R&D for military and other national security purposes—including but not limited to DARPA—do a better job of managing these tensions. In addition, the report may also be of value to individual researchers, whether in the defense community or not, who work on the technologies discussed in this report and who may also be interested in the ELSI dimensions of their work.
The committee assembled for this project included individuals with expertise in risk analysis, perception, and communication; ethics; human rights; military operations; military acquisitions; national security law; organizational behavior; media/communications; bioethics; biomedical sciences; and information technology.
The committee first met in August 2011 and five times subsequently. Its earlier meetings were devoted primarily to workshops and plenary sessions for gathering input from a broad range of experts on a variety of topics related to ethical, legal, and societal issues associated with technology of different kinds used in different contexts; later meetings were devoted primarily to committee deliberations. (See Appendix A for brief biographies of committee members and staff and Appendix B for the agendas for the committee’s information-gathering sessions.) The committee heard presentations related to military ethics and law, emerging contexts for military operations, future military missions and technologies for use in these missions, biomedical ethics and engineering ethics, risk assessment and communication, emerging technologies and ELSI concerns, mechanisms used by government agencies to address ethical, legal, and societal issues, approaches to embedding ethics in research and development, and non-U.S. perspectives on ethics in science and technology. In addition, the committee received input on specific emerging and readily available technologies, including information technology, neuroscience, prosthetics and human enhancement, synthetic biology, cyber weapons, robotics and automated weapons, and nonlethal weapons. Additional input included perspectives from professional conferences, the extant literature regarding ELSI concerns and science and technology, and government reports studied by committee members and staff.
The complexity of the issues explored in this report meant that the committee had much to learn from its briefers. The committee is grateful to many parties for presentations on the following dates:
• August 30-31, 2011. Shannon French (Case Western Reserve University), Ward Thomas (College of the Holy Cross), Judith Miller (formerly of the Department of Defense), Peter Schwartz (Global Business Network), Scott Wallace (U.S. Army (ret.)), George Lucas (U.S. Naval Academy), Patrick Lin (California Polytechnic State University), R. Alta Charo (University of Wisconsin Law School), and Joseph Herkert (Arizona State University).
• November 2-3, 2011. Peter Lee (Microsoft Research), Keith Miller (University of Illinois, Springfield), Gloria Mark (University of Irvine), Simson Garfinkel (Naval Postgraduate School), Scott Grafton (University of California, Santa Barbara), Craig Stark (University of California, Irvine), Martha Farah (University of Pennsylvania), Stuart Harshbarger (Contineo Robotics), Daniel Palanker (Stanford University), Gerald Loeb (University of Southern California), Nicholas Agar (Victoria University of Wellington, New Zealand), James Hughes (Trinity College), George Church (Harvard University), Drew Endy (Stanford University), Nita A. Farahany (Vanderbilt University), Judith Reppy (Cornell University), and George Khushf (University of South Carolina).
• January 12-13, 2012. Deborah Johnson (University of Virginia), Sheila Jasanoff (Harvard University, Kennedy School of Government), David Rejeski (Woodrow Wilson Center), Malcolm Dando (University of Bradford, United Kingdom), Kelly Moore (National Science Foundation), Jean McEwen (National Human Genome Research Institute), Valery Gordon (National Institutes of Health), Fred Cate (Indiana University School of Law), Ray Colladay (DARPA (ret.)), Mark Seiden (Yahoo!), Randall Dipert (University of Buffalo), Neil Rowe (Naval Postgraduate School), Ron Arkin (Georgia Institute of Technology), Peter Singer (Brookings Institution), Jürgen Altmann (Technische Universität Dortmund, Germany), Denise Caruso (Carnegie Mellon University), and Peter Hancock (University of Central Florida).
• April 12-13, 2012. Heather Douglas (University of Waterloo, Canada), Alex John London (Carnegie Mellon University), Nils-Eric Sahlin (Lund University, Sweden), Paul Fischbeck (Carnegie Mellon University), Wandi de Bruin (Carnegie Mellon University), Arthur Lupia (University of Michigan), Adam Finkel (Carnegie Mellon University), William Brinkman (U.S. Department of Energy, Office of Science), Carmen Maher (U.S. Food and Drug Administration, Office of the Chief Scientist), Edward Knipling (U.S. Department of Agriculture), Diana Hoyt (National Aeronautics and
Space Administration), Qiu Renzong (Chinese Academy of Social Science, China), Frans Brom (Utrecht University, The Netherlands), Steven Lee (Hobart and William Smith Colleges), and Montgomery McFate (U.S. Naval War College).
• June 4, 2012. George Perkovich (Carnegie Endowment for International Peace), David Fidler (Indiana University), and Neil Davison (International Committee of the Red Cross).
The committee also appreciates the support of Norman Whitaker from the Defense Advanced Research Projects Agency in the conduct of this project. In addition, the committee acknowledges the intellectual contributions of NRC and NAE staff: Herbert S. Lin (study director and chief scientist of the Computer Science and Telecommunications Board (CSTB)), Jon Eisenberg (director, CSTB), Enita Williams (associate program officer, CSTB), Rachelle Hollander (director, Center on Engineering Ethics), Frazier Benya (program officer, Center for Engineering, Ethics, and Society), Anne-Marie Mazza (director, Committee on Science, Technology, and Law), and Jo Husbands (senior program officer, Board on Life Sciences). Shenae Bradley and Eric Whitaker (both senior program assistants for CSTB) provided administrative support. Special thanks are also due to Patricia Wrightson (associate director of the Board on Global Science and Technology), who contributed time and expertise as a staff consultant.
Jean-Lou Chameau, Co-Chair
William F. Ballhaus, Co-Chair
Committee on Ethical and Societal Implications of Advances in Militarily Significant Technologies That Are Rapidly Changing and Increasingly Globally Accessible
Acknowledgment of Reviewers
This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the National Research Council’s Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following individuals for their review of this report:
Carlos Betha, United States Air Force Academy,
Kathleen Clark, Washington University School of Law,
Nancy Connell, University of Medicine and Dentistry of New Jersey,
David Fidler, Indiana University Maurer School of Law,
Shannon French, Case Western Reserve University,
Paul Gaffney, Monmouth University,
Elizabeth Heitman, Vanderbilt University Medical Center,
Deborah Johnson, University of Virginia,
David Korn, Harvard University,
Miltos Ladikas, University of Central Lancashire,
Maria Lapinski, Michigan State University,
Patrick Lin, California Polytechnic State University,
Lester L. Lyles, United States Air Force (retired),
Richard O’Meara, Rutgers University,
David Relman, Veterans Administration Palo Alto Health Care System,
Robert F. Sproull, Oracle (retired),
Detlof von Winterfeldt, University of Southern California, and
John Weckert, Charles Sturt University.
Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations, nor did they see the final draft of the report before its release. The review of this report was overseen by Julia Phillips from Sandia National Laboratories and Kenneth Keller from the Johns Hopkins University Bologna Center. Appointed by the National Research Council, they were responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the authoring committee and the institution.