FRAMING THE ISSUES
The United States faces a complex array of challenges to its national security. Technology is an essential element of the U.S. strategy for meeting those challenges, and it continues to be U.S. policy to seek technological military superiority over U.S. adversaries. To enhance and expand technological superiority, the Department of Defense and other government agencies invest in science and technology on an ongoing basis. These investments cover a broad range of efforts, from fundamental research that might eventually support national security needs, broadly defined, to specific development and eventual production of weapons and other military materiel intended to address particular national security problems. The U.S. government also adapts technologies originating in the civilian sector, initially without national security purpose, to national security needs.
Developments in science and technology (S&T) for military and national security use have often raised a variety of ethical, legal, and societal issues (ELSI). These ELSI-related challenges are accentuated in a context of emerging and readily available (ERA) technologies, that is, new technologies that are accessible at relatively low cost compared to more traditional militarily relevant technologies, such as nuclear weapons, and thus are within the reach of less technologically advanced nations, non-state actors, and even individuals. This is true because ERA technologies do not require construction of large engineered systems for their exploita-
tion, and in some cases have the potential for doing harm to U.S. interests on a large scale.
In 2010, the Defense Advanced Research Projects Agency asked the National Academies to develop a framework for policy makers, institutions, and individual researchers to use in thinking through ethical, legal, and societal issues as they relate to research and development (R&D) on ERA technologies with military or other national security relevance. What are the ethics of using autonomous weapons that may be available in the future? How should we think about the propriety of enhancing the physical or cognitive capabilities of soldiers with drugs or implants or prostheses? What limits, if any, should be placed on the development of cyber weapons, given the nature and extent of the economic damage that they can cause? Such questions illustrate the general shape of ethical, legal, and societal issues considered in this report.
This report begins with the assumption that defending and protecting national security against external threats are morally sound and ethically supportable societal goals. A related premise is that individuals who are part of the national security R&D establishment want to behave ethically.
That said, the notion of deliberately causing death and destruction, even in defense of the nation from external threats, raises ethical, legal, and societal issues for many. Those who engage in combat, those who support combatants, directly or indirectly, and those whom they defend—that is, the American public at large—all have a considerable stake in these issues and the questions they raise.
Knowledge regarding ethical, legal, and societal issues associated with R&D for technology intended for military purposes is not nearly as well developed as that for the sciences (especially the life sciences) in the civilian sector more generally. (This is generally true, even recognizing that the line between military and civilian technologies is not always entirely clear.) Some of the important differences between the two contexts include the following:
• Unlike civilian technologies, some military technologies are designed with the explicit purpose of causing harm to people and to property.
• Civilian technologies and products may unexpectedly turn out to be relevant to a military need and in that context raise the possibility of heightened and/or new ELSI implications.
• Technologies developed in a military context may turn out to have significant ELSI implications when applied in a civilian context.
• Advancing military technologies may also outpace the evolution of the laws designed to govern their use. For example, cyber weapons offer
the possibility that a nation might be brought to economic ruin without physical death and destruction.
• Some military research is conducted in a classified environment.
A full investigation of ethical, legal, and societal issues associated with technology for military or national security purposes is beyond the scope of this report. To make its task more manageable, the committee explored three areas with respect to ERA technologies:
• The conduct of research, which includes the selection of research areas, the design of particular research investigations (e.g., protocols, experiments), and the execution of those investigations. ELSI concerns relating to the conduct of research focus primarily on the effects of the research on parties other than those who are explicitly acknowledged as being research subjects, such as individuals living close to where the research is being performed, family members of research subjects, and so on. (ELSI concerns related to acknowledged research subjects are important, but there is today a well-developed infrastructure to address such concerns, and the adequacy of this infrastructure is not within the scope of this report.)
• Research applications, which relate to capabilities intended to result from research on ERA technologies. ELSI concerns associated with specified applications fall into two categories: concerns about the intended effects or purposes of the application and concerns about undesired effects (sometimes known as side effects) that might occur in addition to the intended effects. Concerns about technologies that can be used for both military and civilian purposes fall into this category.
• Unanticipated, unforeseen, or inadvertent ELSI consequences of either research or applications; such consequences are usually manifested by something going awry, as when research does not proceed as expected and thus causes harm outside the original bounds on the research or when unanticipated applications raise additional ELSI concerns.
FOUNDATIONAL TECHNOLOGIES AND APPLICATIONS
For illustrative purposes, this report considers three foundational technologies (foundational sciences and technologies) that enable progress and applications in a variety of problem domains: information technology, synthetic biology, and neuroscience. In addition, four application domains associated with specific operational military problems are addressed: robotics, prosthetics and human enhancement, cyber weapons, and nonlethal weapons. These technologies and applications are examples of ERA technologies as defined above—a multitude of state and nonstate
actors, friendly or not, can adopt and adapt such technologies for a multitude of purposes even without large budgets and infrastructures. The report examines each illustrative ERA technology and application domain from the perspective of technology maturity (how close the science or technology is to producing useful applications) and possible military applications, and it highlights some of the ELSI implications that emerge for each technology or application.
SOURCES OF INSIGHT ON ETHICAL, LEGAL, AND SOCIETAL ISSUES AND AN ANALYTICAL FRAMEWORK
A number of ideas, intellectual disciplines, and related efforts are sources of ELSI insight into both new and existing technologies and their applications. These include philosophical and disciplinary ethics; international law (especially the law of armed conflict and various arms control treaties); social and behavioral sciences; scientific and technological framing; the precautionary principle and cost-benefit analysis; and risk science and communication. Considered together, they help to provide an analytical framework consisting of three types of questions:
• Questions regarding various stakeholders that might have a direct or indirect interest in particular ELSI concerns and perspectives. Among these stakeholders are subjects of research, military users of a technology or application, adversaries, nonmilitary individuals or groups that might use a technology or application once R&D has been completed, organizations, noncombatants, and other nations.
• Questions that cut across these stakeholder groups and that cluster around a number of themes reflecting ELSI impacts related to scale, including, for example, degree of harm; humanity, including what it means to be human; technological imperfections; unintended military uses; and opportunity cost, among others.
• Questions that arise from a consideration of the different sources of ELSI insight described in Chapter 4.
Drawing on ELSI-related insights from the consideration of the three foundational ERA technologies and four ERA technology-based applications discussed in Chapters 2 and 3, the report sets forth a framework to help identify ethical, legal, and societal issues that might not otherwise be apparent to program officials. Addressing the relevant questions associated with each stakeholder should help to develop useful knowledge on ethical, legal, and societal issues regarding specific military R&D programs and projects. Such knowledge can be used to determine how and to what extent, if any, a program or project might be modified—or in
extreme cases abandoned—because of ELSI concerns. Use of this framework can thus provide input to policy makers, who will then have to make judgments about how, if at all, to proceed with a particular program or project; such judgments should be undertaken after, and not before, the policy makers have examined the issues raised by the questions posed in the framework.
GOING BEYOND AN INITIAL ANALYSIS
Using the analytical framework offered by this report is likely to bring to light some, although not all, of the ethical, legal, and societal issues associated with R&D on ERA technologies of military significance. Literally anticipating unanticipated ethical, legal, and societal issues is oxymoronic. But the ability to respond quickly to unanticipated issues that do arise can be enhanced by addressing in advance a wide variety of identified issues, because that exercise provides building blocks upon which responses to unanticipated ELSI concerns can be crafted.
In general, the task of anticipating ethical, legal, and societal issues that might emerge in the future would be much easier if the specific path of a given science or technology development were known in advance. However, the history of technology forecasting suggests that inaccurate technology forecasts are not unusual, because a variety of paths for any given scientific or technological development are possible. Also, it sometimes happens that military technologies are used in ways that differ significantly from the original conceptions of use.
Taking an approach that complements predictive analysis, policy makers have sometimes turned to deliberative processes that seek to include a broad range of perspectives and possible stakeholders in discussions of a given issue. From these different perspectives may well come the identification of new risks, questions of fact that have not previously been addressed, and specific knowledge or information that might not have been considered before.
To improve their ability to identify and respond to previously unanticipated ethical, legal, and societal issues that may emerge during the course of an R&D effort, policy makers have sometimes also used adaptive planning that allows them to respond quickly as new information and concerns arise in the course of technology development. Adaptive planning can be a useful way of proceeding despite profound uncertainties about the future. Policies for coping with uncertain environments should take into account the possibility of new information and/or new circumstances emerging tomorrow that can reduce these uncertainties, thus allowing (and indeed including planning for) midcourse corrections.
MECHANISMS FOR ADDRESSING ETHICAL, LEGAL, AND SOCIETAL ISSUES
Various organizations, both public and private, use a number of mechanisms to address different types of ethical, legal, and societal issues. Perhaps the most important mechanism for identifying problematic ELSI concerns that may be associated with a given research project is good judgment. That is, project proposers are expected to exercise good judgment in not submitting proposals that are unethical with respect to either the conduct of the research that would be supported or the applications that might result from that research. The same applies to program officials, who are expected not to approve or support projects that are unethical.
To support, develop, and enhance the judgment of individual proposers and program officials, a number of mechanisms, sometimes topic specific, have been used to address ethical, legal, and societal issues—some apply to research, and some to actual deployments of technology. Mechanisms discussed in Chapter 7 and Appendix D include self-regulation and self-awareness; DOD law-of-armed-conflict review and treaty compliance; codes of ethics and social responsibility in science, engineering, and medicine; ELSI research; oversight bodies (such as institutional review boards); advisory boards; research ethics consultation services; chief privacy officers; environmental assessments and environmental impact statements; and drug evaluation and approval. However, these mechanisms have been developed for use primarily in civilian environments.
Adapting these ELSI mechanisms for the military R&D context must take into account the special characteristics of the military environment. In addition, those responsible must have an awareness of potential ethical, legal, and societal issues in the R&D effort; clear accountability and responsibility for addressing them; access to necessary expertise in ethics, law, and the social sciences, and to ELSI experts who in turn have access to relevant scientific and technical information; time to address ELSI concerns; and finally the involvement of a wide variety of perspectives, as well as comprehensiveness of and cooperation in attention to ethical, legal, and societal issues. Depending on their goals, policy makers will have to decide how far to go in any of these dimensions.
FINDINGS AND RECOMMENDATIONS1
This report finds that some developments in emerging and readily available technologies in a military context are likely to raise complex
1 Boldface below includes findings of the report.
ethical, legal, and societal issues, some of which are different from those associated with similar technologies in a civilian context. ERA technologies by their nature are associated with a very high degree of uncertainty about their future developmental paths, and thus a correspondingly broad range in the ethical, legal, and societal issues that are likely to emerge. Such breadth means that the ELSI concerns that may be associated with a given technology development are very hard to anticipate accurately at the start of that development. Using a diversity of sources of input with different intellectual and political perspectives on a given technology increases the likelihood that relevant ethical, legal, and societal issues will be revealed. Of course, when a particular technology development effort is classified, the universe of sources from which ELSI insights can be derived is more limited, and mechanisms for addressing ethical, legal, and societal issues that are predicated on the relative openness of civilian R&D (that is, unclassified work) are not likely to work as well.
Sustainable policy—policy whose goals and conduct can be supported over the long run—regarding science and technology requires decision makers to attend to the ELSI aspects of the S&T involved. High-quality science is one of the more important and obvious factors that contribute to the success of any particular R&D effort involving that science or technology. But inattention to ELSI aspects of an R&D endeavor can undermine even scientifically sound R&D efforts and call into question policy decisions that led to those efforts, regardless of their initial intent.
Public reaction to a given science or technology effort or application is sometimes an important influence on the degree of support it receives. A lack of support may manifest itself through adverse journalistic and editorial treatment, greater political scrutiny, reduced budgets (especially in a time of constrained finances), additional restrictions on research, and so on. On the other hand, a positive perception regarding the ethics of an R&D project may enhance public support for pursuit of that science or technology, irrespective of the scientific or technical basis for such pursuit.
Finally, any approach to promote consideration of ethical, legal, and societal issues in R&D of military significance will have to address how such plans are implemented at both the program and the project levels. Controversy and concern can easily be fueled by inadequate attention to detail and the manner of implementing oversight processes. For example, it is important that policies for addressing ethical, legal, and societal issues systematically have a “light footprint” when they are implemented by program managers. The intent of the committee’s findings and recommendations is not to impose undue compliance requirements on program managers or agencies, but rather to help well-meaning program manag-
ers in these agencies do their jobs more effectively and to help ensure that basic American ethical values (such as those embodied in the U.S. Constitution’s Bill of Rights) are not compromised. The exercise of common sense, judgment, and understanding of the fundamental intent of policies to address ethical, legal, and societal issues—not simply formal compliance—is the goal and is an important foundation for developing an ELSI-sensitive culture.
The foregoing findings (shown in boldface type) help to shape the committee’s five recommendations, the first four of which are directed to agencies sponsoring research with military significance. The term “interested agency” as used below means agencies interested in addressing ethical, legal, and societal issues associated with the research they support.
Recommendation 1: The senior leaders of interested agencies that support R&D on emerging and readily available technologies of military significance should be engaged with ethical, legal, and societal issues in an ongoing manner and declare publicly that they are concerned with such issues. Such a public declaration should include a designation of functional accountability for ethical, legal, and societal issues within their agencies.
High-level support from senior agency leadership is required if an agency is to seriously address ethical, legal, and societal issues associated with the research it funds. Such support must be visible and sustained over time; in its absence, little will happen. An agency’s senior leadership sets the tone by publicly communicating to the organization and its stakeholders the importance of addressing ethical, legal, and societal issues, the willingness of the agency to learn from outside perspectives, and the intent of the ELSI-related processes. In the long run, these are key elements in creating an institutional culture that is sensitive to ELSI concerns.
Accountability at all levels of an agency, including at the senior management level, is necessary to ensure that attending to ethical, legal, and societal issues is not haphazard and uncoordinated. To maximize the likelihood that ethical, legal, and societal issues will be addressed, an agency’s senior leadership should designate a point of functional accountability for this responsibility. Parties with functional accountability provide a second line of defense against overlooking ELSI concerns that complements the primary role played by project teams in executing a program.
Recommendation 2: Interested agencies that support R&D on emerging and readily available technologies of military significance should develop and deploy five specific processes to enable these
agencies to consider ethical, legal, and societal issues associated with their research portfolios.
2.a–Initial screening of proposed R&D projects
Before supporting any research in a particular S&T area, agencies should conduct a preliminary assessment to identify ethical, legal, and societal issues that the research might raise. In addition, all researchers should identify in their proposals to an agency plausible ELSI concerns that their research might raise. Using such information as a starting point, the funding agency should then make its own assessment about the existence and extent of such issues. Note that this initial assessment should be carried out for all R&D projects (both classified and unclassified).
At this stage, the goal is to identify explicitly whether the research would raise significant ethical, legal, and societal issues that require further consideration. Mostly, the answer will be “no,” and assessment of the proposed research project will proceed without any further consideration of ethical, legal, and societal issues. For the proposals that warrant a “yes,” the process in Recommendation 2.b comes into play.
2.b–Reviewing proposals that raise ELSI concerns
Once an agency has identified research proposals or projects that may raise significant ethical, legal, and societal issues, some closer scrutiny is needed to ascertain how likely it is that such issues will arise, how serious they are likely to be, and whether there are ways to mitigate them. Use of a systematic methodology, such as the analytical framework described in this report, can be helpful for identifying ethical, legal, and societal issues.
If and when such issues are identified, program managers should have the opportunity to take action in response. (Of course, program managers are themselves subject to higher authorities, and the latter may take action as well.) Possible responses include not pursuing a given R&D effort, pursuing it more slowly, pursuing it in a modified form that mitigates the identified ethical or societal concerns, pursuing the original effort but also pursuing research to better understand the ethical or societal impacts, and so on. The responses should not be limited simply to a decision to proceed or not to proceed.
Furthermore, it should be expected that the initial assessment will not be correct in all aspects. But the initial assessment will assemble resources that are likely to be helpful in formulating a response to unforeseen circumstances, even if these resources are used in ways that are very different from what an original plan specified. In addition, the initial assessment is a concrete point of departure for evolving an approach to handling ethi-
cal, legal, and societal issues as circumstances change. Information from the assessment should be made available to modify the research proposal for mitigating ELSI concerns should that be appropriate.
2.c–Monitoring R&D projects for the emergence of ethical, legal, and societal issues and making midcourse corrections when necessary
Perfect prediction of significant ELSI concerns is virtually impossible, especially in an area as fraught with uncertainty as research on emerging and readily available technologies. Projects that seemed to raise significant ethical, legal, and societal issues may turn out to raise none; projects that seemed to have no ethical or societal implications may turn out to have hugely important consequences.
A process for monitoring the course of R&D projects is thus essential to help agencies to adjust to such changing realities. If the perceived ethical, legal, and societal issues change significantly during the course of a project (that is, if and when new issues are identified or previous attempts to address already-identified issues prove inadequate), the program or project plan can be modified accordingly. Such an adaptive approach plans for and relies on continual (or at least frequent) midcourse changes in response to such feedback.
A monitoring process could, in principle, be similar to the initial screening process, with the important proviso that the baseline be updated to take into account what has been learned since the project was last considered. To catch ethical, legal, and societal issues that may have appeared in the interim, the monitoring process should touch all projects in the agency’s R&D portfolio, so that projects that were previously determined not to raise ethical, legal, and societal issues can be reexamined. But the intent of this requirement is not to reopen a debate over a project as initially characterized but rather to see if new issues have arisen since the last examination—and in most cases, a project originally determined to not raise ethical, legal, and societal issues will retain that status upon reexamination. It may also be the case that projects originally determined to raise ethical, legal, and societal issues have evolved in such a way that it becomes clear that they do not.
2.d–Engaging with various segments of the public as needed
With the stipulation that engagement with various segments of the public does not necessarily mean coming to consensus with them, an agency’s ELSI deliberations will often benefit from such external engagement. For example, public concerns about a given R&D project are often formulated in ELSI terms rather than in technical terms. Policy makers
must be prepared for the emergence of unforeseen outcomes and thus must have structures in place that will detect such outcomes and focus attention on them in a timely way. When unforeseen outcomes do emerge, policy makers must be prepared to communicate with the public using proven techniques. A developed strategy for public communication is also useful when anticipated ELSI concerns become public. Government actions in the United States ultimately depend, legally and practically, on the consent of the governed. Building public understanding of an agency’s actions, the reasons for those actions, and the precautions the agency has taken will normally be the best strategy, for democracy and for the agency.
In addition, members of the public (including, for example, technical experts, experts on risk assessment and communication, and those with ELSI expertise broadly defined) may have points of view that were not well represented in an agency’s internal deliberations about a given R&D project. Ongoing engagement throughout the course of a project may reveal the impending appearance of initially unanticipated ethical, legal, and societal issues, and thus provide early warning to program managers and enable a more rapid response if and when these new issues do appear. Finally, the mere fact of consultation and engagement with a wide range of stakeholders helps to defuse later claims that one perspective or another was ignored or never taken into account.
Finally, a relevant stakeholder group is the community of researchers themselves. An agency should not suddenly introduce substantive changes in its requirements for proposals without informing the research community about what those changes mean. What is the rationale for these changes? How, if at all, will research projects have to change? What, if anything, does “attending to ethical, legal, and societal issues” mean in the context of decisions about specific proposals?
For R&D projects that are classified, public engagement is obviously constrained to a certain extent. Nevertheless, even if such projects can be discussed only with the cleared subsets of the various stakeholder groups, the result will still be more robust and defensible than if the project had not been discussed at all.
2.e–Periodically reviewing ELSI-related processes in an agency
Well-meaning policy statements are sometimes translated into excessively bureaucratic requirements. To ensure that ESLI-related processes do not place undue burdens on researchers or on program managers in an agency, these processes should themselves be reviewed periodically to ensure that they are consistent with the intent of high-level policy statements regarding the agency’s handling of ethical, legal, and societal issues.
Recommendation 3: Interested agencies supporting R&D on emerging and readily available technologies of military significance should undertake an effort to educate and sensitize program managers to ethical, legal, and societal issues.
If funding agencies are to screen, assess, and monitor research proposals and projects for possibly significant ethical, legal, and societal issues, they will need people with the ability to recognize those issues. The fields that assess ELSI concerns arising with various technologies have their own vocabularies. At the very least, the agency personnel dealing with these issues will have to understand, at some level, the relevant “language.” At the same time, those with ELSI responsibilities and/or expertise must have some understanding of the underlying research in order to identify issues that may or may not emerge.
One crucial, and easily overlooked, aspect of building internal expertise is building history. If an agency has no institutional memory of what ethical, legal, and societal issues it has faced in its history, how it dealt with those issues, and what the consequences were, its ability to learn from that past is diminished. This diminished capability will be a particular problem for agencies that have frequent turnover. An interested agency needs to make it a priority to collect—and to use—information about how it has dealt with these issues. The agency person or group in charge of screening proposals or projects for ethical, legal, and societal issues might be in a good position to collect and organize that kind of information.
Recommendation 4: Interested agencies supporting R&D on emerging and readily available technologies of military significance should build external expertise in ethical, legal, and societal issues to help address such issues.
Not all expertise should be, or can be, internal to an agency. Agencies should seek advice from external experts because properly addressing some ELSI concerns will require a depth of knowledge that cannot realistically be expected of program managers or scientists. If such expertise is not immediately available, it should be cultivated. Such cultivation would have both immediate and longer-term benefits. It would help the agency directly by providing that expertise, but, in the longer run, it could also build knowledge, expertise, and even trust outside the agency about what it does about ethical, legal, and societal issues, and why.
The committee also makes one recommendation to research-performing institutions.
Recommendation 5: Research-performing institutions should provide assistance for researchers attending to ethical, legal, and societal issues in their work on emerging and readily available of military significance.
Recommendations 1 through 4 address government agencies that fund research on emerging and readily available technologies of military significance. To the extent that these recommendations are adopted, researchers supported by these agencies may need assistance in identifying and responding to ethical, legal, and societal issues with which they may be unfamiliar. The committee believes that universities and other research-performing organizations should provide such assistance when needed by the researchers working under their aegis, in much the same way that they provide other functional support to these researchers.
In addition, many institutions performing research on emerging and readily available technologies with military significance already have in place policies and procedures to address a variety of ethical, legal, and societal issues that arise in S&T research. For example, institutional review boards for research involving human subjects are quite common. Leveraging policies and procedures already in place to address ELSI concerns associated with certain kinds of research will help to minimize unnecessary overhead in institutions performing research on ERA technologies with military significance, and where policies and procedures already exist to address ethical, legal, and societal issues that are common to both military and civilian-oriented research, new ones should not be created to address them.