6

HORIZON SCANNING AND FORESIGHT METHODS

A range of tools can be used to think about future risks and opportunities in a structured manner. As noted by Daniel Flynn from the Office of the Director of National Intelligence, these tools “are for future planning in a world where the future cannot be known.”¹ Such tools are commonly used to help shape policy so that entities (such as governments or organizations) are more resilient and better placed to take effective action (IRM, 2018). As explained by the UK Cabinet Office:

Horizon scanning is therefore not about predicting the future, but focused on the early detection of weak signals as indicators of potential change.

The terminology around relevant tools, techniques, and processes involved in horizon scanning has yet to be standardized, which can lead to confusion. In some cases, for example, the overall process of structured reflection on the future is referred to as “horizon scanning” (UK Government Office for Science, 2013), while in others it is termed “foresight” or “future(s) thinking” (FAO, 2013). In this report, the committee has adopted a definition similar to that used by the Organisation for Economic Co-operation and Development (OECD): horizon scanning is “a

___________________

¹ Mr. Flynn spoke during a webinar held for this study on June 11, 2019.

² See https://www.gov.uk/government/groups/horizon-scanning-programme-team.

technique for detecting early signs of potentially important developments through a systematic examination of potential threats and opportunities, with emphasis on new technology and its effects on the issue at hand” (OECD, n.d.a).

Horizon scanning can be integrated into a broader futures-thinking or foresight framework. This framework describes the overall broader process of assessing and understanding the policy implications of relevant developments, as well as identifying desired futures and specific policy actions that can help realize them (see Annex 6-1 for more detailed discussion of these terms). ETH Zurich developed a model foresight process as part of efforts to strengthen policy making in Switzerland (Habegger, 2009) (see Figure 6-1). This model has three phases. The first involves the identification and monitoring of relevant issues, trends, developments, and changes, accomplished using the tool of horizon scanning. The second phase is assessing and understanding the resulting policy challenges, which makes use of different tools. The third phase involves envisioning desired futures and identifying specific policy actions for realizing them, based on the development of specific scenarios.

This chapter considers horizon scanning in depth, starting with an exploration of how it is used as a policy tool. This is followed by an overview of good practices in horizon scanning. This overview considers potential sources of information, the development of criteria to parameterize the scan or to use for evaluating the outcome, and avenues for improving traditional horizon-scanning methods. Also considered are issues related to communicating the results, connecting the results to specific actions, and learning lessons from the past. To demonstrate how horizon scanning works in practice, the chapter then presents case studies of relevant scans carried out in the past, both in the United States and in other parts of the world. Several of these case studies focus specifically

on biotechnology, while others have been produced by sectors potentially relevant to this study, such as defense, health, food safety, agriculture, and environment and conservation. Next, the chapter places horizon scanning within the broader context of exploring a number of relevant toolkits, handbooks, and guidance, as well as the application of forecasting, or future thinking, by what is termed “superforecasting.” The chapter ends with the committee’s conclusions outlining a possible mechanism for future thinking and horizon scanning tailored to the U.S. bioeconomy, based on existing best practice and making use of current resources.

HORIZON SCANNING AS A POLICY TOOL

Horizon scanning, often as part of a foresight process, can help address a wide variety of policy-making needs (see Annex 6-1 for an overview of one such analysis). It can also generate important information (such as the identification of important trends or developments), and help gain lead time in addressing future issues or serve as an input for scenario-development processes (European Commission, 2015; OECD, n.d.a). It can help ensure that policy making incorporates “thinking outside the box” and that it is able “to manage risk by planning ahead for unlikely, but potentially high impact events” (UK Government Office for Science, 2013). More broadly, benefits accrue from bringing together experts and policy makers from different backgrounds and disciplines (Habegger, 2009). It is important to recognize, however, that horizon scanning operates beyond a firm evidence base and relies on the instincts of those involved in the exercise (UK Government Office for Science, 2017).

The process of horizon scanning can be considered to encompass two separate approaches: “Continuous scanning activities to keep the overview (often with regular newsletters), regular but discontinuous activities (e.g., every 5 years) and ad-hoc Horizon Scanning for a specific purpose, on demand or at a specific occasion” (European Commission, 2015). A number of different horizon-scanning methods have been identified. For example, the Food and Agriculture Organization of the United Nations (FAO) developed a typology that includes best–worst scanning for prioritizing trends or developments, delta scanning for capturing identified trends and developments from other horizon-scanning processes, expert consultations for tapping specialist knowledge, and manual scanning to identify signals of change to track trends and drivers. FAO also provided examples of how each of the methods is commonly used and provided indicative strengths and weaknesses for each (FAO, 2013).

Horizon scanning has been explicitly integrated into policy-making processes in some parts of the world. For example, the United Kingdom has integrated horizon scanning into its central policy making through

its Cabinet Office. The United Kingdom uses horizon scanning as part of a larger foresight process to gather information on relevant trends and developments (monitoring) and explore their possible implications. Horizon scanning is additionally used as a mechanism for engaging people in future thinking and generating an environment conducive to yielding insights into the changing policy environment. Similar efforts have been undertaken, for example, in Singapore (Chong et al., 2007), the Netherlands (European Environmental Agency, 2011), and Switzerland (Habegger, 2009). Efforts in Singapore have focused heavily on automating a horizon-scanning process.

GOOD PRACTICES IN HORIZON SCANNING

The Horizon-Scanning Process

A number of different horizon-scanning processes have been described, including by the UK Government Office for Science (2017), the European Union (EU) Directorate-General (DG) for Research and Innovation (European Commission, 2015), the Institute for Risk Management (IRM, 2018), and several academic groups (Brown et al., 2005; Habegger, 2009; Wintle et al., 2017). An example of a horizon-scanning process is provided in Figure 6-2. In general, these processes share the following features. They start by defining the scope of the scan and then identifying experts likely to have important relevant insights. For example, the IRM process emphasizes the importance of involving a diverse range of participants with open minds (IRM, 2018). Several other models stress that the process can be open-ended, involving as many people as desirable. Of course, increasing the number of people involves additional burdens in terms of tracking and compiling the results and may necessitate a dedicated project manager. Participants are then tasked with compiling a structured scan of a specific issue in a fixed timeframe. For example, the UK process suggests one scan per person per week (UK Government Office for Science, 2017). The issues to be covered can either be pre-identified or identified at the discretion of the participants, thereby drawing on their expertise and insights as to what may be relevant. Each scan describes the trend or development identified, how it relates to the policy or strategy area being explored, why the participants found it important, and what thoughts it stimulated. These descriptions can usefully contain links to original sources or additional information, but preferably are short. For example, the UK process suggests no more than one page (UK Government Office for Science, 2017).

Some processes stop at this point, and their final output is a series of collated issue scans over time, although this output is then sometimes fed

into other activities as part of a larger process, as is the case in the United Kingdom (UK Government Office for Science, 2013). Other processes go further and provide additional steps that involve discussing, refining, rating, or otherwise reviewing the scans within the horizon-scanning process itself. For example, the process developed by the EU DG for Research and Innovation calls for expert dialogue. Some of the academic processes involve a more comprehensive semiquantitative approach, including the need for in-person interaction through a workshop (European Commission, 2015). Some processes then include additional steps to package and frame the results to facilitate their use in policy making. For example, the IRM process highlights the value of visualization (IRM, 2018).

Optimizing a Horizon-Scanning Process

Several factors, such as the sources of information, the decision criteria, methodological tools to tailor the generic process, and the policy impact need to be considered when seeking to optimize a horizon-scanning process (for more detailed discussion of each of these factors, see Annex 6-1).

Sources of information—Information for a horizon scan can come from a number of different sources. Some sources, such as publications, quantitative data, and published opinions, may be more traditional. To reach the limits of current thinking, however, less traditional sources, such as

news outlets, social media, and prepublication servers, may be needed. The process of gathering information can also be increasingly automated as the topic becomes more familiar.

Decision criteria and questions to ask—Either when developing a scan on a topic or when reviewing its potential policy impact, a range of criteria can be applied, such as credibility, novelty, likelihood, impact, relevance, time to awareness (how long before the topic or its impact is widely known), and time to prepare for the development. A number of specific questions for exploring each of these criteria have been proposed (Hines et al., 2018).

Methodological tools to tailor the generic process—A number of recent publications describe methodological tools for horizon scanning. Examples include the use of pre-developed scenarios to aid in the identification of important weak signals (Rowe et al., 2017); more structured approaches for matching specific horizon-scanning tools to the needs of policy makers, including better metrics (Amanatidou et al., 2012); the integration of more comprehensive collaborative review processes to identify appropriate responses by policy makers and practitioners (Sutherland et al., 2012); and mechanisms for assessing the value of different information sources to be used for the horizon scan (Smith et al., 2010).

Increasing the policy impact—A number of good practices for presenting and communicating the results of a horizon scan have been identified, including having a specific sponsor for horizon-scanning and futuring work; translating results in a more accessible manner; tailoring reporting to policy interests; matching timing to political timeframes; selecting experts to increase policy relevance; focusing on potential impacts of events discussed, as well as the timeframes involved; and structuring the results in a logical manner, whether by groups of issues identified or by relevant policy drivers.

Lessons Learned from Past Uses of Horizon Scanning

A number of lessons have been distilled from previous uses of horizon scanning in policy making. For example, horizon-scanning experts consulted by the committee³ discussed (1) the use of expert opinion, (2) sources of bias and approaches to managing them, and (3) options for evaluating the effectiveness of horizon scanning.

On the use of expert opinion, the speakers observed that individuals’ expertise declines dramatically outside the narrow domain of their area of technical specialization or experience, and pointed out that there is also particular value from generalist, nonexpert input. Relatedly, age, number

___________________

³ These experts spoke at a webinar held for this study on June 11, 2019.

of publications, technical qualifications, years of experience, memberships in learned societies, and apparent impartiality do not explain an expert’s ability to estimate unknown quantities or predict future events. However, a number of factors tend to lead to better judgments. An example is experts with experience in fields requiring rapid feedback, such as chess players, weather forecasters, sports players, gamblers, and intensive care physicians. People who are less self-assured and assertive and integrate information from diverse sources also make better judgments. It was noted as well that estimates of risk in many domains can be improved by weighting experts’ opinions by their performance on test questions and that relevant training can improve experts’ abilities to estimate probabilities of events. Lastly, group estimates consistently outperform individual estimates, and diverse groups tend to generate more accurate judgments.

On biases, the experts who spoke to the committee identified the various types of bias and suggested ways to mitigate their effects on the process and outcome of horizon scanning. Gambler’s fallacy (the belief that past events will unduly impact future events) and the availability heuristic (the potential to be overly influenced by more recent memories and events) can be mitigated by identifying and unpacking assumptions inherent in the process, both in the task assigned and on the part of those involved. Confirmation bias (the likelihood of searching for, interpreting, focusing on, and remembering information that confirms preconceptions) can be mitigated by involving participants from a wide range of backgrounds and expertise, drawn from different communities and locations. Projection bias (the belief that preferences will remain the same over time) can lead to focusing on only a subset of issues or options. It can be mitigated by unpacking assumptions and questioning them, as well as by expanding the range of expertise involved in the process. The bandwagon effect, or “groupthink,” increases the likelihood of failing to explore the full range of options or issues, and can be countered by deliberately involving experts from diverse backgrounds and communities. Anchoring bias (the tendency to rely too heavily on a single piece of information, which is often the first obtained) can be mitigated through the use of advocates both for and against a specific issue, as well as multiple rounds of scoring in different orders. Finally, salience bias (the likelihood of focusing on something more prominent or emotionally impactful, especially when particularly vocal or skilled raconteurs are advocating for specific issues) can be managed through rules on advocating positions that are consistently and rigorously enforced, as well as the use of voting and anonymous feedback.

On evaluating the effectiveness of horizon scanning, the experts commented that attempts have been made to review the impact of past horizon scans. As might be expected, these efforts have demonstrated that some

issues were identified in a timely manner and deemed impactful, while others that were identified ultimately had a minimal impact (Sutherland et al., 2012). However, given that horizon scanning is not about predicting the future, assessing the “hit rate” of predictions is an inappropriate metric. The absence of an event is not necessarily the absence of impact. Identifying an early signal and taking effective policy action may result in an apparent null outcome. Metrics for a horizon-scanning or futuring effort might therefore be focused more usefully on exploring whether the effort led policy makers to consider more issues or explore more options. Alternatively, useful insights might be gained by comparing the assessments resulting from a horizon scan against those resulting from other tools with respect to facilitating better policy making.

Publications from other entities, such as the National Intelligence Council (NIC, 2017), the U.S. Forest Service (Hines et al., 2018), the UK government (UK Government Office for Science, 2017), and the EU (European Commission, 2015), have documented reflections, key considerations, rules for implementation, and improvements made through iterative use of horizon scanning (see Annex 6-1 for a detailed discussion of lessons learned).

CASE STUDIES OF HORIZON SCANNING

A number of horizon scans relevant to this study have already been carried out. Both the content of these scans and the communities that produced them could serve as important resources moving forward. The committee noted a paucity of documented horizon-scanning activities performed by U.S. federal agencies. Should relevant federal agencies be carrying out these activities, there is considerable room to enhance transparent reporting of and sharing of experiences from those efforts.

This section provides examples of past scans and the actors undertaking them. The scans reviewed include those directly connected to the bioeconomy, those conducted within the U.S. Intelligence Community, those carried out by agencies with a direct role to play in safeguarding the bioeconomy, and those conducted within a U.S. federal agency. Examples of additional horizon scans are described in Annex 6-1, including efforts that have brought together separate horizon scans from different agencies and subject-specific scans in areas related to the bioeconomy, such as health, food safety, and the environment and conservation.

Example of a Horizon Scan Connected to the Bioeconomy

In 2017, a transatlantic horizon scan was published describing developments in biological engineering likely to have substantial impacts on global society. The process brought together experts in horizon scanning,

biosecurity, plant biotechnology, bioinformatics, synthetic biology, the bioeconomy, biodefense, science policy, nanotechnology, conservation and environmental sciences, industrial biotechnology, and the social sciences. These experts used the process described in the section above on the horizon-scanning process to identify 70 potential issues and then prioritized 20 of these issues, covering such sectors as health, energy, agriculture, and the environment (Wintle et al., 2017) (see Table 6-1).

The 20 prioritized issues were categorized according to their likely timeline for impact. Highlighted as likely to have an impact within 5 years were five issues, including novel approaches to gene drives (which subsequently received notable backing for development from major science

TABLE 6-1 Issues in Biological Engineering Likely to Have Substantial Impacts on Global Society in the Short, Medium, and Long Terms

SOURCE: Adapted from Wintle et al., 2017.

funders) (Wellcome Trust, 2017), human genome editing (2018 saw the birth of the first genome-edited babies) (Cyranoski and Ledford, 2018), and accelerated defense agency research (with novel research programs causing debate within the biosecurity community on the desirability of such research) (Lentzos and Littlewood, 2018). Ten issues were deemed likely to have an impact in 5–10 years, including cyberbiosecurity and corporate espionage and biocrimes (which are directly connected to the aims of this study). Finally, five issues were identified as likely to have an impact in more than 10 years, including securing critical infrastructure needed to deliver the bioeconomy.

Example of Horizon Scanning Within the U.S. Intelligence Community

Shortly after the start of each presidential term, NIC publishes “an unclassified strategic assessment of how key trends and uncertainties might shape the world over the next 20 years to help senior U.S. leaders think and plan for the longer term” (NIC, 2017). Comparatively few details are publicly available about the precise methodology used by NIC, but according to the NIC (2017) report, it involved desk research as well as consultations with experts from inside the U.S. government and from around the world. This enabled the identification of, and subsequent reflection on, key assumptions and trends. Assessment of implications was first carried out at the regional level before being aggregated to identify global trends. The results were structured over different timeframes, ranging from the near term (5 years) to the long term (20 years). Analytic simulations were used to explore future scenarios, in particular how uncertainties and trends might combine to alter outcomes.

The scale and breadth of the consultations reported were also noteworthy:

These expert interviews and the feedback received were then integrated into a scenario-based, policy-oriented foresight approach. The scenario work and backcasting efforts were used to identify choices and policy decisions that could help realize desirable futures and avoid the undesirable (NIC, 2017). Specific tools used in the preparation of the NIC report that might be important for forecasting work relevant to this study include net assessment and analytic simulations. Net assessment is

Net assessment “uses data that are widely available and creates strategic insights that lead to decisive advantage. It offers paths through the increasingly dangerous landscape of national security.” It often makes use of a specific set of tools. “Scenarios, war games, trend analysis, and considered judgment are the methods most widely used in net assessment studies and analyses” (Bracken, 2006).

Analytic simulations, including historical wargaming and analytic path games, have proven useful in military planning for future conflicts. They have allowed commanders to plan for the unknown by both better understanding adversaries and preparing possible responses in advance of events.⁴

Example of Horizon-Scanning Tools Being Developed by an Agency Connected to Safeguarding the Bioeconomy

In 2015, the Office of Technical Intelligence in the U.S. Department of Defense published an assessment of data analytics–enabled technology watch and horizon scanning (TW/HS) for the identification, characterization, and forecasting of known and unknown science, technology, and applications (Office of Technical Intelligence, 2015). According to the assessment report, “data-enabled TW/HS has the potential to improve upon or augment current approaches by expanding the aperture of analyses and decreasing the influence of bias, while at the same time building

___________________

⁴ This observation was made by a participant in the committee’s webinar on June 11, 2019.

institutional capacity.” The report includes a structured framework for integrating new technologies (such as data analytic tools) into existing workflows. This framework reflects components of the generic horizonscanning process described earlier, including the following (all descriptions are from Office of Technical Intelligence [2015]):

Characterizing decisions (see the above discussion of criteria and questions to ask)—Those undertaking the scan need an understanding of the decision itself; the timeline governing their work; and, most important, the evaluation criteria. This understanding “informs the scope, scale and context of the supporting analysis, which enables analysts to provide targeted, actionable inputs into the decision process in time for the information to be actionable.”

Selecting data (see the above discussion of sources of information)—This process “requires careful balancing of relevance and breadth. It is critical to identify sources that are likely to provide signals relevant to the evaluation criteria and to maximize the signal to noise ratio.”

Selecting metrics (see the above discussion of methodological tools and lessons learned from past uses of horizon scanning)—“Evaluation criteria are often complex human ideas which cannot be precisely calculated from data. For example, analytics cannot directly assess the maturity of a technology, but they could analyze the amount of activity which references the technology, growth rates of activity, or identify whether sources discuss prototyping or advanced testing to inform a technology readiness level estimation.”

Conducting analysis (see the above discussion of decision criteria and questions to ask)—“To enable more effective application of metrics, it is often valuable to develop a taxonomy of the field under consideration. Taxonomies allow for the identification of areas at the same level of abstraction.”

Developing decision support products (see the above discussion of increasing policy impact)—“Analysts must integrate the disparate portions of their findings into a cohesive whole in order to make their efforts useful to decision makers… [this] requires understanding what is useful to the decision maker, such as whether the individual metrics or a composite score would be most useful and how to communicate the findings so that they are both clear and most likely to be used effectively.”

Leveraging knowledge management (see the above discussions)—“In order to move from a successful TW/HS project to a TW/HS program, it is important to ensure that products can be kept up to date with manageable amounts of effort and to track the accuracy of analysis.”

Example of a Horizon Scan in a U.S. Federal Agency

In 2018, the U.S. Forest Service’s Strategic Foresight Group and the University of Houston’s Foresight Program published a summary of their efforts “to develop an ongoing horizon scanning system as an input to developing environmental foresight: insight into future environmental challenges and opportunities, and the ability to apply that insight to prepare for a sustainable future” (Hines et al., 2018). The process adopted was similar to that described earlier. It included an initial framing phase in which the domain of interest was mapped (including the identification of key activities, stakeholders, and drivers of change), geographic and timeframe boundaries were set, relevant stakeholders and participants were identified, and guiding questions were developed. The scan itself used a four-step process:

• Find: identify where and how to look for scanning hits.
• Analyze: use cross-level analysis and cross-layered analysis.
• Frame: develop a framework for organizing insights.
• Apply: use the results in work processes.

The criteria used in the scan to determine the relevance of an issue were those described earlier in the discussion of criteria and questions to ask. The authors identify a number of specific lessons learned from attempting to develop a horizon-scanning process within a U.S. federal agency. The study also includes a discussion of future plans for improving the communication of results, integrating the results into the host organization, and linking the results to effective action, as well as making the process self-sustaining.

Examples of Environment- and Conservation-Related Horizon Scans

One example of an international horizon-scanning effort related to the environment and conservation is a 2016 international study by academic authors from 11 countries that focused on issues likely to impact pollinators and pollination positively or negatively in the future and that succeeded in identifying six high-priority issues and nine secondary issues (Brown et al., 2016). A second example is a 2018 international study by academic authors from six countries that identified “15 emerging priority topics that may have major positive or negative effects on the future conservation of global biodiversity, but currently have low awareness within the conservation community” (Sutherland et al., 2019). The latter

is the tenth annual review conducted by this group, and its methodology was employed in the scan of biological engineering described previously.

ADDITIONAL TOOLS FOR FUTURE THINKING

In practice, horizon scanning is rarely used in isolation, but is often combined with a range of other tools and techniques. Sometimes, these tools and technique are combined into a stand-alone exercise (such as the integration of Delphi, a consultation process to gather input from a wide variety of experts and sometimes prioritize the results, and other expert review processes discussed in Annex 6-1). Alternatively, horizon scanning can be embedded in a more comprehensive foresight process that feeds the results of the scan into processes for assessing and understanding the consequent policy challenges, connecting them to possible future scenarios, and identifying specific policy actions designed to steer toward desirable outcomes. See Annex 6-1 for further detail on the additional tools discussed here.

Forecasting Tools

Several studies have catalogued a comprehensive range of forecasting tools. For example, the Handbook of Technology Foresight, published in 2008, explores in depth 19 qualitative tools, 8 quantitative tools, and 9 semiquantitative tools (Popper, 2008) (see Table 6-2). FAO outlined a similar list of tools in 2014, providing a description of each tool, examples of its common use, and its particular strengths and weaknesses (FAO, 2013). And OECD has highlighted four tools as being particularly important: the scenario method, the Delphi method, horizon scanning, and a trends impact analysis (OECD, n.d.a). Many of these tools have been combined into frameworks for forecasting. Box 6-1 describes an example developed by the UK Government Office for Science.

Superforecasting

In 2010, the Intelligence Advanced Research Projects Agency (IARPA) initiated a competition to explore how crowdsourcing can improve forecasting.⁵ Various tools and approaches for making accurate predictions were tested over 4 years of tournaments. IARPA identified a number of

___________________

⁵ See IARPA’s Aggregative Contingent Estimation at https://www.iarpa.gov/index.php/research-programs/ace/baa.

TABLE 6-2 Foresight Tools Identified by Academic Studies and Intergovernmental Organizations

^aIdentified in the Handbook of Technology Foresight (Popper, 2008).

^bIdentified by the UN Food and Agriculture Organization (FAO, 2013).

^cIdentified by the OECD (OECD, n.d.a).

SOURCES: Compiled from FAO, 2013; OECD, n.d.a; and Popper, 2008.

promising tools, but also concluded that (1) some individuals were notably better at making predictions than others, and (2) it is possible to learn how to be better at making predictions. These two conclusions formed the basis of what was to become known as superforecasting. A superforecasting program brings together those with a proven track record in making predictions in a system designed to enhance their abilities and in making use of tools to help interpret the results. Since the conclusion of this program, a successful team of established superforecasters has created the Good Judgment project, which offers superforecasting capabilities and training for commercial entities and public processes.⁶

___________________

⁶ See https://goodjudgment.com.

Roadmapping

Roadmapping “shows how a range of inputs—research, trends, policy interventions, for example—will combine over time to shape future development of the policy or strategy area of interest” (UK Government Office for Science, 2017). A wide range of countries and regions have developed roadmaps for their bioeconomy.⁷

In 2019, the Engineering Biology Research Consortium published “Engineering Biology: A Research Roadmap for the Next-Generation Bioeconomy.” This roadmap was “intended to provide researchers and other stakeholders (including government funders) with a compelling set of technical challenges and opportunities in the near and long term.” It covers four technical themes and explores five application sectors (see Box 6-2).

CONCLUSIONS

During the committee’s webinar on horizon-scanning methodologies, experts highlighted four key questions to consider when developing a horizon-scanning process.

___________________

⁷ See, for example, https://gbs2018.com/resources/other-resources.

• Approach: Does the activity need to enable scenario planning, identify specific issues that could have a policy impact, or both?
• Scope: Will the horizon-scanning efforts envisaged be broad (e.g., mapping issues that might affect the bioeconomy) or narrow (e.g., mapping all the issues emerging in a specific field)? A broad scope will require interacting with a wide variety of experts, while a narrow scope can more readily be attempted using published resources and desk research.
• Process: Will the data being fed into the future-thinking process come from machine-readable sources or be based on expert opinion?
• Timeframe: Is the intent to look at the near term, identifying issues that are emerging now, or further out, including the far horizon of 10–20 years in the future?

Following discussion of the above questions, the committee concluded that best practices for horizon scanning include the considerations laid out below.

Conclusion 6-1: Approach: Policy making for the bioeconomy will be facilitated by both scenario planning and the identification of issues that could have a policy impact. Therefore, future horizon scanning will need to use at least two different approaches.

Ongoing horizon scanning might be integrated into the work of different agencies with specific fields of expertise, using the good practices discussed in this chapter. Encouraging such agencies to share their experiences with each other would help to build relevant capacity as quickly as possible. In some cases, horizon scanning for important policy issues may already be under way. Different issues identified in these field-specific scans could then be fed into a centralized meta-review. This approach would make use of good practice in horizon scanning (as described in this chapter) to compare different issues using a common set of criteria and scoring systems and multiple rounds of voting. These ongoing activities could form the basis of a regular report, similar to NIC’s Global Trends report.

Conclusion 6-2: Scope: In general, the bioeconomy is broad and cuts across different technical fields, agencies’ work, and communities. The U.S. bioeconomy is currently insufficiently characterized to consider a comprehensive mapping exercise. Broad horizonscanning efforts might help further map the bioeconomy. In the meantime, it is possible that narrowly focused horizon-scanning activities could help answer specific policy questions.

One-off horizon scans could be used to answer specific questions or drill down into specific issue areas. Such a process might follow an approach similar to that of the example horizon scan presented earlier in Figure 6-2. It would include modified use of the Delphi method to highlight issues considered most likely to have a policy impact, or highly novel issues that are likely to be omitted from policy-making processes. One issue that could greatly benefit from both one-off horizon scans and continued assessment is the creation and maintenance of bioeconomy-specific satellite accounts (see Chapter 3 for further detail). This combined approach is particularly suitable for the creation of satellite accounts as it serves a policy need, and the bioeconomy is continually changing.

Conclusion 6-3: Process: Given the need to better understand the bioeconomy and factors that may affect it, future-thinking processes are likely to be human-driven in the near term, but there will be opportunities to automate part of the process as improved data sources and metrics become available.

While these horizon-scanning processes are likely to be expert-driven, tools for automated data gathering are advancing and could be integrated into the methodology used for a horizon scan as appropriate. It will be important to involve the widest possible range and diversity of expertise.

The meta-review process, resources permitting, might resemble the scope, scale, and nature of NIC’s Global Trends report, aiming to directly engage thought leaders from different communities around the world.

Criteria to be applied in assessing potential issues to be fed into horizon scans include credibility (e.g., Is the source reputable? Is it confirmed elsewhere?); novelty (Is the issue new, or has it already been widely reported?); likelihood (What are the chances the issue will actually occur?); impact (Will the issue change the future, and if so, how big will that change be?); relevance (How relevant is the issue to the bioeconomy, and is that relevance direct or indirect?); time to awareness (How long is it likely to be before the issue is widely known, and could this change [or be changed]?); and time to prepare (When is the issue likely to have an impact, what could affect its impact, and when would that intervention need to take place?).

Conclusion 6-4: Timeframe: Given the framing of horizon scanning as a tool for identifying weak signals as early as possible, a notable focus will need to be placed on the longer term. By integrating horizon scanning into a broader foresight process, it will be possible to identify policy options in the near term that could help realize desirable future scenarios and avoid the undesirable. The intent would not be to use the longer-term timeframe of horizon scanning as an excuse to avoid efforts to strengthen policy making in the interim, including the recommendations included in this report.

The above conclusions represent the committee’s view of elements for a future-thinking and horizon-scanning mechanism for the bioeconomy. A structured foresight process making use of horizon scanning would help support policy making around the future of the bioeconomy. Chapter 8 considers the establishment of a government-wide mechanism to monitor and oversee the U.S. bioeconomy. Future thinking and horizon scanning should be a tool at this network’s disposal.

Conclusion 6-5: To be effective, a structured foresight process making use of horizon scanning would need a champion with the resources to sustain such an activity, influence to feed the results into appropriate policy-making processes, and leadership buy-in to ensure that neither the process nor its results would be sidelined.

Foresight processes build on horizon scanning intended to identify issues that could have a policy impact, feeding into assessment and scenario-based processes for exploring policy options. How horizon scanning is integrated into broader foresight activities will depend on the ultimate purpose at hand. The committee’s Statement of Task on horizon

scanning includes both (1) identifying gaps in terms of new technologies, markets, and data sources that could provide insights into the bioeconomy; and (2) identifying and helping to prioritize opportunities and threats with respect to safeguarding the bioeconomy. A structured, flexible, and adaptive foresight process is key to identifying additional strategies that might be needed to safeguard these new technologies and data and assess their implications for innovation and biosecurity. A model for such a foresight process that embraces both tasks can be found in two of the pathways included in the UK Government Office for Science’s Futures Toolkit (see Box 6-1): identifying future research and evidence priorities and identifying and prioritizing future opportunities and threats for action. These pathways could usefully be adapted to take advantage of existing foresight resources and approaches and other tools in use within the U.S. government.

Conclusion 6-6: Foresight processes can be used to identify gaps in new technologies, markets, and data sources in addition to identifying and helping to prioritize opportunities and threats for safeguarding the bioeconomy.

The aim of this process, which would need to be integrated into the specific questions asked of participants, would include identifying “known unknowns” and previously “unknown unknowns.” It would be used to begin to formulate hypotheses about the future of the bioeconomy and to shape future research agendas. It would use desk research, interviews, and workshops to produce an evolving roadmap showing how the issues identified could impact the bioeconomy over time. Such a process would need to involve both subject-matter experts and policy makers responsible for relevant areas (see Annex 6-1 for more detail on exactly what such a process might entail).

Horizon-scanning activities would be fed into driver mapping, which could be used to categorize, but not prioritize, drivers. The results of this activity would then be subjected to SWOT (strengths, weaknesses, opportunities, and threats) analysis. That analysis might usefully identify whether the threat or opportunity will impact the bioeconomy in the short, medium, or long term; the potential outcome or implications for the bioeconomy; whether there are control measures that could be implemented; what actions could be taken directly or indirectly to mitigate threats or seize opportunities; and with whom it will be necessary to work to deliver that action. Likely timeframes and impacts also might usefully be addressed using superforecasters. Possible actions, partners, and control measures might be explored using net assessment and analytic pathway games.

Annex 6-1
Defining Horizon Scanning

In this report, the committee uses the terms “horizon scanning” and “future thinking”/“foresight” as developed by the Organisation for Economic Co-operation and Development (OECD):

• Horizon scanning is “a technique for detecting early signs of potentially important developments through a systematic examination of potential threats and opportunities, with emphasis on new technology and its effects on the issue at hand” (OECD, n.d.a).
• Futures thinking is “a method for informed reflection on the major changes that will occur in the next 10, 20 or more years in all areas of social life … [and] uses a multidisciplinary approach to pierce the veil of received opinion and identify the dynamics that are creating the future. While the future cannot be reliably predicted, one can foresee a range of possible futures and ask which are the most desirable for particular groups and societies. A variety of methods—qualitative, quantitative, normative, and exploratory—help illuminate the possibilities, outline policy choices, and assess the alternatives” (OECD, n.d.b).

Use of these definitions is consistent with their use in other settings. The Food and Agriculture Organization of the United Nations (FAO), for example, notes that horizon scanning “generally refers to methodological approaches that scan or review various data sources, while Foresight generally refers to the wider group of more participatory methods” (FAO, 2013).

There have been numerous other attempts to define horizon scanning (European Commission, 2015; IRM, 2018; OECD, n.d.a; UK Government Cabinet Office, 2013). Common components of these definitions include that the tool

• makes use of a standardized, systematic methodology, including a specific set of criteria in the searching or filtering processes to ensure that the results are relevant to the scan’s stated aim (FAO, 2013; OECD, n.d.a; UK Government Cabinet Office, 2013; UK Government Office for Science, 2013);

• focuses on emerging trends rather than specific events or discoveries—such as the trend toward more efficient genome engineering compared with the specific discovery of clustered regularly interspaced short palindromic repeats (CRISPR) as a means of achieving that trend—especially trends that challenge existing assumptions (OECD, n.d.a; UK Government Cabinet Office, 2013);
• utilizes specified data repositories or other sources of information (OECD, n.d.a; UK Government Cabinet Office, 2013);
• attempts to differentiate among types of signals, whether they be constants, changes and constant changes, or weak (or early) signals, as well as trends and wild cards (OECD, n.d.a; UK Government Cabinet Office, 2013);
• looks further ahead than the standard electoral cycle, often into the medium or longer term (UK Government Cabinet Office, 2013; UK Government Office for Science, 2013); and
• results in conclusions that can be tied to specific actions or otherwise be fed directly into policy-making processes (FAO, 2013; OECD, n.d.a; UK Government Cabinet Office, 2013; UK Government Office for Science, 2013).

HORIZON SCANNING AS A POLICY TOOL

According to the Institute for Risk Management, horizon scanning is used as a tool

• “To deepen the understanding of the driving forces affecting future development of a policy or strategy area;
• To identify gaps in understanding and bring into focus new areas of research required to understand driving forces better;
• To build consensus amongst a range of stakeholders about the issues and how to tackle them;
• To identify and make explicit some of the difficult policy choices and trade-offs that may need to be made in the future;
• To create a new strategy that is resilient because it is adaptable to changing external conditions; and
• To mobilize stakeholders to action” (IRM, 2018).

The European Union (EU) Directorate-General (DG) for Research and Innovation has outlined a series of considerations for developing a horizon-scanning process (European Commission, 2015):

• purpose—from providing independent advice as an input to a policy process through legitimizing existing policy decisions;
• scope—from providing an overview of an uncharacterized field through exploring a predefined field;
• degree of automation—from an automated process through an expert-driven exercise;
• duration—from an on-demand activity through an ongoing process; and
• integration—from being a stand-alone activity through being part of a broader policy-making process.

The EU DG notes that determining the needs of a specific horizon-scanning process for each of these considerations will likely have implications for how focused the results will be. The specific needs of each category will also determine the time and resources required (European Commission, 2015).

The United Kingdom provides an example of horizon scanning in policy making, having integrated horizon scanning into its central policy making through its Cabinet Office. The UK process considers three policy horizons (see Figure Annex 6-1). Horizon 1 relates to impacts that will be felt today and tomorrow, where “trends and events stand out against the background and their impacts are clearly signaled to policy makers.” These trends and events can be addressed by actions currently being taken. Horizon 2 comprises trends whose impact will be seen in the short to medium term and can be fed into strategic thinking. Horizon 3 encompasses those trends that will grow in importance in the longer term, for which some planning may be needed. The UK process frames horizon scanning as a tool that “looks towards the long term (Horizon 2 to 3) but is not focused exclusively on it; many H3 developments are the long-term outcome of a range of factors, some of which are in play already” (UK Government Office for Science, 2017).

GOOD PRACTICE IN HORIZON SCANNING

Factors to be considered when developing a horizon-scanning process include sources of information, criteria and questions used to explore them, and policy impact.

Sources of Information

Information for a horizon scan can come from a wide variety of sources, and needs to be tailored to the area of interest of the individual process. Information sources can be traditional, such as publications,

quantitative and qualitative data, and published expert opinions, but it is equally important to consider unique sources that fall on “the margins of current thinking,” ensuring a holistic perspective (Habegger, 2009). As a result, sources can also be less traditional, such as news outlets, social media, and prepublication servers. In addition, the process may need to take into account insights into lifestyles, people’s sociological expectations, or other indicators of potential impact. It will often benefit from including insights from key stakeholders, such as those provided by professional bodies, industry leaders, customers, or those working in the field in question. It is also possible to apply semiquantitative approaches to rating the utility of different sources (Smith et al., 2010).

Efforts are under way to move from manual compilation of information using experts to more automated models. For example, Singapore established the Risk Assessment and Horizon Scanning Experimentation Center to develop better tools for data analytics, modeling, and perspective sharing (Chong et al., 2007). Efforts have been made as well to adapt advances in agent-based modeling in order to automate some of the analysis of the output from horizon scans (Frank, 2016).

Criteria and Questions Used to Explore Them

When a scan of a short timescale on a specific topic is being prepared, it is important for it to describe the trend or development identified,

explain how it relates to the policy or strategy area being explored, and detail why the trend or development is believed to be important and what thoughts it stimulated. The process can include links back to supporting materials and additional information.

To ensure comparability, some processes suggest that those participating in a horizon scan attempt to frame the issues at a similar level of granularity. For example, very specific developments might have a profound impact in one area but be much less likely to have an impact at the level of a policy development. On the other hand, overgeneralization may offer policy relevance but lack specific ties to trends or developments specific enough to be targeted by policy actions (Wintle et al., 2017). Either when developing a scan on a topic or when reviewing its potential policy impact, a number of specific criteria have been suggested, and specific questions have been proposed for exploring each criterion (see Table Annex 6-1) (Hines et al., 2018).

There are also more quantitative approaches for comparing criteria. For example, an analytic hierarchy process can be used to weight the criteria applied in a horizon-scanning exercise (Mehand et al., 2018; WHO, 2017).

Policy Impact

During the committee’s webinar on horizon scanning, speakers indicated the importance of having a specific sponsor for horizon-scanning and futuring work. A sponsor would need to have the resources to sustain relevant work, the ability to feed the results into relevant policy-making processes, and a high-level interest in the work to ensure that neither the process nor the conclusions of the horizon scan would readily be sidelined. Speakers also discussed the importance of carefully considering how the output from foresight processes might best be used to inform decisions, i.e., how the future can be used to inform today’s decisions. That process would likely involve creating a narrative for the future, including through different storytelling approaches. It is also useful to use backcasting (starting with a desirable future and working backwards to highlight decisions and actions that connect it to the present).⁸

The EU has stressed the importance of people in translating the results of a horizon scan into action. It suggests that while parts of the process might be automated, expert involvement is likely to result in more policy-relevant output. It also stresses the importance of understanding who might take action as a result of the scan, what their drivers and priorities

___________________

⁸ Webinar 3, 2019, at http://nas-sites.org/dels/studies/bioeconomy/webinars.

TABLE ANNEX 6-1 Criteria and Questions to Be Considered When Conducting a Horizon Scan

SOURCE: Adapted from Hines et al., 2018.

are, and a clear plan to engage them (or ensure their buy-in from the start) (European Commission, 2015).

The Institute for Risk Management recommends developing a framework for categorizing separate scans to facilitate comparing and reviewing them. It also stresses the importance of highlighting the potential impact of the events and trends identified, in particular describing potential risks and time to impact, which should help an end user better understand the need to take action and how fast it is necessary to act (IRM, 2018).

In its Futures Toolkit, the United Kingdom further elaborates on the importance of a framework for categorizing scans. It proposes two possible approaches: either structuring them according to different change

drivers, such as political, economic, societal, technological, legislative, or environmental factors; or preferably grouping them by themes that emerge from the scans themselves. The toolkit highlights two different formats for presenting the results of a scan: a longer narrative summary providing an overview, broad implications, and specific policy implications; and a shorter structured summary providing a few simple details of impacts, issues, and implications (UK Government Office for Science, 2017).

CASE STUDIES OF HORIZON SCANNING

Examples of Health-Related Horizon Scans

There have been numerous efforts to use horizon scans to identify and prioritize emerging technology in the health sector. Some examples are published snapshots of a single horizon scan, while others are ongoing monitoring processes, and a few track trends in the use of these tools. Examples include the following:

• A joint project of the governments of Australia and New Zealand assessed the potential impact of emerging technologies on public health systems (HealthPACT, 2011).
• A review focused on how horizon scanning has been used to help determine the suitability for public subsidy of new and emerging medical technologies in the Australian private health care sector (O’Malley and Jordan, 2009).
• The Canadian Agency for Drugs and Technologies in Health conducts a horizon-scanning process to identify and monitor new and emerging health technologies that are likely to have a significant impact on the delivery of health care (CADTH, 2015).
• A 2012 review focused on different horizon-scanning approaches used in the United Kingdom’s health system (Miles and Saritas, 2012).
• A 2016 review of the use of forecasting tools identified emerging medical health technologies. The study identified 15 relevant efforts and noted that almost all relied on expert opinion, and only 2 used more complex processes, such as scenario development (Doos et al., 2016).
• A 1999 review examined how horizon scanning can help the United Kingdom’s National Health Service identify and evaluate new technologies and select the most important ones for further support (Stevens et al., 1999).

• A 2003 joint Danish and UK effort was undertaken to analyze how the Internet is used by horizon-scanning systems to systematically identify new health technologies (Douw et al., 2003).

Examples of Food Safety–Related Horizon Scans

FAO identified several organizations that have conducted or continue to regularly conduct horizon scans for food safety (FAO, 2013):

• Canadian Food Inspection Agency (Canada)—This government organization is responsible for safeguarding food in Canada and performs foresight exercises on a semiregular basis.⁹
• Centre for Environmental Risks and Futures, Cranfield University (United Kingdom)—Founded in January 2011, this academic group conducts regular research into foresight methodologies and has been contracted in the past by the UK government to carry out relevant horizon scans.¹⁰
• Horizon Scanning and Futures Team, Department of Environment, Food and Rural Affairs (United Kingdom)—“A leader in horizon scanning work at a global level, this group provides policy advice, identifies future risks and opportunities, and topic specific workshops.”¹¹
• European Food Safety Authority (EU)—This organization is responsible for a wide range of food safety issues in the European Union and carries out assessments of emerging risks that utilize aspects of foresight methodologies.¹²
• Food Standards Agency (United Kingdom)—This is the government agency in the United Kingdom responsible for food safety and hygiene, and it has been exploring the use of foresight methodologies in the area of food safety.¹³
• Strategic Foresight, Department of Agriculture, Fisheries and Forestry (Australia)—Focused on environmental scanning and foresight techniques to identify future issues, this government organization works with local and international partners, and its work includes food safety.¹⁴

___________________

⁹ See http://www.inspection.gc.ca/about-the-cfia/strategic-priorities/cfia-s-strategicpriorities/eng/1521141282459/1521141282849.

¹⁰ See https://theriskexchange.wordpress.com.

¹¹ See https://webarchive.nationalarchives.gov.uk/20070506093923/ http://horizonscanning.defra.gov.uk.

¹² See http://www.efsa.europa.eu/en/efsajournal/pub/5359.

¹³ See http://www.operational-research.gov.uk/recruitment/departments/fsa.

¹⁴ See http://www.agriculture.gov.au/animal/health/strategy.

Example of Combining Separate Horizon Scans

The United Kingdom’s Futures Toolkit includes case studies of how seven different government agencies and ministries make use of futuring tools. Each case study sets out the purpose of the work, the tools used, resources required, the work’s sponsor, specific outputs, particular successes, and challenges. Five of these agencies—the Environment Agency, the Forestry Commission England, the Health and Safety Executive, Revenues and Customs, and Natural England—make specific mention of the purpose of their horizon-scanning work (UK Government Office for Science, 2017). The purposes cited differ and include using horizon scanning to identify new and emerging issues and trends; improve the evidence base for decision making and risk mitigation; help identify risks and opportunities; integrate externalities into business planning; and inform strategy, provoke discussion, and shape thinking.

LESSONS LEARNED FROM HORIZON SCANNING

In addition to lessons identified during the webinar held by the committee, several key actors, including the National Intelligence Council (NIC, 2017), the U.S. Forest Service (Hines et al., 2018), the UK government (Carney, 2018), and the European Union (European Commission, 2015), have distilled lessons from their past use of horizon scanning.

National Intelligence Council Global Trends Report

Improvements in methodology integrated into the most recent iteration of the Global Trends report produced by the National Intelligence Council include (NIC, 2017)

• involving as many experts as possible from a broad range of countries and with a wide variety of backgrounds;
• exploring regional trends first and then aggregating them to create a global picture;
• avoiding connecting conclusions to specific dates, but rather focusing on timeframes relevant to policy making—near-term (5-year), focused on issues confronting the next U.S. administration, and long-term (20-year), to support U.S. strategic planning;
• placing greater focus on difficult-to-measure social and cultural factors that could influence the future events;
• making increased use of analytic simulations, “employing teams of experts to represent key international actors—to explore the future trajectories for regions of the world, the international order, the security environment, and the global economy”; and

• integrating “the potential for discontinuities in all regions and topic areas, developing an appreciation for the types of discontinuities likely to represent fundamental shifts from the status quo.”

U.S. Forest Service

Also in the United States, efforts to establish a horizon-scanning system in the Forest Service led to a number of key reflections, including the following (Hines et al., 2018):

• Background information versus horizon scanning—In general, as horizon scanning is focused on what might happen in the future, the information used in scans should be new (from within the past few years). Older sources may still be useful as background information but not seen as part of an emerging trend.
• New to me versus new to the world—Some information may appear new but be familiar to those well versed in the field. This observation highlights the importance of including subject-matter experts on the issues being scanned.
• How to handle “coaching” of volunteers—Having those undertaking the scans start from the same place and (to the extent possible) use complementary approaches is important. Regular interactions with those undertaking the scans are also important to reinforce guidance provided to them, as is approaching feedback in a positive, constructive manner (as opposed to criticizing participants).
• Focusing on outside issues—Policy makers and decision makers are often well versed in emerging issues in their own field. Participants in horizon scanning can add particular value by looking at events or trends from outside the core field (in this case the bioeconomy) that could also have an impact.
• Staying connected—Whether or not the trend or event identified comes from the core field, its implications for the core field must be clearly articulated. This can be achieved by specifically tasking those undertaking the scan to explicitly address the implications for the core field.
• Stretching into the future—It is important to encourage those undertaking the scan to think further into the future. One approach is asking them to “tag” the scan to one of the three horizons identified earlier in Figure Annex 6-1.
• Tagging discipline—As the number of scans grows, it becomes more difficult to track their content and how they relate to each

• other and the issue being investigated. It is important to add tags, keywords, or relationship indicators to the scans to facilitate their ongoing use.
• Current issues—As discussed above, those familiar with a field (be they technical experts or policy makers) are often aware of current emerging issues. Frequently, these issues are not well articulated or documented. It will greatly improve the utility of scans and increase the value of engaging generalists or specialists from other fields if an effort is made early in the process to map current emerging issues and provide this information to all those involved in the horizon-scanning process.

UK Government

Based on the use of horizon scanning in the UK government, 10 key rules have been identified. Some of these rules have been discussed in this annex and in the main text of Chapter 6—for example, (1) that horizon scanning is not about predicting the future but about challenging assumptions and increasing options, (2) that there is a lack of common understanding about what horizon scanning is or the terms being used, and (3) that focusing on impact and explicitly exploring the implications of the trends or events identified are important. Other rules bear emphasizing here, such as the importance of (Carney, 2018)

• asking the unasked questions (or attempting to explore the unknown unknowns), as opposed to focusing on something that is already known or a specific desirable outcome;
• having a champion or dedicated client for the process—someone that wants the results and is keen and willing to integrate and act upon the results; and
• involving generalists (or at least participants from outside the commissioning domain) and understanding their value in identifying the unasked questions or implications not seen to date, as well as in presenting the outcome of the work.

European Union

Similarly, the European Union has identified a number of key considerations, including (European Commission, 2015)

• having a clear organizational structure (or institutional support) for horizon scanning, such as arrangements for coordination and brokerage with users;

• developing a specific implementation plan to take advantage of the scanning results, or integrating the scan into a more comprehensive foresight process;
• undertaking both continuous horizon-scanning processes in strategically important areas and stand-alone projects designed to answer explicit questions;
• using expert review to help transform information into actionable knowledge;
• tailoring the approach used and people involved to the scan’s end goal, recognizing that processes for understanding a new policy environment will be different from those for considering the implications of emerging trends and new events;
• involving the end user/client of a horizon scan (such as policy makers) in the planning stages, such as the initial sense-making activities; and
• ensuring that the results of the scan are accessible to the eventual end user, likely necessitating that they be “translated” at a suitable stage.

ADDITIONAL TOOLS FOR FUTURE THINKING

Superforecasting

As discussed briefly in the main text of Chapter 6, in 2010, the Intelligence Advanced Research Projects Agency (IARPA) created a program to explore how crowdsourcing can improve forecasting¹⁵:

Similar programs have subsequently focused on developing “innovative solutions and methods for integrating crowd sourced forecasts and other data into accurate, timely forecasts on worldwide issues.”¹⁶ There have

___________________

¹⁵ See IARPA, Aggregative Contingent Estimation: https://www.iarpa.gov/index.php/research-programs/ace/baa.

¹⁶ See IARPA, Geopolitical Forecasting Challenge: https://www.iarpa.gov/challenges/gfchallenge.html.

also been programs created “to develop and test methods for generating accurate forecasts for significant science and technology (S&T) milestones, by combining the judgments of many experts”¹⁷; and “to develop automated methods that aid in the systematic, continuous, and comprehensive assessment of technical emergence using information found in published scientific, technical, and patent literature.”¹⁸

IARPA tested the tools for aggregating crowdsourced forecasting in a 4-year series of tournaments, where

One successful team subsequently identified a number of key findings (Tetlock et al., 2017):

• “Some methods for extracting wisdom from crowds are better than others. Prediction polls yield a probabilistic forecast by aggregating the predictions of individuals…. In contrast, prediction markets rely on forecasters buying and selling contracts whose ultimate value depends on the outcome of a future event.”
• “The winning algorithm across all tournament years was a log-odds weighted-averaging equation that extremized median probability judgments … as a function of the diversity of the views feeding into the median.”
• “Some forecasters are, surprisingly consistently, better than others.”
• “Learning—and therefore improvement—is possible, even though the world of international politics and economics … is not learning-friendly.”

The last two of these findings form the basis of superforecasting (Tetlock and Gardner, 2015). This process brings together in teams those individuals with a proven track record of being able to make more accurate predictions, supported by specialized tools and algorithms so as to further increase their accuracy.

___________________

¹⁷ IARPA, Forecasting Science & Technology: https://www.iarpa.gov/index.php/research-programs/forest.

¹⁸ IARPA, Foresight and Understanding from Scientific Exposition: https://www.iarpa.gov/index.php/research-programs/fuse.

A thorough assessment of the performance of superforecasters during the tournaments demonstrated that they were significantly more accurate in making predictions than other participants and that “tight restrictions on time and information did not erode the superforecaster advantage.” They were also better able to differentiate between signal and noise and were the fastest learners in the tournament. These studies demonstrated that while certain types of people are more likely to become superforecasters, certain skills and organizational arrangements can increase the ability to make accurate predictions. Thus, “superforecasters are partly discovered and partly created.” Mellers and colleagues (2015) identify “four mutually reinforcing explanations of superforecaster performance: (a) cognitive abilities and styles, (b) task-specific skills, (c) motivation and commitment, and (d) enriched environments.”

The first cohorts of superforecasters were identified during the IARPA forecasting tournaments. Efforts to identify and recruit additional individuals have continued through Good Judgment Open.¹⁹ Since the tournaments, the approach has been developed into a commercial service through Good Judgment, which works with governments, the financial sector, and civil society and nongovernmental organizations, providing forecasting, training services, and tools and techniques.²⁰

UK Government Office for Science’s Futures Toolkit

In 2017, the UK Government Office for Science (GO-Science) published a Futures Toolkit that “policy professionals can use to embed long term strategic thinking in the policy and strategy process.” It is intended to be “practical rather than theoretical and … based on GO-Science’s own experience of running futures work and has been developed in collaboration with other government departments and futures practitioners who use these tools regularly in a wide range of settings” (UK Government Office for Science, 2017). The tools in the kit are structured around four common uses for foresight:

• gathering intelligence about the future,
• exploring the dynamics of change,
• describing what the future may be like, and
• developing and testing policy and strategy.

As the task assigned to this committee was to “develop ideas for horizon scanning mechanisms to identify new technologies, markets, and data

___________________

¹⁹ See https://www.gjopen.com.

²⁰ See https://goodjudgment.com.

sources that have the potential to drive future development of the bioeconomy,” our focus was on the use of foresight tools to gather bioeconomy-related intelligence about the future.

The toolkit describes four tools relevant for gathering intelligence about the future (UK Government Office for Science, 2017):

• Horizon scanning—as described in this chapter.
• 7 Questions—This is “an interview technique for gathering the strategic insights of a range of internal and external stakeholders.” It can be used to identify conflicting or challenging views of the future, extract deep information about underlying concerns in a policy area, and stimulate individuals’ thinking in preparation for a futures workshop. It is a fairly quick process, with each interview taking about an hour to conduct and another hour to write up.
• The issues paper—This paper “presents quotes from the 7 Questions interviews to illustrate the strategic issues and choices around the policy and strategy agenda.” It can be used to capture different perspectives from those captured by the 7 Questions interviews about what success in the future will be like and what needs to be done to achieve it. This is another quick process, taking around 30 minutes to process each of the seven questions per interview.
• Delphi process—This is “a consultation process used to gather opinion from a wide group of subject experts about the future and to prioritize the issues of strategic importance.” It can be used to gather opinion from a group of experts, refine thinking on the future, and highlight the potential trade-offs and choices that policy design will need to address. It is a more time-consuming process that can take several weeks.

The tools in the kit are then combined in different ways to meet different needs, as captured in a series of pathways (UK Government Office for Science, 2017):

• “Pathway 1—exploring underlying issues or causes when scoping or defining a policy area;
• Pathway 2—determining a vision for a new policy area;
• Pathway 3—testing policy options for an existing policy area under time constraints;
• Pathway 4—testing policy options for a new policy area;
• Pathway 5—exploring and communicating the complexity of a situation;

• Pathway 6—identifying futures research and evidence priorities; and
• Pathway 7—identifying and prioritizing future opportunities and threats for action.”

Given the focus of this study and the committee’s Statement of Task, Pathways 6 and 7 are of particular relevance. These pathways use additional tools, including (UK Government Office for Science, 2017) the following:

• Driver mapping is used to “identify drivers shaping the future, identify which drivers are most important for the future of the policy area or strategic endeavor, and distinguish between certain and uncertain outcomes resulting from the action of drivers.” It is another quick tool, usually taking 1–2 hours depending on whether it is accomplished in small groups or as a workshop.
• Roadmapping “shows how a range of inputs—research, trends, policy interventions, for example—will combine over time to shape future development of the policy or strategy area of interest.” It can be used to “build a holistic picture of the different elements in a project and how they combine over time” and “deepen understanding of the connections and relationships between different elements.” This tool need not take a long time, and an initial version can be assembled in about 1.5 hours. It can be revisited and improved throughout the life of a foresight program.
• SWOT analysis examines “Strengths, Weaknesses, Opportunities, and Threats. Strengths and Weaknesses [which] are internal factors that need to be taken account of when developing policy or strategy. Opportunities and Threats are external factors that need to be considered.” The analysis can identify what needs to be done to capture and build on opportunities, what needs to be done to mitigate threats, and internal priorities and challenges. A simple SWOT analysis can be accomplished in 1 hour.

Pathway 6, “identifying futures research and evidence priorities,” begins with horizon scanning but feeds the results into 7 Questions, issues papers, driver mapping, and then roadmapping. Pathway 7, “identifying and prioritizing future opportunities and threats for action,” also starts with horizon scanning but feeds the results into driver mapping and SWOT analysis.

REFERENCES

HealthPACT (Health Policy Advisory Committee on Technology). 2011. Welcome to the Australia and New Zealand Horizon Scanning Network (ANZHSN). http://www.horizonscanning.gov.au (accessed October 15, 2019).

Hines, A., D. N. Bengston, M. J. Dockry, and A. Cowart. 2018. Setting up a horizon scanning system: A U.S. federal agency example. World Futures Review 10(2):136–151. https://www.fs.fed.us/nrs/pubs/jrnl/2018/nrs_2018_hines-a_001.pdf (accessed September 4, 2019).

Horton, A. 1999. A simple guide to successful foresight. Foresight 1(1):5–9.

IRM (Institute for Risk Management). 2018. Horizon scanning: A practitioner’s guide. https://www.theirm.org/media/4047721/Horizon-scanning_final2.pdf (accessed September 4, 2019).

Konecny, A. D. 1988. Net assessment: An examination of the process. Thesis. Naval Postgraduate School, Monterey, California. https://apps.dtic.mil/dtic/tr/fulltext/u2/a205104.pdf (accessed October 15, 2019).

Lentzos, F., and J. Littlewood. 2018. DARPA’s Prepare program: Preparing for what? Bulletin of the Atomic Scientists. https://thebulletin.org/2018/07/darpas-prepare-program-preparing-for-what (accessed September 4, 2019).

Mehand, M. S., P. Millett, F. Al-Shorbaji, C. Roth, M. P. Kieny, and B. Murgue. 2018. World Health Organization methodology to prioritize emerging infectious diseases in need of research and development. Emerging Infectious Diseases 24(9). https://wwwnc.cdc.gov/eid/article/24/9/17-1427_article (accessed September 4, 2019).

Mellers, B., E. Stone, T. Murray, A. Minster, N. Rohrbaugh, M. Bishop, E. Chen, J. Baker, Y. Hou, M. Horowitz, L. Ungar, and P. Tetlock. 2015. Identifying and cultivating superforecasters as a method of improving probabilistic predictions. Perspectives on Psychological Science 10(3):267–281. doi: 10.1177/1745691615577794.

Miles, I., and O. Saritas. 2012. The depth of the horizon: Searching, scanning and widening horizons. Foresight 14(6):530–545. doi:10.1108/14636681211284953.

NIC (National Intelligence Council). 2017. Methodological note. In Global trends: Paradox of progress. https://www.dni.gov/index.php/global-trends/methodological-note (accessed September 4, 2019).

OECD (Organisation for Economic Co-operation and Development). n.d.a. Overview of methodologies, futures thinking in brief, future schooling. https://www.oecd.org/site/schoolingfortomorrowknowledgebase/futuresthinking/overviewofmethodologies.htm (accessed September 4, 2019).

OECD. n.d.b. Futures thinking in brief, future schooling. https://www.oecd.org/site/schoolingfortomorrowknowledgebase/futuresthinking/futuresthinkinginbrief.htm (accessed September 4, 2019).

Office of Technical Intelligence. 2015. Technical assessment: Data enabled technology watch and horizon scanning. https://apps.dtic.mil/dtic/tr/fulltext/u2/1010211.pdf (accessed September 4, 2019).

O’Malley, S. P., and E. Jordan. 2009. Horizon scanning of new and emerging medical technology in Australia: Its relevance to Medical Services Advisory Committee health technology assessments and public funding. International Journal of Technology Assessment in Health Care 25(3):374–382. doi: 10.1017/S0266462309990031.

Popper, R. 2008. Foresight methodology. In The handbook of technology foresight: Concepts and practice, edited by L. Georghio, J. Cassingena Harper, M. Keenan, I. Miles, and R. Popper. Cheltenham, UK: Edward Elgar.

Rowe, E., G. Wright, and J. Derbyshire. 2017. Enhancing horizon scanning by utilizing pre-developed scenarios: Analysis of current practice and specification of a process improvement to aid the identification of important “weak signals.” Technological Forecasting and Social Change 125:224–235. https://www.sciencedirect.com/science/article/pii/S0040162517300707 (accessed September 14, 2019).

Schultz, W. L. 2006. The cultural contradictions of managing change: Using horizon scanning in an evidence-based policy context. Foresight 8(4):3–12.

Smith, J., A. Cook, and C. Packer. 2010. Evaluation criteria to assess the value of identification sources for horizon scanning. International Journal of Technology Assessment in Health Care 26(3):348–353. doi: 10.1017/S026646231000036X.

Stevens, A., R. Milne, R. Lilford, and J. Gabbay. 1999. Keeping pace with new technologies: Systems needed to identify and evaluate them. British Medical Journal 319(7220):1291. doi: 10.1136/bmj.319.7220.1291.

Sutherland, W. J., H. Ilary, A. Llison, R. Osalind, A. Veling, I. P. Bainbridge, L. Bnnun, D. J. Bullock, A. Clements, H. Q. P. Crick, D. W. Gibbons, S. Smith, M. R. W. Rands, P. Rose, J. P. W. Scharlemann, and M. S. Warren. 2012. Enhancing the value of horizon scanning through collaborative review. Oryz 45(3):368–374. https://doi.org/10.1017/S0030605311001724 (accessed September 4, 2019).

Sutherland, W. J., S. Broad, S. H. M Butchart, S. J. Clarke, A. M. Collins, L.V. Dicks, H. Doran, N. Esmail, E. Fleishman, N. Frost, K. J. Gaston, D. W. Gibbons, A. C. Hughes, Z. Jiang, R. Kelman, B. LeAnstey, X. le Roux, F. A. Lickorish, K. A. Monk, D. Mortimer, J. W. Pearce-Higgins, L. S. Peck, N. Pettorelli, J. Pretty, C. L. Seymour, M. D. Spalding, J. Wentworth, and N. Ockendon. 2019. A horizon scan of emerging issues for global conservation in 2019. Trends in Ecology & Evolution 34(1):83–94.

Tetlock, P. E., and D. Gardner. 2015. Superforecasting: The art and science of prediction. New York: Crown.

Tetlock, P. E., B. A. Mellers, and J. P. Scoblic. 2017. Bringing probability judgments into policy debates via forecasting tournaments. Science 355(6324):481–483. doi: 10.1126/science.aal3147.

UK (United Kingdom) Government Cabinet Office. 2013. Review of cross-government horizon scanning. https://www.gov.uk/government/publications/review-of-cross-government-horizon-scanning (accessed September 4, 2019).

UK Government Office for Science. 2013. Horizon Scanning Programme: A new approach for policy making. https://www.gov.uk/government/news/horizon-scanning-programme-a-new-approach-for-policy-making (accessed September 4, 2019).

UK Government Office for Science. 2017. The futures toolkit: Tools for futures thinking and foresight across U.K. government. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/674209/futures-toolkit-edition-1.pdf (accessed September 4, 2019).

Wellcome Trust. 2017. Report from the December 2017 gene drive sponsors and supporters’ forum. https://wellcome.ac.uk/sites/default/files/gene-drive-forum-notes-dec17.pdf (accessed October 15, 2019).

WHO (World Health Organization). 2017. Methodology for prioritizing severe emerging diseases for research and development. https://www.who.int/blueprint/priority-diseases/RDBlueprint-PrioritizationTool.pdf (accessed September 4, 2019).

Wintle, B. C., C. R. Boehm, C. Rhodes, J. C. Molloy, P. Millett, L. Adam, R. Breitling, R. Carlson, R. Casagrande, M. Dando, R. Doubleday, E. Drexler, B. Edwards, T. Ellis, N. G. Evans, R. Hammond, J. Haseloff, L. Kahl, T. Kuiken, B. R. Lichman, C. A. Matthewman, J. A. Napier, S. S. ÓhÉigeartaigh, N. J. Patron, E. Perello, P. Shapira, J. Tait, E. Takano, and W. J. Sutherland 2017. A transatlantic perspective on 20 emerging issues in biological engineering. eLIFE, November 14. https://elifesciences.org/articles/30247 (accessed September 4, 2019).