Engage Both Crowds and Experts

Experts are typically better than novices at judging the importance of new signals in an existing forecasting system (Enis, 1995). With the currently available platforms (X2, Techcast, and Deltascan), experts generally provide high-signal and low-noise forecasts. However, academic research (Önkal et al., 2003) suggests that experts are not necessarily better at making forecasts than a crowd. Experts may not catch the full range of alternative solutions from adjacent fields outside their areas of expertise or from the reapplication of technologies developed to solve a different problem. Paradoxically, the narrowness of the knowledge specificity required to achieve expert status can invalidate forecasts generated by experts alone (Johnston, 2003). Thus, it is the committee’s belief that blending input from experts and crowds will lead to better forecasts of disruptive technologies.

The goal of public participation, or crowd sourcing, in a forecasting system is to cast a wide net that gathers a multitude of forecasts, signals, and opinions. This is especially important as technology innovation becomes more diverse and geographically diffuse in its approaches and as regional variations of technology applications flourish. Collaboration technologies, especially those that leverage the power of the Internet, can be used to discover expertise in unexpected places.1 Prediction markets, alternate reality games (ARGs), and relevant online communities are disseminating crowd-sourced methods.

Managing Noise in an Open System

Increasing the diversity of participants will increase the richness of a forecast. Nevertheless, open and public forecasting systems also present challenges to their operators. One of the challenges is the noise and distractions generated by such systems.

There are many different strategies for reducing the noise in crowd-based systems. Some systems limit participation to prescreened invitees. Screening is especially useful if a forecast seeks the opinions of a specific audience based on topic, region, or demographics (i.e., young European postdoctoral fellows studying quantum computing). Another approach is a completely open and public site, with fillers to select those with appropriate reputation, expertise, and credentials. Crowd-sourcing sites can use moderators who are themselves experts to monitor, moderate, and augment the forecasts and discussion. These moderators can be either internal staff members or volunteers from the community of users discovered through the Web.

Incentives for Contribution

It is important that a persistent forecasting system be not only open but also effective and relevant. For a system to generate enough signals and forecasts to be of value and to have adequate global representation, the operators must have a large number of diverse and active participants to cover the range of topics. This suggests that it should have adequate incentives (both financial and nonfinancial) to secure the ongoing participation of a diverse user base, to access technologically and socioeconomically impoverished contributors, and to persuade owners of proprietary data to donate (or sell) their data.

Spann and Skierra, as well as Servan-Schrieber and colleagues, suggested that the most effective incentives might be monetary or nonmonetary, depending on circumstances (Spann and Skiera, 2003; Servan-Schreiber et al., 2004). Incentives could utilize elements of gaming (competition), reputation, and financial rewards. Attention must be paid to the cultural appropriateness of the incentives used to secure reliable and valid data. Much of the world’s population resides in collectivistic and hierarchical societies, where information is much more likely to be shared with one’s own group2 than with strangers (Triandis, 1995). An in-group is made up of people sharing similar interests and attitudes, producing feelings of solidarity, community, and exclusivity.3 An out-group is made


Available at http://itstrategyblog.com/whats-better-crowd-sourcing-or-expert-sourcing/. Last accessed May 6, 2009.


In sociology, an “in-group” is a social group for which an individual feels loyalty and respect, usually because he or she is a member , while an “out-group” is defined as a social group for which an individual feels contempt, opposition, or competition. Available at http://en.wikipedia.org/wiki/Ingroup and http://en.wikipedia.org/wiki/Outgroup_(sociology). Accessed on April 14, 2009.


Available at http://dictionary.reference.com/browse/ingroup. Last accessed May 6, 2009.

The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement