Appendix B Delphi Survey: Methodology and Results

Introduction

As part of its data gathering effort, the National Research Council (NRC) Committee on Visionary Manufacturing Challenges developed and implemented a survey. Using the Delphi method, international experts in manufacturing were surveyed to obtain a forecast of future manufacturing challenges for the year 2020. The Delphi survey was undertaken during a six month period from February to July 1997.

The Delphi Method

The term "Delphi method" refers to a variety of group communication processes for forecasting or decision making. The basic concept originated in the 1950s at the Rand Corporation as a spinoff of Air Force-sponsored research on the use of expert opinion. The original study involved a series of questionnaires with controlled feedback to determine the opinion of a group of experts on the U.S. industrial systems most likely to be targeted by Soviet strategic planners. At that time, the alternative would have been an extensive and costly data-collection process that included programming and executing computer models that could not be handled by available computers (Linstone and Turoff, 1975). During the past three decades, the Delphi method has been used by corporations, universities, government agencies, and nonprofit organizations for planning, technical and strategic evaluations, and forecasting.

The Delphi method characteristically obtains independent inputs from a group of individuals through an anonymous, iterative survey with controlled feedback after each iteration. Delphi participants may be experts or laypersons depending



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 123
--> Appendix B Delphi Survey: Methodology and Results Introduction As part of its data gathering effort, the National Research Council (NRC) Committee on Visionary Manufacturing Challenges developed and implemented a survey. Using the Delphi method, international experts in manufacturing were surveyed to obtain a forecast of future manufacturing challenges for the year 2020. The Delphi survey was undertaken during a six month period from February to July 1997. The Delphi Method The term "Delphi method" refers to a variety of group communication processes for forecasting or decision making. The basic concept originated in the 1950s at the Rand Corporation as a spinoff of Air Force-sponsored research on the use of expert opinion. The original study involved a series of questionnaires with controlled feedback to determine the opinion of a group of experts on the U.S. industrial systems most likely to be targeted by Soviet strategic planners. At that time, the alternative would have been an extensive and costly data-collection process that included programming and executing computer models that could not be handled by available computers (Linstone and Turoff, 1975). During the past three decades, the Delphi method has been used by corporations, universities, government agencies, and nonprofit organizations for planning, technical and strategic evaluations, and forecasting. The Delphi method characteristically obtains independent inputs from a group of individuals through an anonymous, iterative survey with controlled feedback after each iteration. Delphi participants may be experts or laypersons depending

OCR for page 123
--> on the goals of the survey. In most cases, the first questionnaire poses the problem in broad terms and invites answers and comments. Responses to the first questionnaire are then summarized and used to construct the second questionnaire, which presents the results of the first and gives participants an opportunity to refine their responses, clarify issues, identify areas of agreement or disagreement, and develop priorities. This interactive process can be repeated as many times as appropriate (Ziglio, 1996). The Delphi method is widely considered to be effective in situations where no hard data exist and the primary source of information is well informed, learned opinion (A.T. Kearney, Inc., 1988). Experiments carried out in the late 1960s and early 1970s demonstrated that the Delphi method has distinct advantages over traditional, interactive group processes when the best available information is the judgment of knowledgeable individuals, (Dalkey, 1969; Ziglio, 1996). Survey on Visionary Manufacturing Challenges The NRC Delphi survey on visionary manufacturing challenges was designed to forecast manufacturing challenges for 2020 and to identify enabling technologies for research and development. The work of implementing the survey included designing and testing the first questionnaire; identifying, selecting, and contacting potential participants; distributing the first questionnaire; collecting and analyzing responses from the first questionnaire; designing the second questionnaire; distributing the second questionnaire; and collecting and analyzing the responses from the second questionnaire. Designing and Testing the First Questionnaire In February 1996, a Workshop on Methods for Predicting Manufacturing Challenges was held at the Beckman Center in Irvine, California. The workshop was conducted by the BMAED as a means of determining the best methods of gathering data for the study on visionary manufacturing challenges. At the workshop, participants from the United States, Europe, and Japan took part in a roundtable discussion and filled out a trial questionnaire. The results of this workshop and recent questionnaires on manufacturing issues were used to prepare the first questionnaire of the BMAED Delphi survey. The purpose of the first questionnaire was to elicit information on participants' visions of (1) the competitive environment in 2020, (2) characteristics of manufacturing enterprises in 2020, (3) the challenges that would be faced by manufacturing enterprises, and (4) the technological developments that would enable manufacturers to meet the challenges. The committee decided to use questions calling for open-ended responses, as opposed to providing respondents with a selection of answers to choose from. This was done to encourage creative thinking on the part of respondents and to ensure that the scope of survey responses

OCR for page 123
--> was not limited to the committee's knowledge and thinking. A copy of the first questionnaire is provided in Appendix C. The first questionnaire was pilot tested on seven individuals identified by committee members as having suitable manufacturing experience, vision, and familiarity with the project, as well as the ability to complete and return the pilot questionnaires quickly. The results of the pilot questionnaires were incorporated into the instructions and questions in the first questionnaire. Selection and Composition of Survey Participants Potential survey participants were identified using numerous mechanisms. Members of the Committee on Visionary Manufacturing Challenges identified both potential participants and individuals who could suggest potential participants. Members of BMAED and members of Section 8 of the National Academy of Engineering (Industrial, Manufacturing, and Operational Engineering) were contacted and asked to participate. Recommendations were also requested from national and international manufacturing organizations, including the Agility Forum, ASM International, the Consortium for Advanced Manufacturing International (CAM-I), the Coalition for Intelligent Manufacturing (CIMS), the Council on Competitiveness, the Fraunhofer Society of Germany, the Industrial Research Institute, the Institute of Electrical and Electronics Engineers (IEEE), the International Institute for Production Engineering Research (CIRP), Intelligent Manufacturing Systems (IMS), the National Association of Manufacturers (NAM), the National Center for Advanced Technologies (NCAT), the National Center for Manufacturing Science (NCMS), Next Generation Manufacturing Systems (NGM), the Society of Automotive Engineers (SAE), and the Society of Manufacturing Engineers (SME). Recommendations were also requested from the National Science Foundation (NSF). Table B-1 shows the number of participants from each source. The criteria for selecting participants included manufacturing expertise and evidence of visionary thinking. Because of time constraints and to facilitate the analysis, the survey was conducted by facsimile and email. The list of potential participants was, therefore, narrowed to those who had either a working facsimile number or a working email address. The committee believed that the survey should include participants from both industry and academia as well as U.S. and international experts in manufacturing. Special efforts were made to contact a large number of international and industry participants. As shown in Table B-2, the largest representation was from U.S. industry, followed by international and U.S. academia. The international academics were primarily located in Europe. The percentage of respondents from international industry was less than 10 percent. This was probably attributable to the composition of the original lists, which were focused on U.S. industry and international academics, and the difficulty in identifying representatives of international industry.

OCR for page 123
--> TABLE B-1 Number of Potential Survey Participants Identified and Contacted and Number Who Responded to Questionnaires 1 and 2 Organization or Method of Identification Number of Potential Participants Identifieda Total Number Successfully Contacted for Questionnaire 1b Total Number of Respondents to Questionnaire 1 Percentage that Responded to Questionnaire 1 Number of Respondents to Questionnaire 2 Percentage that Responded to Questionnaire 2 Percentage of Questionnaire 1 Respondents that Responded to Questionnaire 2 ASMc 28 28 12 43% 10 36% 83% BMAED 18 18 7 39% 7 39% 100% CIRP 74 44 22 50% 20 45% 91% IMSd 56 54 15 28% 12 22% 80% NAEe 80 58 9 16% 9 16% 100% SMEf 468 149 43 29% 35 23% 81% Individual Referralsg 258 203 73 36% 62 31% 85% Total 982 563 181 32% 155 28% 86% a If an individual appeared on more than one list, he or she was assigned to the list where the name first appeared. b Some potential participants were not contacted because of incorrect or insufficient contact information. c Members of ASM International were recommended by the president of that organization. d Individuals in the leadership of Intelligent Manufacturing Systems (IMS) were selected. e Members of the National Academy of Engineering's (NAE) Section 8 (Industrial, Manufacturing, and Operational Engineering) was contacted. f Fellows of the Society of Manufacturing Engineers (SME) and members of the SME Boards of Advisors were contacted. g Individual referrals came from committee members, the Agility Forum, the Coalition for Intelligent Manufacturing Systems (CIMS), the Consortium for Advanced Manufacturing - International (CAM-I), the Fraunhofer Society, the National Science Foundation (NSF), the National Association of Manufacturers (NAM), Next Generation Manufacturing Systems (NGM), and the Society of Automotive Engineers (SAE).

OCR for page 123
--> TABLE B-2 Distribution of Respondents to Questionnaire 1 by Country and Work Affiliation   Respondents   Industry   Academia   Othera     Number Percent Number Percent Number Percent United States 63 35% 42 23% 4 2% International 16 9% 51 28% 5 3% Africa and the Middle Eastb 0 — 4 2% 0 — Asiac 8 4% 13 7% 0 — Australia 1 — 2 1% 1 — Europed 3 2% 29 16% 0 — North America (non-U.S.)e 2 1% 3 2% 3 2% South Americaf 2 1% 0 — 1 — a This category includes government agencies and trade organizations. b Countries represented were Israel and South Africa. c Countries represented were China, Japan, and Singapore. d Countries represented were Belgium, Denmark, France, Italy, Germany, Norway, Romania, Sweden, Switzerland, and the United Kingdom. e Countries represented were Canada and Mexico. f Country represented was Chile. The committee attempted to elicit a significant contribution from China and South America, two economies that are likely to become increasingly important in manufacturing as 2020 approaches. Efforts were made to identify individuals from China, Brazil, and Chile. Unfortunately, only a few potential participants were identified, and the response rate from them was low. Reasons included the difficulty of contacting these individuals via facsimile or email, problems with language (the questionnaires were not translated), and the cost of responding via facsimile or email. Implementation and Analysis of the Questionnaire 1 A letter describing the project was sent to potential participants prior to the first questionnaire. The purpose of the letter was to familiarize potential participants with the goals of the project and to send a personal request for cooperation from the committee chair. Approximately one week later, the first questionnaires were sent out. The first questionnaire was sent out in batches via facsimile and email between February 15, 1997, and March 17, 1997, and participants were given approximately two weeks to respond. If time permitted, busy facsimile lines were retried and returned emails were resent. One reminder notice was sent to those who did not reply by the original deadline. Responses were received between February 21, 1997, and May 4, 1997. The final number of respondents, including

OCR for page 123
--> the seven pilot respondents, was 181, or 32 percent of the individuals who were contacted (see Table B-1). The responses to the first questionnaire varied in length and detail because of the open-ended nature of the questions. The technique of ''open-coding," developed by Glaser and Strauss (1967), was used to analyze the responses. Using this technique, survey responses were read and reread, and codes, or categories, were inferred from them. Text sections containing similar phrases were grouped according to these codes. The idea behind the method is that the codes are not simply deduced from the analyst's ideas but are inferred from the survey responses. Because of the time-intensive nature of this technique, the committee selected a consultant, Dr. Brian Borys of the School of Public Administration at the University of Southern California, to undertake the analysis. Several measures were taken to ensure that Dr. Borys' coding could be replicated and that the codes where consistent with the study objectives. First, Dr. Borys held preliminary discussions with committee members regarding the nature of the survey, the characteristics of the respondents, and the questions that needed to be answered. Second, Dr. Borys and several committee members separately coded survey responses drawn at random and compared their results. Consistency among the coders was sufficient to conclude that other coders could generate similar interpretations and that Dr. Borys' interpretive scheme would provide sufficient information. As a final check, the committee reviewed Dr. Borys' results after he had coded approximately half of the surveys and before he proceeded to code the rest. When the coding was complete, the committee used the codes, or categories, to distill a list of manufacturing challenges and enabling technologies for 2020 that represented the ideas of the respondents to the first questionnaire. This list was then incorporated into the second questionnaire. Design and Implementation of the Second Questionnaire The Delphi method is an interactive process, i.e., during the process, participants receive feedback on the responses of the group as a whole. In the BMAED Delphi survey, the second questionnaire was used to provide participants with feedback on the results of the first questionnaire. The lists of manufacturing challenges and enabling technologies generated by the first questionnaire were used to construct the first two questions of the second questionnaire, which asked respondents to indicate the challenges and technologies they considered most important. Two additional questions were added asking respondents to list research topics based on their prioritized enabling technologies and the manufacturing challenges that would be addressed by these technologies. A copy of the second questionnaire is attached in Appendix C. The second questionnaire was distributed via facsimile and email between May 16, 1997, and June 30, 1997, to respondents to the first questionnaire. The

OCR for page 123
--> second questionnaire was also distributed to several individuals who had been unable to complete the first questionnaire but had expressed an interest in participating in the second. This questionnaire was also distributed to potential participants from China and South America who had not responded to the first questionnaire in an effort to increase representation from these two regions. Participants were asked to return the questionnaire within two weeks, and two reminders were sent. Responses were received between May 19, 1997, and July 14, 1997. Of the original 181 respondents, 155 (86 percent) returned the second questionnaire. In addition, nine individuals filled out the second questionnaire only. Results from the Second Questionnaire The responses from the second questionnaire were collated to determine the manufacturing challenges and enabling technologies that the respondents considered most important. These results are shown in Tables B-3 and B-4. Table B-5 shows how the respondents correlated enabling technologies and manufacturing challenges. TABLE B-3 Manufacturing Challenges Prioritized by International Experts in Manufacturing Identifier Manufacturing Challenge Votes Rank a Enhancement of workforce performance and satisfaction to address rapidly changing and complex operational requirements and diverse culture-based issues 86 2 b Constant, concurrent development of innovative products, processes, and systems to meet shorter product life cycles, enhance value added, and advance manufacturing capabilities 92 1 c Ability to develop and execute complex and dynamic alliances and collaborations rapidly 52 5 d Response to severe constraints on environmental impact and the increasing scarcity of materials and energy 66 3 e Achievement of the speed and flexibility for cost-effective fulfillment of customer demands for instant satisfaction with customized products 45 6 f Adoption of rapidly developing technologies to increase and/or adapt the core strength of the enterprise to the marketplace 40 7 g Development of an effective global infrastructure to support optimal-scale manufacturing configurations 36 8 h Effective conversion of information to useful knowledge in an environment where the volume of available information is increasing rapidly 64 4

OCR for page 123
--> TABLE B-4 Enabling Technologies Prioritized by International Experts in Manufacturing Identifier Enabling Technologies Votesa Rankb A Adaptable and reconfigurable manufacturing processes and systems (e.g., intelligent, mass customization; rapid creation of new production facilities; ability to accommodate a wide range of product characteristics) 86 1 B Systems model for all manufacturing operations (e.g., real-time synthesis of planning; market demand; product development; distribution; social systems; wealth creation into manufacturing system planning; effective modeling of supply chains) 60 2 C Micro- and nanotechnology for fabrication processes (e.g., atom-by-atom fabrication of assemblies; development of microscale machines) 32 9 D Processes to customize totally new materials with order of magnitude property improvements designed on the atomic scale (e.g., an order of magnitude improvement in strength; defect-free materials; smart materials that can change properties in service in response to changing conditions; materials designed to be reprocessed or reconstructed) 45 7 E Direct machine/user interfaces that enhance human performance and promote intelligent input (e.g., skill-leveraging, human commands transmitted directly to machine; human access to data via "bionic ears") 49 5 F Net-shape, programmable, flexible forming processes that require no hard tooling (e.g., pulsed power autoshaping; forming finished assemblies from the melt) 30 14 G Design methods and manufacturing processes for products that can easily be reconfigured with software or hardware (e.g., products that are easily upgradable in the field for long life; products that can be customized by the customer) 51 4 H Desk-top manufacturing processes (e.g., manufacturing in the home by customer; neighborhood manufacturing service centers; highly distributed manufacturing capacity according to market location; portable manufacturing) 20 18 I Biotechnology processes for manufacturing (e.g., use of biological structures in engineering design; fabrication of parts and assemblies with biological processes; "designer" proteins, enzymes, and tissues; biocatalysts; bioassembly of new foods; biodevices for computer memories) 33 8 J Scientific bases for manufacturing processes (e.g., rapid development of models for simulation) 32 9 K Application of chaos theory to manufacturing (e.g., software that captures emergent behavior; developing basic rules of behavior in manufacturing systems; embedded intelligence software; negotiating and bargaining algorithms) 14 23

OCR for page 123
--> Identifier Enabling Technologies Votesa Rankb L Waste-free manufacturing (e.g., processes designed with no by-products in manufacture, integrated multiple product lines that consume by-products of one line in another) 52 3 M New transportation concepts for rapid movement of materials and products (e.g., friction reduction; antigravity; superfast conveyance) 9 24 N Synthesis and architecture technologies for converting information into desired knowledge (e.g., human memory relational structures; capturing, synthesizing, relating, integrating, and systematizing new knowledge into applications-oriented uses) 47 6 O Design methodologies that process a broader range (by orders of magnitude) of product requirements (e.g., include life-cycle design; producibility; societal requirements; workforce needs) 32 9 P Unified methods of communication and protocols for the exchange of manufacturing enterprise information 31 13 Q Processes for rapid and cost-effective development, transfer, and utilization of technology (e.g., innovation processes; new paradigms for technology development; analysis and synthesis of new technologies) 29 15 R Methodology for quantum jumps in product and process reliability (e.g., variability reduction; new methods for robust design) 14 22 S Low energy consumption processes (e.g., low-inertia machines; catalyst; alternate energy sources; high energy-density batteries) 29 15 T New sensor technology for precision process control (real-time sensors for machine self-calibration; self-verification; self-correction; self-improvement) 32 9 U 360-degree collaboration software (e.g., translate neural knowledge base to language that is personalized to different thinking styles; enable workforce participation in technology design and development; interactive visualization) 16 21 V Low gravity, high vacuum manufacturing (e.g., practical manufacturing in space; earth-bound manufacturing facilities with space environment) 7 25 W New educational methods (e.g., in-home facilities; smart and knowledge pills) 21 17 X New concept manufacturing processes (e.g., ion beam, three dimensional chemical etching) 18 20 Y New software design methods (e.g., methods that are robust, seamless, adaptive, inter-operable, and highly reliable) 19 19 a Respondents could select more than one enabling technology to prioritize. b In case of a tie, items were given the same rank.

OCR for page 123
--> TABLE B-5 Enabling Technologies for Meeting Manufacturing Challenges (Number of Respondents)   Enabling Technologies Challenge A B C D E F G H I J K L M N O P Q R S T U V W X Y a 27 20 2 4 31 0 13 10 1 4 6 8 1 19 17 12 8 3 0 7 7 0 10 2 7 b 51 34 12 21 12 17 26 10 11 15 4 8 0 15 20 12 13 6 3 17 6 2 3 10 14 c 20 22 1 0 6 0 9 5 3 5 6 37 0 14 8 13 5 1 2 0 11 0 3 1 6 d 14 11 8 21 2 7 12 6 18 11 3 3 0 8 11 4 8 1 17 5 4 2 2 3 3 e 50 26 2 14 11 13 21 8 7 9 4 5 1 14 18 13 10 3 2 14 8 0 2 6 9 f 30 16 4 6 4 11 15 2 7 8 3 3 2 10 12 6 15 1 1 8 6 1 3 5 9 g 11 20 1 1 4 0 7 9 1 4 2 6 1 10 7 15 4 1 1 0 7 1 1 0 4 h 16 16 4 2 15 3 12 3 3 6 10 9 1 29 13 19 10 2 3 6 8 0 7 0 11 Total 219 165 34 69 85 51 115 53 51 62 38 79 6 119 106 94 73 18 29 57 57 6 31 27 63 Rank 1 2 9 10 7 16 4 15 16 12 18 8 24 3 5 6 9 23 21 13 13 24 20 22 11 Number 8 8 8 7 8 6 8 8 8 8 8 8 5 8 8 8 8 8 7 6 8 5 8 6 8

OCR for page 123
--> References A.T. Kearney, Inc. 1988. Countdown to the Future: The Manufacturing Engineer in the 21st Century. Dearborn, Mich.: Society of Manufacturing Engineers. Dalkey, N.C. 1969. The Delphi Method: An Experimental Study of Group Opinion. Memorandum RM-5888 PR. Santa Monica, Calif.: Rand Corporation. Glaser, B.G., and A.L. Strauss. 1967. The Discovery of Grounded Theory. Chicago, Ill.: Aldine. Linstone, H.A., and M. Turoff (eds). 1975. The Delphi Method: Techniques and Applications. Reading, Mass.: Addison-Wesley. Ziglio, E. 1996. The Delphi Method and its Contribution to Decision-Making. Pp. 3–33 in Gazing into the Oracle: The Delphi Method and Its Application to Social Policy and Public Health. M. Adler and E. Ziglio, eds. London: Jessica Kingsley Publishers.