Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
3 Research Methodology and Management APPLIED RESEARCH Applied research usually is thought of in terms of the direct transfer of knowledge to application in a particular industry. When there is a clear objective to be met as well as a framework of knowledge that can be brought to bear on the problem, then it seems reasonable to sponsor specific studies that will produce tangible results ready for application to nuclear power plants. However, if this is the only kind of applied research sponsored by the nuclear Industry, then the industry is hostage to the maturity of basic research developments in a particular scientific field. Partial knowledge and approximate models (by the standards of basic research) can have important implications for incremental improvements on significant problems in the applied world. This means that one kind of applied research is the collection and synthesis of existing research results applicable to the nuclear power plant context. This requires people or teams of people that have expertise or experience that spans disciplines. Although at the time of Three Mile Island few behavioral scientists were familiar with the world of nuclear power, this is no longer a critical bottleneck. However, one should note that actually doing this type of review and synthesis is difficult. Too often the result is a dilution of the results of basic research for the noninitiated when what is desired is a distillation of what is really essential and relevant to the basic behavioral science questions that ultimately motivated the research. 30
31 The nuclear industry should also seek to influence research conducted in other institutions so that the research base available to solve the industry's problems will expand. This means that the NRC and the nuclear industry should attempt to influence the research agendas of organizations that perform research (e.g., universities, institutes, laboratories, research companies, and re- search organizations within industrial firms) in the direction of NRC and industry goals and needs. One way to do this is to provide access to experienced nuclear personnel and facilities for such researchers. The types of applied research that should be sponsored or con- ducted by nuclear industry organizations fall into two categories: (1) the utilization of available research and (2) nuclear-specific re- search activities. Activities in the utilization of available research category begins with the decision of what behavioral science is- sues are relevant to nuclear power plants. Research can then be sponsored to identify, collect, and synthesize knowledge. Since the research base changes over time, this category includes tracking research over time. In some cases the existing knowledge base is impoverished; as a result, there is little information to synthesize and apply. In that case what is needed is to encourage and stim- ulate growth on the behavioral science issues relevant to nuclear power plants. This does not necessarily mean sponsoring nuclear- specific research, but fostering through various means research on the sociobehavioral issues that are seen to have relevance to nu- clear power plants. For example, the research base on measuring the quality of computer-based displays is thin. While some interim results are available, the industry needs to encourage more work to identify the elements of an effective display so that results can be transferred to the nuclear industry. The focus of nuclear-specific research activities is the applica- tion of behavioral science and research methods knowledge to the nuclear power plant. There are two parts to this activity. First, there is the need to sponsor, encourage, and stimulate informed innovation in developments intended directly for the nuclear in- dustry. The words encourage and stimulate are used in addition to sponsor because NRC policies can have strong effects on the willingness of industrial organizations to develop new systems, techniques, and on-line or off-line human performance aids. Informed innovation is an open-ended creative process to con- vert the research base into systems and techniques that have an
32 impact. In today's era of computer technology, many develop meets are possible, and it is difficult to decide a priori which will be most effective. For example, behavioral science research can re- veal the potential benefits of exploratory training systems. There may be many different specific systems which could be developed to realize that potential. The NRC and the nuclear industry too "ether need to create an environment that supports and stimulates this process of informed innovation. Second, feedback is needed on the effects of changes to help filter and focus Innovation. Feedback and evaluation complement innovation. In many areas, changes have been introduced into the nuclear power plant as a sociotechnical system, but the effects of those changes have not been assessed and are not yet understood. RECOMMENDED RESEARCH APPROACHES There are many important considerations in managing re- search: how to set the research agenda, how to achieve higher quality, how to achieve more useful research results, and how to achieve maximum leverage from available funds. Peer review of proposals and of draft reports by behavioral scientists with exper- tise in the broad range of disciplines required for an integrated approach of the human-technical system is needed to ensure the quality of sponsored research.* To be effective, a research program must operate coherently for an extended~period rather than change in response to each new; immediate external demand. Since elective research ~ cumula- tive, continuity is as unportant as level of expenditure. Timely research is always assisted by improvement in the quality of re- search archives. At our request the panel was supplied with a complete list of NRC human factors reports. The amount of work and tune required to obtain this list was substantial and reflected a poor cataloging system. If the NRC cannot rapidly and effectively retrieve its own research, how can it be used either for regulation or by industry? Attention should be given to compiling an annual review of relevant research. This review would include not only * We are aware of at least one case where this has occurred success- fully: the NRC-sponsored research on cognitive modeling (NUREG/CR-4862, 1987a), for which a technical review by a distinguished group of scientists was held.
33 NRC work but also work related to regulatory human factors in general. The current NRC report on the status of its programs is primarily for internal administrative use and does not summarize achievements in a way that facilitates successful transfer of results to industry. The task of managing, directing, and conducting behavioral and social science research in the context of the nuclear industry is a challenging one, and our picture of the intricacies of a systems approach supports this claim. Human factors and organizational research requires formal training and sustained experience. It must be conducted by qualified people and overseen by qualified monitors. If human factors research is to be maintained at a level of endeavor and with the quality necessary to cope with the problems encountered, leadership at the branch head level or above must be provided by someone with the necessary experience and qualifications. In this connection, we believe that human factors and social science research ~ not to be managed as a part of human reliability research; rather, human reliability research is one important portion of a broader human factors and social science research program. To date much activity and research related to human factors in the nuclear industry has for the most part been one-shot attempts to generate a final answer to the particular human-related issue in question. The attitude appears to have been one of getting these human issues resolved once and for all. While this research has sensitized many to the importance of human-related issues, it is now time for a serious and continuing commitment to research on the nuclear power plant as a total sociotechnical system. Barriers to sociotechnical research applicable to the nuclear power plant world have as much to do with getting good re- searchers to address behavioral science problems with relevance to nuclear power plants as with having adequate funds for research. Part of taking the interaction between the human and technical systems seriously in nuclear power plant safety means that this access to realistic settings, to facilities such as simulators, and to people such as experienced operators must be greatly enhanced. Research money alone will not provide the needed knowledge base without good access to the nuclear power plant world for be- havioral science experts. For example, actual incidents are an important source of data to calibrate analytic models of human
34 performance, yet to our knowledge no one with expertise in hu- man performance and error is part of either the NRC's or INPO's incident investigation teams. Although difficult, it is important to collect data on human performance with new interfaces and decision aids. While research activities should not conflict with the operational needs of running plants or meeting training requirements, one can bring together for study experienced personnel and examples of new interfaces or decision aids. Ways are needed to allow research and operational needs to coexist and to assist one another. If mechanisms to enhance researchers' access to facilities and people are put in place, the nuclear power plant will become an application world that offers exciting research possibilities from a scientific perspective that can at the same time produce results that will contribute to enhanced safety. Several important general problems arise when conducting research on complex human-machine systems. One is generaliz- ability. In order for the results of a particular research project to be generalizable to an industry population (of nuclear operators, maintenance personnel, or managers), certain statistical criteria must be set. A second problem is a sufficiently large number of trials, people, and treatment conditions are needed to make the statistical power of the results adequate and meaningful. Because of the small population of experienced nuclear power plant person- ne] and their demanding schedules, it is frequently difficult to find a sufficient number of participants for experimental studies. In the human factors area by far the greater number of research reports published by the NRC are not experimental studies but rather reports on methods (such as task analysis), models (such as simu- lations or conceptual models of task allocation), or surveys. Very few are true experiments or well designed observational studies. How can facilities be provided for the use of researchers to investigate important issues? Access for researchers to sites, simu- lators, and personnel in cooperation with the utilities and industry is critical. Pilot studies can often be conducted using part-task simulators or in laboratories, but research that is adequate to convince the utilities and the public and to bear the weight of regulation must have industrial validation. A case could also be made for a national research facility for the study of the human factors of complex systems. A generic or
35 reconfigurable simulator could be instaDed at a laboratory ded- icated to high-technology human factors, in a national research laboratory or university. We understand that steps have been taken by the NRC to acquire such a facility in the form of three simulators. In the meantime, efforts should be made to arrange for utilities and vendors to provide facilities. Efforts should also continue to be made to foster international cooperation as a form of cost-effective research. To summarize these points: the NRC and the nuclear indus- try should try to stimulate research activity in relevant areas of behavioral science, to use outside experts to track and synthesize this research with an eye to opportunities to apply the results to nuclear power plants, to use outside experts to maintain a coherent strategy for research needs on human-related issues, to improve the technical review of programs, to provide researchers access to nuclear power plant personnel and facilities, to sponsor the best people in the relevant behavioral areas, and to emphasize continuity and long-term progress in important areas. SOURCES AND USE OF EXISTING KNOWLEDGE Two major sources of knowledge contain information that is applicable to the problems of improving the human side of nuclear safety. The first source is the published and unpublished research literature from the areas of human factors, behavioral science, organizational theory, management science and computer science and engineering. The second source of knowledge is derivable from the ongoing programs and expertise of those institutions who sponsor or conduct research and information gathering programs related to human performance and nuclear safety. This section reviews these two important knowledge sources, examines current barriers to their use, and suggests actions that, if taken, would reduce these barriers. The Existing Literature Within the past fifty years a large literature has accumulated in the fields of human factors and related behavioral and social science areas. Some of this literature is in the form of published journal articles, theoreticalmonographs, texts, handbooks, design guides, and state of the art summaries. Even though much of this
36 work has been done on other types of systems and organizations than nuclear, it nevertheless constitutes a valuable knowledge base that should be exploited by those concerned with the adm~nistra- tion, planning, and conduct of nuclear safety research. For exam- ple, recent state of the art summaries in the human factors area by Doff, Kaufman, and Thomas (1986), Boff and Lincoln (1987), and Salvendy (1987) contain fundamental data and principles related to human performance In systems contexts that are applicable to nuclear concerns. While it might be assumed that the published literature would be easy to identify and access, it comes from many fields and speciality areas and therefore is covered by many differ- ent indexing, abstracting, and bibliographic services and housed in many different libraries. Unless a potential user is well-versed in information and library science, obtaining information from these diverse sources can be a major problem. Another important source of information exists in the form of technical reports which are often not available from private or federal bibliographic search services and libraries. For exam- ple, many relevant reports published by military laboratories and their contractors; by the NRC, the national laboratories and their contractors; by other organizations such as the Electric Power Research Institute (EPRI); and by the Institute for Electric and Electronic Engineers (IEEE) on human factors standards are not systematically covered by current abstracting and indexing ser- vices or readily available from most libraries. Even less available is work on performance indicators developed by the Institute for Nuclear Power Operations (INPO) which is seldom made public to researchers. The inability of a typical user to quickly and easily locate, evaluate, acquire, and apply information from these published and technical report sources represents a barrier to the use of knowledge that already exists and wastes resources unnecessarily when it is duplicated. Those responsible for the administration and conduct of the human factors research of the type presented in the panel's agenda should be able to access a single bibliographic data base that con- tains abstracts of all of the published and technical report litera- ture applicable to their needs and interests. Such an information search and retrieval system does not now exist and should be developed. Those who administer and conduct research and those who
37 use research results for regulatory analysis and decision making also need access to a periodically updated review of research that is applicable to the problems of improving the human side of nuclear safety. Such a state of the science and art review would distill, summarize, and synthesize research findings in a variety of applicable fields; identify where gaps in knowledge exist; and point to promising application areas. It would provide an objective rather than an intuitive basis for informing regulation. While the development of such a review would be a significant undertaking, involving expertise from a variety of disciplines, its benefits would outweigh its costs. The development of the knowledge access and review mech- anisms suggested here, if jointly undertaken by the NRC, DOE, EPRI, and INPO, represents an ideal opportunity for these insti- tutions to cooperate with one another in an undertaking which does not threaten the integrity of any participant. Ongoing Programs Another major source of applicable information can be derived from a knowledge of the ongoing research programs and expertise of the institutions concerned with the human side of nuclear power safety. We briefly describe these programs below and suggest actions that could be taken to enhance mutual cooperation and information exchange among them. Electric Power Research Institute (EPRI) The long-standing human factors research program of EPRI has produced a series of high quality research reports and research products that are used widely throughout the industry. One rea- son for the success of this program has been EPRI's emphasis on developing mechanisms that ensure that the results of its re- search are transferrable to the industry that EPRI supports. An example of a widely used EPRI report is the Guide to Techni- cal Reports Published 1972 through 1981 (EPRI, 1986a) and the Guide to Technicat Reports Published January 1982-September 1, 1986 (EPRI, 1986b). Because the scope and direction of its research is largely de- fined by the industry, its program will not necessarily cover all topics that require research from a safety perspective and is likely
38 to emphasize research with a relatively short-term payoff. It Is more likely to be concerned with the human factors aspects of control room design, hardware, and software than with research on management and organizational issues. However, several EPRI studies have been concerned with the way in which work and communications within plants is structured. EPRI is also active in developing systems based on techniques from the field of artificial intelligence and expert systems. The NRC should undertake research on methods to evaluate such sys- tems before they are incorporated into nuclear power plant oper- ations, maintenance, and training. Institute of Nuclear Power Operation (INPO) INPO is an important broker for the successful transfer of technology from research to the industry. Since it is not the intent of INPO to undertake research, it cannot be expected to provide significant effort in that area. However, together with the Nuclear Utility Management and Resource Committee (NUMARC), INPO could be the key to the application of research and, through the monitoring of performance indicators, assess the long-term effects of research applications on plant safety. The conduct of several human factors projects recommended by the pane! requires access to industry facilities, such as simula- tors, as well as to industry personnel and industry data. INPO and NUMARC should be encouraged to develop mechanisms where this necessary access could occur. The Institute of Electrical and Electronic Engineers (IEEE) The Institute of Electrical and Electronic Engineers has, through its Nuclear Power Engineering Committee (NPEC), worked for several years on guidelines and standards activities. Although it does not conduct research, NPEC could make use of a well-organized data base. Moreover, the work of this kind of group could be a source of ideas for research, which would help the NRC to target its efforts. It is most important that guidelines and standards agree with the NRC regulations to avoid sending conflicting signals to industry. When they differ research results and not regulatory fiat should resolve the conflict. NRC should
39 have sufficient liaison with groups such as NPEC to ensure good communication. Department of Energy (DOE) Another potentially useful source of knowledge applicable to the commercial nuclear power industry are DOE programs con- cerned with such topics as artificial intelligence and expert sys- tems. We encourage the NRC to maintain an awareness of this research and of its commercial applicability and to develop meth- ods to evaluate these products as they emerge from DOE, EPRI, and other programs. Given the work of DOE on inherently safe reactor designs and its role in identifying candidate systems for the next generation of nuclear power plants, there should be close and continuing cooperation between DOE and the NRC during the development of this program. For example, the panel learned at one of its briefings of a proposal for a small, inherently safe reactor that might allow one operator to control several units at a single site. As interesting as such a proposal may be, it raises questions about the workload on the operator of such a system, maintenance, and other human factors issues. National Laboratories The charter of the national laboratories is to provide support to the NRC and DOE. Since the NRC has lacked the human fac- tors staff to manage and direct all of its research, it has relied on the national laboratories to provide it with the needed exper- tise. Because of this need some national laboratories built up human factors capabilities, which constitute a valuable resource in some areas of human factors for the industry. If support, di- rective, and commitment are provided by the NRC and DOE, it can be presumed that the laboratories, as in the past, will respond appropriately by rebuilding their programs.