In April 1991 Business Week ran a cover story entitled, “I Can’t Work This ?#!!@ Thing,” about the difficulties many people have with consumer products, such as cell phones and VCRs. Today, more than 15 years later, the situation is much the same. At quite a different level of scale and consequence of the disconnect between people and technology are the major large-scale systems accidents for which human error was paramount, such as those at Three Mile Island and Chernobyl. Similarly, a major, expensive console update to the nation’s air traffic control operations was cancelled because the operational personnel concluded that it would be too complicated and difficult to operate. These examples illustrate the pressures on industry and government as the complexity of the systems they seek to develop increase at the same time they are challenged to shorten the development cycle for those systems. These problems are magnified by the increasing prevalence of systems of systems. Systems of systems arise when a collection of different systems, originally designed for their own purposes, are combined and coordinated to produce a very large system with new issues and challenges.
These problems can be traced to a significant challenge—that human capabilities and needs must be considered early and throughout system design and development. One aspect of the challenge has been providing the background and data needed for the seamless integration of humans into the design process from various perspectives (human factors engineering, manpower, personnel, training, safety and health, and, in the military, habitability and survivability). This collection of development activities
has come to be called human-system integration (HSI). A second aspect has been a lack of commitment by funders and program managers to assign priority to these activities. A third aspect has been a lack of effective communication between the system engineers and human-system domain experts.
To address these challenges, the Army Research Laboratory and the Air Force Research Laboratory of the U.S. Department of Defense asked the National Academies, through its Committee on Human Factors, to undertake a study of the current state of methods, tools, and approaches for analyzing human capabilities and needs and to develop a vision for creating an integrated, multidisciplinary, generalizable, human-system design methodology. The Committee on Human-System Design Support for Changing Technology was specifically charged with four tasks:
Provide a comprehensive review of issues involved in design throughout the system life cycle that need to be addressed by a consideration of human cognitive and physical performance characteristics. This review will be used as a framework for further analysis of methodologies.
Evaluate the state of the art in human-system engineering and (a) product development processes, (b) product design methodologies, and (c) product design tools.
Develop a vision for an integrated, multidisciplinary, generalizable, human-system design support methodology and tool set. Identify a set of core methods and tools needed to support design activities associated with a variety of systems.
Recommend a research plan suggesting how to achieve this ideal.
In carrying out its work, the committee’s goal was to make recommendations that are relevant not only to the project’s military sponsors, but also to other government departments and the private sector, including the process control, manufacturing, and service industries.
PRINCIPLES FOR SUCCESSFUL SYSTEM DEVELOPMENT
The committee identified five principles that are critical to the success of human-intensive system development and evolution: (1) satisficing1 the requirements of the system stakeholders—the buyers, developers (including engineers and human factors experts), and users; (2) incremental growth of system definition and stakeholder commitment; (3) iterative system defini-
tion and development; (4) concurrent system definition and development; and (5) management of project risk.
After analysis of several candidate system development models in terms of the five principles, the committee proposes the incremental commitment model as a useful systems engineering approach and as a framework for examining categories of methodologies and tools that provide information about the environment, the organization, the work, and the human operator at each stage of the design process. Although it is not the only model that could be used on future human-intensive systems and systems of systems, it provides a reasonably robust framework for explaining the study’s HSI concepts. A central focus of the model is the progressive reduction of risk through the full life-cycle of system development, to produce a cost-effective system that meets the needs of all the stakeholders. Cost-effectiveness is achieved by focusing resources on high-risk aspects of the development and deemphasizing aspects that are judged to pose a limited risk. All kinds of potential risk, including hardware, software, and HSI risks, must be assessed to identify risk-reduction strategies at each stage in the system development process. The model recognizes that, in very large and complex systems, requirements change and evolve throughout the design process. The approach to acquisition is incremental and evolutionary: acquiring the most important and well-understood capabilities first; working concurrently on engineering requirements and solutions; using prototypes, models, and simulations as ways of exploring design implications to reduce the risk of specifying inappropriate requirements; and basing requirements on stakeholder involvement and assessments. When trade-offs among cost, schedule, performance, and capabilities are not well understood, the model provides a framework to specify priorities for the capabilities and ranges of satisfactory performance, rather than to require precise and unambiguous requirements.
The incremental commitment model has five life-cycle development phases: exploration, valuation, architecting, development, and operation. In each phase, every activity must be considered, from system scoping through goals and objectives requirements and evaluation through operations and retirement. The specific level of the effort on each activity is risk-driven and thus varies across life-cycle phases and from project to project.
The committee concludes that a model such as the incremental commitment model that incorporates the five principles can provide a significant improvement in the design of major systems, particularly with regard to human-system integration. Our policy recommendations follow from this conclusion. These recommendations are followed by an overview of the committee’s recommended research agenda.
Recommendation: The U.S. Department of Defense and other government and private organizations should refine and coordinate the definition and adoption of a system development process that incorporates the principles embodied in the incremental commitment model. It should be adopted as the recommended approach for realizing the full integration of human-related design considerations with systems engineering in organizational policies and process standards, such as the DoD 5000 series and the ISO systems engineering standards.
Recommendation: The U.S. Department of Defense and other government and private organizations should revise current system acquisition policies and standards to enable incremental, evolutionary, capabilities-based system acquisition that includes HSI requirements and uses risk-driven levels of requirements detail, particularly for complex systems of systems and for collaboration-intensive systems.
Recommendation: The U.S. Department of Defense and other government and private organizations should put the operational requirements of human-system integration on a par with traditional engineering requirements at the beginning of the initial requirements analyses to determine which requirements have priority and provide an opportunity for negotiation.
Recommendation: When developing system acquisition programs, the U.S. Department of Defense and other government and private organizations should define potential means for verifying and validating HSI requirements to enable supplier program managers to establish clearly specifiable HSI technical performance measures for contracts.
Recommendation: The U.S. Department of Defense and other government and private organizations should account for HSI considerations in developing the technical, cost, and schedule parameters in the business offer. In particular, contracts need to reflect an understanding of how human-system integration affects the ability to reuse existing technical solutions or the feasibility of inserting new technologies, as well as an appreciation of how anticipated HSI risks may affect meeting program award fee criteria. It is also important that the contractor understand how HSI elements in their product offering contribute to achieving market capture goals and subsequently the viability of their business case.
The committee makes research recommendations that the U.S. Department of Defense and other research funders support (1) the development of shared representations for facilitating effective communication among funders, developers, and users, (2) the extension and expansion of current human-system methods and tools, and (3) the full integration of human systems and engineering systems. Chapter 10 provides details.
Effective and efficient design requires meaningful communication among hardware, software, and HSI designers; among professionals in the domains of human-system design (e.g., personnel, manpower, training, human factors); and among the stakeholders. With a great deal of diversity among the groups tasked with the design of complex systems, the potential for communication and collaboration failures increases if assumptions (and their associated mind sets) are not made explicit. One approach to dealing with such diversity is through shared representations. The production of an explicit representation at various stages in the design process can provide a focus for people from different disciplines to document what they have accomplished and provide a plan for what they will do next. Just as architects provide blueprints, perspective drawings, and physical models to communicate a design, when people from different perspectives collaborate in a design process, they bring the results of various methods and tools to the activity as a shareable representation to communicate design opportunities and constraints. Shared representations can be stories, sketches, models, simulations, prototypes, spreadsheets, or reports in various levels of detail.
The committee recommends research to identify the characteristics of shared representations that communicate effectively across HSI domains and engineering disciplines.
Methods and Tools
There are many human-system methods that inform the system design and development process and many produce shared representations. In this report we review more than 20 categories of methods, many with several variations. Examples include environmental and organizational analysis, task analysis, field observation, participatory analysis and design, event data analysis, physical ergonomics, modeling and simulation, risk analysis, and usability evaluation. Each method is described broadly in terms of gen-
eral characteristics, types of use, shared representations, contributions to the system design process, and strengths, weaknesses, and gaps. Our review is not exhaustive but presents state-of-the-art examples in the categories of methods that the committee agreed are core contributors and central to the provision of needed information about humans and human-system integration. Besides the strength in terms of sheer number of methods, the methods as a whole can also be characterized as highly flexible, fluid, tailorable, scalable, or modifiable—all characteristics that are critical given the current complexity of systems and their associated design uncertainty.
The committee recommends a detailed agenda to extend existing methods and the development of new methods of human-system integration. The recommendations cover seven major areas:
The development of software tools to capture and disseminate the results of context of use analyses so that they can more easily by applied in various phases of system life-cycle development.
The active participation of users in engineering design, the future of unobtrusive, passive data collection, and the ethical considerations of both.
The further development and validation of human-system models to increase usability and expand their application.
The further development of prototypes for training and organizational design.
The identification and communication of human-system development risk.
The further development of cost-effective usability evaluation methods and the more frequent and effective use of usability objectives at the beginning of a system development effort.
The identification and assessment of human-system integration to system adaptability and resilience.
Full Integration of Human Systems and Systems Engineering
The committee recommends research in seven areas to support the full meshing of human-system integration and systems engineering into the system design and development process. These include
Managing integrated system development.
Providing traceability of HSI design objectives, decision points, and the rationale for decisions across life-cycle design phases.
Developing approaches to human-system integration in the context of systems of systems.
Estimating the size of the HSI development effort.
Creating knowledge-based planning tools for including human-system integration in complex system development efforts.
Developing human-system integration as a discipline and preparing HSI specialists to be system development managers.
Fostering more synergy between research and practice.
With the policy and research we recommend, we envision methodology for human-system integration that will be based on anticipated advances in technology in which the products of each design and development activity are manifest in representations that may be shared across the development community. In this approach, each product builds on the reusable components of previous ones. Common threads are provided by storyboards, use cases, scenarios, time lines, models, and system simulations. The stakeholders in a system will cooperate as an integrated team. The resulting design will accomplish much of system integration before implementation begins, and the result will represent a system that is truly responsive to the needs of its users, the ultimate goal of human-system integration.
In addition to the development and application of an integrated methodology, the future would hold the opportunity for the development of a discipline of human-system integration and the opportunity for HSI-led system development, the more active participation by users in system design through the use of new web-based approaches and other technologies, and the development of a set of knowledge-based planning aids to support the sharing of information across domains.