3
Human-System Integration and the System Development Process
An important theme of this report is the integration of human-system methods within the system development process, so that multiple human-system integration (HSI) concerns can be addressed effectively with the least resource expenditure. This reflects the position of Miller (1953) in his initial description of the task analysis method as a procedure that can serve design and training needs analysis equally well. The committee findings indicate that a core set of human factors method classes can serve as integrating links across the diverse HSI concerns of human factors, manpower, personnel, training and system safety, health hazards, and survivability. Furthermore, shared representations of the outputs of these methods can be developed that effectively will communicate findings and conclusions among HSI domains as well with hardware and software developers and other stakeholders.
Three general classes of human factors methods provide a robust representation of the multiple HSI concerns, and are applicable at varying levels of effort throughout the development life cycle. These broad classes include methods to:
-
Define opportunities and context of use: Methods for analyses that contribute to early definitions of opportunities and requirements and that attempt to characterize the context of use, including characteristics of users, their tasks, and the broader physical and organizational environment in which they operate so as to build systems that will effectively meet the needs of users and their work and will function smoothly within the broader physical and organizational context.
-
Define requirements and design solutions: Methods to identify requirements and design alternatives to meet the requirements revealed by prior front-end analyses.
-
Evaluate: Methods to evaluate the adequacy of proposed design solutions and propel further design innovation.
These methods generate objective data concerning critical human-system issues, leading to incremental growth of system definition and stakeholder commitment.
In Chapter 2 we showed how system development activities of the incremental commitment model (ICM) were distributed across system life-cycle phases (see Figure 2-3). Here, Figure 3-1 illustrates broadly how the four major classes of HSI activity relate to these phases. For example, activities related to understanding the context of use are likely be concentrated early in system development, when characteristics of the users, their work, and the environmental context are first being understood. However, because the context of use is constantly evolving and introduction of new technology is likely to produce operational and organizational changes, not all of which will have been anticipated ahead of time, it is important to continue to devote some (albeit lower) level of effort to examination of the context of use and how it evolves throughout the system development and deployment process, both to guide midcourse design corrections and to lay the groundwork for next-generation system development.
HUMAN-SYSTEM INTEGRATION IN THE INCREMENTAL COMMITMENT MODEL
In order to place human-system integration and its associated methods in the risk management context that is central to the incremental commitment model, it is important to distinguish several types of risk. End-state operational system risks include low usability, high rates of human error, low productivity, and safety problems. These types of risks tend to become manifest during the development process as a failure to properly manage HSI risks. They include such problems as specifying the user interface too early in design (or alternatively not considering it all), poorly understood work domain constraints, insufficient stakeholder engagement, and lack of personnel–organizational system interoperability in systems of systems. Often these types of risks are simply accepted or minimized because they pose a threat to maintaining program cost and schedule (program management risks).
Properly balancing these various categories of risk can be accommodated in the incremental commitment model, as it is a risk-driven process that aims to identify and properly manage these various risk categories. By engaging appropriate HSI methods during the incremental development process, risks can be reduced throughout the engineering life cycle, increasing the likelihood of a system’s meeting user requirements and satisficing stakeholders. Appendix Table 3-A1 lists best practices for human-system integration taken from ISO/PAS 18152 (International Organization for Standardization, 2003) and categorized by activity category. Each of these practices is valuable for successful human-centered design. Examples of methods to use in implementing these practices are also shown in the table. A risk assessment can be used to decide how much effort is needed to implement each practice in the context of a particular project.
-
Are the objectives that the user or user organization wants to achieve through use of the system already known, or is some field investigation necessary?
-
How important is it to establish measurable usability criteria for the system in its intended context of use?
-
What are the risks if end-users are not involved in each evaluation?
Figure 3-2 illustrates the links between desired system end-state (stakeholder satisficing), system phases, and HSI activities. This figure conveys the multiple determinants of the ultimate system design goal, stakeholder satisficing. The system development principles identified in Chapter 2 are shown as inputs to the system engineering processes or phases. Each of the phases is conducted iteratively, as described in the incremental com-
mitment model, and requires inputs from multiple HSI methods. This is illustrated by the network of links between HSI activities and the systems engineering phases. As with the level of activity diagram in Figure 3-1, our main point is that HSI activities are concurrent, iterative processes carried out as needed to reduce development risks at various incremental stages of system design.
The role of human-system integration in the management of engineering development risk is a relatively new concept. Human factors engineering methods are traditionally conceived as design-aiding techniques, to be used when it is time to design or test very specific elements of the human-system interface. However, this conception is too narrow for complex systems that increasingly involve multiple teams of distributed operational personnel. Instead, human factors methods can be more broadly conceived both as design-aiding techniques and as methods for progressive risk reduction during the life cycle. In this sense, human factors methods contribute to the development process in much the same way as, for example, prototyping or simulation is employed by systems engineers and can be used during the early to middle stages of development to evaluate alternatives and to narrow design choices based on various constraints. The use of such risk-reduction approaches allows developers to select one design approach over another, gain an understanding of unanticipated effects based on simulation, and generally to have a higher level of confidence that system development efforts are on track to meet requirements and avoid the operational stage risks of disuse, error, and high life-cycle costs.
Extensive experience with major system development efforts by committee members and colleagues in the profession suggests that human factors issues have often been underutilized during system development because of a perception by program managers that the risks of cost and schedule delay associated with human-system integration exceed the benefit to be delivered. Part of this perception is associated with a standard waterfall model of design, in which specific milestones are set in time, and HSI analytic methods tend to be time-intensive if performed in a linear fashion. Program managers often perceive human factors professionals as overly focused on comprehensive application of methods, while the art of engineering is to accommodate the realities of schedule and cost, conducting studies and analyses only as necessary to manage risks. It is thus important for HSI practitioners to adopt an incremental and iterative approach to analysis and design, recognizing that if there are no HSI risks associated with a particular aspect of a project, then it is unnecessary to apply various methods. The HSI profession has seen a trend in this direction with the development of
such approaches as quick look reports1 in which the precision of laboratory experiment or exhaustive observation is traded off with the expedience of providing the most critical design inputs through rapid prototyping, contextual inquiry, and various forms of participatory design.
By incorporating human-system integration as an integral thread within the incremental commitment model, the balance of program risks with HSI risks can be accommodated. This is especially true if HSI professionals are incorporated as members of an integrated product team structure that is involved continuously throughout the design cycle. Early iterations of work domain analysis, for example, may be conducted at a fairly high level to ensure that all appropriate stakeholders are identified and represented. As designs become more elaborated, participatory techniques can be applied to the point of reaching stakeholder consensus for purposes of a specific increment. This linkage of HSI activities to incremental development permits the design and risk management process to serve as a way to select the most appropriate HSI methods (and their extent of application) for a particular phase of the engineering cycle. It is not necessary to apply HSI techniques in a monolithic fashion, and in some cases it may not be necessary at all because there is no risk associated with a particular HSI issue; alternatively, program managers may accept certain identified risks (e.g., commercial off-the-shelf user interfaces) in order to preclude rejecting the use of certain technologies.
An important aspect of the incremental commitment model and the spiral representation for HSI professionals is the notion that methods are not only progressive, but also iterative, and that risk analysis determines the frequency and extent of their application. This has important implications for sizing the HSI effort, since the resources devoted to HSI activities should reflect the requirements for their application. This is a difficult task to accomplish currently, since there are no well-established methods for estimating the resource requirements for human-system integration. Various systems engineering approaches to level of effort sizing include activity-based costing, comparison with previous projects of similar scope, applying a unit-cost basis (as when a request for proposal specifies how many human factors engineers should work on a system), parametric models that link effort to project complexity, expert consensus, and risk trade-off analysis. None of these approaches has been systematically examined for sizing HSI efforts, and this area represents a knowledge gap that could be addressed through research.
COMMUNICATING HSI ISSUES AND OPPORTUNITIES THROUGH SHARED REPRESENTATIONS
A critical concern of HSI professionals and a major theme of the committee’s work is the need not only to communicate effectively within the specific HSI domain but to share findings across all systems engineering domains. This can be a daunting challenge for groups tasked with complex systems design. In addition, there is a clear need to share design and process artifacts at all phases in the systems development process, especially with the software and hardware developers who are actually implementing the system, and who are not only relying on clear specifications for development, but who are also expected to contribute to the generation of those specifications at critical decision points in the process. As a consequence, the committee has pursued the concept of shared representations as a means of addressing this concern. Shared representations—particularly diagrammatic models and other more visual, holistic representations—can serve as the fundamental medium for interactions among individuals, teams, and the organization.
Imagine a scenario from the commercial software development world. A team meets to kick off a new project. Groups of people from business, technology, and human systems discuss the goals of the project, the timeline, and other variables. Notes from the meeting are distributed within a few hours. Each group works on its tasks for several weeks. At the next meeting, team members discover that each discipline has taken its interpretation of the goals in directions different from those of the other groups, and now weeks have gone by with little truly collaborative progress to show for their efforts. Everyone attended the meeting and everyone read the notes, so what happened? The verbal description of goals and the written documentation of the events were not enough to provide common ground for the team. They lacked a shared representation of the event, their views, and what needed to be done by each of the groups.
A shared representation is an artifact or experience that mediates the interaction between and among people coming from multiple perspectives (different organizational roles, distinct technical or business backgrounds, etc.). It can be useful for an individual, a team, or an organization (Curtis, Krasner, and Iscoe, 1988). A representation can provide support or scaffolding for effective collaboration among people in transdisciplinary design teams, and at the same time be used at the organizational level to “communicate up” to forge understandings between and among the various project stakeholders. A shared representation is most powerful when used not only to facilitate activities, but also to make people’s assumptions and individual mind sets explicit.
Shared representations act as a means for synchronization, clarification, and grounding in the socially constructed process of design (D’Astous
et al., 2004; Olson et al., 1992). In the process, solutions are negotiated, the representation acts as mediator, and subsequent modifications made to the representation make explicit the result of those negotiations.
Why Shared Representations Are Useful
Models, diagrams, and other, more visual shared representations are effective for people as they participate in design activities. Don Norman writes: “Without external aids, memory, thought, and reasoning are all constrained. But human intelligence is highly flexible and adaptive, superb at inventing procedures and objects that overcome its own limits.” He goes on to suggest that one can enhance cognitive ability by producing representations or artifacts to help one think (Norman, 1993; see also Hutchins, 1995; Nardi, 1996; Pasztory, 2005). When people externalize their thinking via representations (e.g., get their ideas out on paper or on screen), they produce a representation of their thinking that not only can be examined critically, but also can be used to reduce their working memory load (Nardi, 1996; Suwa and Tversky, 2002).
In addition, by producing the representation and taking the information beyond words into a new form or medium, relationships among meaningful elements of the design must either be made explicit or must emerge, by the simple act of creating an explicit shared representation of the component elements in a well-framed space (e.g., a two-D sketch, a three-D volumetric representation, or some higher dimensional parametric space characterizing the design elements). Seeing these placements can not only lead the reader or the originators to recognize previously unacknowledged connections or relations, but also produce new connections and ideas. Suwa and Tversky (2002) call the activity detection of unintended relations and features. For example, in the applied cognitive work analysis (ACWA) method, the functional abstraction hierarchy is designed to highlight critical domain relationships that define the problem-space confronting domain practitioners. Each subsequent artifact in the process builds on the original model and, through negotiation, points to a model of what the system should be in such a way that it can finally be prototyped.
Shared representations act as mediators in the collaborative and iterative construction of knowledge in the design process. When multiple people build, share, comment, and change a shared information base, they are collaboratively constructing new knowledge (Bucciarelli, 1988; Suthers, 2005). If participants produce different types of artifacts, representations, or models as they engage in the design and development process—these should, as Norman suggests, “help them think” and make their assumptions explicit. In a sense, shared representations work in the same way that blueprints work for architects in moving from what might be to what is built. Many different views are produced. Everyone involved in the pro-
cess—from stakeholders to the various disciplines involved—is able to use the abstraction reflected in the blueprints to make meaningful decisions. For all these reasons, producing shared representations in transdisciplinary teams can be critical to creating innovative solutions because they help teams collectively see and communicate about novel connections, spawn new ideas, and facilitate a more effective design process (Détienne, 2006; Evenson, 2005).
There is a dynamic between the representation’s role in facilitating externalization (making explicit the group’s assumptions and beliefs) and its role in acting as an environment for conversation, to facilitate subsequent negotiations and elaboration. The task of creating the representation initiates the process of making explicit the underlying goals, assumptions, and viewpoints of the different design team members; the process of updating the representation following design team negotiations captures the results of negotiating design meaning out of nonverbal, semiverbal, and verbal conversations engaged in by the various team members while discussing the current model or representation (Suthers, 2005).
Attributes of Good Shared Representations
Nearly every activity in the system development life cycle results in some form of tangible design artifact, but it may not necessarily be a good shared representation. To be useful, a shared representation should
-
establish a shared language that is appropriately aligned with the development or communication problem to be solved.
-
provide a strategically chosen extent of ambiguity versus definition.
-
facilitate the desired social process (e.g., critique and redesign versus accept/reject decisions).
-
make differences and relationships apparent.
-
facilitate group “thinking with” (Norman, 1993) to transform knowledge and create new understandings (Carlile, 2002).
-
provide a meaningful structure, content, and appearance to both the creators of the shared representation and the consumers of that shared representation.
Of the six attributes listed above, the two most important are the shared language and facilitation of a social process.
To establish the language, a shared representation should be easily read by all of its users (creators and recipients)—that is, the structure and content should be easily perceived and comprehended, reflect the structure and content of the ideas or mental representation, and create a sort of resonance. The participants in the construction process should agree that
the thing produced adequately represents what they want it to (Tversky, Morrison, and Betrancourt, 2002).
To facilitate the social process, the representation must stand in and mediate communication between and among people engaged in the collaborative process (Boland and Collopy, 2004). In other words, the representation must be suitable for facilitating negotiation among the participants.
Shared Representations in the Design Process
What is useful as a shared representation can change over time in the systems development process. In practice, the process of constructing the shared representation may be more important for team building than the artifact itself. Early on, mapping out the territory the system is expected to address can draw out existing preconceptions and knowledge held by the various team members, helping to bound research activities. A territory map is an example of a shared representation that captures more a gestalt or overview of the system. It is suggestive of everything the system is and—by virtue of what is left out—everything it is not. A good territory map accounts for all the stakeholder interests in the system; a great territory map provides a picture of the system that is comprehensive, cohesive, and visionary. Completed early enough in the process, a territory map can even serve to mediate the communication of the participants in the acquisition process. Teams that agree on what is in the territory have established common ground that can be carried forward throughout the design and development process.
Documentation that focuses on activities identified from a territory map often becomes successful shared representations. They are produced in the midst of extensive task, process, or environmental research and provide a way to discuss what currently happens and what should or could happen. A standard recording language (UML, activity diagrams, business process modeling notation, etc.) facilitates discussion and contributes to the production of the shared representation. A less common but often effective shared representation early in the process can be developed with a focus on the target users of the system. For example, the findings from user-generated field journals (incorporating a standardized and embedded framework for users to record their observations) helps to extend the language of the team and build a model of system attributes important to the target user group.
Sometimes shared representations can function as a vehicle for the clarification of ideas (e.g., Suthers, 2005) or as an opportunity for groups to combine their different perspectives and knowledge into new insights (Muller, 2003; Muller et al., 1994). This is often the case when a low-fidelity prototype (such as a blank shape that is used symbolically “in place of” a real device or prototype) is used as a candidate for eliciting different
stakeholders’ ideas about what would be done with the product, system, or service, as well as for eliciting and exploring different stakeholder concepts and assumptions. This was successfully demonstrated in the UTOPIA project, in which the implications for working relationships from a new print shop technology were explored by placing low-fidelity mock-ups of the new technology in the existing print shop and by acting out the new work practices around those prototypes (Ehn and Kyng, 1991).
Erickson reviewed the importance of “roughness” in a shared representation, noting that less formal and less finished representations were more likely to elicit useful comment, critique, and improvement, whereas more formalized or polished representations were more likely to lead to simple accept/reject decisions (Erickson, 1996). People feel more open to participating in and refining ideas in sketches and than they do in finished prototypes.
Personas are also often an excellent shared representation category because they are composite user archetypes based on behavioral data gathered from many actual people during discovery research (see Chapter 7). Personas are useful because they build on people’s expectations about other people’s behavior from what is known about that person. Developing solid profiles or personas contributes to serving individual user needs, aids in integration with customer processes, and leads to a design that the various constituents can participate in co-evolving. As a shared representation, personas and profiles are a tool for making user needs explicit, differentiating between and among different stakeholders, and prioritizing different and sometimes competing goals (Cooper, 2004).
Even physical spaces can become successful shared representation of the system or systems to be designed. For example, in a design project intended to reconceptualize 35mm point-and-shoot cameras, a physical design space was built initially to contain the results of qualitative research conducted to understand the existing paradigm (Rheinfrank and Welker, 1994). The space contained images and relevant artifacts that characterized different aspects of use and users of cameras. Initially, each wall of the space individually represented a particular aspect of the experience and was used more as a repository for the information about each dimension. Over time, however, the space was seen as a whole and became a shared representation in the collaborative process to solve a multidimensional camera design problem (Star, 1989). Specifically, the physical space evolved into a shared representation offering the internal and external design teams multiple views of possible 35mm camera futures, in which the view depended on its position on the floor in juxtaposition to the walls of the room (Rheinfrank and Welker, 1994). In later cycles of development, scoping maps (1) illustrate the features, functionality, and content of the designed system, (2) illustrate anticipated user experiences, and (3) enable team members to prioritize a
plan for staged release. These types of shared representations allow the stakeholders to make decisions about what can and should be produced from the potential things that could be built. Eventually, a high-fidelity representation, such as a functional prototype, is a good candidate for validating what the team collectively knows about the system and for communicating a clear idea of the system from one design group to another, as well as “up” the organization.
As the design and development process unfolds, and when groups shift from one type of shared representation to another or change the way they are using a shared representation, the shift signals a qualitative change in the know-how needed for continuing to make progress in the design/development process (Cook and Brown, 1999; Gasson, 2005).
CONCLUSION
Effective use of shared representations depends on understanding the team or organization’s current issues and needs in communication, and in strategically choosing the right kind of shared representation to mediate at the right time.
When used appropriately, shared representations enable the design team to coalesce around a shared view, while providing a capacity for increasing the conceptual complexity that can be attended to—activities that are crucial in the design of complex systems.
Shared representations can provide a bridge among analysis, design, implementation, and training in complex systems design, development, and fielding. The act of producing the representation can help teams detect unanticipated relations and features that can be exploited to lead to new connections and ideas.
Shared representations can be anything from a simple sketch, to a “wizard of Oz” prototype, to a fully active simulation of system design and behavior. Although conventional project planning schedules or spreadsheets can support the design and development process, they can never take the place of consciously planning, producing, and seeding discussion around shared representations to improve the quality of collaboration and productive outcomes of transdisciplinary design teams (Carroll, 2002). Shared representations provide a means for teams to transcend conventional project management paradigms and to coalesce around their ideas to produce work that is a reflection of their shared understanding of the mission to be supported, the user needs, and the best that technology can deliver.
APPENDIX 3-A
TABLE 3-A1 Best Practices for Risk Mitigation
Activity Category |
Best Practices for Risk Mitigation from ISO/PAS 18152 |
Example HSI Methods and Techniques |
|
|
|
|
|
Work context analysis |
|
|
Success-critical stakeholder identification Context of use analysis Work context analysis Investigate required system usability Usability benchmarking |
Activity Category |
Best Practices for Risk Mitigation from ISO/PAS 18152 |
Example HSI Methods and Techniques |
|
|
Define the intended context of use including boundaries Identify staffing requirements and any training or support needed to ensure that users achieve acceptable performance Storyboards Establish performance and satisfaction goals for specific scenarios of use Define detailed user interface requirements Prioritize requirements (e.g., QFD) |
|
|
Function allocation Generate design options |
Activity Category |
Best Practices for Risk Mitigation from ISO/PAS 18152 |
Example HSI Methods and Techniques |
|
|
Human performance model Design for alertness Plan staffing |
|
See (a) System architecting. |
|
|
See (a) System architecting. |
User interface guidelines and standards |
Activity Category |
Best Practices for Risk Mitigation from ISO/PAS 18152 |
Example HSI Methods and Techniques |
|
|
Plan to achieve and maintain usability Plan use of HSI data to mitigate risks Identify HSI issues and aspects of the system requiring user input Develop a plan for user involvement Select and use the most effective methods Customize tools and methods as necessary |
Activity Category |
Best Practices for Risk Mitigation from ISO/PAS 18152 |
Example HSI Methods and Techniques |
|
|
Obtain user feedback on usability Compare with requirements Performance measurement HR |
Activity Category |
Best Practices for Risk Mitigation from ISO/PAS 18152 |
Example HSI Methods and Techniques |
|
|
Value-based practices and principles (identify success-critical stakeholder requirements) Environment/organization assessment |
|
|
User feedback on usability Performance measurement |
|
|
User feedback Work context analysis |
Activity Category |
Best Practices for Risk Mitigation from ISO/PAS 18152 |
Example HSI Methods and Techniques |
|
|
Work context analysis |
Activity Category |
Best Practices for Risk Mitigation from ISO/PAS 18152 |
Example HSI Methods and Techniques |
|
|
Assess and improve HSI capability Develop and maintain HSI infrastructure and resources Identify required HSI skills Provide staff with HSI skills Establish and communicate a policy on HSI Maintain an awareness of usability |
NOTES: Italicized items are methods not covered in Chapters 6-8. HF = human factors. HS = human-system. HR = human resources. QFD = quality function deployment. HCD = human-centered design. |