5
Case Studies

This chapter provides three examples of specific system development that illustrate application of human-system integration (HSI) methods in the context of the incremental commitment model (ICM). The examples are drawn from the committee’s collective experience and specific application of the concepts developed during our work to these particular projects. They represent projects at three stages of development: the early stages of planning, in mid-development, and fully realized.

The first example involves the development of unmanned aerial systems and identifies numerous HSI issues in these systems that will require solution. This example provides a “notional” application of human factors methods and potential implementation of the incremental commitment model. The case study illustrates the theme of designing to accommodate changing conditions and requirements in the workplace. Specifically, it addresses the issue of adapting current unmanned aerial systems to accommodate fewer operators, with individual operators controlling multiple vehicles. The hypothetical solutions to this problem reveal the potential costs of reliance on automation, particularly prior to a full understanding of the domain, task, and operator strengths and limitations. This case study also reveals the tight interconnection between the various facets of human-system integration, such as manpower, personnel, training, and design. In other words, answering the “how many operators to vehicles” question necessarily impacts design, training, and personnel decisions.

The second example focuses on a large-scale government implementation of port security systems for protection against nuclear smuggling. The example discusses the HSI themes and incremental application of methods



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 91
Human-System Integration in the System Development Process: A New Look 5 Case Studies This chapter provides three examples of specific system development that illustrate application of human-system integration (HSI) methods in the context of the incremental commitment model (ICM). The examples are drawn from the committee’s collective experience and specific application of the concepts developed during our work to these particular projects. They represent projects at three stages of development: the early stages of planning, in mid-development, and fully realized. The first example involves the development of unmanned aerial systems and identifies numerous HSI issues in these systems that will require solution. This example provides a “notional” application of human factors methods and potential implementation of the incremental commitment model. The case study illustrates the theme of designing to accommodate changing conditions and requirements in the workplace. Specifically, it addresses the issue of adapting current unmanned aerial systems to accommodate fewer operators, with individual operators controlling multiple vehicles. The hypothetical solutions to this problem reveal the potential costs of reliance on automation, particularly prior to a full understanding of the domain, task, and operator strengths and limitations. This case study also reveals the tight interconnection between the various facets of human-system integration, such as manpower, personnel, training, and design. In other words, answering the “how many operators to vehicles” question necessarily impacts design, training, and personnel decisions. The second example focuses on a large-scale government implementation of port security systems for protection against nuclear smuggling. The example discusses the HSI themes and incremental application of methods

OCR for page 91
Human-System Integration in the System Development Process: A New Look during the iterative development of the system. This case is useful for illustrating application of human factors methods on a risk-driven basis, as they tend to be applied as needed over time in response to the iterative aspects of defining requirements and opportunities, developing design solutions, and evaluation of operational experience. The third example describes development of an intravenous infusion pump by a medical device manufacturer. This example is the most detailed and “linear” of the three cases, in that it follows a sequential developmental process; the various systems engineering phases are discussed in terms of the human factors methods applied during each phase. This case study illustrates the successful implementation of well-known HSI methods, including contextual inquiry, prototyping and simulations, cognitive walkthroughs for estimating use-error-induced operational risks, iterative design, and usability evaluations that include testing and expert reviews. The importance of the incremental commitment model in phased decision making and the value of shared representations is also highlighted. Each of these examples is presented in a somewhat different format, as appropriate to the type of development. This presentation emphasizes one broad finding from our study, which is that a “one size” system development model does not fit all. The examples illustrate tailored application of HSI methods, the various trade-offs that are made to incorporate them in the larger context of engineering development, and the overall theme of reducing the risk that operational systems will fail to meet user needs. UNMANNED AERIAL SYSTEMS Unmanned aerial systems (UASs) or remotely piloted vehicles (RPVs) are airplanes or helicopters operated remotely by humans on the ground or in some cases from a moving air, ground, or water vehicle. Until recently the term “unmanned aerial vehicle” (UAV) was used in the military services in reference to such vehicles as Predators, Global Hawks, Pioneers, Hunters, and Shadows. The term “unmanned aerial system” acknowledges the fact that the focus is on much more than a vehicle. The vehicle is only part of a large interconnected system that connects other humans and machines on the ground and in the air to carry out tasks ranging from UAS maintenance and operation to data interpretation and sensor operation. The recognition of the system in its full complexity is consistent with the evolution from human-machine design to human-system design, the topic of this report. It highlights an important theme of this book: the need for methods that are scalable to complex systems of systems. Unmanned aerial systems are intended to keep humans out of harm’s way. However, humans are still on the ground performing maintenance, control, monitoring, and data collection functions, among others. Reports

OCR for page 91
Human-System Integration in the System Development Process: A New Look from the Army indicate that 22 people are required on the ground to operate, maintain, and oversee a Shadow UAS (Bruce Hunn, personal communication). In addition, there is a dearth of UAS operators relative to the current need in Iraq and Afghanistan, not to mention the U.S. borders. The growing need for UAS personnel, combined with the current shortage, points to another theme of this report: the need for human-system integration to accommodate changing conditions and requirements in the workplace. In addition, this issue has strong ties to questions of manning. The manning questions are “How many operators does it take to operate each unmanned aerial system? Can one modify the 2:1 human to machine ratio (e.g., two humans operating one UAS) to allow for a single operator and multiple aircraft (e.g., 1:4)?” Automation is often proposed as a solution to this problem, but the problem can be much more complex. Automation is not always a solution and may, in fact, present a new set of challenges, such as loss of operator situation awareness or mode confusion. Furthermore, the manning question is a good example of how HSI design touches other aspects of human-system integration, such as manpower, personnel, and training. That is, the question of how many vehicles per operator is not merely one of automation, but also involves the number and nature of the operators in question. A Hypothetical Case This example is based on an ongoing debate about the manning question, which has not been fully resolved. Therefore some aspects of the case are hypothetical, yet not improbable. In this example we assume that the objective of the design is to change the operator to UAS ratio from 2:1 to 1:4. That is, instead of two operators for one UAS there will be one operator for four UASs. This operator to UAS ratio is a requirement of the type that may be promulgated by the Department of Defense with minimal HSI input. It could be too late for human-system integration, which needs to be fully integrated into the engineering life cycle before system requirements have been determined. It could be too late in the sense that up-front analysis might have revealed that an effective 1:4 ratio is beyond the capabilities of current humans and technology under the best of circumstances. If this is the case, then there is a huge risk of designing a system that is doomed to fail. Even worse, this failure may not reveal itself until the right operational events line up to produce workload that breaks the system. In our example, we present another scenario. The design of a UAS with a 1:4 ratio of operator to system is carried through the ICM development process to illustrate the potential role of human-system integration and one of the themes of this book. The Department of Defense is one of many

OCR for page 91
Human-System Integration in the System Development Process: A New Look critical stakeholders in this scenario, all of whom are to be considered in the satisficing process that ensues. Human-System Integration in the Context of the Incremental Commitment Model In the earliest exploration phases of ICM development, the problem space and concept of operations are defined, and concept discovery and synthesis take place. Table 5-1 provides highlights of the entire example. It is often the case that human-system integration is not brought into the development cycle at this point, although at great risk. Up-front analyses, such as interviews of UAS operators, observations of operations of 2:1 systems, examination of mishap reports, understanding of the literature and data, an analysis of the 2:1 workload, event data analysis targeted at communications in the 2:1 UAS system, application of models of operator workload, and work flow analysis are all methods that could be used to explore the HSI issues in the current UAS system. There is much that could come from this kind of up-front analysis. One hypothetical possibility is that the up-front HSI analyses could determine that UAS workload is not constant but peaks in target areas where photos need to be taken or in situations in which the route plan needs to change. One of the key principles of ICM development is risk management, including risk-driven activity levels and anchor point commitment milestones. What are the risks if human-system integration is not considered early in the development life cycle? In this case, the formal requirements that are established may target workload reduction incorrectly. For example, autopilot automation might be developed to help to get multiple UASs from point A to point B and so on. This might have the effect of reducing workload when a reduction was not needed, while providing no relief from the high-workload tasks. Ultimately the neglect of up-front human-system integration could result in a system that is ineffective or prone to error. Consideration of risks like these should guide system development. What if there is not enough time to interview UAS operators and to do a thorough job in the exploration phase? There is also risk associated with application of costly up-front techniques. The up-front methods used often during the exploration phase of the life cycle can be tailored to meet time and budget constraints—another theme of this book. For example, in this case in which the manning question is the issue and automation appears to be a promising solution, it would make sense to focus on aspects of the task that may be automated and the workload associated with each. One caveat is that decisions on how to scope and tailor the methods require some HSI expertise in order to target the aspects of human-system integration that promise the most risk reduction.

OCR for page 91
Human-System Integration in the System Development Process: A New Look As system development progresses, other principles of ICM development come into play, including incremental growth of system development and stakeholder commitment. This part of the development life-cycle synthesis leads to construction, invention, or design that is iteratively refined as it is evaluated. HSI activities that would be useful at this point include function allocation and the development of shared representations, such as storyboards and prototypes. Based on the previous finding of fluctuating workload, it may be decided that human intervention is needed at target areas and during route changes, but that the single operator can handle only one of these peak-workload tasks at a time. It may also be determined that, although automation could handle the routine flight task, an even more important place for automation is in the hand-off between the flight tasks and the human planning/replanning operation. The automation would therefore serve a scheduling and hand-off function, allocating complex tasks to the human operator as they arise and in order of priority (e.g., priority targets first). There could also be automation that serves as a decision aid for the targeting task. Because only one nonroutine task can be handled at a time under the 1:4 scenario, it may also be decided that operators should be relieved of the flight functions completely but be on call for hand-offs from automation. For example, four controllers could handle the prioritized hand-offs from the automation, much as air traffic controllers handle multiple planes in a sector. Note that this new design and staffing plan are completely different in terms of operator roles and tasks from the former 2:1 operation. It is human-system integration that guided the allocation of tasks to human and machine; without it there would have been many other possibilities for automation that may not have produced the same end-state. As the ICM development continues, the system engineers will go from working prototypes to product development, beta testing, product deployment, product maintenance, and product retirement. But there is continual iteration along the way. The incremental growth in the automation for scheduling, hand-offs, and targeting would occur in parallel with the next iteration’s requirements and subsystem definitions (i.e., concurrent engineering). Incremental growth will be influenced by stakeholder commitment. The HSI methods in the later stages include interviews and observations in conjunction with the newly designed system and usability testing. Some of the same methods used in up-front analysis (e.g., event data analysis, participatory analysis) can be again used and results contrasted with those of the earlier data collection. The goal of human-system integration at this stage is to verify that the situation for the user has improved and that no new issues have cropped up in the interim. For instance, it may be determined from testing that the targeting decision aid is not trusted by the human operator (a stakeholder)

OCR for page 91
Human-System Integration in the System Development Process: A New Look TABLE 5-1 Example of Human-System Integration for UASs in the Context of the Risk-Driven Spiral Life-Cycle Phase HSI Activity SE Activities Exploration Observe 2:1 system, interview UAS operators, examine literature/data, examine mishap reports, workload analysis and models, event data analysis, communication, work flow analysis Define problem space, concept discovery, concept of operations, synthesis Valuation and architecting Function allocation, storyboards, prototypes Synthesis, construct, invent, design, refine, hard requirements, working prototypes Development and operation Interviews, observations, usability testing, comparisons with previous system Working prototypes, product development, beta testing, product deployment, maintenance, retirement NOTE: HSI = human-system integration; SE = systems engineering; UAS = unmanned aerial system. and as a result is not used (a risk). Through iterations, a new design will be tested or the decision aid will be completely eliminated (i.e., stakeholder satisficing). Conclusion and Lessons Learned In this example, human-system integration plays a major role throughout the design process and is critical in the early stages before requirements are established. It can be integrated throughout the design life cycle with other engineering methods. It is also clear that the HSI activities serve to reduce human factors risks along the way and make evident the human factors issues that are at stake, so that these issues can be considered as they trade off with other design issues. This example illustrates several lessons regarding human-system integration and system design: The importance and complexity of the “system” in human-system integration compared with “machine” or “vehicle.” Design concerns are often linked to manpower, personnel, and training concerns.

OCR for page 91
Human-System Integration in the System Development Process: A New Look Hypothetical Outcome Risks If No HSI HSI Value-Added Workload not constant; heavy at target areas and for route change Ineffective or error-prone system Requirements targeted at known system strengths and weaknesses Automation takes over flight and hand-offs complex tasks to operator-based on priority Operator who is overwhelmed during high workload and bored during low workload Design takes into account known machine and human strengths and weaknesses Targeting decision aid not trusted by human operator Validation and verification would not consider the human limitations in relation to the new system Testing takes into account usability and comparison to prior system Up-front analysis and HSI input in early exploration activities is critical. Methods can be tailored to time and money constraints, but HSI expertise is required to do so. Risks are incurred if human-system integration is not considered or if it is considered late. In this case the risk would be a system that is not usable and that ultimately leads to catastrophic failure. PORT SECURITY The U.S. Department of Homeland Security (DHS) is in the process of implementing a large-scale radiation screening program to protect the country from nuclear weapons or dirty bombs that might be smuggled across the border through various ports of entry. This program encompasses all land, air, and maritime ports of entry. Our example focuses on radiation screening at seaports, which have a particularly complex operational nature. Seaports are structured to facilitate the rapid offloading of cargo containers from ocean-going vessels, provide temporary storage of the containers, and provide facilities for trucks and trains to load containers for transport to their final destination. The operation involves numerous personnel, includ-

OCR for page 91
Human-System Integration in the System Development Process: A New Look FIGURE 5-1 RPM security screening at seaports involves multiple tasks, displays, and people. ing customs and border protection (CBP) officers for customs and security inspection, terminal personnel, such as longshoremen for equipment operation, and transport personnel, such as truck drivers and railroad operators. Figure 5-1 illustrates the steps involved in the radiation screening process. Design and deployment of radiation portal monitoring (RPM) systems for seaport operations engages the incremental commitment model for ensuring commitments from the stakeholders and to meet the fundamental technical requirement of screening 100 percent of arriving international cargo containers for illicit radioactive material. This example illustrates aspects of the ICM process with specific instances of human-system integration linked to concurrent technical activities in the RPM program. The development of RPM systems for application in the seaport environment entails an iterative process that reflects the overall set of themes developed in this book. We discuss how these themes are reflected in the engineering process. Human-System Integration in the Context of Risk-Driven Incremental Commitments The human factors design issues encountered in this program are very diverse, ranging from fundamental questions of alarm system effectiveness at a basic research level, to very practical and time-sensitive issues, such as the most appropriate methods of signage or traffic signaling for controlling

OCR for page 91
Human-System Integration in the System Development Process: A New Look the flow of trucks through an RPM system. HSI methods have been applied on a needs-driven basis, with risk as a driver for the nature of the application. With the issue of alarm system effectiveness, for example, it was recognized early in the program that reducing system nuisance alarms is an important issue, but one that requires a considerable amount of physics research and human factors display system modeling and design. The ICM process allowed early implementation of systems with a higher nuisance alarm rate than desirable while pursuing longer term solutions to problems involving filtering, new sensors, and threat-based displays. The nuisance alarm risk was accepted for the early implementations, while concurrent engineering was performed to reduce the alarm rate and improve the threat displays for implementation in later versions. A contrasting example involves traffic signage and signaling. Since the flow of cargo trucks through port exits is a critical element of maintaining commercial flow, yet proper speed is necessary for RPM measurement, methods for proper staging of individual vehicles needed to be developed. Most ports involve some type of vehicle checkout procedure, but this could not be relied on to produce consistent vehicle speed through the RPM systems. Instead, the program engaged the HSI specialty to assist in developing appropriate signage and signaling that would ensure truck driver attention to RPM speed requirements. HSI Methods Tailored to Time and Budget Constraints Since the RPM program focus is homeland security, there has been schedule urgency from the beginning. The need for rapid deployment of RPM systems to maximize threat detection and minimize commercial impact has been the key program driver, and this has also influenced how the HSI discipline has been applied. The primary effect of program urgency and budgetary limitations has been to focus HSI efforts in work domain analysis, the modeling of human-system interactions, and theory-based analysis rather than experiment. The work domain analysis has typically focused on gaining a rapid understanding of relatively complicated seaport operations in order to evaluate technology insertion opportunities and to better understand design requirements. In contrast to work domain analysis oriented toward cognitive decision aids, which requires time-intensive collaboration with subject matter experts, the RPM analysis worked at a coarser level to characterize staff functions and interactions, material flow, and operational tempo. Similarly, modeling of human-system interactions (such as responding to a traffic light or an intercom system) was performed at the level of detail necessary to facilitate design, rather than a comprehensive representation of operator cognitive processes—this was not required to support engineering.

OCR for page 91
Human-System Integration in the System Development Process: A New Look Theory-based analysis of alarm system effectiveness has been conducted on a somewhat longer time scale, since the problem of human response to alarms is more complex. This work consisted of adapting traditional observer-based signal detection theory, in which the human is an active component of the detection system, to RPM systems in which the human operator evaluates the output of a sensor system that detects a threat precondition. Various threat probability analyses have been conducted in this effort, and they can be used to guide subsequent advanced RPM designs. This work has been guided by empirical studies, but it has not required an independent data collection effort. Shared Representations Used to Communicate The rapid-paced nature of the RPM program places a premium on effective communication between human-system integration and the engineering disciplines. In this program, fairly simple communication mechanisms that use graphics or presentation methods adapted from engineering have the best chance of successful communication. For example, it is important to evaluate the human error risks associated with new security screening systems so that mitigation approaches can be designed. One approach to describing this to the engineering community might be to simply borrow existing taxonomies from researchers in the field, such as Reason (1990). Alternatively, a more graphic and less verbose approach is to represent the approach as a fault tree, shown in Figure 5-2. This type of representation is immediately recognizable to the engineering community and is less subject to interpretation than abstract descriptions of error typologies. FIGURE 5-2 General model of human error analysis for security screening used as a shared representation to communicate the concept to engineering staff.

OCR for page 91
Human-System Integration in the System Development Process: A New Look FIGURE 5-3 Graphical representation of work flow with a threat-based RPM display. Human-system integration has used graphics to convey fairly abstract design ideas to the engineering staff, as shown in Figure 5-3. This display conveys the concept of a threat likelihood display, which informs the RPM operator about the contents of a vehicle based on processing algorithms. The graphic contrasts the eight-step process shown in Figure 5-1, with a four-step screening process, illustrating the functional utility of the display in a direct way. Accommodation to Changing Conditions and Workplace Requirements The RPM program started with a set of baseline designs for seaports that involved a cargo container passing through an exit gate. As the program expanded to a wider range of port operations, numerous variations in the container-processing operations became apparent. In some instances, the traffic volume is so low that the costs of installing a fixed installation are too high; alternatively, trenching limits or other physical constraints may preclude a fixed portal. Operational differences, such as moving containers direct to rail cars, also present challenges for design.

OCR for page 91
Human-System Integration in the System Development Process: A New Look TABLE 5-2 Methodology Issues and Research Needs Human Factors Engineering Process Step Explanation User profiles All major user categories were analyzed. Task analysis All significant task flows were analyzed. User environment All significant use environments were analyzed. Use-error risk analysis All tasks were analyzed and use-error probabilities were estimated, as were hazard severities and risk priority values. Design mitigations were described. Set and meet usability objectives Objectives were set for all critical tasks using both task completion rate and satisfaction ratings. Prototyping and iterative design Design was iterated many times over through a series of at least 12 usability test cycles. Usability testing A series of formative usability tests were conducted in both usability-testing labs and in the patient simulator. A final summative usability test was completed. Field studies Field studies are planned for the postmarketing period. A clinical device study was conducted in two hospitals to obtain real-world usability and product effectiveness data. NOTE: EEG = electroencephalogram; fMRI = functional magnetic resonance imaging.

OCR for page 91
Human-System Integration in the System Development Process: A New Look Methodology Issues Disadvantages Methodological Research Needs Are personas as descriptive of users as regular user profiles? Can we be sure that we have captured the majority of user profiles? Studies of design impacts of different ways to capture users and their profiles. Can we be sure that all significant task flows have been captured and have they been captured at the correct level of detail? Studies of advantages and disadvantages of different approaches to task analysis, including cognitive task analysis and task modeling techniques. Can we be sure that all significant use environments have been captured and have they been captured at the correct level of detail? Studies to develop methods to understand limitations of current ethnographic methods for capturing information about use environments and ways to improve these methods. When data are not available, then subjective estimates must be made for use-error rate probabilities. Group dynamics can bias the consensus ratings of hazard severity, fault likelihood, and mitigation effectiveness. Validate estimation methods to reduce bias in consensus ratings, e.g., Delphi techniques. Research methods to improve error rate modeling. Human performance usability objectives are usually limited to task completion rates and task times, supplemented by subjective measures of satisfaction. Research better, more reliable, and valid outcome measurements and methods to make these measurements, e.g., fMRI, EEG, or other neuro/physiological measures. Iteration takes time, even with rapid prototyping tools. Develop better, quicker, more efficient methods and tools for rapid prototyping. Usability tests also take a lot of time and resources. It is difficult to select the optimal tasks to include in a usability study. Can summative usability tests be done with fewer subjects and still be valid? Research more efficient usability evaluation methods. Create tools that allow easier selection of the most important and critical tasks to include in a test. Investigate the use of alternative statistical analysis methods such as Bayesian statistics to conduct summative usability tests. Field studies have the advantage of giving real-world validation to lab-based usability evaluations, but are time and resource intensive. Research techniques that are more efficient in providing the kind of postmarket surveillance data that can be obtained from field studies.

OCR for page 91
Human-System Integration in the System Development Process: A New Look FIGURE 5-7 Illustrative task flow diagram from the task analysis.

OCR for page 91
Human-System Integration in the System Development Process: A New Look values exceed a value of 45. RPN values between 8 and 45 require an explanation or justification of how the risk is controlled. The product requirements document (PDR) was formally created at this point to describe the details of the product design. It was considered a draft for revision as testing and other data became available. It was based on the customer needs expressed in the marketing requirements document. This document recorded the incremental growth of system definitions and stakeholder commitment and served as a shared representation of the design requirements for the development team. Prototypes Many prototypes and simulations were created for evaluation: Hardware models and alternatives considered hardware industrial design mock-ups early usability tests of hardware mock-ups. Paper prototypes for graphical user interfaces with wireframes consisting of basic shapes, such as boxes and buttons without finished detail graphic elements. GUI simulations using Flash™ animations.1 Early usability tests with hardware mock-ups and embedded software that delivered the Flash™ animations to a touchscreen interface that was integrated into the hardware case. Flash animations are excellent examples of shared representations because they were directly used in the product requirements document to specify to software engineering exactly how the GUI was to be developed. All team discussions regarding GUI design were focused exclusively on the Flash animation shared representations of the Symbiq™ user interface. Integrated Hardware and Software Models with Integrated Usability Tests As noted earlier, the usability tests performed later in the development cycle were done with integrated hardware mock-ups and software simulations. Usability test tasks were driven by tasks with high-risk index values in the risk analysis, specifically the FMEA. Tasks were also included that had formal usability objectives associated with them. Although the majority of usability test tasks were focused on the interaction with the touchscreen- 1 Flash refers to both a multimedia authoring program and the Macromedia Flash Player, written and distributed by Macromedia, which utilizes vector and bitmap graphics, sound and program code, and directional streaming video and audio.

OCR for page 91
Human-System Integration in the System Development Process: A New Look TABLE 5-3 Excerpts from Symbiq™ Failure Modes and Effects Analysis (FMEA) Task Hazard Use Error (Fault) Fault Probability Risk Hazard Severity Enter concentration for those drugs in library with ___/___ concentration. Patient receives over-delivery of ordered drug. Incorrect concentration entered for drug and confirmed. 2 5 Soft limit override. Patient receives under-delivery of ordered drug. Soft limit unintentionally overridden and confirmed. 2 4 based graphical user interface, critical pump-handling tasks were included as well, such as IV pump mounting and dismounting on typical IV poles. Tests of Alarm Criticality and Alerting The initial alarm formative usability studies, described earlier, had the goal of selecting alarms that would be alerting, attention getting, and properly convey alarm priority, as well as communicating appropriate actions. These formative studies evaluated the subject’s abilities to identify and dis-

OCR for page 91
Human-System Integration in the System Development Process: A New Look Method of Control Detect/Mitigate Risk Outcome (Residual Risk) Reference The completed program screen shows the drug selected. When START is pressed on a program screen, a second screen (confirmation screen) appears with the programmed values on the screen view looking slightly different. A query is displayed and user must select YES or NO. If YES selected infusion begins. If NO selected, user is returned to PROGRAM screen and has opportunity to correct the concentration before beginning delivery. After delivery is initiated, the user can view the entered concentration and correct if necessary by returning to the programming screen (by selecting the appropriate line A or B). The user manual provides instructions. Trailing zeros have been eliminated for whole numbers, e.g., 50.0 will be 50. 1 10 Study report for User Study Protocol #03-03 Version 3 revealed the experienced users read the confirmation screen and discovered and corrected any mistakes. When a value entered is not within specified rule sets, a warning appears and the override soft limits icon appears. The warning indicates the value that is outside the rule sets and the clinician must confirm override, YES or NO? If YES is chosen, the confirmation screen appears with the override icon. Query is displayed and user must select YES or NO. If YES selected, infusion begins and override icon remains on screen during infusion. If NO selected, user remains on the PROGRAM screen and has opportunity to correct. The warning icon remains visible. 1 8 Study report for User Study Protocol #03-03 Version 3 revealed the experienced users questioned the messages before overriding. criminate among different visual alarm dimensions, including colors, flash rates, and text size and contrast. For auditory alarms, subjects were tested on their ability to discriminate among various tones with and without melodies and among various cadences and tone sequences for priority levels and detectability. Subjects were asked to rate the candidate tones relative to a standard tone, which was given a value of 100. The standard was the alternating high-low European-style police siren. Subjective measures were also gathered on the tones using the PAD rating system, standing for perceived tone pleasure, arousal, and dominance, as well as perceived criticality. Data

OCR for page 91
Human-System Integration in the System Development Process: A New Look from these studies enabled the team to make further incremental decisions on system definitions for both visual and auditory alarms and alerts. Tests of Display Readability Another set of early formative usability tests was conducted to validate the selection of the particular LCD touchscreen for readability and legibility. During the evaluation it was determined that the screen angle (75 degrees) and overall curvature were acceptable. The screen could be read in all tested light conditions at a 15-foot viewing distance. Iterative Usability Tests As noted, a series of 10 usability studies were conducted iteratively as the design progressed from early wireframes to the completed user interface with all the major features implemented in a working IV pump. In one of the intermediate formative usability tests, a patient simulator facility was used at a major teaching hospital. Users performed a variety of critical tasks in a simulated room in an intensive care unit, in which other medical devices interacted and produced noises and other distractions. The prototype IV pump delivered fluid to a mannequin connected to a patient monitor that included all vital signs. As the pump was programmed and subsequently changed (e.g., doses titrated), the software-controlled patient mannequin would respond accordingly. The patient simulator also introduced ringing telephones and other realistic conditions during the usability test. This test environment helped in proving the usability of visual alarms and tones, as well as the understandability and readability of the visual displays. Final summative usability tests demonstrated that the usability objectives for the pump were achieved. Focus Groups Focus groups of nurses were also used as part of the usability evaluation process. These were used to complement the task-based usability tests. Many of the focus groups had a task performance component. Typically the participants would perform some tasks with new and old versions of design changes, such as time entry widgets on the touchscreen, and then convene to discuss and rate their experiences. This allowed a behavioral component and addressed one of the major shortcomings of typical focus groups, that they focus only on opinions and attitudes and not behaviors.

OCR for page 91
Human-System Integration in the System Development Process: A New Look Field Studies Field studies in the form of medical device studies have also been incorporated in the design process. Thoroughly bench-tested and working beta versions of the IV pump were deployed in two hospital settings. The hospitals programmed drug libraries for at least two clinical care areas. The devices were used for about 4 weeks. Surveys and interviews were conducted with the users to capture their real-world experiences with the pump. Data from the pump usage and interaction memory were also analyzed and compared with original doctor’s orders. This study revealed a number of opportunities to make improvements, including the problem with the perceived annoyance of the alarm melodies and the data entry methods for entering units of medication delivery time (e.g., hours or minutes). Instructions for Use Development and Testing Usability testing was also conducted on one of the sets of abbreviated instructions called TIPS cards. These cards serve as reminders for how to complete the most critical tasks. These usability studies involved 15 experienced nurses with minimal instructions performing 9 tasks with the requirement that they read and use the TIPS cards. Numerous suggestions for improvement in the TIPS cards themselves as well as the user interface came from this work, including how to reset the air-in-line alarm and how to address the alarm and check all on-screen help text for accuracy. Validation Usability Tests Two rounds of summative usability testing were conducted, again with experienced nurses performing critical tasks identified during the task analysis, including those with higher risk values in the risk analysis. The tasks were selected to simulate situations that the nurses may encounter while using the IV pump in a hospital setting. The tasks included selecting a clinical care area, programming simple deliveries, adding more volume at the end of an infusion, setting a “near end of infusion” alarm, titration, dose calculations, piggyback deliveries, intermittent deliveries, using standby, programming a lock, adjusting the alarm volume, and responding to messages regarding alarms. Usability objectives were used as acceptance criteria for the summative validation usability tests. The study objectives were met. The calculated task completion accuracy was 99.66 percent for all tasks for first-time nurse users with minimal training. The null hypothesis that 80 percent of the participants would rate the usability 3 or higher on a 5-point scale in the overall categories was met. There were a few minor usability problems

OCR for page 91
Human-System Integration in the System Development Process: A New Look uncovered that were subsequently fixed without major changes to the user interface or that affected critical safety-related tasks. Federal regulations on product design controls require that a product’s user interface be validated with the final working product in a simulated work environment. In this instance, the working product was used in a laboratory test, but without having the device connected to an actual patient. Bench testing is also a part of validation to ensure that all mechanical and electrical specifications and requirements have been met. Revised Risk Analysis As part of the incremental commitment model, the risk analysis was iterated and revised as the product development matured. FMEAs were updated for three product areas, which were safety-critical risks associated with the user interface, the mechanical and electrical subsystems, and the product manufacturing process. Explicit analysis of the business risks and the costs of continued financial commitment to the funding of development were also incremented and reviewed at various management and phase reviews. Product Introduction Product introduction planning included data collection from initial users to better understand remaining usage issues that can be uncovered only during prolonged usage in realistic clinical conditions. The many cycles of laboratory-based usability testing typically are never detailed enough or long enough to uncover all usability problems. The plan is to use the company complaint handling and resolution process (e.g., corrective action and preventive action) to address use issues if they arise after product introduction. Life-Cycle Planning The product was developed as a platform for the next generation of infusion pump products. As such, there will be continued business risk assessment during the life cycle of this first product on the new platform as well as on subsequent products and feature extensions. Summary of Design Issues and Methods Used This infusion pump incorporated the best practices of user-centered design in order to address the serious user interface deficiencies of previous infusion pumps. The development process took excellent advantage of the

OCR for page 91
Human-System Integration in the System Development Process: A New Look detailed amount of data that is derived from an integrated HSI approach and used it to improve and optimize the safety and usability of the design. Because of these efforts, the Symbiq™ IV Pump won the 2006 Human Factors and Ergonomics Society award for best new product design from the product design technical group. This case study also illustrates and incorporates the central themes of this report: Human-system integration must be an integral part of systems engineering. Begin HSI contributions to development early and continue them throughout the development life cycle. Adopt a risk-driven approach to determining needs for HSI activity (multiple applications of risk management to both business and safety risks). Tailor methods to time and budget constraints (scalability). Ensure communication among stakeholders of HSI outputs (shared representations). Design to accommodate changing conditions and requirements in the workplace (the use of iterative design and the incremental commitment model). This case study also demonstrates the five key principles that are integral parts of the incremental commitment model of development: (1) stakeholder satisficing, (2) incremental growth of system definition and stakeholder commitment, (3) iterative system development, (4) concurrent system definition and development, and (5) risk management—risk-driven activity levels.

OCR for page 91
Human-System Integration in the System Development Process: A New Look This page intentionally left blank.