Risk management is one of the five key principles underpinning the incremental commitment model of system development because understanding risks and managing them effectively are paramount to effective program execution. At a high level, the concept of risk encompasses subjective or objective determination of an event’s likelihood of occurrence and the detrimental impact of that event’s occurrence. Typically, program execution risk is grouped into three primary categories: (1) technical (i.e., a product’s ability to meet technical requirements), (2) cost (i.e., executing the program within the contracted budget), and (3) schedule (i.e., executing the program within the contracted duration). Risk can be defined as the product of estimated probability of each occurrence and the level of undesirable contingent consequences added across the set of events and consequences under consideration. Essential to the concept of risk is that the probability of occurrence must fall between 0 (i.e., total success—risk will never be realized) and 1 (i.e., total failure). In opposition to the concept of risk, but maintaining the same underlying principles and practices as risk management, is opportunity and opportunity management, wherein an opportunity’s consequence has a positive impact on technical, cost, or schedule program variables.
Human-system integration (HSI) analyses to identify risks are typically conducted at two levels: the finer grain relating to aspects of system design (e.g., safety, product usability), and the coarser grain contributing to overarching program risk management, which is the focus of this chapter. To explain how human-system integration fits into program risk-management efforts, an example methodology is presented; however, it should be noted
that other approaches may also be utilized. In addition, although a Department of Defense (DoD) program is the context for discussion, we note that the risk-management concepts and HSI interwoven thread have direct applicability to commercial development as well. Finally, although the chapter focuses on program risk management, the concepts can also be applied to managing opportunities that arise during program execution.
Before delving into HSI contributions to risk management, it is important to understand the DoD acquisition program context in which these activities are carried out and must be integrated to be effective in achieving system performance goals. Risk, issue, and opportunity management is a concept grounded in a continuous, forward-looking, structured, informative approach that is planned early in the life cycle and aggressively executed. This affords an organized, comprehensive, and iterative means for identifying and assessing the risks and handling options required to ensure technical, schedule, and cost aspects are appropriately balanced and accounted for.
An important aspect of the DoD risk-management approach is the concept of cost as an independent variable (CAIV) (U.S. Department of Defense, 2003b). In this concept, the highest priority in executing an acquisition program is reduction of procurement and in-service costs balanced by maintaining a high level of system performance for the user. In essence, CAIV entails establishing aggressive cost objectives, bound by maximum acceptable risk, so that if costs are too great and viable opportunities exist to reduce them, then the user and developer may compromise system performance requirements to meet cost objections. The important aspect to glean is that, as HSI risks are identified, their subsequent prioritization must be presented in a manner that shows a pragmatic and balanced understanding of the risk/opportunity’s likelihood and consequence traded against acquisition and in-service cost impacts. In essence, the HSI field must be able to concretely demonstrate the cost-benefit trade-offs for the technical, cost, and schedule modifications being proposed (Sager and Grier, 2005). Figure 4-1 depicts an overview of a representative program risk-management process. In each step, the HSI practitioner contributes valuable information enabling development and sustainment of a system that meets CAIV objectives and user needs. These steps should be undertaken with a holistic system view covering hardware, software, the human element, and related systems.
We have chosen to describe program risk management as applied to DoD acquisition programs because of its applicability to development of highly complex systems and systems of systems. The methods can also be tailored and scaled for less complex military or commercial projects.
In addition to the CAIV concept, there are additional program management practices that should be considered to effectively manage HSI
risks and opportunities during program execution. These aspects are not covered in detail in this report, but their importance and relationship to risk management and human-system integration are summarized here. The first practice relates to the business proposal on which the program’s execution is based. The business offer is critical because in this process estimates and assumptions are formulated and agreed on by all stakeholders: they cover the customer’s requirements and value proposition, measures of compliance with technical requirements (technical performance measures), schedule milestones, and requisite baseline program resources. If HSI specialists are not involved in the business offer process, their perspective and knowledge are not accounted for in formulating the program baselines, often making it necessary to use management reserve or negotiated requirements relief to resolve technical risks and issues that may have been otherwise accounted for during proposal activities.
The second important program management practice is creating a program organization—in particular, a product-based organization with clear team charters and integration teams at appropriate organizational levels. From a risk management perspective, understanding the organizational breakdown is important because it is a source of program execution risk
and may hinder proactive risk management if not done properly. From the point of view of human-system integration, understanding where the team resides in the organization is critical because it influences their ability to be effective. In addition, understanding the organizational composition affords the HSI team an opportunity to determine whether they should be represented in integration teams or in other organizational components to create the multidisciplinary skill set that is required to meet that component’s charter.
The final program management practice of interest is a culture of openness in which “help needed” (that flows up the organizational structure) and independent reviews are viewed as positive elements. From a risk-management perspective, help needed and independent reviews are important because they encourage early identification and timely mitigation of risks and issues.
The remainder of this chapter focuses on managing risks; however, it should be noted that considering opportunities is just as important and can be managed similarly to risks. In many cases, the HSI practitioner is well suited to identify and exploit opportunities that yield program execution benefits.
IDENTIFYING AND ANALYZING RISK
Risk identification is the problem definition stage at which risks are identified and quantified in terms of the likelihood of occurrence and detrimental consequences, forming the basis for most risk management actions (Hall, 1998; Boehm, 1991; U.S. Department of Defense, 2003b). Assessments provide insight into the likelihood of achieving desired outcomes in terms of program cost, schedule, and technical objectives and in-service system performance requirements. The risk identification component entails screening candidate risks to ensure validity, deletion of duplicates, clear statement of the risk, and creation of records used to summarize and track risks throughout the program life cycle. Due to the heterogeneity and nuances of program objectives, program life-cycle stages, and skill areas contributing to risk identification, detailed standard approaches are generally not promoted; however, some high-level steps are universal. The steps presented in Figure 4-2 are intended to provide insight into risk identification focus areas.
To effectively identify risks, several qualities of the effort must be in place before focusing on external and internal sources of risk. In the committee’s collective experience, the qualities listed below have been found to be indicators of a healthy and dedicated risk-management commitment:
Involvement of all stakeholders and clear communication of program objectives.
Continual iteration of risk identification until program objectives are met.
Utilizing nonadvocate technical experts to assist with risk identification.
Program management culture encouraging risk identification and recording.
Risk-management processes that afford consistent documentation.
As the HSI practitioner begins risk identification efforts, he or she should be aware that acquisition programs tend to have numerous, often interrelated, risks at all program levels and life-cycle stages that are not always obvious or understood by all skill areas. This can at times mask HSI risks or make it difficult to tease out the HSI component in a larger multi-attribute risk. However, lessons learned from DoD acquisition programs have revealed program aspects containing HSI risk sources that tend to be more critical and should receive heightened attention (U.S. Department of Defense, 2003b). These aspects include
system performance (technical) requirements and characteristics that do not satisfy user requirements.
mismatch of user manpower or skill profiles with system design solutions.
user interface problems (software and hardware).
proper mix (experience, skills) of HSI personnel assigned to the program management office or contractor team (or both); frequent rotation of HSI personnel.
Listed below are additional program risk sources that HSI practitioners should be concerned about that may reside at the procuring agency’s program management office, contractor, and/or suppliers.
Missing, incomplete, insular, out of scope, or uncertain HSI activity plans, schedules, cost estimates, resources, and processes; HSI team budget/staff size.
Program management perspective on the HSI discipline and HSI technical risks.
Inclusion in risk review boards.
Working relationships with stakeholders; access to users.
Clarity and validation of HSI requirements and ability to verify compliance; appropriateness of cited standards.
Supplier HSI processes and requirements.
When reviewing these HSI risk sources, one or more of the identification methods enumerated below can be used to populate a candidate risk list. These methods may be undertaken individually by the HSI practitioner, but they are most effective when multiple methods are executed in the setting of an integrated product team (IPT).
Risk-identification checklists and likelihood/consequence tables built on collective knowledge and lessons learned regarding processes, specific technologies, and design phases.
Earned value management metrics analyzed for plan deviations.
Product operations and program processes analyzed for potential failures/anomalies.
Program statement of work or work breakdown structure analyzed for risk sources in managed items (resources, deliverables, or events).
Once a candidate’s HSI risk has been identified, it is assigned to an HSI team member who becomes the “risk owner” and is responsible for managing it. In some instances, the interrelated nature of HSI risks may require that it is co-owned by more than the HSI team. Candidate risks are subsequently recorded in a risk-management database.
The next step entails establishing a risk review board or risk-management IPT for screening and validating the candidate risk list to avoid duplication,
gaps, and inaccuracies. It is essential that HSI practitioners are actively involved in the risk review board or IPT that serves this purpose—at the very least, a representative voice that understands the HSI perspective. Without HSI involvement in this screening and validation step, it is possible that critical HSI risks may be marginalized or ignored. The completeness of the risk identification activity, as well as the candidate risk itself, should be validated using the following criteria:
Each organization of the program has contributed to risk identification; no gaps.
Candidate risks are deemed significant by the owner and those affected by it.
Risks, their source, and consequence are well defined and consistent with known data. (This can be challenging for human-system integration given the skill’s subjective nuances).
Risk consequences describe the unfavorable effects on the program objectives.
Upon completing the candidate risk validation process, HSI risks are included in a program risk watch list that serves as the deliverable for risk identification activities.
Risk analysis determines the levels of likelihood, consequence, and overall risk for each candidate risk, then categorizes (technical, schedule, or cost) and prioritizes the risks (low, moderate, or high) to select risk-handling actions (Garvey and Lansdowne, 1998; Hall, 1998). The overarching intent is to zero-in on areas of high and moderate HSI risk for which risk-handling actions can have the greatest impact.
Figure 4-3 provides a synopsis of the risk analysis steps in which human-system integration plays an important role championing an end-user’s perspective on risk likelihood and consequence. When progressing through these steps, risk analysis is first conducted on individual risks followed by analysis of their effect on managed items (deliverables or scheduled events) and the entire program to prioritize risk-handling strategies. It is important that HSI engineers are part of the risk review board performing the analysis to describe the risks and lobby for appropriate resources to manage HSI risks effectively, as well as to coordinate with others where a HSI thread is woven into other risks.
Determine the Risk Analysis Method
As noted in Figure 4-3, the first step in risk analysis is to determine the method of analysis for “measuring” the risk. Methods of risk analyses range from simple, qualitative approaches to quantitative methods to hybrids combining qualitative and quantitative techniques. Typically, qualitative and hybrid methods are utilized, whereas complex quantitative methods are applied to specialized, in-depth risk analyses. This is particularly true for HSI risks in which subjective data predominates, thereby necessitating a qualitative approach, as explained below. When deciding which method to employ, it is important to consider the type of likelihood and consequence values that will be generated—ordinal (qualitative, relative values, not an actual numerical difference) or ratio (quantitative, calibrated to a known scale with a fixed zero point). This is critical because mathematical operations performed on results from uncalibrated ordinal scales are meaningless and yield erroneous risk ratings (Conrow, 1995). Quantitative analyses should use only values from ratio scales having calibrated, measurable values.
Assess Likelihood and Consequence Levels
The second step in risk analysis is to determine the likelihood and consequence levels for each identified risk. For HSI risks, assessing their likelihood and consequence is particularly challenging for a number of reasons (subjective nature of the risk, lack of previous examples or analogies to reference, individual differences of end-users, etc.) and is an area in which HSI specialists may lack experience. Aside from the risk that the HSI team may not adequately accomplish the risk analysis on time and within budget,
most HSI risks involve the potential failure of the system’s design specifications or user interface specifications to meet the performance requirements of the end users. Quantifying these risks is difficult and requires the participation of the design team as well as the HSI specialist. In addition, it is paramount that method, tools, and criteria are consistently applied to create an equitable basis for comparison. Chapter 8 discusses in more detail aspects of assessing the likelihood and consequence of a risk using failure modes and effects analysis and fault tree analysis.
Determine the Risk Level
Next, the risk level (i.e., a holistic “measurement” of risk) is determined based on combining the assessed likelihood and consequence levels. Risk levels are described as high, moderate, or low and are often portrayed in risk grids (likelihood and consequence are the axes) with stoplight colors (red represents high risks, yellow for moderate risks, and green for low risks).
Assess Program Impacts
Thus far, risk owners have assessed individual risk levels, but individual risk impacts on the program need to be assessed. The program impact of the individual risks is determined by a risk review board that examines the relationships of individual risks to one another or a higher level program item (managed items, deliverables, or scheduled events) to perform a collective risk assessment. The risk review board also takes into account the dependencies of higher level program items to ensure a holistic program-level risk assessment. HSI risks are often marginalized in this process due to the specialty engineering nature of human-system integration, unless an HSI representative is present to elucidate the issue. If an HSI representative is not a part of the risk review board, the result may be inadequate resources being allocated to HSI risks or disregard of their potential impact when examining higher order dependencies because it is masked by other risks.
The holistic, integrated perspective is important because individual risk analysis treats risks as mutually exclusive and independent of each other; typically this is not the case for HSI risks in complex system development. Individual risks frequently have common elements that make them interdependent despite their independent occurrence; this is especially true for HSI risks. As a result, the potential relations of individual risks to higher level program items may have a collective effect of increasing the risk level of that higher level program item—hence the importance of collective risk analysis in which the first step in assessing program impacts is to determine the relationship of individual risks to higher level program items
and then assessing their collective effect on the risk level of a higher level program item.
The next step in assessing program impacts is an integrated risk analysis to determine risk interdependencies that may amplify one or more of the interrelated risks. In this analysis it is important for HSI practitioners to explain how disregarding HSI risks, particularly those masked by other risks or marginalized due to underappreciation of the HSI risk, may have implications later requiring rework to resolve or additional risk management activities. This need for clearly explaining how HSI risks may significantly contribute to the program’s risk “critical path” is particularly crucial for software development, in which a “we can fix that later, it’s software” mind set, or the fallacy of reusing software to improve program cost and schedule metrics, at times prevails.
Once the risk review board has completed the collective and integrated risk analyses, the program risk critical paths may be determined. The risk-critical path of a program can be identified only by determining the predecessor and successor relationships and success dependencies of managed items and conducting the integrated risk analysis. All individual risks in the risk-critical path should be given the highest priorities in the program for risk handling.
Individual risks should be prioritized for their handling options by individual risk levels, effect on higher level program items, relationship to the program’s risk-critical path, and urgency based on time to when the risk may occur and/or customer priorities and preferences. Prioritization is done by adjusting the risk priorities up or down from the risk level based on the risk prioritization factors. If the risk levels and prioritization factors all have ratio scales with absolute values, the risk level may be a calculated function of these quantities. Otherwise, prioritization must be done by manually assessing the effect of the prioritization factors on risk level.
Successful completion of the risk analysis step culminates in the generation of individual risk likelihood and consequence assessments, risk levels (low, moderate, high) for individual risks and higher level program items, an identified program risk-critical path, and prioritized individual risks to steer subsequent risk-handling activities. Given the extent of influence that risk analysis has on resource allocation, it is important that HSI practitioners are actively involved in this component of risk management. The HSI team is typically working with suboptimal resources, and it is imperative they are not further diminished because the issues associated with HSI risks are not fully understood or appreciated by decision makers.
HANDLING OPTIONS ASSESSMENT
Assessing risk-handling options includes evaluating each significant risk to determine the appropriate handling strategy (Hall, 1998; Boehm, 1991). Typically, only moderately or highly prioritized risks require a risk-handling strategy, whereas low-priority risks are monitored and periodically reassessed. The most appropriate handling strategy for each risk is selected based on its urgency, level of the risk, resources and schedule constraints, and customer satisfaction expectations. The risk-handling options to choose from include avoiding the risk, transferring the risk, assuming the risk, and mitigating the risk; these options are depicted in Figure 4-4. The HSI practitioner is well suited to offer a range of risk-handling options given the flexibility in HSI techniques and bringing a mind set of the impact of those options on user and system performance when the system is operational.
Avoiding the Risk
Avoiding the risk is an option that is usually viable only in the earliest phases of a program, when concept development permits redefining plans and approaches with minimum impact. A decision to avoid the risk and
the method of avoidance typically involves not only the team responsible for the risk, but also program management and the customer. The limited window of opportunity for avoiding the risk and the need for extensive coordination is illustrated by its associated options, which include (1) choosing an alternative approach with a lower risk, (2) deleting specific requirements, (3) changing specific requirements, (4) changing the overall technical solution, (5) modifying the program schedule, and (6) modifying the funding level or funding profile.
An additional, yet untraditional, perspective is that HSI risks can often be avoided by further defining requirements to remove the nebulous (“soft”) attributes (in Chapter 7 see the section on usability requirements methods). Traditionally, HSI requirements are vaguely defined and hard for the contractor to prove compliance; jointly refining them creates clarity in the development process. Before deciding on a risk avoidance handling strategy, options of risk transfer, risk acceptance, and risk mitigation should be evaluated.
Transferring the Risk
Transfer is a strategy to shift the risk elsewhere (e.g., another team, supplier, customer, or requirement) so that overall program risk is optimized and risk ownership is assigned to the party most capable of reducing or accepting the risk. Options for transferring the risk include
reallocating requirements to reduce overall program risk.
developing a research and development (R&D) project that takes the risk with it and substitutes a lower risk contractual alternative with an option to evaluate the R&D results for later product improvement.
pushing the risk to the next program phase and addressing it later if the current program is successful.
Deferring the risk can have dire consequences from an HSI perspective. Delaying the handling of an HSI risk is often the option chosen because decision makers do not fully appreciate the importance of managing HSI risks proactively and early in the program life cycle. This is particularly true in software development and hardware design, in which ergonomic (anthropometry and biomechanics) and human factors (usability) aspects of the system design may necessitate costly fixes because the HSI risks were not given appropriate attention at the proper program life-cycle phase, or it was deemed that transferring the risk by pushing it to a later program phase would be more effective. Having assessed risk avoidance and transfer, the options of risk acceptance and risk mitigation should be evaluated.
Assuming the Risk
Risk assumption is a deliberate decision to accept a known risk, and it should be based on cost-benefit analysis, showing that it is more beneficial to assume the risk than it is to choose any of the other risk-handling options. This decision is made only when all relevant facts have been presented to the decision makers. If assuming the risk is the option chosen, then the risk owner needs to monitor the risk continuously for change, reassess it as appropriate, and take action as needed. As long as the risk’s parameters are within predetermined acceptable ranges, no action is needed. If they fall outside these predetermined acceptable ranges, action is triggered, and a decision must be made for how to bring the risk level back into the acceptable range. At this point, a reassessment of the possibilities of avoiding, transferring, or assuming the risk also is recommended.
Risk assumption is another often-used approach for handling HSI risks, primarily because the decision makers do not fully grasp the implications, or the seriousness of the risk does not surface in demonstrable manner until the system is operational. Human-in-the-loop evaluations of system prototypes and system build releases within the spiral life cycle are the most effective means for demonstrating when an HSI risk should not be assumed, but these activities require competing for highly utilized program resources. It is also crucial that the tolerable bounds for assuming an HSI risk are succinctly documented along with the impact of crossing them. After examining the alternative of assuming the risk, risk mitigation should still be evaluated, especially for risk items that exceed acceptable risk assumption parameters.
Mitigating the Risks
For items judged to be significant risks and avoiding, transferring, and assuming risks are not acceptable options, risk mitigation is chosen. Risk mitigation is a strategy for developing options and alternatives that lower or eliminate the risk by reducing its likelihood or consequences. This usually applies to technical risks (e.g., subsystem design), but it may also apply to schedule risks (e.g., a supplier who has a significant on-time delivery risk). For high-risk items, a fallback plan needs to be identified to cover the possibility of mitigation plans failing. The purpose of the fallback plan is to allow the program to continue while still meeting most of the program objectives.
Human-system integration has an amalgam of tools and techniques for mitigating risk that are integrated into everyday practices—for example, software prototyping, anthropometric modeling, usability evaluations, and cognitive workload modeling. By effectively performing the role of an HSI
engineer, the practitioner is continuously mitigating risk in the design and development of the system, with a focus on the risks that impact the operational effectiveness of both the system and the user.
EXECUTING RISK MITIGATION
The risk mitigation option, when chosen, establishes detailed plans for the risks that require them, covering required resources, schedules, tasks, success criteria, expected resulting risk level for each successfully completed task, and plan approval. In addition, this step creates fallback plans for all high risks, along with decision gates and criteria for their implementation (U.S. Department of Defense, 2003b). Human-system integration will execute mitigation strategies for its own risks; however, it is also important that human-system integration is involved in defining the collateral impacts of its efforts and understanding the impacts on other skill areas’ risk mitigation activities, as well as theirs on human-system integration. Figure 4-5 is a representation of the risk mitigation activity.
Develop a Plan
Effective risk mitigation entails reduction of the risk occurrence likelihood, its consequences if it occurs, or both to complete the effort. Subse-
quently, clearly defining task success criteria is imperative to measure the effectiveness of the risk mitigation plan and determining when to invoke a fallback plan. For human-system integration, defining the success criteria can be tricky due to the subjectivity inherent in a majority of the issues with which the discipline works. In essence, the HSI practitioner is trying to define criteria akin to usability goals with gradations indicating degrees of success and when a fallback plan should be implemented.
When creating a risk mitigation plan, an array of options exists, depending on the nature of the risk; HSI options may include
trade studies; parallel prototyping; early development HSI evaluation; early and extensive simulation.
system and/or task analysis; refining, modifying, or eliminating HSI requirements.
working with suppliers to implement solid HSI processes.
using an alternate design that already meets HSI requirements.
extending the schedule; increasing the budget.
When generating the risk mitigation plan, the items below should be explicitly and succinctly stated.
Task definitions with entry and exit criteria that denote starting and completion points (success criteria defining expected risk-level outcomes and subsequently removing the risk from the significant risk list). Planned start and stop dates with associated cost and required inputs.
Relationships between tasks in the risk mitigation plan and overarching program plan, as well as collateral impacts on required resources.
Means for tracking plan variance.
Identify Fallback Plans
Risks that are categorized as high should have fallback plans as part of the risk mitigation strategy. Fallback plans are requisite to ensure that an alternative approach is available to mitigate risks that have a significant likelihood of occurrence, severe consequences, or both.
Incorporate into Program Schedules
Approved risk mitigation plans need to be a formal part of the program schedule and be reflected in the program’s metrics to garner the requisite attention needed to accomplish planned mitigation tasks. Generally, “off the books” mitigation efforts suffer from lack of visibility, suboptimal coordination, and improper configuration management. It is extremely important
that HSI risk mitigation plans are integrated into the program scheduled because the discipline is often hampered by the perspective of some decision makers that human-system integration is a specialty engineering resource utilized for consultancy purposes. By operating off the books, the HSI team is further isolating itself from the decision makers who control resources.
Evaluate Success Accomplishments and Assess Reductions Achieved
Each mitigation activity has predefined success criteria associated with it. As mitigation tasks are accomplished, they must be evaluated to determine if the expected success criteria were met. In addition, the degree to which the predicted risk reduction was accomplished needs to be assessed. If not fully successful, the plan may need to be adjusted or fallback plan implemented to achieve the planned risk reduction.
Evaluate Remaining Plan Activities
Upon completion of each risk mitigation task, the remaining tasks in the plan should be evaluated to ensure the overall required reduction is still attainable via the defined plan. Decisions are made at this point whether to proceed with the plan as defined, modify it, or implement the fallback plan. As noted previously, any decisions to change the plan need to mandate proper coordination, approval, and incorporation into the program schedule.
Successful completion of the risk mitigation step culminates in detailed mitigation plans documented in team and program schedules, fallback plans, approved resources to execute the risk mitigation plan, and completion of the risk mitigation strategy resulting in the expected reduced level of risk.