3

The Potential of Operational Systems Engineering

Following presentations describing the major features of peacebuilding, two presenters provided examples of the insight that operational systems engineering can offer peacebuilding. Bill Rouse, Alexander Crombie Humphreys Chair in Economics of Engineering at Stevens Institute of Technology, provided an overview of systems thinking and methods built around a multilevel approach to systems modeling. As a starting point for identifying potential applications to peacebuilding, he discussed a computational model of health care delivery that has been used to conduct experiments supporting decision making in health delivery. Robert Ricigliano, director of the Institute of World Affairs at the University of Wisconsin–Milwaukee, demonstrated a relational model of social processes in South Sudan that has been used to identify leverage points where investment or innovation could have high impact. The two presentations set the stage for the subsequent examination of specific cases, described in Chapters 4 through 6.

QUANTITATIVE SYSTEMS ANALYSES

Systems thinking is not new or restricted to engineering, Rouse said. People have used it for more than a century, starting with the scientific management of Frederick Taylor in the late 1800s. Rouse defined systems thinking as “the process of understanding how things influence one another within a whole, and an approach to problem solving that views problems as parts of an over-



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 23
3 The Potential of Operational Systems Engineering F ollowing presentations describing the major features of peacebuild- ing, two presenters provided examples of the insight that operational systems engineering can offer peacebuilding. Bill Rouse, Alexander Crombie Humphreys Chair in Economics of Engineering at Stevens Institute of Technology, provided an overview of systems thinking and methods built around a multilevel approach to systems modeling. As a starting point for identifying potential applications to peacebuilding, he discussed a compu- tational model of health care delivery that has been used to conduct experi- ments supporting decision making in health delivery. Robert Ricigliano, director of the Institute of World Affairs at the University of Wisconsin–Mil- waukee, demonstrated a relational model of social processes in South Sudan that has been used to identify leverage points where investment or innova- tion could have high impact. The two presentations set the stage for the subsequent examination of specific cases, described in Chapters 4 through 6. Quantitative Systems Analyses Systems thinking is not new or restricted to engineering, Rouse said. People have used it for more than a century, starting with the scientific management of Frederick Taylor in the late 1800s. Rouse defined systems thinking as “the process of understanding how things influence one another within a whole, and an approach to problem solving that views problems as parts of an over- 23

OCR for page 23
24 HARNESSING OPERATIONAL SYSTEMS ENGINEERING all system.” Thus, the interdisciplinary field of systems engineering focuses on how complex undertakings should be designed, developed, and managed over their life cycle. Moreover, systems approaches emphasize causality and design in a holistic perspective and attempt to avoid the arbitrary boundar- ies that people often create to overcome complexity in conceptualizing an integrated whole. Initially, systems approaches were used to describe physi- cal systems. More recently, however, systems engineering has been applied to complex systems in biology, medicine, and education. These complex, human-centered systems have traditionally been the province of the social sciences and so were not, in a classic sense, “engineered.” Still, like physical systems, they can be modeled. To understand such complex systems, a widely used approach is to model the interactions between (1) individual workers’ activities, (2) work pro- cesses, (3) organizations, and (4) society. Using Table 3-1, Rouse described the issues that can be understood at each layer in such a model and gave examples of systems modeling approaches or tools that could be used to analyze these issues. For example, techniques to model the macro supply TABLE 3-1  Social Systems Can Be Conceptualized as Hierarchical Networks with Multiple Levels, Issues, and Approaches Level Issues Modeling Approaches Society GDP, Supply/Demand, Policy Macroeconomic Economic Cycles System Dynamics Intra-Firm Relations, Competition Network Models Organizations Profit Maximization Microeconomic Competition Game Theory Investment Discounted Cash Flow, Options Processes People, Material Flow Discrete-Event Models Process Efficiency Learning Models Workflow Network Models People Consumer Behavior Agent-Based Models Risk Aversion Utility Models Perception Progression Markov, Bayes Models SOURCE: Rouse workshop presentation.

OCR for page 23
THE POTENTIAL OF OPERATIONAL SYSTEMS ENGINEERING 25 and demand characteristics of society as a whole include macroeconom- ics, system dynamics, and various network models. At the next level down, game theory and option theory can be use to understand how organizations generate competitive advantage, profit, and returns on investment. Organi- zations can be decomposed into networks of individual work processes that can be described using event models and again network models. Finally, at the lowest level, the behavior of people can be modeled using agent-based models or utility models to capture the incentives that govern the activities of individual workers. Implicit in this table is a major question that must be addressed as the model is being developed. What is the appropriate level of aggregation? In modeling peacebuilding, for example, should each person be simulated, or each group? The answer depends very much on what issue the analyst wishes to address. Comparable questions apply at each level of the system, from individuals to the domain ecosystem. In addition, models can focus on different aspects of a system, from economic cycles or policies at the societal level to consumer behaviors and risk aversion at the individual level, with different approaches answering different questions but having very different data needs. Rouse emphasized that, for a given problem, multiple models based on different modeling approaches may be an effective way to explore different aspects of the problem, as opposed to developing a single model that addresses all questions and information needs. Rouse demonstrated the power of this approach with a model evaluating the Emory University Prevention and Wellness Program to prevent diabetes and heart disease. The model operates on the same four levels shown in Table 3-1. Society provides a set of rules, policies, incentives, and cultural structures that defines the context within which Emory University’s hospital operates. The hospital houses many different business processes, including the Prevention and Wellness Program, each of which may affect the other. Interactions between individual clinicians and patients occurs at the lowest level. As illustrated in the account in Box 3-1, the advantage of this approach is that “what if ” experiments can be conducted on computers, without incur- ring the expense and real risk of actually implementing changes and then measuring their effects. Such models are especially useful for planning policy and evaluating options. The point of such exercises is to generate insight rather than make pre- dictions. Similarly, in peacebuilding, a probability of conflict is not a very useful number, but a compilation of all the things that could go wrong, along with a list of the factors that could contribute to their emergence, could be

OCR for page 23
26 HARNESSING OPERATIONAL SYSTEMS ENGINEERING Box 3-1 Emory University Predictive Health Institute Prevention and Wellness Program Simulation Emory University and its Predictive Health Institute (PHI) modeled a prevention and wellness program that would prevent diabetes and heart disease, including the development of a Simulation Dashboard to simulate alternate outcomes. The program was evaluated with 700 employees to explore whether it should be extended to all 20,000 employees. Almost immediately, the simulation revealed that the system was not sustainable because of increasing costs. To address this prob- lem, the project brought together all of the stakeholders for a formal design exercise in which they could suggest a change and test it us- ing the dashboard. Stakeholders could vary their assumptions about the future, the data they were using, the number of people hired and terminated, the retirement age, and other factors to probe the effects of changes. Unlike a peacebuilding intervention, in which success can not easily be reduced to a single metric, the model used return on investment (ROI) as the principal measure for testing different health care strategies. The project demonstrated that investments in health and wellness for employees could produce a 7 percent ROI. However, this rate of return did not come from scaling up Emory’s current prevention and wellness program, which would produce a substantial negative rate of return. Instead, the positive rate of return came about by focusing on the approximately 10 percent of people who were at highest risk. Based on these results, Emory has moved to reorganize its health care delivery system. The modeling for Emory extended only through age 65, but when extended to age 80 the ROI rose from 7 percent to 30 percent. On a purely economic basis, it does not make sense for Emory to invest in the health of its employees in such a way that they will be healthier after they retire, but it would be of great benefit to Medicare. By in- centivizing employers to produce healthier employees, the Center for Medicare and Medicaid Services (CMS) could substantially reduce the costs of Medicare, Rouse pointed out, but today the two systems are separate and would require a dialogue to link those interests.

OCR for page 23
THE POTENTIAL OF OPERATIONAL SYSTEMS ENGINEERING 27 very useful in improving intervention strategies for peacebuilding. Lead- ing indicators that are predictive of events could also be identified, though human judgment is essential in selecting such indicators. Simulations also can be used to explore potential mitigations. Again, the product would be insights rather than predictions, but they could be trans- lated into well-founded instructions for people on the ground to pay atten- tion to critical factors and to avoid actions that could exacerbate conflict. Simulations could help answer questions such as the following: What can go wrong? How likely are these scenarios? What factors might contribute to their emergence? What are leading indicators of these factors being in play? How effective are potential mitigations of these factors? In the development and use of systems models, Rouse stressed the importance of engaging stakeholders so that they understand what is hap- pening in the model, rather than thinking of it as a black box that produces numbers. With understanding, stakeholders start to take ownership of the results. In that sense, as General Eisenhower observed, planning is far more important than the plans themselves. People begin to understand interde- pendencies rather than assuming that factors are independent, and, in a multi-stakeholder conversation, they begin to understand the perspectives of other stakeholders, which can be as important as the results. Qualitative Systems Analyses In contrast to the computational model described by Rouse, Robert Ricigliano demonstrated a relational model that probes the interactions among components of a system. It, too, produces insights that can be used to aim, implement, or redirect policies more effectively, rather than yielding specific predictions. Part of the Joint Irregular Warfare Analytic Baseline (JIWAB), the model was produced to explore ways of using systems maps to improve joint inter- agency planning. The goal was not to actually produce a plan for intervention but instead to evaluate how systems methods could be used in the inter- agency planning process. The systems map shown in Figure 3-1 and used in the JIWAB sessions broadly describes how security, government capacity, and development issues affect the legitimacy and credibility of the Government

OCR for page 23
28 HARNESSING OPERATIONAL SYSTEMS ENGINEERING Demographic shiŌs Returnees + South Sudan ExpectaƟons + + + gap + Ethnic Communal + divisions conflict + + + R11 Conflict: + + GoSS-local R4 Expect. of Comm. self- reliance - + govt + govt R13 + + + R3 - + R10 - Patronage Local - PoliƟcal Sufficiency, equity poliƟcs + + sec. + + exclusion + - of resource dist. - - + + - Donor + + support + Local conflict Unbalanced Military + Insecurity resoluƟon B9 R1 econ. dev. spending + + + + R2 + + - + Local Church + R12 acƟvity - - - GoSS - legiƟmacy, DDR - capacity - + Splits in - + Economic + + B5 viability SPLA + + R6 + Ability of external + actors to extract Conflict: + Ext. support for resources R14 tradiƟonal and GoSS capacity modern systems + building B7 + + R8 + + + Foreign + Copyright © 2012 by Robert Ricigliano. All rights reserved. investment FIGURE 3-1 A systems analysis of South Sudan centers on the legitimacy and capacity of the Government of South Sudan (GoSS). Each arc describes how change in a node at the tail of the arc will affect the node at the arrow head. Plus signs indicate increases and minus signs indicate decreases. For example, the arc joining GoSS legitimacy and capacity to unbalance economic development in R1 should be read, “Decreases in GoSS legitimacy and capacity tend to increase imbalance in economic development.” SOURCE: Ricigliano workshop presentation. Note: DDR is Disarmament, Demobilization, and Reintegration; SPLA is the Sudan Peo- ple’s Liberation Army. of South Sudan. Ricigliano emphasized, however, that the data-gathering process that produced this map lacked the resources (time, quantitative data, and personnel) necessary to produce a reliable policy tool. Despite that, the map was sufficient to test how systems might be used in interagency planning processes, and he thought it would be sufficient to enhance the potential of systems in planning multi-stakeholder interventions for peacebuilding. At the JIWAB workshop, planning teams from the Department of Defense, the State Department, and the US Agency of International Devel- opment (USAID) were given the systems map and asked to describe what

OCR for page 23
THE POTENTIAL OF OPERATIONAL SYSTEMS ENGINEERING 29 their organization would do in South Sudan to improve the legitimacy and credibility of the government. Working separately, each group discussed a different portion of the system based on their own mission. Initially, Defense focused on security sector reform; State on enhancing government capabil- ity; and USAID on economic development and donor relations. With time, however, the three teams independently focused on the same portion of the map. To understand why, Ricigliano first described in some depth the con- struction of the map. A systems map is not just a prettier way to show relationships, said Ricigliano. It is a visualization technique for building a richer, shared narra- tive that leads to more effective peacebuilding. The systems analysis shown in Figure 3-1 evaluates what factors were affecting the legitimacy and capacity of the Government of South Sudan (GoSS)—from the perspective of the interagency teams, the primary reason for intervening. Fourteen unique loops start and end with the central variable, “GoSS legitimacy, capacity.” These loops are divided into 11 marked R for reinforcing and 3 marked B for balancing. Ricigliano explained that positive feedback that strengthens each variable in the loop characterizes a reinforcing loop. For example, around loop R1, lower government legitimacy causes greater imbalance in economic development. These increasing imbalances lead to greater inequity in resource allocations within South Sudan, which causes greater political exclusion and further undermines the government’s legitimacy. A reinforc- ing loop creates a cycle of change which, depending on one’s perspective, can be good—a virtuous cycle—or bad—a vicious cycle. A balancing loop, on the other hand, contains negative feedback that resists change and stabilizes each variable within the loop. For example, in B6 a decline in government legitimacy encourages external donors to invest in capacity-building projects that improve government legitimacy. Analysis of these loops can reveal multiple feedback mechanisms that may produce unexpected results—positive, negative, or both. In JIWAB workshop, Ricigliano continued, although each group initially focused on their area of competence, after about forty-five minutes of discussion, all three groups were focusing on the portion of the systems map shown in Figure 3-2. Compared to the rest of the map, this subsystem interested the three teams because it contained two of three balancing loops in the sys- tem and because it related foreign investment (both public and private) to changes in government legitimacy. Most interestingly, this subsystem linked foreign investment to the potential for increased local conflict between tra- ditional and modern systems.

OCR for page 23
30 HARNESSING OPERATIONAL SYSTEMS ENGINEERING GoSS - legiƟmacy, capacity - - + B5 + + R6 + Ability of external actors to extract Conflict: tradiƟonal + Ext. support for resources and modern GoSS capacity + B7 + systems building + R8 + + + + Foreign investment FIGURE 3-2­  A subsystem shows how external support for Government of South Sudan Copyright © 2012 by Robert Ricigliano. All rights reserved. (GoSS) capacity building and foreign investment can both foster government legitimacy and exacerbate conflict between the traditional and modern elements in South Sudanese society. SOURCE: Ricigliano workshop presentation. Looking a little more closely at Figure 3-2, Ricigliano pointed to loops B5 and R6 as central to the question of how external agencies should make investments. Loop B5 shows how decreases in the legitimacy and capacity of the government of South Sudan tend to result in increased external support from international organizations. This support rebuilds the government’s capacity by funding new programs to deliver services to the citizenry. At the same time, however, loop R6 shows how external support may also increase conflict between traditional and modern systems. If newly funded govern- ment agents now have the vehicles and gasoline to go to rural villages and say, “We are here to tell you how to do things,” the tribal chiefs and other tra- ditional forces are likely to say, “Why do you think you can tell us anything?” Such conflict—over who has authority over such basic governing activities as the judiciary, security, and business—can further undermine the legitimacy of the government, leading to a reinforcing cycle of increased external invest- ment and heightened conflict between the modern state and traditional local authorities. Loops B7 and R8 describe the same effect for private investment. If the reinforcing loops (R6 and R8) are stronger than the balancing loops (B5 and B7) shown in Figure 3-2, as the experts in the JIWAB work-

OCR for page 23
THE POTENTIAL OF OPERATIONAL SYSTEMS ENGINEERING 31 shop judged them to be, external support and foreign investment will likely increase conflict between the traditional and modern elements of South Sudanese society. For that reason, the planners at the meeting eventually all focused on the conflict between traditional and modern systems, determin- ing that conflict could be avoided if the two systems could work together, Ricigliano said. Each of the three groups in the JIWAB workshop was able to make some contribution in its own domain to reducing the conflict between traditional and modern systems. Thus, this location on the systems map is a leverage point where investments or innovation could have high impact. If multiple efforts could be brought to bear here, positive changes would reinforce each other and be amplified rather than undermined by the system. This is a powerful way in which systems analysis can guide investments, said Ricigliano. A systems map can demonstrate the existence and location of an engine for change that can affect the entire system. Analysis produces a theory of change about implementing programs and about how those programs could affect the system as a whole. discussion The construction and use of systems maps can be complicated, as Ricigliano observed in the discussion session. In many cases, it is not immediately clear whether the signs at the start and end of an arc joining two nodes in a sys- tem map should be positive, indicating an increase, or negative, indicating a decrease, in the value of the node. Constructing and modifying maps there- fore requires ongoing data collection and analysis. Ricigliano thought that, although the peacebuilding community is more comfortable with gathering data than it has been in the past, it still is not at the level of, say, the health care community. Outcome data need to be incorporated into both initial formulations of the analysis and subsequent modifications of the systems map. If an inter- vention backfires, a positive sign on a map may turn to a negative sign. In general, when reality differs from the model predictions, modelers have an opportunity to learn why things turned out differently than expected. The slogan Ricigliano uses is “fail smart, learn, and adapt fast.” But he said the peacebuilding community is not currently set up to fail smart; it tends to bury mistakes, rather than learning from them by validating or invalidat- ing testable hypotheses to improve understanding over time based on both failure and success.

OCR for page 23
32 HARNESSING OPERATIONAL SYSTEMS ENGINEERING Maps also need to be contextualized, which means they will be different for different places, as Ricigliano noted in response to a question from Alfred Blumstein, Carnegie Mellon University. But dynamic systems will have pat- terns that, although varying based on context, will be replicated from place to place. The instability caused by foreign investments in local communi- ties shown in Figure 3-2 is likely one such pattern of behavior. Once these patterns are recognized, learning from conflict to conflict by incorporating previous experience and knowledge becomes possible. By identifying such patterns using systems maps, learning can become more institutionalized. Ricigliano acknowledged, in response to a question from Hrach Gregorian, Institute of World Affairs, that, as in other endeavors, the quality of the data used to build or modify a map affects the quality of the output. It is therefore important to have multidisciplinary teams working on a map to bring different kinds of data to the process of building and testing it. Much of the value of a map derives from the interactions that occur in building it. These interactions result in a collective understanding and sense making that can be captured in the relationships of a map. In the exercises in which Ricigliano has been involved, representatives of the Defense Depart- ment learned the perspectives of the State Department, USAID, the Justice Department, NGOs, and/or academic representatives. These interactions also help shift the emphasis from a linear problem-solution frame, to a sys- tems frame that is both process based and iterative. When this happens, par- ticipants know that they are working with a system and not just a checklist. Rouse also reiterated the importance of the process of building a model, again evoking Eisenhower’s statement that plans are useless but planning is indispensable. A systems approach requires bringing together people with very different backgrounds and expertise to talk with each other, often for the first time. “Getting the different stakeholders to understand [each other’s] concerns and issues is a big step.” Steven Robinson, University of Wisconsin–Madison, observed that a range of outcomes with probabilities attached to each is often more useful than a single outcome. For example, if a policy has a 60 percent chance of good outcomes but a 40 percent possibility of disaster, a policy with a 20 percent chance of good outcomes (and a significantly lower risk of disaster) may be a better choice. Simulations can be run multiple times with changes in assumptions to generate a range of outcomes. Rouse pointed out that models can also reveal which missing data ele- ments make the most difference. Thus they can be used to screen the sensi-

OCR for page 23
THE POTENTIAL OF OPERATIONAL SYSTEMS ENGINEERING 33 tivity of factors, identify which should be the highest priority for research, and direct data collection. As Ricigliano observed, a systems mapping approach can indicate the where and the what, but it does not help much with the how, which requires the use of quantitative tools. Such tools are needed to implement programs effectively. A decision about where to intervene might be best answered by understanding the problem as a system complete with powerful feedback loops that can produce unpredictable second-order effects. On other hand, once the where has been determined, the question of how to intervene is often a fairly straightforward activity that defines goals, programming, and methods to monitor and evaluate impact. Finally, Ricigliano noted that a map is not a quantitative model. The evolution from a map to a quantitative model, to produce the tools described by Rouse, forms a major part of the operational systems engineering disci- pline. Peacebuilding can benefit from qualitative applications such as systems mapping, but greater, substantial benefits can be anticipated as quantitative methods and tools are brought to bear.

OCR for page 23