System Modeling and Simulation
The committee believes that modeling of the air transportation system is best accomplished by a suite of system models.1 This chapter describes the committee’s understanding of system models and how they can be used to design and analyze evolutionary and revolutionary operational concepts, technologies, and other changes to the air transportation system. Given existing modeling and simulation capabilities (and ongoing research), the chapter also suggests what else should be done, especially by government, to provide the long-term systems modeling capability needed to analyze and select changes to the air transportation system.
UNDERSTANDING SYSTEM MODELS
In its simplest description, a suite of system models is a set of models, each self-contained and designed to produce meaningful outputs by itself, where outputs from some models are used as inputs to other models. As a general rule, the suite includes very detailed, high-fidelity, data-intensive, long-run-time models, usually involving individual components of the overall system, as well as higher-level, fast-time, abstract analytic models that, at the highest level, seek to represent how the entire U.S. air transportation system functions and how that functioning impacts the economic vitality of the nation.
Fundamental to the use of a suite of system models is the recognition that it is not possible to capture all of the important variables within a single large model, nor do the models in the suite have to be directly connected or operate simultaneously. Moreover, and of equal importance, outputs from the more detailed models often provide insights into the causes of congestion in the air transportation system and the relevance of potential actions aimed at addressing the problem. If these outputs were simply passed along to a high-level analytic model, such insights might well be lost. A suite of systems models should include a variety of models, some simpler, cheaper, easier, and quicker to run (when they can provide the needed output results with the required level of accuracy) and others more complex, more expensive, more difficult, and slower to run (when more detailed and/or more accurate results are needed and worth the extra effort and expense).
Computer-based simulations range from large-scale, fast-time simulations of the entire U.S. air transportation system to detailed human-in-the-loop simulations of specific aircraft or air traffic management systems. Simulations complement other analytical efforts by helping to (1) determine the feasibility of operational concepts, (2) establish parametric values required by models (for example, the increase in airport capacity that a new operational concept or technology will produce), and (3) validate model assumptions.
Also of importance is the strong interdependency of the many factors that enter into assessments of the air transportation system (see Figure 3-1). Thus, when constructing a suite of models, it is critical to capture logical dependencies and interdependencies and to make sure that the available models accurately simulate each of the areas depicted in Figure 3-1 or that efforts are under way to develop better models.
Detailed models support decisions on improving individual elements of the air transportation system. A suite of system models should be designed to assess the performance of system elements and the system as a whole. Incompatibilities that limit the ability of detailed models to support broader analyses should be avoided. High-level abstract models cannot include many of the variables that are in the more detailed models. There is, therefore, a difficulty associated with mapping the sensitivities of the results of the
more detailed models into the higher-level models, so that higher-level models often cannot reflect such sensitivities, even though the more detailed models may show them to be important for problem identification and resolution. When reporting the complete results of a suite of system models, it is important to include the results of the more detailed models where they are relevant to the solution. A description of four levels of models that could be included in a suite of system models appears in Appendix E.
A particular challenge in using a suite of system models for a sociotechnical system as complex as air transportation will be to capture the nonlinear dynamics of interactions among components, which makes it difficult to combine the results from different models. Additional research is needed to overcome this challenge.
ANALYSIS AND DESIGN TO IMPROVE AIR TRANSPORTATION SYSTEM PERFORMANCE
Improving the performance of the air transportation system requires a good understanding of the operation of the current system and the ability to model and analyze the performance of new operational concepts. The air transportation system, however, is a complex, human-centered system that involves multiple technologies, organizational structures, human behaviors, and competing economic entities. Modeling such a complex system is extremely difficult and requires the ability to model interdisciplinary systems and operational concepts (including cross-functional operational concepts) in terms of system performance (comfort, convenience, costs, and societal impacts) and the ability to satisfy the often-conflicting objectives of various stakeholders. Improving the ability to model and measure systemwide performance and assess risks associated with the development, deployment, and operation of complex new systems will help avoid historical precedents in which large new system projects have been cancelled prior to completion because of delays, cost increases, and/or the inability to meet design requirements.
As system complexity increases, it becomes more difficult to guard against dysfunctional interactions. Problems may arise from unanticipated interactions among automated subsystems and from unanticipated interactions among different organizations and parts of organizations. The ability to develop complex systems while effectively managing problems at the intersections between organizations, disci-
plines, and systems is growing more slowly than the ambition and willingness to attempt the development of such systems. Interdisciplinary research and a systems approach to research are needed. Business as usual, with research segregated by discipline, is insufficient and runs the risk of (1) optimizing short-term performance at the expense of long-term improvements or (2) suboptimizing system performance (i.e., optimizing the performance of a portion of the system in a way that fails to optimize or even degrades total system performance).
Demand and demand allocation do not remain static in the presence of changes to the air transportation system and the world. Passengers, airlines, manufacturers, business aviation, general aviation, and other involved parties will all adjust their behavior in response to increasing capacity and other changes. Models should be able to account for these changes. Overly simplistic modeling in the past missed the surge in demand following deregulation and the advent of hub-and-spoke operations, the emergence of low-price point-to-point carriers, and the rise in regional jets. Models should also be able to account for changes in the behavior of individual airports and regional airport systems, including the construction of new runways, gates, and other facilities. Models like the Total Airspace and Airport Modeler (TAAM) would be highly useful for evaluating some of the above factors, but additional models are also needed. Developing models capable of learning and adapting (e.g., agent-based modeling) is therefore very important. Models that account for mode splits (e.g., competition among air, rail, and automobile travel) in selected corridors will also play an increasingly important role.
In the specific case of the U.S. air transportation system, the above improvements are needed to produce suites of system models to do the following:
Characterize the nature of future demand as a function of possible changes to the price, quality, and availability of complementary and competitive transportation services; the overall performance and structure of the air transportation system; perceptions of aviation security; the personal habits and tastes of consumers; consumer income; and other factors internal and external to the air transportation system.
Identify potential shortfalls and needs in the performance of the U.S. air transportation system due to future growth in demand.
Determine the ability of new technologies, operational concepts, and procedures to meet future shortfalls.
Assess the systemwide aviation impacts of adapting evolutionary and/or revolutionary technologies and operational concepts and determine the overall benefits and costs of various alternatives.
Evolutionary and Revolutionary Approaches
The requirements for and capabilities of technologies, procedures, and operational concepts may be analyzed using two different approaches. The evolutionary approach starts with the operational concept and technologies used in the current air transportation system and determines the impact of incremental changes to them (see Figure 3-2, left-hand side). These changes are understood using technology models, computational human performance models, and human-in-the-loop simulations using increasing parametric abstraction and emulation. Impacts on the operation of the overall air transportation system are then determined in terms of metrics such as delays and flight times, using current demand and projected future demand. The benefits and costs are then evaluated for each of the evolutionary improvements.
In the revolutionary approach, the analysis starts with the top-level functional and performance requirements that are necessary for the system to meet various levels of future demand (see Figure 3-2, right-hand side). This approach may be viewed as revolutionary in the sense that totally new operational concepts, architectural approaches, system characteristics, and technological capabilities may be postulated without first assessing their feasibility or relationship to the existing system. Once the system is defined at the top level, parametrically connected layers of models may be used to allocate functional and performance requirements to system elements and human operators. Trade-off studies of alternative concepts and postulated technological capabilities can then be analyzed in terms of benefits, costs, and risks.
The two approaches differ in terms of the starting point of their analyses, not in the nature of the models used to implement them. In both approaches, system analyses must be iterated to account for interactions among various factors—internal and external to the air transportation system—that affect system performance and demand. Given that the functional description of the air transportation system can be likened in a simplistic way to a network of capacity-constrained links and nodes, it is clear that the detail and fidelity of models used in both the evolutionary and revolutionary approaches need to be similar at corresponding levels. In the revolutionary approach, the links and nodes should have specific capacities to meet potential levels of future demand (that is, capacities are assigned to the links and nodes in a way that achieves the desired level of overall network flow performance). New operational concepts and technologies are evaluated to see if they can achieve the requirement by using more detailed technology models, human performance models, and, ultimately, human-in-the-loop simulations. In the evolutionary approach, we modify existing operational concepts and technologies and then evaluate the extent to which the changes accommodate future demand using fast-time network flow models.
The approach used to analyze human behavior differs as
well in the evolutionary and revolutionary approaches. The evolutionary approach is baselined in current operations, where controllers, traffic flow managers, pilots, and dispatchers all have set roles and procedures. Proposed changes in technology (both hardware and software), roles, and procedures are then tested and refined using human-in-the-loop simulations, and performance improvement is measured. In the revolutionary approach, performance parameters and human roles are allocated to satisfy top-down functional requirements. Progressively more detailed simulations using human performance models can then be used to determine feasibility; ultimately, human-in-the-loop simulations can verify predictions, but their development requires detailed design of interfaces and operating procedures as well as the training of personnel on the revolutionized operational concept, operating procedures, and new technologies.
There exist or are under development today in industry and government a number of models and human-in-the-loop simulations that can fill roles at various levels in the overall evolutionary and revolutionary suite of system model constructs described above. These include, for example, the FAA Technical Center’s Integration and Interoperability Facility, NASA-Ames’s Virtual Airspace Modeling and Simulation Project, the approach used by Boeing in developing its Discrete-Event Simulation Interactive Development Environment (DESIDE), MITRE’s Detailed Policy Assessment Tool (DPAT) model, and the Logistics Management Institute network simulation model (LMI Net). Professional and expert-to-expert consultations among these efforts exist. What is missing is a uniform federal strategy for research investments, interagency coordination, and the use of the modeling and simulation results.
There is a significant difference in the detail, run times, and data requirements for the various models. On the one hand, models such as TAAM and facilities such as MITRE’s real-time air traffic management infrastructure laboratory provide very detailed emulations of the U.S. air transportation system at the expense of long run times and extensive data preparation. DPAT and LMI Net, on the other hand, are fast-time models that permit high-level evaluations of air transportation system performance by sacrificing the ability to produce detailed intermediate data. Research to develop improved intermediate models that close this gap could be of considerable benefit.
The stochastic nature of the air transportation system ensures that no one model gives a precise answer. Results on days that are supposedly equivalent from a scheduled airline viewpoint can differ wildly for a variety of reasons:
Changes in the sequence of actual takeoffs for a number of aircraft scheduled to depart within the same 15-minute window.
Changes in the number of military, business, and general aviation operations.
Changes in wind direction at one or more major airports that require a change in airport configuration.
As a result, key questions remain unanswered: What does it mean to validate a suite of system models? How should the validation be conducted? Who—that is, which entity—should certify the degree of validation?
When linking established models, the validation challenge becomes more significant. For example, connecting two validated models (in run time, or by using the output of one as input to the other) does not guarantee that their combined output is itself valid. In other words, establishing mechanisms for combining models is itself a modeling process that must detect and account for conflicts and gaps that may exist among the assumptions and capabilities of each component model.
Answering the above questions and developing widely accepted validation standards and processes will not be easy, even for an organization with the resources of the federal government. Areas of particular difficulty include predicting strategic investment decisions by industry and government, such as hub selection and location and airport construction projects, which (1) depend on a complex interplay of public and private individuals, organizations, and interests and (2) change the shape of the landscape upon which the rest of the air transportation system rests.
Federal research investments in a suite of system models relevant to the air transportation system need to be better coordinated to avoid unnecessary gaps and overlaps. Widely accepted criteria are needed to validate new models and updates to existing models, and a library of validated models is necessary to moderate user contributions to the models and, ultimately, to support good policy decisions by users inside and outside government. The history of NASTRAN, the NASA Structural Analysis program, provides a good example of the government developing an important software tool and then making it widely available to users and to commercial developers, who used it as the basis for creating additional applications and analysis tools. NASA initiated the development of NASTRAN in the early 1960s to provide its aerospace research projects with a finite element analysis capability. The initial release of NASTRAN, in 1968, was primarily of interest to the large aerospace companies and government laboratories that could afford the multimillion-dollar computers necessary to run the software. Since then, improvements in the NASTRAN code by users have extended the applicability of NASTRAN to almost every kind of structure, and improvements in the capability of widely affordable computers have removed com-
putational capability as a limiting factor in the use of NASTRAN. NASA’s decision to make the NASTRAN source code available to users and developers also contributed to the tremendous expansion of NASTRAN’s capabilities. Emulating the precedent of NASTRAN with a broad suite of air transportation system models would be difficult because of the resources required, the intellectual challenge involved, and the proprietary nature of many models, which are viewed by their developers as a means of maintaining a competitive edge with respect to other modeling organizations. Nevertheless, the benefits of such an effort, if successful, would be substantial.
SYSTEM MODELS AND AIR TRANSPORTATION SYSTEM SAFETY
Safety analysis of air traffic management operational concepts has traditionally been based on chain-of-event models, but other approaches based on systems theory have recently been proposed.
When safety is characterized by a chain of events, it may be analyzed using hazard analyses, in which the events leading to each hazard are identified (e.g., fault tree analysis or failure modes and effects criticality analysis) and deterministic models are built of the combinations of failure events and human errors. The hazard analysis models may be used to redesign the system such that hazards are eliminated or mitigated. In addition, probabilistic analysis of events and chains of events is sometimes used to determine the risk associated with a design. Various types of formal mathematical analysis can be applied to state-based models (both probabilistic and nonprobabilistic) to evaluate various aspects of safety.
In addition to using formal analysis to evaluate safety, simulations might be used. These simulations must include humans, who are an integral part of the air transportation system. One approach to the problem, proposed by, among others, Gore (2000) and Pritchett et al. (2001), is to use large-scale, agent-based simulations spanning one or more traffic sectors. Ultimately, it is hoped that these simulations will be able to simulate with high fidelity the ability of agents to reason and react to unexpected situations, but progress will depend on the ability to build accurate models of human behavior.
Simulation has also been proposed as a way to extend risk assessment methods. Such simulations build on traditional hazard analysis models but enable the use of nontraditional event ordering. By allowing for inconsistent or variable event ordering, which can have a significant impact on whether a set of events leads to an accident, these simulations can examine a larger range of potential chains of events. The underlying models used for these simulations are commonly state-based, but other types of models might be used. Methods using stochastic, state-based models in the simulations have been proposed to substantially reduce the simulation runs needed by classic Monte Carlo methods (Blom et al., 1998).
Alternatives to event-based models have been proposed, primarily based on concepts of systems theory (e.g., Rasmussen, 1997; Svedung and Rasmussen, 2002; Leveson et al., 2003). Systems theory is the mathematical foundation for system engineering, with roots that go back to the 1930s (Checkland, 1981). Systems theory emphasizes the manner in which organized systems (both human and nonhuman) function. It includes the principles, models, and laws necessary to understand complex interrelationships and interdependencies among linked components and subsystems within a system. Systems theory models include organizational and managerial factors that are often omitted from chain-of-event models. Safety models based on systems theory view accidents as arising from interactions among system components (Perrow, 1984), where the interactions may be nonlinear and involve multiple feedback loops. Systems theory models can be used to analyze software-related accidents, complex human decision making, and system adaptation or migration toward an accident over time and can handle dynamic or behavioral complexity in addition to static or structural complexity.
In a systems theory approach to modeling, systems are viewed as interrelated components that are kept in a state of dynamic equilibrium by feedback loops of information and control. A system is not treated as a static design, but as a dynamic process that is continually adapting to achieve its ends and to react to changes in itself and its environment. The original design must not only enforce appropriate constraints on behavior to ensure safe operation, but it must also continue to operate safely as changes and adaptations occur over time. Accidents then are treated as the result of flawed processes involving interactions among people, social and organizational structures, engineering activities, and physical system components. Systems theory approaches to modeling and analyzing safety are new, and it remains to be seen whether the resulting models will be more or less effective than the traditional chain-of-event models.
The FAA has established, maintains, and continues to improve a suite of models that are used for environmental impact studies and assessments of proposed regulatory or market-based measures to control noise or emissions. Community noise models estimate the number of people exposed to high noise levels at a single airport (the Integrated Noise Model) or globally (the Model for Assessing Global Exposure to Noise from Transport Aircraft, MAGENTA) and can reflect the effects of noise abatement procedures. The FAA’s local and regional air quality model (Emissions and Dispersion Modeling System, EDMS) calculates total emissions around an airport based on the number and type of aircraft operations and estimates how emissions are dispersed. The global emissions model (System for Assessing Aviation’s Global Emissions, SAGE) inventories global emissions, summing emissions from each flight as a function of flight
altitude and location. All of these models could be used as part of a suite of system models to evaluate the environmental benefits and trade-offs of measures to improve the air transportation system in terms of capacity and other performance parameters.
Given existing modeling and simulation capabilities and the state of ongoing research, the actions defined in the following recommendations would provide the long-term systems modeling capability needed to design and analyze evolutionary and revolutionary operational concepts and other changes to the air transportation system.
Recommendation 3-1. Value of Modeling and Simulation. Federal agencies involved in modeling and simulation of the air transportation system should make complementary use of field tests, laboratory tests, modeling, analysis, and simulation to improve their ability to (1) measure systemwide behavior of the air transportation system, (2) assess the performance of proposed operational concepts, technologies, and other changes, and (3) make informed investment decisions that reduce the schedule, cost, and technical risk of system improvements.
Recommendation 3-2. Management of System Models. Federal agencies that support research in aviation system models should improve their coordination, especially with regard to the following:
Ensuring that the federal investment for research and development in aviation models focuses on key issues, avoids unnecessary duplication, and encourages cooperation among developers.
Encouraging participation of industry and academia in modeling and simulation research and development relevant to government needs.
Establishing widely accepted criteria for the maintenance and validation of models.
Identifying models that are most important to government policy decisions.
Making those models more widely available to users inside and outside government.
Ensuring that modeling and simulation results are used appropriately by decision makers involved in developing the future aviation system.
Recommendation 3-3. System Modeling Research. The government and other interested parties should support additional research in the following critical areas:
Improving the interoperability of high-fidelity, detailed, data-intensive, long-run-time models of the U.S. air transportation system and the higher-level fast-time, abstract models necessary to analyze overall system performance under a variety of different assumptions so that both types of models can be brought to bear on relevant problems. (It may be feasible to develop models with adjustable resolution that can simplify variables for faster run time when those variables are critical to the analysis being performed.)
Modeling and simulation methods suitable for safety analysis, which inherently require a detailed level of modeling that includes all the factors that contribute to safety, including human performance and sociotechnical aspects of the system. (Additional fundamental research and development is required before these methods can enter widespread use. New approaches should be pursued using systems theory as well as new nontraditional chain-of-event models.)
Modeling demand and demand allocation for air transportation services, particularly as it relates to airline schedule changes, including city-pairs, routes (including altitudes and way points), time of day, and the establishment (or elimination) of hub airports. (Dynamic interactions between changing or radically new operational concepts and technologies and user behavior, as they relate to all modes of transportation and other factors external to the air transportation system, must be better understood to ensure the right problems are being addressed.)
Requirements, methods, and standards for validating individual models and suites of models.
Understanding how to connect models to form a suite of system models that includes nonlinear dynamic interactions and emergent properties.
Understanding the role of humans in the aviation system of the future and how to communicate this understanding in a convincing and supportable way. (Including computational human performance models in current simulations and using human-in-the-loop simulations is critical.)
Blom, H.A.P., G.J. Bakker, P.J.G. Blanker, J. Daams, M.H.C. Everdij, and M.B. Kompstra. 1988. Accident risk assessment for advanced air traffic management. Proceedings of the 2nd USA/Europe Air Traffic Management Research and Development Seminar, Orlando, Fla. Also in G. Donohue and A. Zellweger, eds. Air Transportation Systems Engineering. 2001. Progress in Astronautics and Aeronautics Series. Reston, Va.: American Institute of Aeronautics and Astronautics, pp. 463–480.
Checkland, P. 1981. Systems Thinking, Systems Practice. New York: John Wiley & Sons.
Gore, B. 2000. The study of distributed cognition in free flight: A human performance modelling tool structural comparison. Third Annual SAE International Conference and Exposition on Digital Human Modelling for Design and Engineering, Dearborn, Mich., June 6–8. Available online at <www.engr.sjsu.edu/hfe/hail/airmidas/SAE2KGPaper_2181.PDF>.
Leveson, N., M. Daouk, N. Dulac, and K. Marais. 2003. Applying STAMP [Systems Theory Accident Modelling and Process] to accident analysis. Workshop on the Investigation and Reporting of Incidents and Accidents. Williamsburg, Va., September 16–19. Proceedings to be published by NASA Langley Research Center, Hampton, Va. (see <http://shemesh.larc.nasa.gov/iria03/index.html>).
Perrow, C. 1984. Normal Accidents: Living with High-Risk Technologies. New York: Basic Books.
Pritchett, A., S. Lee, and D. Goldsman. 2001. Hybrid-system simulation for national airspace system safety analysis. Journal of Aircraft 38(5):835–841.
Rasmussen, J. 1997. Risk management in a dynamic society: A modeling problem. Safety Science 27(2/3):183–213.
Svedung, I., and J. Rasmussen. 2002. Graphic representation of accident scenarios: Mapping system structure and the causation of accidents. Safety Science 40(5):397–417.