Education, Training, and Professional Practice in Defense-Related MS&A
Education is central to developing and maintaining a high-quality professional MS&A community and is especially crucial as that community takes on the many challenges outlined in Chapters 3 and 4. Structural issues within the defense establishment affect DoD’s ability to attract and retain well-educated modelers. This chapter assesses the academic preparation of the current MS&A workforce at DoD. In it, the committee makes recommendations for satisfying the education and training needs of the future workforce.
This report’s primary emphasis is on competencies that professionals in various roles within the MS&A enterprise need to possess. Specific elements of a curriculum—courses, training modules—are not viewed as ends in themselves but as a means to develop key competencies. That is, a curriculum geared to a given professional role should give students the competencies they need for capable professional performance in the role and should be evaluated with respect to how well graduates demonstrate these competencies. In addition, MS&A educational programs used by DoD must provide the means for professionals to upgrade their skills as they move through their careers. The DoD human resources system, in turn, should be judged according to how well it matches individuals with positions utilizing their competencies.
The emphasis in this chapter is on the education and training of practitioners of MS&A, as distinct from the education of pure mathematicians or computer scientists. For this reason, a curriculum is judged successful to the extent that its graduates understand the role of MS&A within the overall DoD decision-making process and can play their respective roles within this process. Practitioners should be educated to understand that models and simulations must be designed, built, and used in a manner that provides timely and effective answers to questions of importance to the decision makers.
The ever-increasing complexity of the problems confronted by defense decision makers, together with the increasing ability of technology to solve them, makes it impossible to segment today’s MS&A world into specialized roles played by different individuals. The roles played by MS&A practitioners within the MS&A life cycle were discussed in Chapter 1. In many cases, a single individual will play multiple roles; in other cases, different individuals playing different roles work together to support a given decision-making objective. The various roles played by MS&A practitioners were identified in Chapter 1 and are expanded on here:
Analyst. An analyst formulates models and interprets them. He or she creates a formal representation of a real-world problem in a form that is amenable to computation. In addition, the analyst often serves as the domain expert on the modeling team. In this role, the analyst understands which factors are important and must be included in the model or simulation, the level of fidelity needed, and the types of displays that might be useful. After the simulation is run, the analyst will transform its output into conclusions that inform a decision, assess the strengths of the result and the uncertainties in it, and determine the importance of the assumptions incorporated in the model to the validity of the conclusions. Finally, the analyst is frequently called upon to explain the results to a nonspecialist, often in nontechnical language. Because of the variety of skills needed for this explanatory role, it is often filled by more than one person on the MS&A team.
Modeler/programmer. A modeler/programmer transforms the formal representation created by the analyst into executable form. He or she ensures that the designs are modular, computationally efficient, and well documented and that configuration management is adequate and has well-defined subsystem interfaces. The increasing complexity of systems and the growing drive for reusability and interoperability have given rise to highly specialized implementation environments, so that the modeler and the programmer may be different individuals.
Implementers. An implementer adapts the program for
execution, reviews and amends the experimental plan, runs the M&S through the cases of the experimental plan, observes and iterates, and executes the postprocessing needed to visualize the model outputs and produce comprehensible and usable results for the analyst.
M&S manager. A manager hires, fires, and manages personnel; suggests or approves choices of M&S and related purchases, education, and training; draws upon outside consultants for checks of quality; and interacts with peers in meetings of M&S governance. A manager may practice some quality control in specific analysis projects. Some M&S managers may also have analytical depth and may then function more like the analyst-in-charge.
Consumer. The consumer employs M&S to support military decisions. Some consumers may be military or civilian decision makers and/or managers with little technical training and only a rudimentary understanding of the internal workings of the models and simulations that support their decisions. In other cases, they may be equally competent in one of the other roles or may even have played one or more of the other roles at some point in their careers. In still other cases, they play all the roles, as when a decision maker builds a simple model on his or her desktop computer to support a key decision.
Different roles may require different skill sets, acquired through education or experience. For example, analysts and modelers require different skill sets—mathematics, operations research, and statistics for analysts, computer programming for modelers—which are typically taught in different educational disciplines. Consumers may require only a passing understanding of the details of any of the other roles but need a solid appreciation of the limitations of the MS&A activity, the assumptions made, and the conditions within which results hold.
In addition to depending on the ability of individuals to play their respective roles, the effectiveness of MS&A depends on their ability to work together toward a common objective. However, in practice, individuals in different roles sometimes do not mesh as well as they should, and the quality of decisions can suffer as a consequence. The best educational programs consciously address this issue. The problems that remain are due to a number of factors, including inadequacies in some education and training programs or in the organizational climate and deficiencies in the preparation of some practitioners. The committee agrees that in many cases these concerns are well founded and that well-designed education and training can contribute to a solution.
The committee has identified several competencies that it believes have not been given sufficient emphasis and that should be included in the education of MS&A practitioners in the defense establishment:
Documentation. MS&A professionals need to be capable of developing, or guiding the development of, clear and useful documentation for their models, algorithms, and analyses.
Meaningfulness. MS&A professionals need to be able to describe the meaningfulness of their model outputs. For example, is the output shown one among many possibilities? Is it an expected value of a family of outputs? What statistical properties does it have?
Audit. MS&A professionals need to be prepared to subject their work to detailed audit and criticism and to use the results of such audits constructively.
Comparison with reality. MS&A professionals need to understand the real world being modeled in sufficient detail to compare their results against the results that might be expected and to justify any divergence.
In the sections that follow, the committee addresses the question of how to ensure that education and training prepare people to work effectively with other people playing roles that are different from their own, as well as to recognize the limits of their own competence and to compensate appropriately by drawing on the expertise of people trained for different roles. The talents required of MS&A practitioners and the roles they play in the process change over the course of an MS&A study. These roles and the extensive training associated with imparting the competencies needed for each of them make the effective employment of the MS&A workforce a challenging task and require that DoD use its MS&A workforce optimally throughout the life cyle of each project.
DoD is clearly not the only organization that is dependent on MS&A, nor is it the only one that is concerned with best practices in building and using computational models. A good recent example of a community’s standards is found in Jakeman et al. (2006), in this case the environmental community. Another example, focusing on community standards that are desirable in the area of regulatory analysis, is OMB Circular A-4 (OMB, 2003). In fewer than 50 pages, it lays out standards that all agencies should observe in conducting and presenting regulatory analyses. A similar document promulgated by DoD would be useful by imposing uniform standards of practice for MS&A.
A SURVEY OF TODAY’S EDUCATION AND TRAINING LANDSCAPE FOR MS&A
Here the committee surveys the vast array of programs having some relevance to defense MS&A. This section provides a broad overview of the different roles for which education and training are needed and of the many educational options available to MS&A professionals at various points in their careers. It also considers how those programs mesh with current career progressions and identifies the essential ingredients of a successful MS&A education.
Individuals working in MS&A have undergraduate degrees in a wide variety of disciplines, including computer science, computational science, operations research, statistics, systems engineering, industrial engineering, and other traditional programs in engineering, mathematics, or science. For managers of programs or projects with a significant MS&A component, education in engineering management is highly desirable. Education in human factors and the psychological and social sciences is important, especially for those involved in human-computer interface design, human-in-the-loop simulation, and the modeling of human behavior. Very few people come out of an undergraduate education with broad enough and deep enough knowledge to effectively pursue a career in MS&A. Generally, a B.S. allows entry into an MS&A career only as a technician. Suggested requirements for a master’s-level program are given in the next section, “MS&A Curriculum.” However, with today’s emphasis on lifelong learning and the rapid evolution of M&S technologies, it is expected that all MS&A professionals will upgrade and expand their skills as their careers progress.
Through its internal system of academies and colleges, DoD offers baccalaureate and postgraduate education programs for uniformed personnel aspiring to careers as MS&A professionals. Undergraduate education programs available at the U.S. Military Academy at West Point, the United States Naval Academy, and the United States Air Force Academy include MS&A. Postgraduate programs are available at a number of military academies, including the Air Force Institute of Technology, the Naval War College, the Naval Postgraduate School, and the National Defense University. As an example, the U.S. Army Command and General Staff College has institutionalized simulation into the curriculum for all of its schools. As the college shifts its curriculum toward execution-centric learning, simulations are increasingly being used to expose future general staff officers to real-world scenarios that are relevant to current operations.
Over the past 20 years, a small number of civilian academic programs have emerged that specialize in defense-related MS&A topics. Most have been at the graduate level, although an undergraduate program is offered at Arizona Polytechnic, and a few associate degree programs prepare students to become simulator technicians. Furthermore, many programs in other disciplines such as computer science or operations research have concentrations targeted at individuals entering or working in defense-related MS&A. Thus, a wide variety of options is available to individuals initiating or continuing their education for careers in defense-related MS&A.
An important emerging technology is distance education. With the newer technologies for distance education, including streaming audio and video interaction (e.g., Pullen, 2000), students can interact synchronously or asynchronously with instructors and other students, gaining an experience that is close to classroom education. For mature students in some subject areas, distance education now appears to be at little disadvantage compared with traditional classroom instruction and has the advantage of convenience. However, some of the most critical skills for an MS&A professional include the ability to work in teams and the ability to interact effectively with consumers. To acquire such skills, some degree of in-person interaction is essential. Nevertheless, technology for virtual meetings, asynchronous discussion, and file sharing can, if used appropriately, greatly enhance the effectiveness of teams and can augment traditional avenues for interacting with model consumers. Both the technology base and the experience base for distance education are advancing at a rapid pace, although the educational community still has much to learn about the strengths and weaknesses of distance learning for different types of students and different subject areas.
A vital factor in the health of the MS&A profession is the infusion of new talent. The MS&A community within the defense establishment is affected by two trends in the field at large that are troubling in this regard. First is the perceived decreasing interest in mathematics and science education on the part of native-born American citizens. Second is the difficulty, especially since September 2001, for foreign nationals to obtain visas to study in the United States and to remain here after completing their education. Together, these trends raise concerns about the influx of new young MS&A professionals into the workforce.
The committee examined educational programs offered by a number of institutions of the military establishment, held discussions with MS&A practitioners and consumers, and consulted the M&S Body of Knowledge published by the DMSO Education Consortium in April 2004.
These sources reveal a common core set of competencies that should be possessed by professionals in the field of MS&A. The committee concludes that a master’s-level education for an MS&A specialist should provide three kinds of competency: background, core, and specialized defense knowledge.
An MS&A professional should have fundamental competency in undergraduate-level mathematics, science, and computing:
Mathematical topics include algebra, trigonometry, engineering- or physics-oriented calculus, linear algebra, and differential equations.
Statistical topics include calculus-based probability, inference, and data analysis.
Computing background includes basic computer lit-
eracy (basic competency in word processing and spreadsheets), ability to write programs in a computer language such as C++ or Java, and experience with at least one model-development and/or analysis environment such as Matlab, Mathematica, or a spreadsheet macro language.
Science should include college-level biology, chemistry, and/or calculus-based physics.
These competencies are acquired by most students who complete an undergraduate degree in mathematics, physical science, or engineering. Those with other educational backgrounds should make up the missing elements before embarking on specialized training in MS&A.
Building on the background competency, an educational program for a well-educated MS&A specialist would provide a common set of core competencies:
MS&A life cycle. Students should appreciate the role of MS&A in decision making, be exposed to the history of MS&A, and gain an understanding of different types of modeling methods. Students should be introduced to the life cycle of an MS&A study and gain practical experience with each of the steps in the life cycle. Through the use of both positive and negative exemplars, they should understand the value of an organized, systematic approach to carrying out the phases of the life cycle. Their education should provide practical exercises in which they perform all the activities inherent in the life cycle of problem formulation, conceptual model development, selection of software, simulation design, analysis plan, execution, analysis, presentation of results, and evaluation.
Deterministic modeling and optimization. Despite the stochastic and complex nature of many military situations, many others lend themselves to deterministic models that are capable of being optimized or nearly optimized. In addition, such models are useful as approximations to more complex real-world situations, submodels of larger simulations, quick-turn models to meet short deadlines, or models of situations with small variance where deterministic techniques may be sufficient.
Continuous and discrete event simulation. An important core competency is the ability to construct and execute simulations of both deterministic and stochastic discrete event and continuous systems. The typical undergraduate or graduate degree program would include a course in which students learn to develop a model, develop a computer representation for their model, design experiments to be conducted on the model, implement the model in a computer simulation environment, conduct experiments, and analyze and interpret the results.
Probability and statistics. The changing mission space places greater emphasis on performing effectively in situations characterized by large uncertainties. Thus, a strong grounding in the fundamentals of probability and statistics is essential, as is the ability to apply statistical thinking to military problems. The core background described above includes basic competence in probability and statistics. MS&A practitioners must also be proficient in more advanced areas of statistics such as regression, analysis of variance, experimental design, and data analysis, including data combination, and have some exposure to stochastic processes (e.g., Markov processes). An understanding of the concepts of state spaces, dependency conditions, and elementary principles for dealing with risk and uncertainty is important, as is a knowledge of decision analysis. These topics will not be a part of the undergraduate education of many students but are essential ingredients in the toolbox of an MS&A professional.
Topics in computing. The core background described above includes basic computer programming skills. Additional necessary capabilities include an introduction to data structures, software engineering, basic numerical methods, and analysis of computational complexity.
MS&A evaluation. Evaluating how well a model or simulation serves the purpose for which it is designed is one of the most important activities within the MS&A life cycle. The MS&A community places strong emphasis on validation, verification, and accreditation. While there is good reason for a strong emphasis on effective evaluation, excessive concern with formal certification can detract from the overall purpose of supporting the decision maker. MS&A education should cover VV&A thoroughly and should stress its importance not as an end in itself but as a means for ensuring that models and simulations are properly evaluated and that there are mechanisms to guarantee that the most appropriate methods are being applied to a given problem. MS&A education should also prepare students for dealing with the reality of models that cannot be classically validated (see “Expanded Concepts of Validation” in Chapter 3).
Human-simulation interaction. A simulation is useful only to the degree that humans can interact effectively with it. An MS&A professional should have a basic understanding of perception, cognition, and the interaction of humans with computers. As computing technology advances, a greater variety of interaction modalities will become important. As the state of the art in human-in-the-loop interactive simulations advances, an understanding of human-computer interac-
tion becomes more important. The ability to communicate the results of MS&A to decision makers in ways that highlight its relevance to military decisions is an essential skill. The ability to communicate uncertainty is especially important, not only including error bounds due to well-understood stochastic factors but also model uncertainty and deep uncertainty.
Modeling humans. MS&A has traditionally focused on modeling physical systems or systems in which social and cultural factors can be ignored. Sometimes there are strong pressures to focus resources on modeling well-understood aspects of a problem, leaving other aspects unmodeled. As the importance of including social and cultural factors in models and simulations grows, it will become necessary to provide students with the tools for incorporating these factors when traditional MS&A is inadequate. The appropriate modeling technology differs in important ways from traditional M&S technology. Students should be exposed to the literature on effective technologies for exploratory modeling (e.g., Bankes, 1993) and for modeling human and social systems.
Managing MS&A. Individuals managing programs and/or projects in MS&A require management education. While a course of study in business administration provides useful education for managers, management of highly technical programs requires knowledge that goes beyond what is taught in most business administration programs. Some degree of technical training is necessary for individuals managing highly technical programs. In addition, MS&A in DoD has some unique problems not likely to be addressed in civilian schools. One is the joint management of civilian personnel and military personnel, subject to frequent rotations among the latter. A second is the necessity to break down the modeling stovepipes, discussed in Chapter 2, that are artifacts of previous military requirements.
The above list of core competencies provides a strong grounding in MS&A that is not specific to military systems. Many civilian programs whose target population is students aiming for careers in civilian MS&A cover the bulk of this core. At DoD, practitioners also need background in the kinds of problems to which MS&A is applied in that establishment—for example, training, acquisition, test and evaluation, and combat. Also specific to DoD would be a familiarity with the main DoD simulation tools.
Many students at the master’s level, at least in DoD-sponsored schools such as the Air Force Institute of Technology or the Naval Postgraduate School, might wish to specialize in MS&A applied to particular areas, such as combat, training, or logistics. In addition to competence in the particular subject matter and in MS&A techniques appropriate to that subject matter, they should have at least a rudimentary knowledge of the DoD organization, including the military components, and Joint Force organizations, and the overall planning, budgeting, and acquisition processes.
The above discussions stress the importance of different roles in the overall MS&A process. As was pointed out in Chapter 4, requirements for nontraditional missions require that DoD draw on the body of experience from the larger MS&A community. It has already been pointed out in Chapter 3 that the ability to work on the full range of PMESII phenomena requires building the skills needed to work in cross-disciplinary teams.
As mentioned earlier in this chapter, formulators, implementers, and analysts require stronger mathematical and computer competency than consumers, but even consumers must have some degree of competence if they are to understand the value and the limitations of the results and to apply them effectively to decisions. Conversely, formulators, builders, and analysts must have some degree of competence in the problem domain if they are to avoid inadvertently and inappropriately altering the consumer’s problem in the way they construct the model or analyze the data.
Recommendation 12: DoD must give its MS&A practitioners some exposure to all the topics in the core curriculum: the MS&A life cycle; continuous and discrete simulation; probability and statistics; topics in computing; deterministic modeling and optimization; MS&A evaluation; human-simulation interaction; modeling humans; and managing MS&A. Familiarity with these topics is essential for all practitioners, although the depth of knowledge needed will depend on a practitioner’s particular role.
FOSTERING A STRONG AND EFFECTIVE MS&A COMMUNITY AT THE DEPARTMENT OF DEFENSE
Key to the effective employment of MS&A to support decision making at DoD is an organizational climate that fosters high-quality MS&A. There are many MS&A professionals within DoD, but distributed widely among many offices and programs and subject to different missions, cultures, and priorities. There would be value in DoD working to encourage a more integrated MS&A community. In this section, the committee suggests ways to foster and maintain such an organizational climate.
The educational programs examined by the committee stress the importance of effectively integrating different activities:
Modeling (creating computable and/or manipulable representations—physical, mathematical, logical—of the real world).
Simulating (implementing models that describe system behaviors over time, usually in computer code).
Interfacing with the user, trainee, or analyst.
Analyzing (making sense of the output of the simulation; translating outputs into statements about the real world that are relevant to the decision maker’s problem).
Evaluating (assessing the fit between the model and the real-world problem; identifying assumptions on which results depend; qualifying conclusions appropriately to account for problematic assumptions and/ or inaccuracies).
There is a general agreement, both in academia and in the applications community, about the importance of an education that effectively integrates these activities. Segregating them and treating them as isolated components can result in neglect of important aspects of the problem and can seriously degrade the effectiveness of analyses in support of decisions. In practice, however, these activities are sometimes segregated into different bureaucratic units in an organization. Institutional barriers between units can interfere with the effective use of MS&A to support decisions.
Recommendation 13: DoD should ensure that its educational programs provide experiences in which students integrate the activities of modeling, simulation, analysis, evaluation, and communication to address real-world problems of importance to the consumers of the information.
Education is a career-long process for today’s professionals. As in all fields of science, change is rapid in computing and information technology. With this in mind, the committee believes that mechanisms need to be in place for MS&A practitioners to maintain currency. Civilian MS&A practitioners clearly need continuing education to maintain their technical competence. This need is even more critical to uniformed MS&A practitioners who routinely serve in non-MS&A positions. Because these uniformed MS&A practitioners may have periods during which they do not practice their MS&A skills, they may lose some of their technical skills and be unaware of advances in their associated technologies. The Army, for instance, has two MS&A-related specialty fields: FA57 (simulations) and FA49 (operations research and systems analysis); but officers also need to serve in other assignments that may not involve MS&A in order to advance their careers. This practice of rotating out of MS&A assignments, while potentially disruptive, is actually critically important for keeping the MS&A community connected to the operating forces. With this in mind, the committee stresses the importance of having institutional mechanisms in place to refresh and upgrade the competencies of uniformed MS&A professionals who rotate in and out of MS&A positions. It is essential that MS&A professionals upgrade their skills to cover areas in which their initial education was less than thorough and to keep up with changes in the field.
Maintaining professional currency can involve many other activities in addition to or in place of traditional classroom instruction. Practicing MS&A professionals need to devote a certain portion of their time to reading current literature, attending conferences and lectures, becoming involved in professional societies, and other activities that provide regular interaction with other professionals and exposure to new ideas. It is especially important that they be exposed to uses of MS&A outside their own specialty. Some of the most important advances come from the transfer of technology from one field to another. An awareness of how MS&A is applied in other disciplines can be an important source of new ideas for defense-related MS&A. In this regard, a vibrant professional community with opportunities for exchange of ideas is essential to maintaining the health of the profession. The Military Operations Research Society (MORS) provides many national and local conferences, symposia, and workshops, as well as a journal and newsletters. The Society for Computer Simulation publishes the peer-reviewed archival journal Journal of Defense Modeling and Simulation. Nondefense MS&A professional societies also offer opportunities for the exchange of ideas, and many of them have special-interest groups devoted to defense-related topics, such as the Military Applications Section (MAS) of the Institute for Operations Research and Management Sciences (INFORMS). Unlike MORS, MAS is open to individuals without a SECRET security clearance.
Building and maintaining an effective MS&A community requires fostering communication and collaboration among MS&A practitioners in government, industry, and academia. As noted in the final subsection of Chapter 3, collaboration is impeded by export controls and security regulations. Controls on the flow of information are, of course, necessary to prevent sensitive information from falling into the hands of those who would use it to harm the United States and its interests in the world. Nevertheless, building a strong, effective, and self-critical MS&A community requires drawing on the best available talent. There are actions DoD can take to access this talent that would not compromise national security. For example, DoD could require developers to build unclassified versions of models and unclassified databases, or to segregate classified parts of models in separate modules, with unclassified alternative modules that can be substituted when the model is being used in a nonsecure environment. This practice could actually make an MS&A capability more secure by restricting the use of classified information to situations in which it is necessary to the modeling objective. Furthermore, bringing in the most qualified individuals to perform evaluations enables more thorough and capable validation of models and gives DoD the benefit of the strongest available expertise.
An important issue for DoD is the place of both traditional and continuing MS&A education in the career progression of military officers. As officers with education in MS&A progress up the command chain, they tend to move from doing MS&A themselves to leading and managing others who do it. Officers with a solid education and practical experience in applying MS&A to real-world problems move into the ranks of leadership with a good understanding of how MS&A can be most effectively employed to support end users. It is essential that obtaining this education and practical experience not come at the cost of eventual career advancement. Furthermore, educational opportunities need to be provided in a timely manner to support an officer’s current assignment and to anticipate future career directions.
Bankes, S. 1993. “Exploratory modeling for policy analysis.” Operations Research 41(3):435-449.
Executive Office of the President, Office of Management and Budget (OMB). 2003. “Regulatory analysis.” Circular A-4, September 17. Available at http://www.whitehouse.gov/omb/circulars/a004/a-4.pdf.
Jakeman, A.J., R.A. Letcher, and J.P. Norton. 2006. “Ten iterative steps in development and evaluation of environmental models.” Environmental Modeling & Software 21:602-614.
Pullen, J.M. 2000. “The Internet-based lecture: Converging teaching and technology.” ACM Special Interest Group on Computer Science Education (SIGCSE) Bulletin 32(3):101-104.