|
Items in cart [0] |
Questions? Call 888-624-8373 |
| Copyright © 2009. National Academy of Sciences. All rights reserved. Terms of Use and Privacy Statement |
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 103
6
Conclusions and Recommendations
The committee makes four overall recommendations—one in each of
the four major areas that it considers current impediments to the
widespread use of simulation-based acquisition (SBA) and related concepts
in manufacturing and acquisition:
· Technology and research;
· Infrastructure for modeling and simulation (M&S);
· Modeling and simulation in manufacturing and acquisition,
including developing experience in the use of M&S, learning
lessons from that experience, and institutionalizing those lessons;
· Culture and the human issues inherent in any major change.
Within these overall recommendations are subsidiary
recommendations addressing the four sets of issues. Complicating this
picture are the breadth and depth of SBA. Application of M&S to a single
component is quite different from its application to the life cycle of a
system-of-systems. The goal of the recommendations is to move DOD
toward this all-encompassing objective.
The current state of M&S technology applicable to systems
acquisition is mixed. Some is ready for full-scale implementation. Some is
ready for prototype testing, evaluation, and improvement. Some is not
available at all. To move toward DOD's ultimate goads will require
departing from approaches that dominate today's M&S and rely on single-
point solutions—single-resolution models and "stove-piped" simulations
that cannot be reused and integrated with others. These departures will
103
OCR for page 104
104
MODELING AND SIMULA TION IN MANUFACTURING
necessitate significant improvements in multiresolution modeling; model
integration; model reuse; verification, validation, and accreditation
(VV&A); and multiresolution modeling among others. To achieve DOD's
ultimate goals, the whole modeling paradigm may need to be rethought.
These methodological and technological issues must be addressed to form
a firm scientific basis for M&S in manufacturing and acquisition of
military systems (or systems-of-systems).
DOD must gain experience with modeling and simulation in the
context of SBA. Carefully chosen projects should use M&S in real,
important applications. This first class of applications should not carry
significant technological risk, but should use proven technology to
demonstrate the value of M&S in acquisition even when applied in limited
situations. All of these experiences need to be positive. To continue, a
second set of carefully chosen projects should use M&S in SBA in a
prototyping mode. These projects should carry moderate technological
risk, but low programmatic risk. The prototypes should 'not be on the
critical path of major acquisitions. This set of projects should be used to
understand the application of new M&S technology in real-world
application, and the results should indicate the strengths and weaknesses of
M&S technology in SBA. Successes should be moved into the mainstream
of SBA. Areas that prove difficult, but important, should drive M&S R&D
funding.
A third area of importance is in infrastructure development, both in
M&S and in information technology. Lacking infrastructure, every M&S
application will start over without building on past applications. The
infrastructure is required in order to achieve continuous improvement in
the use of M&S in acquisition. Infrastructure includes shared processes,
databases, standards, and architectures. This infrastructure must largely
evolve from practice. Externally imposed standards, for example, are rarely
effective.
Fourth, improvement in all of the above areas will prove ineffective
without a change in the DOD acquisition culture. One of the committee's
recommendations is for an SBA center of excellence. This resource, which
could be geographically distributed, would help create the new culture of
using M&S in SBA applications. It would also help the defense SBA
community reach out to the academic community and integrate knowledge
and insights from that community into the DOD acquisition world. Finally,
there must be leadership from the top in DOD to encourage the appropriate
use of M&S in acquisition and manufacturing. The risk for program
managers must be changed from worrying about "deviating from the status
quo" to worrying about "not mindfully using new technology to improve
the state of acquisition practice."
OCR for page 105
CONCLUSIONS A ND RECOMMENDATIONS
105
An integrated view of the committee's recommendations is critical.
Enhancement of the technology enables the process. Use of that
technology provides experience that guides further use, as well as pointing
out important opportunities for further R&D. infrastructure allows
improvement over time and an ability to be more consistent and
integrative. Finally, people and culture are the bottom line. If the people
and the business culture do not trust and embrace M&S in manufacturing
and acquisition, use of SBA will not advance. As policy makers read these
recommendations, they need to consider the synergy among these areas.
Each area needs to be addressed to be successful in achieving DOD's goals.
TECHNOLOGY AND RESEARCH
Overall Recommendation. Long-term research and development should be
funded, conducted, and applied to increase the science and technology base for
M&S in areas in which current knowledge falls short ofthat required for
manufacturing, acquisition, and life-cycle support of military systems.
.
Recommendation. In order to realize DOD's vision for the use
of M&S in manufacturing and acquisition generally, and for
SEA in particular, DOD should conduct or support basic
research and development in the following areas:
Modeling methods: scalability, multiresolution and
multiviewpoint modeling, agent-based modeling, semantic
consistency of models, model complexity, fundamental limits of
models and computation, and characterization of uncertainty and
risk in models;
Model integration: interoperability, composability, integration of
heterogeneous processes, and linking of engineering and
effectiveness simulations;
Model correctness: domain knowledge, including
phenomenology of warfare, physics-based modeling, and human
behavior modeling; and general verification, validation, and
accreditation methods;
Standards: M&S standards for interoperability and modeling;
general software standards; and higher-layer standards, including
enterprise engineering;
Methods and tools: for assistance in the translation of system
requirements into system functionality.
OCR for page 106
106
MODELINGANDSIMULATIONINMANUFACTURING
· Domain-specific models: including models for emerging areas
such as information operations and operations other than war.
The current state of scientific knowledge in M&S falls short, in
several areas, of the level needed to realize DOD's vision for the use of
M&S in manufacturing and acquisition generally and for SBA in
particular. Basic scientific research and development (R&D) is required in
those areas to address the knowledge shortfalls. Those areas, identified
and discussed in Chapter 5, are summarized here.
Research in modeling methods is needed. Scalability is an essential
M&S capability to support the range of M&S applications needed.
Multiresolution and multiviewpoint modeling contribute to providing
scalability and flexibility, but they are still understood primarily from an
experimental point of view. Agent-based modeling can support emerging
requirements for systems-of-systems modeling. Models must be
semantically consistent if they are to be composed in a simulation system.
Abstraction and multiresolution families of models can help deal with the
increasing complexity of models. Theoretical limits of modeling and
computation apply to the use of M&S for manufacturing and acquisition,
but the implications ofthose limits are generally not considered in that
context. They must be studied, both to determine the limits and to develop
methods to deal with them. Uncertainty is present in most of the real-
world systems of interest; its influence must be modeled and measured.
Both characterization of uncertainty and risk in models and the
development of models that assist in evaluating uncertainty and risk are
important.
Further advances in model integration are required. Model
interoperability and composability, have the potential to support the
flexible use of models for different phases of the manufacturing and
acquisition process, but they present both theoretical and practical
problems. Substantial effort has been put into interoperability and
composability, with the goal of achieving both reductions in development
cost and increases in credibility, but those goals have generally not yet
been fully achieved. This is especially true with respect to integrating
heterogeneous processes and linking engineering and effectiveness
simulations.
Increased attention must be given to model correctness.
improvements to model correctness will depend on both domain
knowledge in crucial areas and general VV&A methods. More domain
knowledge relevant to model development is needed in the areas of the
phenomenology of warfare, physics-based modeling, and human behavior.
The credibility and utility of M&S for all applications, including
i
OCR for page 107
CONCLUSIONS AND RECOMMENDATIONS
107
manufacturing and acquisition and SBA, depend on validation of models
and simulations, yet the degree and bounds of validity of many models are
difficult to quantify or even qualify.
The development of supporting standards must continue. Standards
for interoperability have achieved significant success, but work remains in
this area. Standards for modeling will directly support interoperability and
composability. General software standards have made simulation
development more predictable and reliable, and further application of them
to simulation would be beneficial. It is important that higher-layer
standards, such as enterprise engineering, develop in a manner compatible
with M&S.
Recommendation. M&S capabilities should be enhanced so
that systems-of-systems have the following capabilities:
.
· To represent possible design variations, operational use patterns,
and engagement scenarios;
· To contain and make available a library of composable sensor,
weapon, and C4ISR (command, control, communications,
computers, intelligence, surveillance, and reconnaissance)
models;
· To manage interactions among component systems efficiently;
and
To support analytic and optimization usage modes with
visualization, experiment definition, and statistical analysis
capabilities.
Military force modernization in the future may depend on the
introduction of novel system and operational concepts to which the present
acquisition process is poorly adapted. Accelerated development and
demonstration of new systems prior to full funding or fielding should be
conducted in order to assess their expected mission effectiveness. M&S
technologies are essential for performing such assessments in complex
warfighting environments. This is especially true for new operational
concepts that involve systems-of-systems.
Systems-of-systems simulation depends on several of the basic M&S
areas listed above for research—especially, multiresolution modeling,
integration and interoperation, and validation. It is therefore a cross-cutting
and integrative application of those basic research areas. Because of its
difficulty, research on systems-of-systems simulation should proceed in
parallel with work in the basic areas, since the systems-of-systems
OCR for page 108
108
MODELING AND SIMULA TION IN MANUFACTURING
simulations research may reveal requirements to consider when studying
the basic areas.
Recommendation. A research initiative should be created at
multiple universities to attract academic attention and expertise
to the M&S needs of DOD.
Such an initiative, commonly known as a multiuniversity research
initiative (MURI), may be useful in attracting academic expertise to the
M&S needs of DOD. An increasing number of universities have programs
that focus on M&S. Each MURI should be configured to include strong
industrial participation, not only in terms of dollars but with time of key
personnel to work with academic researchers. Such programs can focus
efforts and provide assistance in addressing the most critical M&S
technology, infrastructure, and programmatic shortfalls. By encouraging
research with direct funding and by stipulating active involvement of both
industry and academia in shared research, DOD should be able to advance
the state of the art in M&S technologies directly. The Defense Modeling
and Simulation Office (DMSO) should establish mechanisms to acquire
feedback from DOD program offices concerning shortfalls in M&S
research. This information should be used to drive the requirements
process for direct funding within the MURI program.
Recom mendation. Transitioning of research into applications
should be planned and executed as an integral part of the
development process.
A continuing problem has been the transitioning of results from M&S
research into application. Active involvement of industry in M&S
research, together with carefully crafted demonstration projects
implementing research results, will provide lessons learned in real-world
application. Remaining difficulties are to be cycled back into research
programs, while successes should be harvested and developed for
widespread use in robust manufacturing and acquisition pursuit.
Appropriate members of the application community must be made aware
of results via active promotion as well as through papers, workshops, and
symposiums.
Desired End State
To reach the potential benefits M&S could have for manufacturing
and acquisition in general and SEA in particular, research is required, and
OCR for page 109
CONCL USIONS AND RECOMMENDS TIONS
109
clearly a desired outcome is that it be funded and conducted. This is not,
however, simply a general call for M&S research of all types. Rather,
specific research topics have been identified as crucial to the applications
that are of concern here, and it is research into those topics that is sought.
That research should be conducted through one or more research initiatives
with relatively broad university participation, but with active coordination
from a responsible organization to ensure its relevance. Finally, it is not
enough that the research be funded and conducted; it must also be applied
to the issues of manufacturing and acquisition. The test of success in this
technology and research area will be the practical application ofthe results
of the recommended research in actual manufacturing and acquisition
processes. That transition from research into application will likely depend
on cooperation between universities and industry.
INFRASTRUCTURE FOR MODELING AND
SIMULATION
Overall Recommendation. DOD should invest in "common good"
activities to encourage adequate standards and a strong infrastructure for
M&S.
Recommendation. DOD should institute incentives for
program managers to develop M&S elements that contribute to
the general infrastructure, including an annual competition for
the best infrastructure contributions. A handbook that illustrates
and discusses how M&S can be integrated into program
planning documents should be developed.
In current practice, investments by program offices in M&S may
often result in data, models, tools, and environments that have the potential
to be reused across DOD acquisition programs. However, such generally
useful outcomes are incidental to program manager goals and objectives,
and there is no institutionalized process for continually infusing these
results into the common infrastructure. A process of this kind should be
defined and implemented.
DOD should support creation of SBA-related M&S infrastructure.
For example, resources could be set aside annually to reward program
managers in a competition for the best infrastructure contributions. In this
way, program offices would increase the value of their work to the DOD
by developing M&S and information technology applications useful to the
immediate program and readily reused by other DOD program offices.
OCR for page 110
110
MODELING AND SIMULA TION IN MANUFACTURING
DOD should develop a guide or handbook that not only defines
systems engineering as practiced within DOD but that goes on to illustrate
and discuss how M&S may be integrated into program planning
documents, such as the integrated master plan (IMP). This guide should be
made available online and should provide templates and other tools
necessary to support development and implementation of an acquisition
program M&S plan. Defining systems engineering at a high level and
further defining how M&S applies to systems engineering activities within
a total systems engineering framework will provide clearer guidance for
the integration of M&S into systems engineering activities. Providing this
as a Web-based system also provides direct linkage of guidance to
templates and tools within the DMSO and Defense Acquisition Deskbook
infrastructure. The Deskbook is an excellent repository of information, but
it lacks useful products for supporting SBA and use of M&S in the
. . .
acquisition process.
Recommendation. DOD should exploit common elements of
M&S to develop a common infrastructure capable of supporting
consistency and interoperability across programs.
The M&S infrastructure that the committee recommends creating
includes the following:
.
.
Common repositories. These repositories should contain data,
models, tools, and environments that can support multiple phases
of a program and that persist from program to program. Standards
should allow different developers to interoperate for the common
good, while retaining competitive advantage and property rights
where appropriate.
Knowledge base. This knowledge base underlies the right
formulation of M&S infrastructure and workable standards and
represents a well-organized information resource in the theory,
science, engineering, and craft required for successful M&S
development. Properly archived in electronic database form, with
helpful searchability attributes, the knowledge base supports
continued advances though basic and applied research and
development.
Trained M&S workforce. This workforce contains the cadre of
professionals, ranging from specialists in M&S infrastructure to
M&S researchers, needed to support the wide array of activities
and programs that SBA entails. More and better-trained workers
with the knowledge and skills to jump-start industrial and DOD
assimilation of M&S and sustain its development are needed.
OCR for page 111
CONCL USIONS AND RECOMMENDA TIONS
.
111
More detailed discussion of this area is included under the
heading "Culture and Human Issues," in this chapter.
Information technology infrastructure. This infrastructure
contains the computing and networking technology that processes
and routes the flow of information, much as the national highway
system enables the flow of physical goods and services. M&S
technologies are necessary to design and test current and next-
generation computing and networking technologies that promise
exponential increases in power. Conversely, the information
technology structure will drive the advance of M&S
infrastructure, making it possible to tackle issues in greater depth
with increasing confidence in the outcomes. Related to this
infrastructure are standards for its modeling components.
Recommendation. A collaborative effort should be stimulated
among members of DOD, industry, and the academic
community to advance the emergence of standards for
performance simulation and product modeling.
· DOD should remain actively engaged in commercial standards
efforts to ensure that DOD needs are considered in the standards
development process.
· DOD should take the lead in the development of standards that
lack commercial interest.
· DOD should develop standard semantics for the data elements
used in DOD acquisition-related models and simulations, such
as standard nomenclature, definitions, and units of measure.
In its review of previous studies related to M&S in military
acquisition, the committee found a general consensus that standards play
an important role and can be applied to make better use of M&S in
acquisition. The continuing need for a collaborative effort in this area
among DOD, industry, and academia must be reiterated. As robust
standards emerge, they will enhance the interoperability and reuse of
models and simulations for system acquisition. As a cautionary note, ill-
thought-out, mandated standards inhibit progress rather than encourage it.
Standards must emerge from the joint efforts of the user community.
Some areas exist in which significant industry effort outside the DOD
community can be leveraged. For example, although progress has been
relatively slow, significant effort in the international standards community
has been devoted to the development of the Standard for the Exchange of
Product Model Data (STEP) for computer-aided design (CAD)/computer-
aided manufacturing (CAM) and other types of product data. DOD and
relevant industry groups must remain actively engaged in such commercial
OCR for page 112
112
MODELING AND SIMULATION IN MANUFACTURING
standards efforts to ensure that DOD needs for such things as enhanced
CAD/CAM standards are considered in the standards development process.
Significant progress has also been made on standards for information
exchange, such as the extensible markup language (XML). Although
standards are often a moving target during their evolutionary development,
it is important for the DOD community to participate in their development
and attain maximum advantage from commercial efforts.
Because of lack of commercial interest, there will be some standards
for which the DOD community must take the lead in development.
Examples of this in the performance simulation community include the
high level architecture (HLA), now approved as IEEE Standard 1516, and
the synthetic environment data representation and interchange specification
(SEDRIS), both developed with leadership and financial support by
DMSO. Existing standards organizations, such as the Simulation
Interoperability Standards Organization (SISO), can be leveraged to evolve
simulation-oriented standards. Although there is hope that standards such
as these might become predominantly supported and pervasively adopted
by the commercial marketplace, their importance to the acquisition of
DOD systems dictates that DOD be sufficiency involved to ensure their
sustenance.
An area in which less progress has been made is standard semantics
for data elements used in DOD acquisition-related models and simulations.
Work on standard nomenclature, definitions, and units of measure is
needed to ensure valid substantive interactions among models and
simulations as use of federated M&S becomes more prevalent.
Standards that build from current standards are emerging in severa]
M&S areas. Development of such standards should be encouraged.
Related software and system engineering standard developments, such as
those described in Chapter 5, should be monitored closely.
Desired End State
A significant government role is needed to nurture common models
and tools that industry on its own would not develop. The benefits derived
from proactive leadership by DOD in developing such infrastructure for
the common good would include these:
· Fully exploiting the potential of modeling and simulation to
greatly improve products, perfect processes, reduce design-to-
manufacturing-to-fielding cycle time, and reduce system
realization costs;
OCR for page 113
CONCLUSIONS AND RECOMMENDATIONS
113
Developing and managing the knowledge of theory, science,
engineering, and craft required for successful M&S
developments;
Growing and training the large numbers of professionals, ranging
from specialists in M&S infrastructure to M&S researchers, who
are needed to support the wide array of activities and programs
that SBA entails.
The various components of infrastructure potentially constitute a self-
sustaining or autocatalytic process. Each of these constituents is necessary
to make the whole cycle work, and if they are present, each reinforces the
other. A test of the successful implementation of the recommended
infrastructure elements is a noticeable increase in the number, and
capabilities, of M&S professionals; a significant increase in the theory,
methods, and best practices available for M&S projects; and ultimately,
greatly reduced times to develop and acquire the best systems for the
nation's defense.
USE OF MODELING AND SIMULATION IN
ACQUISITION AND MANUFACTURING
Overall Recommendation. Process improvements should be undertaken
to better support integration of M&S within DOD's system acquisition
process.
Recommendation. M&S use should be expanded in the
concept exploration phase. M&S and SBA in DOD must have a
scope that includes not just "building the thing right," but also
"building the right thing."
Approaches to SBA to date have focused on building systems once
the need for those systems has been identified, to the exclusion of
identifying what system should be built. In initial DOD applications of the
principles of SBA, attention has been focused on selected programs (e.g.,
Joint Strike Fighter (JSF), or the Crusader artillery). In each of these
activities, the government/industry team has creatively employed
integrated process teams (IPTs) and M&S tools to facilitate intraprogram
coordination for example, by linking the design and test and evaluation
(T&E), the operations and maintenance (O&M), and the training
communities. Although results are only preliminary, there is some
evidence to suggest that the quality performance of these programs is
OCR for page 116
116
MODELING AND SIMULATIONIN MANUFACTURING
.
Industry agreements on acceptable practices for protection of
electronically stored proprietary data that are made selectively
available to the government.
Development of such guidelines and best practices is probably most
effectively done by a working group of DOD and industry acquisition
professionals, with international representation as appropriate, convened
specifically for this purpose.
Recommendation. A deliberate effort should be undertaken to
define how M&S is to be integrated into the DOD systems
acquisition process, including use of the maturity of the
simulation support plan (SSP) as an element in milestone
decision reviews and establishing specific evaluation criteria.
No single acquisition program has yet demonstrated a comprehensive
use of SBA processes. While program-specific approaches to SBA are in
use in some DOD programs, no comprehensive, cross-program approach is
yet in use. SBA literature addresses the use of M&S in five areas:
1. To define and analyze the requirement for the system,
2. To engineer the system,
3. To define the system development process,
4. To test the system, and
5. To support system training.
In support of SBA, the program manager is required to develop a
simulation support plan (SSP) to define precisely how M&S will be used in
the five areas listed above. The maturity of an acquisition program may be
evaluated by its use of M&S to support these areas. The maturity of the
SSP and its implementation should be made an element of the milestone
decision reviews, along with the achievement of key performance
parameters. Specific criteria should be established that will serve as SSP
evaluation criteria to be applied at the milestone reviews. The criteria
should address the manner in which M&S has been applied to establish the
acquisition program baseline. The criteria should establish SSP maturity
metrics that are reported into a system such as the consolidated acquisition
reporting system (CARS) for cross-program review.
The Director, Defense Research and Engineering (DDR&E) should
charge the Functional Acquisition Area Council of the DOD Executive
Council on Modeling and Simulation (EXCIMS) to oversee development
of an SBA maturity guide to be applied to major defense acquisition
OCR for page 117
OCR for page 118
OCR for page 119
OCR for page 120
OCR for page 121
OCR for page 122
OCR for page 123
OCR for page 124
OCR for page 125
OCR for page 126
Representative terms from entire chapter:
pilot efforts
CONCLUSIONS AND RECOMMENDATIONS
117
programs. The SBA maturity guide should establish the criteria to assess
SSP maturity and M&S integration and should use exit criteria to assess
the status of weapons system programs that will be evaluated at each
program milestone. The exit criteria will assess the degree to which M&S
integration planning has been accomplished and the degree to which M&S
integration has occurred in the system acquisition process. Application of
M&S to the system acquisition process will be evaluated against known
areas of opportunity and success in applying M&S to system acquisition
activities where its use was planned.
The committee found that performance modeling or system
effectiveness simulations are not sufficiently integrated with product
modeling or engineering simulations, which are used to determine how to
build a system.
The DOD acquisition instruction document' requires that the program
manager apply a systems engineering process to translate operational needs
and/or requirements into a solution that includes design, manufacturing,
test and evaluation, and support processes and products. The key systems
engineering activities that must be performed include requirements
analysis, functional analysis/allocation, design synthesis and verification,
and system analysis and control. The document goes on to define 20 areas
that must be considered as part of the systems engineering process. This
high-level discussion ofthe systems engineering process, to be applied for
major systems being acquired by DOD, should serve as the basis for
tailored integration of M&S into the systems engineering processes applied
to acquisition programs.
DOD has developed the Handbook of Work Breakdown Structures
for Major Defense Acquisition Programs and Major Automated
information System Programs. This handbook is out of date and is not in
accord with guidance provided by the previously discussed document. The
handbook should be updated and should provide greater detail concerning
the integration of M&S into the systems engineering portion of the
performance work breakdown statement.
DOD has undertaken an effort, supported by the Software
Engineering institute (SEI), to develop and promulgate the capability
maturity model integration (CMMI) for application to the development of
software-intensive systems. A similar effort should be undertaken to define
the structure of systems engineering processes with detailed descriptions of
how M&S may be integrated into these processes. Systems engineering
maturity levels should be defined across the other six domains of DOD
systems engineering in addition to the definition for software-intensive
systems. Furthermore, a library of process artifacts, perhaps an addition to
' Available at
118
MODELING AND SlMULA TION IN MANUFACTURING
the Defense Acquisition Deskbook, should be established to serve as a
repository for M&S integration plans and related systems engineering
documents that may be referenced by new or existing programs.
Recommendation. Incentive should be created and
implemented for DOD program managers to adopt best
practices for the use of M&S in acquisition and throughout the
life cycle of military systems.
The committee found that DOD program managers have no
incentives to apply resources to interprogram aspects of M&S and SBA-
type processes. In fact, they currently have disincentives to apply
resources to M&S activities that might benefit later phases of their own
programs as weld as other programs. Because the typical tenure of a
program manager is relatively short compared with the life of the program,
investments in M&S made early in a program (such as those that might
reduce total ownership cost or improve supportability) will not produce
returns until years or even decades dater, long after the program manager
has departed. In an environment with severely limited funds, a natural
tendency exists not to invest in activities whose benefit will not be realized
during the program manager's tenure. This is even more the case when the
benefits wild be realized by a different, perhaps yet uninitiated, program.
Several prior M&S/SBA studies have recognized the need for
incentives, although few workable solutions have emerged. The problem is
undoubtedly difficl]t, and this report does not offer detailed
recommendations for its resolution. Perhaps the most necessary incentive
is for enlightened senior acquisition leadership to encourage and provide
positive recognition for individual program managers for adopting
M&S/SBA best practices that are emerging but are not yet
institutionalized. This could be realized by utilizing the aforementioned
acquisition maturity mode] to ensure use of these best practices before a
program is a]]owed to proceed to the next phase of the DOD system
. . .
acquisition process.
Recommendation. Pilot efforts should be defined and
undertaken as a part of advancing the use of and experience in
M&S. They should be sponsored at the ]eve] of the Office of
the Secretary of Defense (OSD) to explore and document the
benefits of co]]aborative acquisition of systems enabled by
advances in M&S and information technologies.
A sma]] number of well-defined pilot efforts should be undertaken.
These pilot efforts should place special emphasis on exploring potential
CONCLUSIONS AND RECOMMENDATIONS
119
cross-program benefits of M&S and information-technology-enabled
collaborative acquisition, and should be set up in a sequence with time
phasing that leads to exploration of system-of-systems issues. This may
lead to the selection of pilot efforts that are aligned with programs
comparatively close in the relative phasing of their acquisition schedules.
Each pilot effort should be constructed to permit the collection of data on
specific metrics in order to estimate potential benefits in performance, cost,
and schedule that could result from more widespread application. They
should also be constructed so as to guide technical development of M&S
and SBA concepts, permitting the necessary risk resulting from an
emphasis on learning, and must persist long enough to ensure that the
desired learning is realized.
Because the nature of activities in DOD increasingly involves joint
services, these pilot efforts should be sponsored at the OSD level.
Although the precise mechanism for managing programs should be
decided by the Under Secretary of Defense for Acquisition, Technology
and Logistics (USD (AT&L)), it is recommended that DMSO and the
Defense Systems Management College (DSMC) participate in an oversight
role to ensure that lessons learned from the pilot efforts are shared
effectively with the M&S and acquisition communities.
The committee found that no single acquisition program has
demonstrated comprehensive use of SBA, and that inadequate resources
have been allocated to support the vision for use of M&S in military
systems. Pilot efforts, if properly focused, can provide the following:
A means of exploring new concepts before large-scale
investments are made,
Opportunities for transition of research,
Identification of areas in which additional research is needed, and
Evidence of benefits and other lessons learned to subsequent
users.
The committee notes that recommendations for pilot programs or
other forms of experimentation were made in at least five previous
M&S/SBA-related studies, none of which appear to have been fully acted
on. The recommendation for pilot programs remains relevant today.
The committee found evidence that some individual acquisition
programs, such as the JSF and Future Combat Systems (FCS) programs,
are planning to make investments in M&S and information-technology-
enabled collaborative acquisition efforts, which can be expected to provide
some valuable lessons learned to other programs. These efforts are to be
encouraged. Necessarily, individual programs must focus on efforts
120
MODELING AND SIMULA TION IN MANUFACTURING
expected to directly benefit their own system performance, cost, and
schedule objectives, making it difficult to explore cross-program issues and
to develop and collect data on well-defined metrics.
Desired End State
The successful implementation of these recommendations on the use
of M&S in acquisition and manufacturing will only be recognized by long-
term efficiencies gained in the acquisition of systems. The near-term
implementation of pilot efforts should result in the establishment of metrics
and areas for further work. The metrics so established will aid in defining
best practices to be followed during acquisition, the use of which can be
promoted and rewarded. Success of the expansion of M&S use in the
concept exploration phase to "build the right thing" will be measured over
time by the degree of reduction in cancellation of systems because of
decreasing relevance. Finally, the better integration of M&S into the
acquisition process and the resolution of ownership rights to facilitate
collaboration and reuse can be measured by the realized decrease in system
acquisition times and by improved avoidance of cost overruns.
CULTURE AND HUMAN ISSUES
Overall Recommendation. DOD must provide leadership to initiate, support,
and sustain a cultural change in the acquisition process if it is to be enabled by
modeling, simulation, information technologies, and development of
appropriate intellectual capital in these fields.
Recommendation. Concerted actions should be taken to
fundamentally transform the current acquisition culture in DOD
into one characterized by collaboration, cumulative learning,
agility, risk tolerance, teaming from failure, and appropriate
rewards and penalties. The following steps should be taken:
· DOD's Senior Executive Council should set the direction by
creating a vision of the desired acquisition culture and
formulating and issuing policy consistent with that vision.
· DOD's Business Initiative Council should institute appropriate
incentives for program managers; address issues of data and
model ownership, proprietary information, and intellectual
property; identify and address policy, legal, and organizational
barriers that inhibit SBA activities; identify and address policy
issues associated with the potential international dimensions of
CONCLUSIONS AND RECOMMENDATIONS
.
121
SBA; provide needed resources to implement SBA programs,
and ensure consistency among service implementations of SBA.
DOD's Business Initiative Council should appoint agents of
cultural change to develop and implement a strategy to bring
about the needed change in culture by implementing and
enforcing rewards, creating a best practices manual, training
stakeholders, and convening conferences.
Based on numerous reports from respected sources, as described in
Appendix B. the committee reaffirmed that the current acquisition culture
in DOD is a fundamental impediment to effective implementation of SBA.
The committee's visits, briefings, discussions, and assessments verified that
the current cultural climate in DOD acquisition is characterized by
organizational and functional stovepipes; a failure to share cross-phase,
cross-program knowledge; bureaucratic inertia; risk aversion; limiting of
goals to minimize the probability of failure; and a lack of appropriate
incentives. There is a need to transform this culture into one that is
characterized by cross-function and cross-program collaboration,
cumulative learning, agility, risk tolerance, learning from failure, and
appropriate rewards and penalties.
The committee is under no illusion that such a cultural transformation
will be either easy or rapid. However, if the right individuals in DOD play
appropriate leadership roles and initiate an appropriate mix of actions,
meaningful cultural change can be initiated and sustained.
First, the highest level of senior policy makers must set the direction.
The appropriate organization to perform this role is DOD's Senior
Executive Council. The council is led by the Deputy Secretary of Defense
and consists ofthe service secretaries and the USD(AT&L) (OSD, 2001~.
The council was created in June 2001 to function as a business board of
directors for DOD (DOD, 2001 a). This organization must create a vision of
the culture that it wants to inculcate in the acquisition community, and then
formulate and issue policy consistent with that vision.
At the next level in the hierarchy, DOD has announced the creation of
the Business Initiative Council, which is headed by the USD(AT&L) and
composed of the service secretaries and the Vice Chairman of the Joint
Chiefs of Staff. The organization is to "recommend good business
practices and implement cost savings that could offset the funding
requirements for personnel programs, infrastructure recapitalization,
equipment modernization and transformation initiatives" (DOD, 2001b).
Consistent with that charge, the Business Initiative Council should
institute appropriate incentives for program managers; address and
ameliorate issues of data and model ownership, proprietary information,
and inte]]ectua] property; identify and address policy, legal, and
organizational barriers that inhibit SBA activities; identify and address
122
MODELING AND SIMULA TION IN MANUFACTURING
policy issues associated with the potential international dimensions of
SBA; and provide necessary resources to implement SBA programs. In
addition, the council should ensure that individual service implementations
of SBA are sufficiently consistent that it will not be necessary for industry
to deal with four (or more) distinctly different SBA infrastructures and
processes. It would be desirable if the Business Initiative Council could
formulate a grand challenge for the acquisition community that would
encourage the military branches to push the boundaries of SBA.
The Business Initiative Council should also appoint, anoint, and
empower agents of cultural change at all levels in the SBA process. These
agents would serve as champions and perform such functions as
developing strategies and plans for SBA, capturing lessons learned from
SBA activities, and convening meetings to support articulation of goals
and to stimulate sharing of lessons learned. These agents of cultural change
should develop and implement a strategy to bring about the necessary
change in culture. The following represents a partial enumeration of
actions that might be taken to implement such a strategy:
· Reward structure. Implement and enforce rewards that encourage
adherence to SBA principles. These might include promoting
successful practitioners and providing adequate resources for
innovative applications of SBA.
Best practices. Create a living manual of best practices for SBA.
This manual would capture lessons learned from both successes
and failures.
Education and training. Institute appropriate education and
training for all of the stakeholders in the process. This would
include creating and delivering short courses on SBA principles
for senior decision makers (drawing on the evolving list of best
practices) and ensuring that both military personnel and civilians
in the acquisition process are exposed to SBA education and
training at all phases in their careers.
Annual conferences. Currently, multiple conferences on SBA are
convened irregularly. Over time, these should evolve to an annual
joint service/combined conference using the interservice/industry
training, simulation, and education conference (I/ITSEC)2 as a
model. Consistent with the I/ITSEC model, this conference should
feature participation by senior decision makers (to articulate
policy, learn state-of-the-art practice); include all stakeholders in
the acquisition process (e.g., users, developers, manufacturers,
support personnel, trainers); demonstrate key acquisition
2 Information concerning this conference is available at .
CONCL USIONS AND RECOMMENDA TIONS
technologies and infrastructure; provide tutorials to educate
participants on the state of the practice in SBA; and facilitate
extensive networking among the participants.
Recommendation. DOD should take the lead in collaborating
with academia and industry to build the intellectual capital
needed to implement SBA.
Currently, the acquisition community lacks the intellectual capital
required to implement SBA. To redress this deficiency, a number of
orchestrated steps must be taken.
Recommendation: Create a center of excellence for M&S in
SBA. This resource would help create and promulgate the
desired acquisition culture and enhance DOD's ties to the
academic community. Additional steps include:
· Academic degree programs in M&S. Ongoing efforts to develop
academic degree programs in M&S should be supported by way
of research funding related to SBA-type topics and the
involvement of university faculty in commercial and DOD-
sector M&S.
Multiuniversity consortium. A multiuniversity consortium
should be set up to continue the development of M&S education
applicable to SBA. These efforts should be made in concert
with the university research efforts described in these
recommendations.
Mentoring program. A mentoring program should be established
to take advantage of the insights developed by experienced
practitioners of SBA. These might include individuals in the
center of excellence.
Career-long learning. Individuals should be encouraged to
maintain and expand their proficiency and expertise in M&S
through continuing education programs and attendance at
relevant conferences, meetings, and workshops.
.
.
.
123
Although a variety of courses related to M&S is offered on most
campuses, these are largely in the context of particular disciplines. For
example, in a typical university, senior undergraduate and graduate courses
are offered by departments such as electrical and computer engineering,
management information systems, and industrial engineering.
Unfortunately, these courses are too narrow in viewpoint and scope to
serve the needs of M&S as a discipline capable of meeting the challenges
124
MODELING AND SIMULA TION IN MANUFACTURING
of SBA. Therefore, at this time, offerings in education and training are not
capable of meeting the current and future demand, and there must be
significant developments in this regard. Previous panels (NRC, 1997a,b;
SBAISG, 2000) have concluded that:
What it is to be an M&S professional must be defined, and
accreditation mechanisms must be developed;
University degrees at the undergraduate and graduate levels must
be defined and institutionalized; and
Professional development, distinct from university degree
programs, must be an essential component of the full education
and training package.
A small but growing number of academic institutions have
recognized the situation described above and the opportunities it presents
for increasing student enrollments (Fujimoto, 2000b; Sarjoughian and
Cellier, 2001~. Individuals and small groups in these institutions have
usually had to overcome the resistance of colleagues and the indifference
of university administrations to establish new centers and degree programs
centered on M&S as a discipline in its own right. The Defense Acquisition
University is also taking steps to integrate M&S more fully into the
education of acquisition personnel and program managers.3 An
accreditation certificate program, Certified Simulation Professional, has
been established that will be available to help assess the degree of
capability that M&S students and working professionals have achieved.4
The M&S professional certification program may be an important
motivator for developing M&S instruction in post-university professional
programs and even more widely in restructuring traditional graduate and
undergraduate curricula to better address the need for M&S professionals.
The stability of such initiatives is not guaranteed. A healthy
enrollment of students and a significant input of research funding will be
required to enable these programs to flourish and to spread to other
institutions. Thus, the committee endorses the following recommendations
of other panels.5
.
Universities need to characterize the discipline of M&S and
clearly delineate the discipline from the neighboring ones such as
systems engineering and computer science/ engineering;
3 Randy Zittel, Defense Systems Management College. 2001. Presentation to this study
committee.
4 Information on this effort is available at
s These are described more fully in Appendix B.
COlICL USIONS AND RECOMMENDS LIONS
125
Universities must work with other sources of professional training
to work out areas in which each should concentrate and identify
combinations of offerings that work as a coherent whole; and
Universities must work with funding agencies to establish
programs of research and education needed to advance the field
and to obtain adequate funding for their implementation.
Further, the committee recommends that industry, in the form of
M&S-based companies and corporations using M&S, should do the
following:
.
Work with universities to characterize the current and future types
of M&S professionals that they will hire and what the educational
background of these professionals should be (Yurcik and
Silverman, 2000~;
Coordinate their education and training programs with those of
universities for a coherent set of offerings; and
Collaborate with universities to establish research teams that can
respond to requests for proposals from government funding
sources.
Desired End State
The desired culture for realization of substantial gains in use of M&S
in manufacturing and acquisition is one in which government, academia,
and industry would seek regularly to collaborate in defining needs,
identifying and funding research programs to further M&S technologies to
fulfill those needs, and implementing M&S in SEA. In that desired
culture, individuals would be encouraged to take risks in application of
M&S and would share [earnings from both successes and failures in
regular forums. It would be the responsibility of program managers to
push the envelope in developing and implementing M&S technologies.
Academic degree programs would exist recognizing M&S as a discipline,
with curricula meeting the needs of M&S in manufacturing and
acquisition. Centers of excellence would provide the community with
resources and leadership.
