|
||||||||||||||||
|
||||||||||||||||
The following HTML text is provided to enhance online readability. Many aspects of typography translate only awkwardly to HTML. Please use the page image as the authoritative form to ensure accuracy. 2
|
||||||||||||||||
 |
 |
Page 16 |
 |
 |
 |
|
| ||||||||||||||||||||||||||||||||
|
|
|||||||||||||||||||||||||||||||
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 16
Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs
2
Findings and Conclusions
As summarized in Chapter 1, the committee deliberated on the results from the four data sources to arrive at the findings and conclusions contained in this chapter. The findings and conclusions are based on information derived from the survey responses, interviews with Air Force and DOD personnel and other stakeholders, earlier studies, and the committee members’ expertise. The committee spent considerable time deliberating to ensure that the findings indeed represent what it had heard. The conclusions then represent a committee consensus of a generalization of the findings. The recommendations in turn (see Chapter 3) resulted from integrating and synthesizing the findings and conclusions in a way that the committee believed would be most responsive to the Statement of Task.
The matrix of reviews that was constructed to help classify the number and types of known reviews, their purpose, and their target audiences in order to identify opportunities for streamlining, integrating, and/or consolidating reviews is shown in Table 2-1, with further information included in Appendix E. The reviews are of four types (see third column of the matrix):
Milestone/programmatic reviews. These reviews are conducted by the milestone decision authority (MDA) (the Defense Acquisition Executive (DAE) for acquisition category (ACAT) ID programs), with support from the Office of the Secretary of Defense (OSD) staff. They provide the basis for the MDA to decide whether to allow a new program to be initiated or an ongoing program to proceed to the next phase. Of the four types, reviews of this type require by far the greatest number of prereviews (reviews conducted by functional offices and intermediate management levels on the way to the MDA review). These reviews are listed in approximately
OCR for page 17
Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs
the order they occur leading up to the ultimate Defense Acquisition Board (DAB) review.
Periodic oversight reviews. These reviews are conducted at a regular frequency—annually or quarterly—to allow the DAE, the Service Acquisition Executive (SAE), or the program executive officer (PEO) to monitor progress. They are listed alphabetically.
Ad hoc reviews. These reviews are initiated by an organization outside the program office either in response to a problem or, in the case of some major programs, periodically (e.g., by the GAO). They are listed alphabetically.
Technical or engineering reviews. In contrast to the external reviews mentioned above, these reviews are generally conducted internally by the program itself, with some support from Service functional assets. They are a principal means by which the program office manages the technical and fiscal execution of the program. They are listed in approximately the order they occur over the life of the program.
The matrix (Table 2-1) summarizes the various types of reviews typically conducted over the lifetime of a program. It names 31 formal reviews of four types, although not all of them or even all types may be conducted for every program. At least four of the reviews require unique documents and 10 have been identified by the committee as duplicating or partially duplicating other reviews.
The matrix does not list every possible ad hoc review. Nor does it list the prereviews or prebriefs generated by these formal reviews, since it was not possible to determine the number of such reviews or prebriefs. Because no data or metrics are required or collected on reviews by DOD in general, or by the Air Force in particular, it was not possible to determine the overlap or duplication of different reviews. (Of note, although no historic information on the number of prereviews or prebriefs is available, interview comments and the committee members’ experience lead one to believe that major reviews at the level of the Joint Requirements Oversight Council (JROC), DAB, and PEO are generally accompanied by multiple (3 to 5) prebriefs or prereviews.)
In the program review process depicted in Figure 1-4, it can be seen that there are 6 MS or MDA reviews in a typical program, plus 6 overarching integrated product teams (OIPTs) reviews before those, plus 6 Air Force Review Board (AFRB)/acquisition strategy panel (ASP) reviews before each of those, plus 10 or 11 mandatory technical reviews at the program level, plus 4 periodic program support reviews (PSRs) for each phase of the life cycle, plus one operational test readiness assessment (OTRA) before IOT&E, plus annual configuration steering board (CSB) reviews (notionally 7), plus annual PEO-level sufficiency reviews (notionally 7), plus at least 2 technology readiness assessments/manufacturing readiness assessments (TRAs/MRAs). This suggests in excess of 50 reviews just at the program level and above without counting
OCR for page 18
Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs
TABLE 2-1 Program Review Matrix
Review
Name
Type
Purpose
Frequency
OPR/Customer
ASP
Acquisition Strategy Panel
Programmatic
Formalize acquisition strategy
MS
SAF/AQ
AFROCC
Air Force Requirement for Operational Capabilities Council
Programmatic
Validate requirements
MS
A5R/CSAF, SECAF
JROC
Joint Requirements Oversight Council
Programmatic
Prioritize Joint requirements
MS
VCJCS/DAB
AFRB
Air Force Review Board
Programmatic
Ready for MS or OSD
MS
SAE
IPA
Independent Program Assessment
Programmatic
Risk assessment
MS
SMC/MDA
PSR
Program Support Review
Programmatic
Risk assessment
MS
Director SSE/USD(AT&L)
IIPT
Integrating IPT
Programmatic
Support OIPT
MS
PM/OIPT
OIPT
Overarching IPT
Programmatic
Ready for DAB
MS
Dir PSA, DASD(NII)/DAB
DAB
Defense Acquisition Board
Programmatic
Advise USD(AT&L)
MS
USD(AT&L)
CSB
Configuration Steering Board
Periodic oversight
Review requirements changes
Annual
SAE/USD(AT&L)
DAES
Defense Acquisition Executive Summary
Periodic oversight
Ongoing performance
Quarterly
PM/DUSD(A&T)
PEO/SR
PEO Sufficiency Review
Periodic oversight
Executability confidence
Annual
PEO
OCR for page 19
Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs
Presenter
Stakeholders
Prereview Briefings
Product/Output
Unique Documentation?
Duplication
PM
SAF/AQ
NDA
Approve acquisition strategy plan
Yes
Partially
A5 staff, MAJCOM representative
Air staff, AF Secretariat
FCBs
Validate requirements
No
NDA
J8 director, Service representative
Services, warfighters
Joint Requirements Board, Panel
JROCM
Yes
No
Requirements representative
PM, user, OSD, service, KTR
Unknown
Readiness to proceed
No
No
PM, staff
DSAB, staff support
Unknown
Briefing and report
No
Partially
PM, PO & KTR SMEs
Dir SSE, USD(AT&L), PM
Coordination meetings
Brief findings, recommendations
No
Partially
Varies
PM, user, OSD, service, KTR
None
Report to OIPT
NDA
Partially
PM; OSD staff
PM, PEO, OSD, Joint Staff
IIPT or equivalent, staff
Report w/recommendations
No
No
OIPT chair, PM, others
USD (AT&L), DAB members
OIPT
Acquisition Decision Memorandum
Yes
No
PM
SAE, USD(AT&L), JCS
Unknown
Approve requirements change
Yes
NDA
PM (with SAE)
PM, SAE, OSD staff
Unknown
Action items
Yes
No
PM, SMEs
PEO, PM, ACE, SMEs
Unknown
Briefing, PoPs update
No
NDA
OCR for page 20
Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs
Review
Name
Type
Purpose
Frequency
OPR/Customer
AFAA
AF Audit Agency
Ad hoc oversight
Varies
Varies
AFAA/SECAF
DOD IG
DOD Inspector General
Ad hoc oversight
Varies
Varies
DoD IG/SecDef
GAO
Government Accountability Office
Ad hoc oversight
Cost, schedule, performance
Varies
GAO/Congress
Other ad hoc
Varies
Varies
Varies
Varies
TRA
Technology Readiness Assessment
Technical
Executability confidence
MS
DDR&E/MDA
ASR
Alternative System Review
Technical
Ready for technology development
MS A
Engineering/PM
SEAM
Systems Engineering Assessment Model
Technical
Validates SE process
Varies
AFCSE/PM
SRR
System Requirements Review
Technical
Executability confidence
One time
Engineering/PM
SDR
System Design Review
Technical
Replaced by the SFR
One time
Engineering/PM
SFR
System Functional Review
Technical
Ready for prelim design
One time
Engineering/PM
PDR
Preliminary Design Review
Technical
Ready for detailed design
One time
Engineering/PM
IBR
Integrated Baseline Review
Technical
Align program expectations
Varies
PM
LHA
Logistics Health Assessment
Technical
Logistics health
Unknown
Unknown; currently AAC
MRA
Manufacturing Readiness Assessment
Technical
Executability confidence
One time
DDR&E/MDA
CDR
Critical Design Review
Technical
Ready for fabrication
One time
Engineering/PM
OCR for page 21
Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs
Presenter
Stakeholders
Prereview Briefings
Product/Output
Unique Documentation?
Duplication
PM, staff
SECAF, staff, AFAA office
Coordination meetings
Report
No
Partially
PM, staff
Varies - wide-ranging
Coordination meetings
Written report w/recommendations
No
Partially
Varies - often PM or rep
Varies - wide-ranging
Coordination meetings
Written report w/recommendations
No
Partially
Varies
Varies
Unknown
Varies
Varies
Yes
AF ST&E
PM, S&T, DDR&E, DAE
Unknown
Tech Readiness Levels/Plan
NDA
Yes
PM, PO & KTR SMEs
PM, User, KTR
Unknown
Rationale for preferred alt
No
No
PM/Engineering
AFCSE, PM
None
Assessment reports
No
Partially
PM/Engineering
Engineering, user, KTR
Unknown
Satisfy exit criteria
No
No
PM/Engineering
PM, contractor
Unknown
Satisfy exit criteria
No
No
PM/Engineering
Engineering, user, KTR
Unknown
Satisfy exit criteria
No
No
PM/Engineering
Engineering, user, KTR, DAE
None
Satisfy exit criteria
No
No
PM
PM, contractor
NDA
Mutual understanding of program baseline
No
Partially
Unknown
PM, user, AF logisticians
Unknown
Log/sustainment assessment
Unknown
Partially
PM/Engineering
PM, S&T,
Unknown
Mfg Readiness Levels/Plan
NDA
NDA
PM/Engineering
Engineering, User, KTR, DAE
None
Satisfy exit criteria
No
No
OCR for page 22
Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs
Review
Name
Type
Purpose
Frequency
OPR/Customer
TRR
Test Readiness Review
Technical
Ready for testing
One time
Engineering/PM
PRR
Production Readiness Review
Technical
Ready for production
MS C, FRP
Engineering/PM
OTRR
Operational Test Readiness Review
Technical
Approve program OT readiness
One time
SAF/AQ
SVR
System Verification Review
Technical
Verifies performance
MS C
Engineering/PM
FCA
Functional Configuration Audit
Technical
Satisfies contract
One time
Engineering/PM
PCA
Physical Configuration Audit
Technical
Satisfies contract
One time
Engineering/PM
Frequency: MS, this review takes place in support of or in conjunction with each formal program milestone (A, B, C, etc.); MS A, this review takes place in support of or in conjunction with formal program milestone A; MS C, this review takes place in support of or in conjunction with milestone C; MS C, FRP, this review takes place in support of or in conjunction with milestone C and the full rate production decision.
Duplication: A review/meeting exactly the same as another review/meeting. By this definition, the use of the same material at more than one meeting does not constitute a duplication of the meeting. The
the prebriefs to staff functional principals and others as noted in the preceding paragraph.
Although the data and information received did not allow the committee to determine specific resources required to accomplish each review, answers to Question 1.5 in the survey and information gathered from the interviews indicated that 10-30 percent of a PM’s time is spent supporting reviews. The committee found that there is little consistency in the way reviews are conducted and concluded that opinions on the contribution of specific reviews to successful acquisition varied widely.
The committee believes that the Air Force could improve the effectiveness of its program review effort and reduce the burden on PMs by thoughtfully combining and scheduling reviews. The committee looked at the policies and processes of the National Aeronautics and Space Administration (NASA), the Department of Energy (DOE), industry, and the other military services. Although review
OCR for page 23
Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs
Presenter
Stakeholders
Prereview Briefings
Product/Output
Unique Documentation?
Duplication
PM/Engineering/Test
Test IPT, User, KTR, DOT&E
None
Prepared for testing
No
No
PM/Engineering
KTR, Subs
None
Requirement met, production ready
No
No
PM
SAF staff, DOT&E
NDA
ASAF(A) approval of program’s OT readiness
Yes
Partially
PM/Engineering
User, KTR
None
Requirements met, funding adequate
No
No
PM/Engineering
PM, Engineering, KTR
None
Audit report
NDA
No
PM/Engineering
PM, Engineering, KTR
None
Audit report or completion
NDA
No
degree of duplication shown in the Duplication column is subjective and is based on review documentation, interviewee comments, and survey results. Given that no review, let alone its process owner, requires any data collection on duplication, coordination, value added, and so on, the conclusions are not rigorous. Review duplication attributes: Yes, the review was judged to be a duplication; No, the review was judged not to be a duplication; Partially, the meeting was judged to be a partial duplication; and NDA, the review is too new or had no data available to allow a judgment on duplication.
structures and practices exist, the differences in implementation make cross-community consolidation or streamlining a significant challenge. The information collected from experienced PMs in the U.S. Army and the U.S. Navy, as well as the U.S. Air Force, and from industry contributed to the set of best practices reported in the recommendations.
REVIEWS HAVE BOTH BENEFITS AND COSTS
The committee found that program and technical reviews have both benefits and costs. In general, reviews provide technical and programmatic support to successfully execute acquisition programs, to inform decisions, to share awareness, and to engender program advocacy. In their answers to Survey Question 2.3, PMs said that reviews facilitated program execution as well as problem discovery and resolution at all levels of the acquisition enterprise, including industry. That said,
OCR for page 24
Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs
support for the increasing number of program reviews and associated prebriefs is costly, as is the lost productivity and attention of both government and industry. For example, DODI 5000.02 of December 2008 calls for systems engineering across all phases and mandates technical reviews. The new regulation mandates competitive prototyping of the system or key system elements during the technology development (TD) phase, and a preliminary design review (PDR) must be conducted for each of the candidate designs, followed by a PDR report to be provided to the MDA either in the TD phase supporting milestone B or, if afterward, a separate MDA review during the engineering and manufacturing development (EMD) phase.
Although some such overhead is undoubtedly necessary, excesses can distract PMs from their primary focus, which is managing their program’s technical and business progress. The findings and conclusions of the committee focus on two key areas: The first is the execution of reviews (sequencing, timing, participation), and the second is their planning.
EXECUTION OF REVIEWS (SEQUENCING, TIMING, PARTICIPATION)
The committee’s investigation confirmed that the number of reviews is growing. The committee recognizes that such reviews are generally valuable for sharing knowledge and serve as “stage gates” for the governance and control of programs, including managing risk. While doing this, however, they most certainly add to program costs, and they also draw management’s attention from the main program effort. Of most concern is that the proliferation of reviews does not appear to have had a positive effect on program cost and schedule outcomes. The committee also realized that there is a significant amount of preparation and coordination for reviews, both vertically in their conduct at multiple levels of responsibility and horizontally across adjacent staff offices. As such, merely depicting the individual reviews does not sufficiently capture the amount of time and effort spent by staff in preparing for and coordinating the reviews.
The committee found that the many disparate concerns of higher-level staffs had an impact on the program manager. For ACAT I programs, many of the written-out responses1 to Survey Questions 4.1-4.4 described DOD staff as a stove-piped bureaucracy, where domain “czars” have purview over a breadth of programs (by virtue of the OIPT structure or their membership on the DAB) but are not horizontally integrated from the standpoint of knowledge sharing or synergy. This means not only that the PMs have to prepare separate information
1
The qualitative write-in comments from the PMs have not been included in this report because several of the comments were written in such specific detail that the authors of the comments could be identified by people familiar with their programs. This would violate the privacy protection methodology for the survey, so the data are not included.
OCR for page 25
Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs
briefs for these higher-ups, but also that the information provided to them is not fully integrated across these domains by the OSD staff for optimal decision making by the MDA.
Programs and technical reviews are often not being optimally synchronized with program events. Speakers gave examples of requirements reviews being conducted after contracts had been awarded and other similar occurrences of inappropriate sequencing. The committee was briefed on emerging acquisition policy with its emphasis on pre-Milestone A activities to ensure a better understanding of alternative technical requirements. The committee believes there is ample opportunity for the Air Force and DOD to improve the timing of these reviews and even to consolidate reviews at the service staff level and below to eliminate redundancy and mitigate the associated burdens and costs.
Finding 1a. A significant share of a PM’s time is spent on preparing for and participating in milestone and other reviews.
Finding 1b. The large number of reviews diverts a PM’s attention from the execution of his or her program.
Finding 1c. Reviews impose significant costs on program leadership teams.
The committee interviewed a number of PMs to obtain their views on the time they spent and the costs associated with carrying out these various reviews. It also sought their assessment of how the conduct of these reviews affects their ability to carry out the day-to-day tasks associated with their programs. The PM survey also asked respondents to address these issues. In every case, those interviewed or surveyed cited significant costs in terms of money and time to carry out the reviews, and most of them also noted an adverse impact on their ability to carry out other PM responsibilities. Sixty-nine percent of survey respondents said they were working 51 or more hours per week (Survey Question 1.4) and on average were able to spend only 46 percent of that time managing their program at wing level or below (Survey Question 1.5). About 20 percent of their time was spent reporting up the chain of command above the wing level (Survey Question 1.5). In addition, many of the respondents cited the need to dedicate hundreds or thousands of program staff and contractor hours to carry out the reviews, with concomitant money and time lost from their everyday duties (based on write-in responses to Survey Questions 3.11-3.14 and 3.26-3.29).
Finding 1d. Many higher-level reviews do not contribute to program execution in proportion to their expenditure of time and effort.
While those interviewed identified benefits they received from reviews that were effectively carried out, a number of survey respondents and interviewees said
OCR for page 26
Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs
they received no significant benefit or improvement to program cost, schedule, or performance (CSP) from the effort. For example, 29 PMs identified what they considered the least beneficial (from the standpoint of program CSP) higher-level review they had been involved in. Eighty percent of them cited no positive impact on CSP from this “least beneficial review” (based on written-in responses to Survey Question 3.24). Some even went so far as to say that the review had actually had a negative impact on CSP by delaying program schedule or increasing program costs in the case of reviews that were ineffectively carried out (based on written-in responses to Survey Question 3.25). It should be noted that the PMs were not questioned on ancillary benefits such as sponsorship, which they may not have viewed as a direct program benefit.
Finding 1e. No one in the Air Force or OSD is responsible for monitoring the number of program and technical reviews or the workload they give rise to or their cost, effectiveness, and cumulative impact on PMs.
Based on the committee’s review of the OSD and Air Force organizational structures and its discussions with senior DOD leadership, no one is responsible for monitoring the direct costs of program and technical reviews, in terms of the time, personnel usage, extra costs, and effects on contractors and the PM.
Conclusion 1. Many reviews add little value and others do not add value in proportion to the effort required. Reducing the number of such reviews or combining them could increase the time available to the PMs to more effectively manage their programs.
Finding 2. The sequencing, timing, and frequency of reviews are often not tied to the program schedule in a manner that most effectively supports the program and its execution.
Forty-three percent of survey respondents commenting on the least beneficial reviews suggested that those reviews could be more effective if they were conducted less frequently and at a more appropriate time in the program’s life cycle (generally earlier) (Survey Question 2.6).
Conclusion 2. Reviews could be more effective if they were sequenced and timed to provide the information needed for program execution. Chapter 3 describes an approach for beginning to achieve this result.
Finding 3a. The program review principals, key stakeholders, and subject matter experts do not always attend program reviews.
OCR for page 27
Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs
Finding 3b. Program and technical reviews are often not attended by the right personnel or, in some cases, are attended by too many personnel.
Many survey respondents noted that more effort should be given to ensuring that the right subject matter experts and appropriate senior officials attend program reviews and that the number of attendees be limited to those who can add value to the meeting (based on responses to Survey Questions 2.6 and 3.15 and written-in responses to Survey Questions 3.30-4.4).
Conclusion 3. Required attendance at program review meetings is not clearly communicated nor is it effectively controlled.
Finding 4a. For some reviews the number of actual reviews and preparatory reviews is excessive and the reviews do not contribute value to the program’s management.
Many of the PMs, both in the survey and in interviews, stated that a proliferation of meetings and premeetings was taking time away from the management of their programs (based on responses to Survey Question 2.6 and written-in responses to survey Questions 3.10 and 3.25). Commenting on the reasons for this increase, a few PMs mentioned the Integrating Integrated Product Team (IIPT) and the NII/AT&L structure. The elimination of IIPT reviews was cited as one factor that led to the need for more individual premeetings with the Joint Staff, program management offices, and OSD, where previously only one meeting was needed. Another reason cited was the recent NII/AT&L reconfiguration that resulted in the sharing of responsibilities between these two offices. Survey respondents cited the amount of time now taken up by such premeetings. One respondent, for example, said “The problem isn’t the review … it’s the numerous premeetings needed to get to the review.”
Finding 4b. Program managers are spending time on multiple reviews with similar objectives.
Many survey respondents believed that selected reviews could be combined (Survey Question 2.7).
Conclusion 4. Streamlining or combining reviews and their associated prebriefs in both the vertical and horizontal directions could increase efficiency.
PROGRAM REVIEW PLANNING
The committee found that in many cases, despite published guidance advocating proper planning, reviews were incompletely planned or conducted. The
OCR for page 28
Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs
potential value of a review cannot be fully realized if objectives are not clearly specified or if the right persons are not in attendance. What information a particular review is supposed to contain or what areas a particular review is supposed to cover are often ill defined or based on presumptions of agendas or issues of the day. Often there are no metrics for assessing the effectiveness of a review, and this can lead to perceptions of disruption and uncertain value. On the other hand, many PMs found that the program support reviews (PSRs) and the independent program assessments (IPAs) were well planned, that they added value, and that they contributed positively to program management.
PSRs and IPAs are comprehensive in nature; have well-defined processes, outcomes, and metrics; are socialized with PMs and staff; are conducted by subject matter experts; and are well documented. A review process that makes timely use of PSRs and IPAs might help to limit the number of reviews needed across the bureaucracy to gather information and make decisions.
Finding 5. The purpose, scope, information needs, key issues, and expected outcomes of many reviews have not been specified.
Survey respondents mentioned a number of ways to improve reviews. Suggestions for improvement to address the problems cited above included narrowing the review’s focus or changing its charter (based on responses to Survey Questions 2.6 and 3.15 and on written-in responses to Survey Question 3.30). PMs noted that they would be asked about issues that had not been previously identified for discussion and would then be required to spend countless hours after the review trying to respond to them. They also said there was no standard approach to how the reviews should be conducted.
Conclusion 5. It is important that program review planning is accomplished in a thoughtful, purposeful manner with a standard approach in order to firmly address the need for communication of the expectations and outcomes.
Finding 6. Reviews focus on a single system instead of on a complex system of systems of which the single system is a part. Further, reviews that attempt to address programs from the larger system-of-systems perspective are often unable to cope with the complex interfaces among programs.
Seventy percent of the ACAT I PMs who responded to the survey characterized the amount of external interface of their programs with other efforts as extensive (Survey Question 1.13, ACAT 1 PM responses only). This answer—combined with survey written-in responses and PM discussions with the full committee noting that some reviews did not take into account connections with and dependencies on other programs for mission accomplishment—allows concluding that the current acquisition and program review process has not adapted to the evolu-
OCR for page 29
Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs
tion of simple systems into systems of systems and fails to take into consideration the additional complexity and interrelationships necessary for effective program management in this new environment (based on written-in responses to Survey Questions 3.10, 3.25, and 4.1-4.4).
Conclusion 6. Program review format and design need to reflect the greater complexity and interrelationships inherent in many current Air Force programs to ensure that a system of systems works across organizational constructs.