Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 16
Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs 2 Findings and Conclusions As summarized in Chapter 1, the committee deliberated on the results from the four data sources to arrive at the findings and conclusions contained in this chapter. The findings and conclusions are based on information derived from the survey responses, interviews with Air Force and DOD personnel and other stakeholders, earlier studies, and the committee members’ expertise. The committee spent considerable time deliberating to ensure that the findings indeed represent what it had heard. The conclusions then represent a committee consensus of a generalization of the findings. The recommendations in turn (see Chapter 3) resulted from integrating and synthesizing the findings and conclusions in a way that the committee believed would be most responsive to the Statement of Task. The matrix of reviews that was constructed to help classify the number and types of known reviews, their purpose, and their target audiences in order to identify opportunities for streamlining, integrating, and/or consolidating reviews is shown in Table 2-1, with further information included in Appendix E. The reviews are of four types (see third column of the matrix): Milestone/programmatic reviews. These reviews are conducted by the milestone decision authority (MDA) (the Defense Acquisition Executive (DAE) for acquisition category (ACAT) ID programs), with support from the Office of the Secretary of Defense (OSD) staff. They provide the basis for the MDA to decide whether to allow a new program to be initiated or an ongoing program to proceed to the next phase. Of the four types, reviews of this type require by far the greatest number of prereviews (reviews conducted by functional offices and intermediate management levels on the way to the MDA review). These reviews are listed in approximately
OCR for page 17
Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs the order they occur leading up to the ultimate Defense Acquisition Board (DAB) review. Periodic oversight reviews. These reviews are conducted at a regular frequency—annually or quarterly—to allow the DAE, the Service Acquisition Executive (SAE), or the program executive officer (PEO) to monitor progress. They are listed alphabetically. Ad hoc reviews. These reviews are initiated by an organization outside the program office either in response to a problem or, in the case of some major programs, periodically (e.g., by the GAO). They are listed alphabetically. Technical or engineering reviews. In contrast to the external reviews mentioned above, these reviews are generally conducted internally by the program itself, with some support from Service functional assets. They are a principal means by which the program office manages the technical and fiscal execution of the program. They are listed in approximately the order they occur over the life of the program. The matrix (Table 2-1) summarizes the various types of reviews typically conducted over the lifetime of a program. It names 31 formal reviews of four types, although not all of them or even all types may be conducted for every program. At least four of the reviews require unique documents and 10 have been identified by the committee as duplicating or partially duplicating other reviews. The matrix does not list every possible ad hoc review. Nor does it list the prereviews or prebriefs generated by these formal reviews, since it was not possible to determine the number of such reviews or prebriefs. Because no data or metrics are required or collected on reviews by DOD in general, or by the Air Force in particular, it was not possible to determine the overlap or duplication of different reviews. (Of note, although no historic information on the number of prereviews or prebriefs is available, interview comments and the committee members’ experience lead one to believe that major reviews at the level of the Joint Requirements Oversight Council (JROC), DAB, and PEO are generally accompanied by multiple (3 to 5) prebriefs or prereviews.) In the program review process depicted in Figure 1-4, it can be seen that there are 6 MS or MDA reviews in a typical program, plus 6 overarching integrated product teams (OIPTs) reviews before those, plus 6 Air Force Review Board (AFRB)/acquisition strategy panel (ASP) reviews before each of those, plus 10 or 11 mandatory technical reviews at the program level, plus 4 periodic program support reviews (PSRs) for each phase of the life cycle, plus one operational test readiness assessment (OTRA) before IOT&E, plus annual configuration steering board (CSB) reviews (notionally 7), plus annual PEO-level sufficiency reviews (notionally 7), plus at least 2 technology readiness assessments/manufacturing readiness assessments (TRAs/MRAs). This suggests in excess of 50 reviews just at the program level and above without counting
OCR for page 18
Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs TABLE 2-1 Program Review Matrix Review Name Type Purpose Frequency OPR/Customer ASP Acquisition Strategy Panel Programmatic Formalize acquisition strategy MS SAF/AQ AFROCC Air Force Requirement for Operational Capabilities Council Programmatic Validate requirements MS A5R/CSAF, SECAF JROC Joint Requirements Oversight Council Programmatic Prioritize Joint requirements MS VCJCS/DAB AFRB Air Force Review Board Programmatic Ready for MS or OSD MS SAE IPA Independent Program Assessment Programmatic Risk assessment MS SMC/MDA PSR Program Support Review Programmatic Risk assessment MS Director SSE/USD(AT&L) IIPT Integrating IPT Programmatic Support OIPT MS PM/OIPT OIPT Overarching IPT Programmatic Ready for DAB MS Dir PSA, DASD(NII)/DAB DAB Defense Acquisition Board Programmatic Advise USD(AT&L) MS USD(AT&L) CSB Configuration Steering Board Periodic oversight Review requirements changes Annual SAE/USD(AT&L) DAES Defense Acquisition Executive Summary Periodic oversight Ongoing performance Quarterly PM/DUSD(A&T) PEO/SR PEO Sufficiency Review Periodic oversight Executability confidence Annual PEO
OCR for page 19
Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs Presenter Stakeholders Prereview Briefings Product/Output Unique Documentation? Duplication PM SAF/AQ NDA Approve acquisition strategy plan Yes Partially A5 staff, MAJCOM representative Air staff, AF Secretariat FCBs Validate requirements No NDA J8 director, Service representative Services, warfighters Joint Requirements Board, Panel JROCM Yes No Requirements representative PM, user, OSD, service, KTR Unknown Readiness to proceed No No PM, staff DSAB, staff support Unknown Briefing and report No Partially PM, PO & KTR SMEs Dir SSE, USD(AT&L), PM Coordination meetings Brief findings, recommendations No Partially Varies PM, user, OSD, service, KTR None Report to OIPT NDA Partially PM; OSD staff PM, PEO, OSD, Joint Staff IIPT or equivalent, staff Report w/recommendations No No OIPT chair, PM, others USD (AT&L), DAB members OIPT Acquisition Decision Memorandum Yes No PM SAE, USD(AT&L), JCS Unknown Approve requirements change Yes NDA PM (with SAE) PM, SAE, OSD staff Unknown Action items Yes No PM, SMEs PEO, PM, ACE, SMEs Unknown Briefing, PoPs update No NDA
OCR for page 20
Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs Review Name Type Purpose Frequency OPR/Customer AFAA AF Audit Agency Ad hoc oversight Varies Varies AFAA/SECAF DOD IG DOD Inspector General Ad hoc oversight Varies Varies DoD IG/SecDef GAO Government Accountability Office Ad hoc oversight Cost, schedule, performance Varies GAO/Congress Other ad hoc Varies Varies Varies Varies TRA Technology Readiness Assessment Technical Executability confidence MS DDR&E/MDA ASR Alternative System Review Technical Ready for technology development MS A Engineering/PM SEAM Systems Engineering Assessment Model Technical Validates SE process Varies AFCSE/PM SRR System Requirements Review Technical Executability confidence One time Engineering/PM SDR System Design Review Technical Replaced by the SFR One time Engineering/PM SFR System Functional Review Technical Ready for prelim design One time Engineering/PM PDR Preliminary Design Review Technical Ready for detailed design One time Engineering/PM IBR Integrated Baseline Review Technical Align program expectations Varies PM LHA Logistics Health Assessment Technical Logistics health Unknown Unknown; currently AAC MRA Manufacturing Readiness Assessment Technical Executability confidence One time DDR&E/MDA CDR Critical Design Review Technical Ready for fabrication One time Engineering/PM
OCR for page 21
Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs Presenter Stakeholders Prereview Briefings Product/Output Unique Documentation? Duplication PM, staff SECAF, staff, AFAA office Coordination meetings Report No Partially PM, staff Varies - wide-ranging Coordination meetings Written report w/recommendations No Partially Varies - often PM or rep Varies - wide-ranging Coordination meetings Written report w/recommendations No Partially Varies Varies Unknown Varies Varies Yes AF ST&E PM, S&T, DDR&E, DAE Unknown Tech Readiness Levels/Plan NDA Yes PM, PO & KTR SMEs PM, User, KTR Unknown Rationale for preferred alt No No PM/Engineering AFCSE, PM None Assessment reports No Partially PM/Engineering Engineering, user, KTR Unknown Satisfy exit criteria No No PM/Engineering PM, contractor Unknown Satisfy exit criteria No No PM/Engineering Engineering, user, KTR Unknown Satisfy exit criteria No No PM/Engineering Engineering, user, KTR, DAE None Satisfy exit criteria No No PM PM, contractor NDA Mutual understanding of program baseline No Partially Unknown PM, user, AF logisticians Unknown Log/sustainment assessment Unknown Partially PM/Engineering PM, S&T, Unknown Mfg Readiness Levels/Plan NDA NDA PM/Engineering Engineering, User, KTR, DAE None Satisfy exit criteria No No
OCR for page 22
Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs Review Name Type Purpose Frequency OPR/Customer TRR Test Readiness Review Technical Ready for testing One time Engineering/PM PRR Production Readiness Review Technical Ready for production MS C, FRP Engineering/PM OTRR Operational Test Readiness Review Technical Approve program OT readiness One time SAF/AQ SVR System Verification Review Technical Verifies performance MS C Engineering/PM FCA Functional Configuration Audit Technical Satisfies contract One time Engineering/PM PCA Physical Configuration Audit Technical Satisfies contract One time Engineering/PM Frequency: MS, this review takes place in support of or in conjunction with each formal program milestone (A, B, C, etc.); MS A, this review takes place in support of or in conjunction with formal program milestone A; MS C, this review takes place in support of or in conjunction with milestone C; MS C, FRP, this review takes place in support of or in conjunction with milestone C and the full rate production decision. Duplication: A review/meeting exactly the same as another review/meeting. By this definition, the use of the same material at more than one meeting does not constitute a duplication of the meeting. The the prebriefs to staff functional principals and others as noted in the preceding paragraph. Although the data and information received did not allow the committee to determine specific resources required to accomplish each review, answers to Question 1.5 in the survey and information gathered from the interviews indicated that 10-30 percent of a PM’s time is spent supporting reviews. The committee found that there is little consistency in the way reviews are conducted and concluded that opinions on the contribution of specific reviews to successful acquisition varied widely. The committee believes that the Air Force could improve the effectiveness of its program review effort and reduce the burden on PMs by thoughtfully combining and scheduling reviews. The committee looked at the policies and processes of the National Aeronautics and Space Administration (NASA), the Department of Energy (DOE), industry, and the other military services. Although review
OCR for page 23
Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs Presenter Stakeholders Prereview Briefings Product/Output Unique Documentation? Duplication PM/Engineering/Test Test IPT, User, KTR, DOT&E None Prepared for testing No No PM/Engineering KTR, Subs None Requirement met, production ready No No PM SAF staff, DOT&E NDA ASAF(A) approval of program’s OT readiness Yes Partially PM/Engineering User, KTR None Requirements met, funding adequate No No PM/Engineering PM, Engineering, KTR None Audit report NDA No PM/Engineering PM, Engineering, KTR None Audit report or completion NDA No degree of duplication shown in the Duplication column is subjective and is based on review documentation, interviewee comments, and survey results. Given that no review, let alone its process owner, requires any data collection on duplication, coordination, value added, and so on, the conclusions are not rigorous. Review duplication attributes: Yes, the review was judged to be a duplication; No, the review was judged not to be a duplication; Partially, the meeting was judged to be a partial duplication; and NDA, the review is too new or had no data available to allow a judgment on duplication. structures and practices exist, the differences in implementation make cross-community consolidation or streamlining a significant challenge. The information collected from experienced PMs in the U.S. Army and the U.S. Navy, as well as the U.S. Air Force, and from industry contributed to the set of best practices reported in the recommendations. REVIEWS HAVE BOTH BENEFITS AND COSTS The committee found that program and technical reviews have both benefits and costs. In general, reviews provide technical and programmatic support to successfully execute acquisition programs, to inform decisions, to share awareness, and to engender program advocacy. In their answers to Survey Question 2.3, PMs said that reviews facilitated program execution as well as problem discovery and resolution at all levels of the acquisition enterprise, including industry. That said,
OCR for page 24
Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs support for the increasing number of program reviews and associated prebriefs is costly, as is the lost productivity and attention of both government and industry. For example, DODI 5000.02 of December 2008 calls for systems engineering across all phases and mandates technical reviews. The new regulation mandates competitive prototyping of the system or key system elements during the technology development (TD) phase, and a preliminary design review (PDR) must be conducted for each of the candidate designs, followed by a PDR report to be provided to the MDA either in the TD phase supporting milestone B or, if afterward, a separate MDA review during the engineering and manufacturing development (EMD) phase. Although some such overhead is undoubtedly necessary, excesses can distract PMs from their primary focus, which is managing their program’s technical and business progress. The findings and conclusions of the committee focus on two key areas: The first is the execution of reviews (sequencing, timing, participation), and the second is their planning. EXECUTION OF REVIEWS (SEQUENCING, TIMING, PARTICIPATION) The committee’s investigation confirmed that the number of reviews is growing. The committee recognizes that such reviews are generally valuable for sharing knowledge and serve as “stage gates” for the governance and control of programs, including managing risk. While doing this, however, they most certainly add to program costs, and they also draw management’s attention from the main program effort. Of most concern is that the proliferation of reviews does not appear to have had a positive effect on program cost and schedule outcomes. The committee also realized that there is a significant amount of preparation and coordination for reviews, both vertically in their conduct at multiple levels of responsibility and horizontally across adjacent staff offices. As such, merely depicting the individual reviews does not sufficiently capture the amount of time and effort spent by staff in preparing for and coordinating the reviews. The committee found that the many disparate concerns of higher-level staffs had an impact on the program manager. For ACAT I programs, many of the written-out responses1 to Survey Questions 4.1-4.4 described DOD staff as a stove-piped bureaucracy, where domain “czars” have purview over a breadth of programs (by virtue of the OIPT structure or their membership on the DAB) but are not horizontally integrated from the standpoint of knowledge sharing or synergy. This means not only that the PMs have to prepare separate information 1 The qualitative write-in comments from the PMs have not been included in this report because several of the comments were written in such specific detail that the authors of the comments could be identified by people familiar with their programs. This would violate the privacy protection methodology for the survey, so the data are not included.
OCR for page 25
Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs briefs for these higher-ups, but also that the information provided to them is not fully integrated across these domains by the OSD staff for optimal decision making by the MDA. Programs and technical reviews are often not being optimally synchronized with program events. Speakers gave examples of requirements reviews being conducted after contracts had been awarded and other similar occurrences of inappropriate sequencing. The committee was briefed on emerging acquisition policy with its emphasis on pre-Milestone A activities to ensure a better understanding of alternative technical requirements. The committee believes there is ample opportunity for the Air Force and DOD to improve the timing of these reviews and even to consolidate reviews at the service staff level and below to eliminate redundancy and mitigate the associated burdens and costs. Finding 1a. A significant share of a PM’s time is spent on preparing for and participating in milestone and other reviews. Finding 1b. The large number of reviews diverts a PM’s attention from the execution of his or her program. Finding 1c. Reviews impose significant costs on program leadership teams. The committee interviewed a number of PMs to obtain their views on the time they spent and the costs associated with carrying out these various reviews. It also sought their assessment of how the conduct of these reviews affects their ability to carry out the day-to-day tasks associated with their programs. The PM survey also asked respondents to address these issues. In every case, those interviewed or surveyed cited significant costs in terms of money and time to carry out the reviews, and most of them also noted an adverse impact on their ability to carry out other PM responsibilities. Sixty-nine percent of survey respondents said they were working 51 or more hours per week (Survey Question 1.4) and on average were able to spend only 46 percent of that time managing their program at wing level or below (Survey Question 1.5). About 20 percent of their time was spent reporting up the chain of command above the wing level (Survey Question 1.5). In addition, many of the respondents cited the need to dedicate hundreds or thousands of program staff and contractor hours to carry out the reviews, with concomitant money and time lost from their everyday duties (based on write-in responses to Survey Questions 3.11-3.14 and 3.26-3.29). Finding 1d. Many higher-level reviews do not contribute to program execution in proportion to their expenditure of time and effort. While those interviewed identified benefits they received from reviews that were effectively carried out, a number of survey respondents and interviewees said
OCR for page 26
Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs they received no significant benefit or improvement to program cost, schedule, or performance (CSP) from the effort. For example, 29 PMs identified what they considered the least beneficial (from the standpoint of program CSP) higher-level review they had been involved in. Eighty percent of them cited no positive impact on CSP from this “least beneficial review” (based on written-in responses to Survey Question 3.24). Some even went so far as to say that the review had actually had a negative impact on CSP by delaying program schedule or increasing program costs in the case of reviews that were ineffectively carried out (based on written-in responses to Survey Question 3.25). It should be noted that the PMs were not questioned on ancillary benefits such as sponsorship, which they may not have viewed as a direct program benefit. Finding 1e. No one in the Air Force or OSD is responsible for monitoring the number of program and technical reviews or the workload they give rise to or their cost, effectiveness, and cumulative impact on PMs. Based on the committee’s review of the OSD and Air Force organizational structures and its discussions with senior DOD leadership, no one is responsible for monitoring the direct costs of program and technical reviews, in terms of the time, personnel usage, extra costs, and effects on contractors and the PM. Conclusion 1. Many reviews add little value and others do not add value in proportion to the effort required. Reducing the number of such reviews or combining them could increase the time available to the PMs to more effectively manage their programs. Finding 2. The sequencing, timing, and frequency of reviews are often not tied to the program schedule in a manner that most effectively supports the program and its execution. Forty-three percent of survey respondents commenting on the least beneficial reviews suggested that those reviews could be more effective if they were conducted less frequently and at a more appropriate time in the program’s life cycle (generally earlier) (Survey Question 2.6). Conclusion 2. Reviews could be more effective if they were sequenced and timed to provide the information needed for program execution. Chapter 3 describes an approach for beginning to achieve this result. Finding 3a. The program review principals, key stakeholders, and subject matter experts do not always attend program reviews.
OCR for page 27
Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs Finding 3b. Program and technical reviews are often not attended by the right personnel or, in some cases, are attended by too many personnel. Many survey respondents noted that more effort should be given to ensuring that the right subject matter experts and appropriate senior officials attend program reviews and that the number of attendees be limited to those who can add value to the meeting (based on responses to Survey Questions 2.6 and 3.15 and written-in responses to Survey Questions 3.30-4.4). Conclusion 3. Required attendance at program review meetings is not clearly communicated nor is it effectively controlled. Finding 4a. For some reviews the number of actual reviews and preparatory reviews is excessive and the reviews do not contribute value to the program’s management. Many of the PMs, both in the survey and in interviews, stated that a proliferation of meetings and premeetings was taking time away from the management of their programs (based on responses to Survey Question 2.6 and written-in responses to survey Questions 3.10 and 3.25). Commenting on the reasons for this increase, a few PMs mentioned the Integrating Integrated Product Team (IIPT) and the NII/AT&L structure. The elimination of IIPT reviews was cited as one factor that led to the need for more individual premeetings with the Joint Staff, program management offices, and OSD, where previously only one meeting was needed. Another reason cited was the recent NII/AT&L reconfiguration that resulted in the sharing of responsibilities between these two offices. Survey respondents cited the amount of time now taken up by such premeetings. One respondent, for example, said “The problem isn’t the review … it’s the numerous premeetings needed to get to the review.” Finding 4b. Program managers are spending time on multiple reviews with similar objectives. Many survey respondents believed that selected reviews could be combined (Survey Question 2.7). Conclusion 4. Streamlining or combining reviews and their associated prebriefs in both the vertical and horizontal directions could increase efficiency. PROGRAM REVIEW PLANNING The committee found that in many cases, despite published guidance advocating proper planning, reviews were incompletely planned or conducted. The
OCR for page 28
Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs potential value of a review cannot be fully realized if objectives are not clearly specified or if the right persons are not in attendance. What information a particular review is supposed to contain or what areas a particular review is supposed to cover are often ill defined or based on presumptions of agendas or issues of the day. Often there are no metrics for assessing the effectiveness of a review, and this can lead to perceptions of disruption and uncertain value. On the other hand, many PMs found that the program support reviews (PSRs) and the independent program assessments (IPAs) were well planned, that they added value, and that they contributed positively to program management. PSRs and IPAs are comprehensive in nature; have well-defined processes, outcomes, and metrics; are socialized with PMs and staff; are conducted by subject matter experts; and are well documented. A review process that makes timely use of PSRs and IPAs might help to limit the number of reviews needed across the bureaucracy to gather information and make decisions. Finding 5. The purpose, scope, information needs, key issues, and expected outcomes of many reviews have not been specified. Survey respondents mentioned a number of ways to improve reviews. Suggestions for improvement to address the problems cited above included narrowing the review’s focus or changing its charter (based on responses to Survey Questions 2.6 and 3.15 and on written-in responses to Survey Question 3.30). PMs noted that they would be asked about issues that had not been previously identified for discussion and would then be required to spend countless hours after the review trying to respond to them. They also said there was no standard approach to how the reviews should be conducted. Conclusion 5. It is important that program review planning is accomplished in a thoughtful, purposeful manner with a standard approach in order to firmly address the need for communication of the expectations and outcomes. Finding 6. Reviews focus on a single system instead of on a complex system of systems of which the single system is a part. Further, reviews that attempt to address programs from the larger system-of-systems perspective are often unable to cope with the complex interfaces among programs. Seventy percent of the ACAT I PMs who responded to the survey characterized the amount of external interface of their programs with other efforts as extensive (Survey Question 1.13, ACAT 1 PM responses only). This answer—combined with survey written-in responses and PM discussions with the full committee noting that some reviews did not take into account connections with and dependencies on other programs for mission accomplishment—allows concluding that the current acquisition and program review process has not adapted to the evolu-
OCR for page 29
Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs tion of simple systems into systems of systems and fails to take into consideration the additional complexity and interrelationships necessary for effective program management in this new environment (based on written-in responses to Survey Questions 3.10, 3.25, and 4.1-4.4). Conclusion 6. Program review format and design need to reflect the greater complexity and interrelationships inherent in many current Air Force programs to ensure that a system of systems works across organizational constructs.