This chapter provides an overview of the Air Force approach to force development, (e.g., training), utilization (e.g., promotions and assignments), and retention of Airmen after they have been recruited and matched to a specific Air Force Specialty Code (AFSC) (see Figure 2-3 for the placement of these focus areas within the ecosystem model). The overview is informed by relevant Air Force policy and information provided by Air Force representatives and was supplemented by the committee’s own expertise. The chapter concludes with a discussion of research areas to support post-accession decisions as the Air Force seeks to strengthen its human capital management (HCM) system to develop and support its future force.
Understanding the history behind how the U.S. military manages its people is helpful in thinking in particular about how the Air Force uses its Airmen. One long-standing and important feature of their approach is an “up or out” system, used by the U.S. Navy in various forms since 1916 and scaled up for military-wide use with Congress’s passage of the Officer Personnel Act (OPA) in 1947. The “up or out” system requires officers to be promoted to the next rank within a certain period of time or leave the service. Officers who are not promoted on-time with their peers are permitted one final opportunity for promotion, and if not selected, they are required to separate.1 The 1947 OPA established a promotion schedule based on year of commissioning, with promotion boards evaluating
1 The FY2019 National Defense Authorization Act provides exception to this through authority to promote and extend the career lengths of officers with critical skills (for further discussion, see Kidder, 2019).
entire cohorts of officers at set times. Congress made tweaks to this system up until 1980, when it made significant revisions to the system with the Defense Officer Personnel Management Act (DOPMA), which established military-wide regulations meant to streamline the process for promotions and also systematized processes surrounding officer separation and retirement. Additionally, DOPMA established the highest percentage of field grade officers (in relation to total number of officers) that each service is authorized at any given time.2
Unlike its approach to the promotion of officers, Congress has not been as directly involved in managing the enlisted force but does legislate the maximum percentage of the enlisted force that can serve in pay grades of E-8 (2.5%) and E-9 (1.25%) (Congressional Research Service, 2019). As a result, each Service establishes its own high-year-of-tenure policies by grade to manage its enlisted force, which generates a type of “up or out” effect. Ultimately, this has a profound impact on how the Services use and manage their military force and also serves as an incentive for those military members who want to progress to take advantage of competitive force development programs that will set them apart from their peers.
Air Force doctrine is very clear on the importance of its people: “People are the Air Force’s most critical asset. Airmen turn competencies into required capabilities. For this reason, the art of employing Airmen with the requisite education, training, and experience is fundamental to the effectiveness of the Service, affecting current operations and future capabilities” (USAF, 2019b, p. 2). This “art” is called force development, a dynamic and deliberate process that ensures Airmen have the required institutional and occupational competencies to meet the challenges of conflict in the 21st century.3 To ensure Airmen possess the right expertise and competence to meet operational needs, the Air Force adopted a continuum of learning: a career-long process of individual development that integrates developmental opportunities through foundational competencies (see Appendix E).4 “Through the continuum of learning the Air Force exposes Airmen to a broad-based education, training, and experience framework that equips them with the competencies to serve as leaders as they progress in rank and responsibility” (USAF, 2019b, p. 5-6). As will be described in
detail below, force development is a comprehensive approach to developing the whole Airman: it includes a full spectrum from highly technical training to obtain specific job-related skills to ensuring Airmen are educated on the core values and ethics of the profession of arms.
The Air Force Training Program
The Air Force defines training as “a set of events or activities presented in a structured manner, for the attainment of skills, knowledge, and attitudes required to meet job performance requirements” (USAF, 2019a, p. 6). The purpose of this training is to ensure that each individual is prepared to meet Air Force mission requirements. As such, the “continuum of learning” principles guide the development of training programs across the board.
Enlisted Basic Military Training (BMT) and Enlisted Airman Orientation Course
All Active Duty, Air Force Reserve, and Air National Guard recruits with no previous military service attend BMT for 8.5 weeks at Joint Base San Antonio-Lackland. Recruits with prior non-Air Force military service attend the Enlisted Airman Orientation Course (optional for Reserve and National Guard). BMT provides an orderly transition from civilian life to military life, constructs all initial military official records, and classifies and assigns personnel following policy (USAF, 2019d). It also discharges trainees who fail to meet minimum standards, the percentage of which varies by year and is closely tracked by Air Education and Training Command (AETC).
Enlisted Skills Training and Development
Upon completion of BMT, Airmen proceed to a technical training school where they learn the technical skills needed to perform in their career field specialties. Technical training is conducted primarily at six installations,5 each of which is responsible for a specific portion of the formal technical training Airmen require to accomplish the Air Force mission. Highly trained instructors conduct technical training in specialties such as aircraft maintenance, missile maintenance, civil engineering, medical services, computer systems, security forces, air traffic control, weather, personnel, cyberspace support, intelligence, firefighting, and space and missile operations.
5 Joint Base San Antonio (JBSA)-Lackland, JBSA-Fort Sam Houston, Goodfellow, and Sheppard Air Force Bases in Texas; Keesler AFB, Mississippi; and Vandenberg AFB, California.
The enlisted force has four defined skill levels: Apprentice, Journeyman, Craftsman, and Superintendent. AETC is working to adapt training so that Airmen can quickly test out of training on skills and knowledge they already possess. During the committee’s visit to Keesler AFB, cyber career field trainers demonstrated their approach to this type of training; another example includes the work being done to shorten Air Force pilot training through the use of technology.6 Each functional area collaborates with AETC regarding the curriculum for its technical training, and some career fields are ahead of others on modernizing their curriculum and pedagogy. The Journeyman and Craftsman skill levels are for Staff Sergeants and above and require a Career Development Course7 (if applicable), time in training (if applicable), and mandatory core tasks identified in their Career Field’s Education and Training Plan (CFETP).8 The Superintendent skill level is for Senior Master Sergeant and above and requires meeting mandatory requirements in the Air Force Enlisted Classification Directory and CFETP as well as getting the supervisor’s recommendation and commander’s approval. Figure 5-1 includes more information about enlisted skills development, as it relates to enlisted education and training programs.
Officer Basic Training
As mentioned in Chapter 4, there are three primary officer basic training pipelines: the U.S. Air Force Academy (USAFA), Reserve Officer Training Corps, and Officer Training School. Additionally, the Air Force classifies its
6 For information about the adaptive training initiative, “Pilot Training Next,” see https://www.aetc.af.mil/About-Us/Pilot-Training-Next/. The topic of adaptive training was discussed with Air Force representatives during site visits to Joint Base San Antonio-Randolph (November 5–8, 2019), Wright-Patterson Air Force Base, Dayton, OH (November 13, 2019), Keesler Air Force Base, Biloxi, MS (January 9, 2020), and Maxwell Air Force Base, Montgomery, AL (January 15, 2020).
7 Career Development Courses are designed to provide basic knowledge across a wide spectrum of subjects pertaining to a career field. When these courses are not available, trainees study the applicable technical references identified by their supervisor and/or CFETP.
8 “The CFETP is the primary document used to identify life-cycle education and training requirements. It serves as a road map for career progression and outlines requirements that should be satisfied at appropriate points throughout the career path. The CFETP also specifies the mandatory task qualification requirements for award and maintenance of an AFSC” (USAF, 2019c, p. 40). See Appendix F for additional information and sample CFETPs. Every career field appoints a career field manager for both its officer and enlisted AFSCs. Career field managers communicate with Major Command Functional Managers and with the Air Reserve Component and AETC Training Pipeline Manager to disseminate Air Force and career field policies and program requirements.
officers into certain areas or professions or appoints intraservice transfers, Health Professions, Enlisted Commissions, Legal, and Chaplain.9
Officer Skills Training and Development
Most officer skill development models include initial skills training; however, not all officers will get their initial skills training before they arrive at their first duty station. The point at which an officer will attend their initial skills training is heavily dependent on the skill and the technical training capacity. Follow-on skills training is governed by each CFETP and will likely include on-the-job training, advanced distributed learning training, and structured schoolhouse training at certain phase points in an officer’s career (see Appendix F for excerpts from two sample CFETPs). The officer corps does not have the defined skill levels that the enlisted corps does (apprentice, journeymen, craftsman, superintendent).
All Airmen, across all AFSCs, receive universal training, guidance, or instruction essential for mission accomplishment, referred to as ancillary training. Although some ancillary training programs are routine annual programs, others may be developed in response to events or to introduce or strengthen core values, institutional knowledge, and behavior standards needed to perform assigned duties and responsibilities. Ancillary training may be accomplished through a variety of methods, including formal courses, mass briefings, Advanced Distributed Learning, credit for civilian equivalent experience, and one-on-one instruction. It can be a one-time event or recurring requirement and falls into four categories:
9 To serve as a commissioned officer one must be appointed by the President and when appointed the Service must select the profession: Health Professional Officer, Chaplains and Chaplain Candidates, Judge Advocates, and Line of the Air Force. Some of this authority has been delegated to the Secretary of Defense for grades of 2nd Lt, 1st Lt, and Captain. The accession sources prepare the appointment paperwork once it verifies that the cadet has met all the commissioning requirements. The paperwork then gets processed for appointment approval to the Secretary of Defense before an officer is commissioned. Appointments do not get further stratified into AFSCs.
10 Functional, occupational, or additional duty training is not included in ancillary training; however, ancillary training includes subjects such as chemical, biological and nuclear warfare training, information protection, human relations training, force protection, law of armed conflict, religious freedom, self-aid and buddy care, substance abuse, suicide prevention, and additional duty training.
- Total force awareness training;
- Selected force training, which is targeted to specific groups, including commanders, civilians, and supervisors;
- Event-driven training, which is triggered by some event, such as moving to a new assignment or duty station; and
- Basic Airman readiness training, which is expeditionary-focused training.
There is a general officer-led committee that evaluates all requests for ancillary training to ensure it is relevant and worth the Airman’s time. Some training is directly driven by legislation (e.g., training for cyber awareness or to combat trafficking in persons) and some is functionally driven (e.g., defense travel system training because audits were finding numerous errors in travel vouchers).
The Air Force Developmental Education Program
Developmental education opportunities allow Airmen to participate in an assortment of formal and experiential resident, non-resident, and blended programs (including time authorized during the normal duty day). As described in more detail below, these opportunities include, for example, graduate education, fellowships, and education with industry, as well as service and joint professional military education.
Enlisted Professional Military Education
In FY2020, there are 265,136 enlisted Airmen, making up nearly 80 percent of the active duty force.11 Central to the enlisted Airman’s career is the enlisted professional military education program that provides essential military education through rank- and competency-based models over five developmental levels: introductory, primary, intermediate, senior, and executive. In-residence selection for all regular Air Force enlisted Airmen is based on grade, date of rank, priority, and eligibility. Once selected for resident enlisted professional education, attendance is required and may not be deferred except for valid mission, medical, or humanitarian reasons. Airmen may decline in-residence attendance with prejudice, meaning they will be ineligible for promotion and participation in the High Year of Tenure extension program. A triennial review committee reviews all enlisted professional military education every 3 years to ensure it remains aligned with Air Force priorities, and force development strategies. As described below,
11 Numbers and percentages here and in the following sections are derived from data contained in USAF Almanac (2019), Air Force Magazine, 102(6), p. 41–42.
training is divided into five categories that span an enlisted airman’s career (see Figure 5-1).
Airmen in grades E-1 through E-3 (roughly 28% of Air Force enlisted active duty) complete foundational development (basic military training, technical training, and First Term Airmen Center Orientation12) to become motivated, disciplined, and prepared to successfully transition into a mission-oriented environment.
Airmen in grades E-4 through E-6 (roughly 59% of Air Force enlisted active duty) complete institutional and occupational development to prepare to be professional, highly skilled technicians, supervisors, and leaders in the employment of air, space, and cyberspace power. This training includes, but is not limited to, distance learning, resident enlisted professional military education, and Joint Professional Military Education.
Airmen in grade E-7 (roughly 10% of Air Force enlisted active duty) continue institutional and occupational development to become leaders with operational competence to lead the enlisted force in the employment of air, space, and cyberspace power. In addition to distance, resident, and joint programs, intermediate (and senior) training programs also allow Airmen to attend equivalent courses through the other services.
Airmen in grades E-8 and E-9 (roughly 3% of Air Force enlisted active duty) further continue institutional and occupational development to become experienced and operational leaders while bridging operational-to-strategic perspectives in the employment of airpower.
Airmen in grade E-9 (roughly 1% of Air Force enlisted active duty) reach the final point of institutional and occupational development to broaden the leadership and utilization of enlisted Airmen through operational competence and strategic vision levels of leadership. In addition to distance, resident, and joint programs, executive training may also include other educational and developmental opportunities (e.g., advanced academic degrees).
12 First Term Airmen Center Orientation takes place at every installation but is grounded in a standard Air Force curriculum. At the installation level, the installation commander is responsible for providing the resources and executing First Term Airmen Center (typically referred to as FTAC).
Community College of the Air Force
Community College of the Air Force (CCAF) is a federally-chartered academic institution that serves the enlisted corps (active, guard, and reserve) of the Air and Space Forces. CCAF’s mission is to “offer and award job-related associate in applied science degrees and other academic credentials that enhance mission readiness, contribute to recruiting, assist in retention, and support the career transitions and professional growth of the Air and Space Force enlisted corps.”13 CCAF is the world’s largest community college system: it partners with 112 affiliated Air and Space Force schools and 300 Education Service Offices located worldwide to serve more than 250,000 enlisted Airmen. Annually, CCAF grants more than 22,000 associate in applied science degrees through 71 different topical programs.
Enlisted Program for Advanced Academic Degrees
Among the Air Force’s senior non-commissioned officers (NCOs), 31.6 percent hold a bachelor’s degree, while another 51 percent hold an associate’s degree (AFPC, 2020). To further their education, there are several opportunities for enlisted Airmen to obtain advanced academic degrees through programs managed by Air Force Career Field Managers.14
Officer Professional Military Education
In FY2020, there are approximately 64,000 officers in the Air Force, making up just 19 percent of the active duty force.15 Each member of this small cadre is functionally developed to be expert in their respective career fields, guided by education and training plans. These plans also convey performance expectations and identify preparation to be taken as rank and responsibilities increase. Expertise is evaluated both qualitatively and quantitatively. Annual performance reports document the performance of
14 Air Force Career Field Managers “ensure development, implementation, and maintenance of CFETP for their assigned AF [Air Force] specialties” (USAF, 2019a, p. 39). They coordinate with functional managers, the air reserve component, and AETC to disseminate relevant service and career field policies and program requirements.
15 Percentages derived from data contained in USAF Almanac (2019), Air Force Magazine, June, p. 41. The remaining 1 percent of the Air Force active duty are comprised of USAF Academy Cadets. Note the Air Force active duty includes approximately 29,000 enlisted and officer medics who often come under different policies for recruitment, accession, training, and retention than those for the remainder of the active duty force. For more information on the USAF Defense Health Program, see https://docs.house.gov/meetings/AP/AP02/20190403/109223/HHRG-116-AP02-Wstate-HoggD-20190403.pdf.
an individual Airmen in their current job and often have a recommendation as to what the Airman should do next. Performance and completion of required training also provide insight into how an officer is tracking on their development path.
For officers, each functional area also brings its senior leaders together three times a year and reviews all the records of every officer in the career field to see where they are on their development path, how well they have done in the positions they have served in, and what their leaders have said about them. The review results in a recommendation on what positions the Airman should consider in the near future and also what development opportunities should be sought. This information is provided to the individual’s current commander and the individual to foster two-way conversation.
According to the Office of the Secretary of Defense, professional military education is defined as a broadening opportunity and, as such, is designed to help prepare officers for an array of potential assignments within and outside their occupational expertise. Limited resources may restrict resident professional military education attendance and is therefore limited to the “best qualified” (USAF, 2020a, p. 91).
Professional military education for officers has three levels: primary, intermediate, and senior (USAF, 2020a, 2020b). Intermediate and senior levels include opportunities for in-residence military education programs that are internal to the Air Force, with the other services, and with American allies. It also includes fellowships across government, think tanks, and with industry. Each level also targets these opportunities to the experience and grade level of the eligible officers. See Figure 5-2 for more information about officer education.
Key to effective force utilization are assignments Airmen receive over the course of their career. Assignments serve the dual purpose of serving the needs of the Air Force and giving experience to Airmen by assigning them to positions with increasing roles and responsibilities as they progress in their careers. Assignment changes can be at a single location or can require moving to a new location. In 2019, the Air Force began using the web-based platform, the Talent Marketplace, for making officer assignments (Lieutenant Colonel and below). The Talent Marketplace was initiated in an effort to meet a stated goal: “To the maximum extent possible, assign individuals on a voluntary basis and in the most equitable manner feasible while meeting mission and commander needs” (USAF, 2018a, p. 2). The Talent Marketplace is discussed in more detail below.
The Air Force Assignment System
The Air Force Assignment System is governed by several Department of Defense instructions (e.g., see DoD, 2015), designed according to Air Force Policy Directive 36-21, Utilization and Classification of Military Personnel (USAF, 2019d), and implemented according to Air Force Instruction 36-2110, with more than 500 pages on Total Force Assignments (USAF, 2018b). Through its assignment system, the Air Force seeks to “Ensure qualified Airmen with the needed skills, are in the right job at the right time to meet the Air Force mission” (USAF, 2019d, p. 2). In concert, these policies direct the Air Force to implement an equitable process that classifies and assigns Airmen to meet mission and commander needs and achieve a high state of readiness. Individuals are also to be assigned voluntarily, to the maximum extent possible. Additionally, assignments seek to achieve the following:
- Provide special assignment consideration for Airmen who have family members with special medical or educational needs.
- Ensure involuntary assignments are equitably distributed among similarly qualified Airmen, to minimize family separations and to avoid creating severe personal hardship.
- Establish Active Duty Service Commitments or Reserve Service Commitments when Airmen are reassigned on a fully funded permanent change of station, or public funds are used for education or training courses, to ensure a return on the investment.
- Assign officers to joint positions who will be promoted to the next higher grade at a rate not less than the rate for officers in the same grade and competitive category who are serving, or who have served, on the Air Staff (USAF, 2019d, p. 2).
Furthermore, the Air Force manages and assigns Airmen without regard to color, race, religious preference (except chaplains), national origin, ethnic background, age, marital status (except military couples), spouse’s employment, educational or volunteer service activities of a spouse, gender, or sexual orientation. Ultimately, the primary factor in selecting an Airman for an assignment is the Airman’s qualifications to fill a valid manpower requirement and perform productively in the position for which he or she is being considered. Compliance with the assignment policy is assessed by measuring the extent to which (a) authorized positions are assigned Airmen (minimizing vacant positions) and (b) established time on station requirements are met (USAF, 2019d, p. 2).16
16 Time on station is measured in months and starts with the month an Airman arrives at their current duty location. The purpose of a minimum time-on-station requirement is to enhance operational readiness by stabilizing Airmen at their current location, to reduce
Assignments for the enlisted force are completed in 3-month cycles and are driven primarily by rotation to and from overseas assignments. This process is conducted via the Assignment Management System that advertises position requirements through the Enlisted Quarterly Assignments List (EQUAL) and uses a much older technology platform than the Talent Marketplace for officer assignments. See Figure 5-3 for the enlisted assignment process.
A small percentage of enlisted positions in the Air Force have more selective matching practices that include interviews with the hiring authority. For example, Command Chief positions match a Chief to an O-6 commander from a list of candidates identified through a board process—commanders often interview their top two or three candidates before completing their preference sheet. The Air Force is investigating implementing the Talent Marketplace to manage enlisted assignments, but a decision is still pending at this time.
There are two officer assignment cycles per year. To support the Officer Assignment System for all assignments of Lieutenant Colonel and below (except for Judge Advocate General officers), the Air Force uses its newly developed Talent Marketplace. The web-based system provides transparency for available positions, provides visibility on an officer’s preferences to their commander, and incorporates gaining commander input as well for the first time. The technology behind it examines officer assignment solutions by incorporating specific prioritizations from both the officers who are eligible to move and the gaining unit (see Appendix D for discussion of preference informed matching). Officers on the vulnerable-to-move list use the Talent Marketplace to indicate a desirability rating for assignment location preferences using a list of locations with jobs to fill, in alphabetic order. There is limited information on each position in the system: duty title, command, and location, but the officer can see how many other officers are interested in the position and make their decision accordingly.17 Any
permanent change of station costs, and to improve the quality of life of Airmen and their dependents by reducing personal and family turbulence. The current policy has 21 separate rules for minimum time on station (USAF, 2018b, p. 182). In general, moving from one continental United States location to another requires 48 months on station; moving overseas is 24 months; moving from overseas back to the United States is completion of the overseas tour, which can range from 12 months to several years.
17 Notice that if the assignment system does not make it safe for officers to list their preferences truthfully, information about how many others are interested in the position might make it wise for an officer not to list a position that he (along with many others) prefers, out of
additional information the officer would like to know about the position is gathered through their own independent research. After the window closes
fear of the competition. But this deprives the position from precisely that competition, and it is (yet another) reason why it is desirable to make it safe for officers to list their preferences truthfully (so that even if they are not assigned to a competitive position their final assignment is not harmed by the fact that they tried).
for officers to bid for positions, the position owners access the system to see the final list of volunteers to fill their positions. The results of the matching algorithm are used as a first step in the process, which is further adjusted as needed and finalized by AFPC. See Figure 5-4 for the officer assignment process that utilizes the Talent Marketplace.
The idea behind this is to add preferences on both sides of the assignment process because preferences can express information held by position owners (commanders) and officers seeking assignments that are otherwise absent from the assignment process. However, for preferences to be both informed and informative, additional information infrastructure is needed.
There must be a way for candidates (individuals who are available or vulnerable-to-move) to inform position owners of their particular skills and interests. (In private-sector job markets, this kind of information is supplied by transcripts, resumes, letters of references, and interviews.) And there must be a way for positions to inform candidates of the particular features of the job on offer. (In private-sector job markets, this kind of information is supplied by job advertisements and interviews.) In the absence of this kind of information exchange, elicited preferences will not be as informative as they could be (e.g., positions information may not be current), which may negatively impact the overall effectiveness of the HCM system (as described in Chapter 3, being “accurately informed and informative” is a key attribute of an ideal system).
Presently (and often without this additional information infrastructure) candidates and position owners are asked to submit preferences (candidates rank-order positions and commanders rank-order candidates). An initial assignment is calculated using the position-offering deferred acceptance algorithm (see Appendix D for further discussion). However, this initial assignment is just offered to the assignment team as a starting point, which it modifies as it deems necessary.
In this section, the Air Force Officer and Enlisted Evaluation Systems are introduced. These systems are based on the rules and regulations provided in Air Force Instruction 36-2406, Officer and Enlisted Evaluation System, which states that military evaluations serve three purposes:
- Effectively communicate performance standards and expectations and provide meaningful feedback on how those standards and expectations are being upheld.
- Establish a reliable, long-term, cumulative record of performance and promotion potential based on that performance.
- Provide sound information to assist in making talent management decisions (USAF, 2019c, p. 9).
To accomplish these purposes, the evaluation system focuses on Airman performance. The Air Force Deputy Chief of Staff for Manpower, Personnel, and Services is responsible for the evaluation policy. The Air Force Personnel Center is responsible for the operational execution of the evaluation system in terms of keeping the data systems, ensuring the deadlines are met, and storing all final evaluations. Additionally, there are personnel experts who execute the evaluation program at the base level, tracking completion and providing guidance as needed.
The evaluation process for both officers and enlisted Airmen is initiated with the Airman’s Comprehensive Assessment (ACA) that allows Airmen to evaluate their own performance and receive feedback prior to completion of annual performance reports, as discussed in more detail in the sections that follow.
The Air Force has four forms to document annual officer performance: (a) general officers (AF Form 78); (b) colonels and below (AF Form 707); (c) training and education documentation (AF Form 475); and (d) promotion potential (AF Form 709). The general officer form includes both a performance summary and a promotion recommendation. The grades of colonel and below use a separate form for the annual performance summary (AF Form 707) and only use the promotion recommendation form (AF Form 709) when an officer is in the window for promotion consideration. The AF Form 475 is used to document officer performance in formal training or education when the scheduled course length is 8 weeks or more, or as authorized when the specific course is fewer than 8 weeks (USAF, 2019c).
“All enlisted personnel in the grade of [Senior Airman] through [Chief Master Sergeant] will receive an evaluation as of the appropriate static close-out date for their grade. [Airman Basic, Airman, and Airman First Class] will receive an evaluation upon completing a minimum of 36 months in service as of the [Senior Airman static closeout date], 31 March” (USAF, 2019c, p. 104). The Air Force has three forms to document enlisted performance in addition to the ACA worksheet: (a) Airman Basic through Technical Sergeant (AF Form 910); (b) Master Sergeant (including selects) through Senior Master Sergeants (AF Form 911); and (c) Chief Master Sergeant (including selects) (AF Form 912).
Appraisal Forms and Processes
Air Force Instruction 36-2406 comprehensively describes several types of appraisal forms18 and processes for different ranks of enlisted and officer personnel (USAF, 2019c). Specifically, this includes the ACA, Officer Performance Report (OPR), and Enlisted Performance Report (EPR), as described below. In the committee’s view, these evaluations are primarily
18 The forms include the Airman Comprehensive Assessment (ACA, chapter 2), Officer Performance Reports (OPRs, chapter 3), Enlisted Performance Reports (EPRs, chapter 4), Letter of Evaluation (chapter 5), Education/Training Report (chapter 6), General Officer Evaluations (chapter 7), Promotion Recommendation and Management Level Review Process (chapter 8), and Retention Recommendation Form (chapter 9).
subjective appraisals of ratee performance in key areas, which collect data using a constructed-response (open-ended) format and, in some instances, an ordered-categorical rating format. Although all these data points are useful for structuring communication between supervisors (raters) and subordinates (ratees) for development purposes, numerical data are more amenable to system-wide analyses aimed at validating tests to predict future performance and improve assignment and promotion decisions.
Although the ACA, OPR, and EPR seem to be the most similar to performance appraisals typically conducted in the private sector, their methods of collecting data are somewhat different (as described below). Furthermore, there are some noteworthy limitations on how the data may be shared and used for purposes beyond the performance evaluation context, as instructed by Air Force policy and summarized below.
Airman Comprehensive Assessment (ACA)
The ACA is mandatory for all active duty and reserve Airmen, Airman Basic (AB) through Colonel (see section 2.3 of USAF, 2019c).
An ACA is a formal, two-way communication between a rater and ratee to discuss standards, responsibilities, expectations, and goals. Raters document the session on the ACA worksheet and use the performance feedback section to assess or discuss the objectives, standards, behavior, and performance with the ratee (USAF, 2019c, p. 54).
There are several times that ACAs must be offered: initial, midterm, followup, ratee requested, and rater directed, and the forms used are based on rank: (a) Airman Basic through Technical Sergeant, (b) Master Sergeant through Chief Master Sergeant, or (c) Second Lieutenant through Colonel. All forms contain the following sections: a self-assessment completed by the Airman; a summary of the Airman’s role in support of the mission completed by the rater; an individual readiness index completed by the unit deployment manager; a performance feedback completed by the rater; and a section called Knowing Your Airman. This last section must be completed in a formal face-to-face discussion when feasible, or otherwise by telephone (USAF, 2019c, p. 63). Raters at all levels are encouraged to distinguish performance among peers and to guard against inflation, which acts as a “disservice” to all (USAF, 2019c, p. 71).
The performance feedback section of the ACA items mirrors the items that are documented on the Airman’s annual performance report and as such should make the rater’s end-of-year evaluation very transparent. This feedback process is started at the beginning of each performance period and revisited again half-way through the performance period. There are also opportunities for follow-up feedback, requested feedback, and directed
feedback. At the end of the rating period the Airman’s supervisor completes a performance report.
It is important to mention that AFI 36-2406 section 2.9 (Disposition and Access) indicates that the ACA is not to be made an official part of any personnel record or Personal Information File; it can only be reviewed by the rater, ratee, and authorized personnel for performance evaluation unless the ratee grants access to it or introduces it in a personnel action (USAF, 2019c). In the committee’s view, although this restricted use of the ACA is not unreasonable for development purposes, it precludes data sharing that would facilitate research on prospective predictors of performance that could augment or replace training and experience qualifiers for assignments and promotions.
Officer Performance Report (OPR)
The purpose of the OPR is to “document performance and potential as well as provide information for making promotion recommendation, selection, or propriety action; selective continuation; involuntary separation; selective early retirement; assignment; school nomination and selection; and other management decisions” concerning officers (USAF, 2019c, section 3.2.1). With some exceptions, all Colonels and below (other than Brigadier General selects) are subject to this evaluation either annually or biennially.
Officers also use an ACA appropriate for the rank to gather information from the Airman and initiate a dialogue about the Airman’s well-being and performance in broad areas, such as Job Knowledge, Leadership Skills, Professional Qualities (includes adherence to standards), Organizational Skills, Judgment and Decisions, and Communication Skills. However, unlike the EPR (discussed below), OPRs use a dichotomous (Meets/Does Not Meet Expectations) judgment. If the ratee does not meet performance factors, it triggers a referral evaluation (USAF, 2019c, section 1.10) “to allow the ratee due process by giving the ratee an opportunity to respond and/or rebut any negative ratings or comments before it becomes a matter of record. Additionally, it allows evaluators to consider all the facts, including any they may not have been aware of, before the evaluation becoming a matter of record” (USAF, 2019c, p. 33). There is also a Rater Overall Assessment, which requires at least one open-ended comment using a bullet format (USAF, 2019c, section 188.8.131.52).
Enlisted Performance Report (EPR)
The purpose of the EPR is to “determine selections for promotions, job, and school recommendations, career job reservations, reenlistments, retraining, and assignments” (USAF, 2019c, section 4.3). The enlisted force also uses the ACA to provide feedback on ratee performance in major areas and subareas.
TABLE 5-1 Enlisted Forced Distribution Panel Scale
|9.0||Few could be better|
|8.0||Slightly above average|
|7.0||Slightly below average|
|6.5||Well below average|
SOURCE: Air Force Instruction 36-2406 (USAF, 2019c), Table 4.1, p. 123.
Ratees placed in the top two promotion categories must be considered by a “Force Distributor” (section 4.18), which is a designated commander or panel (an Enlisted Forced Distribution Panel [EFDP]) that scores the ratees on a best-qualified basis, as shown (AFI 36-2406 section 4.18.9 and Table 5-1).
Military Promotion and Demotion
Promotion requires the identification of Airmen with the “highest potential to fill positions of increased grade and responsibility” (USAF, 2018a, p. 2). Promotion selections are made to ensure Airmen who are promoted are fully qualified19 and best qualified,20 which values duty performance and potential based on that performance but not to the exclusion of other factors. According to Air Force Policy Directive 36-25, “When appropriate, and as permitted by law and consistent with the Department of Defense policy, Airmen may be demoted to a grade commensurate with their demonstrated abilities and skills” (USAF, 2018a, p. 2). Demotions are rare;
20 “The fundamental purpose of the officer promotion program is to select officers through a fair and competitive selection process that advances the best qualified officers to positions of increased responsibility and authority and provides the necessary career incentive to attract and maintain a quality officer force” (USAF, 2020c, p. 12). Furthermore, the “Best Qualified Method of Selection” is defined as a “Requirement that selection boards may recommend for promotion only those officers whose records, when compared with those of other eligible officers, reflect the greatest potential to serve successfully in the next higher grade” (USAF, 1997, p. 2).
when they do occur, they occur more often among enlisted Airmen rather than officers because enlisted demotions are not as heavily regulated in law and the authority to demote is delegated at a much lower level. However, there have been several recent cases where general officer misconduct was substantiated and resulted in a demotion, including retroactive demotions several years after retirement, to the last grade they served in honorably.21
As vacancies occur, the officer promotion program allows the Air Force to promote officers of the desired quality, in the proper grades, in sufficient numbers to carry out the mission. This ensures maintenance of the strength of the Air Force in each grade and also helps to retain a highly qualified and motivated officer force by offering them reasonable career progression. Ultimately, promotion is based on the officer’s readiness to take on the responsibilities of the more senior grade. See Figure 5-5 for an overview of the officer promotion process.
In line with the FY2019 National Defense Authorization Act, a competitive category is “a separate promotion category established by the Service Secretary for specific groups of officers whose specialized education, training, or experience and often relatively narrow utilization, makes separate career management desirable.”22 In October 2019, the Air Force formally established six new officer developmental categories for active component officers in the competitive category, “Line of the Air Force;” in May 2020, these took effect with the board considering those eligible for promotion to Lieutenant Colonel. Previously, the Line of the Air Force competitive category included approximately 80 percent of the officer corps, encompassing more than 40 different officer AFSCs. As such, officers in this category competed for promotion against officers with little consideration of varied job requirements or other factors. Under the new arrangement, officers will compete against officers within specific developmental categories, in career fields that have “similar progression milestones, experiences, and mission area focus.”23 The new Line of the Air Force developmental categories are Air Operations and Special Warfare, Nuclear and Missile Operations, Space Operations, Information Warfare, Combat Support, and Force Modernization. In addition to the six new categories,
21 For example, see Losey (2017), “Retired 4-star demoted after sexual assault investigation,” Air Force Times. Available: https://www.airforcetimes.com/news/your-air-force/2017/02/01/retired-4-star-demoted-after-sexual-assault-investigation/.
23 See Air Force Public Affairs news release, October 23, 2019, available: https://www.afrc.af.mil/News/Article-Display/Article/1996233/air-force-announces-new-officer-developmentalcategories/.
the Air Force has had the following competitive categories for many years: Judge Advocate General, Medical Corps, Dental Corps, Chaplain, Medical Services Corps, Biomedical Sciences Corps, and Nurse Corps (USAF, 2004).
Under current policy, to be eligible for promotion, Airmen must meet merit criteria and requirements for time in the current grade as defined by law and several Air Force policy documents (e.g., 10 U.S.C. §619; USAF, 2004, 2018a, 2020c). Accordingly, Second Lieutenants and First Lieutenants are eligible for promotion upon completing 24 months in their current grade. In most cases, a board considers First Lieutenants for promotion after they have served 9 to 21 months, and if selected, they are promoted to Captain after completing 24 months as a First Lieutenant. Captains, Majors, and Lieutenant Colonels are not eligible for “on-time” consideration, referred to as in-the-zone, to the next higher grade until they serve 3 years in their current grade. Generally, a central selection board considers officers for Major in-the-zone after about 10 years of service; for Lieutenant Colonel after about 15 years of service; and for Colonel after about 20 years of service. The Secretary of the Air Force has the authority to consider officers for early promotion, referred to as below-the-zone promotions. However, in a recent memorandum to the field (Losey, 2019), Secretary Barrett and General Goldfein announced that the Air Force is transitioning out of the current below-the-zone promotion option and, beginning with May 2020 Lieutenant Colonels, has adopted a merit-based approach to pin-on times. This decision was possible as a result of recent changes in the DOPMA, enacted through an amendment to federal law, that now allows for promotions to be made “in the order of the seniority . . . or based on particular merit, as determined by the promotion board” (10 U.S.C. §624(a)(1)). Before this change, regardless of an officer’s order-of-merit on the promotion board list, the timing of the pin-on date was based on their date of rank. This sometimes resulted in an officer in the top 5 percent on the order-of-merit from the promotion board coming at the bottom of the pin-on list because of their date of rank. The new legislation prevents this situation, and officers who are higher on the order-of-merit list from the board will pin-on sooner.
Promotion zone, promotion opportunity, and selection rates
Service Secretaries establish promotion zones for each grade (i.e., early, on-time, late) and competitive category, using the date of rank for officers in the competitive category, and should balance promotion opportunities within categories over the next 5 years. The promotion opportunity is the approximate percentage of officers in an accession year group that a central selection board will select for promotion to the next higher grade independent
of the zone (i.e., on-time, early, and late).24 As such, it is the maximum number of officers that a board can select. The selection rates are the actual percentage of on-time, early, and late promotions that are then selected.
The Secretary of the Air Force convenes selection boards who select officers for promotion across each competitive category for Captain through Major General. Federal law provides specific requirements for the composition of officer selection boards and the processes to be followed (10 U.S.C. §612, 619, and 14102). For the Air Force, the Secretary of the Air Force administers the promotion program, including, for example, appointing, convening, and instructing selection boards and establishing “competitive categories, promotion zones, eligibility and selection criteria, promotion opportunity and selection rates (USAF, 2004, p. 9).
Whenever a selection board25 is convened, each officer in or above the promotion zone is considered for promotion with a few exceptions granted in law (e.g., the Service Secretary can exclude officers with an established separation date within 90 days of the board convening). When the central selection board convenes, board members review each officer’s Headquarters U.S. Air Force Officer Selection Record, which includes: Officer Performance Reports, a Promotion Recommendation Form, an Officer Selection Brief (one-page data summary of officer’s career), a letter to the board submitted by the eligible officer, if applicable, and information not part of the official military record of an officer. (This other information is typically adverse information of a very serious nature that for one reason or another is not in the individual’s record. This information must be approved directly by the Secretary of the Air Force and gets a significant legal review.)
Board members and tasks
Selection boards are convened by and work directly for the Secretary of the Air Force, who provides members with written guidance covering, for example, the board’s purpose, task, methods, and other specific instructions. In a pamphlet to explain officer promotions, the selection, qualifications, and expectations of board members are described as follows:
24 By federal law, there also exists “Authority to Allow Officers to Opt Out of Selection Board Consideration” (10 U.S.C. §619) and “Alternative Promotion Authority for Officers in Designated Competitive Categories” (10 U.S.C. §649).
25 Board members serve in the grade of Colonel or General depending on what grade the promotion board is promoting to. For promotions to Colonel and General all the board members are general officers. The number of members that serve on an individual board depends on how many records the board will review. The system is honed such that it has a good idea of how many board members it needs to complete boards on schedule through scoring a certain number of records each day.
The Air Force selects only highly qualified senior officers with extensive experience and mature judgment to serve as board members. Collectively, they represent the broadest practical scope of Air Force activities. To provide a balanced perspective on the Air Force mission, the Air Force selects officers who mirror, as much as possible, the officers they are considering with respect to race, sex, aeronautical rating, career field, and command of assignment. . . . Board members are required to perform their duties based on the best interest of the total Air Force and take an oath affirming that they will do so (USAF, 1997, p. 10).
To score records, board members are divided into panels and, using secret ballots, each board scores every record assigned to the panel on a 6–10 scale: 6 is below average, 7.5 is average, and 10 is absolutely superior (USAF, 2004, p. 19). If there is a difference of more than 1.5 points between 2 or more board members (referred to as a split), it must be resolved by the members with the split.
After all the records are scored and any splits resolved, the final scores are totaled and sorted by order of merit. Board promotion quotas are computed and adjusted, as necessary, in accordance with Air Force Instruction 36-2501 (USAF, 2004) and its implementation guidance (USAF, 2020c). Ultimately, Airmen are selected for promotion, from the top of the order-of-merit list, working down according to instructions on establishing initial and secondary cut lines to fill, but not exceed, allocated quotas.
Officers not selected for promotion on-time (in-the-zone) are considered again by a second board (their first late look). Generally, officers not selected by a second board for Captain through Lieutenant Colonel are separated or must retire if eligible approximately 7 months after the board results are announced. However, in line with the FY2019 National Defense Authorization Act, there is also a process for selective continuation for officers that possess a critical skill (Kidder, 2019) as defined by the Secretary of the Air Force and based on retention models developed by analysts working under the Office of the Deputy Chief of Staff for Manpower, Personnel, and Services (AF/A1). The AF/A1 reviews the critical skills list for every promotion board and makes a recommendation to the Secretary of the Air Force on which skills to continue, specific to each grade.
The purpose of the enlisted promotion program is also to select enough enlisted Airmen of the desired quality, in the proper grades, to carry out the mission. However, enlisted promotions have very little legislation governing the process, which gives the Services wide latitude to tailor enlisted promotion programs. See Figure 5-6 for an overview of the enlisted promotion process.
To sustain the Air Force at any given end strength, the Air Force uses retention models developed for each unique AFSC (e.g., see Mattock et al., 2019). The Air Force uses these models to determine how many Airmen to access each year and where to apply its many retention levers to address shortages. Modeling by career field is critical because each skill set has a unique retention pattern. The Air Force has many tools, including both monetary and non-monetary incentives, in its toolkit that can be used to retain the talented Airmen it needs to meet the nation’s warfighting requirements. One tool the Air Force has is a robust Special and Incentive pay budget that allows it to target Airmen in critical career fields with monetary incentives at certain points in the Airmen’s career to retain them for a set number of years. These financial incentives are codified in Title 37, Chapter 5, Subchapter I–Special and Incentive pays, and are separated into the following broad categories: Hazardous Duty Incentive Pay, Arduous Duty Pay, Assignments, Career Incentives, Accessions, Proficiency, Retention, Responsibility, Rehabilitation, Skill Conversion, Transfer Between Services, and Medical. The sustainment modeling the Air Force does allows it to efficiently use these monetary incentives to target the right skilled Airmen at the right time to sustain the force. The use of monetary incentives typically focuses on enlisted Airmen (the principal Air Force officer beneficiaries are pilots and medical personnel—a reflection of the investment in their training and the outside competitive labor market).
That said, retaining Airmen to sustain the force is not all about money. Often it is more about Air Force policies than financial compensation. The Air Force’s recent implementation of the Talent Marketplace to provide more transparency to both the Airmen and the hiring authority is one such policy. It leverages modern technology to increase the satisfaction of both the Airmen and the hiring authority in the assignment process and should have positive impacts on retention. Many other policy changes have positively influenced retention. The following list is an example of such policies but is by no means all-inclusive: changes to high-year-of-tenure policies for enlisted Airmen; allowing retired Airmen to come back on active duty; liberal retraining programs to allow greater flexibility in assigned career paths; expanded maternity leave; sabbatical programs; changes to the fitness policy post-pregnancy; and changes to the tattoo policy. The Air Force can also adjust active duty service commitments to secure service for a longer period associated with certain training schools. For example, the Air Force increased the active duty service commitment for pilot training from 8 years to 10 years to get a better return on its investment by locking in 2 additional years of tactical flying capability for every pilot the Air Force trained. Using the many retention tools available in a synchronized, data-informed process has allowed the Air Force to help minimize manning gaps.
Similar to the research support needed to ensure that Air Force entry point decisions are valid and evolve appropriately to meet strategic priorities and reflect new capabilities (as discussed in Chapter 4), post-accessions personnel decisions should also be informed, designed, and validated by internally and externally conducted research efforts. To support post-accessions decisions in the development, utilization, and retention of Airmen, research is likewise needed to sustain the current HCM system’s performance, ensure continuous incremental improvements keep the system in line with best practices, and innovate new processes and assessments for competitive advantage in developing and managing successful Airmen across a career. To effectively achieve this, the Air Force uses many of the same internal and external research organizations described in Chapter 4. In the remainder of this chapter, the committee presents several key research opportunities that are necessary if the 21st century Air Force is to improve its existing policies or programs and develop innovative and agile alternatives for the future. In this section, the committee discusses research opportunities in the following key areas:
- Training and Development Needs Assessments;
- Training and Development Effectiveness;
- Post-Accession Assignment Decisions;
- Enlisted and Officer Promotions; and
- Performance Appraisals.
Based on the committee’s major findings or conclusions presented in boldface throughout the remainder of this chapter, these subsections describe research needed to improve post-accession personnel decisions impacting force development, utilization, and retention, and to inform implementation of the actions called for in this report’s Flight Plan given in the final chapter.
Training and Development Needs Assessments
Training and development needs assessments operate at multiple levels. The most common assessments identify knowledge, skills, and abilities required for a particular job. The Air Force conducts task analysis and competency modeling to support both enlisted and officer training in this regard (see discussion in Chapter 4, especially research and alignment needs in this area), and the committee found evidence that this approach seems to be generally effective in informing adjustments to training curricula, albeit perhaps not at the speed desired.
Needs assessments also can focus on individual Airmen in terms of what each person might need in the way of development. To this point, during the study, the committee learned about the many ways in which the Air Force has moved to more adaptive training, wherein individuals can advance through training at a pace that is more suited to the individual’s needs.26 Airmen need not sit through training on activities, equipment, or procedures for which they are already competent. Such adaptive training might be tested on a trial basis, and the net result could make the overall training more effective and efficient, saving both time and money (as described in Chapter 3, being “agile and flexible” and “innovative yet disciplined” are key attributes of an ideal system). In fact, the committee saw evidence of these types of assessments in use (e.g., schoolhouse testing, officer career development planning), but this could be an area for refinement and expansion through a systematic program of research to inform, design, and validate a complementary suite of training and development programs across all AFSCs and ranks.
First, data accrued on Airmen could directly inform their training to make it more efficient. That is, if it has been verified that Airmen have acquired certain knowledge, skills, abilities, and other characteristics (KSAOs) either through previous experience or education, then they should start training where their KSAOs leave off. Second, developing an expanded suite of self-assessment instruments for key knowledge and skill areas (e.g., cyber, leadership) will allow individual Airmen to understand their own needs and to take greater ownership and direction of their development. Further, an expanded suite of skill assessments would allow the Air Force to track data on skill reserves and deficiencies and make adjustments in training seats as well as in online course development and outlets.
Expanding assessments that individual Airmen can take “anytime, anywhere” will allow for a maximum level of adaptive training, with individuals taking on development activities as needs are identified rather than as determined by a more centralized and inflexible system. Creating and using assessments of focal skills and abilities throughout the Air Force expands this notion of adaptive training even further, to allow (a) individuals to determine whether they would benefit from additional training and development, even after completing required training; (b) individuals to begin self-development activities before entering a formal training program, potentially speeding up their time to training completion; and (c) identification of individuals who already possess higher levels of skills and abilities,
26 The topic of adaptive training was discussed with Air Force representatives during site visits to Joint Base San Antonio-Randolph (November 5–8, 2019), Wright Patterson Air Force Base, Dayton, OH (November 13, 2019), Keesler Air Force Base, Biloxi, MS (January 9, 2020), and Maxwell Air Force Base, Montgomery, AL (January 15, 2020).
thus expanding the potential pool of eligible Airmen for assignments as well as the pool of assignments available to the individual.
Although training and development needs assessment occurs at the level of the job and the level of the individual Airman, the committee also notes the value of continued assessment at the organizational level. Through multiple mechanisms, the Air Force has invested considerably in identifying future force needs to ensure it is capable of providing air superiority in the all-domain battle of a joint forces fight (e.g., see Harrison, 2019; USAF, 2015; and Shlapak, 2006). In fact, this is a major objective of the recently established Air Force Warfighting Integration Capability (AFWIC) (see AFWIC, 2020).27A systematic program of research to integrate needs assessments that focus on the job, on the individual Airman, and on the future force would improve coordination and planning capabilities. For example, integration of data gathered on training needs from different viewpoints may provide valuable information on force readiness to enable more timely and targeted implementation of critical changes in training systems (as described in Chapter 3, being “collaborative” is a key attribute of an ideal system). Addressing this need is particularly time sensitive if it is to influence the design and implementation of the Air Force’s new Virtual Test and Training Center, “a hub for advanced synthetic training that could eventually integrate joint and international partners” (Albon, 2020). The lack of an integrated systematic approach linking (a) needs to research and (b) multiple research programs addressing related topics, results in suboptimal systems (e.g., using different metrics, misaligned foci) and a lack of sharing of data important for research efficiency and the ability to validate results.
Training and Development Effectiveness
One of the most basic research questions in the area of training and development is whether the effort was effective. Evaluating the efficacy of the training and development efforts in the Air Force requires the specification of training objectives (e.g., to improve an individual’s skill in a particular area, to enhance the general levels of skill across the Air Force in terms
27 The committee learned about the mission and vision of AFWIC when it met with key representatives of the office during the committee site visit to Air Force Headquarters at the Pentagon (September 26–27, 2019) and a subsequent virtual meeting with Maj. Gen. S. Clinton Hinote, USAF, who was serving as Director, AFWIC (as of May, 2020, Hinote was selected for appointment to the rank of Lieutenant General, with assignment as Deputy Chief of Staff, Strategy, Integration, and Requirement). Specifically, AFWIC, “guided by the National Defense Strategy, enables the Department of the Air Force to rapidly identify areas for new investment in capabilities that will provide the backbone and connective tissue to allow the Joint Force to conduct true multi-domain operations, even in the most difficult scenarios” (see https://www.afwic.af.mil/).
of future readiness, to allow for greater flexibility in assignment processes). Next, measures of those training objectives (outcomes) are needed for use as criteria when evaluating training and development efforts (e.g., scores of individuals on end-of-training assessments, the fullness of pipelines for certain critical positions, individual Airman satisfaction with potential career paths). Researchers can then evaluate whether those outcomes have been met.
Within the Air Force, there is a healthy focus on end-of-training assessments and whether individuals leaving basic and specialty skills training have the requisite skills and knowledge to perform their jobs. Although the committee heard about ways in which this feedback loop from the quality of job performance to training could be enhanced and made nimbler, these aspects of training evaluation are currently being attended to, particularly in critical areas such as pilot training, cyber, and special operations. Certainly, there are always ways to improve upon these classroom assessments, particularly concerning integrating new technologies and new assessment formats (e.g., more situational and hands-on evaluations using virtual reality). However, there are some areas where the committee heard distinct calls for improvement in the research support and integration related to training effectiveness.
One such area is evaluation following less-than-successful performance. In this instance, the committee found that when Airmen are less than successful (either in training or subpar performance after training completion), systematic and continual evaluation of factors contributing to those failure points is not as integrated or consistent as would be ideal. Sporadic attention occurs when attrition rates spike or when performance lapses become glaring; however, an ideal training and development evaluation system would incorporate continual assessments and monitoring. For example, databases that incorporate information on Airman training performance integrated with performance management information could provide better information on where, when, and how training can be enhanced. As another example, representatives of the USAFA indicated a desire for higher quality data systems that would allow a feedback loop to the Academy on the subsequent success of cadets throughout their career.28 A human capital data superstructure could prove valuable to USAFA in understanding areas of needed curriculum enhancements and relative preparation for various officer career paths in the Air Force.
Next, training evaluations need to provide enough diagnostic information to determine enhancement needs and approaches. High attrition by itself does not indicate whether improvements are needed in the training
28 Representatives of the USAFA discussed this need with committee members during the site visit to USAFA in Colorado Springs, CO (January 17, 2020).
curriculum, end-of-training assessments, classification and placement into training, or in accession standards themselves. An integrated system of evaluation that tracks data from accession through on-the-job performance would certainly help in such an analysis.
Finally, relevant research emphasizes the importance of facilitating the transfer of training through post-training support, refresher training, and the like. For critical positions (e.g., pilot), one can see how skills are maintained and enhanced post-initial training; similar ways of assessing how well individuals maintain and update their knowledge and skills in all assignments would require greater data collection, but most importantly, greater integration of existing data on Airmen.
Post-Accession Assignment Decisions
As described previously, Air Force assignments provide both the manpower needed by the force and the experiences needed by Airmen as they advance through positions with increasing roles and responsibilities. Recently, an increased focus has been placed on modifications to the Air Force Assignment System, particularly for officers through the Talent Marketplace, to promote transparency in the assignment process and facilitate greater consideration of Airmen preferences in making those assignments29 (as described in Chapter 3, being “understood and trusted” is a key attribute of an ideal system). Consequently, this section is focused on the research needed to expand and improve consideration of preferences in post-accession assignment decisions.
As the Air Force expands its use of the Talent Marketplace for officers and develops a more modern approach to the antiquated Enlisted Quarterly Assignments List, it could benefit from considering the research conducted on and the implementation and results of similar marketplace initiatives (see for example, Malia, 2020). The U.S. Army, in particular, recently implemented its Army Officer Assignment Marketplace via Assignment Interactive Module (AIM).30 AIM is a centralized clearinghouse that requires officers and units to finalize preferences for the other side of the market at the same time, typically 6–9 months before officers are expected to move to their next assignment. The Army studies its assignment system in the Office of Economic and Manpower Analysis (OEMA), Department of Social Sci-
29 Information about the Talent Marketplace was provided to the committee during the site visit to Joint Base San Antonio-Randolph (November 5–8, 2019) and a follow-up conference call (November 18, 2019).
30 For more information on how AIM works, see https://www.army.mil/article/221864/five_things_army_officers_and_units_should_know_about_the_assignment_interactive_module.
ences, West Point.31 Information provided to the committee on the design and administration of AIM indicated that it is something of a hybrid model that involves a lot of hand-processing of assignments by assignment officers. However, it was reported that the “Human Resources Command received permission from the Assistant Secretary of the Army for Manpower and Reserve Affairs to test the concept of matching officers to jobs according to a deferred acceptance algorithm similar to that of the National Resident Matching Program,”32 which matches new doctors to residency programs (Roth, 1984; Roth and Peranson, 1999).
One aspect of the Army Cadet Branching process (used to assign new West Point graduates to branches) is that Cadets can bid for preferred assignments by offering to extend their service obligation. Although a detailed review of AIM was outside the scope of this study, the committee notes that analysis has suggested that its use of a deferred acceptance algorithm is seriously flawed in implementation (e.g., Switzer, 2011), and this may be important for the designers and administrators of the Air Force Talent Marketplace to study and understand.
As of May 15, 2019, the Army reports that 90,952 officers of all ranks had participated in AIM. Officers are encouraged to submit resumes, but the majority have not done so (from 7% of Colonels, up to 42% of Majors). Resume-writers are encouraged to include unique Knowledge, Skills, and Behaviors (KSBs), but almost none have done so. Units seeking to fill jobs are encouraged to submit (long) preference lists of officers for each job they are seeking to fill, and so far, participation has been spotty: “The best participation is from [the 10 percent of] units who submit preferences for many jobs, and for multiple officers on each job” (U.S. Army, 2019, p. 2). Although no explanation was provided to the committee, the Army’s overall low participation numbers suggest that the Air Force and Army could benefit from joint efforts to understand the reaction Airmen and Soldiers have had to these initiatives as well as to develop approaches to improve participation and therefore its overall utility.
Turning back to the Air Force, the present (preliminary) use of deferred acceptance algorithms in the Talent Marketplace is not an evidence-based decision; rather it arises from a perhaps too-hasty parallel with the operation of the private-sector clearinghouse for American physicians (see discussion of preference informed matching in Appendix D). To guide Air Force research into post-accession assignment decisions to improve and expand
31 These paragraphs are based in part on a committee phone meeting with Maj. Kyle Greenberg, USA, OEMA on September 23, 2019.
32 Personal email between Maj. Kyle Greenberg, USA, OEMA and Alvin Roth, Committee Member, September 24, 2019.
the effectiveness and validity of the Talent Marketplace, the committee identified the following research questions:
- How do assignments affect separation decisions?33 (Collect data on “Who stays, who leaves, where they go, would they have stayed for a better assignment?”)
- How are preference data related to family data? (Collect family data on jobs of spouses, age, and schooling of children)
- How is the new blended retirement system changing separation choices?
- How do pilot separations interact with airline hiring demands? (E.g., could new arrangements be initiated between the Air Force and commercial airlines to better meet fluctuating needs of both?34)
- Can exit interviews be combined with interventions that might prevent or delay separation by the most effective Airmen?
- How should the Talent Marketplace be organized for minimizing early separation of the most effective Airmen?
- Does the Talent Marketplace assure an appropriate distribution of talent across units?
- How does the Talent Marketplace affect individual and unit performance?
Enlisted and Officer Promotions
As noted earlier, Air Force policy documents outline the general process used to promote enlisted Airmen and officers to various ranks. However, the Air Force lacks a systematic program of research to evaluate and
33 It was reported that within the Air Force, little data are collected regarding resignations after assignment—leaving interpretation to be mostly anecdotal. Indication that this data matters was relayed by Col. Justin Joffrion, USAF, who observed, “We were decimating our members with ‘365’ one-year deployments overseas, filled by a rigid order of who was home stationed the longest. Once selected for a 365-day deployment, members had three days to accept or separate, so the detailer would go down the list forcing separations until a member accepted the deployment. Now we are trying to solicit volunteers with incentives about their next assignment (e.g., come back to your current location, or move to a family-friendly place). But it is challenging to measure the effectiveness of these incentives because we are not tracking those data! For instance, we do not explicitly track who was offered the 365 and who accepted. Even with bonuses, when we looked back at 12 years of enlisted bonuses to determine retention impacts, we had to get pdfs to try to reconstruct who was eligible at the time the bonus was offered, and then see who accepted. Program and policy evaluation requires painstaking data work.” Information received during call between Col. Justin Joffrion, USAF, AF/A1, and Alvin Roth, Committee Member.
strengthen the process its selection boards use to make enlisted/officer promotion decisions and to ensure that they are in line with professional standards and best practices. For example, through the course of this study it was not clear to the committee the extent to which promotion decisions are made based on the information and/or assessments that have been validly linked to performance and retention of Airmen at the rank of interest. For example, for promotions from E-3 to E-4, commanders have the authority to promote 15 percent of their eligible Airmen early using a board process to review enlisted performance reports and personal information files (USAF, 2019e). The Air Force could strengthen its E-3-to-E-4 promotion process by systematically examining the specific types of information commanders and board members use when making these decisions and validating that factors under consideration actually relate to subsequent performance and retention at the E-4 level. During this study, the committee found no evidence that this has been done, nor that there is a system in place for tracking and periodically evaluating whether factors commanders and board members are considering are consistent with job analysis or competency modeling results focused on the E-4 level. Simply documenting the factors commanders and board members consider, and how they combine that information to make a decision will give the Air Force a critical baseline for starting to evaluate and improve the process, with the goal to positively impact downstream outcomes such as Airmen performance, retention, and diversity at the E-4 rank. If the Air Force were to identify factors that were predictive of subsequent Airmen performance and retention at the E-4 level through such an evaluation, it could benefit from screening all E-4 candidates on those factors, not just the 15 percent considered for early promotion under the current system.
The issues above are far from germane to only E-3-to-E-4 promotions. Analogous needs and opportunities for improvement also exist for promotion at higher enlisted and officer ranks. For example, the Air Force currently maintains minimum time-in-grade (6 months) and time-in-service (3 years) requirements for promotion from E-4-to-E-5; however, based on the information available, the committee could not determine whether those thresholds have any defensible basis in actual performance and retention at the E-5 level. For example, the committee was unable to determine how recently, if ever, the Air Force has conducted a study that evaluates the defensibility of those thresholds from the perspective of E-5 performance-retention (e.g., would Airmen with 4 months in grade perform worse or attrit at higher rates from E-5 than those with 6 months in grade?). Similar questions can be posed for time-in-grade and time-in-service requirements for promotion to various officer ranks as well. There are professional standards and processes for establishing and evaluating such minimum requirements. Subjecting all of its promotion minimum requirements (not
just at the E-5 level, but for other enlisted and officer ranks as well) to such standards may expand the pipeline of promotion-eligible Airmen without necessarily lowering expected performance or retention at higher ranks (e.g., see Buster et al., 2005).
For promotion to E-5 and E-6, the current process makes use of the Weighted Airman Promotion System (WAPS), which includes a specialty knowledge test, a general military knowledge test, EPR scores, as well as other components (USAF, 2019e). WAPS was originally developed to replace promotion boards with a system that used information available to those boards to model—in a highly reliable manner—the exact decisions that those boards were making.35Although WAPS does what it was designed to do (in terms of providing Airmen with a completely transparent and highly reliable promotion system), it is not clear that WAPS and its current weighting36 provides an optimal system for promoting those Airmen who will perform the best at the next higher grade. The Air Force would benefit from a systematic review of the WAPS and its components to ascertain whether (a) it assesses characteristics deemed critical to performance or retention at the E-6 level (and that are needed at the entry to the E-6 level); (b) it has any gaps concerning assessing characteristics deemed critical to performance or retention at the E-6 level (and that are expected upon entry to that level [e.g., an assessment of important non-cognitive characteristics]); (c) it assesses any characteristics not deemed critical to performance or retention at the E-6 level (suggesting source invalidity in promotion scores and wasted assessment time); and (d) it is optimally weighting components to predict E-6 performance and retention.
Although the committee understands that WAPS has undergone at least two successful evaluations of its ability to capture the policies of promotion boards (i.e., evaluations of its designed purpose), the committee found no evidence during our study that a systematic evaluation has been done of WAPS’ ability to predict performance at the higher grade, or that there is a system in place for tracking and periodically evaluating how WAPS content or weighting may need to change over time in light of shifting demands at the E-6 level. Research and maintenance work continues on the content of the job knowledge test component of WAPS,37 but that is only one narrow part of the broader WAPS and E-6 promotion process that can impact its
35 In technical terms, WAPS is based on a policy-capturing ordinary least squares regression model.
36 The committee notes that the components included in WAPS and the weighting of those components have changed over time, possibly negatively impacting its policy-capturing optimization.
37 This information was provided during the committee site visit to Joint Base San Antonio-Randolph (November 5–8, 2019).
validity and defensibility for identifying high-performing E-6 Airmen who are likely to remain in the force.
For promotion to various officer ranks and higher enlisted ranks (E-7, E-8, and E-9), the Air Force adopts a promotion board process (as discussed above). But the specific factors that board members consider when evaluating the information on an Airman are not documented, making it difficult to validate whether those factors are reflective of key KSAOs or competencies required for effective performance or linked to retention at the E-7 level and above, or at the target officer rank in question. Again, the committee found no evidence that such an examination has been done, or that there is a system in place for tracking and periodically evaluating how such decisions may need to change in light of shifting demands at higher levels of enlisted or officer leadership or rank (as described in Chapter 3, being “mission responsive” and “accurately informed and informative” are key attributes of an ideal system).
Evaluating the current promotion board processes could yield even stronger cadres of enlisted leaders and officers. For example, with the baseline described above, the Air Force would be in a position to evaluate whether types of information being used by boards to make promotion decisions relate to downstream outcomes such as senior enlisted and officer job performance, retention, and diversity at a target rank of interest. It is important that the Air Force treat promotion decisions made at each enlisted and officer rank as assessments of individuals, subject to the same scrutiny as any test or decision-making process an employer might use to hire people for a job. There are clear professional guidelines and standards that are pertinent for evaluating personnel decision-making processes that the Air Force should consider as it evaluates the validity of its promotion board decision-making process (e.g., AERA, APA, NCME, 2014; SIOP, 2018). A critical component of these guidelines and standards is the link made between the information used to make personnel decisions and outcomes of value for the employer (e.g., typically employee performance and retention in the employment context). Although promotion decisions made via boards can be effective, the committee found little evidence that the Air Force is engaged in a systematic program of research to evaluate and strengthen the process its promotion boards use to make enlisted/officer promotion decisions that are in line with professional standards and best practices.
Without a systematic examination of the issues above, enlisted and officer promotion boards may be using the data available to them in a suboptimal way when making decisions that impact Airmen’s performance or retention. Without such a systematic study, the Air Force has no way to compare its current practice to any potential alternative in determining who gets promoted among a pool of promotion eligible Airmen. For example,
the committee understands that the Air Force values diversity and would like to increase diversity in senior grades. The Air Force provides guidance to promotion boards that diversity is an important goal, and tracks each board’s promotion decisions with regard to diversity. By pursuing the research into the promotion board decision-making process described above, the Air Force would be in a better position to clarify the types of information that boards should consider, determine how that information should be combined to make promotion decisions, and elucidate how the process could be better aligned with professional best practices. Such research could also help guide the Air Force’s progress with regard to diversity.
As described previously, enlisted and officer personnel are evaluated throughout their careers to facilitate personnel decisions such as promotions and assignments, training, or advanced education opportunities. These evaluations include structured tests to measure KSAOs, job knowledge, and core competencies. They also include performance appraisals, conducted at regular intervals throughout an Airman’s career, making them a rich source of longitudinal data about an individual Airman.
First, the Air Force collects a great deal of data on the performance of Airmen at different points during their careers. However, it does not appear that these data are systematically aggregated and made available in de-identified form for personnel research, which could improve decisionmaking and help track proficiency and readiness over time. For example, the EPR yields quantifiable job performance and promotion readiness data that could be used as criteria for validating personnel tests. As introduced in Chapter 4, a “Human Capital Data Superstructure” would be beneficial as a central personnel repository for these and other kinds of data. Such a system could provide different units that need to make personnel decisions (e.g., readiness, deployment, promotion) with access to crucial data and support much-needed research that would determine the effectiveness of personnel decisions.
Second, there is considerable variation in the performance attributes, and in the methods of evaluating these attributes, across Airmen of different ranks as well as between enlisted and officer personnel. Greater consistency in the design of performance appraisals, with emphasis on numerical ratings that are linked to behaviors or critical incidents, should facilitate improved comparisons of individuals across contexts and levels (Murphy and Cleveland, 1995).
Finally, the rich qualitative data that are gathered in the “Comments” section of these appraisals can be aggregated and analyzed using data science methods that can identify recurring themes and patterns. Advances in
text analysis that leverage new computing power and artificial intelligence/machine learning make previously laborious handling of qualitative data potentially usable as a manageable data source. If captured and stored in a usable format, such data could be used in concert with the quantitative performance data and post-accession KSAO tests to forecast individual and group career trajectories and target interventions to prevent derailment, as well as incentivize top performers to remain in the Service.
The Air Force aims for an integrated approach to advancing the careers of its Airmen and managing its force, through its assessment of training, evaluation of performance, and promotion decisions. The committee believes that better use of the data these separate processes produce and further collection of additional data would strengthen integration and conceivably yield improved results, helping the Air Force meet the challenge it faces in implementing the National Defense Strategy through the optimal use of its human capital.
While there is a healthy focus on end-of-training assessment, the committee found that when Airmen are less than successful, systematic, continual evaluation of factors contributing to failure is lacking. A system that tracks performance from accessions to on-the-job outcomes is needed. Likewise, expanding current skill needs assessments across all AFSCs and ranks would help the Air Force adjust training seats and understand its skill imbalances.
Similar to the Army, the Air Force is implementing a “Talent Marketplace” that gives Airmen increased agency in decisions about their assignments, which should improve person-job fit and career satisfaction. But early results suggest the Air Force could benefit from a better understanding of how Airmen view this initiative, perhaps working jointly with the Army, which seems also to be encountering early implementation problems. At a minimum, any review should reconsider whether the deferred acceptance algorithm the Air Force Talent Marketplace currently employs is being deployed as effectively as possible, or whether some other preference-informed matching procedure might better fit Air Force needs. In any case, a better understanding of what information participants require to form their preferences reliably will help support the information exchange needed by any matching and assignment system that incorporates preferences.
Consistent with its “up or out” promotion system, the Air Force invests much time and energy in performance evaluation. But it does not benefit from all the data it collects, and misses data it should be able to assemble. It could exploit the ACA to augment or replace training and experience qualifiers for assignments and promotions. It could use the rich qualitative data
in the comments sections of appraisals to identify recurring themes and patterns. And it could improve the data generated through greater consistency in the design of performance appraisals, emphasizing numerical ratings, which should improve its ability to compare individuals across contexts.
Above all, the committee believes the Air Force needs to ensure promotion decisions fulfill its objectives. Although WAPS does what it was designed to do, it is not clear that it provides an optimal system for promoting those Airmen who will perform the best at the next higher grade. More senior enlisted and all officer promotions are entrusted to boards, a practice that can be effective. The committee found little evidence, however, that the Air Force is evaluating the processes its boards use to ensure they are in line with professional standards and best practices.
The research and analysis implied by the committee’s findings will require establishment of a human capital data superstructure. Only such a system can ensure that those making human capital decisions can reach the evidence-based standard to which the Air Force properly aspires.
AERA (American Educational Research Association), APA (American Psychological Association), & NCME (National Council on Measurement in Education). (2014). Standards for Educational and Psychological Testing. Washington, DC: Authors.
AFPC (Air Force Personnel Center). (2020). Military Demographics. Available: https://www.afpc.af.mil/Portals/70/documents/03_ABOUT/Military%20Demographics%20Jan%202020.pdf.
AFWIC (Air Force Warfighting Integration Capability). (2020). Global Futures Report: Alternative Futures of Geopolitical Competition in a Post COVID-19 World. Available: https://www.afwic.af.mil/Portals/72/Documents/AFWIC%20Global%20Futures%20Report_FINAL.pdf?ver=2020-06-18-124149-070.
Albon, C. (2020). Air Force eyes formal establishment of new synthetic training hub by summer 2021. Inside Defense. Available: https://insidedefense.com/daily-news/air-force-eyes-formal-establishment-new-synthetic-training-hub-summer-2021.
Buster, M.A., P.L. Roth, & P. Bobko. (2005). A process for content validation of education and experienced-based minimum qualifications: An approach resulting in federal court approval. Personnel Psychology, 58(3), 771–799.
Congressional Research Service. (2019). Defense Primer: Military Enlisted Personnel. Available: https://fas.org/sgp/crs/natsec/IF10684.pdf.
DoD (Department of Defense). (2015). Procedures for Military Personnel Assignments. Department of Defense Instruction 1315.18. Available: https://www.esd.whs.mil/Portals/54/Documents/DD/issuances/dodi/131518p.pdf.
Harrison, T. (2019). The Air Force of the Future: A Comparison of Alternative Future Structures. Washington, DC: Center for Strategic and International Studies. Available: https://www.rand.org/content/dam/rand/pubs/technical_reports/2006/RAND_TR322.pdf.
Kidder, K.L. (2019). Fostering Skills and Creating New Pipelines for Military Service: Provisions in the FY 2019 NDAA. RAND Corporation. Available: https://www.rand.org/pubs/testimonies/CT512.html.
Losey, S. (2019, December 9). Air Force to drop below-the-zone promotions for officers. Air Force Times. Available: https://www.airforcetimes.com/news/your-air-force/2019/12/09/air-force-to-drop-below-the-zone-promotions-for-officers.
Malia, A.M. (2020). Data Analysis to Support a Limited Objective Test/Experiment of a Billet Marketplace for Marine Officers. Monterey, CA: Naval Post Graduate School. Available: https://calhoun.nps.edu/handle/10945/65397.
Mattock, M.G., B.J. Asch, J. Hosek, & M. Boito. (2019). The Relative Cost-Effectiveness of Retaining Versus Accessing Air Force Pilots. RAND Corporation. Available: https://www.rand.org/pubs/research_reports/RR2415.html.
Murphy, K.R., & J.N. Cleveland. (1995). Understanding performance appraisal: Social, organizational, and goal-based perspectives. Thousand Oaks, CA: Sage Publications.
Reim, G. (2020). U.S. Air Force considers ways to recruit commercial pilots. Flight Global. Available: https://www.flightglobal.com/fixed-wing/us-air-force-considers-ways-to-recruitcommercial-pilots/138945.article.
Roth, A.E. (1984). The evolution of the labor market for medical interns and residents: A case study in game theory. Journal of Political Economy, 92(6), 991–1016.
Roth, A.E., & E. Peranson. (1999). The redesign of the matching market for American physicians: Some engineering aspects of economic design. American Economic Review, 89(4), 748–780.
Shlapak, D.A. (2006). Shaping the Future Air Force. RAND Corporation. Available: https://www.rand.org/content/dam/rand/pubs/technical_reports/2006/RAND_TR322.pdf.
SIOP (Society for Industrial and Organizational Psychology). (2018). 2018 Principles for the Validation and Use of Personnel Selection Procedures. Available: https://www.apa.org/ed/accreditation/about/policies/personnel-selection-procedures.pdf.
Switzer, T.B. (2011). A tale of two mechanisms: US Army Cadet branching. (Unpublished master’s thesis). Pontificia Universidad Católica de Chile, Santiago, Chile. Available: http://economia.uc.cl/wp-content/uploads/2015/07/tesis_tswitzer.pdf.
Thie, H.J., R.A. Brown, M. Berends, R.H. Ehrenberg, A. Flanagan, C.M. Levy, W.W. Taylor, R. Eisenman, W. Fedorochko, C.M. Graf II, M. Hoyer, P. Bracken, N.T. O’Meara, J.M. Sollinger, J. Larson, K. Mills, N.L. Rizor, B.D. Rostker, G.A. Gotz, C.R. Roll, Jr., & C.O. Halvorson. (1994). Future Career Management Systems U.S. for Military Officers. Santa Monica, CA: RAND Corporation.
USAF (U.S. Air Force). (1997). You and Your Promotions—The Air Force Officer Promotion Program. Air Force Pamphlet 36-2506. Available: http://dopma-ropma.rand.org/pdf/AFPAM36-2506.pdf.
USAF. (2004). Officer Promotions and Selective Continuation. Air Force Instruction 36-2501. Available: https://static.e-publishing.af.mil/production/1/af_a1/publication/afi36-2501/afi36-2501.pdf.
USAF. (2015). Air Force Future Operating Concept: A View of the Air Force in 2035. Available: https://www.af.mil/Portals/1/images/airpower/AFFOC.pdf.
USAF. (2018a). Military Promotion and Demotion. Air Force Policy Directive 36-25. Available: https://static.e-publishing.af.mil/production/1/af_a1/publication/afpd36-25/afpd36-25.pdf.
USAF. (2018b). Total Force Assignments. Air Force Instruction 36-2110. Available: https://static.e-publishing.af.mil/production/1/af_a1/publication/afi36-2110/afi36-2110.pdf.
USAF. (2019a). Air Force Training Program. Air Force Instruction 36-2651.
USAF. (2019b). Force Development. Air Force Doctrine, Annex 1-1. Curtis E. LeMay Center for Doctrine Development and Education. Available: https://www.doctrine.af.mil/Doctrine-Annexes/Annex-1-1-Force-Development.
USAF. (2019c). Officer and Enlisted Evaluations Systems. Air Force Instruction 36-2406. Available: https://static.e-publishing.af.mil/production/1/af_a1/publication/afi36-2406/afi36-2406.pdf.
USAF. (2019d). Utilization and Classification of Military Personnel. Air Force Policy Directive 36-21. Available: https://static.e-publishing.af.mil/production/1/af_a1/publication/afpd36-21/afpd36-21.pdf.
USAF. (2019e). Enlisted Airman Promotion/Demotion Programs. Air Force Guidance Memorandum to Air Force Instruction 36-2502. Available: https://static.e-publishing.af.mil/production/1/af_a1/publication/afi36-2502/afi36-2502.pdf.
USAF. (2020a). Total Force Development. Air Force Instruction 36-2670. Available: https://static.e-publishing.af.mil/production/1/af_a1/publication/afi36-2670/afi36-2670.pdf.
USAF. (2020b). Officer Developmental Education Guide, Version 10.
USAF. (2020c). Air Force Guidance Memorandum to AFI 36-2501, Officer Promotions and Selective Continuation. Available: https://static.e-publishing.af.mil/production/1/af_a1/publication/afi36-2501/afi36-2501.pdf.
U.S. Army. (2019). AIM: Officer Assignment Marketplace Update. Document provided to the committee by Maj. Kyle Greenberg, USA. Document available by request through the study’s public access file.