Click for next page ( 128


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 127
Determining Optimal Types of Incentives ver the years of the All-Volunteer Force, various incentives have been developed and offered to help strengthen and shape mili- ~_ tary enlistments. The effectiveness of these incentives has been addressed and demonstrated, using a variety of evaluation approaches. This chapter considers methodological issues in determining the optimal types of enlistment incentives for specific recruiting goals. We begin by reviewing different types of enlistment options. Next, we consider the various types of effects the incentives may produce, both intended and unintended, and the related methodological issues in assessing their im- pact. We continue with a brief review of the analysis and methods dis- cussed in the earlier chapters, which is followed by a longer consideration of analytical issues applicable to each of the evaluation approaches. Last, drawing on each of these areas, we conclude with a discussion of match- ing potential incentives and their effects to the appropriate assessment goals and evaluation methods. TYPES OF ENLISTMENT INCENTIVES Enlistment options may be grouped in various ways. For example, they can be distinguished in economic terms by whether they provide a cash incentive, cash in kind, or some other feature valued by potential enlisters, such as conditions of service. An alternative is to group incen- tives according to the segment of the recruiting market they are intended to attract, such as youth interested in attending college or those looking 127

OCR for page 127
128 EVALUATING MILITARY ADVERTISING AND RECRUITING for job training. In the discussion that follows we use a mix of these two approaches. We do so because that classification scheme lends itself directly to our discussion of the possible effects of these incentives, appropriate assessment goals, and evaluation methods. Youth also enlist for intrinsic reasons, such as patriotism. Matching recruiting messages and strategies to such motivations was discussed earlier. In developing or assessing the effectiveness of such strategies, considerations analogous to those discussed for the options considered in this chapter apply. More generally, enlistment options can be categorized based on social-psychological or economic theory according to their intended effects on the determinants of enlistment behavior, and their success in achieving these effects can be assessed as part of the evaluation process for the options. One group of incentives consists of cash enlistment bonuses. These bonuses are offered for various purposes: increasing enlistments; channel- ing recruits into hard-to-fill occupational specialties; encouraging recruits to report for active duty quickly; encouraging enlistments or accessions during the seasons of the year, such as late winter and early spring, that pose chronic challenges; encouraging enlistments among persons who served previously but left the military; and so forth. Military pay increases also are used to increase enlistments. Such increases are less targeted or flexible than bonuses and are intended to provide an overall enlistment stimulus. Compensation increases also can be provided through changes in other benefits, such as housing or sub- sistence allowances. Another group of options consists of incentives provided to encour- age youth interested in postsecondary education to see military service as a facilitator of their educational aspirations, rather than as an alternative career option. (As discussed in Chapter 2, this amounts to changing their beliefs about the consequences of enlisting compared with options in the civilian sector, and it can also change the analogous views of their key influencers.) Such education incentives include credit for college courses completed prior to accession (such as credit toward meeting military training requirements); near-term financial incentives, such as entry at an advanced pay grade, student loan repayment, or a bonus; funding for college after service, for example, through the Montgomery GI Bill or the Service's college fund; money for attending college before service, through options such as the Army's College First program (currently being pilot tested); or in-service continuing education programs, in which classes are provided on site or through distance learning. A fourth type of incentive is directed specifically at improving the recruit's job opportunities. Included in this group are such options as the

OCR for page 127
DETERMINING OPTIMAL TYPES OF INCENTIVES 129 Army's Partnership for Youth Success (PaYS) program and the GED Plus pilot program. Another group of incentives involves choices concerning the condi- tions of one's military service. Included in this group are such options as the term of enlistment, location of assignment, and combination of active and reserve forces duty, among others. Of course, the Services can and do use specific enlistment options in combination with others. One example is the combination of enlistment bonuses with postservice educational benefits. Another is the combina- tion of assistance for attending college classes with financial compensa- tion or training credit for having done so. POTENTIAL EFFECTS OF INCENTIVE To structure the analysis of the effects of a current or prospective enlistment option, it is important to first address several key issues. These issues include, but are not limited to, the types of questions highlighted below. Who is the target of the option? For example, is the option intended to increase overall enlistments or specifically those among highly qualified youth (high-aptitude high school graduates)? Similarly, are there key sub- groups of interest in the target population? If the incentive is intended to attract college-bound youth into the military (by encouraging them to see military service as a facilitator of their educational aspirations, rather than as an alternative career option), is it aimed at college graduates, current students, college stopouts (those who attended some college, have not yet graduated, but are not currently enrolled), high school graduates who have not yet attended college, high school seniors interested in college, or some combination of these subgroups? What are the desired effects of the option? These can include such outcomes as increased supply through increased enlistments or market expansion (or improved recruit performance through increased supply among high-aptitude youth), skill channeling into hard-to-fill or new spe- ~As is true for postsecondary educational incentives, the PaYS program is intended to change youths' beliefs about the consequences of enlisting compared with options in the civilian sector in this case by providing job training that can be applied to the civilian sector and helping the youth to be considered for employment in a related job after service- and it can also change the analogous views of their key influencers. Among its other goals, such as increasing training opportunities, GED Plus is designed to facilitate enlistment among high school dropouts with positive propensity while they achieve the qualifications needed to serve on active duty.

OCR for page 127
130 EVALUATING MILITARY ADVERTISING AND RECRUITING cialties, increased man-years, increased affiliation with the reserve forces following active duty, cost savings, and so forth, singly or in combination. What are possible unintended or undesirable effects of the option (for example, on such matters as the quality of enlistments, term of service distribution, attrition, or retention)? What are the potential timing and magnitudes of the desired and unintended effects? What are the option's potential costs? This includes such consider- ations as the direct cost for new recruits, overall cost including economic rent, (discounted) future costs, training costs, costs or savings to the reserve forces? How much do we know about the types and specific features of incentives that might generate effects analogous to those desired from the option? Does the option differ in important ways from past incentives for which we have enlistment data? We consider each of these issues in turn. Who is the target of the option? Enlistment incentives are typically targeted according to the qualifications of the recruit and the occupa- tional specialty in which he or she enlists. Qualifications often refer to whether or not the recruit has a traditional high school diploma and the level of his or her (written) aptitude score on the Armed Forces Qualifica- tion Test (AFQT). Youth with diplomas who score in the upper 50 percen- tile on the AFQT are considered to be highly qualified and are eligible for the largest incentives (subject to meeting the incentives' other require- ments). Alternatively, an option such as GED Plus may be targeted to the non-high school graduate market. Within the target population as a whole, there may be subgroups of special interest. This heightened interest may reflect a Service's desire to improve its recruiting performance for a particular youth subgroup. For example, in the case of GED Plus, one such subgroup consists of Hispanic youth. Alternatively, there may be heightened interest in specific sub- groups because the subgroups require different marketing or recruiting activities to effectively employ the option. In the case of incentives designed to appeal to college-bound youth, as noted, such subgroups may consist of high school seniors, high school graduates without college, college stopouts, youth currently enrolled in college, and college degree holders. Another relevant consideration concerns the option's potential appeal to the targeted youths' key influencers or referents (see Chapter 2~. What are the desired effects of the option? The incentive can be intended to affect a number of domains. These include, but are not limited to, increasing the enlistment rate among subgroups of the youth popula-

OCR for page 127
DETERMINING OPTIMAL TYPES OF INCENTIVES 131 lion who currently are enlisting; expanding enlistments to other sub- groups; channeling enlisters into specific military occupational specialties; increasing the number of years served by the typical enlistee; increasing affiliation with the reserve forces following active duty; or reducing recruiting costs. Defining the desired effects of the option is central to defining the outcome measures needed to assess its effectiveness. What are possible unintended or undesirable effects of the option (on such matters as the quality of enlistments, the term-of-service dis- tribution, attrition, or retention)? In addition to its desired effects, the option may have unintended consequences, which may be significant and thus require assessment. For example, an incentive designed to increase enlistments among high-aptitude GED holders should not do so at the expense of enlistments among high-aptitude high school graduates, which is the primary recruiting market. Alternatively, youth joining the military under an incentive designed to expand the recruiting market by offering shorter terms of service should not be those who would have enlisted for longer terms under existing programs. When incentives have potentially significant adverse outcomes, the unintended effects should be assessed in the evaluation together with those desired. Options may also have more subtle unintended consequences that nonetheless are very impor- tant to address in assessing the options' efficacy. For example, to the extent that the incentive increases youth-propensity to enlist for a particu- lar market or subgroup, recruiters may decrease the effort they expend for that market or for other subgroups (increase environmental con- straints). This is because recruiters may not need to leverage the full potential of the incentive in order to meet their recruiting goals and thus can target their efforts. This can lead to underestimating the option's actual potential to increase enlistments or expand the market, and it needs to be addressed specifically in the analysis. What are the potential timing and magnitudes of the desired and unintended effects? The timing of the option's effects has a direct bearing on the period of the evaluation. Options designed to increase accessions into the military in the near term may require a relatively short assess- ment period, whereas options designed to increase man-years or reserve forces affiliation may require that enlisters under the option be followed up through their first term of service (or even longer). Still other options designed to increase accessions may do so only over a longer period of time. Incentives provided to allow youth to attend college classes follow- ing enlistment but prior to entering active duty fall into this category and thus require a longer evaluation period to cover the increased time between the enlistment and accession points. A longer period between enlistment and accession would be expected to lead to higher loss rates among enlisters before their reporting for active duty. As a result, capturing such

OCR for page 127
132 EVALUATING MILITARY ADVERTISING AND RECRUITING loss rates becomes an important component of the analysis. Options that increase enlistments in the near term may also have longer term conse- quences. Postservice educational benefits might be expected to increase enlistments but also to increase separations if the recipients leave the military at a higher than average rate in order to attend college. The anticipated magnitude of the incentive's effects has a direct bear- ing on the statistical power of the analysis. Consequently, it affects the number of observations required to make a reasonable assessment of the viability of the option. This, in turn, can affect the number of persons to whom the option is offered as well as the required duration of the assessment. What are the option's potential costs (such as the direct cost for new recruits, overall cost including rent, discounted future costs, participa- tion rates, training costs, reserve forces costs or savings)? Enlistment options that have relatively small elasticities in increasing enlistments can nonetheless be very cost-effective recruiting tools depending on their cost. Conversely, the utility of incentives with high elasticities can be limited by their cost. To the extent that the costs of an enlistment incentive are of interest, they need to be captured in the evaluation. This can include the costs (and effects) of alternative values of the incentive or of variations in the number of specialties for which it is offered. The cost of providing the option to qualifying enlisters typically also includes costs in the form of "economic rent." This applies to the payment of the incentive to recruits who would have enlisted even in its absence (a cost we seek to minimize relative to the cost incurred to increase enlisted supply).2 A related cost concerns the impact the option may have on enlistments under other incentives and the resulting overall net cost increase or savings. Incentives that are paid out only after a considerable period of time, such as postservice educational benefits, have true costs well below their nominal value, unlike enlistment bonuses. Thus, their cost needs to be discounted in light of the delay in providing them, and the eventual usage rates for the incentive need to be considered. This can include both the proportion of eligible recipients who use any part of the incentive as well as the proportion of their entitlement they eventually use. Enlistment incentives can also increase or reduce other types of military costs. For example, an enlistment option combining active and reserve forces duty can reduce reserve forces recruiting costs by increasing affiliation rates 2The benefits considered in the rent calculation can include more general benefits to society provided by the incentive, in addition to increased enlisted supply. An incentive that increased college attendance or provided civilian-related job training could be expected to improve the employment opportunities and earnings of the recipients (and the taxes paid by them) over the longer term.

OCR for page 127
DETERMINING OPTIMAL TYPES OF INCENTIVES 133 from the active force into the reserve forces and training costs by encour- aging affiliation in the occupational specialty the soldier held while on active duty. Further, enlistment options can affect attrition and retention rates, which have implications for recruiting and training costs. How much is known about the types and specific features of incen- tives that may generate effects analogous to those desired from the option? Does the option differ in important ways from past incentives for which we have enlistment data? The extent to which a prospective option differs from current and past options has a direct bearing on the requirements for new data collection to assess its effectiveness. At one extreme, consider a new enlistment bonus whose value lies between the values of enlistment bonuses already offered. In such a case, it is very likely that existing data could be used to estimate the effects of offering the new incentive. Alternatively, consider the case in which a prospective enlistment option differs in important ways from those currently avail- able in its basic features or in the portion of the youth market to which it is targeted. An example would be a program with unique features, such as PaYS, or a program designed to attract youth who would not otherwise have joined the military, for example, by providing programs to help them attend college between the enlistment and accession points. EVALUATION METHODS AND ISSUES IN THEIR USE Four major types of evaluation approaches are commonly used in the development and assessment of recruiting incentives: Focus groups, Surveys, Econometric (regression) analyses of existing data, and Experimental or quasi-experimental assessments. Each of these approaches has advantages and limitations. Focus groups lend themselves especially well to exploring in depth the attrac- tive and unattractive features of a limited number of prospective or cur- rent incentives (or to helping develop new types of options), whereas surveys can be especially useful in more general assessments of the rela- tive attractiveness of a set of specific alternative options to the recruiting population as a whole or among particular market segments. Regression analyses are especially helpful in assessing enlistment rates under current (or past) options and in understanding the possible effects of trade-offs among the options or of limited adjustments to the options. When we need to estimate enlistment rates under prospective options that differ significantly in their features or market segment from current and past

OCR for page 127
134 EVALUATING MILITARY ADVERTISING AND RECRUITING options, then experimental or quasi-experimental designs can be particu- larly useful. We next consider the strengths and weaknesses of each approach in detail. We note here that each approach can have both unique advantages and limitations and this often leads to the use of multiple assessment methods in the evaluation of recruiting options. Focus Groups The use of focus groups is especially helpful for shaping prospective recruiting programs by enabling the researcher and policy maker to explore the appeal of a range of incentives or enlistment options to youth or their influencers, including programs currently available to recruits as well as prospective options. Moreover, the focus groups can provide insight into the particular strengths and weaknesses of such programs, including the usefulness and appeal of specific features of the options and the extent to which that appeal varies across key subgroups of the recruit- ing market. In so doing, the focus groups can facilitate the discovery and exploration of new option features or effects that may be helpful in achiev- ing military recruiting goals. A further advantage of this approach is that the information can be gathered relatively quickly and inexpensively. The use of focus groups is not without limitations. The groups must be carefully constructed to represent the population of interest and to facilitate open discussion. Groups can be too large to generate open dis- cussion, can be affected by social desirability concerns or other response biases, or can be dominated by a small number of especially vocal partici- pants (or by the group leader). Even if these potential obstacles are over- come, the groups still have only a limited ability to provide information that is generalizable to the full population of interest, because of the limited number of participants and the semistructured nature of the dis- cussions. This is particularly true when the purpose of the evaluation is to estimate effects of prospective alternative incentives or options such as the preference rates for one option over another or actual enlistment rates. Surveys Surveys also can provide rapid and reasonably inexpensive assess- ments of existing or prospective enlistment options among youth or their influencers. Because they are administered individually and, moreover, the questions can vary across individual respondents, they can be con- structed to cover more ground than what can normally be addressed using focus groups. This is particularly true for surveys administered using computers, such as computer-assisted telephone interviews or web-

OCR for page 127
DETERMINING OPTIMAL TYPES OF INCENTIVES 135 based surveys; in such surveys, an individual respondent's answers to specific questions as he or she proceeds through the survey can be used to determine which succeeding questions he or she is asked. For example, surveys are ideal to assess potential interest of youth in military service; can be designed to facilitate comparisons of the attractiveness and effects (among youth or their influencers) of a substantial number of specified, alternative recruiting options; can be designed to readily assess the varia- tion in propensity and in the effects of such incentives for key youth population subgroups of interest; and can be designed to explore the covariation of enlistment interest and the appeal of various incentives (or their specific features) with a wide variety of other factors, such as demo- graphic factors, educational aspirations, job and career goals, reasons for enlisting and barriers to military service, discussions with influencers, contacts with the military, and awareness of current military benefits. Questions dealing with such matters as media preferences, Internet use, and recreational habits can also be included to provide insights into the most effective ways to market enlistment incentives, both overall and to specific subgroups. As is true for focus groups, there are important issues and limitations in the use of surveys to assess the effects of alternative enlistment incen- tives on military recruiting. Drawing firm conclusions regarding the effi- cacy of alternative enlistment options requires that the sample must be representative of the target population, that the sample size must be ade- quate, and that a sufficient proportion of the surveys are completed and returned. Determining the appropriate sample size involves decisions about the acceptable confidence intervals surrounding the key point esti- mates and differences (in responses to specific items) to be provided by the survey; the statistical variances of the corresponding items; the frac- tion of respondents that will be asked to answer the key questions; the estimated completion rate for the survey; and the acceptable level for the statistical power of the analysis to detect true differences. Even when the sample is adequate and representative, care still must be taken in interpreting survey results concerning the prospective effects of alternative enlistment options. Although statistically significant rela- tionships between one's stated likelihood of enlisting in such surveys as the Youth Attitude Tracking Study (YATS) and Monitoring the Future (MTF) and one's actual enlistment decision have been demonstrated, it also is true that respondents' misestimate their true enlistment probability (Bachman, Segal, Freedman-Doan, and O'Malley, 1998; Orvis, Gahart, and Ludwig, 1992~. Youth seeing themselves as very likely to enlist are, in fact, less likely to actually join the military than they believe, whereas youth seeing themselves as very unlikely to enlist are more likely to do so than they believe. The accuracy of a respondent's stated enlistment inten-

OCR for page 127
136 EVALUATING MILITARY ADVERTISING AND RECRUITING lions varies with a number of factors, such as the temporal proximity of the enlistment decision point and the underlying rate of enlistment in the respondent's demographic group (normative support). When there is a considerable time period between the intention assessment and actual enlistment decision point, for example, changes in the respondent's life, in world events, in the economy, or in the information obtained through discussions with key influencers can alter the youth's enlistment inten- tions. This also limits the ability of surveys of the youth population to capture the possible longer term costs and effects of alternative enlist- ment policies, such as the incentive's effect on attrition and retention, even if the survey includes questions concerning such effects. Past research also suggests that youth overestimate their true respon- siveness to changes in enlistment options. While surveys provide an excellent methodology to assess the relative effects of alternative enlist- ment incentives, obtaining good estimates of the true enlistment rates under the options is likely to require additional data. Moreover, the spe- cific details of the information provided to respondents concerning cur- rent or prospective enlistment options are important and can influence their answers; this involves matters both of question wording and order, as well as social desirability concerns and other response biases. Finally, surveys of enlistment interest in the youth population (the supply side) do not capture the potential behavior of recruiters and recruiting man- agers as the recruiting market changes or new incentives are introduced (environmental facilitators or constraints the demand side); this again limits the ability of survey data to provide accurate estimates of the true enlistment rates that would be obtained under alternative options. Econometric Models As discussed in some detail in Chapter 5, econometric approaches rely on existing data and thus can provide rapid and inexpensive assess- ments and comparisons of policy alternatives. They also can provide statistical controls for confounding factors such as historical differences in enlistments across recruiting areas or the effects of changes in the economy and can allow assessment of incentive effects for subgroups of the youth population. Econometric models can assess the effects and costs of enlistment options over time and in different recruiting environments. They provide data on actual enlistment rates and can directly provide data on the magnitudes of unintended effects; such effects can include market substitution the substitution of enlisters brought in under a pro- gram for other types of enlisters or for enlistments under other options that would have occurred in the absence of the program and can be used to correct for changes in demand-side behaviors (environmental

OCR for page 127
DETERMINING OPTIMAL TYPES OF INCENTIVES 137 constraints or facilitators) that may mask the true potential of an option to increase enlistments or expand the recruiting market (Buddin, 1991; Polich, Dertouzos, and Press, 1986~. Although their reliance on existing data offers advantages of speed and cost in assessing the efficacy of alternative recruiting options, it limits the usefulness of econometric models in shaping and assessing prospec- tive options. Because the models are limited to assessment of previous or current recruiting incentives, the possible effects of new options that differ in important ways from such incentives in their features or the market segments to which they are intended to appeal cannot readily be assessed. Moreover, the accuracy of econometric models in forecasting enlistment rates can be limited by a variety of other factors that merit careful attention by the analyst. These include, but are not restricted to, the often limited amount of enlistment variation they may explain, espe- cially at the individual (versus cross-sectional) level or over time; the possible misspecification of the model (for example, due to the inclusion of an insufficient set of predictors or use of an inappropriate functional form); failure to capture important recruiter mission-setting or manage- ment practices that affect enlistments; failure to capture major shifts in propensity that may occur over time; and the possible correlation between supply- and demand-side variables in the model, such as changes in resources in response to changes in enlistment propensity. If recruiting resources are increased in response to a decline in youth's propensity to enlist and it then takes a while for the resources to improve the recruiting environment, one could underestimate the efficacy of the resources or, in the extreme, obtain incorrect signs for their regression coefficients. Simi- larly, if recruiters reduce their effort in response to the efficacy of an incentive in increasing enlistment propensity, the potential effects of the option on increasing enlistments or creating market expansion can be underestimated. Experimental and Quasi-Experimental Designs Use of experimental or quasi-experimental designs to evaluate the potential effects of enlistment options offers several advantages. The recruiting effects of prospective enlistment incentives can be quantified at low risk compared with full implementation (e.g., by avoiding unneces- sary costs or the difficulties of subsequently withdrawing an established, national incentive). Moreover, the effects of alternative prospective poli- cies or of alternative versions of a prospective policy can be directly assessed and compared. The effects of potential confounding factors in assessing the efficacy of recruiting programs which need to be statisti- cally controlled after the fact in econometric models can be directly and

OCR for page 127
138 EVALUATING MILITARY ADVERTISING AND RECRUITING greatly reduced through balancing of the test areas offering new pro- grams and the control areas not offering the programs on factors that are known to affect enlistment rates. These include, for example, geographical location, local economic conditions, demographic characteristics, past recruiting production, local recruiting missions, such recruiting resources as the number of recruiters in the area, and the alternative enlistment options that may be available locally, particularly to the extent that other Services or reserve components offer related programs (for example, alter- native financial assistance programs for college attendance). By employing an experimental design that compares (1) changes in enlistments in the test areas under the new option relative to enlistments in the same areas during a pretest baseline period with (2) the trend in enlistments in the control areas not offering the new program over the analogous time periods, the approach provides a direct control for tempo- ral and fixed differences in enlistment rates. Experimental and quasi- experimental designs can also directly provide data on the magnitudes of unintended effects; as noted, such effects can include market substitution or changes in recruiter effort or time allocation in response to the change in enlisted supply created by the new incentive (such as reduction of effort as supply increases). Given a test period of sufficient duration, the experiment's interim results, concerning both desired and unintended effects, can be used to recommend adoption of the test program or to provide the basis for changes in the incentive or in recruiter mission for the incentive that are designed to enhance its efficacy. The remainder of the test period can then be used to evaluate the performance of the modified program. Use of experimental or quasi-experimental designs to test the poten- tial effects of alternative options is not without potential problems. Cer- tain recruiting incentives, such as military pay increases, simply do not lend themselves to pilot testing; it is not practicable to limit them to cer- tain areas of the country or to undo them if their effect is not as desired. In cases in which legal authority is required to test prospective options, obtaining such authority may require considerable time, and the specifics of the enabling legislation can compromise the test's results. Similarly, the specific details of the incentive's test implementation such as the pro- portion of occupational specialties eligible for the incentive have a direct bearing on the estimates of the enlistment effects of the incentive that will be derived from the experiment. Such issues require careful attention in the analysis, particularly if comparisons with other incentives that differ in such characteristics are desired (see, for example, Buddin and Roan, 1994~. Additional resources for the test programs may also be needed, particularly if the experiment is to continue over an extended time period. A related issue concerns the . .

OCR for page 127
DETERMINING OPTIMAL TYPES OF INCENTIVES 139 feasibility of conducting a stable test in the operational environment, par- ticularly if recruiting conditions improve or worsen significantly during the test period or if the experiment suggests the usefulness of the pilot program and other recruiting areas (or Services) decide they want it right away. Moreover, there are certain inherent limits when experimental or quasi-experimental designs are used to assess the effects of prospective enlistment options. These include the fact that the program attributes that can be evaluated often are more aggregated or limited in number than is the case for some of the other evaluation methods; this can occur due to resource constraints, policy-based limits on enlistments under the pilot program, or statistical requirements to support the reliability of the test results, among other factors. Also, when new programs are tested in an experiment as compared with econometric analyses of alternative options using existing data, for example their overall cost-effectiveness may be unknown for years; this is because such a determination requires assessment not only of enlistments which can be measured in the rela- tive near term experimentally but also of patterns of attrition, retention, and job performance, which take much longer to unfold. Finally, no matter how well balanced the test and control cells may be at the outset of an experimental assessment, the test does not control for differing inter- temporal changes across the cells, such as those that could occur due to changes in local economic conditions or changes in demand-related factors, such as recruiting behavior or resourcing. Combinations of Evaluation Methods Sometimes a combination of evaluation methods is used to assess the efficacy of alternative enlistment options. One approach that has been used in the past is to combine econometric analysis with the experimental approach in order to help preserve the balance of the test and control cells by adjusting statistically for unexpected, differing intertemporal changes across the cells. The other methodologies also can be used in combination. For exam- ple, focus groups may be used to help explore and define prospective enlistment options that are then tested in large youth market surveys or in pilot tests employing experimental designs (or, conversely, focus groups or surveys are used to explore the effects of policy experiments). Another example is the application of econometric methods or other analytical techniques to existing datasets in order to help interpret survey results. The other approaches could be used to derive a metric to help translate stated enlistment intentions in the survey into estimated enlistment rates.

OCR for page 127
140 EVALUATING MILITARY ADVERTISING AND RECRUITING ASSESSMENT OF INCENTIVE EFFECTS In the final section of this chapter, we provide examples of designing appropriate research to help determine the optimum typed of enlistment incentives for a given policy goal by matching the desired effects of the potential incentives, the corresponding assessment goals, and evalua- tion methods. Consider the case in which the Office of the Secretary of Defense (OSD) or an individual Service would like to expand its recruiting efforts into a new or largely untapped market and is interested in developing enlistment incentives to do so. To the extent that the market is new or exploratory research on new options is desired, econometric modeling using existing enlistment data will not be especially helpful in choosing optimal incentives for recruiting this market. Experimental or quasi- experimental approaches would limit the number of options that could be tested, due to analytical, cost, and legislative considerations, among other factors, and thus are premature. The policy maker also lacks sufficient information at this point to design a survey to gather detailed information on the areas of interest. Focus groups, however, are well suited for this purpose. The Army's interest in materially improving its recruiting perfor- mance among youth who are committed to attending college after high school but would consider enlistment subsequently provides an example. Over the past 20 years, the proportion of youth attending college within a year of high school graduation has risen dramatically, from about half to approximately two-thirds (U.S. Bureau of the Census, 2000, 2002~. How- ever, the military has lagged in adjusting its recruiting strategy and still recruits the vast majority of its enlisted force from youth who have not attended college. In recognition of this mismatch, over the past several years the Army has employed policies that give greater emphasis to recruiting youth from the college market (which includes students, stopouts, and high school seniors). As part of that effort, it has conducted focus groups with youth in the college market to better understand their reasons for or against enlisting and the types of enlistment options they would find appealing. Similarly, OSD recently sponsored a study of the college-bound youth population concerning their enlistment interest and the types of incentives that would stimulate that interest (Kilburn and Asch, 2003~. The initial step in the research consisted of a series of focus groups with each of the subgroups listed above; this helped OSD define and shape incentives that appeared to be promising for further investigation. Next, consider the case in which a policy maker wishes to understand in some detail the relative appeal of an array of alternative recruiting options and wants to be able to generalize the information to the youth

OCR for page 127
DETERMINING OPTIMAL TYPES OF INCENTIVES 141 population. The options may include current incentives, prospective enlistment incentives, or both. The details of interest, for example, may include the usefulness of the incentives for key youth subgroups, such as high aptitude youth and college stopouts; those potentially interested in military service for various reasons, such as job training, assistance for attending school or taking a break from work or school; or those with specific barriers to military service. If the policy maker wants to compare current and prospective incentives, it may also be helpful to assess aware- ness levels in the subgroups of interest for the significant current options. To the extent that research on new options is desired or the youth population characteristics of interest are not captured in the military's enlistment databases for example, starting and then stopping out of col- lege before completing a year, recruits' reasons for enlisting, the level of support for enlisting among their influencers, or youths' awareness levels of specific options existing enlistment data will not be especially helpful in choosing optimal incentives. Experimental or quasi-experimental approaches would again limit the number of options that could be tested, and they might not identify key subgroups of interest without additional data collection procedures. Given the policy maker's interest in system- atic information that can be generalized to the youth population, focus groups lack the required coverage and rigor. In such instances, youth surveys can be used very effectively to address the issues of concern. In the OSD study just discussed, the incentives developed using focus groups were included in a subsequent survey of college-bound youth that pro- vided a variety of detailed, generalizable enlistment-related information, including the incentives' potential effects on enlisted supply. Similar con- siderations apply to the use of influencer focus groups and surveys. As discussed earlier in this chapter, surveys must sample and obtain data from a sufficient number of respondents representative of the popu- lation and subgroups of interest. Care also must be taken in the wording and sequencing of the questions included to represent the enlistment options. The results will be best suited to understanding the relative effects of the alternative incentives. Using propensity measures for which data exist to link responses to enlistments will help to meaningfully interpret the responses to the options in the survey by providing an enlistment index value for each option and subgroup. Even so, past research has shown that respondents misestimate their true enlistment probability to some extent, and there also is the possibility of changes in demand-side factors, such as recruiter behavior in response to new markets or options. Thus, the index values do not equate to the actual enlistment rates that will be obtained under the options. There are various analytical approaches that can be used to construct the samples, survey questions, and enlist- ment indexes (see, for example, the Youth Attitude Tracking Study, 1976-

OCR for page 127
142 EVALUATING MILITARY ADVERTISING AND RECRUITING 1999 Wilson et al., 2000; Kilburn and Asch, 2003; Orvis and McDonald, forthcoming).3 A policy maker's interest in determining the optimal types of enlist- ment incentives may involve wanting to optimize the cost-effectiveness of the mix of current options or to get the greatest return on investment for additional recruiting resources that can be made available in the near term. By analogy, the issue could involve wanting to reduce a particular type of resource for example, recruiters and wanting to know how to best offset that reduction with alternative resources. Answering these types of questions quickly and with confidence requires estimates of actual enlistment rates, costs, and elasticities, based on known past results. Econometric modeling is ideal in such cases (see, for example, Warner, Simon, and Payne, 2001; Buddin and Roan, 1994; Asch and Orvis, 1994~. Using econometric modeling to address such questions also has the advantage of enabling the policy maker to examine in advance the prob- able longer term consequences of making such allocations and trade-offs on such factors as attrition and retention. 3The purpose of an enlistment index is to allow the prospective recruiting effects of alter- native options to be compared more meaningfully than would be possible simply by com- paring the distributions across an intention measure generated by the options. There are various approaches to constructing such enlistment indexes. They all include linking enlist- ment records to stated enlistment intention levels normally, from earlier surveys in order to establish enlistment rates for the intention levels. For example, this can be done for the individual levels (response categories) of the intention measure (e.g., definitely, probably, probably not, definitely not or 0 to 10 likelihood of enlisting); for combinations of the categories (e.g., positive versus negative propensity); or for composite measures that com- bine more than one intention measure (e.g., "definitely" on any of several individual Service measures becomes "definitely" on the composite measure, or an unaided mention of plans to enlist forms the top category, followed by positive propensity without an unaided men- tion, and so forth). Various follow-up periods can be used to determine the enlistment rates such as one year, lifetime, etc. depending on one's purpose (and the ages or school year of the respondents). It is important to match the intention measures to be used in the new survey, the sample composition (e.g., market segments, rules on inclusion of youth in the delayed entry pool), and the enlistment period of interest to those used in the earlier follow-up work to generate the index. The enlistment index value can simply be the enlist- ment rate found in the enlistment records check for respondents stating the given intention level, or it can be transformed or made conditional on other factors (e.g., through inclusion in a regression analysis of enlistments) to help control for the effects of other factors on a person's enlistment decision, which may not be captured fully by the intention measure. It may also be helpful to use the index values to compute a measure of the options' appeal relative to each other, by scaling the index values relative to the most (or least) appealing option (i.e., that which most (or least) increases stated propensity to enlist in the survey). Finally, the index values may be calculated independently or relative to that for the respondent's (baseline) likelihood of enlisting under current options.

OCR for page 127
DETERMINING OPTIMAL TYPES OF INCENTIVES 143 The data collected from focus groups is not sufficiently generalizable or precise to address these types of questions. Although surveys can be useful in comparing the relative appeal of alternative options, they do not provide absolute enlistment (or loss) rates, and they do not directly cap- ture the effects of changes in such factors as the number of recruiters or advertising levels. Finally, given the focus of these questions on trade-offs among existing incentives and resources, using an experimental approach would not provide results in a timely manner and would be unnecessarily restrictive. When the types of incentives to be evaluated are reasonably well defined but they are new in total or vary significantly in their features from current options, assessing their effects requires new data collection. Collecting that information with reasonable precision beyond that afforded by surveys or focus groups but with lower risk than through national implementation can be accomplished using experimental or quasi-experimental pilot tests of the programs. The general features of such tests, as well as their strengths and limitations, were described ear- lier. There are numerous examples of national recruiting tests, and they cover a variety of effects of potential interest to policy makers. For example, the Educational Assistance Test Program compared recruiting effects for the Army, the Navy, and the Air Force of different types and amounts of postservice educational benefits (Fernandez, 1982~. Issues of concern included the overall effect on highly qualified enlist- ments of the incentives; the effects of the type and value of the incentive; and the effect of differences in the value of the incentive across the Ser- vices. The Enlistment Bonus Experiment compared the results of bonuses differing in dollar value and term of enlistment obligations on highly qualified enlistments and the channeling of recruits into hard-to-fill occu- pational specialties (Polich et al., 1986~. The 2+2+4 Recruiting Experiment addressed the effects on active and reserve forces component enlistments among highly qualified youth of a program allowing two years of active duty plus training time followed by two years in the Selective (drilling) reserve forces and four years in the Individual Ready Reserve in return for enhanced postservice educational benefits (Buddin, 1991~. The key effects of interest included active enlistments, changes in job choice and term-of-service obligation, affiliation rates into the Selective Reserve, affili- ation rates into the same occupation held while on active duty, and the overall impact on military man-years. The 2+2+4 test used a true experi- mental design that included random assignment of qualifying youth to eligibility for the program. Another example of employing a quasi-experimental design is pro- vided by the College First/GED Plus National Recruiting Test. These pilot programs are currently in progress (Sellman, 1999; Orvis and

OCR for page 127
144 EVALUATING MILITARY ADVERTISING AND RECRUITING McDonald, forthcoming). The College First program's main purpose is to assess the effect on highly qualified enlistments and market expansion of an incentive allowing recruits to attend college for up to two years after enlisting but before reporting for active duty. The analysis examines such effects both overall and for key subgroups of the college-bound youth market. Based on these analyses, it identifies useful changes in the pro- gram and in recruiter missioning to support it. GED Plus was designed to increase opportunities for and enlistments among qualifying youth with- out traditional high school diplomas. To qualify, youth have to score in the upper half of the AFQT distribution and pass strict screening criteria. Youth without GED certificates have to obtain them through an attendance- based course prior to going onto active duty. Key outcome measures include the program's effects on high-aptitude enlistments among youth without traditional high school diplomas and, in particular, among His- panic youth, who are disproportionately represented in this market; the impact, if any, on highly qualified enlistments (market substitution); first- term attrition rates under the program; and the cost-effectiveness of eligi- bility for alternative enlistment incentive levels among GED Plus recruits, including the extent to which each level is being leveraged by recruiters. Helping policy makers choose the optimal incentives to accomplish their objectives and obtaining reasonable estimates of the incentives' prob- able effects often involves the use of multiple evaluation methodologies. We have noted such combinations in the preceding discussion. For exam- ple, focus groups can provide a highly useful initial step in designing new enlistment options. The main results of the groups can then be used for follow-up assessments that provide more rigorous and generalizable results through surveys or recruiting experiments. We also noted that quantitative assessments linking prior survey results to actual enlistment data can be very helpful in interpreting enlistment propensity informa- tion collected in new surveys and in quantifying the likely enlistment outcomes associated with alternative incentives. As discussed earlier in this volume, econometric modeling can be used in conjunction with experimental or quasi-experimental designs to provide improved estimates of the potential recruiting benefits of new options. The Enlistment Bonus Experiment, for example, used econometric analysis to support a quasi-experimental design. The regressions con- trolled for changes over time in economic and other conditions that could have acted to unbalance the test's experimental and control cells. Based on Dertouzos' approach, they also attempted to control for unobserved reductions in recruiters' effort in response to the improvement in recruit- ing conditions under the test options and the ease of substitution of less- well-qualified versus highly qualified recruits.

OCR for page 127
DETERMINING OPTIMAL TYPES OF INCENTIVES 145 For illustrative purposes, we conclude with an example of using the four evaluation methodologies in combination. The Army is currently assessing the benefits of eArmyU, an online distance learning program allowing soldiers to pursue courses and degrees from more than 20 col- leges and universities and approximately 100 programs, and of soldiers' interest in modified versions of the program. The current program pro- vides a free laptop computer to participants, up to 100 percent tuition assistance, and a number of other benefits, such as an Internet service provider and free books and delivery. In return, the soldier must agree to serve for at least three years from the eArmyU enrollment point and must complete 12 semester hours within two years of enrolling. The evaluation is assessing participation rates in the current program across subgroups (using econometric modeling); retention effects of the current program (econometric modeling and focus groups with partici- pants in the program); and likely enrollment rates and retention effects under alternative versions of the program some with reduced incen- tives matched by reduced service and course completion requirements, in order to allow more soldiers to participate within the resources available for the program (quasi-experimental pilot test, survey of enlisted personnel, and focus groups with participants, supervisors, and education adminis- trators of the alternative versions of the program). The evaluation also is assessing quality of life benefits of the current program to the soldier and his or her family (focus groups) as well as the effects of the current pro- gram on readiness and duty performance (focus groups).