Click for next page ( 160


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 159
9 Conclusions and Recommendations he primary objective of this study is to help the Department of Defense (DoD) improve its research on advertising and recruiting polices. We anticipate that in the coming decade DoD will field and test new advertising and recruiting initiatives designed to improve the recruiting outlook and avoid the shortfalls of the last decade. In order to discover the most promising policies, in the committee's view DoD needs a comprehensive research and evaluation strategy based on sound research principles that will ensure valid, reliable, and relevant results. In this report, we present an evaluation framework that links different types of research questions to different research methodologies. The framework identifies four major categories of research questions and four broad methodological approaches. The first category of research question asks: "What does a target audience see as attractive or unattractive features of a program?" It is well suited to examination via qualitative methods, such as focus groups; unstructured or open-ended surveys; and interviews. The second category of research question asks: "What is the effect of a program on specified attitudes or behavioral intentions?" It is well suited to examination via surveys, experiments, and quasi-experiments. The third category of research question asks "What is the effect of a proposed new program on enlistment?" It is well suited to examination via experiments and quasi-experiments. The final category of research question asks "What is the effect of an existing program on enlistment?" It is well suited to examination via econometric modeling. The committee's work during Phase I led us to conclude that there are a number of critical problem areas or topics needing more intensive study. 159

OCR for page 159
160 EVALUATING MILITARY ADVERTISING AND RECRUITING Some problems arise because of the need for ongoing, up-to-date infor- mation that can serve as early-warning indications of potential recruiting problems or that can point to areas in which improvements are needed. Other problem areas are important because, in our view, they are central to improving the overall recruiting climate. We selected six areas as the central focus of this report. After devoting Chapter 2 to issues of theory as a guide to effective evaluation research, Chapters 3 through 8 each exam- ine one of the six areas. The various chapters also introduce different methodological approaches to evaluation. CHAPTER 2: THEORETICAL APPROACHES The chapter outlines a general framework for thinking about effective program design. The first step is to identify the fundamental factors that impact a target population's enlistment behavior. The second step is to derive strategies (often informationally based) to change, enhance the effect of, or mitigate the effect of those determinants. We outlined a wide range of variables and processes that program designers must potentially take into account, drawing heavily on research from adolescent develop- ment, communications, economics, psychology, and sociology. These per- spectives set the stage for conducting the necessary research to inform program design and program evaluation. There are two distinct theoretical approaches to enlistment behavior: decision theory, based primarily in psychology, and the econometric theory of enlistment supply. While there is some overlap in these two theoretical traditions, they have distinct approaches. Decision theory is more highly developed for the purpose of conceptualizing and measur- ing behaviors that affect individual decision making. Econometric theory is more formally developed with respect to aggregate enlistment out- comes and various exogenous influences. Accordingly, a key objective of this chapter is to build an integrated perspective on the behavioral and econometric approaches. Conclusion: The role of theory is crucial to the design of interventions to increase enlistment behavior. When enlistment programs are developed Theoretically, they run a great risk of being ineffective.

OCR for page 159
CONCLUSIONS AND RECOMMENDATIONS CHAPTER 3: MONITORING TRENDS IN YOUTH ATTITUDES, VALUES, AND PROPENSITY Determinants of Propensity 161 The chapter proposes survey methods as the most suitable research design for tracking changes in propensity as well as for assessing the underlying beliefs that are related to propensity. The information gathered in youth attitude surveys is valuable to many research studies, such as the advertising studies proposed in Chapter 4, but it is also valuable in its own right by providing early warning indicators of changes in the pro- pensity for military service. The chapter reiterates a key point from the committee's earlier report, namely, that as propensity to enlist is the major direct determinant of actual enlistment, increasing propensity in the youth population should be a key objective for the military. We summarize a model of the determinants of propensity that we laid out in the earlier report and build an argument that research on propensity in the youth population should measure the key determinants of propensity. We briefly review ongoing survey efforts dealing with propensity, noting that they do not consistently include these key determinants of propensity. We also note a tendency for survey research dealing with propensity to make use of research designs that do not yield complete data on individuals. If the interest is simply documenting the proportion of respondents choosing each alternative to a survey item, then randomly distributing items among respondents will yield accurate results, as individual-level analysis is not central to the research question. But when there is interest in the pattern of relationships among variables, designs involving complete data at the individual level are needed. Conclusion: Previous survey research examining propensity to enlist has not consistently measured the key classes of determinants of propensity (i.e., attitudes, norms, and self-efficacy), nor has it consistently used research designs permitting analysis at the individual level. This has lim- ited the ability to test complete models of the determinants of propensity. Recommendation: We recommend that survey research examining propensity be designed to incorporate the key determinants of pro- pensity and that it be designed to permit meaningful analysis at the individual level.

OCR for page 159
162 EVALUATING MILITARY ADVERTISING AND RECRUITING Conclusion: We also note that our model of the determinants of propen- sity includes the role of important influencers, such as parents and peers. One important implication of this is that the effects of interventions, such as advertising, are not fully addressed by focusing solely on the direct effects of the intervention on the potential recruit. Recommendation: We recommend that evaluation efforts consider potential effects on key influencers as well as on potential recruits, and that efforts be made to assess such indirect effects on propensity. A Program of Survey Research The second part of the chapter provides a series of concrete recom- mendations for a program of survey research, expanding on ideas set forth in our letter report to the Department of Defense of tune 2000. A program of monitoring surveys, which have the potential to yield very high-quality data about propensity and its determinants, is presented. We propose a cohort-sequential design, in which samples of youth are obtained annually in the 11th grade of school (i.e., age 16-17) and moni- tored through the age of roughly 23. We discuss a range of issues in the design of such a project, including information-gathering format (e.g., self-completed questionnaire versus telephone interview), means of accessing the sample (e.g., school-based surveys versus random house- hold sampling), mechanisms for follow-up surveys over time, issues in the scheduling of surveys, and sampling strategies. We note that such a project is a significant investment and should not be undertaken unless the resources for a minimum of five years can be committed. We note that a variety of options are available with the broad framework we develop. For example, one possibility is to survey 11th graders, with annual follow- up; another is to survey both 11th and 12th graders, with follow-up every two years. The chapter develops the trade-off among the options, noting that many details cannot be specified in advance. Recommendation: We recommend that consideration be given to undertaking a school-based survey, using a cohort sequential design, in which students are sampled in the 11th grade and possibly the 12th grade and regularly resurveyed until the age of 23 or 24.

OCR for page 159
CONCLUSIONS AND RECOMMENDATIONS Item Content 163 The final section of the chapter discusses item formats for assessing the important variables in the model of the determinants of propensity developed in our earlier report and reiterated here. Specific examples of item formats for effectively measuring propensity, attitude, norms, self- efficacy, behavioral beliefs, outcomes, and open-ended queries about out- comes and about influencers are provided. Recommendation: We recommend that surveys dealing with pro- pensity and its determinants assess the variables of interest using established item formats. CHAPTER 4: ADVERTISING PLANNING: GENERATIVE AND EVALUATIVE APPROACHES In order to develop and test effective advertising themes, two types of research designs are required. The first type of design is for the develop- ment of preliminary but promising message strategies. This step requires qualitative exploratory or generative research designs (focus groups, in- depth interviewing, etc.~. After promising themes are developed, the second type of research design is necessary for testing theme awareness and market impact. The best designs for this step are experimental and quasi-experimental studies. This chapter describes the stages in the development and evaluation of an advertising campaign, discussing relevant research methods for each phase. The chapter reviews some findings from the committee's earlier report, including the decline in both the proportion of youth assign- ing high value to duty to country and in the proportion of youth who associate the goal of duty to country with military service. We then develop a framework for developing advertising campaigns that follows a systematic process and builds on sound information about the value structure of youth. That framework involves (1) tracking the competitive environment for military recruitment to detect factors affecting youth understanding and views of military service; (2) examination of audience member beliefs, goals, and language; (3) development of a range of mes- sage strategies for military recruitment, and (4) allocation of resources to advertising message strategies. Conclusion: There is a need for research to provide a more complete picture of the belief and value structure of the youth population, particu-

OCR for page 159
164 EVALUATING MILITARY ADVERTISING AND RECRUITING larly the beliefs relating to public service, duty to country, personal sacri- fice, and concern for others. It is also helpful to study the language used by youth as they think and speak about these issues, so that this informa- tion can be used to develop effective messages on the topic. Recommendation: We recommend a program of research that begins with generative techniques to understand the concepts and language used by youth in considering alternative courses of action (e.g., edu- cation versus military service) and continues with survey research that measures the full range of beliefs, attitudes, and values that emerge as linked to these alternate courses of action. Conclusion: Effective advertising campaigns involve a message strategy strongly linked to beliefs and values that affect decision making. A crucial component of the evaluation of military advertising is an examination of its success in affecting the intended values and beliefs. Using beliefs and values as outcome variables as well as enlistments permits a clearer under- standing of why a given advertising campaign is or is not successful than using enlistments alone. Recommendation: We recommend that advertising message strate- gies be evaluated in terms of their effects on targeted beliefs and values. Such evaluation should make use of experimental designs in controlled settings as well as small-scale, in-market experiments. Recommendation: We recommend that a policy be adopted of regu- larly developing and evaluating alternative approaches that ch~llens, existing message strategies. CHAPTER 5: DETERMINING OPTIMAL LEVELS OF ADVERTISING AND RECRUITING RESOURCES . ~ _ The chapter focuses on econometric methods, as these approaches are most useful for assessing the optimal levels of recruiting programs and resources. Econometric methods can be used to isolate and identify the effects of existing resources, policies, and external factors affecting recruit- ing outcomes as well as their costs. There is by now a relatively well-

OCR for page 159
CONCLUSIONS AND RECOMMENDATIONS 165 developed body of econometric research that has identified some of the most important determinants of enlistment supply as well as the cost and effectiveness of various trade-offs among different recruiting and adver- tising resources. Estimates are based on the natural variation in key recruiting resources and outcomes (usually aggregated) that occur over time and across different geographic locations. Brief overviews of two types of econometric models are provided- models of enlistment supply and models of recruiting cost. We review studies over the past two decades on the effects of recruiting and adver- tising on enlistment and present summary tables comparing the various studies. We note considerable variability in results across studies and suggest a series of methodological features that have not been consis- tently incorporated into the studies and thus may contribute to the variability in results and the difficulty in giving a definitive answer to questions about the elasticity of enlistment with respect to advertising. Conclusion: More sophisticated methods, controlling appropriately for factors affecting enlistment supply, both those that are directly observ- able to the researcher and those that must be inferred, such as recruiter effort, are necessary to obtain efficient, unbiased estimates of the effects of recruiting resources. Moreover, more complete evaluation of the effects of some types of resources, especially advertising content, require estima- tion using more flexible functional forms in the econometric analysis. To apply these methods, however, better data need to be collected and sys- tematically maintained. The specific conclusions and recommendations discussed below are conditional upon the availability of better data. Recommendation: Collect and maintain better data to support the estimation of enlistment supply functions and to evaluate the effec- tiveness of recruiting resources. Conclusion: Recruiter productivity varies with experience, and hence sudden changes in the size of the recruiting force result in declines in average experience. Failure to incorporate recruiter experience in models of recruiter effects may bias study results. Recommendation: We recommend that future research on the effects of recruiters on enlistment supply incorporate the effects of recruiter experience.

OCR for page 159
166 EVALUATING MILITARY ADVERTISING AND RECRUITING Conclusion: Recruiter incentives have been incorporated in supply models via recruiters' quotas, based on the assumption that increasing recruiting quotas increases effort. Recommendation: We recommend that supply models incorporate more complete and realistic models of recruiter incentives that more fully capture the complexities of recruiter incentives. Conclusion: Research to date has not incorporated the effects of reserve forces competition on active-duty recruiting. Recommendation: We recommend that supply models incorporate reserve forces competition for nonprior-service recruits. Conclusion: Econometric estimates of the effects of advertising have focused largely on expenditures of impressions (i.e., the number of relevant individuals who see or hear the advertisement, often measured in terms of gross rating points). Such estimates have not attempted to measure differences in effects by specific advertising content. Recommendation: We recommend that research attempt to evaluate advertising in terms of thematic content in order to determine whether advertising effects vary by content, as well as by impressions and expenditures. Conclusion: The functional forms (i.e., the shape of the relationship between the recruiting incentive and enlistment) of econometric supply models have been relatively restricted. The underlying assumptions (e.g., that each additional advertising dollar has the same effect regardless of the level of total expenditure) may not be correct, and an examination of more flexible functional forms would be fruitful. Recommendation: We recommend that research on supply models make use of flexible functional forms, rather than imposed functional forms.

OCR for page 159
CONCLUSIONS AND RECOMMENDATIONS CHAPTER 6: THE TIMING AND LEVELS OF JOINT AND SERVICE-SPECIFIC ADVERTISING Minimum Advertising Level to Maintain Propensity 167 The chapter first addressed the question of whether there is a mini- mum level of advertising necessary for a cost-effective recruiting program, even if that advertising is not necessary to achieve contemporaneous enlistment contract goals. Historically, when the recruiting climate is good and recruits are plentiful, military planners tend to cut advertising bud- gets, thereby contributing to a reduction in awareness capital and pro- pensity levels. This may possibly set up a boom or bust cycle, in which propensity falls, recruiting becomes more difficult, and then advertising funds have to be restored. We present a model that describes the condi- tions under which it would be cost-effective to advertise in the interests of future enlistment supply and review research to date that speaks to the issue. While extant research suggests that advertising may have effects for only a short period of time, the data available to prior researchers are limited for several reasons. First, they do not permit examining both lagged effects and nonlinear effects within a time period. Second, they focus on advertising aimed at youth at the point of the enlistment deci- sion and do not permit examining possible supplemental advertising approaches, such as those aimed a youth several years prior to an enlist- ment decision, or those aimed at adult influencers, such as parents. Conclusion: Research to date does not permit a definitive answer to the question of the cost-effectiveness of advertising above and beyond that which is necessary to achieve current recruiting goals. Recommendation: We recommend a focused effort to maintain advertising data in a systematic way for purposes of estimating a supply curve that incorporates the potential for both time-lagged and nonlinear advertising effects. Recommendation: We recommend a program of research, incorpo- rating quasi-experimental methods, to examine advertising effects over an extended period of time.

OCR for page 159
168 EVALUATING MILITARY ADVERTISING AND RECRUITING Levels of Joint and Service-Specific Advertising The chapter then turns to the optimal levels of joint and Service- specific advertising. It is our opinion that certain types of advertising themes, such as generic themes designed to increase overall propensity, are best done as a joint program, while advertising themes featuring spe- cific benefits of military service are best done in the Service program. What we do not know is what advertising fund level should be allocated to joint programs. We note that issues of scale play a role in addressing this issue, as certain types of advertising (e.g., television) do not appear to have a constant effect across levels of expenditure. The larger the recruit- ing effort and the larger the budget, the greater the potential value of a multifaceted campaign, with some resources targeted toward providing information about specific Services to those already with a propensity to enlist and others targeted toward increasing propensity among those cur- rently without it. Recommendation: We recommend a program of research aimed at examining the effects and cost-effectiveness of information-oriented versus values-oriented advertising in joint and Service-specific adver- tising programs. CHAPTER 7: DETERMINING OPTIMAL TYPES OF INCENTIVES Over the years of the All-Volunteer Force, various incentives have been developed and offered to help strengthen and shape military enlist- ments. The effectiveness of these incentives has been addressed, and dem- onstrated, using a variety of evaluation approaches. This chapter considers methodological issues in determining the optimal types of enlistment incentives for specific recruiting goals. We begin by reviewing different types of enlistment options. Next, we consider the various types of effects the incentives may produce, both intended and unintended, and the related methodological issues in assessing their impact. We continue with a brief review of the analysis methods discussed in the earlier chapters, which is followed by a longer consideration of analytical issues applicable to each of the evaluation approaches. Finally, drawing on each of these areas, we conclude with a discussion of matching potential incentives and their effects with the appropriate assessment goals and evaluation methods. A central message of the chapter is that each of the evaluation meth- odologies introduced in this volume (qualitative methods, surveys, econo-

OCR for page 159
CONCLUSIONS AND RECOMMENDATIONS 169 metric models, and experiments and quasi-experiments) can play a useful role in addressing different questions that policy makers may ask about current or proposed incentives. The chapter offers concrete illustrations of the research framework introduced in Chapter 1. It identifies focus groups as of particular value in providing insight into the appeal of vari- ous features of proposed incentives and in facilitating the discovery and exploration of new incentive options. It identifies surveys as of particular value in comparing the relative attractiveness of a substantial number of incentive options. It identifies econometric methods as of particular value when examining the effects of existing programs on actual enlistments over time and over differing recruiting environments, providing statisti- cal control for a wide variety of potentially confounding factors (e.g., geographic effects, effects of changes in the economy). It identifies experi- mental and quasi-experimental methods as of particular value when the question of interest is estimating the effects on enlistment of a new incen- tive prior to full implementation. In addition, the chapter emphasizes the value of combining approaches. This might include, for example, the use of focus groups to help explore and define prospective enlistment options that are then tested in large youth market surveys or in pilot tests employing experimental designs. Another example would be the application of econometric methods or other analytical techniques to existing datasets in order to help interpret survey results; for example, the other approaches could be used to derive a metric that can be used to help translate stated enlistment intentions in the survey into estimated enlistment rates. The focus of the chapter is not on specific conclusions and recommen- dations, but rather on illustrating the range of available options for evalu- ating incentives, making a case for the linkage of research methods chosen to the research question of interest and advocating for a combination of research methods as appropriate. CHAPTER 8: PERFORMANCE MANAGEMENT OF RECRUITERS This chapter shifts the focus from influence attempts aimed at the potential recruit to examination of recruiting systems. Service recruiting managers establish systems to select recruiters from among the available pool of Service members, to train and develop those new recruiters, to open recruiting offices in specific locations, to establish production goals for each recruiter, to motivate recruiters with reward and recognition programs, and to monitor and assess recruiter performance. Many options are available for each of these systems, and each is open to evaluation. In some cases (for example, selection of new recruiters), there are continuing

OCR for page 159
170 EVALUATING MILITARY ADVERTISING AND RECRUITING research programs to evaluate the effect of alternative programs. In other cases (for example, the effects of recruiter reward and recognition pro- grams), research or evaluation is rarely attempted. Given the central role that recruiter productivity plays in the recruiting process, all aspects of recruiter performance management should be subjected to evaluation efforts. Recruiter Selection There is a substantial literature, both military and civilian, addressing the problem of selecting people for sales occupations generally and mili- tary recruiting positions specifically. There is a long history in civilian settings of successfully utilizing various selection techniques to identify individuals with a high likelihood of success in sales-oriented positions. Selection in civilian settings involves an applicant pool eager to be selected, which is often not the case in the recruiter selection setting. It would appear worthwhile to consider changes and enhancements in the incentives to take on a recruiter position (e.g., links to career advance- ment) in order to increase the pool of individuals able and willing to serve as recruiters. Conclusion: Given the body of research on selection for sales-oriented positions, it is likely that there are more efficient and effective methods than currently used for choosing those personnel who should be assigned as recruiters. Recommendation: We recommend continued research on the devel- opment of effective recruiter selection strategies, in conjunction with a consideration of career incentives for service as a recruiter. Recruiter Training The committee's earlier report recommended that the Services develop and implement training systems that make maximum use of realistic practice and feedback. We note here the importance of evaluation of training programs, including giving careful attention to the outcome variables of interest. It is important to remember that development takes place in ways other than formal training programs. Often, individual feedback and coaching around certain experiences are very effective ways to shape behavior. Experiential learning and associated coaching, however, assume that there are capable coaches who understand what the desired behavior

OCR for page 159
CONCLUSIONS AND RECOMMENDATIONS 171 is and who can communicate performance deficits and strategies for improvement. Recommendation: We recommend that the Services expand their evaluation of overall training of recruiters to include the study of other informal development opportunities. In particular, assessment and improvement of the supervisory and coaching skills (to include on-thejob training) of those who train recruiters may be a fruitful approach. Performance Goals The question of how best to establish recruiter goals (and the question of whether those goals should be individual or team based) is still open. Given that recruiting duty is often cited as an extremely stressful job- because of the constant pressure to "make goal" there would seem to be high payoff in defining the variables relevant to military recruiting goals and evaluating the goal-setting process. Recommendation: We recommend a program of research aimed at evaluating the effects of goals on recruiter behavior and outcomes. Recruiter Performance Simple outcome measures (e.g., number of contracts) may be subject to a variety of external constraints (e.g., location) and may not capture the full range of important recruiter activities. There is a foundation of previ- ous research on the dimensions of effective recruiter performance that merits updating. A complete and current model of the dimensions of recruiter performance is needed as the basis for an effective performance evaluation system. Recommendation: We recommend research to develop a complete model of recruiter performance and to develop performance appraisal instruments and feedback processes based on this model.