|
Items in cart [0] |
Questions? Call 888-624-8373 |
HARDBACK + PDF HARDBACK PDF BOOK PDF CHAPTERS Free Resources |
||||||||||||||||
|
||||||||||||||||
|
|
|||||||||||||||
| Copyright © 2009. National Academy of Sciences. All rights reserved. Terms of Use and Privacy Statement |
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 159
9
Conclusions and Recommendations
he primary objective of this study is to help the Department of
Defense (DoD) improve its research on advertising and recruiting
polices. We anticipate that in the coming decade DoD will field
and test new advertising and recruiting initiatives designed to improve
the recruiting outlook and avoid the shortfalls of the last decade. In order
to discover the most promising policies, in the committee's view DoD
needs a comprehensive research and evaluation strategy based on sound
research principles that will ensure valid, reliable, and relevant results. In
this report, we present an evaluation framework that links different types
of research questions to different research methodologies.
The framework identifies four major categories of research questions
and four broad methodological approaches. The first category of research
question asks: "What does a target audience see as attractive or unattractive
features of a program?" It is well suited to examination via qualitative
methods, such as focus groups; unstructured or open-ended surveys; and
interviews. The second category of research question asks: "What is the
effect of a program on specified attitudes or behavioral intentions?" It is
well suited to examination via surveys, experiments, and quasi-experiments.
The third category of research question asks "What is the effect of a
proposed new program on enlistment?" It is well suited to examination
via experiments and quasi-experiments. The final category of research
question asks "What is the effect of an existing program on enlistment?"
It is well suited to examination via econometric modeling.
The committee's work during Phase I led us to conclude that there are
a number of critical problem areas or topics needing more intensive study.
159
OCR for page 160
160
EVALUATING MILITARY ADVERTISING AND RECRUITING
Some problems arise because of the need for ongoing, up-to-date infor-
mation that can serve as early-warning indications of potential recruiting
problems or that can point to areas in which improvements are needed.
Other problem areas are important because, in our view, they are central
to improving the overall recruiting climate. We selected six areas as the
central focus of this report. After devoting Chapter 2 to issues of theory as
a guide to effective evaluation research, Chapters 3 through 8 each exam-
ine one of the six areas. The various chapters also introduce different
methodological approaches to evaluation.
CHAPTER 2:
THEORETICAL APPROACHES
The chapter outlines a general framework for thinking about effective
program design. The first step is to identify the fundamental factors that
impact a target population's enlistment behavior. The second step is to
derive strategies (often informationally based) to change, enhance the
effect of, or mitigate the effect of those determinants. We outlined a wide
range of variables and processes that program designers must potentially
take into account, drawing heavily on research from adolescent develop-
ment, communications, economics, psychology, and sociology. These per-
spectives set the stage for conducting the necessary research to inform
program design and program evaluation.
There are two distinct theoretical approaches to enlistment behavior:
decision theory, based primarily in psychology, and the econometric
theory of enlistment supply. While there is some overlap in these two
theoretical traditions, they have distinct approaches. Decision theory is
more highly developed for the purpose of conceptualizing and measur-
ing behaviors that affect individual decision making. Econometric theory
is more formally developed with respect to aggregate enlistment out-
comes and various exogenous influences. Accordingly, a key objective of
this chapter is to build an integrated perspective on the behavioral and
econometric approaches.
Conclusion: The role of theory is crucial to the design of interventions to
increase enlistment behavior. When enlistment programs are developed
Theoretically, they run a great risk of being ineffective.
OCR for page 161
CONCLUSIONS AND RECOMMENDATIONS
CHAPTER 3:
MONITORING TRENDS IN YOUTH ATTITUDES,
VALUES, AND PROPENSITY
Determinants of Propensity
161
The chapter proposes survey methods as the most suitable research
design for tracking changes in propensity as well as for assessing the
underlying beliefs that are related to propensity. The information gathered
in youth attitude surveys is valuable to many research studies, such as the
advertising studies proposed in Chapter 4, but it is also valuable in its
own right by providing early warning indicators of changes in the pro-
pensity for military service. The chapter reiterates a key point from the
committee's earlier report, namely, that as propensity to enlist is the major
direct determinant of actual enlistment, increasing propensity in the youth
population should be a key objective for the military. We summarize a
model of the determinants of propensity that we laid out in the earlier
report and build an argument that research on propensity in the youth
population should measure the key determinants of propensity. We briefly
review ongoing survey efforts dealing with propensity, noting that they
do not consistently include these key determinants of propensity. We also
note a tendency for survey research dealing with propensity to make use
of research designs that do not yield complete data on individuals. If the
interest is simply documenting the proportion of respondents choosing
each alternative to a survey item, then randomly distributing items among
respondents will yield accurate results, as individual-level analysis is not
central to the research question. But when there is interest in the pattern
of relationships among variables, designs involving complete data at the
individual level are needed.
Conclusion: Previous survey research examining propensity to enlist has
not consistently measured the key classes of determinants of propensity
(i.e., attitudes, norms, and self-efficacy), nor has it consistently used
research designs permitting analysis at the individual level. This has lim-
ited the ability to test complete models of the determinants of propensity.
Recommendation: We recommend that survey research examining
propensity be designed to incorporate the key determinants of pro-
pensity and that it be designed to permit meaningful analysis at the
individual level.
OCR for page 162
162
EVALUATING MILITARY ADVERTISING AND RECRUITING
Conclusion: We also note that our model of the determinants of propen-
sity includes the role of important influencers, such as parents and peers.
One important implication of this is that the effects of interventions, such
as advertising, are not fully addressed by focusing solely on the direct
effects of the intervention on the potential recruit.
Recommendation: We recommend that evaluation efforts consider
potential effects on key influencers as well as on potential recruits,
and that efforts be made to assess such indirect effects on propensity.
A Program of Survey Research
The second part of the chapter provides a series of concrete recom-
mendations for a program of survey research, expanding on ideas set
forth in our letter report to the Department of Defense of tune 2000. A
program of monitoring surveys, which have the potential to yield very
high-quality data about propensity and its determinants, is presented. We
propose a cohort-sequential design, in which samples of youth are
obtained annually in the 11th grade of school (i.e., age 16-17) and moni-
tored through the age of roughly 23. We discuss a range of issues in the
design of such a project, including information-gathering format (e.g.,
self-completed questionnaire versus telephone interview), means of
accessing the sample (e.g., school-based surveys versus random house-
hold sampling), mechanisms for follow-up surveys over time, issues in
the scheduling of surveys, and sampling strategies. We note that such a
project is a significant investment and should not be undertaken unless
the resources for a minimum of five years can be committed. We note that
a variety of options are available with the broad framework we develop.
For example, one possibility is to survey 11th graders, with annual follow-
up; another is to survey both 11th and 12th graders, with follow-up every
two years. The chapter develops the trade-off among the options, noting
that many details cannot be specified in advance.
Recommendation: We recommend that consideration be given to
undertaking a school-based survey, using a cohort sequential design,
in which students are sampled in the 11th grade and possibly the 12th
grade and regularly resurveyed until the age of 23 or 24.
OCR for page 163
CONCLUSIONS AND RECOMMENDATIONS
Item Content
163
The final section of the chapter discusses item formats for assessing
the important variables in the model of the determinants of propensity
developed in our earlier report and reiterated here. Specific examples of
item formats for effectively measuring propensity, attitude, norms, self-
efficacy, behavioral beliefs, outcomes, and open-ended queries about out-
comes and about influencers are provided.
Recommendation: We recommend that surveys dealing with pro-
pensity and its determinants assess the variables of interest using
established item formats.
CHAPTER 4:
ADVERTISING PLANNING: GENERATIVE AND
EVALUATIVE APPROACHES
In order to develop and test effective advertising themes, two types of
research designs are required. The first type of design is for the develop-
ment of preliminary but promising message strategies. This step requires
qualitative exploratory or generative research designs (focus groups, in-
depth interviewing, etc.~. After promising themes are developed, the
second type of research design is necessary for testing theme awareness
and market impact. The best designs for this step are experimental and
quasi-experimental studies.
This chapter describes the stages in the development and evaluation
of an advertising campaign, discussing relevant research methods for
each phase. The chapter reviews some findings from the committee's
earlier report, including the decline in both the proportion of youth assign-
ing high value to duty to country and in the proportion of youth who
associate the goal of duty to country with military service. We then
develop a framework for developing advertising campaigns that follows
a systematic process and builds on sound information about the value
structure of youth. That framework involves (1) tracking the competitive
environment for military recruitment to detect factors affecting youth
understanding and views of military service; (2) examination of audience
member beliefs, goals, and language; (3) development of a range of mes-
sage strategies for military recruitment, and (4) allocation of resources to
advertising message strategies.
Conclusion: There is a need for research to provide a more complete
picture of the belief and value structure of the youth population, particu-
OCR for page 164
164
EVALUATING MILITARY ADVERTISING AND RECRUITING
larly the beliefs relating to public service, duty to country, personal sacri-
fice, and concern for others. It is also helpful to study the language used
by youth as they think and speak about these issues, so that this informa-
tion can be used to develop effective messages on the topic.
Recommendation: We recommend a program of research that begins
with generative techniques to understand the concepts and language
used by youth in considering alternative courses of action (e.g., edu-
cation versus military service) and continues with survey research
that measures the full range of beliefs, attitudes, and values that
emerge as linked to these alternate courses of action.
Conclusion: Effective advertising campaigns involve a message strategy
strongly linked to beliefs and values that affect decision making. A crucial
component of the evaluation of military advertising is an examination of
its success in affecting the intended values and beliefs. Using beliefs and
values as outcome variables as well as enlistments permits a clearer under-
standing of why a given advertising campaign is or is not successful than
using enlistments alone.
Recommendation: We recommend that advertising message strate-
gies be evaluated in terms of their effects on targeted beliefs and
values. Such evaluation should make use of experimental designs in
controlled settings as well as small-scale, in-market experiments.
Recommendation: We recommend that a policy be adopted of regu-
larly developing and evaluating alternative approaches that ch~llens,
existing message strategies.
CHAPTER 5:
DETERMINING OPTIMAL LEVELS OF ADVERTISING AND
RECRUITING RESOURCES
. ~ _
The chapter focuses on econometric methods, as these approaches are
most useful for assessing the optimal levels of recruiting programs and
resources. Econometric methods can be used to isolate and identify the
effects of existing resources, policies, and external factors affecting recruit-
ing outcomes as well as their costs. There is by now a relatively well-
OCR for page 165
CONCLUSIONS AND RECOMMENDATIONS
165
developed body of econometric research that has identified some of the
most important determinants of enlistment supply as well as the cost and
effectiveness of various trade-offs among different recruiting and adver-
tising resources. Estimates are based on the natural variation in key
recruiting resources and outcomes (usually aggregated) that occur over
time and across different geographic locations.
Brief overviews of two types of econometric models are provided-
models of enlistment supply and models of recruiting cost. We review
studies over the past two decades on the effects of recruiting and adver-
tising on enlistment and present summary tables comparing the various
studies. We note considerable variability in results across studies and
suggest a series of methodological features that have not been consis-
tently incorporated into the studies and thus may contribute to the
variability in results and the difficulty in giving a definitive answer to
questions about the elasticity of enlistment with respect to advertising.
Conclusion: More sophisticated methods, controlling appropriately for
factors affecting enlistment supply, both those that are directly observ-
able to the researcher and those that must be inferred, such as recruiter
effort, are necessary to obtain efficient, unbiased estimates of the effects of
recruiting resources. Moreover, more complete evaluation of the effects of
some types of resources, especially advertising content, require estima-
tion using more flexible functional forms in the econometric analysis. To
apply these methods, however, better data need to be collected and sys-
tematically maintained. The specific conclusions and recommendations
discussed below are conditional upon the availability of better data.
Recommendation: Collect and maintain better data to support the
estimation of enlistment supply functions and to evaluate the effec-
tiveness of recruiting resources.
Conclusion: Recruiter productivity varies with experience, and hence
sudden changes in the size of the recruiting force result in declines in
average experience. Failure to incorporate recruiter experience in models
of recruiter effects may bias study results.
Recommendation: We recommend that future research on the effects
of recruiters on enlistment supply incorporate the effects of recruiter
experience.
OCR for page 166
166
EVALUATING MILITARY ADVERTISING AND RECRUITING
Conclusion: Recruiter incentives have been incorporated in supply
models via recruiters' quotas, based on the assumption that increasing
recruiting quotas increases effort.
Recommendation: We recommend that supply models incorporate
more complete and realistic models of recruiter incentives that more
fully capture the complexities of recruiter incentives.
Conclusion: Research to date has not incorporated the effects of reserve
forces competition on active-duty recruiting.
Recommendation: We recommend that supply models incorporate
reserve forces competition for nonprior-service recruits.
Conclusion: Econometric estimates of the effects of advertising have
focused largely on expenditures of impressions (i.e., the number of
relevant individuals who see or hear the advertisement, often measured
in terms of gross rating points). Such estimates have not attempted to
measure differences in effects by specific advertising content.
Recommendation: We recommend that research attempt to evaluate
advertising in terms of thematic content in order to determine whether
advertising effects vary by content, as well as by impressions and
expenditures.
Conclusion: The functional forms (i.e., the shape of the relationship
between the recruiting incentive and enlistment) of econometric supply
models have been relatively restricted. The underlying assumptions (e.g.,
that each additional advertising dollar has the same effect regardless of
the level of total expenditure) may not be correct, and an examination of
more flexible functional forms would be fruitful.
Recommendation: We recommend that research on supply models
make use of flexible functional forms, rather than imposed functional
forms.
OCR for page 167
CONCLUSIONS AND RECOMMENDATIONS
CHAPTER 6:
THE TIMING AND LEVELS OF JOINT AND
SERVICE-SPECIFIC ADVERTISING
Minimum Advertising Level to Maintain Propensity
167
The chapter first addressed the question of whether there is a mini-
mum level of advertising necessary for a cost-effective recruiting program,
even if that advertising is not necessary to achieve contemporaneous
enlistment contract goals. Historically, when the recruiting climate is good
and recruits are plentiful, military planners tend to cut advertising bud-
gets, thereby contributing to a reduction in awareness capital and pro-
pensity levels. This may possibly set up a boom or bust cycle, in which
propensity falls, recruiting becomes more difficult, and then advertising
funds have to be restored. We present a model that describes the condi-
tions under which it would be cost-effective to advertise in the interests of
future enlistment supply and review research to date that speaks to the
issue. While extant research suggests that advertising may have effects
for only a short period of time, the data available to prior researchers are
limited for several reasons. First, they do not permit examining both
lagged effects and nonlinear effects within a time period. Second, they
focus on advertising aimed at youth at the point of the enlistment deci-
sion and do not permit examining possible supplemental advertising
approaches, such as those aimed a youth several years prior to an enlist-
ment decision, or those aimed at adult influencers, such as parents.
Conclusion: Research to date does not permit a definitive answer to the
question of the cost-effectiveness of advertising above and beyond that
which is necessary to achieve current recruiting goals.
Recommendation: We recommend a focused effort to maintain
advertising data in a systematic way for purposes of estimating a
supply curve that incorporates the potential for both time-lagged and
nonlinear advertising effects.
Recommendation: We recommend a program of research, incorpo-
rating quasi-experimental methods, to examine advertising effects over
an extended period of time.
OCR for page 168
168
EVALUATING MILITARY ADVERTISING AND RECRUITING
Levels of Joint and Service-Specific Advertising
The chapter then turns to the optimal levels of joint and Service-
specific advertising. It is our opinion that certain types of advertising
themes, such as generic themes designed to increase overall propensity,
are best done as a joint program, while advertising themes featuring spe-
cific benefits of military service are best done in the Service program.
What we do not know is what advertising fund level should be allocated
to joint programs. We note that issues of scale play a role in addressing
this issue, as certain types of advertising (e.g., television) do not appear to
have a constant effect across levels of expenditure. The larger the recruit-
ing effort and the larger the budget, the greater the potential value of a
multifaceted campaign, with some resources targeted toward providing
information about specific Services to those already with a propensity to
enlist and others targeted toward increasing propensity among those cur-
rently without it.
Recommendation: We recommend a program of research aimed at
examining the effects and cost-effectiveness of information-oriented
versus values-oriented advertising in joint and Service-specific adver-
tising programs.
CHAPTER 7:
DETERMINING OPTIMAL TYPES OF INCENTIVES
Over the years of the All-Volunteer Force, various incentives have
been developed and offered to help strengthen and shape military enlist-
ments. The effectiveness of these incentives has been addressed, and dem-
onstrated, using a variety of evaluation approaches. This chapter considers
methodological issues in determining the optimal types of enlistment
incentives for specific recruiting goals. We begin by reviewing different
types of enlistment options. Next, we consider the various types of effects
the incentives may produce, both intended and unintended, and the
related methodological issues in assessing their impact. We continue with
a brief review of the analysis methods discussed in the earlier chapters,
which is followed by a longer consideration of analytical issues applicable
to each of the evaluation approaches. Finally, drawing on each of these
areas, we conclude with a discussion of matching potential incentives and
their effects with the appropriate assessment goals and evaluation
methods.
A central message of the chapter is that each of the evaluation meth-
odologies introduced in this volume (qualitative methods, surveys, econo-
OCR for page 169
CONCLUSIONS AND RECOMMENDATIONS
169
metric models, and experiments and quasi-experiments) can play a useful
role in addressing different questions that policy makers may ask about
current or proposed incentives. The chapter offers concrete illustrations
of the research framework introduced in Chapter 1. It identifies focus
groups as of particular value in providing insight into the appeal of vari-
ous features of proposed incentives and in facilitating the discovery and
exploration of new incentive options. It identifies surveys as of particular
value in comparing the relative attractiveness of a substantial number of
incentive options. It identifies econometric methods as of particular value
when examining the effects of existing programs on actual enlistments
over time and over differing recruiting environments, providing statisti-
cal control for a wide variety of potentially confounding factors (e.g.,
geographic effects, effects of changes in the economy). It identifies experi-
mental and quasi-experimental methods as of particular value when the
question of interest is estimating the effects on enlistment of a new incen-
tive prior to full implementation.
In addition, the chapter emphasizes the value of combining approaches.
This might include, for example, the use of focus groups to help explore
and define prospective enlistment options that are then tested in large
youth market surveys or in pilot tests employing experimental designs.
Another example would be the application of econometric methods or
other analytical techniques to existing datasets in order to help interpret
survey results; for example, the other approaches could be used to derive
a metric that can be used to help translate stated enlistment intentions in
the survey into estimated enlistment rates.
The focus of the chapter is not on specific conclusions and recommen-
dations, but rather on illustrating the range of available options for evalu-
ating incentives, making a case for the linkage of research methods chosen
to the research question of interest and advocating for a combination of
research methods as appropriate.
CHAPTER 8:
PERFORMANCE MANAGEMENT OF RECRUITERS
This chapter shifts the focus from influence attempts aimed at the
potential recruit to examination of recruiting systems. Service recruiting
managers establish systems to select recruiters from among the available
pool of Service members, to train and develop those new recruiters, to
open recruiting offices in specific locations, to establish production goals
for each recruiter, to motivate recruiters with reward and recognition
programs, and to monitor and assess recruiter performance. Many options
are available for each of these systems, and each is open to evaluation. In
some cases (for example, selection of new recruiters), there are continuing
OCR for page 170
170
EVALUATING MILITARY ADVERTISING AND RECRUITING
research programs to evaluate the effect of alternative programs. In other
cases (for example, the effects of recruiter reward and recognition pro-
grams), research or evaluation is rarely attempted. Given the central role
that recruiter productivity plays in the recruiting process, all aspects of
recruiter performance management should be subjected to evaluation
efforts.
Recruiter Selection
There is a substantial literature, both military and civilian, addressing
the problem of selecting people for sales occupations generally and mili-
tary recruiting positions specifically. There is a long history in civilian
settings of successfully utilizing various selection techniques to identify
individuals with a high likelihood of success in sales-oriented positions.
Selection in civilian settings involves an applicant pool eager to be
selected, which is often not the case in the recruiter selection setting. It
would appear worthwhile to consider changes and enhancements in the
incentives to take on a recruiter position (e.g., links to career advance-
ment) in order to increase the pool of individuals able and willing to serve
as recruiters.
Conclusion: Given the body of research on selection for sales-oriented
positions, it is likely that there are more efficient and effective methods
than currently used for choosing those personnel who should be assigned
as recruiters.
Recommendation: We recommend continued research on the devel-
opment of effective recruiter selection strategies, in conjunction with
a consideration of career incentives for service as a recruiter.
Recruiter Training
The committee's earlier report recommended that the Services
develop and implement training systems that make maximum use of
realistic practice and feedback. We note here the importance of evaluation
of training programs, including giving careful attention to the outcome
variables of interest.
It is important to remember that development takes place in ways
other than formal training programs. Often, individual feedback and
coaching around certain experiences are very effective ways to shape
behavior. Experiential learning and associated coaching, however, assume
that there are capable coaches who understand what the desired behavior
OCR for page 171
CONCLUSIONS AND RECOMMENDATIONS
171
is and who can communicate performance deficits and strategies for
improvement.
Recommendation: We recommend that the Services expand their
evaluation of overall training of recruiters to include the study of
other informal development opportunities. In particular, assessment
and improvement of the supervisory and coaching skills (to include
on-thejob training) of those who train recruiters may be a fruitful
approach.
Performance Goals
The question of how best to establish recruiter goals (and the question
of whether those goals should be individual or team based) is still open.
Given that recruiting duty is often cited as an extremely stressful job-
because of the constant pressure to "make goal" there would seem to be
high payoff in defining the variables relevant to military recruiting goals
and evaluating the goal-setting process.
Recommendation: We recommend a program of research aimed at
evaluating the effects of goals on recruiter behavior and outcomes.
Recruiter Performance
Simple outcome measures (e.g., number of contracts) may be subject
to a variety of external constraints (e.g., location) and may not capture the
full range of important recruiter activities. There is a foundation of previ-
ous research on the dimensions of effective recruiter performance that
merits updating. A complete and current model of the dimensions of
recruiter performance is needed as the basis for an effective performance
evaluation system.
Recommendation: We recommend research to develop a complete
model of recruiter performance and to develop performance appraisal
instruments and feedback processes based on this model.
Representative terms from entire chapter:
enlistment supply
