Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 40
3
Monitoring Trends in Youth
Attitudes, Values, and Propensity
n our earlier report (National Research Council, 2003), we argued that
"military readiness may best be served when the first role of military
advertising is to support the overall propensity to enlist in the youth
population and to maintain a propensity level that will enable produc-
tivity in military recruiting" (p. 6~. On this basis, we recommended that "a
key objective of the Office of the Secretary of Defense should be to increase
the propensity to enlist in the youth population" (p. 8~.
Clearly the more one knows about the determinants of propensity,
the more it becomes possible to develop effective communications or other
types of interventions to increase propensity. In this chapter we propose a
cohort-based sequential sample survey with a longitudinal component
that we think will provide the information necessary for both tracking the
determinants of propensity and for developing more effective communi-
cations to increase and maintain the pool of youth with the propensity to
join the military. This survey design has several advantages over current
and recent survey designs that have been used by the Department of
Defense (DoD) to track youth attitudes toward military service:
~ . . . .
1. By assessing an unchanging set of core questions annually, this
survey design provides a consistent set of data that can be used to moni-
tor and track changes in the determinants of propensity, as well as in
propensity itself, over time. Consistent measures from a constant set of
questions are obviously necessary to discern changes in propensity over
time across different cohorts. This perhaps can provide an early warning
to changes in the recruiting market and provide some indication regard-
40
OCR for page 41
MONITORING YOUTH TRENDS
41
ing how advertising and other resources and incentives should be ad-
justed to reflect market changes.
2. One can assess the reasons for differences (cross-section) and changes
(longitudinal) in key outcome variables over time. With structured model-
ing, the Services can determine which of the underlying determinants of
propensity are driving current levels of propensity, and how they may
have differed in the past. This will affect the development of advertising
messages and the levels and types of incentives used to attract youth into
military service.
3. The longitudinal nature of the data in particular, the follow-up of
individuals provides the basis for assessing how critical determinants
of propensity evolve over time. Not only will this allow the assessment of
changes in the relative importance of factors influencing propensity to
enlist over time, but also the follow-up sample will permit analyses of the
relationships among initial intention, subsequent intention, and actual
status (enlistment, college attendance, or entering the civilian workforce).
These latter analyses may suggest ways to help young adults act on their
initial intentions to enlist.
In the previous chapter, we proposed a model that suggested that
propensity to join the military (or a particular Service such as the Army or
Navy) was primarily determined by attitudes, norms, and self-efficacy
(see also National Research Council, 2003~. According to the model, the
relative importance of these three variables as determinants of intention
can vary as a function of both the behavior and the population being
considered. That is, some behaviors may be primarily influenced by atti-
tudes while others may be primarily influenced by norms or self-efficacy.
Similarly, a behavior that is primarily under attitudinal influence in one
population (or population segment) may be primarily under the influence
of norms or self-efficacy in another population.
In order to develop interventions to increase the proportion of the
population with a propensity to enlist at any given point in time, one
needs to assess the underlying determinants of propensity. That is, the
more one knows the values and relative importance of attitudes, norms,
and self-efficacy in a given population, and the more one knows about the
beliefs underlying these attitudes, norms, and perceptions of self-efficacy,
the more likely it is that one can design an effective message (or other
type of intervention) to increase propensity (Fishbein et al., 2001~. How-
ever, this type of information (i.e., the most relevant data for guiding the
development of effective messages or other interventions to increase pro-
pensity) is currently not available. Although some of this information was
assessed by the Youth Attitude Tracking Study (YATS), a major problem
with YATS is that, rather than getting complete data from each respon-
OCR for page 42
42
EVALUATING MILITARY ADVERTISING AND RECRUITING
dent, respondents answered only small, randomly selected subsets of the
questions, making complete analyses at the individual level impossible.
It was for this reason that the committee prepared a letter report
evaluating YATS, which was included as Appendix A in the committee's
Phase I report (National Research Council, 2003), and presenting a num-
ber of findings and recommendations. Among these was the recommen-
dation that "ongoing surveys to assess the critical determinants of
propensity should be conducted on a regular basis." These surveys would
allow for individual-level analyses to identify the psychosocial determi-
nants of propensity. Thus, we recommended that, whenever a survey is
designed, "consideration should be given to randomly assigning inter-
related blocks of information to the same subgroups. Consideration
should be given to maintaining sufficient sample size and content within
a block of relevant questions so that multivariate analysis can be con-
ducted without serious missing data problems" (p. 294~.
In addition to conducting a survey to identify the determinants of
propensity, it is important to assess changes in these underlying beliefs
over time. Thus we also recommended that "DoD consider using a con-
tinuous tracking survey methodology for such issues as propensity to
enlist, advertising awareness, awareness of direct response campaigns,
involvement in high school activities, and perceptions of the military"
(p. 295~. More specifically, we recommended that "a portfolio of surveys
at different time intervals replace the current annual YATS administra-
tion" (p. 294~. Moreover, we suggested that, to be maximally useful, the
formatting of the questionnaire must permit individual-level multivariate
analyses. For example, it should permit assessment of an individual's
complete set of salient beliefs about the consequences (or costs and ben-
efits) of joining the military.
It is important to note that in response to these suggestions, DoD
funded Wirthlin Worldwide to conduct a number of youth and influ-
encer surveys (Bailey et al., 2002; Sattar et al., 2002~. More specifically,
Wirthlin conducted four surveys with respondents between ages 15 and
21 (March and April 2001; fuly and August 2001; October and November
2001; and October and November 2002) and four surveys with adults
ages 22 to 85 (May 2001; September and October 2001; lanuary 2002; and
September 2002~. While the youth surveys provided information neces-
sary for testing some of the relations in the theoretical model, no single
survey obtained information on all of the theoretical determinants of
propensity. Thus while it was possible to examine some relationships
that could not be examined by the YATS data, a full test of the model was
not possible. Moreover, a large number of items focused on "genera-
tional" questions and in particular on variables assumed to be related to
"millennials" (e.g., on team orientation, decision making, life satisfac-
OCR for page 43
MONITORING YOUTH TRENDS
43
lion). There is little evidence that variables of this type are in any way
related to the propensity to enlist or to actual enlistment behavior (Na-
tional Research Council, 2002~.
Turning to the adult surveys, Wirthlin does provide some interesting
and important data concerning adult perceptions of the military and
adults' propensity (or intentions) to "encourage [young people] to join a
military service" or to "get a job" or to "attend a four-year college." It is
interesting to note, however, that there are no questions assessing the
determinants of these intentions to encourage youth to choose any of
these career paths.
In sum, the current set of surveys does not yet provide the data neces-
sary for identifying the critical determinants of propensity, and thus they
do not provide the data necessary for developing effective interventions
(including mass media advertising) to increase the pool of individuals
with a propensity to enlist. In order to assist and provide guidance for
mass media and other interventions to increase this pool of people, com-
plete data concerning the determinants of propensity at the individual
level are needed. Data are also needed to evaluate the effectiveness of
advertising and other interventions. Thus, in addition to assessing
whether a communication campaign has produced changes in propensity
(and its underlying determinants), it is also necessary to track the extent
to which the message is reaching its intended audiences (i.e., it is impor-
tant to assess exposure to message content). In addition, in order to evalu-
ate message (or advertising) effectiveness, it is necessary not only to
determine whether a message is having a direct effect on propensity and
its determinants, but also to look for other paths of effect for example,
messages can influence relevant referent (such as parents) who, in turn,
influence youth; messages may increase conversations among youth vis-
a-vis joining the service; messages may increase the likelihood that youth
will talk to others (people in the military, school counselors, parents)
about joining the military. One implication of these other paths is that,
although a campaign may be effective, one may not always find differ-
ences between those who are and those who are not exposed to the adver-
tising campaign. Thus, surveys should be designed to take these other
paths of effect into account.
USE OF SURVEY RESEARCH
Survey research can be used for many purposes. A survey conducted
at a given point in time can provide important prevalence data about
propensity and its psychosocial determinants. These data also permit the
investigation of relationships among beliefs, attitudes, perceived norms,
self-efficacy, and propensity. Equally important, experimental or quasi-
OCR for page 44
44
EVALUATING MILITARY ADVERTISING AND RECRUITING
experimental designs or both can be built into a given survey. For exam-
ple, in order to investigate how changes in one or more variables (or
policies) could affect propensity, respondents taking the survey could be
randomly assigned to different forms of the survey instrument that pre-
sented different recruitment or enlistment scenarios. Given this manipu-
lation, one could then assess, for example, how different increases in pay
or different education policies influenced the likelihood that one would
join the military. Furthermore, by repeating the survey (or parts of it)
over time, either cross-sectionally or longitudinally, one can observe
changes in any of these variables or in the relationships among them.
Moreover, by repeating surveys one can develop quasi-experimental
(time series) designs to assess the efficacy of a given advertising cam-
paign or recruitment strategy as a means of producing changes in pro-
pensity and its underlying determinants (Campbell and Stanley, 1966;
Shadish, Cook, and Campbell, 2002~.
Even when the major goal of conducting a survey is to provide infor-
mation to increase propensity to join the military, there are a number of
issues that must be addressed prior to developing one or more survey
instruments.
1. What is the target population? What age group (or groups) should
be targeted? That is, should one attempt to increase propensity among
youth, adolescents, or young adults? Alternatively, knowing that there
are important others who may strongly influence a person's propensity to
join the military, one must decide whether to view these influencers as the
target audience. It is therefore important to consider whether different
surveys are needed for different populations and, if so, how often should
each be conducted.
2. With one or more target populations identified, one needs to con-
sider a number of sampling issues: size of samplers), number of (distin-
guishable) samples, response rate, and other biases. In addition, one
should consider questions of timing (annual, quarterly, continuous).
3. What survey methodology should be used and where and how can
one maximize access to the population of interest? More specifically,
should the mode of administration be face-to-face, self-administered,
conducted by telephone (each of which can also be computer assisted),
conducted via the Internet, or should it be a mail questionnaire? Simi-
larly, should it be individually or group administered (e.g., at school)?
4. Turning to the content of the survey, one must consider such things
as question ordering, question coverage, and question wording. Other
issues relate to question types; for example, should the questions be open-
ended (with and without prompts) or closed-ended (multiple choice)?
What should be the order of presentation of the questions?
OCR for page 45
MONITORING YOUTH TRENDS
45
The remainder of this chapter will discuss issues related to survey struc-
ture, implementation, and content.
SURVEY STRUCTURE AND IMPLEMENTATION DECISIONS
The issues and recommendations presented here are broadly compat-
ible with those in our 2000 letter report concerning YATS, but here we
provide a greater degree of specificity along some dimensions. What is
recommended here need not be viewed as a proposed replacement for the
current pattern of telephone surveys, except insofar as cost constraints
dictate some degree of trade-off. That said, it should be acknowledged
that if the series of monitoring surveys recommended here were to be
carried out, the overlap in coverage is such that it would eventually be
possible to phase out much of the telephone survey work now being
done.
The focus of this section is on monitoring trends in youth attitudes,
values, and propensity. Nevertheless, the survey strategies outlined here
could readily be adapted to incorporate other material. One example
would be to measure, and perhaps monitor, reactions to advertising (see
Chapter 5), although we would not propose this to the exclusion of other
survey and nonsurvey methods focusing on advertising. Another exam-
ple, as suggested above, would be to explore reactions to possible new or
modified incentives for military service (see also Chapter 7~.
A number of matters will have to be resolved before undertaking a set
of surveys such as those proposed here. We outline them briefly, and in
each case state the committee's recommendations.
Need for Long-Term Funding
Monitoring survey designs are built with the expectation that fund-
ing will be available to carry out the research over a considerable period
of time. It is not a good investment of federal funds or of investigator
time to initiate monitoring survey efforts only to drop funding after a
short period of time. Careful consideration needs to be given to the spon-
sor's ability and willingness to make funding commitments for multiyear
(e.g., five-year) periods. The survey approach outlined here supposes such
funding commitments, and the committee strongly urges that monitoring
surveys not be initiated until such commitments are in place.
Selecting Target Samples
Earlier work summarized in the committee's Phase I report showed
that, among many young people, enlistment propensity tends to firm up
OCR for page 46
46
EVALUATING MILITARY ADVERTISING AND RECRUITING
by or before the end of high school. Although students provide answers
to propensity questions as early as 8th grade in the Monitoring the Future
(MTF) surveys, the proportions providing "definite" answers are much
higher among 12th grade students. Senior year propensity measures, how-
ever, can be complicated by the likelihood that many seniors who eventu-
ally enlist make their decisions and commitments before leaving high
school; accordingly, their "propensity" answers might better be described
as reports of decisions already taken. This is clearly consistent with the
often reported finding (Fishbein and Ajzen, 1975; Ajzen and Fishbein,
1980; Albarracin, Johnson, Fishbein, and Muellerleile, 2001) that the
shorter the time interval between the assessment of intention (or propen-
sity) and the observation of behavior, the better the prediction (i.e., the
higher the correlation).]
Taking into account these factors, the committee recommends that
the lower age boundary for target samples be about age 16-17. If there is
an interest in propensity or related factors at lower ages, they could be the
subject of limited special studies rather than monitoring surveys. By age
16-17 young men and women have had to confront questions about their
next steps after high school (college, military service, civilian employ-
ment), so this seems a good lower age bound for target samples. Specifi-
cally, for reasons spelled out below, we recommend the 11th grade of
high school as an optimal start point for the target sample age range.
The end point for the target sample is less easy to specify; however,
given the increased recruiting attention to college students, the committee
recommends that inclusion extend to at least age 23 the point by which
most young adults either have completed college or are relatively unlikely
to do so. We leave open the possibility that it may prove useful to extend
the age span a bit farther; however, we think it unlikely that it would be
cost-effective to extend it much beyond age 25.
Use of Self-Completed Questionnaires
There are very large potential cost advantages in surveys employing
self-completed questionnaires. There are also constraints and limitations.
The committee considered several important constraints and judged them
acceptable for the proposed monitoring surveys.
1The fact that YATS excluded respondents who had already committed to the military,
while Monitoring the Future included all high school seniors, helps to explain why,
although both surveys found strong evidence that propensity does predict enlistment, MTF
found stronger intention-behavior relationships than did YATS (see, e.g., Bachman, Segal,
Freedman-Doan, and O'Malley, 1998~.
OCR for page 47
MONITORING YOUTH TRENDS
47
Self-completed questionnaires, particularly when carried out on a
large scale, tend to be limited to closed-ended multiple-choice question
items. In spite of these restrictions in question-and-answer format, self-
completed questionnaires have been used successfully in monitoring sur-
veys (e.g., the Youth Risk Behavior Survey, Monitoring the Future). In
contrast, YATS used telephone personal interview methods, which allow
for open-ended questions and for potentially complex question branching
strategies. Interestingly, even in the YATS telephone interviews, the sur-
vey data that tended to be most useful for monitoring purposes involved
closed-ended items. Accordingly, the committee would find it acceptable
to use self-completed questionnaires for monitoring purposes, even
though that may constrain the survey questions to closed-ended items.
(Open-ended items can be included in self-completed surveys, although
they add costs, complexities, and time delays.) The committee recom-
mends that other approaches (focus groups, smaller scale interview and
elicitation studies, etc.) be used to develop and evaluate the closed-ended
items.
Another extremely important limitation of self-completed question-
naires is that the target respondents must be able to read and understand
the questions. Limiting the target population to literate respondents could
be a serious bias in many studies, and even in the surveys proposed here
this constraint needs to be kept in mind. Nevertheless, given a primary
focus on potential military recruits, and given current and future military
requirements for literacy among all Service members, the committee con-
siders the literacy constraint tolerable.
One other dimension of self-completed questionnaires to be noted
involves respondent motivation. All survey methods require some degree
of motivation on the part of respondents, but arguably there are some
methods that are more inherently motivating than others. Face-to-face
personal interviews can be particularly motivating, because some degree
of personal relationship is established between interviewer and inter-
viewee. Telephone interviews may develop such a relationship to some
extent; however, in a period in which telemarketing has imposed greatly
on the good will of those still willing to answer their phones in person
(rather than use an answering machine), the motivational value of phone
interviewing may be debatable. In any case, one of the factors to be con-
sidered in developing any survey strategy is respondent motivation, and
that will be particularly important for the surveys proposed here. The
committee thinks that topics related to career choice are of great interest
and importance to most of those in the target survey population and that,
with sufficient care, self-administered questionnaires can be developed
that will maintain respondent motivation.
OCR for page 48
48
EVALUATING MILITARY ADVERTISING AND RECRUITING
Use of School-Based Administrations
Why survey in schools? To paraphrase Willie Sutton's bank robber
comment: because that's where the students are. School-based survey
administrations are wonderfully cost-effective at least from the stand-
point of survey researchers. Teachers and administrators, however, are
becoming increasingly dubious about school-based surveys, partly
because they seem to be proliferating, and also because the increased
demands for student performance and school accountability have made
classroom time a more precious commodity. The result is that obtaining
cooperation for school-based surveys is increasingly difficult.
Nevertheless, in spite of the difficulties, there is much to recommend
school-based group-administered self-completed questionnaire surveys:
the sampling is fairly straightforward and can be quite accurately repre-
sentative; response rates can be quite high among students; and, once
again, it can be very cost-effective. Moreover, school-based surveys can
also be treated as a baseline or starting point for panel surveys (repeated
surveys of the same respondents over time). Panel surveys involve some
respondent attrition, to be sure, but the base-year data can provide con-
siderable information about the lost respondents (and how to make statis-
tical adjustments to compensate for the loss).
Matters of Entry, Motivation, and Acceptability
A Pentagon-sponsored survey focused on military propensity, asking
only questions about military jobs and missions, is likely to be acceptable
to some school administrators and less so to others. Moreover, in most
schools there is likely to be a range of parent and student reactions to a
"military survey." Pentagon sponsorship should, of course, be acknowl-
edged; but if the survey were a joint undertaking sponsored by the DoD
and others, and if the content reflected that joint sponsorship, the survey
might be more attractive to all concerned. Moreover, in order to do a good
job, as well as to be broadly acceptable to school personnel, parents, and
students, the military portion of the questionnaire content should be bal-
anced in its items about military service, military working conditions,
duty to country, and the like. If the survey comes across as promilitary
rather than balanced in its approach, it is likely to generate considerable
controversy and resistance in some communities and school districts. An
emphasis on national service might aid in making the survey more
broadly attractive.
A closely related point is that if personal identification of respondents
is requested in order to permit follow-up surveys of some respondents,
great care must be taken to ensure, and communicate, that no identifying
OCR for page 49
MONITORING YOUTH TRENDS
49
information or individual survey responses will be available to anyone
other than those conducting the survey. In particular, it must be clear to
everyone that survey responses and the names of participants will not be
made available to recruiters. This is not the sort of thing that can be
mentioned once with the expectation that everyone will take note and
remember. Rather, it is something that will need to be repeated often, to a
number of relevant audiences.
Proposed Features of a Monitoring Survey Series
The committee considered a number of survey designs and options in
its deliberations. None of the options was ruled out; to the contrary, it was
considered important that multiple methods be employed at various
points. In particular, the use of focus groups and individual personal
interviews with open-ended questions was recommended as means of
forming and improving questionnaire items. Nevertheless, consistent with
the discussion of broad issues outlined above, the committee developed a
design that it recommends for a monitoring survey series. We outline
some key features below.
A Cohort-Sequential Design Using Self-Completed Questionnaires
The monitoring design proposed here would obtain new and rela-
tively large samples of high school students annually and then track sub-
sets of those students on a regular schedule during subsequent years. This
is a cost-effective research strategy for generating descriptive data on
youth and young adults, and it can permit disentangling changes over
time that reflect age differences, cohort differences, and secular (historic)
trends. (See, e.g., Johnston, O'Malley, and Bachman, 2003a, 2003b, 2003c,
for illustrations of all three kinds of changes in substance use.) The design
has a bonus feature: because it tracks the same individuals across time, it
can be the basis for panel analyses exploring individual changes and
growth trajectories, and it can distinguish chronological (and possibly
causal) ordering of events.
Base-Year Surveys of High School Students
The committee recommends that the starting point in the cohort-
sequential design consist of school-based group-administered self-
completed questionnaire surveys of 11th grade students throughout (and
representative of) the United States. As noted earlier, 11th grade is a point
by which many students have confronted choices about their next steps
OCR for page 50
50
EVALUATING MILITARY ADVERTISING AND RECRUITING
after high school but before very many have made firm commitments to
further education or to military service.
Most students in 11th grade are age 16 or 17, thus the choice of 11th
grade as the starting point sets a fairly narrow age boundary. The choice
of 11th grade rather than a particular age is governed, in part, by practical
considerations involved in school-based surveys; however, the choice of a
grade-based rather than age-based starting point also takes account of the
fact that decisions about next career or education steps tend to be firmed
up during 11th grade. In any case, we judge that grouping students by
grade rather than by age is well suited to the purposes of the research.
For the reasons outlined above, the committee recommends 11th
grade as the earliest point for the school-based monitoring surveys.
Although surveys at earlier grades (and ages) could reveal things about
the developmental progression of propensity, such could be the subject of
special studies rather than an ongoing monitoring effort. The committee
did not, however, reach a firm conclusion about whether 11th grade was
the only grade worth including in school-based surveys. An obvious
alternative would be to include both 11th and 12th grades, and we can see
advantages as well as disadvantages of such a strategy. Later in this chap-
ter we outline in greater detail two alternative strategies and consider
some of the costs and benefits of each. One strategy involves sampling
only 11th grade students and then tracking them with follow-up surveys,
perhaps as often as once a year. The other strategy involves sampling
both 11th and 12th grade students, with follow-up surveys every two
years. It is important to note that either strategy will take time to develop
the full range of respondents.
Periodic Tracking of Subsamples
Monitoring the Future tracks its panel respondents by mail every two
years, with half of them resurveyed 1, 3, 5, 7, 9, and 11 years after high
school and the other half resurveyed 2, 4, 6, 8, 10, and 12 years after high
school. (Later follow-ups take place at modal ages 35, 40, and 45.) This
every-other-year schedule works reasonably well for monitoring shifts in
substance use, as well as for most other purposes; however, the commit-
tee notes that follow-up surveys conducted on an annual basis would
provide greater detail in tracking changes in propensity, time of enlist-
ment, and other related factors.
Anything more frequent than annual surveys would probably be
burdensome to many respondents, and in any case seems unnecessary for
monitoring purposes. Mailed self-completed questionnaires, accompanied
by a modest "honorarium" check and coupled with phone follow-up
prompts when necessary, can produce reasonably high response rates
OCR for page 57
MONITORING YOUTH TRENDS
57
somewhat more demanding than an 11th grade survey only (Plan A); per-
school costs would thus be higher for Plan B than for Plan A, but certainly
much less than double. One consideration affecting logistics and costs is
whether identical questionnaire forms would be used for the 11th and
12th grade surveys (in Plan B). We see little need for separate forms
(given that a single question can be used to identify respondents' grade
level, and such a question is a necessary control in any case). The cost and
logistical advantages of limiting to one set of forms are evident.
Response rates. Plan B would produce a better sampling of young
people age 17-18 (i.e., 12th graders) than would Plan A. Under Plan A, the
data from those ages 17-18 would be obtained from mail follow-up sur-
veys of those surveyed as 11th graders a year earlier; even with the best
efforts at tracking target respondents and securing their participation,
roughly one-quarter to one-third of those targeted are likely to be lost. In
contrast, under Plan B data from those ages 17-18 would be obtained by
in-school surveys of 12th graders; losses in student participation would
be much smaller. (The MTF experience in recent years is that 12 percent in
the 12th grade sample was lost, primarily due to absence; the resulting
biases can be corrected, at least in part, by using special weights that take
account of the rates of recent absenteeism reported by students who did
participate in the survey.)
Follow-up participation rates. We noted above that substantial portions
of target samples are lost in mail follow-up surveys. That would occur for
the follow-ups in both Plan A and Plan B; however, there is room for
subtle differences between the two plans. On one hand, the two-year
intervals between surveys in Plan B would presumably result in lower
respondent burden than the one-year intervals in Plan A, and that might
contribute to higher response rates in Plan B. On the other hand, the two-
year intervals permit more time for respondents to become lost to the
researchers and perhaps also to lose interest, and that might contribute to
lower response rates in Plan B. We are not sure how these countervailing
tendencies would play out; they might simply cancel each other. In any
case, we believe that with careful efforts to track panel respondents
(including some form of "keep in touch" mailing no less frequently than
once a year), coupled with some modest financial incentive and an effort
to keep the questionnaire content interesting and not overly burdensome,
the differences in follow-up participation between Plan A and Plan B are
not likely to be large.
Numbers of cases versus amounts of data per case. Let us for the moment
make the simple assumptions that Plan A would involve surveying 10,000
OCR for page 58
58
EVALUATING MILITARY ADVERTISING AND RECRUITING
11th grade students each year and then follow a subset (say, a target
sample of 3,000) of each such sample for five or six annual follow-up
surveys, whereas Plan B would survey 10,000 11th grade students plus
10,000 12th grade students each year and then follow a subset (again,
3,000) of each for three biennial follow-up surveys. Under these assump-
tions, Plan A would yield roughly half as many total respondents as
Plan B. but it would involve nearly twice as many data collections for
each. Once fully under way, each plan would provide appreciable num-
bers of cases for any given year and any age within the range specified-
that is, roughly age 16 to age 23; moreover, except for the 12th grade data,
the numbers of cases at any particular age or year would be roughly
equal. But this would be accomplished differently, with Plan B providing
about twice the total number of individuals but Plan A providing more
fine-grained detail for each.
For panel analyses, the two-year intervals would probably provide
sufficient detail for most purposes. Moreover, key questions, such as those
about entering and leaving military service, could be asked with specific
dates (to the nearest month, etc.) and thus provide nearly as much useable
detail with a biennial follow-up sequence as with an annual sequence.
Accordingly, the committee judges that the loss of fine-grained detail in
the panel data under Plan B is not a severe handicap.
The advantage of having twice as many total cases under Plan B than
Plan A is, however, considerable. The opportunities for cross-validation,
exploration of rare subgroups, and other analytic improvements are
potentially quite valuable, given that sufficient resources are provided for
extensive analyses of the data.
Cost considerations. We have mentioned some of the cost consider-
ations above, but here we focus on them specifically. The first point to be
made is that costs will be determined by a great many factors, most of
which we cannot (and should not) try to specify here. What we can do is
focus on a few of the trade-offs and make some broad observations about
cost considerations. Perhaps the broadest observation to be made is that
we cannot with confidence estimate which plan would cost more than the
other. Clearly, as noted earlier, if 11th and 12th grade students were sur-
veyed in the same schools at the same times, there would be considerable
cost savings quite possibly the Plan B school surveys would cost only
half again what the Plan A surveys cost (i.e., the 12th grade survey might
be added at half price).
One could consider that 12th grade in-school survey cost as being
weighed against the cost of a mail follow-up of the 11th grade sample (in
Plan A) one year later. It is hard to know which would be more expensive;
OCR for page 59
MONITORING YOUTH TRENDS
59
much would depend on those devilish details. Depending on how many
follow-ups were used in each, the follow-up costs might be fairly similar
under the two plans. If Plan A involved six annual follow-ups and Plan B
involved three biennial follow-ups of approximately twice as many people
(e.g., targeting 3,000 each from the 11th and 12th grade in-school survey
samples), the total numbers of targeted follow-ups would be identical
and the costs would be quite similar. The Plan B follow-ups would be a bit
more expensive because of the extra mailings needed to keep track of
respondents in the off years. It should be noted that in the illustration
above there is an extra year of data under Plan B; specifically, the 12th
graders followed up a third time would have modal ages of 23-24, whereas
the 11th graders followed up a sixth time (under Plan A) would be age 22-23.
If the third follow-up of 12th graders were eliminated in Plan B. the over-
all cost comparisons would be more similar; however, there would be less
complete data available on college completion (and possible enlistment
afterward) if only two biennial follow-ups were used for 12th graders.
SURVEY CONTENT
If a major purpose of the survey is to identify the determinants of
propensity or to track changes in propensity and its determinants over
time, then, at a minimum, it will be necessary to assess propensity, atti-
tudes, perceived norms, self-efficacy, and the behavioral, normative, and
control beliefs underlying attitudes, norms, and self-efficacy. These clearly
should comprise the core questions in any survey design.
There are now fairly standardized instruments for assessing these
variables (see e.g., Fishbein et al., 2001; Ajzen and Fishbein, 1980~. How-
ever, as we shall see below, although it's possible to develop measures of
propensity (or intention), attitude, perceived norms, and self-efficacy
without going directly to the population of interest, one must go to that
population to identify the salient behavioral, normative, and control
beliefs underlying these variables. Thus, formative research is necessary
prior to developing a fixed-item survey instrument designed to assess
underlying beliefs.
Propensity, Attitudes, Perceived Norms, and Self-Efficacy
Propensity/Intention: Propensity (or intention) is typically measured
by asking a person how likely it is that he or she will (or will not) engage
in the behavior in question. For example, the respondent could be asked
to indicate the extent to which it is likely or unlikely that:
OCR for page 60
60
EVALUATING MILITARY ADVERTISING AND RECRUITING
I will join the military (Army/Navy/Air Force/Marine Corps) sometime in the
next N (months, years)
extremely unlikely : : : : extremely likely
Alternatively, they could be asked whether they "strongly agree," "agree,"
"neither agree nor disagree," "disagree," or "strongly disagree" with the
statement.
Attitude: Attitude refers to a person's overall feeling of favorable-
ness or unfavorableness toward performing the behavior in question.
Although there are many ways to assess attitude, the most commonly
used instrument to assess attitude is the semantic differential (Osgood,
Suci, and Tannenbaum, 1975~. For example, respondents could be asked
to indicate whether:
My joining the military (Army/Navy/Air Force/Marine Corps) sometime in the
next N months would be:
good
wise
pleasant
enjoyable
. .
bad
foolish
unpleasant
unenjoyable
Perceived Injunctive Norm: The perceived injunctive (or subjective)
norm refers to a person's belief that their important others think they
should or should not engage in the behavior in question. Thus, for exam-
ple, respondents could be asked to indicate the degree to which they
agree or disagree (or think it's likely or unlikely) that:
Most people who are important to me think I should join the military (Army/
Navy/Air Force/Marine Corps) sometime in the next N months.
strongly agree
agree
neither agree nor disagree
disagree
strongly disagree
Perceived Descriptive Norm: The perceived descriptive norm is one's
perception of what important others are actually doing vis-a-vis the behavior
in question. Thus it would be important to ask questions such as:
How many people like you will join the military (Army/Navy/Air Force/Marine
Corps) in the next N months?
none
very few
some
OCR for page 61
MONITORING YOUTH TRENDS
or:
and:
almost all
all
Out of 100 people like you, how many will join the military (Army/Navy/Air
Force/Marine Corps) in the next N months?
How many people do you personally know who have been in or are now in
the military?
61
Self-Efficacy/Perceived Control: Self-efficacy and perceived behav-
ioral control refer to one's belief that he or she has the necessary skills and
abilities to perform the behavior in question, even under a number of
difficult circumstances. That is, it refers to the perception that one could
perform the behavior if one "really wanted to." Items to measure this
either could use the semantic differential format. For example:
My joining the military (Army/Navy/Air Force/Marine Corps) in the next N
months is:
up to me
under my control
or they could be put in terms of a certainty question, such as:
not up to me
not under my control
How certain are you that, if you really wanted to, you could join the military
(Army/Navy/Air Force/Marine Corps) in the next N months?
certain I cannot : : : : certain I can
Underlying Behavioral, Normative, and Control Beliefs
Generally speaking, there are also standardized items that can be
used for assessing behavioral, normative, and control beliefs. For example,
behavioral beliefs are usually measured with items using the following
format.
My performing (Behavior X) will lead to/prevent (Outcome Y).
extremely unlikely
extremely likely
Injunctive normative beliefs are usually measured with items using the
following format:
OCR for page 62
62
EVALUATING MILITARY ADVERTISING AND RECRUITING
(Referent A) thinks I should perform (Behavior X)
extremely unlikely : : : : extremely likely
Descriptive normative beliefs are usually measured with items like:
(Referent A) performed or is currently performing (Behavior X).
yes no
Efficacy beliefs are usually measured with items using the following for-
mat:
How certain are you that, if you really wanted to, you could perform
(Behavior X), even if (Barrier A) were present?
certain I cannot : : : : certain I can
What the above illustrations should make clear is that, prior to devel-
oping a fixed-item instrument to assess the beliefs that underlie attitudes,
norms, and self-efficacy, one must first conduct formative research to
identify the outcomes, referents, and barriers that are salient for the popu-
lation in question.
Thus one could go to a small sample of the population in question
and ask the following open-ended questions:
To Identify Salient Outcomes:
- What do you see as the advantages of your joining the military (or a
particular Service) in the next N months? That is, what are the good things that
would happen if you joined the military in the next N months?
- What do you see as the disadvantages of your joining the military (or a
particular Service) in the next N months? That is, what are the bad things that
would happen if you joined the military in the next N months?
- What else comes to mind when you think about joining the military (or a
particular Service) in the next N months?
To Identify Relevant Referents (and Important Others):
- Please list those people who would approve of your joining the military in
the next N months.
- Please list those people who would disapprove of your joining the military
in the next N months.
- Please list any other people you would talk to or whose opinions you would
consider if you were trying to decide whether or not to join the military in the next
N months.
OCR for page 63
MONITORING YOUTH TRENDS
63
To Identify Salient Barriers and Facilitators:
- Please list those things that would facilitate or make it easy for you to join
the military in the next N months.
- Please list those things that would prevent or make it hard for you to join
the military in the next N months.
Responses to the above questions can be content analyzed and the
most frequently mentioned (i.e., the most salient) outcomes, referents,
barriers, and facilitators can be identified. This information can then be
used to develop fixed-item survey questions to assess underlying behav-
ioral, normative, and control or self-efficacy beliefs. This type of formative
research is also critical as a first step in developing advertising strategy.
Outcome Evaluation
In addition to assessing behavioral beliefs (or outcome expectancies),
it is also necessary to assess the evaluation of the salient outcomes. Most
often this is done as follows:
(Outcome A) is:
good
: bad
However, with YATS, rather than evaluating outcomes, respondents
were asked to indicate the extent to which a given outcome was impor-
tant to them. Thus it's necessary to consider whether to assess impor-
tance, or value, or both.5
Similarly, in assessing behavioral beliefs, should one simply ask if
joining the military (or specific Service) will lead to each of the outcomes,
or should a YATS-type question be retained that asks whether a given
outcome is more likely to be obtained from the military or from a civilian
job? Moreover, if one takes this comparative approach, is the appropriate
comparison military versus civilian or military versus college? Irrespec-
tive of the answer to this question, we would recommend that the core set
of questions includes assessment of both propensity (or intentions) to
continue one's education (e.g., go to college) and to join the civilian work-
force (e.g., get a job).
Finally, YATS assessed beliefs about each Service as well as the mili-
tary in general. Thus another question that must be addressed is whether
Because there were very few changes in ratings of importance over time, and because,
with few exceptions, judgments of importance were not related to propensity, we recom-
mended dropping importance in our earlier letter report National Research Council, 2000~.
OCR for page 64
64
EVALUATING MILITARY ADVERTISING AND RECRUITING
a survey should ask about the military or about each Service separately,
or both? Similarly, should it include the reserve forces?
Distal Variables
Distal variables may be related to propensity, but theoretically they
are assumed to exert their influence indirectly by influencing underlying
behavioral, normative, and control beliefs. For example, gender differ-
ences in propensity should be explained by finding that men and women
hold different behavioral, normative, or control beliefs about joining the
military. Similarly, the finding that young adults from the South are more
inclined to join the military than those from the North should be related
to differences in the beliefs held by young adults in these two geographic
areas. Thus an important question to consider is what distal variables
should be assessed. First, it seems reasonable to consider demographic
variables that are known to be related to enlistment. These would include
such variables as age, gender, ethnicity, socioeconomic status, education,
geographic location (region/urban-rural/state) and employment status.
Second, as pointed out earlier, particularly with respect to youth, a
very important potential influence on propensity is the image (or proto-
type) that one has of the kind of person who pursues a given choice
option (e.g., the image of the kind of person who enlists in the military).
Not only is it important to assess the prototype per se, but of equal inter-
est is the extent to which a person's own self-concept maps onto (i.e., is
consistent or inconsistent with) the prototype. As a result, we recommend
that assessments of both prototype and self-concept be included in the set
of core items comprising the survey instrument. Specifically, participants
could be asked to rate "a person who enlists in the military" as well as
one's self on a series of semantic differential scales, such as: "wise/foolish,"
"aggressive/timid," "works well with others/works best alone," "strong/
weak," etc. Formative research would be necessary to identify the scales
most appropriate for assessing a military prototype. Finally, it may also
be useful consider other distal variables, such as attitudes (e.g., toward
the military, education, or war), as well as personality or individual dif-
ference variables, such as intelligence and sensation seeking (Zuckerman,
1979; Palmgreen et al., 1995)
The above considerations are focused on survey content. We have
tried to identify the types of questions that are needed to identify and
monitor changes in propensity and its determinants. The data obtained
from these questions provide the kinds of information necessary for devel-
oping effective media campaigns as well as for evaluating the effective-
ness of advertising (or other types of interventions) designed to increase
propensity. However, as indicated above, there are many other issues
OCR for page 65
MONITORING YOUTH TRENDS
65
that need to be addressed in designing survey research. For example, in
order to evaluate the effectiveness of various advertising campaigns or
changes in recruitment policy, it is necessary to obtain data at regular
intervals over time.
We have recommended the use of a cohort-based sequential sample
survey with a longitudinal component that in our view will provide the
information necessary for both tracking propensity and its determinants
and for developing more effective communications to increase and main-
tain the pool of youth with the propensity to join the military. Moreover,
by following the cohort on a regular basis, it will be possible, at least in
part, to evaluate the effects of current events, as well as changes in adver-
tising and recruitment policies. At the same time, however, we recognize
that other data need to be obtained in order to fully evaluate advertising
effectiveness and other recruitment initiatives. For this reason, in our
earlier letter report on YATS, we recommended that "a portfolio of sur-
veys at different time intervals replace the current annual YATS adminis-
tration" (National Research Council, 2000~. Let us thus briefly consider
two other possible surveys.
Influencer Surveys
As noted earlier, there is considerable evidence that the decision to
join the military is strongly influenced by other people. While much of
this influence should be assessed through measures of injunctive and
descriptive norms, it is important to recognize that these normative beliefs
are usually quite consistent with reality. One usually is quite accurate in
one's beliefs about the normative proscriptions and behaviors of relevant
others. Thus, it may sometimes be necessary to change the beliefs and
attitudes of influencers. In order to do this, however, one must first under-
stand why influencers (parents, teachers, friends, etc.) support or oppose
joining the military, as well as why they are (or are not) inclined to recom-
mend military service. Since this is a totally different target audience, a
separate influencer survey is recommended.
It is important, however, to distinguish between two roles of an influ-
encer. On one hand, an influencer can encourage/discourage or support/
oppose enlistment (i.e., as injunctive normative influences see e.g.,
National Research Council, 2003; Rutter, 1980~. On the other hand,
influencers may be viewed as transmitters or evaluators of advertising or
other types of interventions. They may influence and shape the way a
person evaluates a given advertisement, advertising campaign, or recruit-
ment policy (Hornik, 1997~. Considerations such as these suggest a sepa-
rate survey to more precisely track exposure to, and to evaluate the
effectiveness of, advertising campaigns or other policy changes.
OCR for page 66
66
EVALUATING MILITARY ADVERTISING AND RECRUITING
Exposure to Media and/or Recruiters
Although there are many ways to monitor exposure, the committee
recommends that, if at all possible, respondents be presented with current
TV/print/radio/Internet ads (plus one or more dummy ads) and be asked
whether they saw or read it and, if so, how many times? This can easily be
done if the survey instrument is computer assisted or presented on the
Internet.6 In addition to these subjective estimates of exposure, we would
also recommend that gross rating points, as well as other indices of time
or space purchased, be tracked.
Ad evaluation: For each of the ads to which the respondent reports
exposure, a series of questions such as the following could be asked:
Did you believe the ad? yes no
Did the ad tell you something you didn't already know?
yes no
Was the ad appropriate for you (or people like you)? yes no
Was the ad interesting? yes no
Did you like or dislike the ad? like dislike
To assess indirect paths of effect: For each ad (whether or not the
respondent reported exposure), a series of questions such as the follow-
ing could be asked:
Which of the following people have you talked to about this ad?
Provide respondents with a list of influencers (based on open-ended
surveys) as well as "none" and "other (please specify)" alternatives and
ask them to check all that apply.
For each influencer checked:
Did (Referent A) like or dislike the ad?
To assess interpersonal contacts: In addition to asking whether the
respondent discussed ads (or policies) with various influencers, it would
also be important to know if they discussed joining the military with
these others. Thus respondents' could also be asked such things as:
6As indicated above, because of changing technology, it is possible that follow-ups of the
cohort sample could eventually be done using the Internet. If this were the case, tracking
could also be accomplished as part of a cohort-based sequential sample survey design.
OCR for page 67
MONITORING YOUTH TRENDS
Who have you talked to about joining the military? Check all that apply.
surveys).
67
Provide respondents with a list of influencers (based on open-ended
For each influencer checked:
Did Referent A support or oppose your join
ing? support oppose
INTERACTION OF PURPOSE, CONTENT, AND DESIGN
In this chapter, we have recommended a survey design and a set of
core questions to assess and monitor changes in propensity and its deter-
minants. We have also recognized that influencer and advertising track-
ing surveys are also necessary to gain a more complete understanding of
advertising effectiveness and the role that advertising, recruiters, and
other influencers play in the recruitment process. We recommend that
there should be at least two types of surveys: an annual or semiannual
cohort-based sequential sample survey with a longitudinal component to
monitor changes in, and provide an in-depth understanding of, the deter-
minants of propensity. In addition, this survey should include a more
general set of questions about exposure to media and interactions with
recruiters and other influencers. In different years (or at different times),
questions about more distal attitudes and values could be assessed.
While such an annual or semiannual survey should address most of
the key questions concerning propensity and recruitment, assessments of
whether one has been exposed to a current advertising campaign, whether
one has talked to others about that campaign, or whether one has talked
to a recruiter will clearly vary as a function of advertising expenditure
and military policies concerning incentives as well as the number and
placement of recruiters. Thus, we recommend a brief, continuous track-
ing survey to assess exposure to specific events, advertisements (or other
recruitment policy changes), and the extent to which the respondent
engaged in discussions with influencers about the campaign or about
joining the military. This survey should also monitor respondents' evalua-
tions of the ads they were exposed to, as well as provide data to allow one
to track changes in propensity, attitudes, norms, and self-efficacy over
time. We recommend that separate studies (or surveys) be conducted to
address specific populations, such as influencers and younger populations.
Representative terms from entire chapter:
youth trends
