Click for next page ( 10


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 9
1 ~ ntroduction During the late l990s, the armed forces began having difficulty meeting recruiting targets, particularly for highly qualified recruits youth with high school diplomas and above-average aptitudes who are essential for an effective fighting force. Recruiting problems were most severe during 1998 and 1999, when most of the Services experienced accession shortfalls. These shortfalls were especially alarming to military planners, given the major force reductions of the l990s and the lower accession requirements that followed (Sellman, 1999~. In 1999, at the request of the U.S. Department of Defense (DoD), the National Research Council formed the Committee on the Youth Popula- tion and Military Recruitment. During Phase I of its study, the committee was asked to examine long-term trends in the youth population and evalu- ate policy options that could improve the propensity for and enlistment in the Services. In its Phase I report, Attitudes, Aptitudes, and Aspirations of American Youth, the committee confirmed the decline in propensity for military service among youth and identified several correlates, especially the increasing trend in college enrollments. The report made specific recommendations about changing advertising programs and recruiting policies to improve propensity and enlistment. During the earlier phase, the committee observed that military re- search on advertising and recruiting programs is often opportunistic and fragmented, lacking coordination and long-term objectives. In this report, the committee aims to fill this gap by proposing a comprehensive evalua- tion framework to assist and improve research on recruiting policies and advertising programs. This report contains results, conclusions, and rec- ommendations from the committee's Phase II study. 9

OCR for page 9
10 EVALUATING MILITARY ADVERTISING AND RECRUITING PURPOSE OF THE STUDY The primary objective of this study is to help DoD improve its re- search on advertising and recruiting policies. It is anticipated that in the coming decade DoD will field and test new advertising and recruiting initiatives designed to improve the recruiting outlook and help avoid the shortfalls of the past decade. In order to discover the most promising policies, in the committee's view the department needs a comprehensive research and evaluation strategy based on sound research principles that will ensure valid, reliable, and relevant results. Good research should begin with developing a clear statement of the problems to be studied, because the nature of the policy questions influ- ences and shapes the most appropriate research designs. While a given research question can be studied using a variety of approaches and tech- niques, some research designs in our view are more promising than others for particular research questions. For example, focus groups are most useful when determining what a target audience finds attractive about a program, whereas econometric methods are appropriate when studying the effects of an existing program on enlistments. Good research requires more than sound methodology; it must also be grounded in solid theory and valid concepts. In the case of recruiting and advertising policies, the most important theory concerns individual decision-making processes, particularly the factors than influence job and career decisions. The research program proposed here reviews and devel- ops the most relevant theories for enlistment decisions. Finally, advertising and recruiting research must produce results that are useful to military planners. Research results must be available in a timely manner to help planners make decisions among competing pro- posals and to assist during the programming and budgeting cycles of the department. In the committee's view, the research and evaluation studies proposed in the following chapters can meet these needs. The remainder of this introduction is an overview of issues and methods and proposes a framework for selecting the most appropriate method to address various types of research questions. . THE EVALUATION PROBLEMS The committee's work during Phase I led us to conclude that there are a number of critical problem areas or topics needing more intensive study. Some problems arise because of the need for ongoing, up-to-date infor- mation that can serve as early-warning indications of potential recruiting problems or that can point to areas in which improvements are needed. Other problem areas are important because they are central to improving

OCR for page 9
INTRODUCTION 11 the overall recruiting climate. We have selected six areas to be the central focus of this report. Monitoring Trends in Youth Attitudes, Values, and Propensity The issue of youth attitudes has, in fact, received considerable em- phasis from DoD. Youth attitudes, values, and propensity for military service were monitored by the Youth Attitude Tracking Studies (YATS) from 1975 to 1999. A new effort to survey youth attitudes on a quarterly basis began in 2001 as part of the Defense Marketing Research program (Sellman, 2001~. Further monitoring of youth attitudes is important for several reasons. First, this information can be used as leading indicators of future changes in enlistment behaviors. For example, YATS documented a downturn in youth propensity that presaged the 1998-1999 recruiting shortfalls by at least six years. Second, the YATS methodology was not optimal, as was documented in the committee's letter report of tune 2000. The once-per-year format hampers the assessment of effects on propensity arising from specific events, such as the Persian Gulf and the latest Iraq wars. In addition, YATS administered different attitude and value questions to different subsamples of youth at different times, making it very difficult to analyze the relationship between attitudes or values and enlistment propensity. Finally, it is not clear that YATS adequately covered the domain of values and attitudes that influence career decisions. DoD should continue to place a high priority on studies of short- and long-term trends in youth attitudes, values, and propensity. The new Defense Marketing Research program (Office of Assistant Secretary of Defense [Force Management Policy], 2000) has improved the frequency of administration, but issues remain regarding the collection of trend data on key values and attitudes relevant to military service. The methodology used in these studies should ensure timely, reliable, and relevant infor- mation that can be used for anticipating potential problems in enlistment supply. Developing Effective Advertising Themes to Increase Youth Propensity The committee's Phase I report concluded that current advertising strategies and themes are not designed to maintain a base level of propen- sity, and this may be contributing to the decline in youth propensity for military service in light of alternatives (e.g., college, employment).

OCR for page 9
12 EVALUATING MILITARY ADVERTISING AND RECRUITING Accordingly, one of our most important recommendations was to develop effective advertising themes to help increase the propensity of youth for military service. Solving this problem requires two types of research. The first is devel- opment of promising advertising themes using generative research tech- niques; the second is thoroughly and rigorously testing the market impact of the selected themes. Again, the Services and DoD have conducted research in this area, but many improvements can be made in the meth- odologies applied to date, and these improvements can have a real impact on the development of effective advertising themes and message delivery strategies. Optimal Levels of Recruiting Programs and Resources The Services have many existing policies and programs designed to maintain adequate levels of enlistment supply. These include traditional and on-line advertising, various incentive packages (education benefits and enlistment bonuses), and number and location of recruiters. The policy problem is how to allocate these resources to provide effective and efficient approaches for maintaining a given supply of highly qualified recruits. Results of optimal mix studies are extremely useful for allocating resources among various existing programs during the budget planning cycles. While the Services have conducted some research in this area, many existing studies have not adopted the strongest research designs or incorporated the most reliable data. Improving the quality and reliability ~ en 1 1 1 1 r- ~ rr 1 1 ~ 1 en of this research would nave s~gn~cant payoff In helping to provide the most efficient mix and level of existing recruiting resources. Optimal Investment in loins and Service Advertising The timing and level of advertising are somewhat related to both advertising planning and optimizing the levels of recruit programs and resources, but we discuss them separately to emphasize two important areas of research that have not been adequately studied to date. First is the issue of level the minimum level of advertising needed to maintain youth propensity even during periods when enlistment supply is good. Service enlistment goals can fluctuate from year to year for vari- ous policy reasons, and enlistment supply is also influenced by changes in external factors (e.g., unemployment). During good supply years, mili- tary planners often reduce recruit advertising below levels necessary for

OCR for page 9
INTRODUCTION 13 maintaining the propensity base, thereby creating an advertising vacuum that lowers propensity and supply in difficult years. The second issue is the question of the optimal combination of joint (DoD-wide) and Service-specific advertising. Some advertising themes, particularly those designed to maintain or increase the base level of pro- pensity (e.g., intrinsic values such as duty to country), may be accom- plished more efficiently as a joint program, while other advertising themes (e.g., specific benefits of serving in a particular Service) are more appro- priate for Service-specific programs. What we do not know, because of a paucity of research, is what combination of joint and Service advertising would be optimal for meeting both types of goals. Improving Enlistment Incentives The Services offer a wide assortment of enlistment incentives at the present time, including benefits for college education and enlistment bonuses for particular military jobs. For the most part, these incentives are similar to those introduced at the inception of the All Volunteer Force in 1974. Given the many competing alternatives facing youth today, par- ticularly the increasingly popular college option, there may be other types of incentives that would enhance enlistment supply. For example, some Services are experimenting with programs that tie military duty more closely to college education, and other incentives being considered in- volve changing the length of active-duty and reserve commitments with the potential for enrollment in civilian national service. Research on new types of incentives, or new combinations of incentives, is very important to meet the increasing competition from college and other work oppor- tunities. Performance Management of Recruiters Existing research has been very suggestive, but not definitive, about the impact of recruiter organization and performance on enlistment sup- ply. By recruiter organization we mean such policies and practices as the number of recruiters, their geographic location, how they are selected from and returned to other military units, how they are trained, their pay and benefits, and the setting of recruiting goals or targets. While there is some existing research on recruiter organization and performance, com- prehensive studies do not exist. There is potential for substantial payoffs from research that focuses on how the performance management of recruiters can increase enlistment supply in a very cost-effective manner.

OCR for page 9
14 EVALUATING MILITARY ADVERTISING AND RECRUITING THE EVALUATION FRAMEWORK Our evaluation framework is based on the fundamental notion that different research designs are suited to answer different types of research questions. What appear to be disagreements about the relative merits of econometric models versus experiments versus survey methods tend to diminish if the question is framed as "What research approaches are best suited to answering the following research question?" rather than "What is the best approach to doing research?" We outline a number of common types of research questions that emerge in the context of military recruiting. Note that this is not intended as a general or complete taxonomy of research questions; other types of questions may arise in other research contexts. The research questions are placed in a two-dimensional framework (Table 1-1~. The first dimension indicates whether the research question focuses on an existing incentive, program, or activity or a proposed new incentive, program, or activity (hereafter summarized generally as "pro- gram"~. For example, an existing program might be a current advertising campaign, while a new program might be a proposed new enlistment . . ncenhve. The second dimension differentiates among three types of assess- ments of new or existing programs. The first type focuses on settings in which the outcome variable of interest is not specified a priori. It addresses what a target audience sees as attractive and unattractive features of a current or proposed program. The second type of assessment focuses on specific attitudinal or behavioral intent variables, such as one's perceived benefits of military service or one's intention to enlist. It addresses the effect of existing or proposed programs on attitudes or behavioral inten- tions. The third type of assessment focuses on actual behavioral outcomes, such as an enlistment decision or contact with a recruiter. It addresses the effect of existing or proposed programs on a behavior of interest. Each of the cells in this two-dimensional framework lends itself to some research methods more than others. Some cells may preclude the use of certain methods. The framework identifies focus groups or open-ended surveys and interviews as the most viable method when the research question seeks opinions about and reactions to a current or proposed program. The use of such methods makes sense when the researcher has not specified in advance the attitudinal or behavioral variables of interest. The reactions obtained from individuals in focus groups may be completely unantici- pated by the researchers. Such a strategy is perhaps most generally useful in the early stages of considering a new program, although it may be usefully applied to existing programs, particularly those that were imple-

OCR for page 9
INTRODUCTION TABLE 1-1 Evaluation Framework 15 Outcome Not Specified A Priori Specific Attitudes or Behavioral Intentions Actual Behavior Question: "What does programs a target audience see as attractive or unattractive features of a program?" Method: focus groups; unstructured or open-ended surveys and interviews Existing programs Question: "What is the effect of a program on specified attitudes or behavioral intentions?" Method: surveys; experiments; . . quas~-exper~men~s Same as above Same as above Question: "What is the effect of a proposed new program on enlistment?" Method: experiments; . . quas~-exper~men~s Question: "What is the effect of an existing program on enlistment?" Method: econometric modeling mented without the use of systematic evaluation in their initial design and implementation. Chapter 5 offers further development and illustra- tion of the use of such qualitative research methods in the context of developing advertising campaigns. The framework identifies surveys as a useful method when the re- search question deals with the effect of existing or proposed programs on attitudinal or behavioral intent variables of interest. Survey questions can be administered in various media, from paper-and-pencil questionnaires to telephone interviews to administration via computer. The key distinc- tion with the previous category is that the researcher has clearly specified in advance the attitudinal or behavioral intent variable of interest. Thus if a portfolio of possible new recruiting incentives is being considered, and the researcher is interested in the effects of each on an individual's stated likelihood of enlistment, the use of survey methods would be particularly appropriate. Note that experimental and quasi-experimental designs can be usefully employed in such settings, with different groups of partici-

OCR for page 9
16 EVALUATING MILITARY ADVERTISING AND RECRUITING pants asked to respond via survey to different proposed incentives. Survey methods are also useful in monitoring whether existing programs continue to generate the same attitudinal reaction over time. It is often the case that the attitudinal or behavioral intent variables of interest are not of primary interest in and of themselves; they are of inter- est because of a documented link to actual behaviors of interest, such as enlistment. Because of this link to behavior, insight into the potential effects of programs can be obtained prior to actual program implementa- tion. Thus, for example, a wide array of different bundles of recruiting incentives could be presented to research participants to obtain insight into which is seen as most attractive. Note, though, that while informa- tion on the relative attractiveness of different options can be quite useful in deciding which ones to implement or to research more extensively, this method does not result in a point estimate of the effects of the program on enlistment (without research linking survey responses to actual enlist- ment, as discussed in Chapter 7~. Chapter 3 further develops and illus- trates the use of survey methods. The final two cells in the framework focus on behavior; in the military recruitment context, enlistment is the behavior of greatest interest. It is in research questions focusing on enlistment behavior that the distinction between the evaluation of new programs and the evaluation of existing programs becomes a critical issue. Econometric modeling methods are well suited to the evaluation of ongoing programs, as they are based on actual variations in a set of observable conditions and relating that varia- tion to behavioral outcomes of interest. For example, the effects of adver- tising expenditures on enlistment can be examined by relating changes in expenditures over time to patterns of enlistment, statistically controlling for other relevant time-varying features. Chapters 5 and 6 further develop and illustrate econometric modeling. In contrast, determining the effects on enlistment of a new program prior to full-scale implementation or the differential effects of alternate possible new programs are not amenable to examination via methods relying on observed variation in the features of interest, because the programs have not yet been implemented. Thus approaches involving manipulation of features of interest are needed, namely, experimentation (manipulation with random assignment) or quasi-experimentation (manipulation without random assignment). For example, different advertising content could be used in different geographic regions to examine effects on enlistment. Chapters 4 and 7 further develop and illus- trate experimental and quasi-experimental approaches. While the framework presented here identifies broad types of re- search questions for which particular research methods are well suited, it is not our intent to draw bright-line distinctions among the methods.

OCR for page 9
INTRODUCTION 17 Another message of this volume is the relevance of multiple methods to many key issues in military recruiting. Chapters 6, 7, and 8 all illustrate areas in which multiple methods are useful. In addition, aspects of different methods can usefully be combined. For example, quasi-experimentation does not ensure the equivalence of conditions that are obtained via random assignment, and some of the quantitative tools of econometric modeling can be very useful in controlling for differences among conditions. In the next chapter we elaborate more fully on similarities and differ- ences between econometric and experimental approaches to research. Again, it is the nature of the research question that should drive the choice of research method or methods, rather than allegiance to any par- ticular methodological orthodoxy.