Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
6 Effectiveness of Labor Market Preparation Programs The YEDPA studies we reviewed represented three basic approaches to labor market preparation for youths: (1) career exploration programs, which usually provided information on occupational opportunities and requirements, work habits and attitudes, and job search techniques and sometimes included ability and interest testing; (2) basic educational skills training, usually remedial, which often included General Equivalency Diploma (GED) preparation; and (3) direct work experience, usually combined with some orientation to the "world of work." Most programs offered some combination of the first two approaches, and a few were designed as systematic variations of work and classroom instruction as tests of their relative effectiveness. The implicit long-term goal of many of the programs, especially the programs focused on out-of-school youths, was to increase the employment opportunities and economic self-sufficiency of youths. It was assumed that certain deficiencies--in work habits and attitudes, basic educa- tional skills, and understanding of job interests and options--were barriers for disadvantaged youths in achieving economic self-sufficiency and that the correction of these deficiencies would solve the problem. Correction or amelioration of these deficiencies, therefore, became the interim objectives of the programs, as well as steps toward the longer-term employment goal. Labor market preparation programs under YEDPA were provided to both in-school and out-of-school youths, but the majority served in-school youths. Unfortunately, most of the studies on the in-school projects did not meet the committee's criteria of scientific evidence and so are not included in this review. For example, we were not able to evaluate summer programs designed to maintain or improve educational skills of youths over the school vacation or the many career exploration programs offered as a supplement to regular school programs. In terms of numbers of participants, the programs that met our criteria for tIt is not entirely clear how some programs should be classified, particularly those that provided several alternative treatments (such as the AYES project). We decided to classify them here. 119
120 inclusion and are reviewed here largely involved out-of-school youths, both dropouts and graduates. Several of the reports reviewed by the committee indicated that the programs being evaluated, while initially designed to provide occupational skills training, were revised to offer more basic educational and prevocational skills. These changes were necessary because the youths enrolled lacked the skills required for more specific vocational training. PROGRAMS FOR OUT-OF-SCHOOL YOUTHS Programs serving out-of-school youths generally served both dropouts and graduates, though the relative proportions varied considerably across the programs. We reviewed four programs for out-of-school youths that we found deserving of mention: Alternative Youth Employment Strategies (AYES), Recruitment and Training Program (RTP) Career Exploration Program, Project STEADY (Special Training and Employment Assistance for Disadvantaged Youth), and the Job Corps Educational Improvement Effort (EIE). Table 6.1 details the characteristics of each of these programs; Table 6.2 details the research design and results of the evaluations of the programs. Alternative Youth Employment Strategies The Alternative Youth Employment Strategies (AYES) project was designed to test the efficacy of three alternative models for providing youth employment programs to a particularly high-risk, hard-to-reach group: unemployed, out-of-school (largely dropout) 16- to 21-year-olds, many referred by the juvenile and criminal justice systems. The model treatments were: (1) full-time work experience, with counseling and placement services; (2) full-time classroom instruction in basic educa- tional, vocational, or prevocational training, with counseling, ser- vices; and (3) a mixed model of part-time work, part-time training, counseling, and placement. AYES was implemented using random assignment at sites in three cities (New York City, Miami, and Albuquerque) and involved about 1,100 youths. The Vera Institute study of AYES was of particularly high quality. Although not free of problems, the research seemed to have been conducted as carefully as conditions permitted, given some problems of implementation. Its major finding was that a 26-week, full-time program that concentrated its services on high-risk youths enhanced their chances of securing full-time employment. Differences in employ- ment rates of approximately 10 percentage points were found between participants and a randomly selected control group approximately 8 months after program participation (see discussion of Career Explora- tion Program below). Positive effects were found at all three sites: the largest program effects were found in New York City and the smallest effects in Albuquerque. At the same time, the demographic nature of the samples varied considerably across the sites, making it
121 U) ·_' U] ,1 ~4 C: S4 o P~ ·e U] S o o o U] 1 o 1 o o V] o o ·_' a' a y ~: o Q ~g ~S E~ U] a, ._! U) o o ~ . - q~ o .. - s ~ . - U] .,, Ul o a' V CD E~ U] ~ a ._, ,~ ~ o cq P4 s o ~4 o k4 P~ a .= o ~ ~ o s 3 O a) ~ ~5 ,' ~ ` O . . . ~ ~ tI5 ^ S ·` Q ~ ~ · O >'~ I ~O Z Z ~ 5 ' ~ ~ 3 C 4~ ~ ~ O ~ ' ·O i, ~ ~ ~O Y o Q 03 u] u] C~ ~ · ~a4 ul ~ ~) a) u ° · · 0) ~ s ~4J Q tO ~ ~ : ~'- ~ ~ O O U] ', , 3 [4 Z ~ P4 ~; O L z m ~ U] .,, Ul U] s UD s 4J :' >' 3 ~ >, Q O O _I :^ :~^ {,q a) ~ 4J ~4~ ' - Q, ~ 3 -1 C ~ Ll[,) ~ O S~ O ~.,' O O ~ ~' - C ~ O ~' - Y ·a Q, Q~ k4Y a~ ~u~ ~ ~ ~ ~ ~ ~ ~ h C - - ~1 ~C ~ ~ ' - ~ S ~ 3 J~ Q := ~ ~ ~ JU Q 3 ~ O U] C ~ Ul -1 ~ -~ dP dg , - y ~,- ~ oP dP dP Ql O ~ ~ Ln ct O ~ ~ ~O ~ ~ ~ ~ ~ ~ ~ O a~ I_ I_ ~0 ns u) ~O ~ O ~ a~ rn ~ ~ tn t) O o o ~S U) Q5 ~ ~U] U] 3 Q5 ~ J~ Q~ O 1 ~ dP 1 dF d~ dP 1 ~:5 ~ ~ d~ ~ C~) ~1 ~S P~ P4 kD ~ ~ ~ ~ns ~ u~ Q, Q, ~ o 0 O S ~CS k0 0 CO _' ~O h S ~3 ~ ~ Q 1 ~1 1 ~ ~ ~ O 4') - · · · .L) · · · J~ - - Q · · · · 3 3 3 O O O 1 y ~ ~, - O S ~5 ~' a' ~-~ {0 ~ ~t ~4 ~ ~ ~ Q S C I U] O ~1 ~ ~ JU O tO JJ O J ~ C) L' aJ ~ 3 0 '- O ,0 C) S q~l O ~ 1 ~U] 3 Q~ J~ tI5 C .- ~ 44 ~ U C U] JJ ~3 ~ a) ~ 4~ u s 0 ' ~ Ll ' - ·. ~ ~ ~ ~ '- O C ~ ~ C Q ~ C C U] '- ~ k4 0 U2 .~1 a) ~0 U2 a) 0 4U 0 ~c ~ c 0 ~ ~ ~ 0 4~ JU J~ O ~ J~ ~ ~ "l c u? ~a~ ~ ~J~ ~ ~ 0 0 0 Q] 1 1 ~ Q ~ 3 C ~ ~ ~Y ~ ' C ~ ~ ~ ~ ~ ~ ~ _1 ~ ~ ~ ~ ~ O ~ Q ~ ' C ~ ~C ~ 3 ~ ~ ' 4J ~ ~ O ~ ~ 3 X ~ ~ ~ O ~ C ' - C ~O ~ U 3 ~ ' ~ C RS 3 ~ O ~ C ~ ~ 4U ~ ~ O C ~ ~ ~ ~ O ~ · 3 0 C ~ q~ L4 - ~ ~ ~ ~ O ~ O~ ' - ~ ~ ~ ~ C ~ ~ ' - 0w q~ '- ~ q~ JU ~ U] ra ~ ~S ~ ~ O · _~ U] 0) · · · ~ c Y c c 0 ttS ~l 0) ~ ~ c 3 = - ~ ~ ~ u, ~ ~n ~ ~ ~ ~ ~ ~ u >4~- ~0 ~ ~ o q~ a) ~ ~ ~- - ~ CD C s~ ~ s C 0 a1 ~ ~ ° E~ 3 U) U] 1 ~ >1 ~ ~ C O ~ ~C ~ O :) ~ O · - 1 S ~ U] J~ C U] O O ~ ~ ~ ~ ~ ~ 0) J~ ~J~ ~J 0) 0 3 Ul 0} J ~0) ·' Q ~ JU ~ ~ ~ c a' ~-' ~ u' ~ ~ a, ~ Q Q5k4 h aJ · - 0) ~1 ~ C O V 0 Q]a~ ~ ~ ~ · - 0 ~ s ~ ·-l ts~ U] O ~ ·- ~ ~ ~ O C Q ~J~ ~ C) C (d Q (24 ~ ~ ~ C ~ ~ Q O E~ I L C a' O P4 ~q~ a O ~E~C ~, ~C ~1 ~(0 ~0) 05 u] ~c H U] a~ a~ ~- -- ~; ~c 0 J~ E ~C a) ~1 a ~ ~ ' - ~ ~C ~ ~ ~ ~ ~ ~ C C ~O ~ - ·- ~ ~ ~ ~ C ~ ~ ~ ~ C ~ ~ ~ ~ ·- · - ~ h h ~ . - . - S ~ U] ._4 4U C ~ ~ O ~ 4~ ~ C O U] '= S ~ ~ O ~ c JU .<s ~ ~ ,' ~ a, ~1 ~ ~ ~ · - ~ ' - ~ ~O ~ ~ O ~ O k4 3 ~4 :~ U] ~ n~ O L~ ~4 0 0) p~ (~5 Q4 U] U] ~O ~ Q4 n5 a) O ~ ~ c ~ i4 L1 n5 X Ll ~m U) ~ ~ ~ ~- O ~ ~ u:-,_, c~ E, ~, r~ ~ ~, 0-E~ ~ <: a ~Q Cz: -/ ~ E~ ~ - ~a~ ~0 ~=; P~ ~
122 ~:5 s v a U] a ~r; o 8 s ~Q 1 o 1 o U) S~ o P4 o Q ~n 3 U] a) ~;U] C o ~C) _. ~q a) a' a' 1 U] O ~ ~n 0 a ·e U] S ~5 UO JJ V q~ O E s~ c O U] ~ o O ~ ~0 O s c: ~ ~ O O Q] ~c O P~ O ~-0 a) Y N C U) . V ·,1 ~ 4 U] C~ m ~ 0~ O E~ U] U] V ~ U] .,. O ~U~ V ~ ~U2 ~ ~ (,] ~ [,, ~ . - . - ~ O v ~ O ~ 3 n c ° ' 3 ~ k.' a' i, a) V (0 V 0) ~D ~ v ~ ~ ~ ~ v 4 ~ =~ s u ~4 ~ c a) v ~ ~ .- ~ .- . - n ~ a' Lb O O `~' a E ~ ~ so ~ ~ C o~ L) ~p, a' 4~ U2 Ll U] ~ 15 - O h u] 1 ~Q ~a' ~ :5 ~u~ E dP d ~E °~ ~P ~ ~ ~v ~ ~ ~ E ~ ~ ~ ~ ~ ~ E ~ v ~ U I' I! o O I' I' 0 ° '' I' ~ ° 11 11 C 0 11 11 (L ~ 4~ ~ ~ O ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ·~ ~ ~ ~ · s ·- H ~00 H `,~, ~c4 dP dP a, Q~ ~ oo ~5 ~n ON & 4~ 4 ~c ~ a) s ~ oP n E v 0 E E O °' ~ OE ~ E . - + v ~ O ~ ~Q ~ ~dP O I u~ ~ 4 ~0 ~ ~ ~ O ~ ~ ~ ~ ~ :, ~ c q~ ~ ~ S s . - .c ~ ~ ~ c u~ 0 0 1- ~ :~ E E 3 ~ Q o > U] V 1S ~ ~ C a) ~ U2 ~ co tJ) 10 .- C ~ ~ · ~ ~ 3 ~ O I O ~q 4~ ~I I ~O ~ U O D v c i) O O a c a ~ · - ~, · - V a) ~ C ~ C ~ ~ V ~ ~ 0 ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ c PJ v Q' ~ ~ ~zo zc ~ ~ ~ v ~r o l - o V ~ ~aJ O ~V ~ V ~ V 4. ~ ~ u) ~ ~1 ~o u) Q ~H ~
123 virtually impossible to distinguish the independent effects of ethnicity and site. Differences in outcomes were not accompanied by any changes in the measured attitudes and orientations reported on a series of tests developed by the Educational Testing Service (ETS) that were adminis- tered at the conclusion of the program. Nor did these differences in measured attitudes affect the types of jobs that participants obtained. Three interpretations are possible: (1) changes in reported orienta- tions toward employment are weakly related to changes in job-related behavior; (2) the tests were administered immediately after program completion (when attrition rates were relatively lower for both the experimental and control groups, 16 and 45 percent, respectively) rather than at 8 months postprogram, when the job placement comparisons were made (when attrition rates among the experimental and control groups were 31 and 42 percent, respectively); and (3) the tests them- selves may be of questionable validity in measuring the attitudes and knowledge they purport to assess. The Vera Institute study of AYES compared the relative effectiveness of the three treatment strategies at each site. No difference in the effects of the alternative treatment strategies could be discerned. In several other studies, similar null findings for alternative treatments were also found. Indeed, this is the one finding that was fairly robust throughout the studies of labor market preparation programs we reviewed. The only exception was the Special Project for Indochinese Youth. It appeared to show English-language training for Indochinese youths to be more effective if based in the classroom instead of in a job context. This null finding parallels a similar finding in educational research that shows little difference in the employment effects of vocational and general education. Three explanations are possible: (1) the types of instruction have equal effects; (2) students select the type of program best suited to their needs (a choice among the three types of programs was generally left to the individual), and effects appear equal because each type of instruction is provided to that group of students for whom it is best suited; and (3) sample sizes are too small to detect small differences in outcomes controlling for site, treatment, and other variables. RTP Career Exploration Program The Recruitment Training Program (RTP) Career Exploration Program provided summer instruction in career preparation to economically disadvantaged, unemployed out-of-school youths, both dropouts and graduates, in several cities. Services, including occupational information, job search information, and basic skills instruction, were provided to 16- to 21-year-olds during the summers of 1979 and 1980. The study of RTP shows positive effects of the program. Partici- pants showed larger gains on a variety of ETS-developed measures of job satisfaction and vocational aspiration than a randomly selected control group. In addition, participants were more likely to be employed full time at program completion than control group members, 89 percent
124 compared with 53 percent. The favorable full-time employment experience of participants persisted at the time of the 3- and 8-month follow-ups and even increased slightly: a difference of 7.5 percentage points at 3 months and of 8.2 percentage points at 8 months. The gains reported for this less intensive summer program were as large as the gains reported for the full 6-month AYES program studied by Vera. The findings could be attributed to the fact that the RTP Career Exploration Program addressed a less disadvantaged population: approximately 25 percent of treatment and control group members were enrolled in college at the 3- and 8-month follow-ups. It is also likely that the assignment of youths to participant and control groups in the Career Exploration Program was not strictly random--sizable differences in the two populations can be discerned on the basis of preprogram characteristics within sites, such as high school graduation, welfare recipiency, ethnicity, and age (20 to 30 percent of partici- pants and controls were 20-21 years of age). These considerations, combined with the fact that the analysis takes no explicit account of the program year or site, suggests that the findings in this case are not as reliable as those from the Vera study. Project STEADY Special Training and Employment Assistance for Disadvantaged Youth, Project STEADY, operated during the summer of 1980. Its purpose was to determine the feasibility and effectiveness of the employment service's local office efforts to increase the employment and employability of unemployed out-of-school (graduate and dropout) youths who had no further educational plans and no immediate employment prospects. Participants, whose ages ranged from 16 to 21 and who were otherwise eligible for the Summer Youth Employment Program, were required to participate 35 hours per week at the minimum wage {$3.10 per hour) for up to 12 weeks. Program activities included aptitude and achievement testing, counseling, labor market information, job search training, and referral to and placement in unsubsidized jobs. Ten sites were selected for the project. The nature of the treatment varied considerably across the sites; site directors had complete discretion in selecting those services, materials, and emphases they thought most appropriate. The evaluation of Project STEADY is based on data for approximately 600 participants and 400 controls at 3 months postprogram. Data from a longer term follow-up are not available. Personal characteristics of both controls and participants varied a great deal among the sites, although the initial assignments to control and participant groups within a site were random: program applicants were initially tested and then randomly assigned to participant and control groups. Attempts were made to make up for participant attrition by accepting control group individuals as participants. The author of the evaluation report had no information as to how control group members were selected to be participants. In addition, attrition in the 3-month follow-up period was significantly higher among control group members (about one-third) compared with participants (one-fifth).
125 Project STEADY employed two measures from the Standard Assessment System (SAS) as appropriate to both the target population and the program objectives: job-holding skills and job-seeking skills. Both a pretest and a posttest consisting of the two SAS measures were adminis- tered to the participants and controls. Performance outcome measures were used in a program completion survey administered to participants after 12 weeks of program participation and in a control group status survey administered to controls at the same time. In addition, performance outcome measures were used in follow-up surveys (3 months) after termination of program participation for participants and at the same time for controls. Information on individual characteristics of participants and controls was taken from the individual participant profile of SAS. There were relatively small numbers of participants and controls at any given site. Alternative statistical tests were used to gauge the effectiveness of Project STEADY. Statistically significant gains for participants relative to controls were found in both job-holding skills and job-seeking skills when sites were pooled. However, on an individual-site basis, only 2 of 10 sites revealed statistically significant gains for participants relative to controls in job-holding skills and job-seeking skills. At most of the remaining sites, the gains of participants relative to controls were not statistically significant. The 3-month follow-up survey revealed that when all sites were pooled, the percentage of participants who were employed full time exceeded that of the controls by a large and statistically significant amount: the adjusted probabilities of full-time employment are estimated to be 29 and 17 percent for participants and controls, respectively. On an individual-site basis, only the three sites with large samples exhibited statistically significant full-time employment differences, all in favor of participants. The percentage of par- ticipants reporting being employed in jobs of higher status exceeded that of the controls by a statistically significant amount when pooling all sites. On an individual-site basis, however, there were no statistically significant differences in job status. There were also no significant differences in earnings or in job satisfaction between participants and controls. After adjusting for heterogeneity between the participant and control groups at the 3-month survey, the evaluators found that of the 40 possible outcomes included, 10 yielded statistically significant differences, all in favor of participants (relative job status was no longer significantly different). These outcomes included full-time employment, financial contribution to one's family, two measures of future job quality, and getting along with one's family. Very little information pertaining to race, sex, or age differences in program gains is available from the evaluation of Project STEADY. A multiple regression analysis was conducted in which posttest scores were regressed on pretest scores and demographic characteristics. For all sites taken together, females experienced statistically significant smaller gains than males in both job-holding and job-seeking skills. Practically, however, these meant small actual differences in scores.
126 On the other hand, there were no statistically significant differences by race and age. There were certain implementation problems associated with Project STEADY as a demonstration project that need not occur under a permanent program. Due to the brief start-up time, there was difficulty in recruiting participants. Lack of time and resources caused difficulties regarding planning, curriculum development, and the necessary outreach to potential participants, as well as the creation of a true control group (as discussed above). Overall, the evidence indicates some positive effects of Project STEADY on the short-term employment prospects of youths. While the precision of the estimated gains is questionable because of data dif- ficulties, the qualitative effects can probably be accepted. However, statistical significance, where found, was typically the result of pooling the data across sites, and therefore we have questions about whether the evaluation of Project STEADY demonstrated that the program could have a substantial positive impact on a significant number of young people facing employment difficulties. Job Corps Educational Improvement Effort The purpose of the Educational Improvement Effort (EIE) was to improve the educational offerings of Job Corps to provide corps members with the best opportunities for learning at all levels. To meet these objectives, new or revised curricula were developed for basic skills in reading and mathematics and high school level skills in all areas. Programs were tested in an experimental design to provide information concerning their effects on educational progress and process (Argento et al., 1982~. The Job Corps Educational Improvement Effort (EIE) is noteworthy in its attempt to use random assignment of Job Corps par- ticipants to treatment and control groups to test alternative teaching techniques. The programs evaluated included: (1) a reading curriculum that used materials revised from earlier Job Corps reading programs; (2) a calculator mathematics program that provided instruction and experience in the use of hand-held calculators; (3) a reading program using "peer aides" to help instructors in the reading program deal with the instruc- tional needs of their students; (4) a program offering participants the opportunity to obtain a regular high school diploma rather than a GED; (5) a GED program that used televised instruction; (6) a computer- assisted education program using the Comprehensive Computer Program to help students with reading and mathematics; (7) the PLATO system of computer-assisted instruction; (8) two curriculums to help students with learning disabilities--one developed by the University of Florida and the second by the University of Kentucky; and (9) two curriculums designed to improve the "employability skills" of participants, the Adkins Employability Skills Series Program and the American Preparatory Institute Program.
127 Over 7,000 Job Corps members in 11 centers took part in one or more of these programs. Because participants could enter or leave the program at any time, attrition in posttest data was substantial for many treatments and sites: for example, 74 percent among controls in the mathematics component of the Comprehensive Computer Program model (Argento et al., 1982:Table 1.4-1~. The assignment of participants to treatment and control groups is described as follows (Argento et al., 1982:1-5~: Potential participants were randomly assigned to either the experimental group or the control group, to the extent possible. Unfortunately random selection was not always possible. In one large Job Corps Center, for example, a decision was made to place all students in the same vocational training area in the same educational classes. Thus all students interested in automobile mechanics were in one mathematics class, all those interested in nursing in another, and so on. With this system, it was impossible to maintain true randomization. . . . While the report is forthright about such problems, it does not present separate analyses for the "true" random assignments, and so it is not possible to estimate the biases that might have been introduced by such administrative decisions to abandon randomization at some sites. The key analysis performed for each of the programs uses preprogram to postprogram differences in Stanford Achievement Test (SAT) scores as the dependent variable in an analysis of covariance in which "treatment" is the independent factor and the covariates include sex, age, race/ethnicity, highest grade completed in school, hometown size, whether family receives welfare, and score on SAT pretest. The reported analyses do not consistently include all variables, apparently because a stepwise inclusion procedure was used. Gains in test scores are measured in "grade equivalent" years in order to gauge treatment effects on educational attainment. Thus, postprogram minus preprogram scores are divided by the number of hours in the program: a 100-hour program that raised performance by two grades would show a gain of 2/100, or .02. Gain scores are then adjusted for the covariates included in the analysis. Few of the treatments produced significant results; for those that do appear significant selectivity bias cannot be ruled out as an important factor. Thus, we did not find the evidence on the differ- ential effectiveness of the Job Corps EIE convincing. PROGRAMS FOR IN-SCHOOL YOUTHS This section discusses three programs that predominantly or exclusively served in-school youths: the Career Exploration Program, the School-to Work Transition program, and Project Redirection. Table 6.3 details program characteristics; Table 6.4 presents the research design and results of the program evaluations.
~n v .-. u) c) o ·e u] s :' o o o s u] 1 H o U] o Ll o ._l a ~: o Q . ~; E~ U) 4U .,' cn )J o ~ ~ o P4 4 o .,' s . - h Ul ,' Q. JU :3 ~n O ' - C' ~ E~ So U] ~ .,- ·- Ll O O )~ ~q P4 ~ SO S~ O ~ Ll O P4 4~ C) a) ·~_, O O :D ~a a' O .,' ~ ' ~ JJ ~ ~ . - ~ ~ ~ ~ 1 ~ O ~ ~ ~ 3 G. O ~ ~ U] O U) Ll ~ ~q O ~1 ~ ul ~q 1 Q' ~ - X O O a) q~ 128 O ~ ~ e ~ ~ 3 ~ ·= Ul·' · C · - ~ ~ N _I ~. Z ~ ~ c: ~ a' · - a' ~· ~ O S S:: 2 X ~· IJ_I · ~c a~ C ~ O · - ~ · ~ 0 ~ 0} 0 3 X <1) S -I ~ ~J ~ O ~ S ·- ~r1 ~ v ~ a) m ~ c~ `:4 ~ ~ ~ `: ~ - - ~ U] ~ U] ~ Ul 0) 0 s ~ ' ' . - u] 0 h ' .Q ~ ~-~ C ) GS S O O ~U' ~ (;s £ ~ CJ, := O ~ 0 3 ·' Ul O ~ ~ U] O q~ ~ ~ ~ ' ·' P: O ~ ~O ~ ~ ~ C S ~ ~ ~S ~ ~ ~ ~ O ~ ~ -/ ~ S ~ ~ ~ ~ ~ ~ ~ · C U) U] C) (.) tI5 t~1 :e U) ~ ~ ~ Z C~ 1 ~ V ° C 11 o i.4 O 3 C u' ~ a,, ~ ~ Y u, ~q ~ u' a' h C' 3 O C GS O L. S . - -S 0) o U) C~ o U] S C o C~ C a S ~q ' C ~4~ ~ U] ~U) ~ ~C C q~ ~- ~J ~ O ~=- - ~ C U] V ~ ~ Y ~ ~ U] S ~ ~q O ~ U] ~ s~ 4~ u, ~ ~4 0 u' a, ~ tO ~tq ~ ~O s a) ~ ~ ~ ~. - O ~O ·- V V ~5 [Q ~ ~ · - C U] V ~ U] ~ O ~ ~O ~V (1) 4 ~· - 4~ S 0) u ~S O ·- t: _I ~ ~·~ O q~ :D C ~ ~ ~ C C ~O ~C O ~:5 :5 P4 ~·- O o' S O C C S O Q~ V ~ aJ ~5 0 0 S V O `) S (d td V Q' V GS v 1 0 lIJ U] ~ ~ ~ ~ O ~ V V S ~ ~ ·- ~n X ~ ~ · - ~ ~ ~ s 0 cn u' ~ ~ ~ ~ ~ ~ I o~ m ~ ~ - a' v ~ ~ ~ ~c Qm 0 0 ~ 0 ~ u, ~ ~ ~s ~ ~n c O O ~ dP oP dp dP op O ~ 3 3 ~ Q~ tD C tO dP oP O S 1 ~ a~ a~ ~ C~ ~ S n5 s ~J 0 0 u~ :~ ·- ~ a~ a~ ~ V V ~ ~ ~ ~ ~ tn V ~ cr ~ ~ ~ ~ O s U] ~U] · - ~1 a) ~ 1 1 0 · ~ ~ - ~ ~1 · S ·~1 · - ~ ~ C ~ · C C S O H ~U] 2 U) 1 ~1 q~ 1 _I C a) · - 0 a Ul ~ Ul J~ C ~ ~ ~ ·~1 U] C O ul a U] p4 C ~ X 0 a S ~ ~ U] ~ ~ ~ Y 3 ~ O ~ ~ a) r, Q ;~ a' nj U ~ V O l5 ~ 4 ~ O a~ ~ ~ X U] ~n 1 a ~ C ~ V a' ~ c S C C >. a O U] ns Q _I C ~ ~V O .,1 ~ a., 4 ~ £ ~v ~ v ~ C a' ~4 C o v ~ .,, ~ O q~ · - O U] C ~1 O U) ~ O C ~ O S x a) v C 14 ~ · - ~ ~ O U eq ~ ~ O ~ ~ ~ ~ 3 ut a V C · - · - C Ul U] ·~4 C ~ ~ a) a) ,- s u) o a) c Q k4 ~ Q Q, O ~ V o ~I cr; 0) 0 ~1 V H ~ 0 ~a' ~: C C ~q ~ ~ O ~ ·~ ~ Ll O ~ ~ ~ S C E~ :^ ·-~ ' au ~ ~ ~ ~ ~ v 0 ~ ~ o ~ c' a~ ~ ~ ~ ~ 0 u ~ ~ :3 a) a~ ~ :^ ~ ~ ~ ~ ~ u' ~£ O ~ U] O V ~ C ~ O Ul U] ~C ~ ~ 3 C t7) ~ ~ ~V C ~: Q, O ~4 ~O V P~ ~ O l- U~ ~o O X Ll O ~ O ~ ~ Z O ~ ~ ~ E~ ~=- - a. tk~ t-~ P, 4-~ ~ -4-~ E~ P., ~ ~ u ~O H ~Ll 0 2 P4 V ·~' O ~ O ~1 C O C U] ~ V V C 1 a, c C ~ · - C 4~ 0 C V m- - C ~ q~ C ~ ~ q~ . - U2 ~ ~ ~ns c ~q Q, c ~ 0 1 a) ~ u' q~ ~ ·~ v 0 ~ c a, q" Q,-,1 Q, u]
129 U] - s C) S~ a) ~n a) o o s U) 1 y 1o .j4~ [QE C~ ~a ._. U]~ a3 C O o ~, U] O ·e ~a 3 U] r' q~ E 0 o ~O ~ ~ O O/IS O P4~ ~o O OO ·,' 4~ :, 0= 0 C, ~O P~O C) 4~ N `: ,~ ~ U] 04 .,~ a' 0 ~ _' OE v E ·54 0m ~O E ~ - 1, ~ ' ,] C C: Q. X u 0 a) q~ ~ ~a Ll n5 , 4~ ~ C: ~q c ~ ~ O .,. ~ 3 ~ .,. ~ c: ~ ~ O m -~1 U] dF dP U] ~ o s ~ a C 8 CO 11 11 . ~ C) t] O 4 (IJ )- .' c: u~ J~ a :' ~ ~C a' ~ ~ 0 ~ U] . - JJ 4 a o 0 c a) ~a 0 a' q~ s ~ q~ O ~ a ] ~ - ~ c m o . - o a' L4 Q. s 4~ c 3 P4 3 U' U) ~ 54 ~ O C~ ~ C ~ C U a' c O U' ~ ~ c~ 4 q~ O ~ C u o C) d~ dP u~ 0 a, s U~ 4J C 11 11 o ~ ~ C~ CD U) C C~ O ._, ~ 1 U] ~ · - Qi y · - U1 L4 [0 C~ ~ a~ c Q 1 V Ql ~ C m- - ~ C ·_~ i ~ C~ ~ · - · - JJ S ~ 4 C Co ~ C ~ a) ~ ~ 0 CL ~ Co ~ c a' c q~ ~ ~ ~ · - 1~ 0 0 .,. ~, ~, ·- ~5 ~ ~i ~ Q U) 4~ U] Y . - u' n a) ~ ~ · - C C C~ ~ 0 4~ U) C~ ~ ~ co ~ 3 _ c H U] S 4~ C o £ O ~ .,' C~ C O .,. ~ O JJ C aJ ~JJ c a) 0 a, a' dP dP ~ ~C N CD ~ ~ tD . - h U] ~1 _ ~q~ C~ 0 11 11 . - ~ ~ O C C ~ O S ~ Co m. - U) C ~.- U] U) 4 o C O 1 u' 1 .~4 0 U] ~C a) ~ O Ll U) O ~ ~ O ~ O CO ~) O O U~ C Ll O JJ L' ns ~, L. ~ C ) O Ll ~ ~ O ~ X Ll C) O U) O U: . Ll _' C U] ~4 JU O O O C: O · - V O Q~ J~ C U] V ~ U] L. ~ - ~- - ~ a) 4~ c u~ Q, q~ U' fO ~ O ~ · - · - L. U] 4~ C O C ~ V ~ U] C ~ ~c ~ ~ L. ·-I ~ O V ~ V c · - ·-~ S . - l 0 L. ~1 V JU ~ ~ ·,1 L. I Q. C C ~' - .,4 cn L' L. a q~ ·a c ~ U] ~ ~ ~ U] S ~ 0m S C O C 0 i4 11 11 0 C aJ ~ C~ ~r C C) U] C U] ~ U] S~ ~ - ~ · - O O Co O ~1 · - S V Ll U] C ~ 0 ~ a' tn c · - O L' C U] C O - - o r~ U] 4~ ~i C C c: C O O O O .,' 4~ ~ a V C V a' ~ a) a' q~ JJ ~ JJ q~ ~ ~I C ~l a) ~1 ~q a) ~ ~ ~ a' a) ~ ~ ~ a' ~C O ~ O . - ~ ' - O ' - ~ Ql ~ S J~ -4 .,1 ~ ~ C~ . - U] ~ ~ ~ U) 0 a' 0 P4 Z P' o U ~O U~ O (D O o 4 1 :o ~ L. O O O ~ 124 S L V C ) P4 z C ~>1 0 ~ .,' ~:' ~ 4 ~ ·-t U] U] U] c a ~0 L. a, ~ _ E~ ~ u~ V C C a' h C} a U] U' a' S Ll C~ c a.' s0 N q~ o U] C ~O O C , h ·-l J~ n5 C V C ~ O O ·' o, O ~ ~ 3 O ~ ~s c ~, C ~ Ll O O Q, oC V z o ~1 4~ V v a, 3, L' r~ .,~ O ~ L. ~ PM ~; J J~\ C a U~ o ~ 11 C C a' ~ ~ C O ~h L. O ~[ ~O C ~4 a) ~ c O ~ ~ a) ~ -l ~ Q~ ~n O O ~ ~4~ V ~ S'- CD V C X O V ~ 4J U] ~ ~ C V c a) ~ ~ u' Q s :' V C ~ ~ ~ U] u' u, ~ ~ 0 C c q~ ~ ~ 0 ~s a' ~ u~ ~ u' p, ~ ~ s C 4~ ~ U] C~ Co ~ ~ V ~ C~ c c c ~ ~ 0 ~ a u' ~ ~ c c . . . ~ ~ · ~ U] ~4
130 OIC/A Career Exploration Program The Opportunities Industrialization Centers of America, Inc. (OIC/A) operated the Career Exploration Program in seven sites during 1980. The design of the OIC/A program was similar in many respects to that of programs offered by RTP and other community-based organizations; it involved a 10-week summer program providing classroom instruction for 2 hours per day and career exposure site experience (work experi- ence) for 4 hours per day. A follow-up component extended for 8 months after the summer program and included review classes, counseling, referral services, and a newsletter designed to reinforce skills learned in the program. The OIC/A program served a predominantly minority clientele (78 percent black, 13 percent Hispanic) of 16- to 21-year-olds; nearly half of the participants were female (48 percent). Because an express aim of the program was to serve high-risk youths, about 24 percent of the participants were ax-offenders and 19 percent were dropouts; 75 percent of the participants were high school students, 4 percent were graduates, and 1 percent had received GED degrees. No data were provided on cross-site differences in participant characteristics. While participant and control group members were similar in terms of age and economic status, they differed significantly in other preprogram characteristics. Participants were more likely to be female (48 percent of participants compared with 37 percent of controls); attending high school {75 percent compared with 68 percent), and black or Hispanic (90 percent compared with 84 percent). By design, controls were more likely to be youth offenders than were participants, though the actual proportions differed from the planned 50 percent and 33 percent: 49 percent of controls were offenders while only 24 percent of participants were. The differences between the two groups suggest that random assign- ment was not strictly followed, at least at some of the sites, and the evaluation report itself suggests as much. While overall attrition at 8 months was reasonable for studies of this kind (23 percent for par- ticipants and 20 percent for controls), differential attrition among black participants eliminated the significant difference in racial composition that had existed earlier between the experimental and control groups. The evaluation results indicate that the participants' school attendance improved and their criminal recidivism was reduced at 3 and 8 months postprogram compared with a randomly assigned control group. The analysis by the Center for Studies in Social Policy of the data for nearly 1,500 participants and 800 controls at 8 months postprogram indicates that continued school attendance was significantly higher for participants than controls: 73 percent compared with 62 percent. Given the initial differences between the two groups on school and offender status, however, and the fact that being in school at program entry was found to be positively associated with being in school at the follow-up, it may not be surprising that participants compare favorably with controls on this measure. Because it is not clear that random assignment was effectively carried out and the results are confounded
131 with preprogram characteristics, it may be erroneous to treat postprogram differences as program outcomes. National Puerto Rican Forum The National Puerto Rican Forum's (NPRF) School-to-Work Transition program was designed to serve approximately 150 (largely) Puerto Rican in-school youths in each of various sites to "enable the participants to better understand and identify their strengths and weaknesses, facilitate the transition from school to work, and enhance their ability to select a career" {Murphy and Appel, 1981:13~. Services were intended to include workshops in self-awareness, preemployment skills, and job exploration for 5 hours per week during the school year. Actual contact hours averaged fewer than 30 in each site. In the initial year of the program, 1979, services were provided to high school seniors attending two schools in each of three sites, Chicago, Jersey City/Hoboken, and New York (South Bronx). In the following year, apparently due to a concern that many needy Puerto Rican youths leave school before senior year, the program shifted its focus to serving high school freshmen, lost one of the Jersey City schools, and added four high schools, two in Hartford and two in San Juan. Because of the program's focus on high school seniors, it may be appropriate to view the 1979 program as similar to job placement efforts (such as Jobs for Delaware Graduates or Project BEST, discussed in Chapter 9~. As such, the nature of the comparison group and the relative experiences of the two groups become important. While initially based on an experimental design, random assignment was abandoned due to insufficient sample sizes, and all youths expressing an interest in the program were allowed to participate. Comparison group members were high school seniors from the same schools as participants. No information is available on the procedures used to select comparison group members, who differed considerably from participants: comparison group members included fewer Hispanics and more blacks, had higher family economic status, and had less prior employment. Attrition among both participant and comparison group members at the 8-month follow-up (50 and 62 percent, respectively) was par- ticularly high for participants who were Hispanic and from low economic status households, while attrition among comparisons was highest for black males. Due to the attrition pattern, the initial differences in characteristics between the two groups largely disappeared. Moreover, the resulting sample sizes, 102 participants and 130 comparisons, were inadequate for reliable quantitative analysis. Therefore, no valid inferences can be drawn about the effectiveness of the 1979 NPRF program. The program that operated during the 1980-1981 academic year served high school freshmen for an average of 33 contact hours during the school year (Trismen, 1982~. About 83 percent of participants were Hispanic, 57 percent were female, and 85 percent were from families
132 with incomes no higher than 70 percent of the lower living standard of the Bureau of Labor Statistics. While no information is provided on how comparison group members were chosen, the demographic character- istics of the comparison group very closely matched those of partici- pants. Participants and comparisons in the second year program were, of course, younger than those who participated the previous year; they were also of lower economic status, had less prior employment experi- ence, and scored lower on pretests on a variety of cognitive measures. At program exit, participants exhibited significantly larger gains than did comparisons on each of the seven items in the SAS test battery. 2 This is in contrast to a finding of no significant differences in gains for participant and comparison group members during the previous year's program. Postprogram results are available only at 3 months after program completion for 61 percent of participants (260) and 65 percent of comparison group members (302~. Statistically significant findings that favored the participant group related to the degree of job knowledge, the proportion working full time or part time (40 percent and 29 percent), and the extent to which family members felt good about the program (or "how you've been doing n for controls). As seems appropriate for a program serving in-school youths, the employment outcome largely reflects part-time employment. At the time of the 3-month follow-up in early fall of 1981, about 86 percent of the respondents were in school. While participants were significantly less likely to be in school at the follow-up than comparison group members, other things equal, the actual difference was small: 85 percent compared with 87 percent. Overall, participants in the 1980-1981 program performed significantly better immediately postprogram than nonparticipants on a variety of cognitive measures. Three months postprogram, participants were somewhat more likely than comparisons to be employed full time (2 percent compared with 1 percent) and much more likely to be employed either part time or full time (40 percent compared with 29 percent), but they were also somewhat less likely than comparison group members to be enrolled in school. Thus whether the program intended to or not, it did not increase school retention. Although\the evaluation of the 1980-1981 NPRF program provided promising results, it provided insufficient information on how the comparison group was formed. Therefore, we cannot be confident about the results for either year of the NPRF program's operation. 2 Gains were measured in terms of percentages of standard deviations, and the magnitudes averaged 34-65 percent across all sites (Murphy and Appel, 19811. Gains of 10 percent or more were considered to have practical significance.
133 Project Redirection Project Redirection was designed to provide pregnant and parenting women aged 17 and younger who had not yet graduated from high school with educational, health, family planning, and employment-related services for up to 18 months. Its goal was long-term personal and economic self-sufficiency. It operated in four sites--Boston, New York (Harlem), Phoenix, and Riverside (California)--from mid-1980 to 1983. About 48 percent of participants were black, 38 percent were Hispanic, and 13 percent non-Hispanic whites. The average age of participants was 16, 56 percent were pregnant (not yet parents), and 52 percent were in school at the time of the baseline interview. Project Redirection was noteworthy for its development of a comprehensive program of counseling and supportive services for young women from low-income backgrounds who were pregnant or mothers. While the program seemed innovative and promising, the evaluation findings are unclear. The interim report on program effectiveness considered about 180 participants and 200 comparisons at only 12 months after enrollment in the program, when most young women were still partici- pating or had only recently left the program. The findings indicated that participants were less likely to have a repeat pregnancy (17 percent compared with 22 percent) and more likely to be enrolled in school or have completed school or a GED program (66 and 50 percent, respectively). However, a later report at 24 months after enrollment in the program showed Project Redirection youths on the whole fared no differently than comparison group youths on a variety of outcomes. There were no significant differences in the number of repeat pregnancies, in school enrollment or completion, or in employment. While the evaluation design of Project Redirection was superior to that of other programs for pregnant and parenting youths, several shortcomings limit our confidence in the findings. The comparison group approach used matched sites in the same regions, but several significant differences between comparison and participant group members at baseline suggest that the two groups were not comparable. For example, controls were more likely to be attending school (70 percent compared with 52 percent), had had more pregnancies, and had previously enrolled in a teen parent program (44 percent compared with 23 percent). Attempts to adjust for selectivity bias produced no difference in the results. In addition, the comparison group members received many of the same services provided to participants in Project Redirection. Thus, rather than being a test of the effect of providing services, per se, the demonstration is more appropriately seen as the test of the relative effect of the Project Redirection service provision strategy compared with others. The possibility therefore exists that the results understate the true program effects. In order to enlarge the participant sample by nearly one-half (from 180 to 350 participants), a second sample was formed with treatment group members who participated in the program between March 1981 and January 1982, about a year after the original sample members were given baseline interviews. They were added to the analysis along with
134 TABLE 6.5 Selected Project Redirection Outcome Differences Outcome First Sample Second Sample Percentage with pregnancy subsequent to baseline at 12-month interview at 24-month interview Percentage with live birth subsequent to baseline at 24-month interview Percentage ever enrolled in school between baseline and 24-month interview Percentage employed at 24-month interview - _ 9* 7 9 +11* - 8 + 2 - 2 +25* NOTE: Adjusted participant group mean minus comparison group mean. *Statistically significant at the .05 level or below in a two-tailed test. SOURCE: Polit et al. (1985~. additional comparison group members. The numbers of additional sample members were not uniform across the sites. Various techniques were used to overcome the lack of comparable baseline data for members of the second sample, including the use of retrospective data and estimation. The data for the two samples are pooled in most of the analyses even though the treatment period covered different time periods (and therefore possibly somewhat different program offerings) and members of the second sample participated for a significantly shorter period of time, 9.9 compared with 12.9 months. The pooling of the two samples may account, in part, for the apparent decline in program effects between the 12-month and 24-month results. The interim report results rely exclusively on data from the initial sample while the 24-month results include data from the second sample, which account for a significant share (40 percent) of the total. When results at 24 months are reported separately for the two samples, participant outcomes for the first sample are generally more favorable with respect to the comparison group than those of the second sample: for example, outcomes are better in terms of the number of repeat pregnancies, the number of live births, whether the participant is employed, and a variety of other measures (see Table 6.5~.
135 Although the overall attrition rate of 14 percent for the first sample was relatively low for a study covering 2 years, there was significantly more attrition among participants (21 percent) than among comparison group members (7 percent). Attrition rates for members of the second sample are more problematic: only 55 percent of those who participated were given the 24-month interview. Attempts to adjust for attrition bias using the Heckman (1979) procedure produced no changes in estimated program effects, although this was as likely due to the difficulty of modeling attrition as to any other explanation. 3 Finally, due to the nature of the sample design, site effects are confounded with race/ethnicity effects. The Harlem site was largely black (92 percent), Boston was predominantly Puerto Rican (96 percent), and Phoenix and Riverside were the only sites with white non-Hispanic participants (9 and 40 percent, respectively) and with significant numbers of Mexican-American participants (42 and 24 percent, respectively). Because of the many methodological difficulties inherent in the evaluation, we do not believe that reliable conclusions about the effectiveness of Project Redirection can be drawn. CONCLUSION Programs offering labor market preparation were the largest single category of programs we reviewed. Although 15 reports met the com- mittee's standards of evidence for determining program effectiveness, many suffered from serious methodological deficiencies that led to questions about their results. The results of several studies (mostly of programs for out-of- school youths) were of sufficient reliability to be examined for their implications for youth policy: the Alternative Youth Employment Strategies (AYES) project, some career exploration programs (those operated by OIC/A and RTP, in particular), and Project STEADY. Overall, the results of these studies suggest that most labor market preparation programs for out-of-school youths have at best only marginal effects on employment, and there is some hint that the effects may decay fairly rapidly (3 to 8 months) after participants leave the program. A comparison of the 26-week program with programs of 10 to 12 weeks suggests that the same marginal gains in employment can be 3 As a practical matter, it is when the use of these techniques produces change in the estimated results that the presence of selection bias is indicated. When selection bias adjustment techniques produce no change in the estimates, it can either be due to the absence of selection bias or the inability to properly identify the factors that differentiate participants from nonparticipants. Thus, when the application of these techniques produces no change in the estimated effects, it implies nothing about the presence or absence of selection bias.
136 achieved as well by a shorter program, although differences in target group characteristics and treatments suggest caution in generalizing this particular finding. The effects of the programs on job attitude and orientation, if reliably measured, are also marginal. The lack of a relationship between these measures and employment gains raises interesting issues about the goals of these programs (many of which were supposed to have focused on changing youths' attitudes and motivations) about attitude measurement, and about the relation of job attitudes to employment. The results of these studies raise many other interesting questions that, unfortunately, we cannot answer because of deficiencies in the research. Programs were operated in many sites, with variations in program approach and target group characteristics, but when the data were analyzed and the results presented, those differences were not examined, often because of insufficient sample sizes. Across and within sites, different groups of participants received similar services, perhaps with varying effects, but again most evaluations did not include separate analyses: for instance, for in-school compared with out-of-school youths; for dropouts compared with high school graduates; for males compared with females; or by race and ethnic subgroups. When a program has an effective outcome, we know little about why it works or for whom. Similarly, when there is no effect or no difference in effect, as in the AYES project, we cannot identify the reasons for the particular finding. Yet such information would help identify possible effective approaches to youth employment problems. Many of the labor market preparation studies produced by the YEDPA knowledge development effort are not discussed in this report because we found their methodological deficiencies too serious to allow reliable interpretations of their results. The most common shortcomings in these evaluations were the inadequacy of the control or comparison groups and sample attrition. In most cases, the "control" group was sufficiently different from the participant group in important char- acteristics that there was reason to suspect differences in unmeasured characteristics as well, which makes the attribution of changes in outcomes as due to the program questionable. In many other cases, although a requirement for random selection was stated as a planned feature of the research design, the randomization of participants and controls was abandoned. In still other cases, the comparison group consisted of participants in another program, resulting in probable underestimates of program effects relative to those based on a truly untreated control group. Appropriate techniques for following up program participants and comparison group members were rarely used. Not only was attrition in the 3- to 8-month period following program completion often in excess of one-third, but differential patterns of attrition raised serious questions about the validity of purported results. - - . . . · · .
