Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Summary, Conclusions, and Recommendations In response to high levels of unemployment and other employment- related problems of American youth, the federal government enacted the Youth Employment and Demonstration Projects Act (YEDPA; P.L. 95-93) in 1977. This legislation established a variety of employment, training, and demonstration programs for youth. With the passage of YEDPA, federal spending on employment programs earmarked for youth approxi- mately doubled, bringing the total to about $2 billion per year. Besides this substantial commitment of funds, YEDPA was unique in its explicit commitment of substantial resources to research and evaluation efforts intended to test alternative ways of meeting the needs of youth. YEDPA programs ended in 1981 with the change in presidential administration. At that time the products of YEDPA's research had not been comprehensively evaluated, and there were questions about what had been learned from this undertaking. Two years later the U.S. Department of Labor (DOL) requested that the National Research Council (NRC) review the products of the YEDPA research effort. The following eight chapters detail the findings from that review; this chapter summarizes our findings, conclusions, and recommendations. THE NATURE OF THE YOUTH EMPLOYMENT PROBLEM In focusing on the nature of the youth employment problem at the end of the 1970s, as YEDPA began, it is helpful to note that the majority of out-of-school youths found jobs and, when they lost or left a job, found another one without a long period of unemployment. It has long been recognized, however, that youth employment is more sensitive to the cycles of economic activity (recession and expansion) than is that of adults: the percentage decline in employment during a recession is generally greater for youths than for adults, but the percentage increase in employment during recovery is also greater than for adults. What was much more disturbing was the worsening long-term trend--which emerged clearly in the 1970s--in the employment rates of youths relative to adults, even when measured from the peak of expansion of one business cycle to the next. Furthermore, the data reveal substantial differences in that trend according to the race and sex of the youths; the long-term 1
\ 2 trend in employment was much worse for minority, principally black, groups than it was for the white majority. The most revealing picture emerges from data on employment-to- population rates that separate subgroups of the youth population by race, sex, and school status. Those data show that from 1964 to 1978 there was a growing gap between the employment-to-population ratios of white and black youths, for both in-school and out-of-school youths. For in-school youths, employment rates for whites were growing while rates for black males were falling and those for black females were not growing as fast, yielding an increasing black-white gap in employment rates. While some may regard in-school employment opportunities of lesser importance, researchers have found that, holding measured char- acteristics constant, those youths who work during school years have higher employment rates and wages after their school years. This finding may simply reflect that youths who are more motivated (an unmeasured characteristic) both seek work more energetically while in school and seek, find, and perform work better after school, but the possibility cannot be excluded that the in-school work experience per se enhances later employment and earnings. If so, the growing black- white in-school employment gap foreshadows a later out-of-school black-white gap in employment and earnings. The black-white employment gap for out-of-school youths also grew during this period for both males and females. And this occurred while the previously existing gap between blacks and whites in years of schooling attained was substantially reduced. Given the generally acknowledged positive relationship between years of schooling attained and employment and earnings, this reduction should have narrowed the employment gap between blacks and whites; however, it did not do so, or not sufficiently to offset other factors widening the gap. Research further shows that within this out-of-school group, employment problems (lower chances of getting a job, lower wages when a job is obtained, higher chances of losing a job, longer periods of remaining without a job having lost one) are highly concentrated among minority-group, inner-city, low-income, and high school dropout youths. For those with combinations of these characteristics the problems are compounded. Finally, it is apparent that young unwed mothers have very serious and special problems in qualifying for, finding, and holding jobs, espe- cially at earnings sufficient for their families' economic viability. Since the end of YEDPA in 1981 the United States has experienced both the deepest recession since the 1930s, which reached its trough in 1983, and a sharp economic recovery. There has also been a notable decline in the absolute size of the youth population since it reached its peak in the early 1980s. It seems reasonable to ask in light of these events if the nature of the youth employment problem has sub- stantially changed, in its general configuration, from what it was in 1978, as outlined above, when YEDPA began. Although exact comparisons cannot be made (comparable data are not yet available), it appears that at the beginning of 1985 the employment problems of youths were of about the same magnitude and configuration as they were in 1978, including racial differentials.
3 LIMITATIONS OF THIS REVIEW Our ability to respond to our charge was necessarily constrained by the nature of the material we had to work with. Although we searched the literature available beyond the reports generated as part of the YEDPA process and consulted with people experienced with youth programs and related research, we had to rely almost exclusively on the reports of particular YEDPA youth demonstration projects to assess the effec- tiveness of youth programs. The exceptions were studies of three programs that began before YEDPA, the Job Corps, the Summer Youth Employment Program, and Supported Work. We have attempted to test the individual YEDPA research reports against reasonable standards of scientific quality with respect to both the data collected and the methods used to measure program effects. The reports that met such standards were not necessarily evenly dis- tributed over the range of youth programs and target groups. Thus we could not address certain issues with respect to the role and effec- tiveness of youth employment and training programs because of a lack of reliable evidence. Since we were always in the position of examining these programs through the lens of their respective research reports, it is important that we clearly distinguish between the quality of the research and the (sometimes unobservable) quality of the programs themselves. In some instances we found reliable evidence, both positive and negative, from which to draw conclusions; in other instances the available evidence was not sufficiently reliable for us to draw any conclusion, one way or the other. Readers should be careful not to confuse a conclusion about the failure of research to provide adequate evidence with a conclusion that a particular program, itself, was ineffective or failed in some manner. A conclusion of noneffectiveness requires evidence, just as a conclusion of effectiveness does. In the absence of reliable evidence, no inference is possible. In addition to the above limitations, our ability to draw firm conclusions was further constrained by two conditions that affected the implementation of YEDPA and, particularly, the conduct of the research. First, YEDPA programs and research were mounted in considerable haste and in a period in which many other employment and training and research efforts were going on, so that both program and research resources were stretched very thin. (There are a few notable exceptions to this generalization, e.g., Job Corps and Supported Work.) Second, with the change of administration in 1981, less than 3 years after YEDPA's quick start-up and troubled implementation, both programs and evaluation efforts were abruptly halted. As a consequence of these two factors, most of the data on which evaluations were based, again with a few notable exceptions, were gathered at a stage at which programs had not been stabilized. As a further consequence, relatively few program evaluations provide data for long postprogram periods: virtually all of the YEDPA project evaluations had only 3- to 8-month postprogram follow-ups. Only two evaluations had as long as a 3-year follow-up (Job Corps and Supported Work). Our review suggests that longer-term follow-ups are important
4 in determining the time pattern of program effects, some of which decay rapidly and some of which emerge slowly. Further limiting our ability to draw firm conclusions were the serious problems of researchers in creating reasonable comparison groups and preventing sample attrition over waves of the data collection. m ese problems sharply reduced the number of studies we could review and put in question the reliability of the results of several others. As a result of these limitations, our coverage of YEDPA programs is uneven and not necessarily representative of overall youth program activity during that period. In many cases conclusions about the effects of specific types of programs are based on only one or two evaluations, in other cases there is no reliable evidence for any con- clusion of program effect. In addition, the quality of evidence varies, sometimes supporting strong conclusions, sometimes merely suggesting the direction of program effects. In presenting our conclusions, therefore, we try to indicate the source of the evidence and the degree of its reliability, and we distinguish lack of reliable evidence from lack of evidence per se. A final limitation of this review concerns the very magnitude of YEDPA and Comprehensive Employment and Training Act (CETA) programs from 1977 through 1981. It has been estimated that in 1979 as much as two-fifths of all jobs held by black teenagers were in government employment and training programs. Thus, even when comparison groups were reasonably created, there may well have been substantial amounts of employment, training, and related effects from federal programs among the comparison group members. To the degree this problem is serious and undetected in the evaluation data, the participant- comparison contrasts will underestimate the impact of the programs. FINDINGS AND CONCLUSIONS ON PROGRAM IMPLEMENTATION The conditions under which YEDPA was implemented severely con- strained both the potential effectiveness of the programs themselves in reaching their objectives and the related research and demonstration activities that sought to evaluate program effects and to create a reliable knowledge base for future youth programs. Implementation was affected by: (1) the size and implicit duality of the YEDPA service- research mandate; (2) the congressional and executive time schedules imposed on YEDPA program operations and research results; and (3) the instability of the service delivery system due to fluctuations in employment and training policy, regulations, and funding levels. The combination of these three factors was significant in determining the course of YEDPA at both the national and local levels. The duality of the YEDPA mandate, which was inherent in the enabling legislation, stemmed from the charge to mount new and very large service delivery programs quickly and at the same time to design and conduct comprehensive research and evaluation. Either of these tasks by itself would have been a sizable and complex endeavor; taken together they burdened the system not only by their sheer magnitude,
s but by their diversity of purpose, at times pitting the interests of program delivery against those of research and knowledge development. The imposition of two consecutive time limits also constrained the implementation of YEDPA programs and research. The first, imposed by the legislation itself, required that YEDPA be sufficiently operational within 1 year of passage to warrant congressional reauthorization; and the second, imposed by the executive office, required that within 2 years the results of YEDPA program research be adequate to inform the Vice President's Task Force on Youth Employment for its subsequent report to Congress. These limits put tremendous pressure on the national office of the Department of Labor to get both the programs and the research under way immediately and foreshortened the time available for careful planning of either the programs or the research on their effects. The third major factor constraining YEDPA was one that overrides YEDPA itself, and of which, in fact, YEDPA is a prime example: the tremendous fluctuations from one administration to the next, and often from one fiscal year to the next, in employment and training policy, regulations, and most importantly, funding levels. This instability, perhaps more than any other factor, undermined the employment and training system, particularly at the local level, where in response to such changes adjustments in all aspects of program operations ultimately have to be made. Such fluctuations precluded a more stable and orderly development and institutionalization of the youth employment system. Given the instability of the employment and training system, together with the implementation requirements of YEDPA, it was somewhat unreal- istic to expect that within 3 years these programs would be fully operational and ready to prove their effectiveness. CONCLUSION: The YEDPA legislation created a program that combined too short a time schedule with too many different program elements and objectives. The demand to quickly implement the full range of elements impaired the quality of many of the Programs. In addition, the pressure to obtain a wide range of research results on those programs within a short time compounded the problem and resulted, in many cases, in poor research on hastily constructed programs. It may be that the lack of proven effectiveness of many programs is due as much to the instability of the system as to the inherent nature of the Programs. National Office Management of YEDPA The tasks of designing and implementing YEDPA programs and research activities strained the capacity of DOL's Office of Youth Programs (OYP) given its very small staff and limited research capability. In response to these demands OYP created a system of indirect management, delegating substantial responsibility for the design, implementation, management, and evaluation of major YEDPA demonstration programs to private nonprofit intermediary organizations. In addition OYP extended
6 the staff's research capability through agreements with other research units within the Department of Labor, with the Educational Testing Service (ETS), and with Brandeis University's Center for Employment and Income Studies (CEIS). Agreements with other federal agencies to operate other portions of YEDPA were another means of expanding the YEDPA management structure. As a means of quickly disbursing funds and implementing programs under severe time constraints, the agreements with other parties were expedient. As a means of managing programs and research, however, that approach was not very effective. Of the four intermediary organiza- tions, only two, the Manpower Demonstration Research Corporation (MDRC) and the Corporation for Public/Private Ventures (CPPV), produced competent research on program Impacts on parclalpanc~. The results of programs operated by the other two intermediaries, the Corporation for Youth Enterprises (CYE) and Youthwork, organizations created to manage YEDPA demonstrations, are largely unknown because those organizations, although they did produce reports, did not attempt to assess program impact in a quantitative manner that could be evaluated. The results of programs operated under interagency agreements are also unknown either because no program research was conducted or because research reports failed to meet our criteria of scientific evidence. In general, considering the amount of YEDPA funds channeled through intermediaries and interagency agreements, remarkably little is known about the programs or their results. The results of OYP's other agreements were also mixed. Agreements with other ETA research units to incorporate a youth sample in the National Longitudinal Survey (NLS) and YEDPA samples in the Continuous Longitudinal Manpower Survey (CLMS) data bases produced useful results. m e agreement with the Educational Testing Service to establish a large national data base on YEDPA programs and participants, however, was poorly planned and implemented. The support provided by the Center for Employment and Income Studies was effective in documenting and assessing YEDPA programs, but CEIS's technical assistance and oversight of YEDPA research were not--and given the scale of the task, could not have been--sufficient to ensure the comprehensiveness of its research design or the quality of its execution, at least as evidenced by our review. CONCLUSION: The resources provided to the Office of Youth Programs were woefully insufficient to its charge to mount and manage YEDPA programs and research. Lacking research staff and resources, GYP delegated responsibility for the design and evaluation of large portions of the YEDPA demonstration research agenda to parties outside the Department of Labor. The resulting management structure lacked sufficient control at the center to provide coherence in program objectives and policies, to monitor developments, and to ensure accountability. These conditions had their greatest impact at the local level. With its additional reporting requirements, increased federal control over program design and target groups, increased services to youths, and the demands of research and demonstration, YEDPA imposed substan
7 tial administrative burdens on local prime sponsors. The competing demands of a substantially increased Public Service Employment Program and the regular programs under the Comprehensive Employment and Training Act (CETA) placed YEDPA in a strained local environment. The demands of the YEDPA agenda and the time schedule for their implementation severely hampered local program planning and assembly. The increased federal program requirements compressed the already shortened planning and approval process, which was complicated by the lack of clear guidance from the national office. To the extent that these conditions interfered with the careful planning of services, selection of program operators, coordination with other parties in the local service delivery network, identification and recruitment of participants, and assessment of local need for these programs, there were consequences for each subsequent stage of program operations and, ultimately, for the success of the programs. Despite these problems, YEDPA did succeed in mounting new programs, at double the size of previous youth programs, and in providing large numbers of disadvantaged youths with jobs. Furthermore, the evidence indicates that those jobs were generally well supervised and worthwhile experiences for both the participants and their employers. Targeting, Recruiting, and Retaining Youths Research on YEDPA programs cites numerous problems with targeting and recruiting sufficient numbers of eligible youths from the designated target groups: in-school and out-of-school youths meeting criteria of economic disadvantage. This problem was attributed in part to the short planning time and the resulting tendency of prime sponsors to base needs assessments on outdated information and to overestimate target group size. In addition, a legislative requirement that youths in regular CETA programs be served at the same levels as previously and the strict eligibility requirements for some YEDPA programs may have resulted in a shortage of eligible youths in some local areas. A related problem cited in many reports was the tendency of program operators to serve the least disadvantaged of the eligible youths, leaving the most disadvantaged and needy without services. This phenomenon, known as "creaming, n reflects the tradeoff that many program operators perceived between serving those most in need versus those more likely to succeed. It raises both equity and efficiency issues to the extent that the less disadvantaged might have achieved the same employment results without benefit of the programs. An example of this tradeoff, and one representing a major dilemma for YEDPA, was the targeting of services for dropout versus in-school youths. An inherent tendency of many YEDPA Programs was to Gravitate to the in-school population. A 22 percent YEDPA set-as~de for linkages with the schools was an additional incentive for local prime sponsors to target in-school youths. The dilemma was that the group most in need of employment services--the dropout population--was also the group that was hardest to recruit and to serve successfully. Conversely, the group most easily recruited and served--the in-school population--was , , _ _ _ , _ , _ _ . _ , _ _
8 the group for which services were less critical {or at least for whom the problem was less clearly defined). Though it is widely recognized that of all youth employment problems those of school dropouts are the most serious, there appears to be a tendency for employment and training programs to avoid serving this group. Many programs designed specifically to serve dropouts (either through school-conditioned work or through alternative education, training, or work settings) often had difficulties recruiting them and, once they were recruited, experienced difficulties retaining them in Other projects designed to serve either in-school or the programs. _ _ out-of-school youths, facing similar difficulties, evolved toward serving in-school youths, recasting the dropout problem in terms of prevention instead of remediation. CONCLUSION: In meeting the increased enrollment and rapid implementation requirements of YEDPA, prime sponsors when given the option of serving either in-school or out-of-school youths tended to focus resources on the in-school population. Even when programs were specifically targeted to dropouts, they often had difficulty in recruiting and retaining them. As a result, the question of how to reach and serve dropout youths effectively was largely unanswered by YEDPA. Enrollment of Young Women Most of the youth programs we reviewed enrolled substantial numbers of young women, and evidence from some programs suggests more positive effects for young women than for young men. Many of the programs, how- ever, encountered difficulties maintaining enrollment of economically disadvantaged young women, apparently because of the high incidence of teenage pregnancy and childbearing. Most program operators and evaluators apparently overlooked this characteristic of the target population, and so there is little direct evidence on the effect of pregnancy and childbearing on program participation or on the effect of program participation on subsequent pregnancy and childbearing. Evidence from one demonstration program designed to serve pregnant and parenting teenagers under the age of 18 (Project Redirection) is equivocal on the effect of a service-coordination strategy in reducing pregnancy and increasing subsequent employment and earnings. Neither is there evidence to date that would allow clear distinctions to be made as to the effects of alternative decisions about pregnancy resolu- tion, i.e., birth, adoption, or abortion, on other program outcomes. CONCLUSION: Most youth programs had substantial enrollments of young women. Many, especially those serving older teenagers, encountered difficulties maintaining enrollment of economically disadvantaged young women because of the high incidence of childbearing.
9 FINDINGS AND CONCLUS IONS ON PROGRAM EFFECTIVENES S Our conclusions are based on a review of 28 programs, including both~demonstration projects and regular youth programs. In this review we proceed by type of program and within type of program by target group, out-of-school youths (both dropouts and graduates) or in-school youths. Occupational Skills Training Programs Our conclusions on skills training are based on evaluations of only two programs, the residential Job Corps program for out-of-school, mainly dropout, youths and an apprenticeship program for in-school youths. Occupational Skills Training Programs for Out-of-School Youths The Job Corps The Job Corps stands out in our review as a program for which there is strong evidence regarding program effectiveness. The quality of the evaluation reviewed, in terms of sample sizes, comparison group method- ology, sample attrition, and the measurement of outcome variables, lends confidence to these conclusions. The Job Corps is a comprehensive program providing occupational skills training, basic (and remedial) education, counseling, health care, and job placement to youths more disadvantaged than typical participants in youth programs. Although the contribution of each component part of the program is not known, it is clear that as a whole the program has provided positive benefits to participants in terms of employment, earnings, and education. On average, for up to 3-1/2 years after participation, Job Corps enrollees earned 28 percent more per year ($567 in 1977 dollars) and worked 3 weeks more per year than nonparticipant comparisons. In addition, participation reduced receipt of welfare and unemployment by 2 weeks and 1 week per year, respectively. CONCLUSION: Job Corps participation resulted in gains in employment and earnings in the postprogram Period and in declines in receipt of welfare and unemployment payments. These positive effects persisted at a relatively stable rate for up to 4 years after youths left the program. Participation in the Jobs Corps increased the probability that Job Corps enrollees would receive a high school diploma or a General Equivalency Diploma (GED). Specifically, the probability was .24 for participants compared with .05 for comparisons. CONCLUSION: Job Corps participation resulted in gains in educational attainment during the program as measured by completion of GEDs.
10 The Job Corps evaluation measured criminal activity and found that Job Corps participation significantly reduced the criminal activity of participants. CONCLUSION: Jobs Corps participation resulted in decreases in criminal activity, as indicated by rates of arrest during program Participation and decreases in seriousness of crimes in the postprogram period. In addition, although the Job Corps by far exceeded the per- participant costs of other youth programs, the benefit-cost analysis indicated a net benefit. CONCLUSION: The benefits of Job Corps participation in terms of increased employment and earnings and decreased crime and transfer payments exceeded the costs by a sizable margin ($2,300) per enrollee. Other Occupational Skills Training Programs Although there is substantial evidence on the effectiveness of the Job Corps, it is not known which of its several component parts con- tribute to which effects; how much (if any) is due to the self-selection factors of youths who enroll in the program; or how program components and participant characteristics interact. The residential requirement of the Job Corps, in particular, is untested as a factor in explaining the program's effectiveness and precludes generalizing its results to nonresidential settings. Nonresidential skills training would certainly be less expensive to operate than the Job Corps; however, YEDPA Produced no reliable evidence on the effectiveness of occupational skills training Provided in a nonresidential setting for out-of-school , youths generally or for the severely disadvantaged population of out-of-school youths served by the Job corps. - Occupational Skills Training Programs for In-School Youths m e committee found few studies of occupational skills training programs operated under YEDPA. This was due, at least in part, to a concern that participants require a sufficiently high level of academic preparation to be able to gain from such training. We reviewed only one program providing occupational skills training to in-school youths. This program (New Youth Initiatives in Apprentice- ship) was designed to prepare high school seniors for registered apprenticeships after graduation. CONCLUSION: m ere is only very limited evidence from YEDPA on the effectiveness of skills training for in-school youths. The only program that provided evidence of reasonable quality showed no effect on participants' postprogram earnings or
11 employment rates. However, this Program was so special in nature and participant characteristics that one would not wish to base general conclusions about skills training for in-school youths on the evidence from this program alone. Labor Market Preparation Programs The programs classified as labor market preparation programs in our review were very heterogeneous in terms of their services and activities, but they shared the long-term goal of preparing youths for their future work lives by improving their personal skills, knowledge, and attitudes toward the work place. Activities ranged from career exploration and job search assistance to remedial education and combinations of work experience and classroom training. The programs also varied greatly in intensity and duration, ranging from 5 to 35 hours per week and from 10 weeks to 1 year. Programs for Out-of-School Youths The Studies of labor market preparation programs serving out-of-school youths tended to provide sounder evidence on program effectiveness than did studies of programs serving in-school youths. In the 3- to 8-month postprogram period, participants often exhibited significantly better employment outcomes than nonparticipants. It is particularly trouble- some, however, that the term "out-of-school youths" is used to refer to high school graduates as well as dropouts: the programs providing reliable evidence served varying mixtures of the two groups and did not produce separate analyses of effects. This lack of separate analysis for dropouts and graduates conditions our confidence in the evidence because program outcomes (e.g., employment and earnings, and educational attainment) might be influenced by whether the youths had completed high school. CONCLUSION: YEDPA programs providing labor market preparation for out-of-school youths resulted in some positive effects on employment in the 3 to 8 months following Program Participation (Alternative Youth Employment Strategies, the Recruitment and Training Program, Project STEADY). There are no reliable data, . however, to determine whether these short-term gains are sustained over the long run or whether such programs had any effects on educational attainment or other coals, such as . reduced crime and substance abuse. Programs for In-School Youths Some of the reports on in-school programs we reviewed indicated that program operators did not expect to directly affect the youths' postprogram earnings or employment; instead, they concentrated on other
12 program goals, such as improving reading, mathematics skills, work attitudes, and other skills measured by the Standardized Assessment System, a battery of tests developed by the Educational Testing Service. ~ market Preparation services to in-school youths, the research on the effects ~ ~ ~ _ of these programs was of low quality. Even for the few research reports of sufficient quality to be reviewed in depth (Opportunities Industrialization Centers of America, National Puerto Rican Forum (NPRF), Project Redirection) each was so seriously flawed in one respect or another that we could draw no conclusions regarding the effects of such in-school labor market Preparation programs on employment, earnings, educational attainment, or other goals. Temporary Jobs Programs Programs providing temporary subsidized employment have until recently, with the passage of the Job Training Partnership Act (JTPA), been a major focus of employment and training policy for youths. The objectives of these programs were to solve the immediate employment (and income) needs of disadvantaged youths and to provide them with work experience which would be a basis of future employment. In addition, under YEDPA a major thrust was the testing of an entitlement to subsidized jobs for economically disadvantaged youths (Youth Incentive Entitlement Pilot Projects), which was designed to encourage them to remain in or return to school. Programs for Out-of-School Youths Two subsidized jobs programs operated under YEDPA, Ventures in Community Improvement (VICI) and Supported Work, showed that partici- pants experienced increased employment and earnings during the program compared with a similar group of nonparticipants. CONCLUSION: Temporary jobs programs for out-of-school youths that were operated during the YEDPA period were effective in increasing participants' employment and earnings during the period of program participation. The types of subsidized job opportunities provided in YEDPA demonstration programs varied widely. Most, as required under CETA, were in the public sector. YEDPA, however, for the first time allowed work experience placements in private for-profit businesses. Evidence from the Public Versus Private Sector Jobs Demonstration Project tentatively indicates that while it was possible to create subsidized employment in the private sector, substantially more effort was required to do so. This finding is not surprising considering the lack of experience of prime sponsors in developing private sector jobs and the lack of experience of the employers in working with government- sponsored programs. Evaluations of Supported Work and VICI programs
13 - indicate that jobs provided were often of high quality and produced output that was of value to the employing agency and to society in general. CONCLUSION: Subsidized jobs programs for out-of-school youths can be effectively operated in the public and private sectors, though substantially more effort is required to enlist private sector employers. Such jobs need not be "make-work" but can provide output of positive social and economic value. Probably the most important goal of temporary jobs programs for out-of-school youths is to provide an experience that will raise the postprogram employment and earnings of participants over what they would have been without that experience. Unfortunately, the only evidence on this issue that we found reliable was that for Supported Work. The Supported Work program served a severely disadvantaged segment of dropout youths and, therefore, the degree to which its findings can be generalized to the wider out-of-school youth population is not known. However, the long postprogram follow-up study (15 to 67 months) provided strong evidence of no long-term gains in postprogram employment and earnings. CONCLUSION: m e evaluation of the Supported Work program for severely disadvantaged school dropouts provided strong evidence of no long-term gains in employment and earnings. In addition to their value as a source of immediate employment and as a bridge to longer-term unsubsidized employment, temporary jobs programs have sometimes been seen as yielding social value in the form of return to school by dropouts or entrance in alternative education (GED) programs or through deterrence of crime and substance (drug and alcohol) abuse. The only temporary jobs program for out-of-school youths reviewed here for which evidence on these measures was found reliable (Supported Work) seemed to have no such effects. CONCLUSION: A temporary jobs program for severely disadvantaged out-of-school youths (Supported Work) had no effect on educational attainment nor on reducing crime or alcohol and drug abuse. The entitlement program, which guaranteed jobs for dropouts if they returned to school, provides evidence on the effects of temporary jobs programs on school completion. Its effectiveness is discussed in the next section. Programs for In-School Youths Temporary jobs programs for in-school youths under YEDPA were provided to meet the immediate part-time employment needs of school youths both during the school year and in the summer. Like temporary
14 jobs programs for out-of-school youths, the in-school programs we reviewed, i.e., entitlement and the Summer Youth Employment Program (SYEP), were effective in increasing participants' employment and earnings during the period of program participation. These findings, and the entitlement results in particular, refute the hypothesis that the employment problems of these youths are a function of their high reservation wage (that is, that jobs are available but not sought or filled by these youths because the wages are too low), since large numbers took up these jobs when they were made available through the programs. Overall, the entitlement program significantly reduced unemployment. m ere was also some evidence that under such programs jobs of reasonable quality could be provided. As with the out-of-school programs, work experience placements in the private sector were developed, but they required greater effort than public sector placements and resulted in some degree of job substitution and hi Al Ant (more in private than in Public sector jobs). -~r-~-~- ~--I -a ¢~ CONCLUSION: Temporary jobs programs for in-school youths operated during the YEDPA period (entitlement and SYEP) were effective in increasing participants' employment and earnings during the period of program participation. The entitlement program demonstrated the ability of a government program to eliminate black-white employment rate differences among in-school youths. This effect may be viewed by some as contributing to the goal of greater social equity and as of sufficient merit, in itself, to justify the program regardless of other benefits. Others may believe, however, that an important objective of temporary jobs programs, even for in-school youths, is improvement in postprogram employment and earnings. CONCLUSION: For one program (entitlement) effects were of sufficient magnitude to eliminate the employment and unemployment differences between black and white youths who were eligible for the program. In addition, there is evidence that these jobs were of reasonable quality. In the immediate postprogram period observed for the entitlement program (2 months after entitlement ceased operation), there was some evidence of increased weekly earnings of eligible youths. When examined by urban versus rural sites, however, it was apparent that this effect was due largely to the results in the rural Mississippi site. CONCLUSION: A major temporary jobs program (entitlement) appears to have been effective in increasing the earnings of eligible youths in the immediate postprogram period, but this effect was dominated by results in one rural site.
15 Unfortunately, none of the research on the YEDPA temporary jobs programs for in-school youths produced reliable long-term follow-up data that would permit determination of the long-term effects on employment and earnings. CONCLUSION: YEDPA produced no reliable evidence on the effect of temporary jobs P-r--ograms on the long-run postprogram employment and earnings of in-school youths. One of the purposes of temporary jobs programs for in-school youths was to prevent youths from dropping out of school in order to find employment. The entitlement project in fact set school enrollment as an eligibility requirement for a subsidized job and had as a major objective the retention of youths in school. Comparison of school retention rates in entitlement and nonparticipant sites, however, indicates that the program had no effect on school retention. The Summer Youth Employment Program also emphasized return to school as an objective but there are no reliable data on the effect of SYEP on school return or retention. CONCLUSION: A major program (entitlement) offering temporary jobs to youths on the condition that they remain in school was not effective in increasing school retention. An express aim of the entitlement program was not only to prevent dropouts but also to encourage those who had already dropped out to return to school by offering them a subsidized part-time job during the school year and a full-time job during the summer, both conditional on continued enrollment in school. This incentive, though it did encourage some dropouts to return to school initially, was ineffective in retain- ing them--comparison of entitlement and nonparticipant sites indicates no differences in school completion rates. CONCLUSION: A major program (entitlement) offering temporary jobs to school dropouts on the condition that they return to school did not increase school completion among these youths. Job Placement Programs Job placement programs, which provided job search career information, and job placement to youths, were assistance, designed to ease the labor market exchange between potential employees and employers. Such programs were thought to increase employment and earnings of youths through more efficient operation of the labor market. Although job placement was often a component of other more comprehensive programs, the programs considered here offered only job placement. The conclusions drawn, therefore, do not necessarily apply to job placement as a support service in other programs. Conclusions on the effectiveness of job placement programs for out-of-school youths are based on evaluation of two projects, 70001 and
16 the Job Factory. Conclusions on programs for in-school youths are based on evaluations of Jobs for America's Graduates (JAG) and Jobs for Delaware Graduates (JDG). Both the clientele and the costs of the programs we reviewed varied widely. Jobs for Delaware Graduates served largely nondisadvantaged, noncollege bound in-school youths and offered no stipends for partici pation. The 70001 program, funded with YEDPA money, served disadvantaged youths, both in-school and out-of-school, and provided stipends. In general, the effect of these job placement programs for both in-school and out-of-school youths was to produce short-term increases in the rates of employment and earnings of participants over those of similar youths not receiving such services. There is evidence that these effects lasted for the first year after program participation, decayed, and then disappeared by the end of the second year. CONCLUSION: Programs designed to provide job placement assistance to in-school and out-of-school youths were effective in increasing employment for the first year after program participation. This effect decayed gradually and disappeared entirely by 2 years after program completion. Benefit-Cost Estimates After a review of the effects of YEDPA programs, it ask about the efficiency with which the programs achieved is natural to those ~ _ ~ ,. _ effects. The most rigorous way to formulate a response to this question is in terms of a comprehensive benefit-cost analysis. In order to make a complete benefit-cost analysis of a program it is necessary, first, to have reliable estimates of the effects of the program and, second, to have an appropriate accounting framework that translates those estimates of effects into benefits and costs. Many of the programs we reviewed lacked the first element and, except for Job Corps and Supported Work (programs that predated YEDPA), even when reliable estimates of effects were available, they were not translated into benefit and cost estimates. For Job Corps and Supported Work, we found the benefit-cost analyses reliable and comprehensive: for Job Corps, the social benefits were found to exceed social costs; for Supported Work, the social costs were found to exceed social benefits. With the exception of two programs (Job Corps and Supported Work), we could not find sufficient reliable evidence to assess the efficiencY, in terms of social benefits and social costs, with which youths programs achieved their effects. FINDINGS AND CONCLUSIONS ON YEDPA RESEARCH We often found it difficult, as noted above, to reach firm conclusions about the effectiveness of YEDPA programs given the quality of the available evidence. Indeed, the difficulty in making inferences
17 arose largely from the paucity of reliable evidence from the YEDPA research effort. Careful readings even of the few reports that did meet minimal standards of methodological adequacy frequently revealed important flaws in the reported research. It would be unfortunate, however, to conclude that rigorous research cannot be conducted on youth programs. On the contrary, we found several examples of research studies on youth employment and training programs that provided strong evidence, both positive and negative, on program effectiveness (Job Corps, Supported Work, and Alternative Youth Employment Strategies). These examples indicate that given reasonable time for design and planning, attention to research standards, care in coordinating with program operators and in program implementation and follow-up, good evaluations of employment and training efforts can be executed for both small- and large-scale programs. While it is relatively easy to point out the methodological problems in YEDPA research programs, it is more difficult to identify their causes. In the interests of improving the quality of future research, we describe below some of the common inadequacies of the YEDPA research effort and some of the conditions that, we believe, contributed to those failings. m ere were a wide variety of problems in YEDPA research. Many are obvious and reflect the difficulty of meeting established research standards for complete reporting of research results, documentation of procedures for sample definition, presentation of details of statistical analyses, adequate coverage of the target population in data gathering, and minimal sample attrition. Low rates of initial sample coverage and high rates of sample attrition, for example, were among the most common flaws of the YEDPA research reports we screened. Other problems, however, are less easily resolved and appear to be inherent in public policy research. Comparison and Control Group Methodology Difficulties in identifying equivalent comparison or control groups flawed many YEDPA studies. Very few of the studies we reviewed used random assignment to create participant and control groups. A more frequent strategy was to define or construct a comparison group that, except for program participation, was presumed to differ in no important way from the participant group. Differences in measured outcomes were then attributed to the effects of program participation. In some cases, statistical techniques were used to control for measured differences between participants and nonparticipants. Our review of research that used such constructed comparison groups revealed two basic problems that repeatedly jeopardized the validity of the inferences to be drawn. The most frequent problem was the use of comparison groups that differed markedly in background characteristics from participant groups. ences in However, ~ _ Given an adequate theoretical model, differ- measured characteristics can be controlled for statistically. when there are substantial differences in measured character
18 istics one becomes uneasy about the adequacy of the theoretical assumptions and statistical adjustments and is left wondering about major differences in unmeasured characteristics that may be correlated with the outcome measures. A second common problem was the use of treated comparison groups, i.e., groups receiving services similar to those provided to the YEDPA participants but in a different program. Both of these problems undermined the basic purpose of a comparison or control group and made the attribution of program effects questionable. The research reviewed in this report demonstrates the causes for concern about bias in estimated effects from use of such comparison groups. Because of these problems we frequently found that purported evidence of YEDPA program effects might plausibly be ascribed to the non- equivalence of the comparison and participant groups, even when statistical methods to control for such differences were used. Even in the methodologically sophisticated work conducted on the CLASS data base, there is convincing evidence that the constructed comparison groups that were equated on common socioeconomic variables could differ markedly on important unmeasured variables. The Job Corps evaluation also, despite efforts to correct for biases in the con- structed comparison group, was ultimately less convincing in its estimates of effects than would have been the case had random assignment been used. CONCLUSION: YEDPA research did not make sufficient use of random assignment in defining participant and control groups. Our review of the research on YEDPA shows dramatically that control groups created by random assignment yield research findings about employment and training programs that are far less biased than results based on any other method. The fact that some studies successfully used random assignment suggests that this procedure is feasible and presents no serious technical difficulties in execution. It is evident that if random assignment had consistently been used in YEDPA research much more would have been learned. (The use of random assignment in public policy research is discussed in Appendix C.) Measures of Implementation In a real-world experiment involving a social program one does not have a single, standardized treatment. Hence, evaluation of the outcomes of such social experiments must carefully take into account the nature of the treatments given and variations between the treatments received by different individuals. Without paying attention to matters of implementation one cannot know whether a given program was successfully delivered. If no program approximating that intended by its designers was implemented at a given site, for example, there is no reason to treat the results as a "test" of the outcomes arising from the program. We found that although YEDPA produced several studies of program implementation, YEDPA research gave insufficient consideration
19 to measures of the extent of program implementation in interpreting or qualifying program results. The YEDPA research plan envisioned gathering a standard set of data about the process of program implementation. This plan included infor- mation on the nature of the services offered by different programs and sites, the gearing up and scaling down of operations over time, the turnover of staff, the distribution and magnitudes of program expendi- tures, and so forth. Unfortunately, this effort was neither well planned nor well executed (see Appendix A). Length of Follow-up Period A major shortcoming of theoretical importance in YEDPA research was the very short postprogram period over which outcomes were measured. For most programs, follow-up measures were available only at 3 or 8 months after program completion; Job Corps and Supported Work, with over 3-year follow-ups, were notable exceptions. Given the employment objectives of most YEDPA programs, a more reasonable test would have considered longer-term outcomes. The Job Corps experience of initial declines in employment and earnings in the first months after termination, followed by significant longer-term gains, suggests that, at least for some programs, additional time may be required for effects to emerge. On the other hand, there are examples (70001 and JAG) indicating that early program effects may decay rapidly. Thus, there is evidence that short-term follow-ups may lead to incorrect inferences, both positive and negative. CONCLUSION: For the most part, YEDPA program evaluation research suffered seriously from the lack of outcome data collected over a sufficiently long period following program completion. For some programs it appears that there was never any serious intention to collect data beyond 8 months post- program; for other programs, the abrupt cutoff of funds in 1981 precluded any attempt at further follow-up. It is clear that the long-term effects of programs cannot be measured when there are no data beyond 8 months. Neither policy makers nor researchers are well served by such truncated evaluation studies. Use of Subjective Measurements as Proxy Variables Many YEDPA research projects used subjective measurements, such as measures of work-related attitudes and job-holding skills, as proxies for longer-term outcomes. As with all such measures, the link between the proxy variable (the social attitude or skill) and the unmeasured long-term outcome variable (increased employment and earnings) is subject to empirical verification. It is important to ask how well these variables serve as proxies for the variables of primary interest. (Appendix A reviews the correlations between the subjective measures
20 used as proxies in YEDPA research and measures of employment and earnings variables.) We found that these measurements showed relatively low correlations with the objective variables for which they were intended to serve as proxies and that the reliability of these YEDPA measurements was rather low. Furthermore, an earlier analysis of data on the same measures collected prior to the 3- and 8-month data collection periods was predictive of these low correlations and reliabilities. CONCLUSION: YEDPA evaluations invested large amounts of resources in measurement of proxy variables rather than the real variables that were the goals of the programs. Moreover, most of these were subjective measures known to have only modest levels of association with the objective outcomes that were the implicit long-term goals of the YEDPA programs--future employment and earnings. Sample Undercoverage and Attrition Sample undercoverage and sample attrition were two of the most critical and frequent problems with YEDPA research. Indeed, attrition between waves of a study was second only to lack of impact data as a reason for excluding reports from our review. In addition, over the range of YEDPA programs covered by the Standardized Assessment System, it is estimated that data were gathered from less than half of the target sample. This undercoverage makes it impossible to generalize to the total population of participants in YEDPA demonstration projects unless one makes highly unrealistic assumptions, for example, that nonresponders were a random sample of program participants. CONCLUSION: Many YEDPA research projects gave inadequate attention to sample design and execution, including defining the sampled population, obtaining data from all members of the target sample, and preventing sample attrition between waves of the longitudinal data gathering. It is important that the foregoing catalog of problems with YEDPA research not lead readers to the despairing conclusion that such problems are unavoidable for all evaluation research in the employment and training field. We found that these problems were not predestined by the nature of the questions being posed, but rather were caused by the demands of Congress for research results and the failure of the researchers to recognize the limits imposed by YEDPA conditions and scale the research accordingly. The research tools available are ade- quate to the task, but suitable conditions for their use must be provided. These conditions include adequate time for design and execution of the research, reasonable stability of the programs being evaluated, provision of adequate resources for the research, and a political commitment on the part of those commissioning the research to allow adequate time not only for careful planning at the beginning, but . .
21 also for enough time at the end to see the research through to its conclusion and to obtain the full benefits. OVERALL CONCLUSIONS: DID YEDPA ACHIEVE ITS GOALS? The YEDPA legislation had four major objectives that broadly defined its scope: (1) to address the immediate employment needs of youths through job creation efforts; (2) to conduct research and demonstration as a means of identifying methods of dealing with the structural employment problems of youths; (3) to involve other agencies, nationally and locally, in the planning and delivery of youth employment services as a means of exploring the effectiveness of alternative service delivery mechanisms; and (4) to increase cooperation between the schools and the employment and training system as a means of addressing the problems of disadvantaged youths. We present here our conclusions on the extent to which these objectives were met. Job Creation A key objective of YEDPA, the one on which the most funds were spent, was providing young people with jobs. In response to high youth unemployment in the 1970s, federal outlays for youths in the first year of YEDPA doubled what they had been, and in its course YEDPA served over 6 million youths. Previous efforts at large-scale jobs programs (SYEP and Public Service Employment) were widely criticized for their poor administration and poor job quality (U.S. General Accounting Office, 1979~. Evaluations of massive jobs programs under YEDPA (SYEP and entitlement) showed major improvements in job supervision and job quality. There is evidence not only that very large numbers of jobs were provided to youths under YEDPA, but also that the quality of the jobs was more than "make-work." CONCLUSION: A major achievement of YEDPA was that it succeeded in providing large numbers of disadvantaged Youths with jobs _ that were more than make-work. YEDPA demonstrated the capacity of the employment and training system to mount and run large-scale jobs programs for young people. Research and Demonstration Although the employment and training system under YEDPA was able to demonstrate its capacity for implementation and operation of large- scale programs, it was hampered in its response to its second major objective--conducting research and demonstration aimed at solving the structural employment problems of youths. Our review suggests that the institutional capacity for research and demonstration was insufficient to the objectives of YEDPA knowledge development. The knowledge
22 development agenda itself was too ambitious given the rapid time schedule of YEDPA and the other demands of large jobs programs. In terms of the results of YEDPA research, the youth employment situation today is not very different from what it was before YEDPA began. CONCLUSION: Rapid program expansion in a period of considerable social program activity severely hampered the planning, imple- mentation, and evaluation of YEDPA demonstration research. The abrupt halt of research activity in 1981 did not help. As a result, despite the magnitude of the resources ostensibly devoted to the objectives of research and demonstration, there is little reliable information on the effectiveness of the programs in solving youth employment problems. Developing the System m e youth employment and training system has long ~_ _, ~. ~ ~ _ ~ suffered from its This isolation from major economic and educational Institutions. isolation is due to a number of factors, most importantly the character of its client population and its lack of a stable institutional base. m e YEDPA legislation required the involvement of private employers, the schools, and other established agencies, both public and private, In planning and delivering youth services in order to explore alter- native mechanisms for more effective service delivery and to encourage the integration of employment and training services into the mainstream of the society. Our review of numerous reports on program effectiveness and implementation suggests that although the involvement of these parties was effective in making better use of local resources, it complicated the planning and delivery process and did not on the whole achieve its goals of improving service delivery and integrating either the programs or their participants into the larger society. Given the nature of the youth employment problem, the goals of equity and efficiency argue for targeting services to disadvantaged youths. At the same time, however, our review suggests that the youth employment and training system continues to suffer from a marginal status that is due in part to such targeting. CONCLUS TON: Despite YEDPA efforts to improve the service delivery system, employment and training services for youths remain economically, socially, and racially isolated from the larder society and its institutions. This isolation has - sti matized the ~roarams and undercut their abilitY to recruit g 17 J - staff, motivate participants, and involve employers. The Schools and School Dropouts The YEDPA requirement that 22 percent of the funds for YETP be set aside for programs coordinated with the local school system was an
23 attempt to influence the schools to broaden their constituency to include disadvantaged youths. Despite the agreements negotiated under YEDPA to involve the schools in youth employment programs, we found little evidence of successful mutual efforts. CONCLUSION: The relationship between the employment and training system and the school system remains problematic. Despite some common objectives and client groups and efforts to bring the two systems together in the service of those objectives and clients, they remain largely separate, and the potential benefits of mutual efforts are largely unrealized. An example of this problem, and one of great importance to this committee, is the problem of school dropouts--a group of legitimate concern to both the schools and the employment and training system and a group that neither has been able to serve effectively. No other youth group faces the employment problems, both immediate and long term, faced by school dropouts, and particularly those who are minority group members. The YEDPA approach to the dropout problem was twofold: (1) prevention of the problem by targeting services to youths at risk of dropping out and giving them incentives to remain in school and (2) remediation of the problem by targeting services directly to dropouts in a way that encouraged return to school or an alternative education. Our review of youth programs found no evidence of effective means of either dropout prevention or remediation. We observed instead the severe problems of schools and other program operators in recruiting and retaining dropout youths and the tendency of those programs to focus their services on the more easily recruited and served population of in-school youths. CONCLUSION: Of all youth groups, school dropouts face the most serious employment problems. Because of problems in recruiting and serving dropouts, however, the focus of youth research and demonstration under YEDPA was unduly directed to in-school youths and high school graduates. As a result, the question of how to reach and serve dropout youths remains unanswered. IMPLICATIONS AND RECOMMENDATIONS Two basic issues--the problem of school dropouts and the relations between the schools and the employment and training system--remain, in the committee's view, fundamental dilemmas confronting the youth employ- ment and training system in the United States. We begin our discussion of implications with our recommendations for future youth policy with regard to dropouts. We recognize that there is a long history of research and program attempts to understand and deal with the problems of school dropouts. And yet, as our review strongly suggests, dropouts, particularly minority group dropouts, remain as the segment of the youth population
24 with the most serious employment problems. Attempts, both preventive and remediative, to address the employment problems of this group, with the exception of the Job Corps, have been ineffective. Despite these efforts we know little more about dropouts now than we did before YEDPA, not only in terms of their responses to employment and training programs, but more fundamentally in terms of the factors--economic, social, and psychological--affecting their dropout and subsequent educational and employment behaviors. RECOMMENDATION: The committee strongly recommends that school dropouts be given priority status for employment and training programs and research. Program efforts should be shaped to test systematically the alternative methods of addressing the education and employment needs of these youths, and research should focus on the underlying determinants of the dropout phenomenon. Another major implication of our review concerns the marginal role of the youth employment and training system, its relation to the school system, and the two systems' relation to the broader society in addressing youths' educational and employment needs. We recognize the inherent tension between the above recommendation to give priority status to dropouts and the suggestion that the employment and training system, partly because of such targeting of services, has isolated itself from the broader society. This is a complex problem to which we have no solution. We believe, however, that it is an important aspect of the youth employment problem and that it bears serious consideration and further study. RECOMMENDATION: _ In order to rid employment and training Pro grams of the stigma which has plagued them and their partici- pants, the committee strongly recommends that attempts be made to target services for disadvantaged youths in ways that will not isolate them but rather integrate them into mainstream institutions and activities. The role of the school system and the relation between the schools and the youth employment and training system are critical in resolving this problem. The committee therefore recommends a direct study of the appropriate role of the youth employment and training system, and its relation to the educational system, in alleviating the employment problems of those youths most in need of assistance. Youth Programs Committees such as ours invariably recommend further program research and testing. Unless the problems addressed by the programs have disappeared or been substantially ameliorated or unless social priorities have shifted sharply, such recommendations should in good conscience be made. We are hesitant, however, to prescribe program approaches and techniques as lessons from experience. In our con
25 sidered judgment, the clearest lesson from the YEDPA experience concerning effective programs is that much remains to be learned about dealing with youth employment problems. In our judgment, it is better by far to admit that knowledge is lacking than to assume on the basis of scant knowledge that we know what works best for whom. Therefore, our recommendations on programs and program research are closely tied to the evidence we reviewed. In this section we first present our program recommendations for out-of-school youths, then for in-school youths, and finally for programs serving women. Programs for Out-of-School Youths The results of our review and the present conditions of the new Job Training Partnership Act (enacted in 1982) suggest the following program areas for research for out-of-school, dropout youths: basic remedial education, occupational skills training, and financial assistance. Basic Education Although there is no evidence on the effect of labor market preparation programs on basic skills acquisition, there is evidence from the Job Corps and 70001 that programs placing a strong emphasis on GED training can substantially increase the educational attainment of out-of-school youths, as measured by GED attainment. RECOMMENDATION: The importance of basic education as a component of programs for out-of-school youths should be tested systematically. Many programs have placed heavy emphasis on the attainment of a GED (or other educational interventions such as competency-based instruction) for this group and a serious attempt should be made to determine whether the increase in basic education provided through programs does in fact have long-term payoffs for these youths. Occupational Skills Training The results of the Job Corps evaluation suggest that occupational skills training programs can be an effective means of solving some of the structural employment problems of dis- advantaged out-of-school youths, at least of that population of disadvantaged dropout youths served in residential Job Corps centers. The fact that the research to date has not explained the Job Corps' effects in terms of the individual contribution of its many program components or the totality of its residential services, limits the generalizability of the results to other disadvantaged youths in other settings. RECOMMENDATION: Opportunities to enroll in the Job Corps should continue to be provided for the out-of-school youth population.
26 RECOMMENDATION: The effectiveness of the Job Corps should be further studied through systematic evaluations using random assignment. These evaluations should attempt not only to assess the overall effectiveness of the program but also to determine which components of the program are most effective for which groups of youths. Attainment of this latter objective will require some use of random assignment to alternative components within the Job Corps program. RECOMMENDATION: Nonresidential skills training programs for out-of-school youths should be systematically tested and evaluated. Financial Assistance There is a serious question of whether employment and training programs can effectively enroll the out-of-school youths most seriously in need of assistance and hold them in the program for a reasonable amount of time without providing some form of financial assistance. In our review we found no good evidence on this question. RECOMMENDATION: An attempt should be made to test how the number and characteristics of those enrolled in youth employment Programs are affected by the Provision of financial assistance and whether the length of participation varies according to whether assistance is provided. Programs for In-School Youths General research, sponsored in part by YEDPA, highlighted two important facts regarding in-school youths and employment. First, in the last few decades, employment for in-school youths has grown sub- stantially for white youths while black in-school youths have not experienced a similar growth in employment (and for black males the extent of employment while in school actually declined). Second, there is a high correlation between employment while in school and employment and earnings after school. The second point is recognized to be correlational and possibly not causal, but it raises the question of whether providing the means for increasing employment while in-school would reduce the incidence of employment problems after school. The entitlement program provided the potential to give this hypothesis a meaningful test, but that potential could not be fully realized. The entitlement program did show, however, that meaningful, minimum-wage jobs could be provided and that youths would take the jobs in sufficient numbers to change the relative black- white in-school employment rates. Given this step, it seems eminently worthwhile to test the hypothesis further. A test of the effects of in-school employment on later employment need not, however, necessarily come in the form of an entitlement-type program. Indeed, in terms of testing for the effects of such an experience, the research inferences can be more powerful if access to the job experience is provided through random assignment of individuals
27 to either the program or to a control group. Even with random assign- ment to a limited number of part-time jobs for in-school youths during the school year, if the program is focused on those areas where ethnic differentials in employment of in-school youths are high, the results of the test programs would provide evidence as to whether a narrowing of in-school employment differentials will lead to a subsequent narrowing of postschool employment differentials. RECOMMENDATION: Programs should be designed to test whether increased in-school employment leads to greater postschool employment. The tests should be conducted in a form that requires random assignment of individuals to the program or a control group. The evaluation of the test programs should provide sufficiently long-term follow-up data for both participants and control group members to determine long-term postschool effects. m e Summer Youth Employment Program seems to have sufficient political popularity to survive regardless of evaluation research findings or nonfindings. Therefore, attempts should be made to restructure segments of SYEP to provide an opportunity to learn more about whether its resources can be used more effectively. RECOMMENDATION: Attempts should be made to restructure some elements of the Summer Youth Employment Program to sYstemati- cally test whether SYEP elements can be used to enhance basic education sufficiently to reduce school dropout rates or, at least, improve employment chances for those who do drop out anyway. Elements of SYEP could be structured so some skills training is added to the pure work experience in order to determine whether such training enhances the long-term employment effects of the program. Women in Youth Employment Programs Although women constitute half of the participants in employment and training programs, little attention has been given to sex differences either in terms of program needs or outcomes. Yet it is clear that the distinct needs and characteristics of this group have implications for program design. RECOMMENDATION: Programs should be designed to recognize more fully the fact that teenage parenthood often results in restricting the educational, training, and employment oppor- tunities of young women. The benefits of providing child care to encourage greater participation of teenage mothers and more favorable program outcomes should be rigorously tested. In addition, while there is some indication that women benefit more from participation in employment and training programs than men, there
28 is also evidence that such programs may perpetuate occupational segregation. RECOMMENDATION: More research should be done on selective factors that affect the recruitment of women into youth employment programs and the differential treatment by sex in occupational training once in programs. More research is also required on potential nonemployment outcomes of job training for women, such as increased educational attainment, reduced welfare dependency, and reduced early childbearing. A General Research Strategy Under JTPA Having made a series of recommendations regarding types of programs which might be tested, we must hasten to state that we are fully aware of the changed environment in which employment and training programs currently operate under the Job Training Partnership Act (JTPA). Recom- mendations for demonstrations and research must be framed in light of that environment. When considering JTPA from the perspective which has been the primary concern of this committee, three features stand out: 1. The resources devoted to employment and training are con- siderably less than those devoted to YEDPA even though the magnitude of the youth employment problem is at least as great today as it was in the year preceding YEDPA. 2. The continued concern with the employment problems of youths is indicated by the fact that a substantial proportion of the greatly reduced training resources are earmarked for programs enrolling youths. 3. The control of the content of programs and any research to be done concerning them is placed almost completely in the hands of the local Private Industry Councils and Service Delivery Areas (and the state agencies guiding them). In light of these features, we must ask what is likely to be learned from JTPA about how to alleviate the employment problems of youths, "what works for whom" among the youth population. Our answer must, in all honesty, be "very little." The YEDPA experience amply demonstrates that completely decentralized research efforts executed with a minimum of central coordination and technical assistance are likely to yield very little hard evidence on program effectiveness. On the other hand, the very fact that there is a considerable reduction in program activity in the field may create an opportunity for more careful planning and execution of evaluation research than was possible under YEDPA and a greater likelihood of finding the sort of research resources which will generate evidence of high quality. We believe that with relatively small amounts of central resources, a strategy and mechanism for evaluation research under JTPA can be implemented which will considerably enhance the likelihood that reliable evidence on youth programs will be derived from JTPA.
29 The strategy would be to provide through some central mechanism the research designs and technical assistance that would be necessary to add evaluation components to youth programs being underwritten by local Private Industry Councils (PICs) and Service Delivery Areas (SDAs) through JTPA. Adding a small amount of program funds, allocated on a discretionary basis to those PICs and SDAs that agree to cooperate, might induce the localities to make the slight alterations in their program content or procedures necessary for evaluation research. The central agency would have the overall perspective the local PICs and SDAs may lack as to the range of program types that are being undertaken in various localities and would ensure that a reasonable portfolio of quality evaluations was being mounted so that the relative effectiveness of different program types could be assessed. Having this perspective would also enable the central agency to provide infor- mation and technical assistance to the local PICs and SDAs concerning alternative program types, better program procedures, and so on. The central agency would also help to ensure that at least a central core of the evaluation research information collected across sites was reasonably comparable so that cross-program comparisons could eventually be made. The central agency need not be in the federal government itself. The experience with the use of intermediary organizations to organize research and technical assistance under YEDPA, while it was not all positive, was sufficiently good in a number of cases to suggest that this might be an effective medium through which to interject this evaluation research into the JTPA framework. Such organizations now have experience in negotiating with local operating agencies, adapting research designs to local constraints, and combining technical assistance with research guidance. While the major activity could be carried out through an intermediary, some guidance and oversight from the Department of Labor is necessary, as we noted in our earlier discussion of YEDPA; it is not wise to devolve responsibility totally to an intermediary. But with a level of research and evaluation activity that would be only a small fraction of that undertaken under YEDPA, the Department of Labor staff required to oversee inter- mediaries' activities could be quite small. We note also that good evaluative research could provide a sounder basis for the setting of performance standards, a key feature of JTPA. What is important for performance is value added, the improvement in employment and earnings over what it would have been in the absence of the program, and we would argue that this can best (and perhaps only) be established through evaluation research using randomly assigned control groups. These then are just the rough outlines of a strategy and mechanism for evaluation research on youth programs under JTPA. We believe they are compatible with the basic design of JTPA itself and could yield good evidence from the JTPA experience about how an employment and training system can better help alleviate the continuing serious employment problems of sizable proportions of our youths. We must stress that in the absence of such a strategy and mechanism, we believe that several years from now the nation will find itself with several
30 years of JTPA experience but knowing little more about whether or how such programs might help reduce youth employment problems. Research Methods Beyond the general strategy and mechanism just suggested for research and demonstrations under JTPA to test the types of program components we have recommended, we have specific recommendations to make about the actual conduct of research and evaluation activities. The results of our review of YEDPA programs make it obvious that quality in the design and execution of a research project affects the quality of the data and the reliability of any conclusions that are drawn from those data. Poor research designs can make programs look worse than they are, or better than they are, or yield uninterpretable evidence. Poor execution will compromise even the best design. Random Assignment Our review of YEDPA research strongly suggests that much more could have been learned, and more confidence placed in the results, if random assignment had more frequently been used. We believe that not only has the feasibility of random assignment in program research been demon- strated, but that in situations in which program resources are scarce and program effectiveness unproven, it is ethical (see Appendix C). . RECOMMENDATION: Future advances in field research on the efficacy of employment and training programs will require a more conscious commitment to research strategies using random assignment. Randomized experiments should be explicitly authorized as a device for estimating the effects of new projects, program variations, and program components. Furthermore, funding authorities should back this explicit - authorization with firm indications that this is the method of evaluation which is expected. Implementation Research The need for measurement of program implementation in evaluation research is clear. It is as important in social program evaluation as is measurement of dose level in evaluating new drugs. Federal agencies have had substantial experience in eliciting such information, but this information has not always been reasonable in quality, judging from our review of youth employment program evaluations. RECOMMENDATION: Systematic and verifiable information on program implementation should be collected in future research. Better and less expensive methods for obtaining and reporting such information need to be developed.
31 Use of Subjective Measurements as Proxy Variables While one cannot fault a research program for using subjective measurements as proxies for other outcomes, it is a theoretical and methodological challenge to develop measures that have substantial validity and reliability. Indeed, the treatment of subjective measurements has been reconsidered in recent years, and it is generally recognized that common research practices ignore the complexity of the relationships between objective and subjective measurements. RECOMMENDATION: Future researchers should avoid overreliance on subjective measures of program outcomes and devote more resources to studying the relationships that exist between subjective indicators and key objective outcome variables. Postprogram Follow-up m e Job Corps evaluations suggest that some program effects that are not apparent at short-term follow-ups may emerge in the longer term. Research on job placement programs suggests that some, more immediate, postprogram effects may decay rapidly. Together, these pieces of evidence suggest that short-term follow-up data may err either positively or negatively in predicting longer-term program effects. RECOMMENDATION: Future research on the effectiveness of youth employment and training programs should, at least in selected studies, estimate the longer-term effects of these programs by collecting follow-up data for at least 2 years postprogram e Benefit-Cost Studies When evaluations demonstrate that programs have a positive outcome, researchers should recognize that the next question' raised will be whether this positive outcome was worth what it cost to produce it. Thus evaluators should anticipate the eventual need for benefit-cost studies. Such studies, however, need not be a component of every evaluation nor of entire programs, since doing adequate benefit-cost studies is both difficult and costly. The Job Training Longitudinal Survey Data Base One feature of JTPA that is important in regard to program research is the plan to rely heavily on analysis of national data bases to determine the effectiveness of JTPA programs. Present plans are to model the Job Training Longitudinal Survey (JTLS) data base after the previous CLMS. Although its data gathering appeared technically excellent, the CLMS strategy of using nonrandom comparison groups for
32 program evaluation suffered from substantial problems with potential unmeasured biases in its comparison groups. Furthermore, we believe that the plan to use the same strategy in designing the JTLS as a means of obtaining evidence on "what works for whom" is misguided. The evaluations we reviewed that are based on constructed comparison groups is provide strong evidence that this approach to program evaluation seriously flawed; the question of bias . . · · . · · . . ~ In comparison groups so con- structed is virtually impossible to dispel. We believe that the planned JTPA evaluations using the new JTLS will suffer from the same problems. RECOMMENDATION: Planning for the JTLS should give very serious consideration to the selection of randomized control groups. In conclusion, we believe that quality research ought to be recog- nized and ought to be explicit in congressional and agency oversight policy. Special efforts should be made to improve the quality of research and evaluation designs for estimating the impact of youth employment projects. Existing professional guidelines can be used to influence quality of design as well as quality of research execution and reporting. RECOMMENDATION: The committee recommends the following conditions as necessary but not sufficient for quality research: (1) the use of random assignment to participant and control groups and to program variations; (2) reasonable operational stability of the program prior to final assessment of effec- tiveness; (3) adequate sample coverage and low rates of sample attrition; (4) outcome measures that adequately represent the program objectives, both immediate and longer term; and (5) a follow-up period that allows adequate time for program effects to emerge or decay. The General Conduct of Public Policy Research One of the major implications of our review of YEDPA programs and research concerns the conduct of national public policy research and demonstration programs. It was very apparent in our review that many of the problems we faced in attempting to draw inferences from YEDPA research resulted from the fact that under YEDPA attempts were made to combine numerous research objectives with massive service delivery. The consequent tensions, conflicts, and overload on the system inter- fered with the careful planning and conduct of the research and demonstration activities, with the result that the research findings fall short in informing the public policy issues from which YEDPA originated. RECOMMENDATION: In future efforts the objectives of research and demonstration should be more clearly and selectively focused on essential public policy issues and clearly separated_
33 from the objectives of massive service delivery. magnitude of the effort and the expectation of results should be more in scale with limitations of time, money, and staff resources. Dilemmas Confronting the Employment and Training System In closing, we return to the two fundamental dilemmas with which we began this discussion of implications and recommendations for the system of employment and training in the United States. The employment and training system is trying in large part to do what the education system should be doing but, for some significant segment of the youth population, apparently fails to do. Yet the employment and training system has not attained stability of funding, professionalization of staff, and delineation of authority, in short, institutionalization of the sort that has given the educational system its accepted place in the mainstream of American life. As a result, in most communities, organizations involved in employment and training are considered marginal. The educational system, on the other hand, should not be taken as an exact model for the institutionalization of the employment and training system, since it has not yet found an effective way to prepare a substantial segment of the youth population for later employment. For the most part, the youth programs of the employment and training system have been specifically targeted toward special segments of the youth population, often those perceived as most disadvantaged. Given that the major rationale for youth programs within the employment and training system is to assist those whom the educational system has failed to prepare for work, this target seems a sensible means to focus resources on those in greatest need. The problem is that this very targeting tends to create an image of the programs as designed only for "failures;" both the programs themselves and their clientele become stigmatized in the process. The staff of the programs may come to feel stigmatized as well and this can exacerbate Problems of recruitment. . retention, and management. Even the potential target group members can come to share the views of the broader community about the inherent marginality of these programs and the stigmatizing effects of partici- pating in them, and it becomes increasingly difficult to enlist them in the programs and to keep them participating for sufficient time for the "program treatment to take hold." Yet, experience has shown that when programs are not targeted, the resources tend to be shifted rapidly to the more advantaged, better prepared, easier to handle segment of the youth population--those who have far less need for help with potential employment problems. These fundamental dilemmas pose a major impediment to solving the serious unemployment problem of youths, and we emphasize again the need for a direct study of the roles and relationships of the education and the employment and training systems.
