Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 1
Summary, Conclusions, and Recommendations In response to high levels of unemployment and other employment- related problems of American youth, the federal government enacted the Youth Employment and Demonstration Projects Act (YEDPA; P.L. 95-93) in 1977. This legislation established a variety of employment, training, and demonstration programs for youth. With the passage of YEDPA, federal spending on employment programs earmarked for youth approxi- mately doubled, bringing the total to about $2 billion per year. Besides this substantial commitment of funds, YEDPA was unique in its explicit commitment of substantial resources to research and evaluation efforts intended to test alternative ways of meeting the needs of youth. YEDPA programs ended in 1981 with the change in presidential administration. At that time the products of YEDPA's research had not been comprehensively evaluated, and there were questions about what had been learned from this undertaking. Two years later the U.S. Department of Labor (DOL) requested that the National Research Council (NRC) review the products of the YEDPA research effort. The following eight chapters detail the findings from that review; this chapter summarizes our findings, conclusions, and recommendations. THE NATURE OF THE YOUTH EMPLOYMENT PROBLEM In focusing on the nature of the youth employment problem at the end of the 1970s, as YEDPA began, it is helpful to note that the majority of out-of-school youths found jobs and, when they lost or left a job, found another one without a long period of unemployment. It has long been recognized, however, that youth employment is more sensitive to the cycles of economic activity (recession and expansion) than is that of adults: the percentage decline in employment during a recession is generally greater for youths than for adults, but the percentage increase in employment during recovery is also greater than for adults. What was much more disturbing was the worsening long-term trend--which emerged clearly in the 1970s--in the employment rates of youths relative to adults, even when measured from the peak of expansion of one business cycle to the next. Furthermore, the data reveal substantial differences in that trend according to the race and sex of the youths; the long-term 1

OCR for page 1
\ 2 trend in employment was much worse for minority, principally black, groups than it was for the white majority. The most revealing picture emerges from data on employment-to- population rates that separate subgroups of the youth population by race, sex, and school status. Those data show that from 1964 to 1978 there was a growing gap between the employment-to-population ratios of white and black youths, for both in-school and out-of-school youths. For in-school youths, employment rates for whites were growing while rates for black males were falling and those for black females were not growing as fast, yielding an increasing black-white gap in employment rates. While some may regard in-school employment opportunities of lesser importance, researchers have found that, holding measured char- acteristics constant, those youths who work during school years have higher employment rates and wages after their school years. This finding may simply reflect that youths who are more motivated (an unmeasured characteristic) both seek work more energetically while in school and seek, find, and perform work better after school, but the possibility cannot be excluded that the in-school work experience per se enhances later employment and earnings. If so, the growing black- white in-school employment gap foreshadows a later out-of-school black-white gap in employment and earnings. The black-white employment gap for out-of-school youths also grew during this period for both males and females. And this occurred while the previously existing gap between blacks and whites in years of schooling attained was substantially reduced. Given the generally acknowledged positive relationship between years of schooling attained and employment and earnings, this reduction should have narrowed the employment gap between blacks and whites; however, it did not do so, or not sufficiently to offset other factors widening the gap. Research further shows that within this out-of-school group, employment problems (lower chances of getting a job, lower wages when a job is obtained, higher chances of losing a job, longer periods of remaining without a job having lost one) are highly concentrated among minority-group, inner-city, low-income, and high school dropout youths. For those with combinations of these characteristics the problems are compounded. Finally, it is apparent that young unwed mothers have very serious and special problems in qualifying for, finding, and holding jobs, espe- cially at earnings sufficient for their families' economic viability. Since the end of YEDPA in 1981 the United States has experienced both the deepest recession since the 1930s, which reached its trough in 1983, and a sharp economic recovery. There has also been a notable decline in the absolute size of the youth population since it reached its peak in the early 1980s. It seems reasonable to ask in light of these events if the nature of the youth employment problem has sub- stantially changed, in its general configuration, from what it was in 1978, as outlined above, when YEDPA began. Although exact comparisons cannot be made (comparable data are not yet available), it appears that at the beginning of 1985 the employment problems of youths were of about the same magnitude and configuration as they were in 1978, including racial differentials.

OCR for page 1
3 LIMITATIONS OF THIS REVIEW Our ability to respond to our charge was necessarily constrained by the nature of the material we had to work with. Although we searched the literature available beyond the reports generated as part of the YEDPA process and consulted with people experienced with youth programs and related research, we had to rely almost exclusively on the reports of particular YEDPA youth demonstration projects to assess the effec- tiveness of youth programs. The exceptions were studies of three programs that began before YEDPA, the Job Corps, the Summer Youth Employment Program, and Supported Work. We have attempted to test the individual YEDPA research reports against reasonable standards of scientific quality with respect to both the data collected and the methods used to measure program effects. The reports that met such standards were not necessarily evenly dis- tributed over the range of youth programs and target groups. Thus we could not address certain issues with respect to the role and effec- tiveness of youth employment and training programs because of a lack of reliable evidence. Since we were always in the position of examining these programs through the lens of their respective research reports, it is important that we clearly distinguish between the quality of the research and the (sometimes unobservable) quality of the programs themselves. In some instances we found reliable evidence, both positive and negative, from which to draw conclusions; in other instances the available evidence was not sufficiently reliable for us to draw any conclusion, one way or the other. Readers should be careful not to confuse a conclusion about the failure of research to provide adequate evidence with a conclusion that a particular program, itself, was ineffective or failed in some manner. A conclusion of noneffectiveness requires evidence, just as a conclusion of effectiveness does. In the absence of reliable evidence, no inference is possible. In addition to the above limitations, our ability to draw firm conclusions was further constrained by two conditions that affected the implementation of YEDPA and, particularly, the conduct of the research. First, YEDPA programs and research were mounted in considerable haste and in a period in which many other employment and training and research efforts were going on, so that both program and research resources were stretched very thin. (There are a few notable exceptions to this generalization, e.g., Job Corps and Supported Work.) Second, with the change of administration in 1981, less than 3 years after YEDPA's quick start-up and troubled implementation, both programs and evaluation efforts were abruptly halted. As a consequence of these two factors, most of the data on which evaluations were based, again with a few notable exceptions, were gathered at a stage at which programs had not been stabilized. As a further consequence, relatively few program evaluations provide data for long postprogram periods: virtually all of the YEDPA project evaluations had only 3- to 8-month postprogram follow-ups. Only two evaluations had as long as a 3-year follow-up (Job Corps and Supported Work). Our review suggests that longer-term follow-ups are important

OCR for page 1
4 in determining the time pattern of program effects, some of which decay rapidly and some of which emerge slowly. Further limiting our ability to draw firm conclusions were the serious problems of researchers in creating reasonable comparison groups and preventing sample attrition over waves of the data collection. m ese problems sharply reduced the number of studies we could review and put in question the reliability of the results of several others. As a result of these limitations, our coverage of YEDPA programs is uneven and not necessarily representative of overall youth program activity during that period. In many cases conclusions about the effects of specific types of programs are based on only one or two evaluations, in other cases there is no reliable evidence for any con- clusion of program effect. In addition, the quality of evidence varies, sometimes supporting strong conclusions, sometimes merely suggesting the direction of program effects. In presenting our conclusions, therefore, we try to indicate the source of the evidence and the degree of its reliability, and we distinguish lack of reliable evidence from lack of evidence per se. A final limitation of this review concerns the very magnitude of YEDPA and Comprehensive Employment and Training Act (CETA) programs from 1977 through 1981. It has been estimated that in 1979 as much as two-fifths of all jobs held by black teenagers were in government employment and training programs. Thus, even when comparison groups were reasonably created, there may well have been substantial amounts of employment, training, and related effects from federal programs among the comparison group members. To the degree this problem is serious and undetected in the evaluation data, the participant- comparison contrasts will underestimate the impact of the programs. FINDINGS AND CONCLUSIONS ON PROGRAM IMPLEMENTATION The conditions under which YEDPA was implemented severely con- strained both the potential effectiveness of the programs themselves in reaching their objectives and the related research and demonstration activities that sought to evaluate program effects and to create a reliable knowledge base for future youth programs. Implementation was affected by: (1) the size and implicit duality of the YEDPA service- research mandate; (2) the congressional and executive time schedules imposed on YEDPA program operations and research results; and (3) the instability of the service delivery system due to fluctuations in employment and training policy, regulations, and funding levels. The combination of these three factors was significant in determining the course of YEDPA at both the national and local levels. The duality of the YEDPA mandate, which was inherent in the enabling legislation, stemmed from the charge to mount new and very large service delivery programs quickly and at the same time to design and conduct comprehensive research and evaluation. Either of these tasks by itself would have been a sizable and complex endeavor; taken together they burdened the system not only by their sheer magnitude,

OCR for page 1
s but by their diversity of purpose, at times pitting the interests of program delivery against those of research and knowledge development. The imposition of two consecutive time limits also constrained the implementation of YEDPA programs and research. The first, imposed by the legislation itself, required that YEDPA be sufficiently operational within 1 year of passage to warrant congressional reauthorization; and the second, imposed by the executive office, required that within 2 years the results of YEDPA program research be adequate to inform the Vice President's Task Force on Youth Employment for its subsequent report to Congress. These limits put tremendous pressure on the national office of the Department of Labor to get both the programs and the research under way immediately and foreshortened the time available for careful planning of either the programs or the research on their effects. The third major factor constraining YEDPA was one that overrides YEDPA itself, and of which, in fact, YEDPA is a prime example: the tremendous fluctuations from one administration to the next, and often from one fiscal year to the next, in employment and training policy, regulations, and most importantly, funding levels. This instability, perhaps more than any other factor, undermined the employment and training system, particularly at the local level, where in response to such changes adjustments in all aspects of program operations ultimately have to be made. Such fluctuations precluded a more stable and orderly development and institutionalization of the youth employment system. Given the instability of the employment and training system, together with the implementation requirements of YEDPA, it was somewhat unreal- istic to expect that within 3 years these programs would be fully operational and ready to prove their effectiveness. CONCLUSION: The YEDPA legislation created a program that combined too short a time schedule with too many different program elements and objectives. The demand to quickly implement the full range of elements impaired the quality of many of the Programs. In addition, the pressure to obtain a wide range of research results on those programs within a short time compounded the problem and resulted, in many cases, in poor research on hastily constructed programs. It may be that the lack of proven effectiveness of many programs is due as much to the instability of the system as to the inherent nature of the Programs. National Office Management of YEDPA The tasks of designing and implementing YEDPA programs and research activities strained the capacity of DOL's Office of Youth Programs (OYP) given its very small staff and limited research capability. In response to these demands OYP created a system of indirect management, delegating substantial responsibility for the design, implementation, management, and evaluation of major YEDPA demonstration programs to private nonprofit intermediary organizations. In addition OYP extended

OCR for page 1
6 the staff's research capability through agreements with other research units within the Department of Labor, with the Educational Testing Service (ETS), and with Brandeis University's Center for Employment and Income Studies (CEIS). Agreements with other federal agencies to operate other portions of YEDPA were another means of expanding the YEDPA management structure. As a means of quickly disbursing funds and implementing programs under severe time constraints, the agreements with other parties were expedient. As a means of managing programs and research, however, that approach was not very effective. Of the four intermediary organiza- tions, only two, the Manpower Demonstration Research Corporation (MDRC) and the Corporation for Public/Private Ventures (CPPV), produced competent research on program Impacts on parclalpanc~. The results of programs operated by the other two intermediaries, the Corporation for Youth Enterprises (CYE) and Youthwork, organizations created to manage YEDPA demonstrations, are largely unknown because those organizations, although they did produce reports, did not attempt to assess program impact in a quantitative manner that could be evaluated. The results of programs operated under interagency agreements are also unknown either because no program research was conducted or because research reports failed to meet our criteria of scientific evidence. In general, considering the amount of YEDPA funds channeled through intermediaries and interagency agreements, remarkably little is known about the programs or their results. The results of OYP's other agreements were also mixed. Agreements with other ETA research units to incorporate a youth sample in the National Longitudinal Survey (NLS) and YEDPA samples in the Continuous Longitudinal Manpower Survey (CLMS) data bases produced useful results. m e agreement with the Educational Testing Service to establish a large national data base on YEDPA programs and participants, however, was poorly planned and implemented. The support provided by the Center for Employment and Income Studies was effective in documenting and assessing YEDPA programs, but CEIS's technical assistance and oversight of YEDPA research were not--and given the scale of the task, could not have been--sufficient to ensure the comprehensiveness of its research design or the quality of its execution, at least as evidenced by our review. CONCLUSION: The resources provided to the Office of Youth Programs were woefully insufficient to its charge to mount and manage YEDPA programs and research. Lacking research staff and resources, GYP delegated responsibility for the design and evaluation of large portions of the YEDPA demonstration research agenda to parties outside the Department of Labor. The resulting management structure lacked sufficient control at the center to provide coherence in program objectives and policies, to monitor developments, and to ensure accountability. These conditions had their greatest impact at the local level. With its additional reporting requirements, increased federal control over program design and target groups, increased services to youths, and the demands of research and demonstration, YEDPA imposed substan

OCR for page 1
7 tial administrative burdens on local prime sponsors. The competing demands of a substantially increased Public Service Employment Program and the regular programs under the Comprehensive Employment and Training Act (CETA) placed YEDPA in a strained local environment. The demands of the YEDPA agenda and the time schedule for their implementation severely hampered local program planning and assembly. The increased federal program requirements compressed the already shortened planning and approval process, which was complicated by the lack of clear guidance from the national office. To the extent that these conditions interfered with the careful planning of services, selection of program operators, coordination with other parties in the local service delivery network, identification and recruitment of participants, and assessment of local need for these programs, there were consequences for each subsequent stage of program operations and, ultimately, for the success of the programs. Despite these problems, YEDPA did succeed in mounting new programs, at double the size of previous youth programs, and in providing large numbers of disadvantaged youths with jobs. Furthermore, the evidence indicates that those jobs were generally well supervised and worthwhile experiences for both the participants and their employers. Targeting, Recruiting, and Retaining Youths Research on YEDPA programs cites numerous problems with targeting and recruiting sufficient numbers of eligible youths from the designated target groups: in-school and out-of-school youths meeting criteria of economic disadvantage. This problem was attributed in part to the short planning time and the resulting tendency of prime sponsors to base needs assessments on outdated information and to overestimate target group size. In addition, a legislative requirement that youths in regular CETA programs be served at the same levels as previously and the strict eligibility requirements for some YEDPA programs may have resulted in a shortage of eligible youths in some local areas. A related problem cited in many reports was the tendency of program operators to serve the least disadvantaged of the eligible youths, leaving the most disadvantaged and needy without services. This phenomenon, known as "creaming, n reflects the tradeoff that many program operators perceived between serving those most in need versus those more likely to succeed. It raises both equity and efficiency issues to the extent that the less disadvantaged might have achieved the same employment results without benefit of the programs. An example of this tradeoff, and one representing a major dilemma for YEDPA, was the targeting of services for dropout versus in-school youths. An inherent tendency of many YEDPA Programs was to Gravitate to the in-school population. A 22 percent YEDPA set-as~de for linkages with the schools was an additional incentive for local prime sponsors to target in-school youths. The dilemma was that the group most in need of employment services--the dropout population--was also the group that was hardest to recruit and to serve successfully. Conversely, the group most easily recruited and served--the in-school population--was , , _ _ _ , _ , _ _ . _ , _ _

OCR for page 1
8 the group for which services were less critical {or at least for whom the problem was less clearly defined). Though it is widely recognized that of all youth employment problems those of school dropouts are the most serious, there appears to be a tendency for employment and training programs to avoid serving this group. Many programs designed specifically to serve dropouts (either through school-conditioned work or through alternative education, training, or work settings) often had difficulties recruiting them and, once they were recruited, experienced difficulties retaining them in Other projects designed to serve either in-school or the programs. _ _ out-of-school youths, facing similar difficulties, evolved toward serving in-school youths, recasting the dropout problem in terms of prevention instead of remediation. CONCLUSION: In meeting the increased enrollment and rapid implementation requirements of YEDPA, prime sponsors when given the option of serving either in-school or out-of-school youths tended to focus resources on the in-school population. Even when programs were specifically targeted to dropouts, they often had difficulty in recruiting and retaining them. As a result, the question of how to reach and serve dropout youths effectively was largely unanswered by YEDPA. Enrollment of Young Women Most of the youth programs we reviewed enrolled substantial numbers of young women, and evidence from some programs suggests more positive effects for young women than for young men. Many of the programs, how- ever, encountered difficulties maintaining enrollment of economically disadvantaged young women, apparently because of the high incidence of teenage pregnancy and childbearing. Most program operators and evaluators apparently overlooked this characteristic of the target population, and so there is little direct evidence on the effect of pregnancy and childbearing on program participation or on the effect of program participation on subsequent pregnancy and childbearing. Evidence from one demonstration program designed to serve pregnant and parenting teenagers under the age of 18 (Project Redirection) is equivocal on the effect of a service-coordination strategy in reducing pregnancy and increasing subsequent employment and earnings. Neither is there evidence to date that would allow clear distinctions to be made as to the effects of alternative decisions about pregnancy resolu- tion, i.e., birth, adoption, or abortion, on other program outcomes. CONCLUSION: Most youth programs had substantial enrollments of young women. Many, especially those serving older teenagers, encountered difficulties maintaining enrollment of economically disadvantaged young women because of the high incidence of childbearing.

OCR for page 1
9 FINDINGS AND CONCLUS IONS ON PROGRAM EFFECTIVENES S Our conclusions are based on a review of 28 programs, including both~demonstration projects and regular youth programs. In this review we proceed by type of program and within type of program by target group, out-of-school youths (both dropouts and graduates) or in-school youths. Occupational Skills Training Programs Our conclusions on skills training are based on evaluations of only two programs, the residential Job Corps program for out-of-school, mainly dropout, youths and an apprenticeship program for in-school youths. Occupational Skills Training Programs for Out-of-School Youths The Job Corps The Job Corps stands out in our review as a program for which there is strong evidence regarding program effectiveness. The quality of the evaluation reviewed, in terms of sample sizes, comparison group method- ology, sample attrition, and the measurement of outcome variables, lends confidence to these conclusions. The Job Corps is a comprehensive program providing occupational skills training, basic (and remedial) education, counseling, health care, and job placement to youths more disadvantaged than typical participants in youth programs. Although the contribution of each component part of the program is not known, it is clear that as a whole the program has provided positive benefits to participants in terms of employment, earnings, and education. On average, for up to 3-1/2 years after participation, Job Corps enrollees earned 28 percent more per year ($567 in 1977 dollars) and worked 3 weeks more per year than nonparticipant comparisons. In addition, participation reduced receipt of welfare and unemployment by 2 weeks and 1 week per year, respectively. CONCLUSION: Job Corps participation resulted in gains in employment and earnings in the postprogram Period and in declines in receipt of welfare and unemployment payments. These positive effects persisted at a relatively stable rate for up to 4 years after youths left the program. Participation in the Jobs Corps increased the probability that Job Corps enrollees would receive a high school diploma or a General Equivalency Diploma (GED). Specifically, the probability was .24 for participants compared with .05 for comparisons. CONCLUSION: Job Corps participation resulted in gains in educational attainment during the program as measured by completion of GEDs.

OCR for page 1
10 The Job Corps evaluation measured criminal activity and found that Job Corps participation significantly reduced the criminal activity of participants. CONCLUSION: Jobs Corps participation resulted in decreases in criminal activity, as indicated by rates of arrest during program Participation and decreases in seriousness of crimes in the postprogram period. In addition, although the Job Corps by far exceeded the per- participant costs of other youth programs, the benefit-cost analysis indicated a net benefit. CONCLUSION: The benefits of Job Corps participation in terms of increased employment and earnings and decreased crime and transfer payments exceeded the costs by a sizable margin ($2,300) per enrollee. Other Occupational Skills Training Programs Although there is substantial evidence on the effectiveness of the Job Corps, it is not known which of its several component parts con- tribute to which effects; how much (if any) is due to the self-selection factors of youths who enroll in the program; or how program components and participant characteristics interact. The residential requirement of the Job Corps, in particular, is untested as a factor in explaining the program's effectiveness and precludes generalizing its results to nonresidential settings. Nonresidential skills training would certainly be less expensive to operate than the Job Corps; however, YEDPA Produced no reliable evidence on the effectiveness of occupational skills training Provided in a nonresidential setting for out-of-school , youths generally or for the severely disadvantaged population of out-of-school youths served by the Job corps. - Occupational Skills Training Programs for In-School Youths m e committee found few studies of occupational skills training programs operated under YEDPA. This was due, at least in part, to a concern that participants require a sufficiently high level of academic preparation to be able to gain from such training. We reviewed only one program providing occupational skills training to in-school youths. This program (New Youth Initiatives in Apprentice- ship) was designed to prepare high school seniors for registered apprenticeships after graduation. CONCLUSION: m ere is only very limited evidence from YEDPA on the effectiveness of skills training for in-school youths. The only program that provided evidence of reasonable quality showed no effect on participants' postprogram earnings or

OCR for page 1
11 employment rates. However, this Program was so special in nature and participant characteristics that one would not wish to base general conclusions about skills training for in-school youths on the evidence from this program alone. Labor Market Preparation Programs The programs classified as labor market preparation programs in our review were very heterogeneous in terms of their services and activities, but they shared the long-term goal of preparing youths for their future work lives by improving their personal skills, knowledge, and attitudes toward the work place. Activities ranged from career exploration and job search assistance to remedial education and combinations of work experience and classroom training. The programs also varied greatly in intensity and duration, ranging from 5 to 35 hours per week and from 10 weeks to 1 year. Programs for Out-of-School Youths The Studies of labor market preparation programs serving out-of-school youths tended to provide sounder evidence on program effectiveness than did studies of programs serving in-school youths. In the 3- to 8-month postprogram period, participants often exhibited significantly better employment outcomes than nonparticipants. It is particularly trouble- some, however, that the term "out-of-school youths" is used to refer to high school graduates as well as dropouts: the programs providing reliable evidence served varying mixtures of the two groups and did not produce separate analyses of effects. This lack of separate analysis for dropouts and graduates conditions our confidence in the evidence because program outcomes (e.g., employment and earnings, and educational attainment) might be influenced by whether the youths had completed high school. CONCLUSION: YEDPA programs providing labor market preparation for out-of-school youths resulted in some positive effects on employment in the 3 to 8 months following Program Participation (Alternative Youth Employment Strategies, the Recruitment and Training Program, Project STEADY). There are no reliable data, . however, to determine whether these short-term gains are sustained over the long run or whether such programs had any effects on educational attainment or other coals, such as . reduced crime and substance abuse. Programs for In-School Youths Some of the reports on in-school programs we reviewed indicated that program operators did not expect to directly affect the youths' postprogram earnings or employment; instead, they concentrated on other

OCR for page 1
23 attempt to influence the schools to broaden their constituency to include disadvantaged youths. Despite the agreements negotiated under YEDPA to involve the schools in youth employment programs, we found little evidence of successful mutual efforts. CONCLUSION: The relationship between the employment and training system and the school system remains problematic. Despite some common objectives and client groups and efforts to bring the two systems together in the service of those objectives and clients, they remain largely separate, and the potential benefits of mutual efforts are largely unrealized. An example of this problem, and one of great importance to this committee, is the problem of school dropouts--a group of legitimate concern to both the schools and the employment and training system and a group that neither has been able to serve effectively. No other youth group faces the employment problems, both immediate and long term, faced by school dropouts, and particularly those who are minority group members. The YEDPA approach to the dropout problem was twofold: (1) prevention of the problem by targeting services to youths at risk of dropping out and giving them incentives to remain in school and (2) remediation of the problem by targeting services directly to dropouts in a way that encouraged return to school or an alternative education. Our review of youth programs found no evidence of effective means of either dropout prevention or remediation. We observed instead the severe problems of schools and other program operators in recruiting and retaining dropout youths and the tendency of those programs to focus their services on the more easily recruited and served population of in-school youths. CONCLUSION: Of all youth groups, school dropouts face the most serious employment problems. Because of problems in recruiting and serving dropouts, however, the focus of youth research and demonstration under YEDPA was unduly directed to in-school youths and high school graduates. As a result, the question of how to reach and serve dropout youths remains unanswered. IMPLICATIONS AND RECOMMENDATIONS Two basic issues--the problem of school dropouts and the relations between the schools and the employment and training system--remain, in the committee's view, fundamental dilemmas confronting the youth employ- ment and training system in the United States. We begin our discussion of implications with our recommendations for future youth policy with regard to dropouts. We recognize that there is a long history of research and program attempts to understand and deal with the problems of school dropouts. And yet, as our review strongly suggests, dropouts, particularly minority group dropouts, remain as the segment of the youth population

OCR for page 1
24 with the most serious employment problems. Attempts, both preventive and remediative, to address the employment problems of this group, with the exception of the Job Corps, have been ineffective. Despite these efforts we know little more about dropouts now than we did before YEDPA, not only in terms of their responses to employment and training programs, but more fundamentally in terms of the factors--economic, social, and psychological--affecting their dropout and subsequent educational and employment behaviors. RECOMMENDATION: The committee strongly recommends that school dropouts be given priority status for employment and training programs and research. Program efforts should be shaped to test systematically the alternative methods of addressing the education and employment needs of these youths, and research should focus on the underlying determinants of the dropout phenomenon. Another major implication of our review concerns the marginal role of the youth employment and training system, its relation to the school system, and the two systems' relation to the broader society in addressing youths' educational and employment needs. We recognize the inherent tension between the above recommendation to give priority status to dropouts and the suggestion that the employment and training system, partly because of such targeting of services, has isolated itself from the broader society. This is a complex problem to which we have no solution. We believe, however, that it is an important aspect of the youth employment problem and that it bears serious consideration and further study. RECOMMENDATION: _ In order to rid employment and training Pro grams of the stigma which has plagued them and their partici- pants, the committee strongly recommends that attempts be made to target services for disadvantaged youths in ways that will not isolate them but rather integrate them into mainstream institutions and activities. The role of the school system and the relation between the schools and the youth employment and training system are critical in resolving this problem. The committee therefore recommends a direct study of the appropriate role of the youth employment and training system, and its relation to the educational system, in alleviating the employment problems of those youths most in need of assistance. Youth Programs Committees such as ours invariably recommend further program research and testing. Unless the problems addressed by the programs have disappeared or been substantially ameliorated or unless social priorities have shifted sharply, such recommendations should in good conscience be made. We are hesitant, however, to prescribe program approaches and techniques as lessons from experience. In our con

OCR for page 1
25 sidered judgment, the clearest lesson from the YEDPA experience concerning effective programs is that much remains to be learned about dealing with youth employment problems. In our judgment, it is better by far to admit that knowledge is lacking than to assume on the basis of scant knowledge that we know what works best for whom. Therefore, our recommendations on programs and program research are closely tied to the evidence we reviewed. In this section we first present our program recommendations for out-of-school youths, then for in-school youths, and finally for programs serving women. Programs for Out-of-School Youths The results of our review and the present conditions of the new Job Training Partnership Act (enacted in 1982) suggest the following program areas for research for out-of-school, dropout youths: basic remedial education, occupational skills training, and financial assistance. Basic Education Although there is no evidence on the effect of labor market preparation programs on basic skills acquisition, there is evidence from the Job Corps and 70001 that programs placing a strong emphasis on GED training can substantially increase the educational attainment of out-of-school youths, as measured by GED attainment. RECOMMENDATION: The importance of basic education as a component of programs for out-of-school youths should be tested systematically. Many programs have placed heavy emphasis on the attainment of a GED (or other educational interventions such as competency-based instruction) for this group and a serious attempt should be made to determine whether the increase in basic education provided through programs does in fact have long-term payoffs for these youths. Occupational Skills Training The results of the Job Corps evaluation suggest that occupational skills training programs can be an effective means of solving some of the structural employment problems of dis- advantaged out-of-school youths, at least of that population of disadvantaged dropout youths served in residential Job Corps centers. The fact that the research to date has not explained the Job Corps' effects in terms of the individual contribution of its many program components or the totality of its residential services, limits the generalizability of the results to other disadvantaged youths in other settings. RECOMMENDATION: Opportunities to enroll in the Job Corps should continue to be provided for the out-of-school youth population.

OCR for page 1
26 RECOMMENDATION: The effectiveness of the Job Corps should be further studied through systematic evaluations using random assignment. These evaluations should attempt not only to assess the overall effectiveness of the program but also to determine which components of the program are most effective for which groups of youths. Attainment of this latter objective will require some use of random assignment to alternative components within the Job Corps program. RECOMMENDATION: Nonresidential skills training programs for out-of-school youths should be systematically tested and evaluated. Financial Assistance There is a serious question of whether employment and training programs can effectively enroll the out-of-school youths most seriously in need of assistance and hold them in the program for a reasonable amount of time without providing some form of financial assistance. In our review we found no good evidence on this question. RECOMMENDATION: An attempt should be made to test how the number and characteristics of those enrolled in youth employment Programs are affected by the Provision of financial assistance and whether the length of participation varies according to whether assistance is provided. Programs for In-School Youths General research, sponsored in part by YEDPA, highlighted two important facts regarding in-school youths and employment. First, in the last few decades, employment for in-school youths has grown sub- stantially for white youths while black in-school youths have not experienced a similar growth in employment (and for black males the extent of employment while in school actually declined). Second, there is a high correlation between employment while in school and employment and earnings after school. The second point is recognized to be correlational and possibly not causal, but it raises the question of whether providing the means for increasing employment while in-school would reduce the incidence of employment problems after school. The entitlement program provided the potential to give this hypothesis a meaningful test, but that potential could not be fully realized. The entitlement program did show, however, that meaningful, minimum-wage jobs could be provided and that youths would take the jobs in sufficient numbers to change the relative black- white in-school employment rates. Given this step, it seems eminently worthwhile to test the hypothesis further. A test of the effects of in-school employment on later employment need not, however, necessarily come in the form of an entitlement-type program. Indeed, in terms of testing for the effects of such an experience, the research inferences can be more powerful if access to the job experience is provided through random assignment of individuals

OCR for page 1
27 to either the program or to a control group. Even with random assign- ment to a limited number of part-time jobs for in-school youths during the school year, if the program is focused on those areas where ethnic differentials in employment of in-school youths are high, the results of the test programs would provide evidence as to whether a narrowing of in-school employment differentials will lead to a subsequent narrowing of postschool employment differentials. RECOMMENDATION: Programs should be designed to test whether increased in-school employment leads to greater postschool employment. The tests should be conducted in a form that requires random assignment of individuals to the program or a control group. The evaluation of the test programs should provide sufficiently long-term follow-up data for both participants and control group members to determine long-term postschool effects. m e Summer Youth Employment Program seems to have sufficient political popularity to survive regardless of evaluation research findings or nonfindings. Therefore, attempts should be made to restructure segments of SYEP to provide an opportunity to learn more about whether its resources can be used more effectively. RECOMMENDATION: Attempts should be made to restructure some elements of the Summer Youth Employment Program to sYstemati- cally test whether SYEP elements can be used to enhance basic education sufficiently to reduce school dropout rates or, at least, improve employment chances for those who do drop out anyway. Elements of SYEP could be structured so some skills training is added to the pure work experience in order to determine whether such training enhances the long-term employment effects of the program. Women in Youth Employment Programs Although women constitute half of the participants in employment and training programs, little attention has been given to sex differences either in terms of program needs or outcomes. Yet it is clear that the distinct needs and characteristics of this group have implications for program design. RECOMMENDATION: Programs should be designed to recognize more fully the fact that teenage parenthood often results in restricting the educational, training, and employment oppor- tunities of young women. The benefits of providing child care to encourage greater participation of teenage mothers and more favorable program outcomes should be rigorously tested. In addition, while there is some indication that women benefit more from participation in employment and training programs than men, there

OCR for page 1
28 is also evidence that such programs may perpetuate occupational segregation. RECOMMENDATION: More research should be done on selective factors that affect the recruitment of women into youth employment programs and the differential treatment by sex in occupational training once in programs. More research is also required on potential nonemployment outcomes of job training for women, such as increased educational attainment, reduced welfare dependency, and reduced early childbearing. A General Research Strategy Under JTPA Having made a series of recommendations regarding types of programs which might be tested, we must hasten to state that we are fully aware of the changed environment in which employment and training programs currently operate under the Job Training Partnership Act (JTPA). Recom- mendations for demonstrations and research must be framed in light of that environment. When considering JTPA from the perspective which has been the primary concern of this committee, three features stand out: 1. The resources devoted to employment and training are con- siderably less than those devoted to YEDPA even though the magnitude of the youth employment problem is at least as great today as it was in the year preceding YEDPA. 2. The continued concern with the employment problems of youths is indicated by the fact that a substantial proportion of the greatly reduced training resources are earmarked for programs enrolling youths. 3. The control of the content of programs and any research to be done concerning them is placed almost completely in the hands of the local Private Industry Councils and Service Delivery Areas (and the state agencies guiding them). In light of these features, we must ask what is likely to be learned from JTPA about how to alleviate the employment problems of youths, "what works for whom" among the youth population. Our answer must, in all honesty, be "very little." The YEDPA experience amply demonstrates that completely decentralized research efforts executed with a minimum of central coordination and technical assistance are likely to yield very little hard evidence on program effectiveness. On the other hand, the very fact that there is a considerable reduction in program activity in the field may create an opportunity for more careful planning and execution of evaluation research than was possible under YEDPA and a greater likelihood of finding the sort of research resources which will generate evidence of high quality. We believe that with relatively small amounts of central resources, a strategy and mechanism for evaluation research under JTPA can be implemented which will considerably enhance the likelihood that reliable evidence on youth programs will be derived from JTPA.

OCR for page 1
29 The strategy would be to provide through some central mechanism the research designs and technical assistance that would be necessary to add evaluation components to youth programs being underwritten by local Private Industry Councils (PICs) and Service Delivery Areas (SDAs) through JTPA. Adding a small amount of program funds, allocated on a discretionary basis to those PICs and SDAs that agree to cooperate, might induce the localities to make the slight alterations in their program content or procedures necessary for evaluation research. The central agency would have the overall perspective the local PICs and SDAs may lack as to the range of program types that are being undertaken in various localities and would ensure that a reasonable portfolio of quality evaluations was being mounted so that the relative effectiveness of different program types could be assessed. Having this perspective would also enable the central agency to provide infor- mation and technical assistance to the local PICs and SDAs concerning alternative program types, better program procedures, and so on. The central agency would also help to ensure that at least a central core of the evaluation research information collected across sites was reasonably comparable so that cross-program comparisons could eventually be made. The central agency need not be in the federal government itself. The experience with the use of intermediary organizations to organize research and technical assistance under YEDPA, while it was not all positive, was sufficiently good in a number of cases to suggest that this might be an effective medium through which to interject this evaluation research into the JTPA framework. Such organizations now have experience in negotiating with local operating agencies, adapting research designs to local constraints, and combining technical assistance with research guidance. While the major activity could be carried out through an intermediary, some guidance and oversight from the Department of Labor is necessary, as we noted in our earlier discussion of YEDPA; it is not wise to devolve responsibility totally to an intermediary. But with a level of research and evaluation activity that would be only a small fraction of that undertaken under YEDPA, the Department of Labor staff required to oversee inter- mediaries' activities could be quite small. We note also that good evaluative research could provide a sounder basis for the setting of performance standards, a key feature of JTPA. What is important for performance is value added, the improvement in employment and earnings over what it would have been in the absence of the program, and we would argue that this can best (and perhaps only) be established through evaluation research using randomly assigned control groups. These then are just the rough outlines of a strategy and mechanism for evaluation research on youth programs under JTPA. We believe they are compatible with the basic design of JTPA itself and could yield good evidence from the JTPA experience about how an employment and training system can better help alleviate the continuing serious employment problems of sizable proportions of our youths. We must stress that in the absence of such a strategy and mechanism, we believe that several years from now the nation will find itself with several

OCR for page 1
30 years of JTPA experience but knowing little more about whether or how such programs might help reduce youth employment problems. Research Methods Beyond the general strategy and mechanism just suggested for research and demonstrations under JTPA to test the types of program components we have recommended, we have specific recommendations to make about the actual conduct of research and evaluation activities. The results of our review of YEDPA programs make it obvious that quality in the design and execution of a research project affects the quality of the data and the reliability of any conclusions that are drawn from those data. Poor research designs can make programs look worse than they are, or better than they are, or yield uninterpretable evidence. Poor execution will compromise even the best design. Random Assignment Our review of YEDPA research strongly suggests that much more could have been learned, and more confidence placed in the results, if random assignment had more frequently been used. We believe that not only has the feasibility of random assignment in program research been demon- strated, but that in situations in which program resources are scarce and program effectiveness unproven, it is ethical (see Appendix C). . RECOMMENDATION: Future advances in field research on the efficacy of employment and training programs will require a more conscious commitment to research strategies using random assignment. Randomized experiments should be explicitly authorized as a device for estimating the effects of new projects, program variations, and program components. Furthermore, funding authorities should back this explicit - authorization with firm indications that this is the method of evaluation which is expected. Implementation Research The need for measurement of program implementation in evaluation research is clear. It is as important in social program evaluation as is measurement of dose level in evaluating new drugs. Federal agencies have had substantial experience in eliciting such information, but this information has not always been reasonable in quality, judging from our review of youth employment program evaluations. RECOMMENDATION: Systematic and verifiable information on program implementation should be collected in future research. Better and less expensive methods for obtaining and reporting such information need to be developed.

OCR for page 1
31 Use of Subjective Measurements as Proxy Variables While one cannot fault a research program for using subjective measurements as proxies for other outcomes, it is a theoretical and methodological challenge to develop measures that have substantial validity and reliability. Indeed, the treatment of subjective measurements has been reconsidered in recent years, and it is generally recognized that common research practices ignore the complexity of the relationships between objective and subjective measurements. RECOMMENDATION: Future researchers should avoid overreliance on subjective measures of program outcomes and devote more resources to studying the relationships that exist between subjective indicators and key objective outcome variables. Postprogram Follow-up m e Job Corps evaluations suggest that some program effects that are not apparent at short-term follow-ups may emerge in the longer term. Research on job placement programs suggests that some, more immediate, postprogram effects may decay rapidly. Together, these pieces of evidence suggest that short-term follow-up data may err either positively or negatively in predicting longer-term program effects. RECOMMENDATION: Future research on the effectiveness of youth employment and training programs should, at least in selected studies, estimate the longer-term effects of these programs by collecting follow-up data for at least 2 years postprogram e Benefit-Cost Studies When evaluations demonstrate that programs have a positive outcome, researchers should recognize that the next question' raised will be whether this positive outcome was worth what it cost to produce it. Thus evaluators should anticipate the eventual need for benefit-cost studies. Such studies, however, need not be a component of every evaluation nor of entire programs, since doing adequate benefit-cost studies is both difficult and costly. The Job Training Longitudinal Survey Data Base One feature of JTPA that is important in regard to program research is the plan to rely heavily on analysis of national data bases to determine the effectiveness of JTPA programs. Present plans are to model the Job Training Longitudinal Survey (JTLS) data base after the previous CLMS. Although its data gathering appeared technically excellent, the CLMS strategy of using nonrandom comparison groups for

OCR for page 1
32 program evaluation suffered from substantial problems with potential unmeasured biases in its comparison groups. Furthermore, we believe that the plan to use the same strategy in designing the JTLS as a means of obtaining evidence on "what works for whom" is misguided. The evaluations we reviewed that are based on constructed comparison groups is provide strong evidence that this approach to program evaluation seriously flawed; the question of bias . . · · . · · . . ~ In comparison groups so con- structed is virtually impossible to dispel. We believe that the planned JTPA evaluations using the new JTLS will suffer from the same problems. RECOMMENDATION: Planning for the JTLS should give very serious consideration to the selection of randomized control groups. In conclusion, we believe that quality research ought to be recog- nized and ought to be explicit in congressional and agency oversight policy. Special efforts should be made to improve the quality of research and evaluation designs for estimating the impact of youth employment projects. Existing professional guidelines can be used to influence quality of design as well as quality of research execution and reporting. RECOMMENDATION: The committee recommends the following conditions as necessary but not sufficient for quality research: (1) the use of random assignment to participant and control groups and to program variations; (2) reasonable operational stability of the program prior to final assessment of effec- tiveness; (3) adequate sample coverage and low rates of sample attrition; (4) outcome measures that adequately represent the program objectives, both immediate and longer term; and (5) a follow-up period that allows adequate time for program effects to emerge or decay. The General Conduct of Public Policy Research One of the major implications of our review of YEDPA programs and research concerns the conduct of national public policy research and demonstration programs. It was very apparent in our review that many of the problems we faced in attempting to draw inferences from YEDPA research resulted from the fact that under YEDPA attempts were made to combine numerous research objectives with massive service delivery. The consequent tensions, conflicts, and overload on the system inter- fered with the careful planning and conduct of the research and demonstration activities, with the result that the research findings fall short in informing the public policy issues from which YEDPA originated. RECOMMENDATION: In future efforts the objectives of research and demonstration should be more clearly and selectively focused on essential public policy issues and clearly separated_

OCR for page 1
33 from the objectives of massive service delivery. magnitude of the effort and the expectation of results should be more in scale with limitations of time, money, and staff resources. Dilemmas Confronting the Employment and Training System In closing, we return to the two fundamental dilemmas with which we began this discussion of implications and recommendations for the system of employment and training in the United States. The employment and training system is trying in large part to do what the education system should be doing but, for some significant segment of the youth population, apparently fails to do. Yet the employment and training system has not attained stability of funding, professionalization of staff, and delineation of authority, in short, institutionalization of the sort that has given the educational system its accepted place in the mainstream of American life. As a result, in most communities, organizations involved in employment and training are considered marginal. The educational system, on the other hand, should not be taken as an exact model for the institutionalization of the employment and training system, since it has not yet found an effective way to prepare a substantial segment of the youth population for later employment. For the most part, the youth programs of the employment and training system have been specifically targeted toward special segments of the youth population, often those perceived as most disadvantaged. Given that the major rationale for youth programs within the employment and training system is to assist those whom the educational system has failed to prepare for work, this target seems a sensible means to focus resources on those in greatest need. The problem is that this very targeting tends to create an image of the programs as designed only for "failures;" both the programs themselves and their clientele become stigmatized in the process. The staff of the programs may come to feel stigmatized as well and this can exacerbate Problems of recruitment. . retention, and management. Even the potential target group members can come to share the views of the broader community about the inherent marginality of these programs and the stigmatizing effects of partici- pating in them, and it becomes increasingly difficult to enlist them in the programs and to keep them participating for sufficient time for the "program treatment to take hold." Yet, experience has shown that when programs are not targeted, the resources tend to be shifted rapidly to the more advantaged, better prepared, easier to handle segment of the youth population--those who have far less need for help with potential employment problems. These fundamental dilemmas pose a major impediment to solving the serious unemployment problem of youths, and we emphasize again the need for a direct study of the roles and relationships of the education and the employment and training systems.