Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
4 Assessment of Vocational Education R&D What has been gained from the $250 million invested by the U.S. Office of Education in vocational education R&D over the past decade? The Committee's assessment of the vocational education R&D program fo- cused on two questions: What contributions have been made by R&D to knowledge about vocational education? What has been the impact of R&D on students? Both of these questions have been difficult to answer. In its attempt to identify the accumulation of knowledge on various topics related to vocational education, the Committee was hampered by the absence of reports on projects funded by USOE. Because the work funded only by USOE could not be reviewed as a separate body of litera- ture, the Committee reviewed research related to vocational education-- regardless of funding agency. (The results of that review are summarized in Appendix A, and a related discussion of research issues is presented in Chapter 2.) In looking for evidence of impact of R&D on students, the Committee and staff interviewed many people involved in vocational education and its R&D, conducted hearings, and reviewed evaluation re- ports. (The individuals and organizations that were contacted are listed in Appendix B.) R&D conducted by federal agencies to address social problems can serve primarily either the agency or the practitioners throughout the country. For example, the Department of Labor conducts its manpower R&D program to produce information for its own use and to help solve national manpower problems, while the vocational education R&D pro- gram in the Once of Education serves primarily the needs of practition 28
Assessment of Vocational Education R&D 29 ers. USOE allocates R&D funds by state in order to meet the differing needs of the numerous vocational education programs, which are operated by state and local education agencies. Evaluation of vocational education R&D iS quite difficult because of the complexities of the educational and R&D programs: the thousands of R&D products should be evaluated ac- cording to the differing needs of practitioners and in settings appropriate for their intended use. Assessing the impact of the R&D program on students has been difficult because there are virtually no data that permit a rigorous evaluation of all the R&D and its outcomes. Project evaluations have been rare, and the impact measures used have been weak, superficial, and transitory. Evalu- ation of demonstration projects has been especially problematic, partly due to lack of criteria for measuring success. In addition, follow-up or longitudinal data on graduates from various vocational education and other programs have not been collected. The Committee has therefore based its assessment of vocational education R&D on incomplete and sometimes subjective information, program evaluations, and project de- scriptions. The Committee had difficulty finding evidence that R&D has had a measurable impact. Although we do know that some R&D products have been disseminated and used by practitioners, we do not know what pro- portion of the total they are or what proportion is either reasonable or desirable. Further, we do not know how well these products fit the needs of potential or actual users. Even complete information about implemen- tation of R&D products does not accurately reflect impact. Since research often involves testing ideas or products, discovering that a promising idea or product is, in fact, unworkable is still a valuable contribution. Many ideas or products should not be implemented or adopted, and researchers perform a valuable service by finding evidence that prevents wasteful application. In short, there are no hard data to substantiate the belief that the vocational education R&D prompted by the legislation of 1963 and 1968 has improved vocational education. Even though there are other benefits from R&D (such as accumulation of knowledge, building R&D capability) these have rarely been measured, documented, or validated. LARGE-SCALE EVALUATIONS OF R&D There have been major efforts by two organizations (Project Baseline and Development Associates, Inc.) to evaluate broad impact of vocation- al education R&D. Project Baseline, located at Northern Arizona Univer- sity, is funded under the federal share of Part C of the 1968 Amendments
30 ASSESSING VOCATIONAL EDUCATION RESEARCH AND DEVELOPMENT to collect data and report on several aspects of vocational education in the United States. Project Baseline, which usually receives over $200,000 yearly, has produced four comprehensive annual reports and several sup- plemental reports on such topics as preparing vocational education teachers, women in vocational education, and the impact of manpower training programs on the labor market. Project Baseline attempted to evaluate state-administered Part C and D projects and federally adminis- tered Part D projects; this evaluation of R&D iS just one of their recent activities. Development Associates, Inc., a private management consult- ing firm specializing in evaluating social programs, attempted to evaluate Part D programs, focusing more sharply on the federal than the state projects. These evaluations have had only limited success and produced rather narrow and superficial definitions of impact. There is not more evidence of impact partly because impact is not usually immediately manifest, and this Committee is studying a time span that is both short and recent. The Committee presents the findings of Project Baseline and Develop- ment Associates both for their inherent value and to explain the need for more and higher-quality evaluative data. These two projects present the only large-scale evidence of impact of vocational education R&D known to the Committee. However, the reader should be cautioned that the data collected by these organizations and presented here are not adequate for a rigorous evaluation. Project Baseline surveyed people who had been deeply involved in the projects evaluated, so the probability of judg- ments biased in favor of the R&D iS extremely high. Development Associ- ates' conclusions are tenuous partly because of problems in obtaining unambiguous enrollment data on the students sampled. PROJECT BASELINE Project Baseline attempted to determine the impact of research, develop- ment, and exemplary projects on vocational education funded under the state-administered Part C program, the state-administered Part D pro- gram, and the first round of the Commissioner's share of the Part D program. In Project Baseline's Fourth National Report (Lee 1975) the term "impact" is used to mean the extent to which local districts have maintained and implemented previously developed products and results. This measure of impact was used to assess project outcomes because Baseline did not have nor could it obtain accurate measures of changes in student skills and other evaluative data. The impact of federally administered Part D projects is classified in five categories:
Assessment of Vocational Education R&D 1. Implementation in many states 31 2. Extensive implementation within a state (Extensive is operationally defined as enough acceptance that the majority of schools in the state are using some or all of the concepts, materials, and other products.) 3. Some implementation in several areas of a state 4. Implementation within a district 5. Minimal or no implementation A similar classification scheme is used for state-administered projects: i. Wide implementation or extensive use within a state 2. Implementation in the original site and in a few additional sites 3. Implementation only in the original site 4. Use of results as a basis for additional work 5. Little or no known current use An effort was made by Baseline to assess 54 federally administered Part D projects by reading final reports and interviewing project di- rectors by telephone. Interview questions solicited information regarding the extent of the dissemination and utilization of R&D results and prod- ucts. Baseline found that 40 of the 54 projects fell into the first four categories; they "have had some impact in bringing about changes in at least a few additional school districts within their states and in additional schools within their own districts" (Lee 1975, p. 61~. Twelve of the 40 projects have had extensive impact within the state (category 2), and four have been adopted in whole or part by other states (category 1~. Stratified random samples of state-administered Part C and D projects were selected for assessment by Project Baseline. Stratification of Part C projects was based on statewide versus local focus and whether or not impact was primarily on policy, administrative practices (including guid- ance), or instruction. Part D projects were stratified according to the following topics: "cooperation between education and manpower, post- secondary education programs directed toward out-of-school youth, and programs for young people in school" (p. 63~. Information about projects was obtained by telephone contact with research coordinating unit di- rectors, project directors, or school district personnel. Respondents were asked about the extent to which projects, methods, and results had been implemented in additional schools or school districts. There was no fol- low-up to determine the duration of newly adopted programs or prod- ucts or to verify the responses independently. Baseline found that 94 percent of the sample of 96 state-administered Part C projects had had some impact, at least locally (categories 1-4~; 29 percent were "widely
32 ASSESSING V=ATIONAL EDUCATION RESEARCH AND DEVELOPMENT implemented or extensively used" (category 1~. There were similar findings for state-administered Part D projects: 97 Dercent kind .c~m~ impact (categories 1-4) and 20 Sent Wears with 1m~l=~A AN (category 1). 1 - -~^A^_ _ _ ~ ~^ _ ~ ^~} $~1~111~11 ~V1 US Problems in obtaining student outcome information were experienced in the Project Baseline evaluation. The lack of baseline data made it impossible to assess accurately indicators of student growth and prog- ress. Instead, the study describes several state-administered Part C and D projects that were judged by state R&D personnel to have had the greatest Impact. Again, success appears to have been determined by the extent to which the product is used by other schools, generates interest among students, teachers, and the community, and provides information for use by teachers and administrators. The benefits of using research results were not assessed. DEVELOPMENT ASSOCIATES In their evaluation, Development Associates attempted to assess the Part D program for USOE, using pre- and post-treatment measures of career development. For each of the 50 projects selected for evaluation, experi- mental groups were drawn from students in grades 6, 9, and 12 partici- pating in Part D programs, and control groups were drawn from students not participating in those programs. The study was hampered by difficul- ties in sample identification: " . . . most projects were not able to iden- tify precisely either the students who were 'participants' in the program or what constituted 'participation'" (Development Associates 1975, p. 23~. Questionnaires and tests of student outcomes were given to experi- mental and control groups. The 13 questions used to measure student outcomes included: Are student participants able to identify a greater number of occupa- tions than non-participants? Do students demonstrate more familiarity with tasks and functions associated with selected occupations than the comparison groups? Are a greater number of students who have graduated from school and who participated in the Part D project employed full-time or engaged in further training than students who did not participate? The questionnaires produced mixed results. Five questions were elimi- nated from the summary due to factors such as insufficient or invalid
Assessment of Vocational Education R&D 33 data. Significant differences between participants in the Part D projects and non-participants were found for only some grade levels on only three outcome questions. In light of the problems of identifying the study sample as well as the inconclusive findings, it is impossible to conclude that the Part D projects significantly improved student outcomes. Several of the problems of evaluating vocational education R&D after projects have been completed are evident in the work of Development Associates: participation in the projects was hard to define since it is not a simple yes/no variable. Because of inadequate records, administrators had difficulty in determining which students had participated in the pro- jects; the measures of impact that could be collected were somewhat weak. In addition, placement and follow-up data, which Development Associates did not have, can be collected only with expenditures of much time and money. Even though the study appeared to be very carefully planned, it could not overcome the shortcomings imposed by these other factors. EXAMPLES OF PROJECTS JUDGED TO BE SUCCESSFUL Definitive conclusions on the whole body of vocational education R&D cannot be drawn on the basis of the two national evaluations described above. Therefore, in an attempt to find some substantiation of the impact of R&D, the Committee analyzed individual projects on the basis of their descriptions. The Committee was able to locate a number of projects with some measure of impact. Six projects judged to be successful were drawn from lists compiled by USOE and the Southwide Research Coordinating Coun- cil; other examples were supplied by Committee members and a consul- tant. The nine examples are described here. U.S. OFFICE OF EDUCATION Eighteen federally funded projects were cited by Howard F. Hj elm and Glenn C. Boerrigter (1974) as examples of visible and useful accomplish- ments of vocational education R&D over the past decade. Three projects taken from their list are described below. The Committee selected these projects as examples because they demonstrate three different objective measures of impact. In the initial phase of the Aviation Mechanics Project (reported in Allen 1968), a core curriculum for aviation mechanics was identified. During Phase II, airlines and aviation companies were surveyed in order to identify industry requirements. Phase III consisted of two parts: Dur
34 ASSESSING VOCATIONAL EDUCATION RESEARCH AND DEVELOPMENT ing Part I, 100 teachers were involved in curriculum development and teacher training. During Part II, 30 percent of the companies surveyed in Phase II were resurveyed. The second survey showed the necessity for readjusting school curricula in major tasks such as painting and welding. As a result, the core curriculum was updated and a method by which the Federal Aviation Administration could continue periodic updates was established. Recommendations made on the basis of the second survey reflect current requirements of the aviation industry. Establishing a method for industry to update vocational curricula may provide more relevant training for vocational students, but the actual improvement in student job prospects and the advantage to employers in hiring graduates trained with this curriculum have not been measured. The objective of the Electro-MechaIlical Equipment Technology Proj- ect (reported in Roney 1971 ) was to assist two-year colleges in establish- ing training programs by developing the necessary planning and instruc- --tional materials and by providing direct program planning assistance. Researchers developed and tested an integrated system of instruction built around discrete technical concepts that are basic to more than one technology. Student achievement was measured and recorded at all stages of the project. Instructional materials were tested on students and revised. Case studies were written for the purpose of identifying adminis- trative problems. By the end of the project, materials were being used in 30 states and planning assistance had been provided to 375 schools. Ma- terials were disseminated widely after the project was completed. Evaluation of the electro-mechanical equipment curricula showed that students successfully learned skills and knowledge in electro-mechanical equipment technology. However, it is not clear how many of the students who used the curriculum materials were placed in jobs or how many used the acquired skills and knowledge on the job. The success of the project cannot be fully determined until it can be shown that the stu- dents trained in electro-mechanical equipment technology gained an ad- vantage in finding and keeping good jobs because of their training. "The Kingdom of Could be You" (reported in Sutherland Associates 1974) consisted of 16 short cartoon and real-life films related to occupa- tional clusters. The project had three objectives: to develop awareness of future job opportunities in young children; to enlarge the vocational self- concept by encouraging children to see themselves in a variety of occu- pational roles; and to engender a work ethic in children. The films were originally presented on the Captain Kangaroo Show. The film evaluation project involved 124 children, three to seven years of age, with appropriate ethnic, sex, and socioeconomic distribution. Pre- and post-tests were used to measure gain in career awareness. Children
Assessment of Vocational Education R&D 35 showed a remarkable gain in awareness of occupational opportunity as judged by comparison of interviews before and after exposure to the films. The 16 films are available from a commercial distributor. SOUTHWIDE RESEARCH COORDINATING COUNCIL The Southwide Research Coordinating Council, consisting of the re- search coordinating units in 14 states in the southeastern United States, compiled a casebook of R&D activities they judged to have had impact (1975~. The report is not intended to assess all vocational education R&D, but to demonstrate that some projects have had a positive impact in the Southeast. Although the sources of information are not explicitly de- scribed in the report, it appears that project directors or staff members of projects supplied impact data on their own projects. Therefore, the ob- jectivity of the information is open to question. A sample of three of these 26 projects demonstrates the kinds of projects included and ways in which impact is defined. The Alabama Vocational Management Information System was de- signed to provide manpower data for management of vocational educa- tion programs in the state. In addition, an accounting system provides fiscal data as well as student enrollment, completion, and follow-up in- formation by teacher, school, program, and occupational objective in order to evaluate programs. As a result, an increase in planning activities has been noted as the data are made available to assist local planning efforts. Special data requests from teachers, administrators, and planners have indicated that the data are needed, and the management informa- tion system is reported to have resulted in more accurate and useful information. However, the accuracy of the data supplied by the system has not been evaluated objectively. A Part D project in Florida, "An Exemplary Model for a Total Eco- lo~ical ADDroach to Non-~raded Vocational Programs in Senarate Edu 0 1 1 C.7 .. . , , .. . . . . . . . . . .... . . . rational Centers, was designed to Improve students attitudes toward their environment, improve academic achievement, and develop occupa- tional skills, as well as to develop new techniques for teaching disadvan- taged students. Project staff reported that achievement measures indicat- ed that all of these objectives were met. This project was the forerunner of career education in the state, and eventually the model was adopted by all education districts in the state. A career education project in Mississippi exposed students to occupa- tional education in elementary school and continued preparing students for the world of work through high school and post-secondary school. An evaluation using experimental and control student groups showed
36 ASSESSING VOCATIONAL EDUCATION RESEARCH AND DEVELOPMENT that students in the project had more positive attitudes toward work and a greater knowledge of careers than did students in the control group. Project staff reported that the project's career education methods have spread to 22 school systems in the state. Most of the other projects listed in the Southwide Research Coordinat- ing Council's report have impact measures similar to those in these three examples. For most placement services, management information sys- tems, and evaluation models, adoption by a certain number of schools or school districts is reported as the measure of impact. Impact of curricu- lum materials and instructional techniques is reported by adoption and by the number of teachers and students involved. Occasionally, the re- port of a third-party evaluation is cited. The measures of impact that are reported are generally not objective and do not convey information about changes directly related to students. OTHER EXAMPLES Some projects can be considered successful if research findings are used to change program policy. An example was provided by a member of this Committee, the Texas Director of Vocational Education. A Part C project entitled "A Survey of the Occupational and Educational Needs of the American Indian in Dallas County" was designed to collect in- formation for use in program planning. Findings indicated specific edu- cational needs, and on the basis of recommendations in the final report, the Dallas County Community College District (1973) initiated a cooper- ative education program for Dallas County Indians. Study findings also provided a basis for the development of a pre-school program and a youth program for Indians later funded by USOE. The Inter-Tribal Center and Clinic also used the findings to develop a manpower training pro- gram funded by the Department of Housing and Urban Development. Finally, as an outgrowth of the study, the Dallas County Community College District began operating three outreach programs designed to recruit and counsel American Indian students and other minority group members. However, the impact of the pre-school, youth, manpower training, and outreach programs has not been studied in formal evalua- t~ons. A nationwide project on allied health occupations (reported in Ander- son 1973 and Fielstra 1973) developed, validated, tested, and disseminat- ed instructional material for more than 20 allied health occupations. Nationwide conferences and national advisory committees were in- volved in developing the instructional material. Twelve of the final in- structional material documents were published by commercial publish
Assessment of Vocational Education R&D 37 ers. One of the publications (nursing) has sold over 80,000 copies, and another (dental assisting) has sold nearly 35,000 copies. These materials have been used by every state in the nation and by 35 foreign countries. Other materials not published commercially are in great demand three years following the completion of the basic project. Again, however, stu- dents trained with these materials have not been followed up to deter- mine how helpful the materials are in preparing students for jobs. Krumboltz et al. (1967, 1968) constructed "job experience kits" in accounting and six other occupations in order to test the social learning theory that career interests are learned as a result of successfully master- ing occupationally related tasks. The kits were designed to stimulate young people to explore career opportunities by exposing them to simu- lated occupational tasks and easily solved problems. In one of the studies the theory was tested by administering the accounting kit to a random sample of high school students and giving control groups other info~a- tion about accounting or general occupational information. Follow-up interviews and questionnaire responses revealed that interest in account- ing was increased by the problem-solving accountant kit significantly more than by other treatments. Interest in 41 other occupations changed no more than would be expected by random fluctuation; however, the experience of using the accounting kit seemed to stimulate later inquiries about other occupations. Other experimental results showed that prob- lem solving was an elective method of stimulating interest, particularly for students from communities of low socioeconomic status. The devel- opers of the job experience kits contracted with an outside agency to develop kits in 13 more occupational areas and to distribute them na- tionally. The kits have been distributed in every state. CONCLUSIONS Although the examples of projects cited here represent only a fraction of the thousands of projects funded, they demonstrate some of the potential contributions of vocational education R&D to vocational education. The actual contributions have not been well-documented: most of the proj- ects described as successful have not been rigorously evaluated. Often, they are said to have impact because research products are "widely disse- minated" or "interest is generated" among students and teachers. Proj- ects are also said to be successful if their reports are published commer- cially and large numbers of copies are sold. It should be noted that few of the projects cited as successful have addressed the needs of students in secondary schools, where vocational education enrollments have been concentrated.
38 ASSESSING VOCATIONAL EDUCATION RESEARCH AND DEVELOPMENT Rarely has it been shown that students have benefited from the proj- ects cited as successful. In most cases it cannot be determined that R&D results or products are actually used, and, if they are used, it cannot be determined whether they have any significant or long-lasting erects on students. The lack of evidence implies that much vocational education R&D has probably not had a widespread effect on classroom activities and student outcomes. PERCEPTIONS OF R&D EFFECTIVENESS In the interviews and hearings conducted by the Committee and staff, respondents were requested to give information on the impact of R&D on various groups of vocational education consumers, especially students. Virtually all of the people interviewed had great difficulty in judging the effectiveness of the research and development familiar to them. The effects of R&D are diffuse: R&D may affect the work and attitudes of federal and state administrators of both vocational education and R&D, other researchers, teachers, other school personnel, students, and em- ployers of vocational graduates. All respondents felt it nearly impossible to trace changes in students' acquired abilities and attitudes to research projects. Most could cite advances in vocational education that had ac- companied progress in research but were not necessarily caused by re- search. Some improvements that were mentioned often are an increase in student enrollment, "better" and "more meaningful" programs, im- proved teacher attitudes, and greater student enthusiasm. One research coordinating unit director stated that vocational educa- tion R&D has enhanced the stature of other types of research in his state; people recognize that research can be practical and is, therefore, valu- able. There have been similar improvements in the image of vocational education as a result of increased awareness of the field by the public. Several people have noted that research may have benefits that are less visible than the accumulation of knowledge about particular topics. Scientific research is often described as a way of testing existing hy- potheses, but it can also raise new questions, or help to map unexplored territory as well as test existing hypotheses. Researchers can reformulate old questions on the basis of new evidence. Some of the people interviewed have observed innovative demonstra- tions and development programs and believe them to be more interesting than traditional programs. In one state, the teaching efficiency in a nurs- ing program has increased due to an innovative project, resulting in a shorter program covering the same amount of material. This sort of measure is much more quantifiable than many, for example, measures of 'A Acme +~ an_ :~
Assessment of Vocational Education R&D 39 students' enthusiasm or the effectiveness of programs. Some of those who often visit the schools are convinced that exciting things are happening partly as a result of vocational education R&D efforts. It is as difficult to refute such statements as it is to use them as hard evidence that R&D has made a difference. Virtually no hard data on the impact of R&D projects were presented at the hearings conducted by the Committee. Some witnesses noted that impact is very difficult to assess since many factors, not just R&D, affect events. Some noted that knowledge has increased in certain areas, but they were not asked to substantiate such statements and did not do so. One local administrator stated that he had observed very little impact as a direct result of R&D. He noted that many local vocational education administrators share his perception of R&D. Most of the witnesses, how- ever, were very enthusiastic about R&D and its value in improving voca- tional education. The mere existence of vocational education R&D is felt by many to have had some notable benefits not specifically related to changes in the structure and content of vocational education and the R&D program. Funding for vocational education R&D has most certainly helped develop research capability and increased the prominence of vocational educa- tion. Research personnel have been drawn to vocational education, vo- cational educators have gained experience in research, and a community of vocational education researchers has developed. In addition, research institutions supported by R&D funding have added to the visibility of vocational education R&D and have provided a setting that potentially increases research capability. Most of those interviewed noted changes in the supply of useful data as a result of research. Perhaps equally impor- tant is the seldom-noted fact that as the supply of information increases, administrators tend to rely less on intuition and more on data. Finally, as the data supply and demand increase, so do the standards for judging the quality of research. LACK OF EVALUATIVE MATERIAL There are no evaluative data or even judgmental statements that would make possible a comprehensive assessment of the vast majority of voca- tional education R&D over the last ten years. A comprehensive evaluation is virtually impossible for two major reasons. First, the priorities and objectives of vocational education R&D have been poorly defined and have rapidly changed so that there are no goals against which to measure actual accomplishments. Second, there has not been a consistent, broad- ly based, and long-term concern with collecting, analyzing, and reporting
40 ASSESSING VOCATIONAL EDUCATION RESEARCH AND DEVELOPMENT evaluative data on R&D products and their effects on students and society in general. Rigorous evaluations and follow-up studies are not only costly, but quite difficult to conduct. As is the case with all social science research, imposing laboratory conditions on real-life situations is all but impossi- ble for both ethical and economic reasons. Project evaluations in voca- tional education typically have been self-reports or post hoc, third-party reports rather than evaluations built into project designs. There have been very few follow-up studies to determine the eventual impact of R&D projects on students. The problems associated with evaluation are by no means unique to vocational education. Egon G. Guba explained many shortcomings of the methods of evaluation, but expressed optimism that the problems would soon be solved (Guba 1969, p. 29~: The American educational establishment is currently making a massive effort at self-improvement. Unprecedented resources, stemming mainly from the federal government . . . , are being expended on a variety of promising but as yet un- proved programs. To assure the elective and efficient uses of these resources and, even more importantly, to determine the real utility of the innovative ap- proaches, it is necessary to gather hard data about their performance. Evalua- tion is the process best suited for this purpose. The traditional methods of evaluation have failed educators in their attempts to assess the impact of innovations in operating systems. Indeed, for decades the evidence produced by the application of conventional evaluation procedures has contradicted the experiential evidence of the practitioner. Advances have been made in evaluation technology since Guba's arti- cle was published in 1969, but there are still problems to be solved, and the advances already made must be put into practice. More recently, Jerome Moss, Jr., and Ernst W. Stromsdorfer (1971, p. 261) concluded: It is apparent from this review of studies, both non-economic and economic, that the methodological issues facing any analysis of the effe~.~ of v~rntir~n~] education are formicl~hl~ , _ , ~ _ ~ - , ~ . ~ an. . . . [I]t is our opinion that the existing analyses, tak- en as a whole, do not use effectively the methodological knowledge which is cur- rently available. For instance, the concept of use of a control group would ap- pear obvious; yet in studies which pose hypotheses whose testing clearly implies the need for a control group, none is employed. And, in other evaluations, where recommendations are made which involved extensive commitment of eco- nomic resources, there is no treatment of costs along with benefits. These are only examples. Unfortunately, they do not represent uncommon errors.
Assessment of Vocational Education R&D CONCLUSIONS AND RECOMMENDATIONS CONCLUSIONS 41 The Committee's assessment of vocational education R&D has been limit- ed because a collection of final reports of all R&D projects is not avail- able. Further, data for program evaluation have not been collected. Therefore, the Committee's assessment has been based on partial evalua- tions conducted by others, small-scale surveys of researchers and admin- istrators, some reports of R&D projects, impressions of acknowledged experts, and~udgments of Committee members. The Committee has con- cluded that vocational education R&D of the past decade has not had documented, widespread impact. The available data do not indicate that vocational education R&D findings and products have had an influence on the knowledge, skills, or employability of large numbers of students. There are insufficient data to allow for a comprehensive evaluation of vocational education or its supporting R&D. Program success has most often been measured in terms of initial job placement, and little attention has been given to assessing the effects of programs over an extended period of time. The impact of R&D has been measured most often in terms of user acceptance, defined as frequency of requests for informa- tion rather than frequency of implementation of findings and installation of products. Impact measures have often been subjective and difficult to validate. They have sometimes been superficial and have failed to con- sider the long-term consequences of vocational education programs or R&D. In addition, the quantified elects of different programs or projects often cannot be compared because measurements were not standardized across projects. The Committee recognizes that vocational education R&D iS relatively new and that it faces many of the difficulties of all educational research and, in fact, of all social science research. One difficulty is that models for evaluation are not readily available. Further, the effects of R&D proj- ects cannot be isolated: many social, psychological, and economic fac- tors can confound or weaken the impact of R&D. Despite these difficulties the Committee believes that vocational edu- cation R&D has added to the body of knowledge about vocational educa- tion and its students. Vocational education R&D has also produced new programs and classroom techniques for use across the nation. Many curriculum materials have been published commercially and have been purchased by large numbers of people, although the extent to which these products or other research results are beneficial or are actually used by practitioners cannot be adequately determined at this time. Funding
42 ASSESSING VOCATIONAL EDUCATION RESEARCH AND DEVELOPMENT for vocational education R&D has also increased research capability. Na- tional and state institutions exist to facilitate and coordinate the conduct of R&D, and numerous researchers have been trained in or drawn to vocational education. RECOMMENDATIONS The Committee recommends several interrelated strategies that are nec- essary for a comprehensive evaluation of vocational education R&D. First, the goals and objectives of the R&D program, which should logical- ly follow from the objectives of vocational education, should be clearly defined. Only then can evaluation measure the degree of success in at- taining these objectives. The U.S. Commissioner of Education and state directors of vocational education should begin to fund studies of the objectives and priorities of vocational education programs and related work. Second, in order to identify in the future contributions made by R&D, funding agencies should be able to provide access to final reports of all R&D projects. Research synthesis documents would be especially helpful in reporting and assessing what has been learned. Third, USOE should develop, with ample input from researchers and practitioners, a plan for evaluation of R&D that includes the collection of longitudinal data. A planned mix of self-evaluation, agency evaluation, and third-party evaluation is suggested. In general, post hoc evaluations should be avoided. A sample of R&D projects should receive funding for evaluations, planned at the same time as the projects themselves. Not all R&D projects need extensive external evaluation: a few have adequate evaluation built into the research design, and others are so small that adequate evaluation would cost more than the original research. USOE should determine exactly what factors are to be measured by evaluations. Presently, impact on the knowledge, skills, and employabili- ty of students is the most widely accepted factor. Also of interest are the qualifications of the researchers; factors related to research design (quali- ty and appropriateness, implementation, data analyses, interpretations, and the extent of validation of results); the methods of dissemination; and the degree of utilization of products. A comparative, evaluative data base is needed for determining what vocational education programs and R&D have been effective and should receive continued support. Evalua- tion criteria should be studied and standards developed so that evalua- tions of different projects would be as nearly comparable as possible, given their inherent differences. However, important questions that are
Assessment of Vocational Education R&D 43 unique to certain programs or projects should not be eliminated simply because of their uniqueness. In order to measure the effectiveness of vocational education pro- grams, USOE should collect long-term, follow-up data on a national sam- ple of all people who enroll in vocational education programs and of comparable people who do not enroll. The data collected should include measures such as job satisfaction, upward job mobility, wages, satisfac- tion of the employer with the worker, and continuation of education (vocational as well as academic, both degree and non-degree). If longitu- dinal data were collected on new and traditional vocational programs, their differential effectiveness could be studied. Finally, long-term, longi- tudinal studies of potential client groups could be a useful, albeit expen- sive, means of gathering data and might be used by USOE for evaluation of vocational education. Because vocational education R&D is relatively new and rapidly chang- ing, the ongoing program of R&D should be supplemented by the work of an advisory panel charged with studying all of vocational education, including its R&D. This panel should be convened every five years to assess the accomplishments and failures of vocational education and to make recommendations in terms of goals and objectives for the future. The broad, long-range perspective of such an advisory panel would strengthen and give guidance to continuing work in vocational education R&D.
