Assessing Evaluation Studies The Case of Bilingual Education Strategies (1992) / Chapter Skim
Currently Skimming:
3 The Longitudinal Study
3 The Longitudinal Study
Pages 24-66
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 24... ...
The Longitudinal Study itself consisted of two components, a baseline survey and a series of follow-up studies In the subsequent 2 years. The study began late in 1982, and data collection for the descriptive phase occurred during Me [all of 1983.
|
From page 25... ...
Findings from the Descriptive Phase (1984 1987) include: The need for LEP services is not evenly distributed geographically across states and districts.
|
From page 26... ...
Spring 1984 Longitudinal Study Phase of the National Evaluation of Services for Language-Mirzoriry Limited-English Proficient Students: Overview of Research Design Plans for, report by Development Associates, Inc. Describes
|
From page 27... ...
December 1984 The Descriptive Phase Report of the National Lon git~inal Evaluation of the Effectiveness of Services for Lar~guage-M~nority Limited-English-Proficient Stu dents, report by Development Associates, Inc. and Re search Tnangle Institute.
|
From page 28... ...
To identify clusters of instructional services provided to LM-LEP students in Grades Key; and 9. To obtain information useful in designing a longitudinal evaluation of the differential effectiveness of the identified clusters of services provided to LM-LEP students.
|
From page 29... ...
A planning questionnaire provided data from school personnel for planning the longitudinal phase. Analysis Methods The Descriptive Phase Report (Development Associates' 1984a)
|
From page 30... ...
categorizes service pattems into service clusters, or "sets of instructional services provided to one or more LM-LEP students at a particular school or schools, based on their most salient features," thereby defining clusters of similar types of service. There is no description of the methodology usedfor the categorization.
|
From page 31... ...
First graders were near the national norms. Bow firstgrade and third-grade students were rated by teachers as being below grade-level proficiency In mathematics, English language arts, and native language arts, but third-grade students were rated as being closer to grade-level proficiency.
|
From page 32... ...
Overall, field researchers found a positive attitude in most schools toward serving the needs of LM-LEP students. Panel Critique of the Descriptive Phase Study The descriptive phase study was based on a national probability sample of students and teachers.
|
From page 33... ...
If it was important to limit We number of states to 20, Den resmcting the study to Me 20 states containing the largest proportion of LM-LEP students might well have been more desirable. Such a sample would have contained 92.7 percent of Me LM-LEP population and would have constituted a worthwhile study population in its own nght.
|
From page 34... ...
The reported problems appear to have been minor, ~us, it was still possible to weight the data appropriately. However, in Me longitudinal phase, Me problems were so severe that the data could not be weighted.
|
From page 35... ...
The report notes that there is a distinct underrepresentation at grade 6, observed by comparing school-level data to distnct-level data, which were not subject to such bias. The Longitudinal Study Descriptive Phase Report is deficient in that Were is no description of the method of calculating sampling errors.
|
From page 36... ...
Although missing data problems in the descriptive phase were encountered, they do not appear to have been severe, particularly in comparison with those experienced during Me longitudinal phase of the survey. The report does not document in detail the sources and problems of missing data.
|
From page 37... ...
What are the effects of the special services provided for LM-LEP students in Grades 1-5 in terms of the LM-LEP student's ability to function effectively in an all-English-medimn classroom?
|
From page 38... ...
Study l;) esign and Data Collection The panel found much less documentation for the research design for the longitudinal phase Han for He descriptive phase.
|
From page 39... ...
The LEP group consisted of virtually all students ire the respective cohort who were classified as limited-English proficient by local cntena. The English-proficient group consisted of those students not classified as LEP but who were receiving special services because they were placed in classes win LEP students.
|
From page 40... ...
(If these assumptions are not satisfied, the resulting estimates are inconsistent, meaning Hat even with very large sample sizes, the parameter estimates are not necessarily close to the actual parameter values.) These assumptions are described in Appendix B to the OveIview of the Research Design Plans (Development Associates, 1984b)
|
From page 41... ...
~ _` (321) Y1 Hypo~enca1 Model 2 GNU RE ~1 Sample Pam Diagrams from Research Plan 41 Y2
|
From page 42... ...
For this reason, a second assumption is required if ordinary least squares regression is to be used tO estimate the parameters of this equation: namely that the random errors in the first and second equations are uncorrelated with each other. The research plan notes (Development Associates, 1984b, Appendix B)
|
From page 43... ...
Estimates for four sets of linear regression models were included in the report: · Receipt of services was modeled as a fUnChOn of student and home background measures, ability/achievement measures, and educational history measures (Chapter 4~. · English language arts achievement was modeled as a function of ins~uchor~a1 exposure measures, student and home background measures, ability/achievement measures, educational history measures, school/classroom measures, and measures of background characteristics and teaching style of the main language arts teacher (Chapter 5~.
|
From page 44... ...
Summary of Results The final report for We longitudinal phase (Burkheimer et al., 1989) includes a summary of major study findings.
|
From page 45... ...
45 The yearly achievement of LEP students in mathematics and English language arts is not facilitated by a single approach; rather, different approaches seem more appropriate depending on the characteristics of the student. LEP students who are assigned to sernces for which they are ready generally show increased achievement (in both mathematics, as measured on an English test, and English language arts)
|
From page 46... ...
and when provided more instruction in English language arts andlor indirect exposure to English through instruction provided in English in other courses, LEP children show greater yearly achievement in English language arts. · When LEP children are weak in English and/or strong in their native language, English language arts instruction in the native language facilitates yearly English language arts achievement; to a lesser extent, mathematics instruction in the native language facilitates mathematics achievement under the same conditions.
|
From page 47... ...
This suggests Hat some programs were rehang children in LEP services on the basis of some criterion Hat increased over subsequent years (i e., perhaps achievement In English language arts) , while others were mainlining services to all or a subgroup of LEP students irrespective of external cntena.
|
From page 48... ...
Greater prior achievement in English language arts and/or initial oral proficiency in English (typically as modified by certain combinations of instructional services) were also found to increase the probability of exit for children in all but the least powerful of the models (i.e., in the later grades)
|
From page 49... ...
: When LEP children are weak in English arid/or strong in their native language, English language arts instruction in Me nanve language facilitates yearly English language arts achievement; to a lesser extent, mathematics instruction in Me native language facilitates mathematics achievement under the same conditions. What can defensibly be said is that a positive partial correlation was observed between achievement and the respective instructional variable when the effects of other variables in the model were taken into account.
|
From page 50... ...
Resource restrictions always require tradeoffs among objectives, and dropping infeasible or low-priority objectives early in a study enables resources to be concentrated on ensuring that high-pnority objectives will be met. To illustrate this point, consider the descriptive phase results that revealed that a large majority of LM-LEP students were Spanish speaking, that these students differed in many important ways from other LM-LEP students, and that there was a strong relationship between native language and service charactenstics.
|
From page 51... ...
Of course, the existence of clusters in the population restricted the range of independent variation oftreatments, thus restricting the inferences that could be made about distinct effects. (For example, use of Me native language in mathematics instruction tended to correlate with use of the nanve language in language arts instruction because both tended to occur more in programs emphasizing a gradual transition to English; therefore, inferences about separate influences of these variables are problematic.)
|
From page 52... ...
All test scores were obtained from students who had been receiving one or another treatment prior to We beginning of the study. Selection of Units into the Sample The sample of schools for the longitudinal phase consisted of those schools from the descriptive phase sample with a sufficient number of LM-LEP students.
|
From page 53... ...
No consideration appears to have been given to augmenting the sample to ensure the necessary variability even though data available from the descriptive phase could have identified problems of insufficient variation in treatments and noncomparability of units. Determination of Sample Size Required for Reliable Inferences Having a large overall sample is not sufficient for drawing conclusions when the goal is to infer relationships among variables.
|
From page 54... ...
It does not appear that this was done in the Longitudinal Study. The term "missing data" was mentioned only once in the Overview of Research Design Plans (Development Associates, 1984b, Appendix B)
|
From page 55... ...
The descriptive phase thus should have provided information about the necessity of applying methods for controlling or adjusting for measurement error. In view of the important effect of measurement error on Me ability to estimate parameters of interest in path analysis models, the lack of attention devoted to planning for and controlling measurement error is a .
|
From page 56... ...
The panel found that the data provide a picture of the state of bilingual education in the early to mid-1980s and might be useful in planning fumier studies. Descriptive Phase Data The descriptive phase of the Longitudinal Study presents a large set of data mat can be reliably projected to the national population of (locally defined)
|
From page 57... ...
The data were collected about dismcts, schools, teachers, classrooms, and students from a subset of those schools and districts surveyed during the descriptive phase win sufficient LM-LEP students to make the Longitudinal Study data collection effort cost effective. Some schools declined to participate, and for some of these substitutions were made, but no weighting adjustments were made to reflect this nonresponse.
|
From page 58... ...
, but might also be informative about whether certain types of teacher services and practices are predominantly available in classrooms with mostly LEP students, or in those with few LEP students. IN summary, it is unclear from the Descriptive Report how restricted the population for inference is from the longitudinal phase.
|
From page 59... ...
The descriptive phase study provides information concerning the natural range of variation in services in He population and about correlations between service types and various background characteristics. The longitudinal phase adds to this information base: variables were measured in the longitudinal phase that were not in the descriptive phase, measurements were made over tune, and information was collected about the factors related to exit from LM-LEP services.
|
From page 60... ...
lIere, we note the critical importance of identifying key data items for which it is essential to have complete data, monitoring Me completeness of these items as the survey progresses, and devoting resources to follow-up on these items if the extent of missing data becomes too high. Fifth, there must be control of measurement error.
|
From page 61... ...
Strategies for Control of Missing Data Missing data problems can be greatly reduced (although seldom eliminated entirely) by proper planning and execution of a data collection effort.
|
From page 62... ...
How is a LM/LEP student's oral proficiency in the native language related to He reaming of English when He student's nanve language Is: a) A language linguistically related to English?
|
From page 63... ...
B How do the various combinations of special services, or "service clusters," provided for LM/LEP students in grades 1 - compare In terms of the effectiveness with which recipients subsequently can function in all English medium classrooms?
|
From page 64... ...
2) How are the effects of the student's level of oral proficiency in the native language related to teaching academic subjects in the student's native language?
|
From page 65... ...
The descriptive phase report of He national lor~gitudinal study of the effectiveness of services for LMLEP students. Technical report, Development Associates Inc., Arlington, Va.
|
From page 66... ...
The Condition of Bilingual Education in the Nation: A Report to the Congress and the President. Office of the Secremy.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.