Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 161
7 Decision Criteria and Recommendations In this chapter we offer our recommendations for a set of criteria that the U.S. Department of Education (DoEd) can use in supporting a decision on which of the allowable data sources to use for allocating Title III funds. We present the decision criteria in the form of a list of desirable characteristics for data for formulas that allocate federal funds. Based on the analysis in Chapters 2 through 6, we relate this list of characteristics to the American Community Survey (ACS) estimates and the state-provided counts of English language learner (ELL) children and youth. Taking into account the decision criteria and mindful of the weight of evidence described in the analysis provided in this report, we also offer several recommendations to the DoEd and the U.S. Census Bureau with regard to the use of the allowable data sources for allocating Title III funds. DESIRED CHARACTERISTICS OF ALLOCATION FORMULAS It is useful to have objective criteria to assist in determining the appropriateness of a data source that is to be used in developing a formula for the allocation of federal funds. Fortunately, the committee has a starting point for assessing the two data sources that are allowable under Title III. Several considerations for evaluating the adequacy and appropriateness of data sources and their data elements for service in determining the allocation of federal funds were outlined in a National Research Council (NRC) report on formula allocations (National Research Council, 2001, p. 6): (1) the conceptual fit between currently available data and the formula elements, as defined in enabling legislation or administrative regulations; (2) the level of geographic detail for which data are provided; (3) the timeliness of the data; (4) the quality of the data; and (5) the cost of collecting or compiling new data to
OCR for page 162
provide inputs to the formula. To these five considerations, we add five others that have emerged during our discussions with data users and producers and through our own examination of the necessary characteristics of data for use in allocating federal funds: (1) fairness, (2) stability from year to year, (3) insensitivity to differences in state policies and methods, (4) transparency, and (5) comparability. The rest of this section describes each of these criteria. Conceptual Fit A data element used in an allocation formula should meet the conceptual objectives of the program for which the allocation is aimed. In the case of allocating Title III education funds to states, a data element with a good conceptual fit is one that meets the definition provided in the legislation—the number of limited English proficient and immigrant children and youth in a state. In a larger sense, however, considering the overall objective of the allocation of federal funds, a conceptually fitting data element would provide state and local governments with federal funding that is proportional to their need and circumstances. Level of Geographic Detail The Title III legislation stipulates that the federal funds for the ELL program should be allocated to the states. Thus, the state government is the key level of detail for which the data should be available. Timeliness The elapsed time between the reference period for the estimates and the period for which the allocations are being made should be as short as possible so that the allocation would appropriately reflect the need at the time that the allocation is made. Quality Data quality is broadly defined as “fitness for use” (Statistics Canada, 2009, p. 6; Organisation for Economic Co-operation and Development, 2003, p. 6). In turn, fitness for use is generally characterized in terms of six attributes that are expected of the information provided by the data products: utility: the usefulness of the information to its intended users; objectivity: whether information is accurate, reliable, and unbiased, and is presented in an accurate, clear, and unbiased manner; interpretability: the availability of documentation that includes a presentation of the underlying concepts and their definitions; descriptions of the methods used to collect, process, and analyze the data; and a discussion of the limitations imposed by the methods used to aid customers in understanding and using the data; integrity: the security or protection of information from unauthorized access or revision; accuracy: the difference between an estimate and its true value, characterized in terms of systematic error or bias, and random error or variance; and comparability: similarity across geographic and demographic dimensions.
OCR for page 163
Of these, integrity, interpretability, and comparability are largely covered by other characteristics on our longer list, so under the heading of “quality” we focus on utility and accuracy. Cost The benefits from improvements in conceptual fit or other aspects of data quality have to be weighed against the costs. Even when existing data sources are used, it is desirable to avoid incurring significant costs of obtaining data in a format suitable for the allocation process. Fairness The allocation formula should be perceived as being fair. By fair, it is generally meant that the data used in the formula should be free from perverse manipulations, open to review, and should distribute resources equitably across governmental units. The formula itself should be replicable (see “Transparency,” below). Stability The data should be relatively stable over time. They should not be subject to extreme volatility or to large, unexplainable variation. However, in the context of Title III allocations, an appropriate balance needs to be struck between the stability of the data series and responsiveness to real annual changes in the size and characteristics of the ELL or immigrant population. Insensitivity to Policies and Methodological Differences The data series should be relatively insensitive to differences that arise from administrative practices and policy differences between agencies and jurisdictions that provide the data and that benefit from those differences. In the context of allocations for Title III funding, if states X and Y have the same distribution of English language proficiency, but state X sets standards for program entry and exit that result in a larger fraction of its students designated as eligible for Title III services, the data series should take those differences into account so that they do affect the allocations. (This is not to suggest that states are or could be “gaming” the system. In fact, there is little incentive to game the system, since the cost to states and local education authorities of administering and conducting the ELL program generally exceeds the funds received from the federal government. In each year of the reauthorized Title III Program, the amount of money involved was small enough that it didn’t create an incentive to states to this kind of strategic behavior.) Transparency Users should be able to have access to and be able to understand the assumptions, methods, and results so that a knowledgeable user could readily reproduce the information, within the constraints of protecting the confidentiality and privacy of the subjects (see U.S. Census Bureau, 2010, p. 167). Comparability The methodology by which the estimate is derived should be similar across geographical units.
OCR for page 164
The 2001 NRC report on this topic concluded that “there are many trade-offs among these considerations, and it is likely that no one data source will be superior to the others on all counts” (National Research Council, 2001, p. 6). The ability of the two allowable data sources to fulfill these desired characteristics is discussed in the next section. COMPARING THE ALLOWABLE DATA SOURCES In our judgment, both the ACS estimates and the state-provided counts meet each of these criteria to some extent, although each has strengths and weaknesses that need to be taken into account when considering their use for specific applications and at specific times. Table 7-1 shows our analysis of each of the 10 desired characteristics for each data source, discussed above. We present our ratings in the form of a scorecard, with the assignment of plus (+) marks, the highest rating being “++.” The panel particularly notes that prior issues with the relative volatility of the ACS data for smaller states have diminished as the survey has matured and as 3-year data have become available. Similarly, as discussed in Chapter 4, prior concerns about the accuracy and transparency of the state-provided data and the effects of different state practices on those data have abated over time as system-wide submission standards for data have been implemented. There are also some signs that states, as they are working to implement the guidelines of NCLB, are migrating toward more commonality in their approaches to identifying, testing, and educating the ELL population (as discussed in Chapters 1, 3, and 4.) On the basis of within-state regression analysis (which has the effect of eliminating between-state differences), the panel found that, under a set of uniform state procedures, the ACS and state counts tracked very well. This result shows that the ACS serves as a relatively good proxy of the constructs addressed by the state English language proficiency (ELP) tests, and it has the advantage of being more uniform between the states. However, we could not demonstrate a similar relationship of the ACS to a more comprehensive ELP assessment at the state level because there is no measure of the latter that is uniform across states for such comparisons. We found significant differences between the ACS and state-provided counts at the state level, and we attempted to explain these differences as a function of state policies. However, because state policies, procedures, and criteria differ along numerous dimensions that cannot be quantified in any parsimonious way, and the variables that described the states procedures and tests are so numerous, we were unable to identify any predominant cause of the differences in the state-level regressions reported in Chapter 5. We find the conceptual fit of the state-provided counts to be particularly compelling in contrast to the ACS definition which is only a rough proxy for the official ELL definition. At the same time, we are concerned about the lack of state-to-state comparability in the policies, practices, and criteria for classifying students as English
OCR for page 165
TABLE 7-1 Comparison of ACS and State-Provided Data on Desired Characteristics for an Allocation Formula Desired Characteristic Evaluation ACS State Provided Conceptual Fit The ACS estimates define need in terms of the numbers of children and youth who are eligible for being served by virtue of their skill in speaking the English language. The state-provided counts define need in terms of the number of those identified by schools as being eligible by virtue of surveys and assessments that are becoming increasingly standardized. The state-provided data are considered to be more accurate and relevant assessments of individual students as well as of the intensity of need as defined by the policies of the various states. + ++ Geographical Detail The ACS estimates and the state-provided counts are available for both states and local education agencies (LEAs). ++ ++ Timeliness The ACS, state-level estimates for use in the allocation formula are available approximately 9 months following the reference period. The state-provided counts are submitted by the states to the Department of Education about 6 months after the school year data are collected in the fall and publicly released in July, which is also about 9 months after collection. + + Quality The data from the ACS meet statistical reliability standards as described in this report and are of acceptable precision. State-provided counts are based on administrative data and are not subject to sampling error, although there may be some different interpretation of the instructions for data collection. State-provided counts on immigrant children and youth very much rely on LEA judgments, they and fall short of the quality of the ELL counts or the ACS estimates. ++ + Cost Both the ACS estimates and state-provided counts of the ELL population are available at minimal extra cost. + + Fairness The Census Bureau has an excellent reputation for assuring that the data in its charge are free from manipulation. State data systems and submission procedures have improved such that the data are similarly free from manipulation, but states still have discretion over the timing of submissions and other policies that may affect perceptions of fairness. ++ + Stability The state-provided counts are relatively stable from year to year. The annual ACS estimates for smaller states have been subject to greater variation due to small sample sizes, but they are comparable. The 3-year estimates are more stable than both the 1-year ACS estimates and the state counts. ++ ++
OCR for page 166
Desired Characteristic Evaluation ACS State Provided Insensitivity to Policy and Methodological Differences The ACS estimates are not sensitive to administrative practices or policy differences, although they may be sensitive to differences in demographic composition of the respondents. The state-provided counts are somewhat sensitive to state decisions regarding identification, testing, and program entry and exit policies. The panel has no evidence that these state decisions are made in any way to influence the federal government’s allocation of Title III funds. Nonetheless, the decisions would tend to influence the allocation. ++ + Transparency ACS data are collected by professional staff using highly standardized, well-documented methods. State data are collected by methods that vary from state to state and rely on implementation by local authorities; consequently, documentation of the methods as they are implemented across the country is not readily available. ++ + Comparability The ACS is comparable across geographic and demographic dimensions. The state-based counts conform to definitions promulgated by the U.S. Department of Education but are not comparable in their constructs due to differing state tests and classification and reclassification criteria. ++ + language learners and reclassifying them as former English language learners that we have documented in Chapter 4. We therefore believe that the DoEd should consider a new approach for the 80 percent of the funding that is based on the number of limited English proficient children in the state under Title III—one that uses both data sources, building on the strengths of each of them and recognizing their unique contributions. Because the panel concluded that the allocation formula would gain strength by using data from both data sources in the allocation formula, we discussed various means of doing so. We concluded that both data sources should be used in the allocation formula, with the predominant view that eventually the data sources should be given equal weight—balancing an emphasis on the current need in the state and an opportunity to dampen some of the variability in the ACS measure, and with an equal emphasis on a standardized measure across the states that the ACS offers. For the foreseeable future, the desirable characteristics of the ACS insofar as the quality of the data, the perception of fairness of the ACS, the insensitivity of the estimates to policy changes, and the comparability between geographic areas commend its continued use in the allocation formula. However, as discussed above, the conceptual fit of the state-provided counts strongly commends their use in the allocation formula. Unfortunately, states’ tests,
OCR for page 167
practices, and procedures are not standardized across the country; and there is still much work to be done to improve the quality of the counts (see Chapters 3 and 4). Hence, we recommend a compromise approach that gives some weight initially to the state counts that can increase when the state data are judged to be of sufficient quality (considering the criteria enumerated in Table 7-1) for taking on the burden of constituting an equal share of the allocation. In evaluating these criteria, the DoEd might consider, as a matter of policy, whether the variation across the states in methods and ELP assessments are acceptable for use in determining allocations with the justification that they represent the procedures actually used to determine eligibility of students for services. The department also may wish to consider whether basing a component of the allocation estimates on counts based only on ELL students in public and charter schools is acceptable, given our results indicating that there may be a modest but not insignificant impact on allocations by limiting the counts to this population. RECOMMENDATION 7-1 As soon as technically possible, the U.S. Department of Education should begin to incorporate state-provided counts of English language learner (ELL) students into Title III formula allocation calculations. Initially, the state-provided data should be given a weight of 25 percent of the ELL allocation, with the remaining 75 percent weight given to the American Community Survey data. As discussed in Chapters 3 and 4, states annually report four sets of counts to the DoEd: the number of students who (1) are English language learners, (2) received Title III services, (3) took an ELP test, and (4) scored at the “English proficient” level on that test. Of these four counts, we recommend using those based on the results of the ELP test: that is, the count of ELL students who took an English proficiency test (3, above) and scored below the level the state defines as “English proficient (4, above).” We conclude that this count provides a relatively objective criterion within each state. Furthermore, in our judgment, this count is potentially the least likely to be affected by differing state policies and practices because even if nontest criteria vary (in specification and implementation) across states, the tests share a number of common features that provide a foundation for establishing comparability across them. RECOMMENDATION 7-2 In the portion of the allocation that is based on state-provided data, the U.S. Department of Education should use the state-provided count of the number of students who are determined not to be English proficient on the basis of the state’s English language proficiency test. For the 20 percent of the allocation formula that is determined by the count of immigrant children, as discussed in Chapter 6, the committee does not find that the state-provided estimates have any significant benefits over those from the ACS.
OCR for page 168
RECOMMENDATION 7-3 The U.S. Department of Education should continue to use the American Community Survey estimate as the basis for allocating the 20 percent of the Title III funds that are to be based on the population of recently immigrated children and youth (relative to national counts of these populations). This report has documented a number of areas in which both the ACS and the state-provided data would benefit from additional improvements. With regard to the ACS, we have highlighted the need for more research on the responses of the English speaking ability questions and the need to bring the ACS source questions used in defining the ELL population (currently based only on speaking ability) into closer alignment with professional and legal standards for determining limited English proficiency. Accordingly, we offer two recommendations about the ACS. RECOMMENDATION 7-4 The U.S. Census Bureau should conduct research on the accuracy of the American Community Survey language item for assessing population prevalence of English language learner children and youth, including the strength of its association with more comprehensive English language proficiency (ELP) measures. With the objective of evaluating and improving the item, researchers should examine the effects on responses of situational, cultural, demographic, and socioeconomic factors, placement of the item in the questionnaire, and the ability of adult responders to make ELP distinctions. With regard to the state-provided estimates, a program of research, evaluation, and enhanced data collection is likely to facilitate transition to more extensive use of state-provided data in funding allocations. This work should focus both on studies designed to improve the cross-state comparability of the performance levels that define which students are considered to be English proficient on an ELP test and on strategies for improving the quality of state-provided data. There are several strategies the DoEd could consider for improving the cross-state comparability of state-provided data. One possibility would be to undertake quantitative studies to statistically link the ELP tests, although such studies are likely be of little value because many of the basic assumptions for the strongest form of linking (equating) have not been met. Although weaker linking methods with less stringent assumptions might be possible, we are not optimistic about their utility. Moreover, such studies would be resource intensive. In contrast, qualitative approaches, such as the “crosswalk” analyses we described in Chapter 3, may be useful in evaluating the comparability of the performance levels. These studies could focus on the performance levels set by states to define “English proficient,” seeking to evaluate the extent to which the skills required by the different states are comparable and determining a strategy for setting comparable performance levels across the states.
OCR for page 169
We also note that the current policy environment may foster increased comparability among the ELP tests. The current efforts directed at developing and adopting common English language arts and mathematics content standards across the states and collaborating to develop common assessment systems to measure these standards may affect the comparability of ELP tests. Specifically, changes to states’ English language arts content standards are likely to trigger changes in states’ ELP standards. We anticipate that as English language arts standards, math standards, and the language and literacy aspects of other content standards become more similar across states, so will states’ ELP standards. As a result, the ELP tests that are used by states to measure these standards will likely become more similar and will more easily lend themselves to defining comparable cross-state performance standards for “English proficient.” With regard to improving the quality of state-provided data, there are several steps that the DoEd might explore. First, the department might consider asking states to provide documentation of the technical quality of their assessments, particularly information to document the procedures used to set the performance levels and to determine the “English proficient” level, as well as information to document the accuracy and validity of decisions based on the assessment. This type of information has been required for the English language arts and mathematics achievement tests used by the states to meet the accountability provisions of Title I of the Elementary and Secondary Education Act. These requirements might also be extended to the tests used for Title III, and they would help to ensure and enhance the quality of the data that states provide on their ELL students, particularly the composite and domain-performance levels used to define the “English proficient” standard. Second, the DoEd could continue its efforts to improve the quality, consistency, and completeness of data collected from the states on the Consolidated State Performance Reports and maintained in the Education Data Exchange Network system. As a strategy for improving the comparability of state-provided estimates of students who are considered to be English proficient on the basis of an ELP test becomes available and is implemented—and as evidence of the quality, consistency, and completeness of state-provided data improve—the state-provided data can be accorded more weight in the allocation formula. RECOMMENDATION 7-5 When the quality and cross-state comparability of state-provided data have reached an acceptable standard, the weight given to the state-provided counts should be adjusted upward to the point at which the American Community Survey estimates and the state-provided counts contribute equally to the 80 percent portion of the allocation formula. State-provided counts should continue to be based on the number of students who are determined not to be English proficient on the basis of the state’s English language proficiency test, in a way that is comparable across states.
OCR for page 170
This page intentionally left blank.