Small-Area Estimates of School-Age Children in Poverty: Interim Report 2, Evaluation of Revised 1993 County Estimates for Title I Allocations (1998)

On the recommendation of the Panel on Estimates of Poverty for Small Geographic Areas (National Research Council, 1997), the Department of Education allocated Title I funds to counties for the 1997-1998 school year by averaging estimates of poor school-age children from two sources, the 1990 census and the Census Bureau's original county model for 1993.¹ The panel's recommendation was designed to meet the need for an immediate decision on allocating funds given that there had not been time to thoroughly evaluate the county model and resulting estimates. The recommendation took advantage of the Census Bureau's work to develop model-based estimates that are more up to date than the census estimates but reduced the impact of possible limitations in the model. The panel further recommended a series of evaluations of the Census Bureau's model and alternatives to it, many of which the Census Bureau had begun but had not had time to complete.

Between June and October 1997 the Census Bureau carried out an extensive set of evaluations of its model and a range of alternative models with input and review from the panel. On the basis of those evaluations, the Census Bureau made some changes to its county model and provided to the panel and the Department of Education a revised set of 1993 county estimates of poor school-age children in late October.

The panel commends the Census Bureau staff for their work to investigate thoroughly the properties and pros and cons of a number of alternative county model specifications, some of which represented innovative and methodologically challenging formulations. The work was carded out under severe time constraints to meet the schedule requested by the Department of Education.

Thorough evaluation is a critical component of any estimation approach. The evaluation of county models for estimating the number and proportion of poor school-age children pinpointed strengths and weaknesses of alternative models and identified immediate solutions for some problems. The evaluation results also suggested useful avenues for future research and development (see Chapter 6).

The Department of Education requested that the panel assess the revised 1993 county estimates of poor school-age children for use in Title I allocations for the 1998-1999 school year. (The alternative would be to use the average of the 1990 census estimates and the 1993 estimates from the Census Bureau's original county model for a second round of allocations, that is, not only for the 1997-1998 school year, but also for the 1998-1999 school year.)

The extensive evaluations of the model and alternatives to it that were completed between June and October 1997 provided the information that the Census Bureau needed to assess and refine its county model and for the panel to judge the suitability of the resulting revised estimates for the next round of Title I allocations. The panel's recommendation is based on its review of the county model evaluation results.

The revised estimates should not be averaged with estimates from the 1990 census, as was done for the allocations for the 1997-1998 school year.

The evaluation work initially considered a broad range of alternative formulations and then focused on four similar models that were considered practicable candidates for use in producing revised 1993 county estimates of poor school-age children by October 1997. The panel's recommendation follows from analysis of the final round of evaluations:

• The four candidate models are similar in their performance when evaluated internally in terms of the features of the underlying regression models. When evaluated externally by estimating each model for 1989 and comparing the results with 1990 census estimates of numbers and proportions of poor school-

age children, the four candidate models also show broadly similar performance. However, considering all of the measures of model-census differences that were used in the 1990 census comparisons, two of the four models—the log number (under 18) model (b) and the log rate (under 21) model (c)—perform better than the other two models.² Furthermore, model (b), which is the model the Census Bureau used to produce the revised estimates of poor school-age children in 1993 that were provided to the panel in October 1997, performs somewhat better than model (c).

• When compared with the 1990 census, the Census Bureau's revised county model, model Co), performs substantially better than simple estimation procedures that are based solely or largely on data from the previous (1980) census. (The other three candidate models also perform better than the simple estimation procedures.) Moreover, the revised model performs better than a procedure in which 1989 county estimates are developed by averaging 1980 census estimates with 1989 estimates from the Census Bureau's original log number model (under 21)—the same sort of averaging procedure that was used for the 1997-1998 Title I allocations.

The conclusions from comparisons with census estimates are necessarily based on evaluations for a single year (1989) that use 1990 census estimates of poor school-age children for comparison purposes; there is no comparable basis of evaluation for 1993. Nevertheless, the performance of the Census Bureau's revised county model (and the other three candidate models) in comparison with the estimation procedures that rely more heavily on 1980 census estimates to predict poor school-age children in 1989 makes the panel confident that a model-based approach for 1993 is preferable to using the outdated 1990 census estimates or to averaging 1990 census estimates with model-based estimates for 1993. The major changes in the distribution of children in poverty that have occurred since 1989 make heavy reliance on 1990 census estimates for current allocations highly problematic. There was a 20 percent increase in the number of children in poverty between 1989 and 1993, and there is clear evidence that the geographic distribution of such children changed markedly in those years (National Research Council, 1997:Table 2-3).

The justification for use of the revised 1993 county estimates of poor school-age children in the Title I allocations for the 1998-1999 school year is clear. At the same time, continuing work on research, evaluation, and development for the

county model (and state model) is warranted to further improve the methods for producing updated county estimates of poor school-age children in the future (see Chapter 6). The evaluation work identified areas of potential improvement in which progress could be made in the near term—for example, in the estimation of the sampling error variance of the county regression model and in clarifying the relationship between the state and county regression models. The evaluation work also identified promising alternative specifications for the county model, such as the bivariate formulation, that will require longer term research to develop.

Over time, changes in the nature of the available data for model-based estimates will require evaluation and are likely to lead to modification of the county model. For example, changes in welfare programs may affect the comparability of food stamp program data across states and even across counties within states. Also, data from the 2000 census and the possibility of a continuing large-scale household survey in the next decade, the American Community Survey, will require rethinking the best approach for producing county estimates of poor school-age children. Consequently, research and development for the county estimates will need to continue even while the Census Bureau works on the challenging task of developing updated estimates of poor school-age children for school districts.

The Title I allocations include Puerto Rico, which is treated as a county equivalent. While the commonwealth's 1990 decennial census provides estimates for 1989, no estimates of Puerto Rican children in poverty can be made for 1993 from the Census Bureau's model because the appropriate IRS and food stamp program data are not available for Puerto Rico. The Census Bureau computed 1993 estimates for Puerto Rico from data collected in a Family Income Survey that was conducted in the commonwealth in February and March 1995. Several adjustments had to be made to produce the estimates of school-age children in poverty in 1993. The original 1993 estimates for Puerto Rico, which were averaged with 1990 census estimates for Title I allocations for the 1997-1998 school year, have not been revised.

The panel concluded in its first interim report that the approach adopted by the Census Bureau for producing 1993 estimates of poor school-age children in Puerto Rico seems a reasonable one given the data available, although there is limited information about the quality of the data (see National Research Council, 1997:App. F). Because there is no alternative at this time and for consistency with the treatment of U.S. counties, the panel recommends that the original 1993 estimates for Puerto Rico be used in the Title I allocations for the upcoming 1998-1999 school year. They should not be averaged with 1990 census estimates.

The Puerto Rico income survey was repeated in 1997 and is expected to be conducted at 2-year intervals in the future. It will presumably be the basis of updated estimates of poor school-age children in Puerto Rico for 1995 and later years. Further investigation should be carded out of the quality of the estimates, and their comparability with the model-based estimates for U.S. counties, to determine if there are ways in which the data and estimation procedures for Puerto Rico can be improved.