6
Next Steps

The panel commends the Census Bureau for establishing a research program to develop methods to provide county estimates of school-age children in poverty that are more timely than those from the decennial census so that Title I funds can be more appropriately allocated, as Congress intended. The work that has been completed by the Census Bureau makes a strong case that model-based county estimates can be produced that are preferable to the estimates derived from the decennial census.

Though we unequivocally support a model-based approach, more analytical work is needed before we can endorse a specific model for allocating Title I funds. For this important purpose, there should be evidence that the model has been evaluated as fully as possible and that there are no systematic biases that might favor one population group over another. Before users can be confident in the model-based small-area estimates and before the panel can unequivocally recommend them as the sole basis for fund allocation, the behavior of the selected model and the resulting estimates should be well understood, and alternative models should have been thoroughly evaluated.

Evaluation of estimates can typically be separated into two kinds of activities. First, in an ''external" evaluation, estimates can be compared with "comparison" values or values that serve as substitutes for true values. The Census Bureau designed its model and expended substantial resources for the purpose of enabling a fair comparison with 1990 decennial census estimates of school-age children in poverty in 1989. Unfortunately, there are no additional comparison values for other time periods that can be used for this purpose, thereby limiting this type of evaluation to 1989.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 41
Small-Area Estimates of School-Age Children in Poverty: Interim Report I: Evaluation of 1993 County Estimates for Title I Allocations 6 Next Steps The panel commends the Census Bureau for establishing a research program to develop methods to provide county estimates of school-age children in poverty that are more timely than those from the decennial census so that Title I funds can be more appropriately allocated, as Congress intended. The work that has been completed by the Census Bureau makes a strong case that model-based county estimates can be produced that are preferable to the estimates derived from the decennial census. Though we unequivocally support a model-based approach, more analytical work is needed before we can endorse a specific model for allocating Title I funds. For this important purpose, there should be evidence that the model has been evaluated as fully as possible and that there are no systematic biases that might favor one population group over another. Before users can be confident in the model-based small-area estimates and before the panel can unequivocally recommend them as the sole basis for fund allocation, the behavior of the selected model and the resulting estimates should be well understood, and alternative models should have been thoroughly evaluated. Evaluation of estimates can typically be separated into two kinds of activities. First, in an ''external" evaluation, estimates can be compared with "comparison" values or values that serve as substitutes for true values. The Census Bureau designed its model and expended substantial resources for the purpose of enabling a fair comparison with 1990 decennial census estimates of school-age children in poverty in 1989. Unfortunately, there are no additional comparison values for other time periods that can be used for this purpose, thereby limiting this type of evaluation to 1989.

OCR for page 41
Small-Area Estimates of School-Age Children in Poverty: Interim Report I: Evaluation of 1993 County Estimates for Title I Allocations Second, an "internal" assessment would include, to the extent possible: (1) a comparison of the estimated variances of the updated estimates with variances and estimated biases of the decennial census estimates; (2) an evaluation of the model's assumptions, including the assumptions underlying the above variance estimates; and (3) a determination to the extent possible of the systematic errors (biases) in the model's estimates and how these biases could be reduced. The Census Bureau has performed a number of these internal assessments, including examination of estimated variances for the estimates (developed with variance estimation methods that reflect recent contributions to the literature in this area); residual plots and other techniques to investigate homogeneity assumptions; and regional indicator variables to investigate the possibility of regional biases. A substantial start has been made. As part of both external and internal evaluations, it is useful to compare the performance of the selected model with alternative models, not only to determine whether the selected model is preferred to the alternatives, but also to gain more understanding of the performance of the selected model. This kind of analysis using alternative models is particularly needed in this case. Some of this activity has been carried out by the Census Bureau for some models that predict change in poverty over time, for models that predict poverty rates, for models that constrain the coefficients of the predictor variables (on the logarithmic scale) to sum to 1, and for simple improvements to the decennial census poverty estimates (e.g., controlling them to the Census Bureau's state estimates of the number of school-age children in poverty); however, more needs to be done to evaluate these and other models in comparison with the selected model. Finally, more evaluation using comparative analyses with other sources of information is needed. The Census Bureau's estimates indicate substantial changes in the number of school-age children in poverty by counties since 1989. There has been little analysis to determine whether these changes correspond to what is known about those counties, both locally and regionally. FURTHER EVALUATION OF THE COUNTY-LEVEL MODEL AND ESTIMATES We outline here some steps that could be taken in the near future to achieve a greater understanding of the properties of the new county estimates and the model that generated them (see Appendix E for more details). Alternative Modeling Approaches Alternative approaches to modeling the number of children living in poverty could be more fully examined. As noted in Section 3, the Census Bureau adopted a cross-sectional approach rather than a change model in order to avoid difficulties arising from changes in tax and transfer programs. For the time period in question, no substantial programmatic changes occurred. Therefore, it is reasonable to examine more fully a change

OCR for page 41
Small-Area Estimates of School-Age Children in Poverty: Interim Report I: Evaluation of 1993 County Estimates for Title I Allocations model for production of 1993 estimates. The Census Bureau also decided to model poverty levels rather than poverty rates to facilitate estimation of variances of model predictions. However, the panel believes that adequate variance estimates can be produced for a model of poverty rates and that the primary job of producing estimates of the highest quality dominates the need to produce precise estimates of variances. Therefore, we believe that models of rates also need to be examined more fully. Other kinds of models that could be investigated include models that borrow information from similar counties or close-by years and models that use information on state poverty levels as part of an integrated state-county modeling approach, rather than controlling the county estimates from the model to agree with the state estimates from a different model. Interaction terms for the independent variables and other indicator variables—especially to account for geographic heterogeneity, growth in poverty, or other potential sources of bias—could be fully examined. In addition, bias resulting from the log transformation could be examined through the use of generalized linear models that link estimates of poverty numbers or rates to the predictor variables. Other Evaluation Activities Five other kinds of further analyses would be useful. First, CPS-census differences could be more fully examined to understand the effect of a change from one measurement system to the other. In addition to studying the quantitative effect of the differences, the qualitative differences between the two measurement systems could also be evaluated. This examination would help to explain the degree to which differences between 1989 model-based estimates and the 1990 decennial census estimates for 1989 were due to CPS-census differences and would help support measurement error models (discussed below). Second, the weights used in the weighted regressions for the county estimates are based on assumptions concerning model error and sampling variability that could be more fully explored. Although moderate failure of these assumptions may not be a concern, severe failure could result in lowering the quality of the estimates. At least two exercises could be carried out: (1) in validating the county-level estimation of the CPS sampling variances, the directly estimated sampling variances for large counties could be compared to those determined by subtracting out the model error from the county-level model; (2) on the basis of directly estimated variances, models could be developed to estimate sampling variances for all counties, and these estimates could be used to estimate the model error for 1989 and 1993. These estimates could then be compared to the model error from the 1990 census regression (see Appendix E). This method, and the method used by the Census Bureau, should produce similar weights. If they do not, the reason for the difference needs to be understood. Third, some anomalies could be further examined: (1) The sums of the initial county estimates, before controlling to the state estimates, often differ from

OCR for page 41
Small-Area Estimates of School-Age Children in Poverty: Interim Report I: Evaluation of 1993 County Estimates for Title I Allocations the state estimates to a substantial extent, and these differences should be investigated; (2) When the county-level model is reestimated as a model of poverty rates, the residual is correlated with population size. This feature has a beneficial impact on the model for reasons that are not understood. Fourth, counties that experienced large changes in the estimated number of children in poverty from 1989 to 1993 could be examined to see whether there are other indications that such changes actually occurred. In addition, for the counties included in the 1989 and 1993 CPS regressions, the counties with the largest residuals (especially from a robust fit) could be subject to further review to see whether there are characteristics that these counties share. This technique is often helpful for identifying additional useful predictor variables in regression models. It is important, however, to note that with any model, there will be some units of observation—in this case, counties—for which the predictions are relatively poor. This fact is simply a property of the methodology and is not a basis for rejecting the results. The finding of several dozen counties with poor predicted values is to be expected and is acceptable, as long as these counties have nothing in common. Random error tends to balance out over time. In contrast, systematic error persists: the primary goal is to root out systematic error, which is indicated by a pattern of similar counties with poor predicted values that are in the same direction. Fifth, the county-and state-level models make use of predictor variables from administrative records, but the quality of these variables is unclear and so is of concern for this approach. The Census Bureau has performed careful analysis of the definitional, coverage, and nonresponse issues raised through the use of the administrative records (see Coder et al., 1996). More work could be done, however, to further understand the uniformity of the relationships to poverty across counties and the likelihood of misresponse. In this regard, we note that in 10 percent of counties, the number of reported child exemptions on income tax returns exceeded that of the estimated total population under age 21. Although this anomaly is not entirely unexpected because of differences between reporting on income tax returns and reporting in the census, which is the basis of population estimates, it needs further study.1 The panel expects that after all the above analyses have been carried out, it will not only be clear that model-based county estimates are preferable to the decennial census estimates for current allocations, but it will also be relatively clear which particular model—the Census Bureau's current model or one of the alternatives suggested in this report—is preferable for providing updated estimates of school-age children in poverty. We say relatively clear because there is 1   For example, addresses on tax returns are not always the county of residence as defined for the census (e.g., the address may be that of the tax preparer); tax filers may report exemptions for children who do not reside with them; and some child exemptions are for children aged 21 or older.

OCR for page 41
Small-Area Estimates of School-Age Children in Poverty: Interim Report I: Evaluation of 1993 County Estimates for Title I Allocations at present only a single opportunity for the use of comparison values. Therefore, there will always be some degree of uncertainty as to which particular set of model-based estimates is preferable. OTHER RESEARCH The panel encourages the Census Bureau to proceed along lines of innovative research that it has already begun. The rush to produce estimates for use in fiscal 1997 and 1998 allocation forced the Census Bureau to put off some interesting work related to a time-series approach that uses correlations for 5 years of CPS data to produce state estimates. It is possible that this approach could be used to produce county estimates as well. These kinds of time-series models have only recently been proposed in the literature and have not been sufficiently tested. The Census Bureau is commended for its initial research into this area and is encouraged to continue, although such work will likely represent a long-term effort and not a contribution to the immediate problem of evaluating and improving the county estimates in the short term. The Census Bureau has also begun work on a measurement error model to provide more understanding of CPS-census differences. This is an important effort. In addition to assisting in the understanding of the comparison of the 1989 model estimates with the 1990 decennial census estimates, such work could also be useful when developing estimates for 1999 by providing a way to produce poverty estimates from the decennial census that are consistent with the way poverty is measured by the CPS. Another area for investigation concerns timeliness of the estimates. Although the Census Bureau's model-based estimates of poor school-age children are more up to date than the 1990 census estimates, the estimates for 1993 lag the allocation year (1997) by several years. It would be useful to consider possible methods to make model-based estimates yet more timely. Such methods could range from basing the county estimates on 1 year of March CPS data, instead of 3 years (which could reduce the lag by a year at the cost of increased variability in the estimates), to research into the dynamics of poverty across time and geographic areas that would make it possible to project the model-based estimates several years forward. Such research is likely to be difficult to carry out, but it and other methods to improve timeliness should be identified along with their possible advantages and disadvantages as a first step in developing a research plan in this area. Still another area for investigation includes evaluation and refinement of the methods used to develop postcensal population estimates for children aged 5–17, which for 1994 represented relatively crude adjustments of estimates from the 1990 census. Such work should include research on methods to estimate the errors in the population estimates. There should also be related work to evaluate and improve the estimates of total and poor school-age children in Puerto Rico

OCR for page 41
Small-Area Estimates of School-Age Children in Poverty: Interim Report I: Evaluation of 1993 County Estimates for Title I Allocations with regard to their quality and comparability with the estimates for U.S. counties. A major thrust of the next phase of the Census Bureau's work will be to determine whether updated estimates of poor school-age children at the level of school districts can be produced that are of adequate quality and preferable to decennial census estimates. Certainly, given the small CPS sample sizes for most school districts, the paucity of good administrative record data, the frequent revisions of school-district boundaries, the dynamics of school district-level poverty, and other factors, this work will be more difficult than county-level estimation, and evaluation will be critically important. The panel looks forward to working with the Census Bureau on this task as part of its continuing study. CONCLUSION The Census Bureau has made an excellent start on a difficult problem. The updated estimates it has produced of the number of school-age children in poverty for counties represent substantial progress in meeting an important need. It is the panel's strong belief that with the evaluations and other analyses we have outlined, a model-based approach will produce updated poverty estimates that are superior to those from the decennial census. What the panel has outlined represents a considerable amount of work, but such work is necessary to properly support the use of model-based estimates for the important purpose of allocating funds under Title I. We stress that it is critical that users be provided with information that will enable them to fully understand the properties of model-based estimates for use for fund allocations and other purposes. Therefore, when official estimates of the number of school-age children in poverty are released by the Census Bureau, they should be accompanied by formal documentation that details the methods used, the analyses conducted, the estimates of reliability, and the detailed results of evaluations, in order to inform users of the basis of the estimates and to give the research community opportunities for further analyses.