National Academies Press: OpenBook

Improving Crop Estimates by Integrating Multiple Data Sources (2017)

Chapter: Appendix B: Routine External Evaluation Protocol

« Previous: Appendix A: NASS County-Level Survey Programs
Suggested Citation:"Appendix B: Routine External Evaluation Protocol." National Academies of Sciences, Engineering, and Medicine. 2017. Improving Crop Estimates by Integrating Multiple Data Sources. Washington, DC: The National Academies Press. doi: 10.17226/24892.
×

Appendix B

Routine External Evaluation Protocol

Dorfman (2017) proposes that the Routine External Evaluation Protocol (REEP) be used as part of external evaluation.1 He argues that, as the demand for small-area estimation is ever-increasing, statisticians have the responsibility to develop a protocol aimed at setting criteria and enabling detection of “a tipping point” at which estimates produced using small-area models cannot be regarded as satisfactory. This is particularly difficult to determine in groups of areas where there is little or no sample and that may behave differently from larger areas.

For NASS, REEP would be based on comparing county model estimates with the corresponding direct sample-based estimates where the latter are based on a large enough sample. The procedure would be implemented by selecting a random sample of counties within which a supplementary sample of farms would be selected and used to prepare direct estimates.

___________________

1 If NASS decides to research the feasibility of the above REEP, a number of questions will have to be answered before the implementation:

  • the procedure for selecting the random sample at Step 1, the number of counties needed, and the size of the sample selected within each county;
  • specific procedure to amend the current adaptive design for data collection;
  • specific method of evaluation and diagnostics, including developing formal criteria;
  • estimated additional cost;
  • estimated benefits from having the additional sample (beyond the goal of the model evaluation);
  • possibility of evaluation (simulations?) based on historical data;
  • whether it is possible to construct a procedure in such a way as to be able to modify the set of publishable counties based on the test results, and in the case where a model fails, whether it can still be applied to a subset of counties.
Suggested Citation:"Appendix B: Routine External Evaluation Protocol." National Academies of Sciences, Engineering, and Medicine. 2017. Improving Crop Estimates by Integrating Multiple Data Sources. Washington, DC: The National Academies Press. doi: 10.17226/24892.
×

Step 1. Select at random a certain number of counties from the set of all counties that are expected to be estimated using models because the regular sample size does not meet publication standards.

Step 2. From each county selected at Step 1, take a supplementary sample. This sample should be large enough so that even with nonresponse, the direct sample-based estimates meet publication standards. These direct estimates and their variances will serve to test the quality of the model estimates.

Remark 1: For the County Agricultural Production Survey (CAPS), the current sample design already attempts to sample enough units from every county (the target sample size for each county is 70 for each commodity, with the goal of having 30 positive reports or at least 25% coverage within the county). The adaptive design approach used by NASS to target counties for survey nonresponse follow-up may need to be modified to give priority to the counties selected for external evaluation.

Remark 2: In the presence of high nonresponse, this supplementary sample may also be potentially useful in assessing the nonresponse bias. Compare the estimates in selected counties with all the other counties. Since the counties at Step 1 were selected at random, the average estimate that includes the supplementary sample can be compared with the rest of the sample; if there is no bias due to nonresponse, the estimates should be close.

Step 3. Use the “main sample” to compute model-based estimates and the full sample for evaluation and diagnostics. Note: The advantage of this “external” test is that it is formal (objective) and contemporaneous.

Step 4. Once the goal of model evaluation is accomplished, the supplementary sample can be combined with the main sample to produce better direct survey estimates and model-based estimates. Thus, even in the situation where model-based estimates fail, the estimates for counties selected at Step 1 will still be published.

Suggested Citation:"Appendix B: Routine External Evaluation Protocol." National Academies of Sciences, Engineering, and Medicine. 2017. Improving Crop Estimates by Integrating Multiple Data Sources. Washington, DC: The National Academies Press. doi: 10.17226/24892.
×
Page 123
Suggested Citation:"Appendix B: Routine External Evaluation Protocol." National Academies of Sciences, Engineering, and Medicine. 2017. Improving Crop Estimates by Integrating Multiple Data Sources. Washington, DC: The National Academies Press. doi: 10.17226/24892.
×
Page 124
Next: Appendix C: Small-Area Modeling in Space and Time with Multiple Data Sources »
Improving Crop Estimates by Integrating Multiple Data Sources Get This Book
×
Buy Paperback | $65.00 Buy Ebook | $54.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The National Agricultural Statistics Service (NASS) is the primary statistical data collection agency within the U.S. Department of Agriculture (USDA). NASS conducts hundreds of surveys each year and prepares reports covering virtually every aspect of U.S. agriculture. Among the small-area estimates produced by NASS are county-level estimates for crops (planted acres, harvested acres, production, and yield by commodity) and for cash rental rates for irrigated cropland, nonirrigated cropland, and permanent pastureland. Key users of these county-level estimates include USDA’s Farm Services Agency (FSA) and Risk Management Agency (RMA), which use the estimates as part of their processes for distributing farm subsidies and providing farm insurance, respectively.

Improving Crop Estimates by Integrating Multiple Data Sources assesses county-level crop and cash rents estimates, and offers recommendations on methods for integrating data sources to provide more precise county-level estimates of acreage and yield for major crops and of cash rents by land use. This report considers technical issues involved in using the available data sources, such as methods for integrating the data, the assumptions underpinning the use of each source, the robustness of the resulting estimates, and the properties of desirable estimates of uncertainty.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!