ensure that the survey is grounded in relevant concepts, applying the most up-to-date statistical methodology, and invested with the necessary design, estimation, and analytical techniques to ensure a quality product.

ARMS is a complex undertaking. From its start as a melding of data collected from the field, the farm, and the household in a multiphase, multiframe, and multiple mode survey design, it has increased in complexity over the decade of its existence as more sophisticated demands for its outputs have been made. Today, the survey faces difficult choices and challenges, including a need for a thorough review of its methods, practices, and procedures.

The Panel to Review USDA’s Agricultural Resource Management Survey was established to conduct such a review, and this report is the product of its efforts. The panel’s specific recommendations appear in context throughout the report and are presented together in Chapter 9.

DATA QUALITY

The panel focused on the elements of quality, broadly defined as “fitness for use,” which for ARMS means relevance, accuracy, timeliness, accessibility, interpretability, and coherence. Assessing these elements of quality for ARMS required a review of its concepts, organization, sampling, questionnaire design, data collection, data processing, and dissemination. In doing so the panel has addressed issues of concern for ARMS and its uses for policy analysis and private-sector decision making in order to identify specific needed improvements, to outline testing and research to keep the survey current with data needs and state-of-the-art methods in the future, and in making it accessible to potential users.

The central qualitative dimension of survey data is accuracy. This is the element the panel was least able to assess with confidence, because knowledge is inadequate about the true values of many data items that ARMS estimates. Obtaining better information about the accuracy of survey-based information from the survey is a central reason for our call, sounded throughout this report, for a systematic program of methodology research and development, which would focus on questionnaire design, survey management, bias resulting from nonresponse, quality checks on responses, editing and imputation procedures, calibration to data from sources other than ARMS, and statistical procedures for calculating confidence intervals on estimates derived from ARMS data.

CONCLUSIONS

ARMS is an invaluable source of information on the current state of American agriculture, as well as the sole source of some important infor-



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement