The Agricultural Resource Management Survey (ARMS) is the federal government’s primary source of information on the financial condition, production practices, and resource use on farms, as well as the economic well-being of America’s farm households. ARMS data are important to the U.S. Department of Agriculture (USDA) and to congressional, administration, and industry decision makers when they must weigh alternative policies and programs that touch the farm sector or affect farm families.
ARMS was initiated in 1996 as a synthesis of existing USDA surveys on cropping practice, chemical use, and farm costs and returns. The survey is managed jointly by two USDA agencies: the Economic Research Service (ERS) and the National Agricultural Statistics Service (NASS). The three-phase annual survey is large, complex, and costly, with a budget of nearly $19 million in fiscal year 2006.
ARMS is unique in several respects. As a multiple-purpose survey with an agricultural focus, ARMS is the only representative national source of observations of farm-level production practices, the economics of the farm businesses operating the field (or dairy herd, greenhouse, nursery, poultry house, etc.), and the characteristics of the American farm household (age, education, occupation, farm and off-farm work, types of employment, family living expenses, etc.). No other data source is able to match the range and depth of ARMS in these areas.
American agriculture is changing, and the science of statistical measurement is changing as well. As with every major governmental data collection with such far-reaching and important uses, it is critical to periodically
ensure that the survey is grounded in relevant concepts, applying the most up-to-date statistical methodology, and invested with the necessary design, estimation, and analytical techniques to ensure a quality product.
ARMS is a complex undertaking. From its start as a melding of data collected from the field, the farm, and the household in a multiphase, multiframe, and multiple mode survey design, it has increased in complexity over the decade of its existence as more sophisticated demands for its outputs have been made. Today, the survey faces difficult choices and challenges, including a need for a thorough review of its methods, practices, and procedures.
The Panel to Review USDA’s Agricultural Resource Management Survey was established to conduct such a review, and this report is the product of its efforts. The panel’s specific recommendations appear in context throughout the report and are presented together in Chapter 9.
The panel focused on the elements of quality, broadly defined as “fitness for use,” which for ARMS means relevance, accuracy, timeliness, accessibility, interpretability, and coherence. Assessing these elements of quality for ARMS required a review of its concepts, organization, sampling, questionnaire design, data collection, data processing, and dissemination. In doing so the panel has addressed issues of concern for ARMS and its uses for policy analysis and private-sector decision making in order to identify specific needed improvements, to outline testing and research to keep the survey current with data needs and state-of-the-art methods in the future, and in making it accessible to potential users.
The central qualitative dimension of survey data is accuracy. This is the element the panel was least able to assess with confidence, because knowledge is inadequate about the true values of many data items that ARMS estimates. Obtaining better information about the accuracy of survey-based information from the survey is a central reason for our call, sounded throughout this report, for a systematic program of methodology research and development, which would focus on questionnaire design, survey management, bias resulting from nonresponse, quality checks on responses, editing and imputation procedures, calibration to data from sources other than ARMS, and statistical procedures for calculating confidence intervals on estimates derived from ARMS data.
ARMS is an invaluable source of information on the current state of American agriculture, as well as the sole source of some important infor-
mation on the linkages between fields, farms, and families that serves to illuminate the challenges faced by agriculture policy makers and farm families. Because the survey is so critical to understanding agriculture, it carries a special burden. Its methods, practices, and procedures must be designed to yield data of impeccable quality in view of their uses, and the data must be made available to the research community both inside and outside the federal government in order to generate the improved analytical knowledge the data makes possible.
At several points in this report, some of the methods and practices used in ARMS are characterized as “unique” or “unconventional.” In large part, the unique nature of the survey is due to its complexity, with multiple modes and phases and with a goal to collect, classify, and aggregate several types of information from three interrelated but not entirely overlapping reporting units. ARMS also reflects some unique practices that are part of USDA’s way of doing business, such as its board review process, which are not within the panel’s purview to assess. Nonetheless, we have been able to document and assess the adequacy of the survey’s design, data collection, analysis, and dissemination.
The panel concludes that ARMS has been carried out with admirable attention to achieving high standards on most of its elements. Nonetheless, there is room for improvement in all of them. In addition to identifying areas of needed improvement in current methods and practices, our review identifies several emerging challenges. These challenges are associated with the changing structure of farming, overall trends in federal surveys—such as the growing difficulty of obtaining satisfactory survey response—and the growing sophistication of survey data users, both inside and outside the federal government. NASS and ERS have attempted to respond to these challenges with some foresight—adding new questions, testing such initiatives as incentives for increasing reporting, developing proposals to collect longitudinal data, and enhancing the provision of microdata files in a protected environment. Our review leads to the conclusion that a number of areas need still further attention, and our recommendations can be considered a roadmap to the future for ARMS.
The panel also examined the appropriateness of the methods that ERS is using to fit statistical models to data from ARMS. The panel concludes that the current practice of NASS to provide survey weights with the ARMS data set, as well as the NASS and ERS recommendation to use the design-weighted approach for many of the analytical uses of the data, are appropriate and should be continued. However, the current one-size-fits-all approach for analytical inference for ARMS is somewhat restrictive, and users need more specific information on the sampling design and on nonresponse patterns and adjustments applied to the data sets. The panel concludes that a guide for researchers on how to approach certain com-
monly used analytical methods would improve the quality of analysis of the ARMS data.
Although appropriate attention is being paid to the basic elements of survey quality, much more can be done to improve important aspects of the survey. Issues that need more attention include the employment of analytical tools to investigate the quality of survey responses, additional control and further automation of the interview process, shift of focus from nonresponse rates to nonresponse bias, introduction of new methods of imputation of missing values and documentation of the results of imputation, improvements to variance estimation that will be more compatible with the types of data analysis that are now employed, and more attention to facilitating access to the data files for research and analysis.
The panel did not explicitly prioritize either the issues or our recommendations, although we do draw special attention to the need for an ongoing, joint, and appropriately funded methodology research and development program. We draw attention to special needs for a research and development program to support improvements in questionnaire design and development, unit and item nonresponse, and imputation and estimation. Such a program needs adequate resources both to support current and future research, development, and statistical analysis needs throughout the implementation of ARMS and to assess and manage the quality of the data. We also call attention to the need for better channels of communication with providers and users of the data. These initiatives will require an infusion of funding, and, in the case of ERS, enhancement of staff expertise in mathematical statistics and data analysis skills.
We are aware that our list of recommendations is long and that some of them will be costly to implement. Full implementation of all of them would require a significant fraction of the ARMS budget. In our view, if additional funds cannot be obtained, at least those recommendations involving research and development directly related to data quality assurance should be undertaken, even at the expense of reducing the size or scope of the survey. For other costly recommendations, notably the training programs and other services for data users outside USDA, additional funding could reasonably be sought, even from unconventional sources in the user community. For example, the land grant universities could be asked to support and perhaps to assist in implementing the training and data access improvements. The universities rely on other sources of USDA funding through the Cooperative Research, Education, and Extension Service, which might be interested in funding competitive National Research Initiatives or other grants for these purposes.
We recommend improvements to ARMS in all aspects of the survey:
Data integration and relevance, including survey integration and longitudinal data.
Survey management, including establishment of a research and development program, planning for changes in survey content, review by the NASS Advisory Committee on Agriculture Statistics, and stakeholder feedback and discussion.
Sample and questionnaire design, including expanded use of generic clearance for survey testing from the Office of Management and Budget, cognitive or usability laboratory facilities, designation of subsamples for research and testing, consistency of variables, and formal collection of auxiliary information.
Data collection, including paradata (i.e., data on the data collection process) and metadata (i.e., data on data items), the consequences of departures from standardized interview techniques, using available analytical tools to assess survey response quality, and moving to computer-assisted interviews and web-based data collection.
Nonresponse, imputation, and estimation, including providing a foundation for appropriate response rate calculations, adjustment for response rates in the multiple phases, nonresponse bias, making the survey mandatory, changes to reduce item nonresponse, approaches to imputation, flagging missing data used for analysis, and clarifying the estimation process.
Methods of analysis, including provision of sampling weights, using the design-weighted approach, jackknife replicates, design and nonresponse characteristics, in-house survey statistics expertise at ERS, and developing a guide for researchers.
Dissemination, including improving the ARMS web tool, extending the availability of ARMS microdata, training for new data users, and database management practices.
INVESTMENT IN THE FUTURE
The ARMS program represents a significant investment of time, talent, respondent burden, and cost. It responds to congressional mandates to produce estimates of income for farms as business establishments by type of operation, to monitor the status of family farms, and to make annual estimates of the costs of producing commodities, such as wheat, feed grains, cotton, and dairy, covered under farm support legislation.
ARMS also enables USDA to meet its responsibility to support the U.S. national accounts by producing estimates of farm sector value-added and net income and the development of regional (state and county) accounts by provision of the state-level estimates of net farm income. It provides
the data to implement USDA program requirements and to formulate and evaluate USDA polices and programs. The analytical program of ERS informs both public- and private-sector decision makers, through research and analysis on a wide variety of farm, farm household, environmental, and other rural issues, shedding light on important national issues like energy usage by supporting analysis of the impact of energy prices on farm production costs and, consequently, on the price of commodities.
As it begins a second decade of operation, improvements are needed in all aspects of the survey’s operations to keep it vital. ARMS is too valuable to the nation for the U.S. Department of Agriculture not to make efforts now to increase its value for the future.