Conclusions and Recommendations
The Agricultural Resources Management Survey (ARMS) is a complex undertaking. It began, just over a decade ago, as a melding of data collected from the field, the enterprise, the farm, and the household, in a multiphase, multiframe, and multiple mode survey design, and it has increased in complexity in the ensuing years as more sophisticated demands for its outputs have expanded. Today, at the outset of its second decade of operations, the survey faces difficult choices and challenges, including a need for a thorough review of its methods, practices, and procedures. This report contributes to that necessary endeavor.
This chapter summarizes the panel’s review in a short statement of our major conclusions and presents the recommendations that appear in Chapters 2 through 8.
ARMS is an invaluable source of information on the current state of American agriculture, as well as the sole source of some important information on the linkages between fields, farms, and families that serves to illuminate the challenges faced by agriculture policy makers and farm families. Because the survey is so critical to understanding agriculture, it carries a special burden. Its methods, practices, and procedures must be designed to yield data of impeccable quality in view of their uses, and the data must be made available to the research community both inside and outside the federal government in order to generate the improved analytical knowledge the data makes possible.
The panel’s review of several aspects of the survey’s statistical quality was challenging. At several points in this report, some of the methods and practices used in ARMS are characterized as “unique” or “unconventional.” In large part, the unique nature of the survey is due to it complexity, with multiple modes and phases and with a goal to collect, classify, and aggregate several types of information from three interrelated but not entirely overlapping reporting units. ARMS reflects some unique practices that are part of the U.S. Department of Agriculture’s (USDA) way of doing business, such as its board review process, which are not within the panel’s purview to assess. Nonetheless, the panel has been able to document and assess the adequacy of the survey design, data collection, analysis, and dissemination.
The panel concludes that appropriate attention is being paid by the National Agricultural Statistics Service (NASS) and the Economic Research Service (ERS) to the basic elements of survey quality, although much more could be done to improve important features of the survey. Several aspects of survey operations need more attention, including the employment of analytical tools to investigate the quality of survey responses, additional control and further automation of the interview process, shifting the focus from nonresponse rates to nonresponse bias, introduction of new methods of imputation of missing values and documentation of the results of imputation, improvements to variance estimation that are more compatible with the types of data analysis uses that are now employed, and more attention to facilitating access to the data files for research and analysis.
In addition to identifying areas of needed improvement in current methods and practices, our review identifies several emerging challenges. These challenges are associated with the changing structure of farming, overall trends in federal surveys—such as the growing difficulty of obtaining satisfactory survey response—and the growing sophistication of survey data users, both inside and outside the federal government. The agencies have attempted to respond to these challenges with some foresight—adding new questions, testing such initiatives as incentives for increasing reporting, developing proposals to collect longitudinal data, and enhancing the provision of microdata files in a protected environment. Our review leads to the conclusion that several areas still need attention, and the recommendations that follow may be considered a roadmap to the future for ARMS.
We are aware that our list of recommendations is long and that some of them will be costly to implement. Full implementation of all of them would require a significant fraction of the ARMS budget. In our view, if additional funds cannot be obtained, at least those recommendations involving methodological research and development directly related to data quality assurance should be undertaken, even at the expense of reducing the size or scope of the survey. For other costly recommendations, notably the training
programs and other services for data users outside USDA, additional funding could reasonably be sought, even from unconventional sources in the user community. For example, the land grant universities could be asked to support, and perhaps to assist in implementing, the training and data access improvements. The universities rely on other sources of USDA funding, through the Cooperative State Research, Education, and Extension Service, which might be interested in funding competitive National Research Initiative or other grants for these purposes.
Although the panel did not attempt to prioritize the issues or our recommendations, we do draw special attention to the need for an ongoing, joint, and appropriately funded methodology research and development program. Such a program needs adequate resources both to support current and future research, development, and statistical analysis needs throughout the implementation of the ARMS and to assess and manage the quality of the data. We also call attention to the need for better channels of communication with providers and users of the data. These initiatives will require an infusion of funding and, in the case of ERS, enhancement of staff expertise in mathematical statistics and data analysis skills. The panel believes that all of the recommendations are feasible and important, but some are more important than others and are worded to convey that immediacy.
In the pages that follow, we present the recommendations that appear in context throughout the report.
Data Integration and Relevance
Recommendation 2.1: The National Resources Conservation Service, NASS, and ERS should engage in a focused research and testing program and use experience with integrating the Conservation Effects Assessment Project and ARMS to assess the feasibility of integrating ARMS with other surveys and data sources.
Recommendation 2.2: In preparation for funds becoming available for a longitudinal design of ARMS, ERS and NASS should systematically conduct research and explore the need for and feasibility of obtaining panel data from ARMS. Furthermore, as a test of the power of such information, more use should be made of the existing longitudinal microdata from the Census of Agriculture. One possible approach would be to create a pseudopanel of such data. Another would be to make a retrospective link between the Census of Agriculture and a year of ARMS.
Recommendation 3.1: The ARMS program should have structured mechanisms in place for stakeholder feedback and discussion on ARMS, beyond what is currently done, such as organized stakeholder forums, with some obligation to respond. Specifically, USDA should solicit input in developing the survey from stakeholders from within USDA and from other government agencies, universities, professional associations, and the private sector.
Recommendation 3.2: The NASS Advisory Committee on Agriculture Statistics should expand its scope to include an annual review of ARMS.
Recommendation 3.3: ERS and NASS should establish an ongoing, jointly sponsored, and appropriately funded methodology research and development program. Such a program should provide adequate resources to support current and future research, development, and statistical analysis needs throughout the implementation of ARMS and to assess and manage the quality of the data. If new funds cannot be obtained, funds from existing programs must be reallocated.
Recommendation 3.4: NASS and ERS should commit resources to developing a five-year plan tied to the Census of Agriculture for ARMS content, coverage, and methodology. The agencies should develop measures to control changes during the five-year period to minimize disruptions to the time series of the core content in ARMS.
Sample and Questionnaire Design
Recommendation 4.1: The methodology research and development program the panel recommends should systematically (1) evaluate current instruments and practices, (2) collect data that inform both the revision of existing items as well as the creation of new items, (3) test revised instruments before they are put into production, (4) use experimental control groups to evaluate the differences between the old and new questionnaires, (5) improve understanding of respondent record-keeping practices and their effect on survey quality, and (6) designate a subsample of the existing ARMS sample for research and testing purposes. Key parts of this work would best be conducted in a cognitive or usability laboratory facility. It would be enabled by obtaining a generic clearance from the Office of Management and Budget for testing of all phases of the survey to allow for broader cognitive testing, evaluate the quality of data reported in response to each question, and evaluate the impact of mode of data collection across the three phases.
Recommendation 4.2: ERS and NASS should improve the consistency of variables across ARMS versions and over time.
Recommendation 4.3: NASS and ERS should explore the collection of auxiliary information on a formal basis, as well as feasibility of enriching the ARMS data files with information from administrative data sources, geospatial data, and the like.
Recommendation 5.1: ARMS should use automated means to collect paradata on interviewer assignments to cases, the relationship between the interviewer and the sample farm respondent (i.e., whether they know each other), demographic characteristics of the interviewer and the characteristics of the sample farms for nonrespondents that are coordinated with information obtained for respondents, either through the interview or interviewer observation. These paradata could be used to determine the need for additional research on the impact of the relationship between the interviewer and the respondent on the quality of answers. This data collection can best be facilitated using computer-assisted technologies.
Recommendation 5.2: NASS should systematically explore the consequences of interviewer departures from standardization in the interview. To facilitate this, NASS should collect paradata on the frequency with which interviewers follow the order of the questionnaire, read questions as worded, provide clarification, and similar indications of departures from standardized procedures.
Recommendation 5.3: NASS should use available analytic tools, for example, cognitive interviews, interviewer debriefing, recording and coding of interviews, and reinterviews, to investigate the quality of survey responses.
Recommendation 5.4: NASS should move to computer-assisted interview and possibly web-based data collection, after research and testing to determine possible effects of the collection mode on the data. Computer-assisted personal interviews and web-based data collection will provide opportunities to increase timeliness, improve data quality, reduce cost, and obtain important paradata.
Recommendation 5.5: NASS and ERS should develop a program to define metadata and paradata for ARMS so that both can be used to identify measurement errors, facilitate analysis of data, and provide a basis for
improvements to ARMS as part of the broader research and development program the panel recommends.
Nonresponse, Imputation, and Estimation
Recommendation 6.1: NASS should routinely report ARMS case dispositions linked across survey phases to provide the foundation for appropriate response rate calculations for Phases II and III.
Recommendation 6.2: All published ARMS response rates for Phase II and III should be calculated to reflect the nonresponse from the preceding phase(s).
Recommendation 6.3: The nature of the ARMS nonresponse bias should be a key focus of the research and development program the panel recommends. This research and development program should focus, initially, on understanding the characteristics of nonrespondents.
Recommendation 6.4: The research and development program should continue NASS’s work on both public relations and incentives, and it should do so with a focus on nonresponse bias, not simply nonresponse rate.
Recommendation 6.5: The research and development program should analyze whether there are differences in ARMS unit and item nonresponse rates between census and noncensus years, with an eye toward deciding whether making ARMS mandatory would improve data quality.
Recommendation 6.6: The research and development program should examine how questionnaire design and interviewing changes could reduce item nonresponse.
Recommendation 6.7: NASS and ERS should consider approaches for imputation of missing data that would be appropriate when analyzing the data using multivariate models. Methods for accounting for the variability due to using imputed values should be investigated. Such methods would depend on the imputation approach adopted.
Recommendation 6.8: All missing data that are imputed at any stage in the survey should be flagged as such on files to be used for analysis.
Recommendation 6.9: NASS and ERS should provide more clarification and transparency of the estimation process, specifically the effect of calibrations on the assignment of weights and the resulting estimates.
Methods of Analysis
Recommendation 7.1: NASS should continue to provide sampling weights with the ARMS data set, combined with replication weights for variance estimation.
Recommendation 7.2: NASS and ERS should continue to recommend the design-weighted approach as appropriate for many of the analyses for users of ARMS data and as the best approach for univariate or descriptive statistics.
Recommendation 7.3: NASS should investigate and implement improvements to the current jackknife replicates to make them more useful for the types of analyses performed by users in ERS and other organizations. In particular, NASS should increase the number of replicates and apply bounds to the magnitude of the weight adjustments.
Recommendation 7.4: NASS and ERS should investigate the feasibility of providing sufficient information on the design and nonresponse characteristics of ARMS, in order to perform design-based statistical analysis without using the replicate weights and to allow users to incorporate design and nonresponse information in model-based analyses.
Recommendation 7.5: ERS should build an enhanced level of in-house survey statistics expertise, in cooperation with NASS. The specialized expertise in both econometrics and survey statistics needed to accomplish this is currently not present in ERS and is likely to require a significant effort in recruiting and training.
Recommendation 7.6: ERS and NASS should collaborate on writing a Guide for Researchers for performing multivariable analyses using data from complex surveys, particularly data from ARMS. In areas in which expertise is not available for writing parts of such a guide, expertise should be sought from the statistics and economics community, especially those with experience in the analysis of survey data from complex survey designs.
Recommendation 8.1: ERS should continue to improve the ARMS web tool by providing summaries on more variables and more subsets from ARMS, and to improve the ARMS extranet web tool by adding the ability to link over years and to more sophisticated models.
Recommendation 8.2: USDA should consider extending the availability of ARMS microdata through the Census Bureau research data centers to increase access opportunities for using additional data sets and enabling researchers to match ARMS files with other data sets.
Recommendation 8.3: ERS should provide more training for new data users, including developing a data user manual, which also includes the recommended guide on statistical estimation, and offering training workshops.
Recommendation 8.4: Database management practices should include a system for managing and reporting errors found by users, for ensuring the consistent labeling of the codes for raw variables, and for using consistent names of the ERS-created summary variables over time.