Practice 9: Commitment to Quality and Professional Standards of Practice
THERE IS A WIDE RANGE of actions a statistical agency should undertake to ensure the quality of its products and practices:
- develop strong staff expertise in the subject areas relevant to the agency’s mission, in the theory and practice of statistics, and in data collection, processing, analysis, and dissemination techniques;
- keep abreast of and use modern statistical theory and sound statistical and computational practice in all technical work;
- publish and implement formal quality standards;
- maintain quality assurance programs to improve data quality and the processes of compiling, editing, and analyzing data;
- develop an understanding of the validity and accuracy of the agency’s data and how to convey the resulting measures of quality (both uncertainty and bias) in comprehensible ways to users;
- document concepts, definitions, and data collection methods and discuss possible sources of error in data releases to the public (see Practice 4); and
- develop continuing relationships with appropriate professional organizations in statistics and relevant subject-matter areas.
The best guarantee of high-quality data is a strong professional staff, which includes experts in the subject-matter fields covered by the agency’s program, experts in statistical methods and techniques, and experts in data collection, processing, and other operations (see Practice 11). A major function of an agency’s leadership is to strike a balance among these staff
and promote working relationships that make the agency’s program as productive as possible, with each group of experts contributing to the work of the others.
An effective statistical agency keeps up to date on developments in theory and practice that may be relevant to its program—for example, new techniques for imputing missing data (see, e.g., National Research Council, 2004a:App. F, 2010e) or for combining data from more than one source and estimating error in the resulting statistics (see National Academies of Sciences, Engineering, and Medicine, 2017b:Ch. 5); new technologies for data collection, processing, and dissemination; and new kinds of and uses for paradata (see, e.g., National Research Council, 2013a). Paradata, not only from data collection processes, but also from tracking how users work with an agency’s data products on its website, can help improve methods in each of these important areas.
An effective statistical agency is also alert to social and economic changes that may call for changes in the concepts or methods used in particular datasets.81 The need for change often conflicts with the need for comparability with past data series, and the latter need can easily dominate consideration of proposals for change. Agencies have the responsibility to manage this conflict by initiating more relevant series or revising existing series to improve quality while providing information to compare old and new series, such as was done when BLS revised the treatment of owner-occupied housing in the Consumer Price Index.82
An effective statistical agency devotes resources to developing, implementing, and inculcating standards for data quality and professional practice. Although a long-standing culture of data quality contributes to professional practice, an agency should also seek to develop and document standards through an explicit process. The existence of explicit standards and guidelines, regularly reviewed and updated, facilitates training of new in-house staff and contractors’ staffs. Statistical Policy Directive No. 2 (U.S. Office of Management and Budget, 2016a), on survey standards and guidelines, is helpful in that it covers every aspect of a survey from planning
__________________
81 Reviews of concepts underlying important statistical data series, which have identified areas in which change is needed, include: poverty (National Research Council, 1995, 2005c); cost-of-living and price indexes (National Research Council, 2002); “satellite” accounts for nonmarket activities—e.g., home production, volunteerism (National Research Council, 2005a); food insecurity and hunger (National Research Council, 2006a); usual residence in the decennial census and surveys (National Research Council, 2006c); disability (National Research Council, 2009b); health satellite accounts (National Research Council, 2010a); medical care economic risk and burden (National Research Council and Institute of Medicine, 2012); happiness, suffering, and other dimensions of experience (National Research Council, 2013b); civic engagement and social cohesion (National Research Council, 2014b); and innovation (National Research Council, 2014a:Ch. 4; National Academies of Sciences, Engineering, and Medicine, 2017a).
82 See, for example, Gillingham and Lane (1982).
through data release (see Appendix A).83 It recommends that agencies develop additional, more detailed standards that focus on their specific statistical activities.84
To ensure the quality of its data collection programs and data releases, an effective statistical agency has not only formal quality assurance programs (e.g., well-developed methods for detecting outliers and other errors in raw data and errors from editing and other data processing steps), but also mechanisms and processes for obtaining both inside and outside reviews (see Practice 12). Such reviews should address various aspects of an agency’s operations, including the soundness of the data collection and estimation methods and the completeness of the documentation of the methods used and the error properties of the data. For individual publications and reports, formal processes are needed that incorporate review by agency technical experts and, as appropriate, by technical experts in other agencies and outside the government.85
Finally, an effective statistical agency builds strong ties with relevant professional associations. It encourages professional staff to participate in relevant associations to refresh their human capital and develop networks of experts from other statistical agencies, academia, and the private sector. It also uses professional associations as one source of advice on ways to keep its data collection programs as relevant, accurate, timely, and cost-effective as possible.
__________________
83 Data quality guidelines of statistical agencies in other countries are also helpful (see, e.g., Statistics Canada, 2009; United Kingdom Office for National Statistics, 2007).
84 For examples, see National Center for Education Statistics (2012) and Bureau of Transportation Statistics (2005). These standards work within but go well beyond the broad data quality guidelines adopted by statistical agencies in response to the 2000 Information Quality Act (see Appendix A).
85 See Appendix A for a description of OMB guidelines for peer review of scientific information; reviews at a program or agency-wide level are considered in Practice 12.
