2
Existing Guidelines Related to Transparency
PETER MILLER: AAPOR TRANSPARENCY INITIATIVE
The goal of the first session was to consider existing guidelines or standards regarding transparency in the U.S. federal statistical system, specifically those on surveys from the American Association for Public Opinion Research (AAPOR) and the standards for the federal statistical system issued by the Office of Management and Budget (OMB). Peter Miller (U.S. Census Bureau), who was AAPOR president from 2009 to 2010, began with some background on AAPOR and a summary of its Transparency Initiative.
In 1947, AAPOR was founded as an organization of individual survey research professionals with the goal of advancing survey research. From its inception, AAPOR included government, academic, and commercial members and was intended to serve not only public opinion researchers but all survey professionals. Miller explained that standards of methodological disclosure were discussed for many years before being formally adopted in 1967. Subsequently, procedures for enforcement of the disclosure standards were adopted in 1975. Disclosure was the focus of the standards and the enforcement, rather than procedures or performance. Following the adoption of the standards, the association set up a mechanism to receive complaints about failure by survey organizations to disclose methods and initiated a set of procedures to investigate such complaints. A standards case could result in the association publicly censuring an organization for failing to disclose useful methodological details of a study that it had released. It is important
to mention, Miller said, that the standards and procedures are “living documents,” having been revised on multiple occasions since they were adopted.
In its requirements for disclosure, Miller explained, AAPOR distinguishes between items that can be reported immediately when a survey is published and those that can be reported within 30 days. The items that have to be reported immediately include who sponsored, conducted, and funded the research; the exact wording of survey questions; the specification of the population of study; the geographic location of interest; a description of the sample frame; the sample design; the sample size and error; weighting and cluster adjustments; and method(s) and dates of data collection. The types of items that have to be reported within 30 days are the instructions for interviewers and respondents; relevant stimuli (such as show cards); the coverage of the sampling frame; methods of panel recruitment (if the sample is drawn from a pre-recruited panel); sample design details (e.g., eligibility rules, screening procedures, and oversampling); use of incentives; weighting details; information for calculating response rates; and information on efforts to validate the data.
In the years since the standards were adopted, AAPOR has been involved in a number of standards cases, Miller said, which put a burden on the organization because of the time needed to adjudicate them and because they can become legally contentious. He discussed two cases that clarified for him the importance of disclosure and the difficulties faced by those who enforce it. First, in the late 1990s, Arianna Huffington, a journalist, asked the major public opinion polling organizations to report on their response rates. One major national organization complained that doing so would be difficult. Miller said he was surprised to learn that it—and other organizations—did not have a routine system for documenting such information. Educating organizations about embedding transparency in their normal workflows can be useful. Some years later, AAPOR had to process a number of standards cases connected to 2008 pre-election polling, one of which led to censuring an organization for refusal to disclose methodological information. (The organization was subsequently cited by Nate Silver from FiveThirtyEight for falsifying data.)1 The other standards cases were resolved once organizations were able to pull information together from records. The experience reinforced belief in the need for education on routinized transparency and on the importance of public censure for organizations that refused to adhere to this key norm.
When he became AAPOR president, Miller proposed that, in addition to maintaining sanctions for failure to disclose, AAPOR should have a program that incentivized systemic, routine disclosure practices. Thus,
___________________
1 See Dewan, S. (2009). Polling firm’s reprimand rattles news organizations. The New York Times, October 2.
the Transparency Initiative was launched in 2010. After long discussions and pretests, organizations wishing to join the initiative were vetted and enrolled beginning in 2014. The initiative included training modules on disclosure practices, along with periodic auditing and feedback. AAPOR’s public recognition of the member organizations includes a seal of approval and media releases. The original concept envisioned an archive of methodological disclosures, but due to costs, it remains a future possibility.
Currently, Miller said, the AAPOR Transparency Initiative has nearly 90 member organizations, large and small, from all sectors of survey research. AAPOR assumes all of the program expenses, and there are no membership fees to join. Investment in their own documentation processes could be useful to organizations. AAPOR conducts audits every 2 years to assess compliance, and recent audits have described the need for enhanced and sustained continuing education for survey organizations since their staff members turn over, Miller said. He noted that, over time, membership may change as audits continue. Organizations may drop out, or AAPOR may decide that they are not fulfilling the initiative’s requirement and cease to recognize them.2
Federal statistical agencies are not members of the initiative, but some large contractors of the agencies are. Initially, some contractors questioned whether the requirements of the Transparency Initiative would either conflict with or duplicate federal standards of disclosure. To allay such concerns, in 2015 the Federal Committee on Statistical Methodology (FCSM) sent a letter to AAPOR detailing the federal commitment to transparency in a variety of standing policies,3 noting that AAPOR’s Transparency Initiative requirements align with the disclosure requirements of statistical agencies.
John Eltinge (U.S. Census Bureau) added that FCSM is currently exploring questions of enhancing transparency for reporting on the integration of multiple data sources, and this workshop is salient to that discussion.
Miller stressed that the level of disclosure in the Transparency Initiative is incomplete in the sense of achieving reproducibility or replication. He said that the AAPOR standards create the opening for a discussion and further public exploration of methodology, which might need to become much more comprehensive to fully achieve that goal.
In summary, Miller presented three points for discussion concerning transparency and reproducibility: (1) AAPOR’s experience with the Transparency Initiative to date suggests that public recognition and education can incentivize documentation and disclosure of methodological elements; (2) the level of disclosure required in AAPOR’s Transparency Initiative is one step toward what may be required for replication or reproducibility;
___________________
2 See https://www.aapor.org/Transparency_Initiative.htm [January 2018].
3 See http://www.aapor.org/AAPOR_Main/media/AM15/Federal-Transparency-Guidances_31DEC15.pdf [January 2018].
and (3) the process of implementing an effort like the Transparency Initiative may inform the process of establishing norms and methods for replication or reproducibility.
BRIAN HARRIS-KOJETIN: RELEVANT OMB DIRECTIVES
Turning to OMB, Brian Harris-Kojetin (Committee on National Statistics [CNSTAT]), former senior statistician in OMB’s Statistical and Science Policy Office, provided details for the agency’s standards, regulations, and guidance related to transparency. He began by noting two relevant documents: (1) the Paperwork Reduction Act (PRA), which requires public comment periods and documentation for any new survey data collection; and (2) OMB policy directive No. 2, Standards and Guidelines for Statistical Surveys, which directs what is essential for survey documentation. Harris-Kojetin explained that the purpose of the PRA was to improve the quality and utility of information that is useful to the federal government and to reduce the resulting paperwork burden on the public. Given this authority, OMB reviews agency requests to collect information to ensure that activities have practical utility, reduce duplication, meet a specific agency need, and minimize burden. He detailed the five steps of the PRA process: (1) an agency’s publication in the Federal Register of a notice of the intent to submit a survey proposal to OMB, which has a 60-day comment period; (2) revisions by the agency to address public comments from that notice; (3) the agency’s publication of a notice in the Federal Register that it is submitting to OMB a request for approval, which has a 30-day comment period; (4) OMB review and feedback to the agency; and (5) OMB approval.
To demonstrate compliance with the PRA, Harris-Kojetin explained, an agency has to provide the following information to OMB:4 (1) type of submission, (2) title, (3) purpose, (4) number of respondents, (5) estimate of burden, (6) frequency of collection, and (7) use of statistical methods. He also provided details about the necessary two-part supporting statement: part A covers justification for information collection, and part B covers statistical methods.
In part A, an agency is asked to provide the necessity of information collection, the use to which the information will be applied, the use of information technology, efforts to identify duplication, minimizing burden on small entities, consequences if the information is not collected, a record of consultation with the public and public comments, assurances of
___________________
4 This submission is referred to as the RISC and OIRA Consolidated Information System core data. Available: https://georgewbush-whitehouse.archives.gov/omb/gils/oira-gils.html [January 2018].
confidentiality, questions of a sensitive nature, time and monetary burden estimates, total annual cost burden, annual cost to the federal government, explanation of any program changes, and plans for publication and tabulation. In part B, which covers statistical methods, an agency is asked to provide the sampling universe and respondent selection, procedures for collecting information, methods to maximize response, tests of procedures, and contacts for statistical aspects and data collection.5
In addition to documents related to the PRA, in 2006 OMB issued Standards and Guidelines for Statistical Surveys,6 which includes 20 standards, Harris-Kojetin said. The standards are supplemented with guidelines or best practices to help the statistical agencies interpret and fulfill the goals of the standards. The standards call for details on seven elements: (1) the development of concepts, methods, and design, (2) the method of data collection, (3) the method for processing and editing data, (4) the production of estimates and projections, (5) data analysis, (6) procedures for review, and (7) dissemination of information products.
Turning to the issue of survey documentation, Harris-Kojetin pointed out that the standards call for an agency to provide documentation that includes information both to understand how to analyze every survey’s data and to replicate and evaluate each survey’s results. That documentation must be readily accessible to users, he said, unless it is necessary for access to be restricted to protect confidentiality. There are associated guidelines to this requirement, he noted, which are detailed in Guideline 7.3.1.7 This information must be made available within 30 days.
OMB’s survey documentation standards also contain three other detailed guidelines, Harris-Kojetin said. Guideline 7.3.2 concerns survey replication and evaluation; Guideline 7.3.3 requires a periodic evaluation report, such as a methodology report, that itemizes all sources of identified error; and Guideline 7.3.4 calls for the retention of all survey documentation consistent with appropriate federal records disposition and archival policy.8
Harris-Kojetin pointed out that OMB’s standards and guidelines are clearly survey-centric, but not exclusively so. For example, he noted, Guideline 4.1.6 concerns the development of estimates and projections, requiring that agencies document the methods and documents used; ensure objectivity, utility, transparency, and reproducibility of the estimates and projections; and archive the data and models.9
___________________
5 For details, see https://www.reginfo.gov/public [January 2018].
6 See https://obamawhitehouse.archives.gov/omb/inforeg_statpolicy [January 2018].
7 U.S. Office of Management and Budget. (2006). Standards and Guidelines for Statistical Surveys, pp. 26–28. Available: https://www.ftc.gov/system/files/attachments/data-quality-act/standards_and_guidelines_for_statistical_surveys_-_omb_-_sept_2006.pdf [January 2018].
8 See reference in fn. 5.
9 See reference in fn. 5
In addition, Harris-Kojetin noted that OMB has recently provided additional guidance regarding administrative records. That guidance states that, to the extent possible, the previously mentioned standards also cover the compilation of statistics based on information collected from administrative records. In addition, he noted, OMB memo M-14-06 provides guidance on providing and using administrative data for statistical purposes.10 However, Harris-Kojetin said that there has not been a systematic effort to develop the same kind of guidelines and standards and regulations for the use of administrative records and for the products of complicated statistical models as there has been for surveys and the usual survey estimates.
A participant asked Harris-Kojetin whether, when OMB established these standards, there was a problem when providing the documentation for electronic survey instruments, given the complicated skip patterns and other characteristics. Harris-Kojetin responded that OMB wrestled with that issue, since there is no paper form for computer-assisted telephone interviewing (CATI) and computer-assisted personal interviewing (CAPI) instruments. He said one way to document such an instrument is to provide the instrument itself, along with the software to run it, but this raises the issue of what to do about the documentation for edits and imputations. He said that often there is no codebook or an annotated code; in some cases, only a few people really know what is done and its justification, and other people just keep it going.
Jennifer Madans (National Center for Health Statistics [NCHS]) agreed that this is often the case. The documentation is often incomplete, and one has to have someone explain what is done and why it is done that way. What is documented is a black box that varies greatly by survey and agency, she said. She mentioned another standards group that she served on concerning cognitive aspects of survey interviewing. The majority of the standards were about documentation, which is often missing, and how and what to document. However, there still is no consensus about how comprehensive the documentation should be and where it should be posted.
David Barraclough (OECD) noted that it is a national initiative and asked if there is any kind of effort to promote international alignment on the use of standards and recommendations with other statistical offices. Also, he asked, how are the OMB guidelines being implemented? Since publication in 2006, are there efforts to continuously renew them?
In response, Miller said that AAPOR is really an international organization with many foreign members. Therefore, AAPOR does have reach outside the United States. He noted that it has had discussions with the European Society for Opinion and Marketing Research, although AAPOR’s
___________________
10 See https://obamawhitehouse.archives.gov/sites/default/files/omb/memoranda/2014/m-14-06.pdf [January 2018].
work is more related to procedures and performance work. He said there is no concrete linkage between AAPOR and the guidelines of other organizations, though there have been attempts at harmonization.
In response to the question about the OMB standards, Harris-Kojetin said that the PRA requires that ongoing data collections be submitted for renewal every 3 years, so there is definitely follow-up on certain aspects of the data collection. However, he noted, in his tenure at OMB he never asked for all 18 items of survey documentation, since a fair number of things are subsumed in agencies’ general documentation. He explained that OMB’s focus is on sources of error, especially the results of nonresponse bias analyses, and the evaluations that have been carried out concerning the quality of the estimates. In terms of updating over time, Harris-Kojetin noted that when the OMB standards were promulgated in 2006, there had not been any revisions for 30 years. He said that there is no current plan to systematically update them; however, OMB is aware of some areas that could be looked at in the near future. One such area is cognitive interviewing.
A participant pointed out that many surveys are one-off surveys, and so the idea of improving the next repetition is not relevant.
Another participant asked Harris-Kojetin whether the PRA is the only federal regulation or if there were other contexts outside of OMB for which there are regulations on similar topics. Harris-Kojetin responded that OMB may not be the exclusive place for such regulations.
Miron Straf (Virginia Tech) asked about a situation in which surveys need to make quick changes, especially pilot surveys, and the PRA has been criticized for making things difficult under such circumstances. Harris-Kojetin responded that he did not know of a single federal agency that would want to quickly change its CAPI or CATI instrument 1 week before it was going to the field or while in the field. There are, however, provisions for agencies to make small changes without having to go through the 60-day public comment period. There are also generic clearances for doing cognitive testing and pretesting, he said, as well as what are called nonsubstantive changes for small modifications and provisions for emergency procedures.
The next question concerned OMB Memo M-14-06. Harris-Kojetin said that when dealing with administrative records, each agency is doing what it believes is in compliance since guidance has not stabilized. He was then asked whether there are some best practices that he has seen regarding documentation. He answered that he has not seen much documentation of administrative records. He asked whether Madans or Eltinge had seen this at NCHS and the U.S. Census Bureau, respectively.
Madans responded that several FCSM groups are looking into administrative data and how to use them, document their use, assess their quality, and to consider what paradata and metadata mean in the context of
administrative records. There was a big push to carry out quality evaluation of administrative data, but there is still no agreement about how to do this. At the U.S. Department of Health and Human Services, Madans said, staff have been using administrative data for a long time, for example, from the Medicare program. In the early days, the collection of Medicare data was not well documented, but things are much better now. However, Madans said, there has not been sufficient thought concerning best practices about the use of these records. As a result, there are some examples where problems occurred, but there have also been some successes. The big question, she said, is what the value is of an administrative records database for a specific analytic problem.
Eltinge described the administrative records quality checklist developed by the Federal Committee on Statistical Methodology’s (FCSM’s) Administrative Records Subcommittee, which is available on the Web.11 At the same time, FCSM is looking at the integration of multiple data sources.
Referring back to an earlier question about making changes in the field, John Abowd (U.S. Census Bureau) said that the question was likely referring to adaptive design, which means that the procedure in the field changes in ways that are not transparent. He said that there might be standards for adaptive design, and he hopes that the U.S. Census Bureau can start to encode some of these in its procedures. Turning to a broader issue, he said that although standards have required agencies to store the historical metadata associated with a survey, the standards that are used to guide how to do this could be reexamined. In particular, he said, such information could be linked to permanent digital object identifiers (DOIs). Otherwise, it becomes difficult to find such historical documents. The U.S. Census Bureau has become a member of the Inter-university Consortium for Political and Social Research in order to obtain their professional DOIs and services for metadata curation. This is in reaction to the problem exemplified by the current difficulty in finding the 1983 Survey of Income and Program Participation metadata, and the fact that various American Community Survey datasets exist.
The last comment in the session was from Connie Citro (CNSTAT), noting that documentation and having good metadata allows users to evaluate the quality of the data and know what they are dealing with. Therefore, updating, archiving, and curation standards are extremely important. Currently, she said, no statistical agency regularly makes this information available to users in a single location, so the issue is not simply what an agency does internally or within the statistical system but also getting that information out to external users.
___________________
11 See https://nces.ed.gov/FCSM/index.asp [January 2018].
