9
Possible Next Steps
The discussion in the final session focused on how the federal statistical system can move toward greater transparency to build trust on the part of its user community. The panelists for this session were John Abowd (U.S. Census Bureau), Hermann Habermann (CNSTAT), Tom Louis (Johns Hopkins University), and Bill Eddy (Carnegie Mellon University).
HERMANN HABERMANN: SUMMARY REMARKS
Hermann Habermann started by providing some historical context for the workshop. He noted that several years ago, the National Science Foundation (NSF) recognized that new and emerging technologies were changing the way that research was conducted and how it was communicated. So, NSF provided support to the National Academies of Sciences, Engineering, and Medicine to have a forum on open science. That forum existed for a few years, with high-level discussions about the issues that many of participants talked about during the workshop, including transparency, reproducibility, and how science can be conducted differently in this era when technologies are changing. At the same time, the Royal Society in the United Kingdom produced a report on science as an open enterprise. One of the recommendations from that report was that scientists should communicate the data they collect and the models they create to allow free and open access in ways that are intelligible, accessible, and usable to other specialists in the same or linked fields. Universities and research institutes need to play a major role in supporting an open data culture by recognizing data communication by their researchers as an important criterion for career progression.
Because the forum discussions were at a very high level, the organizers of the forum, CNSTAT, and NSF agreed that a workshop should be convened to try to bring those ideas down to a more manageable level and focus on federal statistics.
Habermann noted that further work is needed on the curation and operationalization of standards of transparency, and on elaboration of the costs and benefits. He said that Canada and the United Kingdom have taken the lead and now the United States might examine what is happening and decide how to go forward.
TOM LOUIS: SUMMARY REMARKS
Tom Louis began his remarks by stating that transparency is in the eye of the beholder. In other words, it depends on the stakeholder and consumer, as well as the level of communication to the recipient. For example, it would be opaque for many audiences if Wesley Basel (U.S. Census Bureau statistician) simply provided the formulas for Markov Chain Monte Carlo models, but for some people it might be exactly right. The discussions at the workshop have been restricted to talking about code, microdata, metadata, and paradata. Statisticians also would have, at least in areas where things are in development or are new, not just computer code but the mathematics sitting right next to it. When he is doing research, Louis insists on first writing out assumptions and the model in standard statistical language, then translating it into computer code. This approach is computer code independent and avoids confusion. For example, some computer languages take variance as input, while others require precision. Also, good programmers optimize with matrix processing, but that can make the program less transparent, even with good annotation. For Louis, the opaqueness is reduced by having the statistical model in standard format, and then one has to trust that it is being translated correctly. Louis reminded everyone not to forget having documentation and transparency of the basic approach that is not just embodied in the computer code.
Louis also echoed a comment from H.V. Jagadish that one really cannot afford not to take these steps. It is challenging, but other fields, such as genomics, for example, had such documentation and transparency measures from the beginning. Louis went on to describe an example from Duke University where things were not documented or made transparent. He believes that the field of statistics can learn from genomics and other fields where the luxury was starting from zero. Louis said that statistical agencies are at the front end of a new kind of data collection enterprise and that they have no excuse not to include such things from the beginning.
Louis also provided another example to illustrate how being transparent can help an institution facing criticism. He noted that in the
1990s through the 2010s, the Health Effects Institute (HEI) funded Johns Hopkins University to carry out what was called the National Mortality and Morbidity Air Pollution Studies, which entailed collecting publicly available data on hospital admissions and deaths, air pollution, demographics, etc. (Louis served on the HEI-constituted advisory board for the project.) The reports from the study showed the effects of air pollution exposure on deaths and morbidity, but industry tried to discredit the analyses. In response, the staff at Johns Hopkins posted all of their data and analysis programs on the Web, but they also required critics to post their methods online.
Louis believes that federal statistical agencies may face similar situations where the data or some synthetic version of the data could be made publicly available, which would allow users who do not like the agency’s numbers to review the analysis provided by the agency. They would then need to show how they arrived at the alternatives.
Louis added that agencies have to have people who are not only comfortable with but also seek peer review at all levels. Without that, when forced to open up code, a lot of people will get nervous that mistakes will be found. This is not identical to submitting to the peer-reviewed literature, but it is the same cultural issue. Success may require professional rewards, training, and experience.
In conclusion, Louis noted that obviously resources are needed, including dollars and staffing, and that it is essential to have the right people, including the right computer scientists. Habermann observed that more conferences or workshops showing how to include transparency could be useful, and learning what people are doing in other domains (e.g., genomics or geology) would be beneficial.
JOHN ABOWD: SUMMARY REMARKS
John Abowd invited Sally Thompson (BEA), Jennifer Madans (NCHS), and John Eltinge (U.S. Census Bureau) to make comments from the perspectives of their agencies. Thompson said that she has worked at two statistical agencies, the Economic Research Service (ERS) and BEA, and that they have very different cultures. ERS prides itself on peer review for everything they do. BEA is less focused on peer review, but it does have advisory committees and external people working with it. Everyone likes the idea of increased transparency for science, for knowledge creation, and for all of those other ideals, but even at BEA they do not open up completely to show how every one of their estimates came from the source data and became published statistics. Thompson said that she loves the approach that Jagadish presented the day before: providing enough detail so that people are comfortable. She believes that BEA provides that level of detail
in their methodology. She noted that in this academic setting, transparency is readily embraced and perhaps it would be politically incorrect to speak otherwise.
Eltinge began by noting that much of what participants are advocating for or considering here amounts to a public good or collective action activity. Statisticians know that transparency will be good for the field overall, but understanding more about practical ways in which to incentivize people, and whether management structures are aligned to move this forward as opposed to worrying about perfectionism, are key. He believes that it is important to pay attention to realistic ways in which agencies can allocate a modest amount of resources and get some kind of practical benefit that is truly observable by others. Referring to Louis’s Johns Hopkins case, instead of having infinite levels of cost attached to responding to each individual inquiry, they were able to effectively modify the costs by putting all of the code and the data available online. That is a very prudent way to manage the taxpayers’ money and improve both quality and cost and risk profiles.
Eltinge noted that an example of this in the past decade is the security requirements, which have had a huge impact on the federal system as a whole, not just the statistical system. Experts may or may not like those, or they may not think that the requirements are aimed in the right direction, but clearly there were very strong perceptions about a compelling public need to move in a certain direction. There is a risk management issue here, and although agencies may not agree with what was done, they are stating what they are doing and why.
Madans said that from NCHS’s point of view, looking across the whole system and hearing what was said at the workshop, her agency probably does a pretty good job at maximizing transparency. However, it is not perfect and can do better. NCHS has not been involved in some of these new modeling efforts, and the issues for these models are the most complicated. She still sees a lack of consensus on the objectives that agencies are trying to reach. She asked the group a few questions, including: Should we only provide information when there are questions or all of the time? Should we focus on the “big stuff” and then wait on the “little stuff” until someone complains? Making it all available at the beginning has a lot of advantages but is a big investment, she said. They could do an internal cost-benefit analysis and say it is not worth it. No one is going to ask about it, and since it is going to cost the agency something then it does not put it out. If someone does ask, NCHS can at that point release more information. She asked if that is how the U.S. statistical system is going to work—separate ways—or if there is going to be a general rule. She noted that there seems to have been interest within some of the agencies, but not across all of the agencies.
Madans suggested that everyone look at the original OMB standards document sections on documentation and access to determine if new standards and best practices are needed in this area, and if so whether agencies want to develop them jointly. Alternatively, they could have some common metrics so that there are not 14 different systems. She believes that it would be very useful to have that conversation; otherwise, agencies will each do something that may not be the best thing given their individual limited resources. Eddy noted that different groups can have completely different perspectives on these topics, so any effort on this should be multidisciplinary.
Victoria Stodden followed up on something Louis said, which is the distinction between how far a user can go with the use of antiquated research practices, like the use of Excel, or other methods that do not lead to reproducibility before he or she starts heading into territory that starts to look like low integrity. She said that this was debated in the National Academies’ Committee on Fostering Research Integrity, which referred to these as detrimental research practices. Louis said that he would like to clarify that this was extreme malpractice. Another important reason to have embodied procedures and reproducible code is that agency employees know then what the “it” is. He said that we need to know the “it” to understand the output and to study its properties. On the other hand, there can be a risk to locking things down too early. Louis believes that agencies have to have a system that encourages continual prodding of the system. Levenstein said that can be addressed by having transparency about what the automation does, because then people can get in there.
Henry said that this workshop was held not just to discuss transparency and reproducibility, but to work better. Agencies should think of who the users are. If statisticians over-engineer things, then it is harder to explain them, making it harder to be transparent about them and reproduce them. She added that she cannot imagine what it must be like having 13 different organizations as colleagues, and thought that should they come together more, it will be a lot easier to answer some of the most complex social issues because silos do not help with communication.
Connie Citro added that since contractors do a lot of data collection, if the agencies say that they should be using DDI or some other program, that needs to flow through to the arrangements made with their contractors. Adding the complication that there are now multiple data sources such as administrative data, surveys, and Internet sources, what is really happening when combining data sources? She argued that it is imperative that the various standards that are already out there and have had a lot of use be adopted, so that users can trace the process and describe whether they are using administrative records. She said that asking respondents for their income still means that they are used in the imputations, etc. She believes that it is necessary to have the internal transparency that some of these
tools and systems will make possible to migrate successfully into the world of multiple data sources. Hopefully the internal documentation will inform the external documentation, as having the internal documentation amounts to an industry standard. She recalled the discussion about craft production versus more automated state-of-the-art production, and suggested that agencies have to move in this direction. There will be some areas that need craft production, for instance, business surveys with very large companies that skew the data, and a statistician has to be sure that they are looking at that. But many other companies could be automating and saving time. She said that the U.S. Census Bureau’s economic programs spend a large amount of staff time editing, and it is not really doing anything for them. For the big companies, if they have account managers, which Statistics Canada does, they could be working not just to edit their data, but to improve how they are getting it in the first place. Meanwhile, the vast bulk of the data is being handled in a much more automated way where one may be trading off a bit of quality but probably not trading off very much, and an agency gains the ability to reprogram its resources.
Citro added that she does hope that the Interagency Council on Statistical Policy makes this a priority and builds in documentation, process analysis, and process flow rather than waiting until after the fact.
Habermann noted that the discussions have covered transparency a lot, but only about data transparency and method transparency, not about the users. For example, the agencies say to users that they may correct answers for improvement, but they do not say that they are going to do this by taking information from tax forms. How transparent are agencies when they do that? What are the implications of becoming more transparent? Also, agencies are now increasingly collecting genomic data. How transparent can they be, do they want to be, and should they be to the person who provides that data about what may happen to him or her regarding the collection of those data now and 20 years in the future?
Audris Mockus said that Abowd is working on a business case for transparency, and wanted to note a couple of relevant items First, when an agency explicitly has version control systems, it is an extremely effective way to transfer knowledge, and so there is no risk of losing it when somebody leaves quickly. In addition, when something is made public, an agency will get contributions from external contributors that may improve the quality and fix bugs that the agency would otherwise not have enough resources to do, which is another huge business value.
David Barraclough said that he has heard several times that it would be good to become more involved in the working groups that were mentioned. He said that for the SDMX working groups, there are many countries that
are involved; however, he is not aware that there is any U.S. involvement, so his working group would certainly like to have U.S. representation. He also said that he would be interested in knowing what the next steps are after this workshop.
This page intentionally left blank.
