Facilitating Innovation in the Federal Statistical System: Summary of a Workshop (2011)

A healthy and vigorous program for innovation is fundamental for the continued success of any large-scale organization, including the statistical agencies of the U.S. government.¹ The concept of innovation for statistical agencies is broad. It includes traditional subjects of innovation, such as improvements in survey design and data collection procedures, including editing and imputation for missing and incorrect data in surveys and administrative records. And it also includes those less usually considered, such as questions about the usefulness of federal statistics to policy officials and whether new approaches to bridge the interface between users and statisticians are required.

Policy makers often have a different time horizon for needing information (a few days, a few months, perhaps as long as a year) from that of ongoing statistical series, many of which provide information on a long time frame. For policy makers, there may be a tradeoff between the timeliness of information and its usefulness. If they can obtain the approximate answer to their question in a timely manner, they may find it sufficient for their needs, rather than a more accurate answer months or even years later.

Designing and fielding a survey and producing results often take years. Less traditional types of information-gathering modalities can reduce this time frame dramatically, although without the generaliz-

ability and properties of traditional survey methods. For example, the National Academy of Public Administration (NAPA) has partnered with government agencies to conduct more than a dozen online dialogues—web-based discussion forums in which stakeholders and the public can log on, discuss ideas for addressing one or several related issues, and express their perspectives and priorities for government action.² One of those dialogues was conducted on health information technology and privacy. During the week-long online discussion, the dialogue received more than 4,000 visits from across the country, generating hundreds of ideas and comments. It provided what policy officials deemed was sufficient information in a short amount of time.

These types of information collections give rise to several questions. Should federal statistical agencies play a role in these types of information gathering, and, if so, how? If the federal statistical system does not become an active participant in such approaches, is it in danger of becoming irrelevant? From the opposite perspective, is there a danger in straying too far in the direction of approximate answers and away from the traditional rigor of statistically valid information collections? Given these questions and the evolving ways of gathering information, what kinds and extent of innovation are needed for the federal statistical system to be able to play an appropriate role in meeting the needs of the public and policy makers for high-quality, timely, and relevant statistics to address new and changing social issues and questions?

On May 8, 2009, the Committee on National Statistics of the National Research Council and the American Academy of Political and Social Science jointly sponsored a symposium called “The Federal Statistical System: Recognizing Its Contributions, Moving It Forward,” in Washington, DC. One of the topics considered at that symposium was the health of innovation in the federal statistical system.³ A consequence of the symposium was an agreement by the Committee on National Statistics to hold a workshop on the future of innovation in the federal statistical system. That workshop was held on June 29, 2010.

The original statement of task for the workshop focused on three challenges to the statistical system: (1) the obstacles to innovative, focused research and development initiatives that could make statistical programs more cost-effective; (2) a gap between emerging data visualization and

communications technologies and the ability of statistical agencies to understand and capitalize on these developments for their data dissemination programs; and (3) the maturation of the information technology (IT) discipline and the difficulties confronting individual agencies in keeping current with best practice in IT regarding data collection, processing, estimation, and dissemination, all the while protecting data confidentiality.

It was envisioned that the workshop would include invited presentations and discussions to consider these challenges and the potential to address them. However, the steering committee decided that it would not be possible to consider all three topics in a one-day workshop. It was also decided that the time at the workshop would be devoted solely to discussions, without any presentations, although there would be background papers.

Thus, the workshop proceeded under the following task statement:

A major purpose of the workshop would be to generate a wide spectrum of views on the state of innovation in the federal statistical system and possible ways to facilitate it.

The workshop agenda appears in Appendix A. The workshop attendees, who included representatives from the federal statistical system and the academic and private sectors, are listed in Appendix B. Two papers were prepared specifically for the workshop: “Challenges to the Federal Statistical System to Continue to Provide Data Relevant to Policy Issues” by Robert P. Parker (2010) and “Barriers to Innovation and Possible Remedies” by Hermann Habermann (2010a). In addition, a previously published paper by Don Dillman (1996), “Why Innovation Is Difficult in Government Surveys,” was provided as background material. Thomas Louis (Johns Hopkins School of Public Health), chair of the workshop steering committee, suggested including the Dillman paper since its main points are as relevant in 2010 as they were in 1996.

This report is a descriptive summary of what transpired at the workshop. It is therefore limited to the views and opinions of the workshop participants. However, it does not strictly follow the agenda of the workshop, which had four sessions. Instead, it is organized around the themes of the discussions, which migrated across the four sessions. For example,

comments on the barriers to innovation were made in Session I, and comments on barriers and remedies were made in Session II. Moreover, there was no clear distinction between the need for innovation and the scope of innovation.

The introductory remarks that opened the workshop and a list of workshop highlights are summarized below. Chapters 2-5 cover, respectively, the scope and importance of innovation in federal statistics, barriers to innovation in federal statistics, possible remedies, and next steps.

Introductory comments were made by Thomas Louis, Constance Citro, and Katherine Wallman. In his introductory comments, Louis noted the impressive history of innovation in the federal statistical system, yet he stressed that it is time to consider how to move forward. He referred to the background paper by Habermann, which provides some examples of outstanding innovation and research accomplishments of the federal statistical system.

As discussed in that paper, the terms “research” and “innovation” are both used in this summary. They are related to one another, but complementary. “Research” is used, as in any field, in reference to systematic inquiry to discover facts or frame theories, and “innovation” is oriented to applications—that is, to design, invention, or development that yields products or services creating new value. For innovation to occur, the fruits of research must be applied to existing processes. Research may not always be necessary for innovation to occur, and even when new research results are produced, they may not be sufficient for innovation. Furthermore, the necessary research need not come from the federal government. The federal statistical system, however, must have the vision and the commitment for use of research to drive innovation. In discussing innovation, then, one is of necessity considering a complex process, usually involving research, whose end result is a change in existing processes of a statistical agency.

Louis said that among the most prominent accomplishments of the federal statistical system is the work of Morris Hansen and his colleagues at the Census Bureau. Although such seminal work occurs only infrequently, the innovation environment in the federal statistical system is still rich. Some examples include work on the seasonal adjustment of time-series data (such as unemployment rates); models for small-area estimates of poverty; fully outfitted, mobile medical testing facilities for the National Health and Nutrition Examination Survey; and the development of a generalized and integrated data warehouse by the U.S. Department of Agriculture to provide easy access to historical survey and census data

from farmers and ranchers. These innovations have often been accomplished by, or in collaboration with, academic or contractor institutions.

The federal statistical system continues to be aware of the need to foster an environment that supports and contributes to innovation, Louis said. For example, the Interagency Committee on Statistical Policy, chaired by the U.S. Office of Management and Budget (OMB), has created a subcommittee on innovation, which has worked in such areas as sampling methods and dissemination procedures and engaged in discussions with the academic community on the need for statisticians in the federal government.

However, although the environment for innovation is rich and the research accomplishments of the federal system are formidable, Louis echoed the point in Habermann’s paper that the demands of the nation require ever more innovation. New technologies have opened doors that did not exist even 10 years ago. Users are examining the cost and time for traditional survey approaches and asking if quicker, cheaper approaches with less accuracy are acceptable.

Robert Groves (U.S. Census Bureau) noted, for example, that the federal statistical system would not be able to continue its current business model of surveys with its current methods much longer because costs are escalating beyond the tolerance of the taxpayer to meet these costs. Constance Citro (Committee on National Statistics) pointed out that, although it is rare indeed for businesses to reinvent themselves, and even rarer for government agencies to do so, the data needs and the challenges to provide for such needs are growing. Consequently, the federal statistical system needs to assess how it is going to meet the demands and challenges of the future: indeed, that issue is the purpose of the workshop.

Louis also introduced the theme of competition between the operational demands of an agency’s day-to-day activities and the need to refocus these activities. He noted that innovation for tomorrow and beyond is often bumped by operational pressures and that it is important to keep in sight the demands of the future—no matter how pressing today’s responsibilities. In order to be able to meet the ever-changing and increasing demands discussed above, Louis said that it is critical to have the right culture, to have the right people in place, and to have a reward system that encourages risk and innovation. With respect to the inherent risk of trying to innovate, Citro said that the federal statistical system should embrace the process of innovation even though some ideas will be failures.

With respect to having the right people and reward system in place, Katherine Wallman (U.S. Office of Management and Budget) asked what the system would have to do in order to recruit the people who will be needed. She also introduced the themes of confidentiality, burden, and

OMB’s role in the redesign of household surveys. Although preserving the confidentiality of individually identifiable data is of paramount importance, innovation is needed to provide more data to the public, including data on small areas. The survey system depends on the willingness of potential respondents to participate. Work is needed both to encourage their participation and to reduce the burden on them. The traditional “stovepipe” approach to surveys (with agencies independently designing surveys without considering the data requirements of other agencies), as Groves also mentioned, may not be adequate for the future. Wallman noted that OMB is engaged in initiatives to understand how surveys can best be integrated and their costs reduced.

At least three major forces are currently affecting the federal statistical system. The first is the ever more difficult environment that data collection organizations face. For example, because of increasing resistance to survey participation, the Census Bureau has said it would be a significant accomplishment if the mail return rate for the 2010 census equaled that of 2000. The second is the need to respond ever more quickly to the needs of the business community, the public, and the political community. The third is the gap between emerging data visualization and communications technologies and the ability of the statistical agencies to incorporate and capitalize on these developments. Exacerbating these factors is the absence of a central focal point or agency with statistical research as its mission. Research now is scattered among the statistical agencies—and few have the critical mass of research statisticians that may be needed to deal with these factors.

As intended, the workshop was a forum for the free exchange of ideas—primarily on barriers to innovation, possible remedies, and next steps. There were no attempts to arrive at a consensus, nor were any conclusions drawn. To provide some structure, several of the participants presented their views of some recurring themes, and these are presented here.

• Some of the most creative people in an organization are often considered misfits. In fact, the ideas developed by these “creative misfits” are often the most innovative.

• The federal statistical system is a mature system, the field of sta­tistics has become very specialized, and future innovation could require major initiatives. For example, administrative records show great promise and are widely believed to be critical for future

data collections. However, without a major effort, their use seems always to be a year away.

• Sometimes the problem in fostering innovation is a lack of ideas, and sometimes it is the inability to implement new ideas.

• Competition among federal statistical agencies can be an impedi­ment to innovation, so collaboration is critical.

• A key to innovation is the willingness of the senior managers of statistical agencies to provide the necessary leadership and to fol­low through on the ideas discussed at the workshop.

• The Office of Management and Budget is responsible for providing leadership in eliminating bureaucratic barriers in contracting and recruitment.

• A system-wide marketing plan to academic institutions could stim­ulate academic work on federal statistical problems.

• Case studies of best practices could be useful in providing guid­ance on how to stimulate innovation.

• Communication within and between agencies could be improved.

• Progress in innovation needs to be measured periodically, perhaps through developing and disseminating annual or biannual reports on key innovations and research.

• The federal statistical system could develop a joint federal statistics research agenda.

• The Interagency Committee on Statistical Policy could take the lead in developing a marketing program with academic institutions and in establishing a culture of innovation in the federal statistical system.

• It is important for at least a subset of the agencies to work on spe­cific innovation projects while discussion proceeds on the larger issue of innovation in the federal statistical system.

• The federal statistical system could consider a cross-cutting cen­tralized research approach, although with each agency having the ability to retain local creativity.

• Although there are many opportunities to encourage innovation, it is not clear that the federal statistical system has the necessary will to engage them.

This page intentionally left blank.