Perhaps the most challenging innovation in Groves’ vision of the future of surveys is performing real-time estimation during data collection. Groves (2011b) envisions implementing real-time estimation routines—including imputation, nonresponse adjustment, and standard error estimation—after every 24 hours of data collection. Part of this progress would entail assessing whether the standard error increase due to imputation was acceptable or additional nonresponse follow-up was necessary. In this context, imputation can, in effect, be viewed as another mode of data collection. To make trade-off decisions about whether to terminate nonresponse efforts for a case using a particular mode, key statistics on the fully imputed estimates and measures of the imputation standard error and sampling standard error of the estimates would be actively tracked. Groves believes successfully implementing this real-time estimation and decision process at the Census Bureau would take at least 5 years.

In this vein, one issue that needs to be explored is the feasibility of blending the use of administrative records, scientometric tools, and survey techniques to improve the accuracy of data on STI human capital measures and other indicators that NCSES produces, such as research and development (R&D) input and performance measures. A multimodal approach would help in creating longitudinal series using existing and new information. In the near term, the topic could be explored through a workshop designed specifically to discuss the conceptual framework and feasibility of blending data acquisition techniques and using this mixed-methods approach to develop new indicators.3 This approach could be useful for developing real-time maps of networked scholars while measuring return on investments from federal research funds as they are used and linking them to outputs (paper and patents). At the same time, this approach would include periodically assembling basic data on education, employment, work activities, and demographic characteristics.

Data from administrative records and web-based sources—termed “business practice data” (see Chapter 4)—have been used for many years at federal agencies with two purposes: to benchmark sample survey data and, along with sample survey data, to produce official statistics. Horrigan (2012, 2013) gives several examples of sources being used by the Bureau of Labor Statistics (BLS), including the Billion Prices Project data; retail scanner data; the J.D. Power and Associates used car frame; stock exchange bid and ask prices and trading volume data; universe data on hospitals from the American Hospital Association; diagnosis codes from the Agency for Healthcare Research and Quality, used to develop the producer price index; Energy Information Agency administrative data on crude petroleum for the International Price Program; Department of Transportation administrative data on baggage fees and the Sabre data, used to construct airline price indices; insurance claims data, particularly Medicare Part B reimbursements to doctors, used to construct health care indices; and many more sources of administrative records data from within the U.S. government, as well as web-based data. According to Horrigan (2013), in addition to the development of price indices, administrative records and web-scraping data are used to “improve the efficacy of estimates … the Current Employment Statistics (CES) Survey uses administrative data from the Quarterly Census of Employment and Wages (QCEW)….” BLS also is “using web-scraping techniques to collect input price information used to increase the sample of observations we use to populate some of our quality adjustment models” (Horrigan, 2013, p. 26). Horrigan cautions, however, that “the principle of constructing an inflation rate based on the rate of price increase for a known bundle of goods with statistically determined weights lies at the heart of what we do. While research may show the viability of using a web-scraped source of data for a particular item, it needs to be done within the framework of this methodology” (Horrigan, 2013, p. 27).

The statistical methodology related to sampling and weights must be developed if these multimodal techniques are to be fully relied upon to deliver bedrock STI indicators. The panel must stress, moreover, that business practice data must be regularly calibrated using sample survey data. Business practice data contain a wealth of detailed and rapidly changing information that is not practically acquired using surveys. However, businesses and government enterprises generally do not maintain the sort of consistency across organizations, categories, and time that would enable cross-sectional and longitudinal comparisons. In time, and with appropriate financial and human resources, NCSES and other statistical agencies should be able to publish indicators based on business practice data, but only if the raw data are adjusted using a well-designed program of sample surveys. Indeed, the challenge will be to design the most efficient combination—financially and statistically—of traditional sample surveys and administrative and web-based sources.


NCSES needs to determine now how it will handle the above changes if they materialize and how the types and frequencies of various STI indicators will be affected. During the panel’s July 2011 workshop, Alicia Robb of the Kauffman Foundation encouraged NCSES to explore the use of administrative records to produce STI indicators. She also cautioned, however, that ownership issues associated with the use of those data will have to be addressed before they can become a reliable complement to traditional survey data.


3Statistical Neerlandica has prepublication views of a series of articles on the use of administrative records for analytical purposes, including regression analysis; see [December 2011]. For theoretical foundations of combining information from multiple sources of data, see Molitor et al. (2009). Also see Eurostat (2003).

The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement