coverage of frames; falling participation rates; increasing reliance on nonresponse adjustments; and for surveys with high response rate targets, inflated costs.” His proposed solution set for what agencies should do to address these issues is to develop an approach of a “blended data world by building on top of existing surveys.”1 Groves (2011b) envisions multimodal data acquisition and manipulation of data, including: “Internet behaviors; administrative records; Internet self-reporting; telephone, face-to-face, paper surveys; real-time mode switch to fill in missing data; and real-time estimation.”

NCSES needs to determine now how it will handle these changes if they materialize and how the types and frequencies of various STI indicators will be affected. During the panel’s workshop, Alicia Robb (of the Kauffman Foundation) encouraged NCSES to explore the use of administrative records to produce STI indicators, but she also cautioned that ownership issues associated with use of those data will have to be addressed before they could become a reliable complementary data source to traditional survey data. Stefano Bertuzzi (of the National Institutes of Health and the STAR METRICS Program) also presented techniques of using administrative records at universities to determine the impact of federal research funds on scientific outputs and the development of human capital in the physical and biological sciences.

There are also foresight questions that STI indicators can inform. Demographic, economic, technological, and organizational changes will all influence the subjects being measured, the mechanisms used to measure them, and the products offered by NCSES. STI indicators will be called on to answer the following questions: How will demographic shifts affect the science, technology, engineering, and mathematics (STEM) workforce, nationally and internationally? Will those shifts change the locus of the most highly productive regions? Will global financial crises slow innovation activities or merely change the locus of activities? When will emerging economies be integrated into the global ecosystem of innovation, and what effects might they have on the system? However, as cautioned above, indicators are not predictors. They can be used in isolation or in groups to show tendencies, voids, and at times what additional information is needed.

All of this suggests a shift in emphasis over time for NCSES’s indicators program. The agency will have to make decisions on whether and how to adopt the new techniques. Although NCSES is not expected to eliminate all traditional survey methods, it is expected that the prolonged austerity of federal budgets will necessitate increased reliance on web-based techniques and databases. On the horizon, the panel believes that NCSES will have to use surveys more efficiently and increase use of web-based tools for harvesting data, particularly on human capital measures and output measures related to scientific discoveries and innovation, and databases from other government agencies and private providers.


At the panel’s workshop, presentations by Erik Brynjolfsson (Massachusetts Institute of Technology), Lee Giles (Pennsylvania State University), Carl Bergstrom (University of Washington), and Richard Price ( provided insights about tools that can be used


1 For further comments on this point, see Census Bureau discussions: [December 2011]; [December 2011]; [December 2011].

The National Academies of Sciences, Engineering, and Medicine
500 Fifth St. N.W. | Washington, D.C. 20001

Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement