6
Perspectives on the Science of Science Policy
The conference provided opportunities for observers with a range of perspectives to offer their reflections on the work funded by SciSIP and the primary issues facing the community of researchers concerned with the science of science and innovation policy. Several presenters were asked to reflect formally on particular perspectives, and wrap-up sessions at the end of each day provided an opportunity for broad reflections from members of the steering committee. This chapter summarizes the points made by these speakers.
M-H. Carolyn Nguyen, Microsoft Corporation
M-H. Carolyn Nguyen expects there will be “a radical transformation of our individual relationships with technology—computers—and what we expect from them.” She was asked to address upcoming trends in information technology research and the relationship between technology and policy, from the perspective of an executive at Microsoft Corporation. For her, one of the most important changes underway is a shift from computer systems that are driven by the technology itself to systems that are driven by the user. She suggested that several elements support a new understanding of what personal computing means, in which “an ecosystem [of technology] will work together on your behalf.”
A “seamless cloud connection,” she explained, will enable users to have access through their devices to the applications and services they need “at the right time, in the right context.” Thus, for example, a physician might have access to patient records whenever he or she is within the perimeter of the hospital, but not elsewhere. Part of what will make this seamlessness possible, she added, is technology that allows interactions to segue across different devices, so that users can continue an interaction as they move from their home computer to their car, to an office computer, or to a portable device.
The technology will adapt to the user’s needs, not the other way around, she emphasized, referring to this adaptability as “natural user interfaces or interactions.” The idea is that the technology adapts without the user having to learn how to operate or direct it. This capacity has already been applied in systems for people with disabilities—the systems are beginning to develop what she described as “human-like perception.” She noted that, “more and more systems can use and establish context to understand us and do things on our behalf, and then they can also, in order to refine that perception,… look at your interactions with peers to understand your behaviors.”
Another important trend Nguyen identified is that people’s increasingly complex interactions with technology “are generating massive amounts of data:” in 2011, the amount of information created was more than 1.8 zetabytes, or 1.8 trillion gigabytes. Ninety-percent of that data is unstructured, she added, noting that data are a source of innovation and economic growth. “Data is the fuel that drives all these powerful technologies,” she commented, but there is also “tremendous potential for abuse.” In Nguyen’s view, this is a key policy issue. She suggested that the best way to achieve balance between the benefits and potential harm will be to establish a “complete data ecosystem” in which individual users, policy makers, industry, and researchers from many disciplines, including the social sciences, work together to develop policies that balance the needs of all of these stakeholders.57
Elizabeth Wilder, National Institutes of Health (NIH)
Elizabeth Wilder described her thoughts about transformation in science, based on her experiences as director of the Office of Strategic Coordination at the NIH. That office is charged with identifying areas of science in which transformation is needed and using its funds to support researchers in those fields in overcoming challenges and pursuing opportunities likely to foster the needed changes. Wilder noted that transformation often takes place spontaneously in biomedical research: a remarkable, fortuitous discovery may open up entirely new fields. Her office, however, is focused on circumstances in which transformation can be pursued.
She and her colleagues have learned that most of the research at the NIH is initiated by creative investigators. In order to bring about transformation in a field, or engineered transformation, however, it is necessary to begin with a process in which a group of experts focuses on defining where the field needs to go and what needs to happen for it to reach those goals. For example, a series of NIH programs, funded over several years, were designed to make it easier to conduct interdisciplinary team research on biomedical topics. The programs focused on breaking down departmental barriers, and they were accompanied by administrative changes within NIH. Wilder believes that the result has been a “culture shift” and that many more people do now spontaneously consider interdisciplinary research.
Another program was designed to engineer an entirely new field of research on the extent to which the microbes that live inside and on the human body influence health. Developing this field of study was a daunting challenge, Wilder explained, because it involved the generation of new biocomputational informatics strategies. It was necessary to engage many researchers in sampling the biome in many healthy people and developing new computational and analytic methods. Demonstration projects were needed to shape understanding of what can be learned from differences across people in the composition of their biomes.
For this effort to work, Wilder concluded, it was necessary to have many researchers thinking collectively about the steps that would be needed to answer
________________
57For more details on this research, see Bus and Nguyen (2013); and Nguyen (2013).
fundamental questions about the microbes, and also about how the investigators could work together to draw conclusions.
Jason Boehm, National Institute of Standards and Technology (NIST)
Jason Boehm described the sorts of data that are needed at NIST and how they are used. NIST is “the nation’s measurement institute,” he explained, and its mission is to promote innovation and industrial competitiveness by advancing measurement science, standards, and technology. A federally funded, but non-regulatory agency, NIST runs collaborative institutes that address basic research in physics, biotechnology, quantum physics, and marine science. Their goal is to apply the research in ways that are useful to industry and help the nation solve problems and develop and deploy new technologies. The research expertise at NIST, for example, supports innovation in technology associated with lasers, memory, global positioning systems, and wireless communication.
This work puts NIST “right in the middle of big policy challenges,” Boehm explained. For example, NIST is helping to develop measurement tools and standards to promote cybersecurity, nanomanufacturing, and energy security. NIST has developed significant programs in these areas, but, as Boehm explained, there are still questions about the best ways to distribute funding, to develop public-private partnerships, and to assess results.
BIG DATA, SCIENCE METRICS, AND SCIENCE POLICY
Julia Lane, American Institutes for Research
Julia Lane noted that countries and agencies around the world are using evidence to inform policy decisions about public expenditures in such areas in health and education. This is not the case for science. However, the federal government has begun to require that agencies demonstrate their use of evidence in making investment decisions, and the Office of Science and Technology Policy has formed a working group to develop tools, data, and models that can be used to provide a more scientific, empirical basis for science and technology policymaking.
However, Lane noted, there is a significant gap in the data and tools policy makers need to do their jobs. An interagency group charged in 2006 with investigating the state of the science available to support policy decisions reached several conclusions:
- There is a well-developed body of social science knowledge that could be readily applied to the study of science and innovation.
- Although many federal agencies have their own communities of practice, the collection and analysis of data about the science and scientific communities they support is heterogeneous and unsystematic.
- Agencies are using very different models, data, and tools to understand their investments in science and technology.
- The data infrastructure is inadequate for decision making (National Science and Technology Council, 2010).
One of the problems, Lane suggested, is that the performance metrics used for science are not adequate. There is “an almost maniacal focus on counting publications,” she commented, which has “put the focus on publication rather than on thoughtfulness” (Lane, 2010). For example, she noted, it has been suggested that only about 10 to 15 percent of pharmaceutical assays published in peer-reviewed journals can be replicated (Young, Ioannidis, and Al-Ubaydli, 2008). An intellectually coherent theoretical framework will be needed, in Lane’s view, to provide the basis for accurate and valid measures. Research funded by SciSIP has played an important role in this discussion, she added, and it is one that needs to engage not only social scientists, but also scientists from physics, chemistry, and other domains.
The desired framework, according to Lane, should encompass measures that are timely, can be generalized and replicated, and are low in cost but of high quality. “That’s why the promise of ‘big data’58 in the context of science policy is so interesting,” Lane commented. In her view, “the biggest single contribution NSF can make [through the SciSIP program] is to help build a data infrastructure that can be used by many science of science policy researchers.” The core of that effort, she added, would be the capacity to obtain disambiguated data on individuals and to develop new text-mining approaches.
This is a significant challenge, but computer scientists have been developing new ways to analyze and summarize text, Lane noted. For example, the NIH has collaborated with the National Science Foundation, the U.S. Department of Agriculture, the U.S. Environmental Protection Agency, and the White House Office of Science and Technology Policy to develop a program called STAR METRICS59 to study the effects of research on innovation. That group has been working on automatically capturing information about scientists’ work through building on SciSIP-funded data collection activities.
According to Lane, one of the biggest challenges in this sort of work is building an infrastructure that is people-centered rather than document-centered (see Figure 6-1).
________________
58In this context, “big data” refers to collections of data that are too large to be processed or managed by most data processing tools, such as data that can be collected in an automated way. New methods of processing and analyzing such data include the harnessing of large numbers of servers.
59For more information, see https://www.starmetrics.nih.gov [January 2014].
FIGURE 6-1 STAR METRICS conceptual framework.
SOURCE: Presentation to SciSIP Principal Investigators’ Conference by Julia Lane, September 2012.
SCIENCE, TECHNOLOGY, AND INNOVATION POLICY IN JAPAN
Asako Okamura, Japan Science and Technology Agency
Science for Redesigning Science Technology and Innovation Policy (SciREX) is a Japanese program modeled in part on SciSIP, explained Asako Okamura, but it also reflects concerns specific to Japan. The 2011 earthquake and tsunami that affected Japan recently gave rise to a critical examination in that country of the relationship between scientists and the government. A growing expectation that science and technology will provide responses to economic and social challenges, she noted, has focused attention on the importance of evidence in policy. The development of evidence as a shared social resource that can serve as a foundation for public participation in policy formation was thus a primary goal for SciREX.
There are four elements to SciREX, according to Okamura: a policy-oriented investment investigation program, a research funding program funding similar to SciSIP, fundamental research and human resource development, and a data and information
infrastructure. As it gets underway, SciREX has begun to engage universities in its network and also to reach out to international research partners.
Okamura went on to explain that the program faces a number of challenges. SciREX has not yet been able to map the various research fields the program hopes to address, and methodologies still need to be developed to “structure and synthesize outputs.” Doing so will be critical for effectively integrating research findings into the policy process. At the same time, she added, the program needs to better promote the value of research findings, both to policy makers and to the general public. Okamura closed by noting that SciREX is already collaborating with U.S. institutions and is seeking other international opportunities to collaborate.
Several speakers were asked to reflect on the issues and ideas raised at the conference. Irwin Feller, Pennsylvania State University and chair of the conference steering committee, cited the intellectual return NSF has gotten on its investment in these SciSIP-funded projects, and noted that “the real intellectual payoff is yet to come in larger research communities.” He suggested that the dialogue that has begun among policy makers and researchers is a “great open invitation to the research community” to continue to tease out the policy implications of their work and how past work has influenced policy environments.
For Feller, an important unresolved issue is how to address uncertainty about research findings. Researchers are very aware of limitations to their findings. “What is the value to the policy world,” he asked “of an answer that is cautious and conservative?” He noted that Jason Owen-Smith, in his presentation, had suggested that when one is challenged about the degree of confidence to have in a body of work, a solution may be to put it in probability terms. “Policy makers know there is no single definitive study,” Feller noted, “as long as the researcher is transparent, they are glad to have the findings.”
Benjamin Martin, Sussex University and member of the conference steering committee, provided a European perspective on what the U.S. science policy community is trying to do. He commended those involved for having built a science policy community in the United States. He noted that “there wasn’t one before 2002—at least it was dispersed and fragmented.” He noted, however, that science and innovation policy had a good start in the late 1950s, when the RAND Corporation in California gathered a “galaxy of future economics stars … to look at the future of economics and the management of R&D.” There was a long interval between that and SciSIP, however. During the years following the RAND gathering, a whole host of dedicated science policy research centers were set up in multiple northern European countries. There were a few attempts in the United States during the intervening decades, and he said he wondered why those did not work.
Martin also had a few observations about the interaction between policy makers and researchers. He suggested there is not yet an optimal model for how this interaction can work. There is “pull” from policy makers who need information and “push” from researchers who have work they believe can be useful, as well as policy “entrepreneurs” who can mediate between the two, he noted, but added that “we are not there yet.” A final
question for him was the extent to which a researcher is “permitted to simplify and perhaps compromise and tailor what you have to say based on politics.” Clearer rules about this may be needed, he suggested, noting that “we all have our own views. Some have gone glib and crossed a boundary—others never surface and get their message across.” An effective model for the interaction will clarify this question, he concluded.
James Turner, Association of Public and Land-grant Universities and member of the conference steering committee, provided the final reflections on the conference, focusing on the policy relevance of the body of work SciSIP has funded. He noted that the work has both scientific and political merit and expressed appreciation for its contributions. “What SciSIP is really about is bridging cultures,” he observed. As discussion about the STAR METRICS program suggested, engaging the field of anthropology in the study of science policy has been a particularly useful development.
Turner asserted that “you can’t do your work right” unless the social sciences and the natural sciences learn how to talk to each other and cooperate on problems. This effort can be “quite a chore,” however, because even within subgroups, researchers do not always understand one another. Like Martin, he believes that the more difficult challenge is making physical and biological sciences translatable to policy makers. One challenge is that there are different sorts of policy makers in the executive branch of the federal government, Congress, industry, state and local government, and think tanks.
Turner explained that his years working on Capitol Hill helped him to understand the “huge chasm” between scientists and the political world, especially Congress. One key to understanding the political world, in Turner’s view, is the rewards system. Circumstances reward senators and representatives for pursuing reelection and for party loyalty because these are the routes to accomplishing the goals that led them to seek office. It is also important to understand the culture. The House of Representatives and the Senate, he noted, are “dominated by lawyers—the procedures and mores are lawyers’ mores.” Moreover, “a good story is what you need on Capitol Hill to win the debate.” The committee system provides a way of involving specialists in the topics of legislation, but these people are “experts within the congressional framework.” That means that “having more anecdotes” or stories based on evidence is a key to success. The Congressional Research Service was designed to be an in-house bridge to the science community, Turner noted. But, he added, “everybody is trying to influence them [senators and representatives]: lobbyists, the media, campaign contributors, constituents. They need to sort out within their value framework what truth is.”
“How can SciSIP break in?” Turner wondered. He described a brochure on climate change he had seen that focused on changes the insurance industry will need to make to survive as the climate changes. It was well done, but no Ph.D. scientists were directly involved in preparing it. This would be an ideal SciSIP topic, he observed, but to influence policy makers, the SciSIP community will need stories based on social science research analysis to inform policy makers.
Finally, Turner noted, “this enterprise is data driven,” and it is critical to focus on data sources of the 21st century, particularly big data. The separation of data from the physical world so that it can be manipulated in new ways “is almost a third revolution, after industrial,” Turner concluded; “getting the right data and knowing how to use it will be the way to sharpen research and reach customers.”
