6
Understanding the Research Enterprise as a Complex System
KEY POINTS IN THIS CHAPTER
- With a holistic understanding of the system of research, government can enable greater benefits from research through policies that address three pillars of the research system: a talented and interconnected workforce, adequate and dependable resources, and world-class basic research in all major areas of science.
- New and existing measures could be used to assess each of the three pillars. These measures might include, for example, indicators of human and knowledge capital, indicators of the flow of knowledge in specific fields of science, indicators with which to track the flow of foreign research talent, portfolio analyses of federal research investments by field of science, international benchmarking of research performance, and measures of research reproducibility.
The committee’s findings reveal that the pathways from research to innovation are multiple, diffuse, and interconnected. As described in Chapter 1, innovation is an emergent phenomenon, consistent with the principles of systems theory, which depends on the actions of the system as a whole rather than those of one or two components in particular. Making a change to one component of a complex system can affect other
components, often in unpredictable ways. On the other hand, a desired change in the behavior of a system—for example, increased output of new technologies—is unlikely to be achieved by changing one or even a few components without regard to the critical pillars of the system and the relationships among them. That is why a focus solely on technology transfer at universities or on which particular research disciplines to fund might not result in the desired effect and could have potentially undesirable consequences.
We hold that the highly productive American research enterprise rests on three critical pillars—a talented and interconnected workforce, adequate and dependable resources, and world-class basic research in all major areas of science. To understand how these pillars interact to produce research discoveries, one must also understand how knowledge flows among domestic and global networks of individuals and institutions; how research is influenced by the availability of scientific infrastructure, funds, and other resources; how the quality, including the usefulness, of research discoveries is affected by management, research environments, institutions, and peer review; and how all of these aspects interrelate. These topics are not well understood and need to be addressed with future research. Nonetheless, we attempt in this report to enhance understanding of these elements.
As if the complexity of the research system were not challenging enough, it is also necessary to understand how that system interrelates with the innovation system if the development of innovations from research discoveries is to be enhanced. But the story does not end even there. Both the research and innovation systems interact with manufacturing, commercial, legal, political, economic, and other systems.
Despite all this complexity, however, patterns do emerge that can inform policies aimed at achieving further societal benefits from research, and in particular at keeping the United States at the forefront of global competition for new technologies and innovations. Moreover, measures can be developed to guide the effective implementation of those policies. Taking this perspective reveals promising opportunities to increase the benefits of federally funded research to society. This chapter presents an argument for cultivating a greater understanding of the research system through a focus on talent, resources, and basic research, and suggests how measures might be created or adapted to support these three pillars.
UNDERSTANDING THE SYSTEM OF RESEARCH
A complete understanding of the system of research may be elusive, but there are many important aspects of the system that, with proper measures and assessments, can be better understood. Within this under-
standing, the role of failure must be properly valued. As discussed in Chapter 4, scientific research projects that fail to achieve their original objective nonetheless play extremely valuable roles in the overall research system by providing important training experiences, by contributing to the stock of scientific knowledge, and by redirecting research in what ultimately may be transformative directions.
The committee finds that the key to understanding the research system is how knowledge is
- Generated—Research produces value through the generation of a stock of knowledge, which occurs at research universities and other organizations, such as government and industry laboratories, and through team-oriented collaborative mechanisms, such as research consortia and clusters. This knowledge stock is a societal resource whose value depends on developments in the uncertain future. Which knowledge will ultimately prove useful may not be immediately clear, but all knowledge—including that from failures—must remain accessible.
- Utilized by well-trained and highly talented people—The workforce trained at research universities—their talent, abilities, knowledge, skills, and experience and the networks of professional connections they have made—is one of the most valuable products of the system of research. These people make use of the stock of knowledge and adapt it to their specific needs, often to society’s benefit.
- Disseminated through networks of researchers and institutions—The stock of knowledge becomes valuable when it flows to, from, and among people engaged in all forms of research and development (R&D) and when it flows at the right times to the particular places where it is needed most. This flow is made possible by partnerships and networks, as well as by dissemination of information through publications and at conferences. Much knowledge in the early phases of the R&D cycle is somewhat tacit in nature and often is embodied in individual researchers and transferred through interpersonal contact. However, knowledge also is codified to varying degrees in published papers, formal databases, and patents as a research field matures.
- Affected along the way by external variables—By producing talented people, networks, partnerships, and other assets, the systems of research and innovation provide almost everything needed to ensure the continued generation, flow, and use of knowledge. But external variables—such as investment and infrastructure, intellectual environment, management, motivations, and incentives—
-
can enhance or hinder the ultimate success of the research enterprise. And many of the factors that influence the translation of research advances into societal benefit (e.g., labor markets, financial factors, regulation) are themselves well beyond the boundaries of most conventional definitions of the research system.
- Absorbed and used for economic and other societal benefits—New societal benefits are realized through a diverse range of public and private entities that provide the complementary assets needed to transform knowledge into products and services and then penetrate markets.
The key to an understanding of the innovation system is how knowledge is used effectively to produce new technologies and other innovations of economic value.
SUPPORTING THE THREE PILLARS OF THE RESEARCH SYSTEM
With a more nuanced understanding of the system of research, government can enhance the public returns on its research investments through policies that address the system’s three pillars: a talented and interconnected workforce, adequate and dependable resources, and world-class basic research in all major areas of science. As described by the National Research Council’s (NRC) Committee on Science, Engineering, and Public Policy (National Academy of Sciences, 2000) and noted earlier in this report, “major areas” refers to broad disciplines of science and their primary subdisciplines, as well as emerging areas of science. Each of the three pillars supports the research system as a whole, rather than a particular type of research (e.g., basic, applied, or proof-of-concept). The ultimate economic and societal impacts of the research system depend largely on wise and coordinated investment in and management of each of these pillars.
A research system based on talent of high-caliber, adequate and dependable resources, and excellence in basic research is necessary for successful innovations, but it is not sufficient. Also necessary is an innovation system that supports a culture of innovation within firms and among individuals, so that firms and entrepreneurs value creative and unconventional ideas and are willing to take risks, in particular with research investments, and accept failures (Mote, 2013).
Metrics and other measures could be developed to help in understanding whether the government is supporting the three critical pillars successfully. Whereas existing metrics provide limited assistance in answering broad questions about the research system on a national scale (see Chapter 4), the measures described in the following sections could
provide valuable insights into trends, gaps, and opportunities for each pillar. In particular, it may be useful to develop a national set of research portfolios and refine existing methods of international benchmarking to better assess the relative global leadership of the United States in these three critical areas.
Many measures for assessing the performance of policies intended to strengthen the three pillars of the research system are identified in the National Research Council (2014) report Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. That report provides guidance to the National Science Foundation’s (NSF) National Center for Science and Engineering Statistics (NCSES) on how its data collection could be improved to guide research and innovation policy and in particular to allow for international comparisons. In this chapter, we draw heavily on that report’s discussion of data gaps and measure development, particularly with regard to the strength of the nation’s knowledge and human capital.
Below, we describe in more detail the three pillars of the research system. We also present potential measures for assessing the vitality of each.
A Talented and Interconnected Workforce
A talented and interconnected workforce is a critical input to the research system. The U.S. domestic pool of talent relevant to innovation includes individuals that benefit from science, technology, engineering, and mathematics (STEM) education and training, as well as career-technical (i.e., vocational) training. But it also encompasses many other aspects of the system as well, including immigration, professional networks and partnerships, and a supportive and creative research environment that nurtures the creativity and ingenuity of talented researchers (National Academy of Engineering and National Research Council, 2012; National Research Council, 2008, 2010a, 2011a, 2012b; OECD, 2012a). The interconnectedness of this talented workforce, as discussed below in the section on networks, also is key to the success of the research enterprise.
To compete globally, the United States must be able to leverage the expertise of world-class researchers, which will in turn amplify and expedite the nation’s capacity for innovation. This can be accomplished by maintaining a strong pool of scientists and engineers familiar with research at the cutting edge, whose networks can broaden their expertise. A large body of empirical and theoretical economic literature has linked innovation activity, including the absorption of technologies discovered elsewhere, to the levels (Benhabib and Spiegel, 2005) or the composition (Manca, 2011; Vandenbusche et al., 2006) of the human capital of an economy. The translation of research findings into new technologies requires
people who truly understand research in diverse fields; can make unexpected connections; and can devise counterintuitive solutions to problems related to health, defense, communication, the environment, the economy, and other areas of national concern. A critical variable is the flow of talent from abroad to U.S. research institutions and firms, which can be affected by such variables as the research environment and immigration policies.
The STEM Workforce
American research universities differ from the centralized university systems in many other nations. They must supply highly trained STEM graduates in the numbers and fields needed to support the demand of the U.S. research enterprise. Their ability to do so depends in turn on the pool of K-12 students who prepare for and pursue careers in science and on the foreign students and workers who can be attracted to study and remain in the United States. Their contributions to the U.S. research enterprise further its world-class nature, thus creating a self-reinforcing cycle.
The balance of talent in the STEM workforce remains a controversial topic. Federal agencies such as NSF and the National Institutes of Health (NIH) track information on scientists who receive training awards, but insufficient data are available for determining how best to balance STEM talent by field of science, for example. Some large information technology (IT) firms have encouraged the immigration of people with technology expertise to the United States, claiming a shortage of STEM talent. However, opinions on this strategy differ, and there is ample room for further study of the issue, as evidenced by recent publications (National Research Council, 2012c; The Research Universities Futures Consortium, 2012; Salzman et al., 2013; Stephan, 2012; Xie and Killewald, 2012). An article in Science notes the increasing difficulty of retaining students in STEM fields, as many students start but do not finish college with a STEM major (Graham, 2013). Weaknesses in the domestic K-12 system, including those that have the effect of excluding historically underrepresented groups from benefiting from postsecondary STEM education, ultimately diminish the diversity and viability of this talent pool.
Networks
As discussed earlier, basic and applied research leads to the development of national and international networks of researchers, which increase the system’s connectivity by linking research groups, disciplines, and institutions across and within national boundaries. For example, one of the important assets of a new graduate is the network of researchers that he or she has developed. Ideas, instrumentation, and analytical methods
often are freely shared through these networks. Industries draw on what they learn from these networks to develop new technologies and other innovations and to obtain new ideas and approaches for addressing technological problems they might not otherwise pursue. Research networks increase the stock of knowledge and broaden the range of technological opportunities available for commercialization. And through research networks, particularly peer-to-peer collaborations, the nation can draw on the results of research conducted throughout the world. For a nation to tap into this stock of knowledge effectively, however, it must maintain an enterprise of scientists and engineers conducting world-class research.
Measures for Assessing Talent
The NRC report on science, technology, and innovation indicators (National Research Council, 2014, pp. 6-14) provides guidance on the development and use of metrics to measure networks, as well as human and knowledge capital. It may be possible to create indicators of human and knowledge capital based on existing longitudinal data from agencies and organizations such as the U.S. Census Bureau, NCSES, and the U.S. Bureau of Labor Statistics. Doing so, however, would require the ability to link datasets from each agency.
Indicators could be generated, for example, to track the flow of knowledge in specific fields of science. In addition, indicators of STEM labor mobility could help answer questions about the career progression of scientific researchers and recent STEM graduates by following the movement of individual researchers to posts in industry, government, and academia. In addition, data from full-text dissertation databases could be mined to create indicators for emerging research topics. Doing so might allow for a better match between STEM training and the demand for particular skills.
Adequate and Dependable Resources
Certainly research depends on adequate and dependable funds. But resources encompass much more—in particular, scientific infrastructure, or the tools that allow for research excellence, and world-class research universities, national laboratories, and other research institutions. Dependable resources thus provide critical support for the research process. Resources make it possible for the United States to maintain cutting-edge IT and other scientific infrastructure, the best possible pool of talent, and world-class scientific institutions and means of communication. The case study in Box 5-1 in Chapter 5 illustrates how the combined resources of government and industry can drive an innovation’s success.
Key Features of Adequate and Dependable Resources
Key features of adequate and dependable resources include government support for proof-of-concept research, resource stability, and resources to support all fields of science.
Government support for proof-of-concept research. Policies providing businesses with incentives to undertake long-term or high-risk research, in particular, can help support the pathway from research to innovation. Private industry has a good sense of short- and medium-term needs for which proof-of-concept research would be useful. Government regulations, policies, or incentives, as well as increased public-private partnerships, may encourage private industry to help fill the gap between a research discovery and investment in its use by industry. Industry often is reluctant to fund proof-of-concept research when the risk is high, although there are exceptions, such as the insulin inhaler Exubera® developed by Pfizer and withdrawn from market within the first year of sales (Johnson, 2007). In the current climate, a number of federal research funding agencies have assumed increased responsibility for supporting applied research, particularly high-risk and proof-of-concept research, to bridge this gap (see Box 6-1).1 The government’s role also includes continued support for research that leads to technologies such as those needed by the Departments of Energy and Defense (Mazzucato, 2011).
Some examples of government support for proof-of-concept research are described in Box 6-2. Although Box 6-2 includes a number of these programs, most are relatively new, and their ultimate effectiveness (or survival) is uncertain. Moreover, few such programs that have been in existence for more than 5 years have been rigorously or systematically evaluated to determine their effectiveness.
Resource stability. Policies that help maintain the predictability and stability of federal research funding can boost the infusion of talent into the U.S. research enterprise by encouraging students to pursue STEM careers and discouraging established researchers from leaving their careers. Stable federal funding also helps the U.S. research enterprise attract and retain
_____________
1In fact, federal programs such as the R&D and extension programs of the U.S. Department of Agriculture (USDA) have long supported R&D in more applied areas that is designed in part to accelerate the adoption as well as the creation of new technologies. And in the field of aeronautics, the National Advisory Committee on Aeronautics, established in 1919 and the forerunner of the National Aeronautics and Space Administration (NASA), supported “proof-of-concept” R&D in civilian and military aircraft that underpinned such major technological advances as the DC-3. See Ruttan (2001) for a discussion of USDA agricultural R&D or Mowery and Rosenberg (1982).
BOX 6-1
Public-Private Funding and the 3D Printing Boom
The technology for creating three-dimensional objects from a digital model, known as “3D printing,” has existed since the 1980s. In 2012, however, this innovation achieved a new level of commercial success thanks to a $70 million combined investment from the federal government and private industry that established the Midwestern town of Youngstown, Ohio, as a manufacturing innovation hub.
In 1984, an engineer named Chuck Hull developed a technology called stereo-lithography, which uses a robot to stack layer after layer of a material such as plastic, resin, or titanium in an additive process until it produces a three-dimensional object. Hull later cofounded 3D Systems Corporation, which in the early 1990s produced the first stereolithographic machine. The original printer used an ultraviolet laser to solidify each layer of photopolymer, and it demonstrated that complex objects could be manufactured in a matter of hours. By the late 1990s and early 2000s, 3D Systems had established collaborations with academic researchers in North Carolina to create synthetic human organs.
The 3D printers are now used to create everything from synthetic human tissues to footwear, even food. Companies such as General Electric use the printers to create turbine components. A free and open-source software printer produced by the RepRap Project can print individual parts of the printer that can be assembled to generate a continuous supply of the machines. And the company Defense Distributed offers a printable AR-15-type rifle. The Economist (2011) has predicted that the long-term impacts of 3D printing will be akin to those of the printing press in the 1400s, the steam engine in the 1700s, and the transistor in 1950.
In 2012, the technology’s already booming commercial success was boosted still further when private industry and five federal agencies, led by the Department of Defense, established Youngstown as a hub for 3D manufacturing as a pilot program of the National Network for Manufacturing Innovation, initiated by President Obama to spur the development and adoption of pioneering manufacturing technologies. The program aims to “help to strengthen the competitiveness of existing U.S. manufacturers, initiate new ventures, and boost local and state economies.”* The hub in Youngstown is expected to attract venture capitalists and research professionals to the area.
_____________
*More information is available at http://manufacturing.gov/nnmi.html [August 2014].
foreign talent. Stability is particularly important in the wake of fluctuations in research funding due to the American Recovery and Reinvestment Act of 2009 (ARRA) and the recent sequester (see Box 6-3).
Resources to support all fields of science. Priorities for funding research must be established with care so as to sustain the entire U.S. research enter-
BOX 6-2
Federal Government Support for Proof-of-Concept Research
- National Institutes of Health (NIH), National Center for Advancing Translational Sciences (NCATS) (budget request, fiscal year [FY] 2013: $639 million): Established in 2011, NCATS initiates collaborations among government, academia, industry, and nonprofit patient organizations to enable faster and more effective translational interventions that improve human health.
- NIH-Larta Partnership (budget request, FY 2013: not available): NIH has partnered with Larta to design and deliver a program that helps accelerate Small Business Innovation Research (SBIR) and Small Business Technology Transfer (STTR) Phase II awardees’ commercialization outcomes. Larta’s mentors (mainly venture capitalists) are matched with NIH awardees to develop and deploy business plans for commercializing their NIH-funded technology. Larta also has a Web portal for participating companies that serves as a repository for program deliverables and performance tracking, and provides tools, communications, and updates.
- National Science Foundation (NSF), Industry/University Cooperative Research Program (I/UCRP) (NSF contribution, FY 2011: approximately $15 million):* NSF’s I/UCRP Program allows industry, government, and other organizations to leverage R&D investments with more than 60 cooperative research centers known for their innovative research capabilities. The program provides an opportunity for research universities to partner with other institutions to conduct industrially relevant research.
- NSF, I-Corps (budget request, FY 2013: $18.8 million): I-Corps fosters entrepreneurship to promote the commercialization of technology that has previously been supported by NSF-funded research. The program matches an entrepreneur and an NSF awardee to develop a business plan for commercializing a technology, and provides financial support to the team for the development of a prototype or proof of concept.
- NSF, Engineering Research Centers (ERCs) (budget request, FY 2013: $69 million): In 1985, NSF began sponsoring ERCs at universities across
prise, encompassing all fields of science, over the long term. International benchmarking has the potential to reveal scientific areas pursued elsewhere that may not be adequately supported in the United States (National Academy of Sciences, 2000).
The shift in funding over the past 25 years toward biomedical research is shown in Figure 2-3 in Chapter 2. A recent article in Issues in Science and Technology (Merrill, 2013) analyzes federal funding by field of science from 2001 to 2011 using data from NSF’s federal funds survey. The article shows that since 2001, despite the push to double funding in the physical sciences and engineering under the America COMPETES Act, funding
-
the United States, each in close partnership with industry, to foster technological breakthroughs for new products and services and to prepare U.S. engineering graduates for successful participation in the global economy. The ERCs provide a forum for industry to collaborate with faculty and graduate and undergraduate students on the commercial advancement of technologies in the focus areas of manufacturing, biotechnology and health care, energy/sustainability/infrastructure, and microelectronics/sensing/information technology.
- Federal Small Business Innovation Research Program (budget request, FY 2013: 2.8 percent of agency’s extramural R&D budget): The SBIR Program is a set-aside program to enable domestic small business concerns to engage in research/R&D with the potential for commercialization. Federal agencies with extramural R&D budgets of more than $100 million are required to allocate 2.8 percent of their R&D budget to this program. Twelve federal departments and agencies participated in 2013: U.S. Department of Agriculture (USDA), National Institute of Standards and Technology (NIST), National Oceanic and Atmospheric Administration, U.S. Department of Defense (DoD), U.S. Department of Education, U.S. Department of Energy (DoE), U.S. Department of Health and Human Services (HHS), U.S. Department of Homeland Security, U.S. Department of Transportation, Environmental Protection Agency, National Aeronautics and Space Administration (NASA), and NSF.
- Small Business Technology Transfer Program, modeled after the SBIR Program (budget request, FY 2013: 0.3 percent of agency’s extramural R&D budget): STTR is a highly competitive program that reserves a specific percentage of federal R&D funding for award to small business and nonprofit research institution partners. Federal agencies with extramural R&D budgets of more than $1 billion are required to allocate 0.3 percent of their R&D budget to this program. Five departments and agencies participated in 2013: DoD, DoE, HHS, NASA, and NSF.
_____________
*Available: http://www.nsf.gov/eng/iip/iucrc/program.jsp [June 2014].
for the physical sciences has remained flat, and that for engineering has declined. In an attempt to reverse this trend, Congress boosted the budgets of NSF, National Institute of Standards and Technology (NIST), and the U.S. Department of Energy’s Office of Science, under the presumption that this money would flow to the physical sciences and engineering. However, U.S. Department of Defense-funded engineering research, which accounts for one-third of all federal investments in engineering, declined steeply, down 26 percent from 2001 to 2010. Meanwhile, almost 75 percent of the $13.1 billion (in 2005 dollars) in research funding under
BOX 6-3
Instability in National Institutes of Health (NIH) Funding
This report emphasizes the important role of federal research funding in supporting the training of future scientists and engineers. But the importance of this funding in supporting the human capital component of the U.S. research enterprise extends well beyond research grants and stipends provided to graduate students. In many scientific fields, federal research funding supports the postdoctoral fellowships that are the first positions for new degree holders. In addition, federal funds are crucially important sources of support for junior faculty seeking to launch their laboratories and research careers. In other words, federal funds support a complex multiyear training regime for virtually all scientists in U.S. universities, a pipeline that extends from graduate school through postdoctoral training and the establishment of a scientific laboratory and research agenda.
This extended training process is vulnerable to disruption from fluctuations in research funding. When research grants for senior faculty are not renewed, support for graduate students and postdoctoral fellows is likely to be reduced, and when these funding reductions occur suddenly, the disruptive effects are all the greater. But the human capital pipeline also may be destabilized by unexpected surges in research funding, which attract more students to pursue graduate studies in a given field, lead senior faculty to open up more postdoctoral fellowships, and in some cases lead university administrators to support hiring of additional junior faculty. Funding upswings that are followed by cuts or even extended periods of flat growth in inflation-adjusted funding are especially disruptive in this context, and have the potential to degrade the efficiency of both the training and research supported by federal funds.
With this background in mind, it is sobering to observe the wide swings in federal funding for NIH, by far the largest single federal supporter of academic research in the United States, during fiscal years (FY) 1998-2014 (see the figure below). During the first 5 years of this period, a bipartisan coalition succeeded in doubling NIH funding, producing an average annual growth rate of 12 percent in constant-dollar NIH funding during FY 1998-2003 (see Freeman and van Reenen [2008], who use the Biomedical Research and Development Price Index to convert nominal to constant dollars). This period of rapid funding growth was followed by flat budgets that translated into declines in constant-dollar funding: Freeman and van Reenen estimate that by FY 2007, the real NIH budget was nearly 11 percent lower than in FY 2004, and by FY 2009, funding had dropped by roughly 13-14 percent. These reductions in funding were disruptive to the training pipeline described above. In the words of Freeman and van Reenen (2008, p. 28):
The deceleration caused a career crisis for the young researchers who obtained their independent research grants during the doubling and for the principal investigators whose probability of continuing a grant or making a successful new application fell. Research labs were pressured to cut staff. NIH, the single largest employer of biomedical researchers in the country, with more than 1,000 principal investigators and 6,000 to 7,000 researchers, cut the number of principal investigators by 9 percent.
NIH’s budgetary fluctuations continued. In early 2009, passage of the American Recovery and Reinvestment Act (ARRA) provided a 2-year increase in temporary funding for NIH of more than $10 billion, or nearly one-third of the agency’s 2008 budget, triggering another sudden funding upswing. NIH and university administrators strove to minimize the destabilizing effects of this temporary funding surge on training, research, and education by allocating much of the increase to construction or one-time equipment purchases. The 2-year funding increase spanned FY 2009-2010 and was followed by a decline in constant-dollar NIH funding of more than $4 billion—from roughly $32 billion in FY 2010 to approximately $27 billion in FY 2011 (National Science Foundation, 2013).
The FY 2010-2011 funding reduction was followed by the sequestration budget cuts in FY 2012, which imposed an additional $1.5 billion in reductions on the agency’s FY 2013 budget. As a result, the budget’s real-dollar total was reduced to roughly $26 billion.
The figure below depicts NIH’s budgetary fluctuations for the FY 1990-2014 period, using the Biomedical Research and Development Price Index to convert nominal to constant dollars. The wide swings in funding depicted in the figure indicate the magnitude of funding instability experienced by this key federal supporter of U.S. academic research during the past 20 years. Comparison of the long-term
How the National Institutes of Health is being unintentionally defunded.
SOURCE: White (2013). Reprinted with permission.
trend line with the lower growth indicated by the actual funding for FY 1998 versus FY 2014 reveals the reduction in long-term growth in NIH’s real budget. The effects of this reduction were greatly exacerbated by the wide swings in funding, which destabilized the training pipeline that produces future generations of biomedical researchers.
Although the NIH experience is unusual among federal research funding agencies—the initial doubling of the NIH budget during 1998-2003 and the large temporary increases under the ARRA reflect the political popularity of biomedical research within Congress—similar trends are apparent in the budgets of other important federal research funding agencies. The costs of these unstable funding trends are large and lasting, and undermine the performance of the overall U.S. research system.
the ARRA in fiscal years 2009 and 2010 supported NIH-funded research in the biological and medical sciences.
Performance Measures for Portfolio Management
Science must be managed not only through particular programs but also through portfolios of programs, some of which may entail research directed toward a particular national goal. Performance measures, whether based on research outputs, such as publications or patents, or on progress toward a particular goal, can be useful for managing programs within a portfolio. Some of the measures developed for evaluations of research funding programs (see Chapter 5) may be useful for performance measures.
Research projects often have predictable outcomes, in that most funded projects are designed to test a well-reasoned hypothesis and have a high likelihood of accomplishing their originally stated goals. Nonetheless, the ultimate utility of much research is unpredictable because research findings may eventually be used in unforeseen ways. Investing in research without a clear understanding of its utility is taking a risk. Just as with a financial investment, one cannot predict the winners and losers. But a broad financial investment portfolio can be relied upon to do well over the long term. Measures of risk, even if only qualitative, can be used to spread the risk of investments across the programs in a portfolio. The same is true for investments in research. A well-managed portfolio—by which we mean one that spreads risks, explores a diversity of approaches and topics within a field, invites unconventional thinking, and rewards
long-term vision—can manage risks and lead to discoveries from some research projects that more than justify investment in the entire portfolio.
The U.S. government currently invests a large sum in a very broad research portfolio spanning many scientific and engineering disciplines. As we have noted, however, no agency, office, or committee within the Executive Branch or Congress systematically monitors the breadth of federal research investments across disciplines and scientific fields in ways that can support the goal of balance and sustainability of the overall scientific research enterprise.
Some recent policy experiments in developing portfolio analyses of federal research investments in specific fields have recently been launched. In 2011, NIH established an Office of Portfolio Analysis to enable NIH research administrators and decision makers to evaluate and prioritize current, as well as emerging, areas of research. In 2012, the National Science and Technology Council delivered a mandated report to Congress that provides a “national strategic plan for advanced manufacturing” (National Science and Technology Council, 2012). Partly because of this report, the White House directed the NIST to establish an Advanced Manufacturing National Program Office to coordinate management of government-wide research portfolios in manufacturing. In 2013, that office produced a concept paper describing both a conceptual model for management of research portfolios and specific metrics (Advanced Manufacturing National Program Office, 2013).
World-Class Basic Research in All Major Areas of Science
A successful research system is one in which the performance of basic research is characterized by excellence and high intellectual merit. World-class basic research in all major areas of science is important for at least three reasons.
First, research discoveries often rely upon insights in many scientific areas. For example, mathematics, statistics, and computer sciences advance discoveries in other sciences, while the social sciences contribute to effective uses of other sciences, including the adoption of innovations. Research in different areas of science interrelates in the systems through which discoveries and resulting technologies and innovations benefit society. Truly transformative scientific discoveries often depend on research in a variety of fields, from which connections can be made that lead to new ideas.
Second, in today’s rapidly connected world, a discovery made somewhere is soon known everywhere. The competitive advantage may go not to the nation in which the discovery was made but to a nation that can leverage the productivity of follow-up research more effectively to pro-
duce commercially viable technologies, which ultimately drive domestic economic growth. Reaping these benefits from research discoveries throughout the world requires a highly sophisticated domestic research enterprise built on people, infrastructure, and funding. In particular, awareness of scientific discoveries may travel quickly, but sufficient understanding to extend them or to apply them for the development of new technologies or other innovations often requires that the nation’s researchers possess considerable fundamental knowledge derived from diverse basic research.
Third, cultivating a system of world-class basic research attracts students and scholars from around the world to the United States. Their contributions to the U.S. research enterprise enhance its world-class stature, thus creating a self-reinforcing cycle.
Moreover, as discussed earlier, an additional benefit of research—particularly basic research—is its contribution to scientific infrastructure. Basic research generates—and benefits from—new methods of observation, measurement, data collection, analysis, and experimentation, enabling the quality of research to improve continually and the extent of research to expand (see Box 6-4). These new methods are most likely to have beneficial effects on both basic and applied research when they are widely adopted (Darby and Zucker, 2003). One example is basic statistical research in experimental design, which has enabled well-designed experiments in agriculture, medicine, and engineering. Another example is the Sloan Digital Sky Survey, as described in Box 6-5.
The Importance of Partnerships
Partnerships can help the research system produce world-class basic research. Discoveries often arise from collaborations and partnerships among individuals with different training, experience, and perspectives, such as researchers in academia and in industry. University-industry partnerships often are the means by which industry invests in university research, and they provide opportunities for the commercialization of research discoveries. Effective partnerships strengthen scientific and technological research in both universities and industry; inventions and even manufacturing feed back into research, and vice versa. NSF’s Industry-University Cooperative Research Centers Program (discussed in Chapter 5) offers many successful examples, although evaluations of this program are at best incomplete. It should be noted that, while public-private partnerships are known to play a central role in the success of the American
BOX 6-4
Examples of How Scientific Infrastructure
Enables Research Progress
Instrumentation: Electron Microscopy
Developed in 1931 by German researchers who received the Nobel Prize in Physics more than half a century later, in 1986 (Bellis, 2013), the modern electron microscope can magnify objects millions of times and allow researchers to view atomic-scale detail. This instrumentation is now used in materials research, biological research, semiconductor research, and industrial research for applications as diverse as mineral analysis in mining, forensics, and three-dimensional tissue imaging for medical analysis.
Technologies: Nuclear Magnetic Resonance (NMR) Spectroscopy, Magnetic Resonance Imaging (MRI)
NMR helps determine a liquid or solid material’s molecular structure and identifies the various compounds that make up the material. The technology was discovered in the 1940s, in part as the result of a Massachusetts Institute of Technology (MIT) researcher’s experiences during World War II with the detection of radio frequency power and the absorption of that power by matter to produce radar. The technology is now used in a variety of applications, ranging from chemistry quality control, to molecular physics, to the study of crystals and noncrystalline materials, to natural gas exploration and recovery. The technology was a critical step in the discovery of MRI, which allows for the early detection of countless health conditions, including cancer, Alzheimer’s disease, multiple sclerosis, and stroke.
Observational Studies: General Social Survey
Initiated in 1972 by NORC at the University of Chicago, the General Social Survey (GSS) collects data on attitudes and behaviors in contemporary American society. These data allow scholars, students, and policy makers to draw comparisons between the United States and other societies. According to the NORC Website,* the GSS is the second most frequently analyzed source of data in the social sciences, after U.S. census data. The data are reported by journalists, considered by legislators and policy makers, and used as a major teaching tool in universities. According to the NORC Website, “More than 20,000 journal articles, books and Ph.D. dissertations are based on the GSS; and about 400,000 students use the GSS in their classes each year.”
_____________
*Available: http://www.norc.org/Research/Projects/Pages/general-social-survey.aspx [June 2014].
BOX 6-5
Big Data in Astronomy: The Sloan Digital Sky Survey
The Sloan Digital Sky Survey (SDSS)—named after the Alfred P. Sloan Foundation, which provided significant funding—is an ambitious astronomical survey that has been in progress since 2000 and will continue through 2014 (SDSS-I, 2000-2005; SDSS-II, 2005-2008, and SDSS III, 2008-2014). The systematic release of open-access data from the SDSS has accelerated the rate of findings and innovations in the field of astronomy. These datasets include spectra of 930,000 galaxies, 120,000 quasars, and 460,000 stars. The data are calibrated, checked for quality, and made available on an annual basis to researchers through online databases.
The various SDSS releases include a range of tutorials appropriate for audiences ranging from elementary school children to professional astronomers. The raw data also are available through other platforms, such as the National Aeronautics and Space Administration’s (NASA) World Wind Program. The availability of SDSS data has supported a vast range of scientific investigations by astronomers and other researchers around the world. “Half of these achievements were among the original ‘design goals’ of the SDSS, but the other half were either entirely unanticipated or not expected to be nearly as exciting or powerful as they turned out to be” (Sloan Digital Sky Survey, 2008). In hindsight, the release of these data appears to be an obvious approach; in 2000, however, this approach was questioned by many who thought that the public release of these data was neither important nor relevant.
research enterprise,2 additional studies are needed to fully characterize their role.
Measures for Assessing Basic Research
The ability to achieve world-class basic research can be tracked with international benchmarking of a nation’s leadership status by field of science. This qualitative metric was suggested by the NRC in 1993 (National Academy of Sciences, 1993) and tested with experimental panels that examined institutional and human resource factors influencing world leadership status in three areas of research (National Academy of Sciences, 2000).
International benchmarking makes it possible to track research performance, recognize niche areas in which each nation excels, and identify strengths and weaknesses as a means of improving the quality and impact
_____________
2See http://sites.nationalacademies.org/PGA/guirr/index.htm [August 2014].
of each nation’s research program. In its 2000 report, the NRC’s Committee on Science, Engineering, and Public Policy identifies eight factors predicted to have the greatest influence on the quality of future U.S. research performance relative to that of other nations: (1) the intellectual quality of researchers and the ability to attract talented researchers; (2) the ability to strengthen interdisciplinary research; (3) the ability to maintain strong, research-based graduate education; (4) the ability to maintain a strong technological infrastructure; (5) cooperation among the governmental, industrial, and academic sectors; (6) increased competition from Europe and other countries; (7) a shift in emphasis toward health maintenance organizations in clinical research; and (8) adequate funding and other resources (National Academy of Sciences, 2000). Measures focusing on these eight factors could help sustain the world-class quality of basic research as an essential pillar of the research system.
Research funding agencies strive to achieve world-class research by awarding competitive grants based on the caliber of the project personnel, the innovativeness of the research, and the strength of the research design, among other factors. Increasingly, however, agencies also consider the impact of the research.
Typical measures of world-class stature focus on outputs: publications, patents, citations, and other bibliometrics. Peer review also is used to judge research quality, as a supplement to quantitative measures. But there is nearly universal agreement that research excellence can be measured not only by outputs but also by inputs, such as the caliber of scientific talent, the quality of research facilities, a balanced national investment in research among fields and disciplines, the working environment, and how research is planned and managed. Measures of these inputs can be relatively easy to obtain (National Research Council, 2013b, 2013d).
Measures of research reproducibility could also help in assessing the world-class stature and long-term impacts of basic research. The use of independent laboratories and the implementation of journal or funding agency requirements that data be made available to other researchers have been suggested as ways to facilitate the reproducibility of research findings (Lehrer, 2010; Little, 2011; Siegfried, 2013; Wadman, 2013b).
Finally, measures of research performance could begin to capture the trends in international research performed by American companies, at least some of which may be directly tied to corporate research centers in the United States, as a means of facilitating the rapid translation of new knowledge into product or process innovations.
CONCLUDING OBSERVATIONS
Together, the three pillars described in this chapter interact to drive the performance of the research and innovation systems. But how best to support these pillars requires further research and the development of improved measures. Supporting these three pillars will lead to more cutting-edge research and stronger connections among world-class researchers, which in turn will allow the United States to attract even more talent and garner even more benefits for society and the economy.
