Chapter 1

NAKFI Origins and Evolution

“To launch a new field of inquiry like this, across all walks, with such major support is an incredible privilege.”

—Jody Deming, Member, National Academy of Sciences and NAKFI Challenge award recipient

The National Academies Keck Futures Initiative (NAKFI) was launched in 2003 with generous support from the W. M. Keck Foundation. It was a 15-year experiment to catalyze interdisciplinary research across the fields of science, engineering, and medicine. The prestige of the convening power of the National Academy of Sciences, the National Academy of Engineering, and the National Academy of Medicine coupled with a commitment to an evolving model informed by rigorous evaluation were essential to NAKFI’s success.

NEED AND CONTEXT

At the turn of the 21st century, the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine (referred to as the National Academies of Sciences, Engineering, and Medicine starting on July 1, 2015) were committed to leveraging the value of interdisciplinary research and collaboration that had been identified by major thought leaders, agencies, and institutions. The idea of interdisciplinary efforts was not new. Research achievements such as determining the structure of DNA, working out the movements of tectonic plates, mapping the

human genome, and many others were the result of cross-disciplinary collaborations. Major research agencies, including the National Science Foundation (NSF) and the National Institutes of Health (NIH), were beginning to focus on interdisciplinarity as a top priority. Recognition was growing that adjustments to institutional behavior and funding mechanisms were needed to embrace interdisciplinary research if major scientific challenges were to be effectively tackled. There was a realization that opportunities to start conversations across disciplines could lead to unique and valuable breakthroughs. However, despite increased support for interdisciplinary research and institutes, the parking lot remained the most likely location where the leading researchers in science, engineering, and health were encountering each other on university campuses.

The Futures Initiative arose out of a documented understanding of the systemic obstacles to conducting interdisciplinary research. It was clear that many factors both internal and external to the university environment played a role in whether interdisciplinary research was done and how well. The team developing the Futures Initiative believed that talent and individual perseverance would be acknowledged by organizations and funders if there were sufficient recognition and support along the way. While structural and interpersonal obstacles were impediments, many talented individuals, when motivated by a good question or an important problem, were finding ways to succeed.

The overarching goal of the Futures Initiative was to stimulate new modes of scientific inquiry and break down barriers to interdisciplinary research in academic and other research settings as well as in funding agencies and academic publishing. The goals set out in the original proposal implied an impact far beyond the program itself:

• Structural change in research funding, both public and private, to provide significantly increased support for interdisciplinary and cross-professional research;
• Structural change within research universities to foster more interdisciplinary research;
• Changes in the career paths of Futures Initiative participants resulting from greater interdisciplinary and cross-professional opportunities; and
• Increases in the opportunities for graduate student (PhD-level) research in interdisciplinary and cross-professional areas.

In 1999, the W. M. Keck Foundation hosted roundtable discussions of scientists from diverse disciplines to inform program directors about high-potential areas of research and the barriers to fulfilling that potential. The culmination of those conversations resulted in the Promising Directions report (W. M. Keck Foundation, 1999), which also informed the development of the proposal that created the NAKFI program (National Academies, 2002). Dr. Marvin Cassman, a member of the Keck Foundation’s roundtable discussions was acknowledged as identifying multiple shortfalls in advancing interdisciplinary research: lack of common cultures, language, and understanding among disciplines; the absence of meetings where the best talent across disciplines focused on the most important questions; a lack of seed grants to bridge the gap between new ideas and traditional funding sources; and a need to train and support researchers interested in interdisciplinary research.

The intent of the Futures Initiative was to “invest in the human capital of the research enterprise, by identifying the best talent and providing these individuals with the opportunity to explore the horizons of knowledge together” (National Academies, 2002, p. 9). Interdisciplinarity and boundary crossing were essential components of NAKFI and the approach would need to address identified challenges to working in these ways. The program included three components, the first two of which were informed by ongoing internal and external evaluation:

• Futures Conferences
• Futures Grants
• National Academies’ Communication Awards

The program would provide a prestigious forum for generating new connections and ideas, the type of high-level recognition from the National Academies that could move science, engineering, and medicine forward in exciting ways. The program would bring together researchers from various disciplines and career stages, proving opportunities for unexpected interactions. Seed grant funding amplified the collaborative conference experience allowing participants to turn conversations into projects.

FACILITATING AN INTERDISCIPLINARY RESEARCH CONSENSUS REPORT

During the first 18 months of the Keck Futures Initiative, the National Academies undertook a study on facilitating interdisciplinary research. The study examined the current scope of interdisciplinary efforts and provided recommendations as to how such research can be facilitated by funding organizations, scientific societies, and academic institutions.

According to report co-chairs Nancy C. Andreasen and Theodore L. Brown, “[Facilitating Interdisciplinary Research] is a ‘call to action’ for all those who perform, administer, support, and organize interdisciplinary research and training. Its purpose is to facilitate collaborative practices that can increase the productivity of science and engineering” (NAS et al., 2005, p. xii). Though most of the recommendations suggested “incremental” changes that would facilitate interdisciplinary research, one chapter was devoted to “transformative” changes for innovative institutions. The report was clear that the intent is not to privilege the pursuit of interdisciplinary inquiry over disciplinary research, but to acknowledge that structures and reward systems built for disciplinary scholarship may not be effective for institutions that seek to optimize and strengthen research that crosses disciplinary boundaries.

NAKFI’s working definition of “interdisciplinary research” was also informed by the report (see Box 1-1).

Facilitating Interdisciplinary Research continues to serve as a valuable resource for those striving to improve the climate for interdisciplinary and collaborative research. Published in 2005, it is in the top 1% of downloads of all National Academies publications. It is available through the National Academies Press website (www.nap.edu/catalog/11153).

GOVERNANCE AND ORGANIZATION

NAKFI required involving people with diverse expertise to design, manage, and implement activities of a complex, evolving program. Interdisciplinary boundary crossing was built into the program’s governance and oversight. The National Academies’ presidents provided general program oversight and steering committees were responsible for overseeing each year’s activities. Every conference steering committee included members elected to each of the three academies. Working groups and expert panels were appointed as necessary to accomplish specific projects and evaluation activities. Information on committee members can be found in Appendix I.

FUTURES CONFERENCE MODEL

Innovation is not predictable. It is possible, however, to identify compelling topics and provide the space, tools, and support for the most promising talent to address them. The Futures Conference Model is informed by nearly 15 years of experimentation and a comprehensive environmental scan of the current state of innovative research.

A hallmark of the Futures Conference Model is the selection of topics at the intersection of science, engineering, and medicine that could serve as the starting point for the trajectory of innovation and impact. The first three conference themes were included in the proposal submitted to the Keck Foundation. Subsequent themes were identified through a rigorous process involving input from the National Academies’ vast network. Themes were chosen based on their potential to stimulate innovation at the intersection of science, engineering, and medicine and NAKFI’s unique ability to make a contribution to the topic.

Prospective conference topics came from a variety of sources and were developed by staff with input from experts before being submitted to the oversight committee for selection. By continually refining the themes and questions during the planning process, Futures Conferences could incorporate emerging ideas that characterize nascent areas of research. This flexibil-

ity improved the timeliness, relevance, and impact of NAKFI’s cutting-edge themes. The few less effectively focused conferences prompted program improvements including guidelines to assist conference steering committees during the topic refinement process. For example, a committee might focus a broad theme by exploring it across different scales (nano to mega) or by breaking the topic down into subtopics. Alternatively, conference at-

tendees could be engaged prior to the conference to narrow the theme. As the program progressed, conference participants had more opportunities to refine the themes in real time during the conference.

The first conference in 2003, at which attendance was by invitation, used the familiar structure of moderated panel presentations interspersed with networking opportunities. Two regional conferences designed to stimulate campus, local, and regional research collaborations were held the following spring. This was an experiment to see if follow-up research occurred more frequently when participants lived and worked in closer geographic proximity.

The program evolved after the inaugural year. For subsequent conferences, a call for applications publicized the conference and the conference steering committee selected the attendees across relevant disciplines from academic, industry, and government laboratories. A list of all Futures Conference participants by year is available in Appendix II. Conference communications made clear that this was not an ordinary academic conference. Attendees would be expected to contribute their intellect and curiosity to tackle pressing, real-world challenges during a 3-day inquiry-driven conference. Beginning in 2009, a series of transdisciplinary orientation questions provided a tool for applicants to reflect on their interest in participating in NAKFI’s unique conference format. The committee also extended a small number of invitations to attendees whose perspectives would contribute to the richness of the conference (e.g., individuals working in funding agencies, university administration, foundations, or science media). The applicants selected by the committee always included participants who were diverse in a number of factors, including discipline, professional sector, career stage, gender, and geographic location.

Adding pre-conference tutorials helped bridge communication gaps and gave attendees a common language for their conversations at the conference. Pre-conference tutorials and engagement activities equipped participants to engage in deep conversations, transcend boundaries, and build bridges. NAKFI experimented with using a variety of pre-conference engagement strategies, including live bi-coastal webcasts; podcast interviews with researchers and well-known members of the scientific media; academic-style presentations; curated material from popular media such as TED Talks and NPR; and exercises to prepare attendees for the unfamiliar (and sometimes uncomfortable) conference process. Tutorials containing different levels of information were provided for each topic and attendees were encouraged to select their own pre-conference learning experience.

NAKFI paid all costs for participants, creating a micro-sabbatical during which curiosity, risk, and ingenuity were encouraged. Participants were assigned to small groups (referred to variously over time as “task groups,” “interdisciplinary research teams,” and “seed idea groups”) to consider hypothesis-driven challenges connected to the conference theme. During some conferences, each group explored a different question. In other years, the same questions were explored by several groups. In all cases each group was aided by one or more conference steering committee mentors and reported on its work midway through and at the end of the conference. In addition to working in these small groups, participants had time for informal conversations, networking, and in later years, “creative engagements” designed to generate unexpected ideas.

Even though attendees were assigned to groups based on their interests, NAKFI anticipated that not every group would work well together. Some attendees would find it difficult to deal with the lack of a specific goal or the absence of a designated facilitator. A dominant personality could leave little room for others’ views. These experiences mirror what can happen in longer-term interdisciplinary and cross-professional teams and informed NAKFI’s understanding of successful team collaboration. In the early years, attendees were encouraged to stay with their assigned group throughout the conference. Participant feedback led to rethinking this approach, and attendees in later years started in one group and could self-organize into new groups around emerging ideas midway through the conference. This evolution provided more space for exploration and self-direction without diluting the scientific content of the meeting. For the final Futures Conference, attendees generated their own questions around four themes and self-organized to explore the most promising topics.

NAKFI also experimented with the role of committee member mentors, who had varying levels of experience, skill, and comfort helping participants navigate group dynamics and connecting them to resources. Guidelines were developed to help committee members provide appropriate feedback about the teams’ emerging ideas. Mentors were encouraged to commit themselves to being open to solutions outside of their own disciplines. Even if participants began with a seed idea with which the mentor had great familiarity, at some point the idea would stray into unfamiliar territory. Mentors were cautioned to let go of any subconscious desire to steer participants toward a solution related to their own discipline, or down a path they felt comfortable traveling. Mentors’ contributions were to facilitate the development of ideas that were innovative, important to team

members, and had the potential for big impact. Like participants, mentors were asked to demonstrate the ability to think outside of their own discipline and be open to new ideas and perspectives.

For the first several conferences attendees gathered in plenary session midway through and at the end of the meeting for formal presentations from each group. In later years, mid-conference presentations were distilled into 1-minute verbal pitches given during breakfast. As groups were encouraged to “think beyond the PowerPoint” and develop messages that could be understood by a variety of audiences, final presentations included demonstration, audience participation, role-playing and dance.

The small group format encouraged co-inquiry, -discovery, and -creation throughout the meeting, and served as the basis for the conference publication. As one facet of NAKFI’s commitment to science communication, graduate students from science writing programs across the country were engaged to report on the work of the small groups. Students were nominated by their departments and prepared for the conference by reviewing the pre-conference tutorials and suggested reading and by selecting the small group in which they wanted to participate. Students worked with a mentor, the seed idea group to which they were assigned, and NAKFI staff to finalize their reports following the conference. These reports and a conference overview were compiled into a conference publication. In 2016, students from a design program participated in each group (see the “Futures Communications” section).

The group discussions remained confidential until after the Futures Grants were announced and posting content-related discussions to social media during the meeting was strongly discouraged. This provided protected time for participants to crystallize their ideas, apply for funding from NAKFI and other sources, or progress toward other next steps. Embargoed meeting content was released in the spring following each November conference. The National Academies Press published the conference summary and the final seed idea group presentations were posted to NAKFI’s website. In the last years of the program, NAKFI learned that some participants would have preferred that all conference discussions remain off the record indefinitely. The program did not have an opportunity to evaluate this suggestion with a broader pool of participants or to thoroughly consider its implications on the NAKFI experience and program outcomes.

Ideas from the conferences continued to flourish long after the final presentations and attendees returned home. Data showed that in the 3 months following the conference nearly 75% of participants reported inter-

acting with someone they did not know previously. The type and number of interactions varied and included one participant inviting another to speak on campus, working on a seed grant proposal, sharing unpublished research findings, conducting email conversations, and collaborating on publications.

Data suggest the conference model was an effective starting point that led to an abundance of possible outcomes, including:

• better questions that lead to new ways of thinking, collaborations, models, technologies, or research methods;
• increased interest in the conference theme and related outcomes;
• broad, diverse, and better prepared networks of interdisciplinary and cross-professional researchers, scholars, and practitioners;
• individual and team-based work on scientific, educational, social, and cultural projects;
• transformation of participants’ career paths, especially for early career participants;
• new streams of funding from agencies and other institutions;
• identification of high-quality participants, ideas, and projects for public engagement activities; and
• a better understanding of the support and tools necessary to stimulate innovation and public engagement.

Some of the most important outcomes came from serendipitous interactions sparked by an environment that encouraged people to explore their curiosity, take risks, have new experiences, and work together on challenges of shared interest. Many professional conferences and workshops accomplish some of these goals. What made NAKFI truly unique was the possibility of the participants receiving funding to act on the inspiration generated by the conference.

FUTURES GRANTS

Funding for regional conferences, which were found not to be as valuable as had been hoped, was reallocated to support additional Futures Grants to pursue collaborations and ideas ignited during the conference. NAKFI continued to respond to attendee feedback and incorporate relevant research throughout the duration of the program.

The Futures Grants provided seed funding to Futures Conference participants on a competitive basis to enable them to pursue important new ideas and connections stimulated by the conferences. These grants filled a critical missing link between bold new ideas and major federal funding programs. Futures Grants provided flexible funding enabling researchers to start developing a line of inquiry by supporting the recruitment of students and postdoctoral fellows, the purchase of equipment, and the acquisition of preliminary data, which in turn positioned them to compete for larger awards from other public and private sources.

Approximately $1 million in “seed venture grants” was awarded annually on a competitive basis to conference attendees at U.S.-based institutions and organizations. NAKFI did not assign an academic or a professional rank requirement on principal investigator eligibility, which provided an exceptional opportunity for junior investigators (whose institutions also did not have a rank constraint) to pursue projects about which they were personally passionate.

Futures Grants were typically awarded in the amounts of $25,000, $50,000, $75,000, and $100,000. This funding served as an incentive for participants to collaborate after the conference and provided resources for start-up projects. Futures Grants promoted and supported ingenuity, risk taking, and experimentation on research, cultural, social, and educational projects judged most promising to stimulate innovation on the conference theme. They fulfilled the need for investigators to pursue ideas that sprung from their own creativity rather than from the latest government funding announcement. The application process was straightforward and reporting requirements kept to a minimum. Applicants had already been vetted by NAKFI for attendance at the conference and the NAKFI conference steering committee looked for projects with the greatest potential to succeed. Grants were provided for up to a 2-year period and no-cost extensions were occasionally allowed. Grantees came together 1 year after receiving their funding to provide progress updates and share reflections on what was working or not working and why. NAKFI encouraged grantees to learn as they went and to make changes to their projects as appropriate.

Funding seed projects was similar to venture capital in that NAKFI invested in early stage, high potential projects that had a substantial element of risk. If a problem calls for a truly original response it is because no single individual can know in advance what that response should be. Otherwise, the solution would have already been described and attempted. NAKFI’s approach to seed funding for scientific research was different from venture capital in that it did not specify minimum returns. It is not always possible, nor advisable, to pre-define success. For NAKFI, project successes and noble failures, which were tracked through a rigorous evaluation process, were regarded as equally valuable. Grantees’ reflections on what worked, what did not work, and why informed organizational learning and funding strategies.

NAKFI employed a number of methods to understand the overall impact of the seed grants, including surveys, staff review of grantee final reports, and analyzing scientific articles produced from seed grant projects. Missing from the approach was a process for conducting reviews of the seed

TABLE 1-1 Futures Conference and Seed Grant Program Model

grant program by experts able to judge the scientific merit and potential impact of the projects. After 2 years of development, in 2011, a panel of experts conducted appraisals of the creativity, intellectual quality, disciplinary scope, integrative quality, and likely scientific and societal impacts of each portfolio of Futures Grants. Seed grant review working groups concluded that Futures Conferences and Grants filled an important gap in the landscape of scientific discourse and funding, yielding a variety of outputs and results.

The peer-review process revealed the importance of aligning grant application criteria with the selection, reporting, and evaluation processes. In some years it was difficult to fully assess the scientific and societal impact of grants because there was a misalignment among the application, reporting, and evaluation processes. This often required research to obtain missing information or requests for grantees to provide additional details. It is also difficult to predict the potential societal impact of a highly speculative research project. It may take years or decades for a promising technology to be ready for widespread use and even longer to measure its value. In the later years of the program, the grant application, reporting, and evaluation processes were fully aligned, providing data to support optimal evaluation of the Futures Grants program. Futures Grantees and their project descriptions at the time of the award are listed in Appendix III.

The Futures Grants program demonstrated potential for not only scientific, but also educational, societal, or cultural impacts. NAKFI’s seed grants resulted in more than $158 million in verified follow-on funding, 205 publications, and 6 patents, as well as other notable successes. NAKFI funding also supported the work of 94 graduate students (master’s and PhD) and 42 postdoctoral fellows. The full potential of these projects is only beginning to be realized and will continue to develop over time.

At the heart of NAKFI was a commitment to an evolving model. Opportunities surfaced each year to create a conference and a seed grant program that would meet the needs and interests of the participants while maintaining the key features that made NAKFI a transformational experience.

FUTURES COMMUNICATIONS

The National Academies’ Communication Awards were a part of the NAKFI program designed to recognize, promote, and encourage the effective communication of science, engineering, medicine, and interdisciplinary

work within and beyond the scientific community. Awards of $20,000 each were presented to individuals or teams who made outstanding contributions to the public’s understanding and appreciation of science, engineering, and/or medicine. From 2003 to 2007 awards were given in three categories: books, film/radio/television, and magazine/newspaper. An online category was added in 2008.

By the time of NAKFI’s 5-year review in 2008 the communication awards program had established itself as one of the largest awards in science writing and communication. Winners and finalists suggested that even more important than a substantial cash prize was the recognition, visibility, and encouragement the award provided. Consideration was given to modifying the program to include acknowledgment of finalists, thereby extending encouragement and recognition to more deserving teams. Beginning in 2010, winners and finalists were honored during a ceremony held at the Keck Center or National Academy of Sciences building in Washington, DC, and included videos highlighting the award winners, followed by a reception.

In 2018 a total of 364 nominations were received in 4 categories: 90 books; 58 film/radio/television; 91 magazine/newspaper; and 125 online. The large cash prize and recognition connected with receiving the award has made a meaningful difference in the careers of award recipients. Winning the book award in 2006 helped author Charles Mann fund the research and travel necessary to write his next book. At least twice winners have given their award money to the organizations where they work (a regional newspaper and a public radio station), providing crucial financial support. More recently, reporters at FiveThirtyEight, a site known for predicting election results, won in the online category for reporting on gun deaths in the United States. A member of that team donated his winnings to an organization dedicated to conducting research on gun violence.

NAKFI also cultivated future science writers by inviting graduate students from science writing programs across the country to attend and report on the conference. Students were nominated by the department director or designee and selected by program staff. They received mentoring and wrote summaries of each small group’s discussion, which were combined to create a conference publication.

In 2016 the conference model included a partnership with the Designmatters program of the ArtCenter College of Design in Pasadena, California. Designmatters students participated in the conference by sup-

porting visualization, ideation, and synthesis strategies and by documenting the processes and outcomes. Science writers floated among attendees and in and out of the groups to collect the stories that unfolded during the conference. Science writing and design students prepared for the conference by reviewing the pre-conference tutorials and suggested reading and by selecting a seed idea group in which they wanted to participate. They worked with expert mentors before, during, and after the conference to finalize their materials for publication. Material from design students and science writers formed the basis of the summary Discovering the Deep Blue Sea, which is available through the National Academies Press. Previous workshop summaries featured only the work of science writing students.

A list of the 2003–2018 Communication Awards committee members and recipients is available in Appendix IV.

PROGRAM EVALUATION FOR AN EVOLVING MODEL

NAKFI’s commitment to an evolving model for interdisciplinary research was paired with an evaluation approach adapted to the ongoing development of the program. Evaluation and routine environmental scans made it possible to continuously reflect on the development of the program and make data-based decisions for improvements. This approach, referred to as developmental evaluation, “facilitate[d] ongoing innovation by helping those engaged in innovation examine the effects of their actions … and test their hypotheses about how to foment the change in the face of uncertainty in situations characterized by complexity” (Patton, 2011). NAKFI’s approach was adaptable and responsive to the complex and dynamic environment in which it operated. The program sought to find new ways to understand and document its unique approach to fostering and harnessing innovation. An internal evaluator was embedded within the program and external evaluation expertise provided additional support with an outside perspective. In addition to routine and targeted program component evaluations, the program was reviewed after the first 5 years and again at the 10-year mark. The interaction between the program and the evaluation went both ways. The program adapted and changed to leverage what was learned through evaluation and the evaluation strategies, methods, and reporting evolved to support the needs of the program and stakeholders. See Appendix V for more information on the 5- and 10-year program reviews.

Surveys, Scanning, and Developing New Measures and Approaches

Surveys were used throughout the program to provide timely data and feedback from conference attendees and committee members about what worked and what needed improvement. NAKFI’s approach to gathering data from applicants and participants informed decision making and program improvement. Three surveys were conducted each year of conference applicants and conference participants. The survey of conference applicants asked about their application experience and collected data on applicants’ research activities and beliefs about collaboration. An immediate post-conference survey assessed attendees’ experiences and captured suggestions for improvements. The survey asked respondents to rate the effectiveness of each conference element toward stated goals and captured narratives of peak experiences that occurred during the meeting. The development of pre-conference tutorials was directly based on an early evaluation finding that attendees needed a common language with which to facilitate conversations at the conference. Assessing the delivery methods and effectiveness of the tutorials for various levels of attendee knowledge was an ongoing part of the program evaluation. A survey sent three months after the conference assessed attendees’ post-conference activities and collected reflections on the value of the experience after some time had passed. Some survey items were consistent each year allowing for comparisons across conferences. An important early evaluation finding that was consistent across all conferences was that those who applied to attend and attended Futures Conferences were already inclined toward interdisciplinary research before their experience with the program.

In 2006, the program surveyed various stakeholders to understand the climate for interdisciplinary research. Four key groups provided information on their policies and activities related to interdisciplinary research. Included were provosts from the American Association of Universities’ (AAU’s) member organizations and primarily undergraduate institutions (PUIs); editors of leading science, engineering, and medical journals; foundations providing funding for scientific research; and executive directors of professional associations of scientists, engineers, and medical doctors.

The survey of provosts was delivered by mail in July and August 2006 to all 61 members of the AAU and a random sample of 70 PUIs. The response rate was 61% for the AAU and 56% for the PUIs. In most areas, the AAU institutions reported a higher likelihood to engage in specific strategies supporting interdisciplinary research (see Table 1-2). Provosts surveyed for

TABLE 1-2 Summary Results from 2006 NAKFI Survey of AAU and PUI Provosts

the Facilitating Interdisciplinary Research report ranked promotion criteria among the top impediments to interdisciplinary research. The NAKFI survey results indicate that in 2006 only 21% of responding AAU institutions had explicit language in their promotion and tenure criteria for how to assign credit for interdisciplinary scholarship. A Consensus Study Report released in 2015, Enhancing the Effectiveness of Team Science, found most universities still lacked comprehensive criteria for evaluating individual contributions to team-based research projects (NRC, 2015).

NAKFI’s evaluation used multiple measures and complementary designs whenever feasible. The program’s early focus on bibliometric methods proved useful in documenting the number, degree of integration, and reach of publications produced by seed grantees (Porter et al., 2007). Bibliometric analysis is a methodology that uses journal articles as a data source. It is highly scalable, allowing for analysis of small and large groups of papers. Other organizations and researchers were using and continue to use these methods to assess the interdisciplinary nature of research products. The 2007 paper describing the bibliometric approach has been cited 133 times. NAKFI evaluators developed and tested a computed measure of scholarly integration based on the references in a published journal article (Porter, Roessner & Heberger, 2008). The intent was to develop benchmarks measuring the interdisciplinarity in scientific disciplines and compare the level of integration found in articles by Futures Conference participants or

seed grant recipients to the benchmark disciplines. This article has been cited 53 times. Bibliometric measures continued to be a component of the program’s evaluation approach and were used in assessing the impact of Futures Grants.

More information on the bibliometric methods and how they were used by NAKFI is provided in Appendix VI.

In 2008, the American Journal of Preventive Medicine released a special supplement focusing on the science of team science. The volume drew from the proceedings of a National Cancer Institute (NCI)–National Institutes of Health (NIH) Conference on the Science of Team Science held in 2005. Work was being done to evaluate the interdisciplinary qualities of research grant proposals developed by scientists participating in the NCI’s Transdisciplinary Research on Energetics and Cancer (TREC). The NCI-TREC Written Products Protocol was adapted from an earlier version of the instrument first developed to gauge the interdisciplinary qualities of doctoral dissertations and undergraduate research fellowship term papers.

In 2009, the NAKFI Oversight Committee approved forming an expert interdisciplinary working group to conduct reviews of the NAKFI seed grants awarded following the Genomics (2005) and Smart Prosthetics (2006) annual conferences. The plan was to:

1. Develop a NAKFI Written Products Protocol (WPP) to be used by expert peer reviewers to derive reliable appraisals of the scientific originality, generativity, cross-disciplinary scope, integrative quality, and likely scientific and societal impacts of Futures Grants projects;
2. Analyze the characteristics, scientific outcomes, and impacts associated with Futures Grants; and
3. Identify markers of the most successful Futures Grants.

The new approach was implemented for the first time in 2011 and was used to evaluate six rounds of seed grants. The NAKFI Seed Grant Review Working Groups were comprised of a small, dedicated group of individuals with an affinity for working in an interdisciplinary environment. They had familiarity with the NAKFI program, experience attending Futures Conferences, and a willingness to participate in an exploratory and potentially exciting process. Each working group was trained on the NAKFI WPP, which assessed projects on multiple dimensions, including multiple facets of interdisciplinarity, scientific impact, societal impact, creativity,

and overall quality. The protocol also asked reviewers to identify outputs from each project and conceptual links to the topics explored at the Futures Conference the investigators attended. This activity built on the cumulative knowledge base generated by prior evaluations of the NAKFI program. At the same time, it developed new evaluative criteria for gauging the scientific impacts of products generated by NAKFI’s grantees.

Five-Year Review

The NAKFI Five-Year Review Panel was charged with assessing the extent to which the Keck Futures Initiative was progressing toward achieving the program’s goals. The panel met in 2007 and 2008 and delivered its report to the oversight committee in July 2008. A list of panel members is provided in Appendix II.

As part of its charge the panel reviewed the founding assumptions of the NAKFI program, including the original objective to achieve structural change in research funding, research institutions, and the career paths of investigators. The panel reviewed program and evaluation data and relevant literature and interviewed several participants. While the climate for interdisciplinary inquiry appeared more favorable than anticipated at the time the NAKFI program was originally proposed, there remained obstacles to interdisciplinary research both at the institutional and investigator levels that merited continuation of the program strategies. The need for a program like NAKFI remained.

The panel concluded that the original objectives of the NAKFI program (e.g., driving fundamental change in funding agencies, universities, and careers) would be difficult to assess in view of the rapid growth in interdisciplinary research, even during the brief period during which the NAKFI proposal was developed and implemented. The study and subsequent report on facilitating interdisciplinary research had informed and encouraged interest in interdisciplinary research but the impact of such a report on government agencies, institutions, and individual scientists would be impossible to ascertain. Furthermore, NAKFI’s own survey of leaders in academia, funding agencies, professional associations, and publishing found that minimal progress had been made to reduce the barriers to interdisciplinary research identified in the Facilitating Interdisciplinary Research report. Perhaps more importantly, it was apparent that the program had developed some powerful new approaches for organizing interdisciplinary conferences and stimulated important new research strategies and activities

by individuals involved in these experiences. Indeed, many individuals that the panel heard from believed that the nature of NAKFI as an experiment to examine, stimulate, and promote interdisciplinary research in exciting areas was more important than its capacity to drive institutional change.

Based on its review the panel suggested that goals for impacting organizations across the nation would be difficult to achieve through a strategy that focused primarily on individual researchers. Furthermore, two factors made a systematic investigation of the program’s institutional impact infeasible. First, the scale of the NAKFI program was dwarfed by the research expenditures of U.S. academic and funding institutions. Shifts in academic and federal funding would make greater and more far-reaching changes than NAKFI could accomplish. Like a small boat in a big ocean, the NAKFI program was more likely to be impacted by waves of change emanating from federal funding agencies than it was to shift the tides. Additionally, NAKFI had developed new models for interdisciplinary conferences that were not in use by others. Identifying a control group that could be used in an experimental evaluation would be difficult. A quasi-experimental design would have been cumbersome and costly. Rather than pursue an experimental approach to evaluation, the panel suggested adding narrative evaluation of participant experiences to the program’s survey research.

A high-level summary of the panel’s conclusions addressed the climate for interdisciplinary research, success of NAKFI’s conference and seed grants, and overall value of the program. Specifically, the panel concluded:

• Interdisciplinary and cross-professional research was increasingly well funded and encouraged by both research sponsors and academic institutions. While the climate for interdisciplinary inquiry appeared considerably more favorable than anticipated at the time the NAKFI program was originally proposed, there remained obstacles to interdisciplinary research both at the institutional and investigator levels that merited continuation of the NAKFI program.
• Futures Conferences and Grants had evolved into an unusually powerful framework for drawing together investigators from widely diverse disciplines and exploring important scientific topics requiring an interdisciplinary approach. This had been due in large part to the evolving structure of the conferences, careful organization (e.g., drawing on the experience of previous conferences), pre-conference preparation (e.g., tutorials, reading lists), provocative task group topics, the quality of participants, and the

• exceptionally strong support and credibility provided by the National Academies.
• The NAKFI program had already been of immense value not only in stimulating important conversations on key interdisciplinary topics but also in fostering innovative research in several areas. It has also developed important new paradigms for the consideration and conduct of interdisciplinary research.

The panel’s overall recommendation was to continue the program with its current structure and philosophy.

Key programmatic recommendations included striving for greater diversity of women, traditionally underrepresented groups, and younger scientists; expanding the role of social and behavioral scientists; and exploring the expansion of the Futures Conference process by using new communications technologies, collaborating with other programs to increase interdisciplinary research, and continuing to target topics with the potential for identifying strategies for transformative research. The impact of the recommendations of the 5-year review panel resulted in changes to the program, including

• Creation of an online network to encourage alumni engagement
• Broadening the scope of conference topics to include social and behavioral scientists
• Expanded promotion of the call for conference applications to facilitate the selection of a diverse group of participants
• Connecting with other programs supporting interdisciplinary research
• Separating the communication awards from the conference
• Adding mid-cycle grant meetings for grantees and their collaborators
• Refocusing the evaluation on the scientific and societal impact of seed grants

Information about the 5-year and 10-year reviews is available in Appendix VI.

Ten-Year Evaluation and Planning Panel

The NAKFI Ten-Year Evaluation and Planning Panel was charged with reviewing NAKFI’s progress toward achieving the program’s goals and de-

veloping recommendations for the next generation of a NAKFI program. The panel met in 2013 and 2014 and delivered its report to the oversight committee in July 2014.

Working with the panel, the program examined what it had accomplished since the 5-year review and conducted a scan of how the environment for interdisciplinary, transformative, and innovative research had changed. In early 2014, the NAKFI network was used to gather ideas on what would make research more innovative. Those ideas were sorted into conceptually similar groups to create a conceptual model for more innovative research that included familiar areas: institutional support, responsive funding, ingenuity and risk taking, incentives, education and training, and data accessibility. NAKFI alumni then rated a set of more than 90 ideas on two scales: importance and the degree to which each idea is commonly practiced. Interestingly no areas were rated as highly important and commonly practiced (see Figure 1-1). Participants’ rankings were consistent across fields (science, engineering, and medicine), sectors (academia versus

non-academia), ages, and genders. More details on the 10-year review are provided in Appendix V.

Many of the most important areas identified during the 10-year review were leverage points NAKFI had already successfully engaged.

• The program directly addressed the core issue of interdisciplinary collaborations by drawing together researchers across science, engineering, and medicine, as well as members of media, government organizations, foundations, and the general public by providing them the space, time, and funding needed to explore and act on new solutions to intractable intellectual challenges.
• For graduate students and postdoctoral scholars their participation provided an invaluable opportunity for engagement with interdisciplinary trailblazers across the career spectrum and if involved in a Futures Grant, direct experience conducting interdisciplinary research.
• According to the NAKFI alumni who participated in the 10-year review, “more funding” was not the only answer to the question of how to support more innovative research. A variety of funding mechanisms were called for, including support for proof-of-concept studies, small awards that do not require preliminary data, and longer-term funding for especially promising young investigators. NAKFI’s flexible seed grant model was succeeding in these areas.

Some of the areas identified were not ones that NAKFI or the National Academies could address directly. For example, NAKFI was unlikely to be able to influence the policies and infrastructure support of other institutions. However, the National Academies could play a role in providing consensus-based recommendations to higher education around innovative promotion and tenure guidelines that support and facilitate cross-disciplinary research collaboration. Areas that scored low on the importance/practice scales—collaborations and engagement—might be more important to advancing innovative research than how they were ranked at the time of the NAKFI Ten-Year Review.

NAKFI developed a program model and also a collection of methods and tools for measuring and monitoring the impact of interdisciplinary research, team science initiatives, and venture science funding. The NAKFI

program and its evaluation evolved and developed in tandem, each piece informing and shaping the other.

NAKFI CHALLENGE COMPETITION: THREE SEEDS FOR THE FUTURE

As the program concluded, the National Academies announced the NAKFI Challenge, a competition to continue the essential work of crossing boundaries in the spirit of collaboration and scientific advancement. Open only to NAKFI alumni, the NAKFI Challenge awarded three projects in the amount of $500,000 each in 2018.

These projects carry forward NAKFI’s work beyond its 15 years as an activity of the National Academies. The call for proposals (open November 2017 to March 2018) generated 78 applications. Applications underwent a round of peer-to-peer community judging by fellow applicants. The 30 highest scoring proposals were then judged by an expert panel consisting of members of NAKFI conference organizing committees. The three winners were chosen by the presidents of the National Academy of Sciences, the National Academy of Engineering, and the National Academy of Medicine.

The recipients, with NAKFI alumni in bold, and their project titles are:

Jody Deming, University of Washington

* Fiscal agent for grant, Djerassi Resident Artists Program

Daniel Kohn, Kohnworkshop

Heather R. Spence, Marine and Bioacoustics Programs, Michelle’s Earth Foundation (Global Research and Arts Center for the Investigation and Advancement of Sustainability Solutions)

Jonathan Berger, Stanford University

Timothy J. Broderick, Institute for Human and Machine Cognition

Margot H. Knight, Djerassi Residents Artists Program

Timothy W. Weaver, University of Denver

Ocean Memory: A New Cross-Disciplinary Approach to Global-Scale Challenges

Memory involves the recall of events, pruned and processed from countless recordings by neural networks and thereby shaping future behavior. The ocean and its inhabitants hold memories of events throughout the evolution of the planet, awaiting our cognition. This proposal established a

thriving community exploring and expressing ocean memory, a new line of highly evocative scientific inquiry, aiming for a sea change in our ability to address challenges of the Anthropocene. The approach builds on NAKFI best practices, spanning disciplines required to address agents of memory and adding novel elements of distributed interactive spaces and grants for cross-disciplinary mentoring.

Bonnie Keeler, Humphrey School of Public Affairs, University of Minnesota

Jessica Hellmann and Fred Rose, Institute on the Environment, University of Minnesota

Elena Bennett, McGill University

Peter Kareiva, University of California, Los Angeles

Lydia Olander, Nicholas Institute for Environmental Policy Solutions, Duke University

Institutionalizing Interdisciplinarity: A Cross-Institutional Network to Synthesize What Is Working (and Not) in the Pursuit of Transformative Sustainability Science

Interdisciplinary research is necessary to tackle urgent societal challenges, yet institutional barriers hinder these efforts. Conventional systems reward individual work over collaboration and conformity over risk taking, disincentivizing engagement with “real-world” problems. While universities have invested in institutes, graduate programs, and other cross-cutting mechanisms, they remain uncoordinated experiments with limited contribution to systemic change. This project aims to change that by creating a cross-university network to review and synthesize three central challenges to institutionalizing interdisciplinarity: measuring impact, supporting students, and fostering co-development. The models will be drawn from sustainability science, an emerging field that spans natural and social sciences.

Kentaro Toyama and Mustafa Naseem, University of Michigan School of Information

Melissa Densmore, University of Cape Town

Agha Ali Raza, Information Technology University

Digital Street Theater for Global Maternal and Child Health Education

High or rising rates of maternal and child death are an urgent global issue—shockingly, including the United States. Causes differ by country, but mothers’ health knowledge and habits play a key role in reducing mortality in all contexts. Mortality rates, however, are highest among low-literacy families; those who most need health education are least equipped to absorb it from mainstream sources. This proposal combines old traditions with new technology—street-theater health skits shared via locally available digital channels—in three countries (Pakistan, South Africa, and the United States) so as to identify globally generalizable solutions.

Two proposed projects were named as honorable mentions:

Robert Froemke, New York University School of Medicine

Maryam Sanechi, University of Southern California

Jonathan Viventi, Duke University

Smart Neuroprosthetics: Brain–Machine Interfaces for the 21st Century

The goal of this project is to build an interface between the human brain and electronic systems. Currently, all devices that interface with the body use passive, individually wired electrodes, connected to remote electronics. The proposal aims to develop a new generation of brain–machine interfaces that use active, flexible electronics to interface with the brain at 1,000× higher spatial resolution than available today. Functional capacity will be tested in two domains: sensory restoration with hearing aids and recovery of motor control after paralysis or spinal cord injury. This technology opens an entirely new window into understanding brain function and building truly high-performance brain–machine interfaces.

Jose L. Contreras-Vidal and Dario Robleto, University of Houston

Juliet King, Indiana University School of Medicine and Herron School of Art and Design

Aesthetic Neurotherapeutics: Toward a Safe, Effective, and Noninvasive Arts Prescription (ArtRx) Program to Treat Physical, Neurological, and Mental Disability

An interdisciplinary team of engineers, clinicians, and art professionals have joined together with the goal of developing an innovative closed-loop system that treats physical, neurological, and mental health disorders by noninvasively modulating brain activity via precise and intelligent aesthetic (music, visual, dance) stimulation (ArtRx). The ArtRx team will validate the concept of aesthetic stimuli as powerful neurotherapeutics via personalized aesthetic-driven modulation of brain activity. This will be achieved by testing the brain’s response to a range of aesthetic “drivers,” including visual art, music, and expressive movement (dance), to quantify the dose–response and neuromodular preference to creative arts therapy interventions.

THE NAKFI LEGACY

Over the past 15 years NAKFI has promoted interdisciplinary research and collaboration through its annual conferences and seed grant program. It challenged the boundaries between disciplinary silos by providing opportunities, support, and tools for the best researchers to move fluidly across disciplines, fields, and sectors. Through these strategies, the program achieved the National Academies’ vision to advance research across the scientific spectrum while developing a model for innovation that occupies a unique space in the landscape of efforts to support innovative, cross-disciplinary research. Some of the brightest minds across the sciences, engineering, medicine, and arts and design disciplines have participated in deep conversations and developed bold approaches to the real-world challenges of our day.

Much of the world has changed since NAKFI’s inception but its impact will continue to resonate. During its 15 years it has inspired a diverse network of more than 2,000 researchers, artists, designers, students, teachers, university administrators, representatives of funding institutions, the scientific media, and disciplinary societies. In this same time, the network was supported with $14.6 million in seed funding to innovate along the scientific spectrum: HIV prevention, the evolution of language, forecasting epileptic seizures, the liquidity of financial markets, tactile sensors for robotic hands, and citizen science projects to address complex socio-ecological issues facing Gulf Coast communities and ecosystems. It has inspired its diverse network to “think big” at the frontiers of science, engineering, and

medicine. In addition to these successes is the creation of a program that is road tested and ripe for replication. As noted by the NAKFI Ten-Year Evaluation and Planning Panel, “one of the most important contributions NAKFI has made to advancing interdisciplinary cross-professional research is the program model itself.”

The legacy of the Futures Initiative is the program’s adaptive, rigorous, and original approach to fostering collaboration. The propagation of the Futures Model—through the winners of the NAKFI Challenge and those who initiate new efforts based on the learnings of this unique program—will support efforts to address the changes taking place in the conduct of science, engineering, and medicine, and in the scope, breadth, and interconnected nature of pressing issues facing the nation and the world. The profound implications of these dynamics create new imperatives to challenge the traditional boundaries between scientists and non-scientists, competition and collaboration, process and outcome, inquiry and knowledge, and art/design and science through strategies that engage the global brain power and spark the human curiosity necessary to forge new paths of discovery at the most exciting frontiers. The vision and support of the W. M. Keck Foundation and the National Academies launched the Futures Initiative and the enthusiastic support of its program participants contributed to its success and the creation of its legacy. With the successful completion of its 15-year mission and the publication of this volume, NAKFI’s legacy is available to address the challenges of today and tomorrow.