Appendix F
Request for Data from ARPA-E, Input from Other Offices at the Department of Energy, and Agendas for Committee’s Public Information Gathering Sessions
This appendix contains three collections of information. The first is the committee’s formal request for specific data and information from ARPA-E to make this evaluation possible. The second is written responses from the Department of Energy’s Office of Fossil Energy answering questions regarding that office’s coordination with ARPA-E and adaption of any ARPA-E practices. The third collection is all of the agendas for public information gathering sessions the committee held during the course of this study.
September 1, 2015
To:
Ellen Williams, Director
Advanced Research Projects Agency-Energy
U.S. Department of Energy Washington, D.C.
From:
Pradeep Khosla
Chair of the Committee for the Evaluation of ARPA-E
Dear Director Williams:
Thank you for providing the National Academies of Sciences, Engineering, and Medicine Committee for the Evaluation of ARPA-E with the important information contained in your initial presentations and in the separately provided documents. The Committee appreciates all of the effort and care that you and your staff have already put into this project.
As stated during its last meeting in July 2015, the Committee requires additional information and data in order to fulfill its mandate from Congress to conduct an independent evaluation of the appropriateness and effectiveness of ARPA-E’s structure to achieve its mission and the most significant technical accomplishments of the program. We would like to accept your offer to provide additional information and data.
Some of the information the Committee needs includes program and project descriptions that you have already provided or is available through the ARPA-E website. I wish to verify that the Committee has the complete set of information. The Committee and I also recognize that you and your staff are working to complete an internal self-appraisal of the accomplishments and impacts of ARPA-E and plan to provide a presentation at our next meeting in October 2015.
Listed below is the Committee’s initial request for additional information. The committee may wish to obtain additional information through the course of its work. In obtaining needed information and data, the Committee and I wish to work with ARPA-E in any way we can – such as by utilizing independent consultants or NAS staff – to minimize the burden on your agency’s limited resources while still maintaining our independence. We also want to ensure that there is no compromise of ARPA-E’s necessary confidentiality or other legal obligations while fulfilling the Academies’ Congressional mandate to conduct an external evaluation. I note that many of these information sources were made available for the Government Accountability Office’s review of ARPA-E that was completed in 2012.
To complete its work, the Committee requires the information and data detailed below. I respectfully request your assistance in helping them obtain these items.
Definitions
The Committee would like to know how ARPA-E:
- Defines ‘success’ and ‘failure’ for both programs and projects; and
- Captures and propagates lessons learned from program or project ‘successes’ and ‘failures’ institutionally and across program directors and, where appropriate, to other offices and programs within the Department of Energy.
Program Awards, Metrics, and Milestones
The Committee respectfully requests:
- Description of Project Performers – we would like to verify that we have complete descriptions on the full portfolio of both open and focused programs and projects, including information to distinguish whether this is a seedling research effort intended to help develop new focused programs or a “regular” project intended to, eventually, produce a specific commercializable device, the description and location of the performer, and the funding level;
- Access to ARPA-E internal program documents for all focus programs and broad area announcements to date including official Funding Opportunity Announcements, submitted projects, and selection processes;
- For each full application received, detailed project and PI information (e.g., lead-PI, Co-PI and other associated staff and personnel) whether the project was funded or not, including information used to make a fund/don’t-fund decision (such as applicant scores);
- For each project funded, a detailed description of milestones and metrics and individual projects’ performance against those milestones and metrics;
- A list of instances where project milestones were revised during the course of the project, including descriptions of why ARPA-E and the performer changed the milestones and how those changes impacted the project;
- A list of projects restructured, particularly in response to revising milestones;
- A list of projects terminated, including descriptions of why and how ARPA-E and the performers terminated the projects; and
- A description of the process for terminating or restructuring, including:
- An explanation of how ARPA-E decides it must discuss restructuring or termination with a project performer; and
- Whether and how input from outside experts (at dedicated ARPA-E focus program events or the ARPA-E summit, for example) is utilized when making a decision to restructure or terminate a project.
Project Outputs and Intermediate Outcomes
The Committee respectfully requests:
- Copies of final project reports for all completed projects;
- Lists of journal and other scientific publications derived from ARPA-E funded projects;
- Lists of patents that derive from ARPA-E funded projects;
- Data on commercialized outcomes, revenues generated, and follow-on funding acquired by ARPA-E grantees/awardees;
- Data tracked by ARPA-E on performers following completion of individual projects; and
- Graduate students and postdocs funded by ARPA-E programs, and their career trajectories (e.g., academics, industries, startups, or government).
Collaborations and Communities of Interest
The Committee is interested in ARPA-E’s work in developing new communities and respectfully requests:
- ARPA-E’s description of “new communities” that have been built or enhanced as a result of ARPA-E programs, and evidence to support ARPA-E’s role and contribution in forming those communities;
- A description of how these new communities support or have contributed the goals of ARPA-E;
- Programs and agendas from past and planned ARPA-E pre-program workshops, program kickoff meetings, program annual meetings, and annual innovation summits;
- Lists of attendees present at ARPA-E pre-program workshops, program kickoff meetings, program annual meetings, and annual innovation summits;
- Lists of potential collaborations that have derived from, and can be traced to, ARPA-E meetings;
- Proposals (funded/nonfunded) that show evidence of collaboration; and
- Lists of technologies and/or commercialized outcomes, revenues, proposals funded by ARPA-E or other entities, prizes awarded, and submissions by collaborators or communities resulting from or linked to ARPA-E organized meetings.
Names and Points of Contact
To facilitate contact with necessary individuals and organizations, the Committee respectfully requests:
- List of ARPA-E program directors for consultations or to participate in focus groups;
- Résumés of past and present program directors and deputy directors, as well as personnel within ARPA-E; and
- List of ARPA-E stakeholders to interview including other federal programs or industries that benefit from ARPA-E funded research.
I would appreciate your best efforts to provide the requested information by October 15, 2015. The Committee understands that several of the items above may require further discussion and that some will require careful attention and proper handling to protect confidential information. As I mentioned earlier, the Committee and I stand ready to work with ARPA-E to ensure that protection and note that the National Academies has policies and procedures in place to work with such information, providing the protection that ARPA-E and its applicants require.
I look forward to your response.
Sincerely,
Pradeep Khosla, Chair
Committee for the Evaluation of ARPA-E
Questions from members of the National Academies Committee on the Evaluation of ARPA-E for
Christopher Smith, Assistant Secretary for Fossil Energy, U.S. DOE
- Based on your experience how would you describe ARPA-E’s specific niche and mission and how does ARPA-E’s mission relate to DOE’s mission and mandate?
ARPA-E’s mission is an important one for the DOE, and covers important space that the applied program offices do not (e.g., gas-to-liquids conversion).
- Are there aspects of the ARPA-E operations that can (or should) be transferred to DOE? If so, which aspects are transferable? No.
- Has the Office of Fossil Energy adopted operations and procedures from ARPA-E such as the use of funding opportunity announcements (FOA), the methods for awarding of contracts, or the project reporting requirements?
No. FE has a well-established FOA process, including workshops in advance of new initiatives or programmatic efforts (akin to ARPA-E’s approach). These are administered by NETL for FE, which has a large, dedicated staff with deep experience in solicitations, awards, and contracting.
- Does ARPA-E coordinate with or engage with Office of Fossil Energy? In particular, ARPA-E worked on projects related to carbon capture and sequestration that they describe as being followed-on in work done within your office. To what extent was there coordination between ARPA-E and the Office of Fossil Energy on the carbon capture and sequestration projects?
Coordination has been poor and partial. Despite several face-to-face discussions between FE and ARPA-E, and interest from both sides in several topics (e.g., advanced carbon capture technology or water treatment and reuse) we have not yet converged. As such, we have had no joint workshops, no joint solicitations, no coordinated launches, and no cooperative roadmapping. While there is benefit in such activities, to date they have not begun.
- We also note that ARPA-E has funded projects in other areas that might be of interest to your office, such as the advanced laser mechanical drilling technologies of Foro Energy. Do you have any
thoughts on how well other ARPA-E projects were developed or the results disseminated to the Office of Fossil Energy?
There are several areas for ARPA-E’s consideration which are currently outside of FE’s formal missions space and which would be of great interest to FE. These include:
- Advanced technology for drilling and completions
- Advanced technology for water treatment and reuse
- CO2 mineralization and polymerization technology
- CO2 capture and separation from the atmosphere (potentially including direct air capture, application of biochar, and afforestation).
FE carries technology through to maturation and commercialization. This is carried in different programs in different ways:
- Use-Inspired basic programs (Pascal’s quadrant), lie in the applied R&D programs of Coal and Oil and Gas. Core R&D in geoscience and Discovery of Carbon Capture Substances (DOCCS) are examples, and fund both National Labs and universities.
- TRL maturation and development lie in those programs as well. These include our conventional granting programs, and commonly involve a mix of academic, industrial, and National Lab partnerships. These grants commonly are designed to move a technology 1-2 TRL levels forward, heading towards market. IP is managed in these efforts, and is commonly held and matured by industrial channel partners.
- FE supports scale-up and demonstration explicitly. These include small pilots (~1 MW), large pilots (10-50 MW), and commercial demonstrations. Almost all of these pilots and demos are done with industrial partners aimed at market deployment.
- Finally, FE has supported and continues to support R&D at commercial sites to assess technical performance in a commercial setting. These include efforts such as the National Carbon Capture Center and the Regional Carbon Sequestration Partnerships.
In this context, we do not see ARPA-E’s efforts as similar to FE’s but rather as complementary. They reach a technology ecosystem different from FE’s, often including more SME’s and a different set of practitioners.
AGENDA
Evaluation of ARPA-E Mission and Goals
1st Meeting
May 28, 2015
Room 206
Keck Center, 500 Fifth Street, NW
Washington, DC 20001
MAY 28, 2015
OPEN SESSION – Open to the Public
12:30 PM | Congressional History and Intent of Evaluation |
Hon. Bart Gordon, Chair of House Committee on Science and Technology (former), original sponsor of America COMPETES Act of 2007 |
|
Christopher J. King, Staff House Committee on Science and Technology (former) |
|
Adam Rosenberg, Democratic Staff Director, House Committee on Science, Space, and Technology, Energy Subcommittee |
|
Mark Marin, Republican Staff Director, House Committee on Science, Space, and Technology, Energy Subcommittee |
|
Emily Domenech, Republican Staff, House Committee on Science, Space, and Technology, Energy Subcommittee |
|
Lindsay Garcia, General Counsel, Office of Senator Lamar Alexander |
|
John Rivard, Legislative Fellow, Office of Senator Lamar Alexander |
|
Franz Wuerfmannsdobler, Senior Energy Policy Advisor, Office of Senator Coons |
|
Allison Schwier, Congressional Science Fellow, Office of Senator Coons |
|
1:15 PM | Sponsor Presentation I: An Overview of ARPA-E |
Hon. Ellen Williams, ARPA-E Director |
|
2:15 PM | Sponsor Presentation II: ARPA-E Operations |
Shane Kosinski, ARPA-E Deputy Director for Operations |
3:15 PM | Break |
3:30 PM | Sponsor Presentation III: ARPA-E Programs |
Eric Rohlfing, ARPA-E Deputy Director for Technology |
|
4:30 PM | Sponsor Presentation IV: ARPA-E Tech to Market Activities |
Dave Henshall, ARPA-E Acting Deputy Director for Commercialization |
|
5:30 PM | Break |
5:45 PM | Establishing ARPA-E |
Hon. Arun Majumdar, first ARPA-E Director |
|
6:30 PM | Adjourn Open Session. Enter Closed Session |
AGENDA
Evaluation of ARPA-E Mission and Goals
2nd Meeting
July 28-29, 2015
Room 206
Keck Center, 500 Fifth Street, NW
Washington, DC 20001
JULY 28, 2015
OPEN SESSION – Open to the Public
8:55 AM | Welcome and Introduction |
Pradeep Khosla, NAS Committee Chair |
|
9:00 AM |
Franklin (Lynn) Orr, Undersecretary for Science and Energy |
10:00 AM |
Arati Prabhakar, DARPA Director |
11:00 AM | Break – End of Open Session for July 28, 2015, Enter Closed Session |
JULY 29, 2015
OPEN SESSION – Open to the Public
8:25 AM | Welcome and Introduction |
Pradeep Khosla, NRC Panel Chair |
|
8:30 AM | Opening Comments |
Hon. Ellen Williams, ARPA-E Director |
|
8:50 AM | Program Directors &Tech to Market Program presentation on GENI (Green Electricity Network Integration) |
Tim Heidel & Josh Gould, ARPA-E |
9:35 AM | Program Directors & Tech to Market Program presentation on AMPED (Advanced Management and Protection of Energy Storage Devices) |
Pat McGrath & Ryan Umstattd, ARPA-E |
|
10:20 AM | Break |
10:35 AM | PD&T2M Program presentation on METALS (Modern Electro/Thermochemical Advances in Light Metals Systems) |
James Klausner, ARPA-E |
|
11:20 AM | Program Directors &Tech to Market Project presentation on Baldor Electric Co.’s REACT (Rare Earth Alternatives in Critical Technologies) project |
Pat McGrath & Kacy Gerst, ARPA-E |
|
11:40 AM | Program Directors &Tech to Market Project presentation on Arizona State University’s FOCUS (Full-Spectrum Optimized Conversion and Utilization of Sunlight) project |
Eric Schiff & James Zahler, ARPA-E |
|
12:00 PM | Additional Q&As and Concluding Remarks |
12:30 PM | Adjourn Open Session, Enter Closed Session |
AGENDA
Evaluation of ARPA-E Mission and Goals
3rd Meeting
October 29-30, 2015
Room 103
Keck Center, 500 Fifth Street, NW
Washington, DC 20001
OCTOBER 29, 2015
OPEN SESSION – Open to the Public
1:00 PM | Welcome and Introduction |
Pradeep Khosla, NAS Committee Chair |
|
1:05 PM | Assessment of ARPA-E’s Impacts and Tracking of Future Outcomes |
Hon. Ellen Williams, ARPA-E Director |
|
2:35 PM | Perspectives from ARPA-E Alumnus - Sonic Joule, LLC |
Robert Keolian, President, Sonic Joule LLC |
|
3:20 PM | Break |
3:35 PM | Perspectives from ARPA-E Alumnus – ForoEnergy |
Joel Moxley, Founder & EVP Business Development |
|
4:20 PM | GAO Staff – Results and Methodology used in 2012 GAO evaluation of ARPA-E |
GAO Staff Analyst |
|
5:10 PM | Adjourn Open Session, Enter Closed Session |
AGENDA
Evaluation of ARPA-E Mission and Goals
4th Meeting
December 8-9, 2015
Board Room
Beckman Center, 100 Academy Drive
Irvine, CA 92617
DECEMBER 8, 2015
OPEN SESSION – Open to the Public
10:00 AM | Perspectives from ARPA-E Alumnus Program Director |
Hon. David Danielson, Assistant Secretary for Energy Efficiency and Renewable Energy |
|
11:00 AM | Adjourn Open Session |
CLOSED SESSION – BECKMAN CENTER BOARD ROOM
11:15 AM | Enter closed session |
OPEN SESSION – Open to the Public
1:00 PM | Welcome and Introduction |
Pradeep Khosla, NAS Committee Chair |
|
1:05 PM | Perspectives of the former ARPA-E Director |
Arun Majumdar |
|
2:05 PM | Perspectives from ARPA-E Alumnus Program Directors |
Ilan Gur, Jonathan Burbaum, Howard Branz, Dane Boysen |
|
2:45 PM | Q&A and Discussion with ARPA-E Alumnus Program Directors |
Ilan Gur, Jonathan Burbaum, Howard Branz, Dane Boysen |
|
3:15 PM | Break |
3:30 PM | Perspective on Government Agencies for Technology Innovation |
David Mowery and Shane Greenstein |
|
4:30 PM | Adjourn Open Session |
AGENDA
Evaluation of ARPA-E Mission and Goals
5th Meeting
February 29–March 1, 2016
FEBRUARY 29, 2016
OPEN SESSION
3:00 PM | Committee members visit ARPA-E Technology Innovation Summit |
7:00 PM | Depart Summit and Adjourn Open Session, Enter Closed Session |
This page intentionally left blank.