The National Institute of Standards and Technology (NIST)—recognizing that information and insights gained through continual examination of practices for organizational assessment are useful for decision makers at organizations across the federal, industrial, academic, and national laboratory sectors—recently requested that the National Research Council (NRC) organize a panel to review best practices in assessment of research and development (R&D) organizations.1 In response, the NRC established the Panel for Review of Best Practices in Assessment of Research and Development Organizations. The panel was charged to consider means of assessing the following, in a manner that satisfies the requirements of NIST to perform effective assessments but also identifies assessment methods that can be applied selectively to other R&D organizations:
• Technical merit and quality of the science and engineering work
• The adequacy of the resources available to support high-quality work
• The effectiveness of the agency’s delivery of the services and products required to fulfill its goals and mission and to address the needs of its customers
• The degree to which the agency’s current and planned R&D portfolio supports its mission
• The elements of technical management that affect the quality of the work
• The extent to which the agency is accomplishing the impact it intends
• The agency’s flexibility to respond to changing economic, political, social, and technological contexts
As one means of data gathering, among others that the panel is performing toward development of a final report of its findings, the panel organized a planning committee for a workshop on best practices in assessment of R&D organizations. The workshop was conducted at the Keck Center of the National Academies in Washington, D.C., on March 19, 2012.
1Appendixes A, B, C, and D in this report present, respectively, the agenda of the workshop, a list of the attendees, and biographical sketches for the planning committee and panel members and for those who made presentations at the workshop.
This report has been prepared by the workshop rapporteur as a factual summary of what occurred at the workshop. The planning committee’s role was limited to planning and convening the workshop. The views contained in the report are those of individual workshop participants and do not necessarily represent the views of all workshop participants, the planning committee, or the National Research Council.
The workshop addressed the broad subject of best practices in assessment, with a focus on elucidating two key aspects of organizational assessment: (1) evaluation of the technical quality of an organization’s R&D work and (2) assessment of the effectiveness of the organization in addressing its mission and the needs of customers and stakeholders. Appreciating the importance of individual differences across organizations, the committee set as a desideratum for the workshop the identification of a variety of assessment questions and methods for addressing them, which might then constitute a tool kit of assessment questions and methods that could be tailored for application by individual organizations.
During the morning session of the workshop, six distinguished individuals each provided a presentation. The collective expertise of the presenters, listed below in the order of their presentations, spans the management of R&D activities within congressional, federal, industrial, and academic environments (see their biographical sketches in Appendix D). The audience of workshop participants consisted of approximately 100 representatives of organizations within those sectors (see Appendix B for the list of participants).
• James H. Turner, Counsel and Director of Energy Programs at the Association of Public and Land-grant Universities, and former chief counsel to the U.S. House of Representatives Committee on Science and Technology;
• John C. Sommerer, Head, Space Sector, and Johns Hopkins University Gilman Scholar, Johns Hopkins University Applied Physics Laboratory;
• J. Stephen Rottler, Chief Technology Officer and Vice President for Science and Technology, Sandia National Laboratories;
• William F. Banholzer, Executive Vice President and Chief Technology Officer, The Dow Chemical Company;
• Roy Levin, Distinguished Engineer and Managing Director, Microsoft Research, Silicon Valley; and
• Gilbert F. Decker, Consultant, former Assistant Secretary of the Army for Research, Development and Acquisition.
During the afternoon session of the workshop, attendees formed seven separate groups, six of which included one of the morning presenters. Each group discussed best practices in assessment, with attendees sharing their insights and experiences. At the conclusion of the workshop, a rapporteur from each of the groups provided a brief oral summary of the group’s discussion.
McGee, director of the NRC’s Laboratory Assessments Board, summarizes each of the six presenters’ talks. In Chapter 3, the questions raised by the discussion groups are organized according to elements of the statement of task of the Panel for Review of Best Practices in Assessment of Research and Development Organizations.