ASSESSMENT AT THE NATIONAL INSTITUTE OF STANDARDS
AND TECHNOLOGY
Ongoing assessments have been provided for more than half a century by the National Academies’ National Research Council (NRC) through contracts with the National Institute of Standards and Technology (NIST). The assessments are focused primarily on the technical and scientific quality of the scientific and technological research of the laboratories. Recognizing the broader mission of NIST, however, and in order to provide actionable feedback to management, the review panels have almost always received a multifaceted statement of task that typically has included adequacy of the facilities and/or budget and alignment with mission and/or desired impact.
The review process has been approached on a peer-to-peer basis, with the panel for each laboratory consisting of about 15 to 20 individuals, from academia, industry, and other scientific and engineering environments, divided into thematic subgroups of approximately 3 individuals aligned with laboratory divisions (generally about five or six divisions). NIST suggests membership for the panels, but the final selection is made by the NRC, reflecting expertise, balance, potential conflict of interest, diverse characteristics, and willingness to serve.
Each laboratory prepares and posts to a dedicated website review material and references to published papers and a set of projects for detailed review. Not all projects can be reviewed during each assessment, and so a negotiation determines which will be reviewed, primarily reflecting NIST management needs and concerns. Among the projects reviewed, each panel’s goal is to portray an overall impression of the laboratory. Generally, the subgroups meet with their NIST counterparts for a day, attending presentations, taking tours, and engaging in interactive sessions with laboratory staff and managers. Then the panel as a whole meets for another day and a half, mostly in closed session, to deliberate on findings and define the contents of the assessment report. Observations are written by the subgroups and then blended into a cohesive report from the panel as a whole.
Panels as technical experts in the areas being reviewed generally depend on the experience, technical knowledge, and expertise of their members. The Director of NIST provides a charge to the assessment panels, identifying the general aspects of the laboratories’ R&D that he wishes assessed (this may include, for example, the technical quality of the work, its impact, and the adequacy of supporting resources), but rigid metrics and criteria are not mandated for the review. The panels focus on evaluation of the quality of the research, the number of publications and quality of the journals, timeliness of the research, knowledge of relevant research being conducted elsewhere, and other metrics such as patents, presentations, professional committee service, extramural awards, and so on. Overall, however, the assessment is more accurately characterized as qualitative rather than quantitative.
ASSESSMENT AT THE ARMY RESEARCH LABORATORY
For nearly two decades, the Army Research Laboratory (ARL) has sponsored assessments by the National Research Council of the technical quality of the work at the laboratory’s six directorates, which perform research and development (R&D) in the areas of computational and information sciences, human research and engineering, sensors and electron devices, vehicle technology, weapons and materials research, and survivability/lethality analysis. Accordingly, the NRC appoints six panels of experts that provide annual assessments of the scientific and technical quality of the work at the ARL. These panels provide biennial reports that summarize their findings and recommendations related to the quality and appropriateness of R&D for each of ARL’s technical business areas. The reports are delivered to the Army sponsor and to Army and Department of Defense (DOD) stakeholders and are made available to the public.
The panels are charged to address the following items:
• Is the scientific quality of the research of comparable technical quality to that executed in leading federal, university, and/or industrial laboratories both nationally and internationally?
• Does the research program reflect a broad understanding of the underlying science and research conducted elsewhere?
• Does the research employ the appropriate laboratory equipment and/or numerical models?
• Are the qualifications of the research team compatible with the research challenge?
• Are the facilities and laboratory equipment state of the art?
• Does the research reflect an understanding of the Army’s requirement for the research or the analysis?
• Are programs crafted to employ the appropriate mix of theory, computation, and experimentation?
In addition to the panel’s addressing of the items listed above, the NRC selects from the panel membership individuals who form ad hoc groups to respond to specific issues identified by the Director of the ARL. The assessments are conducted by annual panel visits to the ARL facilities and review of supporting documentation describing technical projects and programs, equipment and facilities, and staff backgrounds and characteristics. The interactions between the panels and the ARL staff include dialogue during presentations, tours, demonstrations, poster sessions, and other means of presenting the ARL technical work, during which panel members seek clarification of facts and additional contextual information relevant to the assessments. Dialogue also occurs at the end of each review meeting, and panel chairs often engage in follow-up discussions with ARL staff. The purpose of these dialogues is to provide the opportunity for panel members to gather (and for ARL staff to provide) accurate, substantive, relevant information and to clarify contextual factors that may relate to the impressions under formulation by the panels; the dialogues are important so that subsequent findings, conclusions, and recommendations in the biennial report are based on accurate facts and adequate understanding.
As can be discerned from the examples of NIST and ARL assessments, there are various forms of peer review, each tailored to the specific type of organization. Considerable advantage to the process can be gained from the continuity of peer committee membership over time to
allow for the assessment of change within the organization. Staggered appointments allow for some carryover of expertise in subsequent reviews.
ASSESSMENT AT THE SANDIA NATIONAL LABORATORIES
At a National Research Council workshop on best practices in assessment of research and development organizations on March 19, 2012 (a data-gathering workshop conducted under the auspices of the Panel on Review of Best Practices in Assessment of Research and Development Organizations), Dr. J. Stephen Rottler, Chief Technology Officer, Vice President, Science and Technology, at Sandia National Laboratories, described Sandia’s assessment processes. The following is taken from the summary of his presentation, titled “Assessing Sandia Research,” provided in the NRC workshop summary.1 Jordan and coauthors also provide a detailed description of the assessment methods employed at Sandia.2
Sandia National Laboratories has undergone a continuous evolution in the assessment of quality, relevance, and impact, with quantitative assessment evolving into qualitative assessment that is informed by data. Organizations are complex systems, composed of interconnected parts. The properties of the whole organization are not necessarily perceived by looking at individual parts. Systems behave in nonlinear ways that are difficult to predict. Assessors must probe, watch behavior, probe, and watch behavior, iteratively, noting that the assessment impacts behaviors. There has been a need to shift from quantitative to qualitative assessment informed by data.
Organizations that traditionally have been stovepiped are increasingly evolving strategies and funding approaches that acknowledge the importance of multidisciplinary research organizations.
At Sandia, there are three assessment categories: (1) Self-assessments try to be objective, but they are inherently limited. All successful organizations have mature self-assessments that are objective and that promote responsive behaviors. (2) External peer reviews and visiting committees (external advisory boards) are used to examine quality, relevance, impact, and responsiveness to customers. (3) Benchmarking compares the organization being assessed to other organizations and is accomplished by formal assessments (through the visiting of other organizations) and less formal interactions as well.
Self-assessment at Sandia has become increasingly more formal and disciplined. Quarterly assessments are opportunities for leaders to examine with their teams whether their expectations about quality, relevance, and impact are being met. These assessments are performed at all levels of management.
Independent assessments are performed through a research advisory board that meets twice a year. The board is composed of senior individuals drawn from across academia and the public sector. The board is used in a broad sense to assess technical quality using external measures and comparison against other organizations. The assessment examines whether Sandia is meeting the criteria for its roles as fast follower or first researcher. It also examines the health
____________________________
1 National Research Council, 2012. Best Practices in Assessment of Research and Development Organizations: Summary of a Workshop. The National Academies Press, Washington, D.C.
2 G. Jordan, P. Oelschaeger, A. Burns, R. Watkins, and T. Trucano, 2010. Description of the Sandia National Laboratories Science, Technology and Engineering Metrics Process. Sandia Report SAND2010-0388. Sandia National Laboratories, Albuquerque, N. Mex.
of the research environment and connections with internal and external customers. It elucidates what is working or what is getting in the way in terms of innovation. The board also meets with customers of the organization and examines the impacts of prior investments. It assesses whether investments have enabled the laboratories to continue fruitful work or to initiate new work. Laboratory Directed Research and Development (LDRD) funds are an important element of the laboratories. The laboratories’ director is permitted to decide how the LDRD funds are allocated across projects consistent with the laboratories’ mission. The National Nuclear Security Administration provides oversight for this program, which captures principal-investigator-generated ideas within the management context. The program includes five or six grand challenges projects; each of these larger projects has an assigned external advisory board. Historically, these larger projects have transitioned successfully to have impact within the laboratories or have achieved follow-on external funding—these impacts have been achieved with the help of the external advisory boards.
Assessments have been traditionally performed according to a balanced scorecard that guides the selection of data to support assessment decisions. Metrics are defined to assess three areas of measurement: value to customers, outputs, and inputs. Within each area, metrics are defined to support the assessment of what the organization is doing and how it is doing it. To assess value to customers, the value and impact in terms of leadership, stewardship, and mission satisfaction are addressed by examining measures of the effectiveness of strategic partnerships with industry and technical collaborators. To assess outputs, the excellence of scientific and technical advances is addressed within the context of management excellence, which involves measuring elements of the work environment and management assurance. To assess inputs, the capabilities of staff, technology infrastructure, and facilities are addressed by examining the science, technology, and engineering strategy through measurements of parameters indicative of the portfolio and the technical planning process.
The evolving assessment processes increasingly include the examination of qualitative factors informed by the quantitative data. The following elements are assessed: clarity, completeness, and alignment of the research strategy; alignment of the research with the organization’s missions; the quality and innovation of the research; the vitality of the organization’s scientists and engineers; and long- and short-term impacts of the research with respect to the organization’s missions and to advancing the frontiers of science and engineering.
In summary, successful organizations and their assessors strive to be clear about the purposes of the assessment and its context, carefully decide what data to collect and what the assessment framework is, and link the assessment to the organization’s concept of what makes a great organization.