National Academies Press: OpenBook

Best Practices in Assessment of Research and Development Organizations (2012)

Chapter: Appendix J Assessment at the National Institute of Standards and Technology, Army Research Laboratory, and Sandia National Laboratories

« Previous: Appendix I Sample Crosscutting Assessments
Suggested Citation:"Appendix J Assessment at the National Institute of Standards and Technology, Army Research Laboratory, and Sandia National Laboratories." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations. Washington, DC: The National Academies Press. doi: 10.17226/13529.
×

Appendix J

Assessment at the National Institute of Standards and Technology, Army Research Laboratory, and Sandia National Laboratories

ASSESSMENT AT THE NATIONAL INSTITUTE OF STANDARDS
AND TECHNOLOGY

Ongoing assessments have been provided for more than half a century by the National Academies’ National Research Council (NRC) through contracts with the National Institute of Standards and Technology (NIST). The assessments are focused primarily on the technical and scientific quality of the scientific and technological research of the laboratories. Recognizing the broader mission of NIST, however, and in order to provide actionable feedback to management, the review panels have almost always received a multifaceted statement of task that typically has included adequacy of the facilities and/or budget and alignment with mission and/or desired impact.

The review process has been approached on a peer-to-peer basis, with the panel for each laboratory consisting of about 15 to 20 individuals, from academia, industry, and other scientific and engineering environments, divided into thematic subgroups of approximately 3 individuals aligned with laboratory divisions (generally about five or six divisions). NIST suggests membership for the panels, but the final selection is made by the NRC, reflecting expertise, balance, potential conflict of interest, diverse characteristics, and willingness to serve.

Each laboratory prepares and posts to a dedicated website review material and references to published papers and a set of projects for detailed review. Not all projects can be reviewed during each assessment, and so a negotiation determines which will be reviewed, primarily reflecting NIST management needs and concerns. Among the projects reviewed, each panel’s goal is to portray an overall impression of the laboratory. Generally, the subgroups meet with their NIST counterparts for a day, attending presentations, taking tours, and engaging in interactive sessions with laboratory staff and managers. Then the panel as a whole meets for another day and a half, mostly in closed session, to deliberate on findings and define the contents of the assessment report. Observations are written by the subgroups and then blended into a cohesive report from the panel as a whole.

Panels as technical experts in the areas being reviewed generally depend on the experience, technical knowledge, and expertise of their members. The Director of NIST provides a charge to the assessment panels, identifying the general aspects of the laboratories’ R&D that he wishes assessed (this may include, for example, the technical quality of the work, its impact, and the adequacy of supporting resources), but rigid metrics and criteria are not mandated for the review. The panels focus on evaluation of the quality of the research, the number of publications and quality of the journals, timeliness of the research, knowledge of relevant research being conducted elsewhere, and other metrics such as patents, presentations, professional committee service, extramural awards, and so on. Overall, however, the assessment is more accurately characterized as qualitative rather than quantitative.

Suggested Citation:"Appendix J Assessment at the National Institute of Standards and Technology, Army Research Laboratory, and Sandia National Laboratories." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations. Washington, DC: The National Academies Press. doi: 10.17226/13529.
×

ASSESSMENT AT THE ARMY RESEARCH LABORATORY

For nearly two decades, the Army Research Laboratory (ARL) has sponsored assessments by the National Research Council of the technical quality of the work at the laboratory’s six directorates, which perform research and development (R&D) in the areas of computational and information sciences, human research and engineering, sensors and electron devices, vehicle technology, weapons and materials research, and survivability/lethality analysis. Accordingly, the NRC appoints six panels of experts that provide annual assessments of the scientific and technical quality of the work at the ARL. These panels provide biennial reports that summarize their findings and recommendations related to the quality and appropriateness of R&D for each of ARL’s technical business areas. The reports are delivered to the Army sponsor and to Army and Department of Defense (DOD) stakeholders and are made available to the public.

The panels are charged to address the following items:

•  Is the scientific quality of the research of comparable technical quality to that executed in leading federal, university, and/or industrial laboratories both nationally and internationally?

•  Does the research program reflect a broad understanding of the underlying science and research conducted elsewhere?

•  Does the research employ the appropriate laboratory equipment and/or numerical models?

•  Are the qualifications of the research team compatible with the research challenge?

•  Are the facilities and laboratory equipment state of the art?

•  Does the research reflect an understanding of the Army’s requirement for the research or the analysis?

•  Are programs crafted to employ the appropriate mix of theory, computation, and experimentation?

In addition to the panel’s addressing of the items listed above, the NRC selects from the panel membership individuals who form ad hoc groups to respond to specific issues identified by the Director of the ARL. The assessments are conducted by annual panel visits to the ARL facilities and review of supporting documentation describing technical projects and programs, equipment and facilities, and staff backgrounds and characteristics. The interactions between the panels and the ARL staff include dialogue during presentations, tours, demonstrations, poster sessions, and other means of presenting the ARL technical work, during which panel members seek clarification of facts and additional contextual information relevant to the assessments. Dialogue also occurs at the end of each review meeting, and panel chairs often engage in follow-up discussions with ARL staff. The purpose of these dialogues is to provide the opportunity for panel members to gather (and for ARL staff to provide) accurate, substantive, relevant information and to clarify contextual factors that may relate to the impressions under formulation by the panels; the dialogues are important so that subsequent findings, conclusions, and recommendations in the biennial report are based on accurate facts and adequate understanding.

As can be discerned from the examples of NIST and ARL assessments, there are various forms of peer review, each tailored to the specific type of organization. Considerable advantage to the process can be gained from the continuity of peer committee membership over time to

Suggested Citation:"Appendix J Assessment at the National Institute of Standards and Technology, Army Research Laboratory, and Sandia National Laboratories." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations. Washington, DC: The National Academies Press. doi: 10.17226/13529.
×

allow for the assessment of change within the organization. Staggered appointments allow for some carryover of expertise in subsequent reviews.

ASSESSMENT AT THE SANDIA NATIONAL LABORATORIES

At a National Research Council workshop on best practices in assessment of research and development organizations on March 19, 2012 (a data-gathering workshop conducted under the auspices of the Panel on Review of Best Practices in Assessment of Research and Development Organizations), Dr. J. Stephen Rottler, Chief Technology Officer, Vice President, Science and Technology, at Sandia National Laboratories, described Sandia’s assessment processes. The following is taken from the summary of his presentation, titled “Assessing Sandia Research,” provided in the NRC workshop summary.1 Jordan and coauthors also provide a detailed description of the assessment methods employed at Sandia.2

Sandia National Laboratories has undergone a continuous evolution in the assessment of quality, relevance, and impact, with quantitative assessment evolving into qualitative assessment that is informed by data. Organizations are complex systems, composed of interconnected parts. The properties of the whole organization are not necessarily perceived by looking at individual parts. Systems behave in nonlinear ways that are difficult to predict. Assessors must probe, watch behavior, probe, and watch behavior, iteratively, noting that the assessment impacts behaviors. There has been a need to shift from quantitative to qualitative assessment informed by data.

Organizations that traditionally have been stovepiped are increasingly evolving strategies and funding approaches that acknowledge the importance of multidisciplinary research organizations.

At Sandia, there are three assessment categories: (1) Self-assessments try to be objective, but they are inherently limited. All successful organizations have mature self-assessments that are objective and that promote responsive behaviors. (2) External peer reviews and visiting committees (external advisory boards) are used to examine quality, relevance, impact, and responsiveness to customers. (3) Benchmarking compares the organization being assessed to other organizations and is accomplished by formal assessments (through the visiting of other organizations) and less formal interactions as well.

Self-assessment at Sandia has become increasingly more formal and disciplined. Quarterly assessments are opportunities for leaders to examine with their teams whether their expectations about quality, relevance, and impact are being met. These assessments are performed at all levels of management.

Independent assessments are performed through a research advisory board that meets twice a year. The board is composed of senior individuals drawn from across academia and the public sector. The board is used in a broad sense to assess technical quality using external measures and comparison against other organizations. The assessment examines whether Sandia is meeting the criteria for its roles as fast follower or first researcher. It also examines the health

____________________________

1 National Research Council, 2012. Best Practices in Assessment of Research and Development Organizations: Summary of a Workshop. The National Academies Press, Washington, D.C.

2 G. Jordan, P. Oelschaeger, A. Burns, R. Watkins, and T. Trucano, 2010. Description of the Sandia National Laboratories Science, Technology and Engineering Metrics Process. Sandia Report SAND2010-0388. Sandia National Laboratories, Albuquerque, N. Mex.

Suggested Citation:"Appendix J Assessment at the National Institute of Standards and Technology, Army Research Laboratory, and Sandia National Laboratories." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations. Washington, DC: The National Academies Press. doi: 10.17226/13529.
×

of the research environment and connections with internal and external customers. It elucidates what is working or what is getting in the way in terms of innovation. The board also meets with customers of the organization and examines the impacts of prior investments. It assesses whether investments have enabled the laboratories to continue fruitful work or to initiate new work. Laboratory Directed Research and Development (LDRD) funds are an important element of the laboratories. The laboratories’ director is permitted to decide how the LDRD funds are allocated across projects consistent with the laboratories’ mission. The National Nuclear Security Administration provides oversight for this program, which captures principal-investigator-generated ideas within the management context. The program includes five or six grand challenges projects; each of these larger projects has an assigned external advisory board. Historically, these larger projects have transitioned successfully to have impact within the laboratories or have achieved follow-on external funding—these impacts have been achieved with the help of the external advisory boards.

Assessments have been traditionally performed according to a balanced scorecard that guides the selection of data to support assessment decisions. Metrics are defined to assess three areas of measurement: value to customers, outputs, and inputs. Within each area, metrics are defined to support the assessment of what the organization is doing and how it is doing it. To assess value to customers, the value and impact in terms of leadership, stewardship, and mission satisfaction are addressed by examining measures of the effectiveness of strategic partnerships with industry and technical collaborators. To assess outputs, the excellence of scientific and technical advances is addressed within the context of management excellence, which involves measuring elements of the work environment and management assurance. To assess inputs, the capabilities of staff, technology infrastructure, and facilities are addressed by examining the science, technology, and engineering strategy through measurements of parameters indicative of the portfolio and the technical planning process.

The evolving assessment processes increasingly include the examination of qualitative factors informed by the quantitative data. The following elements are assessed: clarity, completeness, and alignment of the research strategy; alignment of the research with the organization’s missions; the quality and innovation of the research; the vitality of the organization’s scientists and engineers; and long- and short-term impacts of the research with respect to the organization’s missions and to advancing the frontiers of science and engineering.

In summary, successful organizations and their assessors strive to be clear about the purposes of the assessment and its context, carefully decide what data to collect and what the assessment framework is, and link the assessment to the organization’s concept of what makes a great organization.

Suggested Citation:"Appendix J Assessment at the National Institute of Standards and Technology, Army Research Laboratory, and Sandia National Laboratories." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations. Washington, DC: The National Academies Press. doi: 10.17226/13529.
×
Page 69
Suggested Citation:"Appendix J Assessment at the National Institute of Standards and Technology, Army Research Laboratory, and Sandia National Laboratories." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations. Washington, DC: The National Academies Press. doi: 10.17226/13529.
×
Page 70
Suggested Citation:"Appendix J Assessment at the National Institute of Standards and Technology, Army Research Laboratory, and Sandia National Laboratories." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations. Washington, DC: The National Academies Press. doi: 10.17226/13529.
×
Page 71
Suggested Citation:"Appendix J Assessment at the National Institute of Standards and Technology, Army Research Laboratory, and Sandia National Laboratories." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations. Washington, DC: The National Academies Press. doi: 10.17226/13529.
×
Page 72
Next: Appendix K Examples of Peer Review Conducted at Federal R&D Organizations »
Best Practices in Assessment of Research and Development Organizations Get This Book
×
Buy Paperback | $37.00 Buy Ebook | $29.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Research and development (R&D) organizations are operated by government, business, academe, and independent institutes. The success of their parent organizations is closely tied to the success of these R&D organizations. In this report, organizations refers to an organization that performs research and/or development activities (often a laboratory), and parent refers to the superordinate organization of which the R&D organization is a part. When the organization under discussion is formally labeled a laboratory, it is referred to as such. The question arises: How does one know whether an organization and its programs are achieving excellence in the best interests of its parent? Does the organization have an appropriate research staff, facilities, and equipment? Is it doing the right things at high levels of quality, relevance, and timeliness? Does it lead to successful new concepts, products, or processes that support the interests of its parent?

This report offers assessment guidelines for senior management of organizations and of their parents. The report lists the major principles of assessment, noting that details will vary from one organization to another. It provides sufficient information to inform the design of assessments, but it does not prescribe precisely how to perform them, because different techniques are needed for different types of organizations.

Best Practices in Assessment of Research and Development Organizations covers three key factors that underpin the success of an R&D organization: (1) the mission of the organization and its alignment with that of the parents; (2) the relevance and impact of the organization's work; and (3) the resources provided to the organization, beginning with a high-quality staff and management.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!