National Academies Press: OpenBook
« Previous: Summary
Suggested Citation:"1 Introduction." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations. Washington, DC: The National Academies Press. doi: 10.17226/13529.
×

1

Introduction

Research and development (R&D) organizations are operated by government, business, academe, and independent institutes. The success of their parent organizations is closely tied to the success of these R&D organizations. In this report, organization refers to an organization that performs research and/or development activities (often a laboratory), and parent refers to the superordinate organization of which the R&D organization is a part. Where the organization under discussion is formally labeled a laboratory, it is referred to as such. The questions arise: How does one know whether the organization and its programs are achieving excellence in the best interests of its parent organization? Does it have an appropriate research staff, facilities, and equipment? Is it doing the right things at high levels of quality, relevance, and timeliness? Does it lead to successful new concepts, products, or processes that support its mission and the interests of its parent?

How does one assess a research organization to answer these questions? This report offers assessment guidelines to senior management of R&D organizations and of their parent organizations and other stakeholders. The report lists the major principles of assessment, noting that details will vary from one organization to another. The report provides sufficient information to inform the design of assessments, but it does not attempt to prescribe precisely how to perform assessments.

CALL FOR THE STUDY BY THE NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY

The National Institute of Standards and Technology (NIST), a major, multiprogram government organization, promotes U.S. innovation and industrial competitiveness by advancing measurement science, standards, and technology in ways that enhance economic security and improve the quality of life for all Americans. Over its history there has always been at NIST the belief that a strong peer assessment of its programs is a critical aspect of ensuring that NIST is effectively fulfilling its mission. This assessment has traditionally been provided by the National Research Council (NRC) of the National Academies. Since 1959, the NRC has provided assessments of the technical quality of the scientific and technical work performed at NIST (formerly the National Bureau of Standards) laboratories.

In October 1, 2010, NIST implemented its first major reorganization in over two decades. The primary objective of the reorganization was to sharpen the focus of NIST’s programs on their respective missions and to optimize their ability to deliver both the cutting-edge research and the related services critical to the U.S. economy. The most notable impact of the NIST reorganization is its effect on the NIST laboratory programs: The reorganization shifted the NIST laboratories from being activity or discipline-based organizations to being mission- and outcome-oriented organizations. The reorganization condensed the number of NIST laboratories and focused them along the distinct NIST mission lines of metrology, technology, and the provision of unique user facilities. This reorganization prompted the need for a reevaluation of the NIST

Suggested Citation:"1 Introduction." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations. Washington, DC: The National Academies Press. doi: 10.17226/13529.
×

processes for organizational assessment, to ensure that NIST is positioned to have the benefit of relevant feedback relating to all of its programs and mission areas.

NIST’s Visiting Committee on Advanced Technology (VCAT), an external advisory board, highlighted in 2010 the need for NIST to reevaluate its assessment and review processes. The VCAT 2010 Annual Report contained the following recommendation relevant to program assessment:1

Following the reorganization of NIST into mission-focused laboratories, NIST should develop a comprehensive assessment program that includes effective peer review of scientific quality, [and] customer satisfaction for measurement services and for effectiveness in meeting the needs of the particular measurement program.

STATEMENT OF TASK

In 2011, NIST charged the National Research Council to conduct a study of best practices in assessing R&D organizations and to prepare a report providing NIST with information on peer-review and performance-evaluation systems. The NRC formed the Panel for Review of Best Practices in Assessment of Research and Development Organizations to carry out this study. The Panel was charged by the Director of NIST to consider means of assessing specific organizational aspects in a manner that satisfies the requirements of NIST to perform effective assessments but also identifies assessment methods that can be applied selectively to other R&D organizations. The organizational aspects are:

•     Technical merit and quality of the science and engineering work,

•     The adequacy of the resources available to support high-quality work,

•     The effectiveness of the agency’s delivery of the services and products required to fulfill its goals and mission and to address the needs of its customers,

•    The degree to which the agency’s current and planned R&D portfolio supports its mission,

•    The elements of technical management that affect the quality of the work,

•  The extent to which the agency is accomplishing the impact it intends, and

•  The agency’s flexibility to respond to changing economic, political, social, and technological contexts.

In this report, an R&D organization is readily understood to consist of a group of scientists, engineers, and support staff, with appropriate facilities and equipment, working to accomplish some stated mission goal(s). An R&D organization could be the Department of Defense (DOD) laboratories system, or it could be a single laboratory, or even a directorate, division, or group within a laboratory. It could be the entire array of Department of Energy (DOE) National Laboratories or any one of these. It could be a laboratory within an industrial organization. It could be an organized research center at a university, or a research center involving collaboration among several universities and industries. Even more broadly, an R&D

____________________________

1 Visiting Committee on Advanced Technology, 2010. 2010 Annual Report of the Visiting Committee on Advanced Technology of the National Institute of Standards and Technology. U.S. Department of Commerce, Washington D.C., p. 7.

Suggested Citation:"1 Introduction." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations. Washington, DC: The National Academies Press. doi: 10.17226/13529.
×

organization might include such organizations as the National Science Foundation or the DOE funding offices.

THE PANEL’S STUDY PROCESS

To address the study charge, the NRC appointed 12 expert volunteers to the panel. These volunteers, whose biographical sketches appear in Appendix A, represent expertise across research, development, and management activities in the following organizational sectors: the federal government, industry, academia, and national laboratories. Several members of the panel had previously engaged in activities as members of NRC committees that assessed R&D activities at NIST and at other federal agencies.

The panel gathered inputs from relevant literature and from formal discussions at panel meetings with representatives of the following organizations: NIST, the Defense Laboratory Office, the Office of Science and Technology Policy, the Office of the Director of National Intelligence, and the IBM Corporation. On March 19, 2012, the panel steered a workshop at the Keck Center of the National Academies in Washington, D.C., at which representatives from the following organizations provided formal presentations on assessment practices: Johns Hopkins University Applied Physics Laboratory, the Association of Public and Land Grant Universities, former congressional staff, Sandia National Laboratories, The Dow Chemical Company, and Microsoft Research; a presentation was also made by the former Assistant Secretary of the Army for Research, Development, and Acquisition. The workshop also garnered inputs through small group discussions with approximately 100 representatives from organizations of the federal government, industry, academia, and national laboratories. A summary of the workshop was published in September 2012.2

The panel conducted four meetings at which it considered the literature reviewed, the presentations made and the discussions conducted, and information derived from the extensive personal experience and expertise of the panel members, and the panel drafted this report summarizing their findings. The report underwent rigorous review by a committee of experts appointed by the National Research Council. When that review committee expressed satisfaction with the responses of the Panel to their review comments, and when the report was subsequently approved by the National Research Council, it was made available to the public.

ORGANIZATION OF THE REPORT

This report is organized to reflect the elements in the panel’s statement of task. The body of the report consists of the Summary and six chapters. This first chapter describes the formation and study process of the panel. Chapter 2 presents issues that formed the framework within which the panel examined assessment practices. Chapters 3, 4, and 5 present assessment considerations applicable to the three foci of assessment considered fundamental by the panel: management practices, the technical quality of the R&D, and the impact of the R&D. Chapter 6 presents a summary of guidelines for consideration during an assessment.

The report also includes 12 appendixes. In addition to the biographical sketches of the panel members (Appendix A), they present the following: a summary of a workshop presentation highlighting the importance of assessing the alignment between an organization’s vision and its

____________________________

2 National Research Council, 2012. Best Practices in Assessment of Research and Development Organizations—Summary of a Workshop. The National Academies Press, Washington, D.C.

Suggested Citation:"1 Introduction." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations. Washington, DC: The National Academies Press. doi: 10.17226/13529.
×

people (Appendix B); a discussion of considerations pertaining to the validation of assessments (Appendix C); an example of the effective application of peer advice during the planning phase of a research project at the Army Research Laboratory (ARL; Appendix D); a discussion of relevant statutes and requirements documents that may be involved in some types of assessments (Appendix E); a description of the process applied by the ARL to assess the relevance of its R&D (Appendix F); a discussion of stakeholder relationships for laboratories at the DOD, DOE, and NIST (Appendix G); a list of questions pertaining to the assessment of leadership and management (Appendix H); examples of recent cross-organizational assessments (Appendix I); a summary of assessment processes at NIST, ARL, and Sandia National Laboratories (Appendix J); a summary of assessment processes at selected other government organizations involving peer review of technical quality (Appendix K); and a set of assessment metrics and criteria applied by NRC panels that review the ARL.

Suggested Citation:"1 Introduction." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations. Washington, DC: The National Academies Press. doi: 10.17226/13529.
×
Page 11
Suggested Citation:"1 Introduction." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations. Washington, DC: The National Academies Press. doi: 10.17226/13529.
×
Page 12
Suggested Citation:"1 Introduction." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations. Washington, DC: The National Academies Press. doi: 10.17226/13529.
×
Page 13
Suggested Citation:"1 Introduction." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations. Washington, DC: The National Academies Press. doi: 10.17226/13529.
×
Page 14
Next: 2 Framework for Examination of Assessment Processes »
Best Practices in Assessment of Research and Development Organizations Get This Book
×
Buy Paperback | $37.00 Buy Ebook | $29.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Research and development (R&D) organizations are operated by government, business, academe, and independent institutes. The success of their parent organizations is closely tied to the success of these R&D organizations. In this report, organizations refers to an organization that performs research and/or development activities (often a laboratory), and parent refers to the superordinate organization of which the R&D organization is a part. When the organization under discussion is formally labeled a laboratory, it is referred to as such. The question arises: How does one know whether an organization and its programs are achieving excellence in the best interests of its parent? Does the organization have an appropriate research staff, facilities, and equipment? Is it doing the right things at high levels of quality, relevance, and timeliness? Does it lead to successful new concepts, products, or processes that support the interests of its parent?

This report offers assessment guidelines for senior management of organizations and of their parents. The report lists the major principles of assessment, noting that details will vary from one organization to another. It provides sufficient information to inform the design of assessments, but it does not prescribe precisely how to perform them, because different techniques are needed for different types of organizations.

Best Practices in Assessment of Research and Development Organizations covers three key factors that underpin the success of an R&D organization: (1) the mission of the organization and its alignment with that of the parents; (2) the relevance and impact of the organization's work; and (3) the resources provided to the organization, beginning with a high-quality staff and management.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!