National Academies Press: OpenBook

Best Practices in Assessment of Research and Development Organizations (2012)

Chapter: Appendix L Metrics Applied by National Research Council Panels to Assessment of the Army Research Laboratories

« Previous: Appendix K Examples of Peer Review Conducted at Federal R&D Organizations
Suggested Citation:"Appendix L Metrics Applied by National Research Council Panels to Assessment of the Army Research Laboratories." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations. Washington, DC: The National Academies Press. doi: 10.17226/13529.
×

Appendix L

Metrics Applied by National Research Council Panels to Assessment of the Army Research Laboratory

Chapter 4, “Assessing Technical Quality,” provides a discussion of metrics in the assessment of research and development (R&D) organizations. This appendix provides a set of assessment metrics and criteria applied by National Research Council (NRC) panels that review the Army Research Laboratory (ARL).

1. Community

a. Papers in quality refereed journals and conference proceedings (and their citation index)

b. Presentations and colloquia

c. Participation in professional activities (society officers, conference committees, journal editors)

d. Educational outreach (serving on graduate committees, teaching/lecturing, invited talks, mentoring students)

e. Fellowships and awards (external and internal)

f. Review panel participation (Army Research Office, National Science Foundation, Multidisciplinary University Research Initiative, etc.)

g. Recruiting new talent into ARL

h. Patents and intellectual property (IP) (and examples of how the patent or IP is used)

i. Involvement in building an ARL-wide cross-directorate community

j. Public recognition—for example, in the press and elsewhere for ARL research

2. Impact to Customers

a. Documented transfer/transition of technology, concepts or program assistance from ARL to research, development, and engineering centers (RDECs) or RDEC contractors for both the long term and short term

b. Direct funding from customers to support ARL activities

c. Documented demand for ARL support or services (Is there competition for their support?)

d. Customer involvement in directorate planning

e. Participation in multidisciplinary, cross-directorate projects

f. Surveys of customer base (direct information from customers on the value of ARL research)

3. Formulation of the Project’s Goals and Plan

a. Is there a clear tie to ARL Strategic Focus Areas, Strategic Plan, or other ARL need?

b. Are tasks well defined to achieve objectives?

c. Does the project plan clearly identify dependencies (i.e., successes depend on success of other activities within the project or outside developments)?

Suggested Citation:"Appendix L Metrics Applied by National Research Council Panels to Assessment of the Army Research Laboratories." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations. Washington, DC: The National Academies Press. doi: 10.17226/13529.
×

d. If the project is part of a wider activity, is the role of the investigators clear, and are the project tasks and objectives clearly linked to those of other related projects?

e. Are milestones identified, if they are appropriate? Do they appear feasible?

f. Are obstacles and challenges defined (technical, resources)?

g. Does the project represent an area in which application of ARL strengths is appropriate?

4. Methodology

a. Are the hypotheses appropriately framed within the literature and theoretical context?

b. Is there a clearly identified and appropriate process for performing required analyses, prototypes, models, simulations, tests, etc.?

c. Are the methods (e.g., laboratory experiment, modeling/simulation, field testing, analysis) appropriate to the problems? Do these methods integrate?

d. Is the choice of equipment/apparatus appropriate?

e. Is the data collection and analysis methodology appropriate?

f. Are conclusions supported by the results?

g. Are proposed ideas for further study reasonable?

h. Do the trade-offs between risk and potential gain appear reasonable?

i. If the project demands technological or technical innovation, is that occurring?

j. What stopping rules, if any, are being or should be applied?

5. Capabilities and Resources

a. Are the qualifications and number of the staff (scientific, technical, administrative) appropriate to achieve success of the project?

b. Is funding adequate to achieve success of the project?

c. Is the state of the equipment and facilities adequate?

d. If staff, funding, or equipment are not adequate, how might the project be triaged (what thrust should be emphasized, what sacrificed?) to best move toward its stated objectives?

e. Does the laboratory sustain the technical capability to respond quickly to critical issues as they arise?

6. Responsiveness

a. Have the issues and recommendations presented in the previous report been addressed?

Suggested Citation:"Appendix L Metrics Applied by National Research Council Panels to Assessment of the Army Research Laboratories." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations. Washington, DC: The National Academies Press. doi: 10.17226/13529.
×

This page intentionally left blank.

Suggested Citation:"Appendix L Metrics Applied by National Research Council Panels to Assessment of the Army Research Laboratories." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations. Washington, DC: The National Academies Press. doi: 10.17226/13529.
×
Page 78
Suggested Citation:"Appendix L Metrics Applied by National Research Council Panels to Assessment of the Army Research Laboratories." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations. Washington, DC: The National Academies Press. doi: 10.17226/13529.
×
Page 79
Suggested Citation:"Appendix L Metrics Applied by National Research Council Panels to Assessment of the Army Research Laboratories." National Research Council. 2012. Best Practices in Assessment of Research and Development Organizations. Washington, DC: The National Academies Press. doi: 10.17226/13529.
×
Page 80
Best Practices in Assessment of Research and Development Organizations Get This Book
×
Buy Paperback | $37.00 Buy Ebook | $29.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Research and development (R&D) organizations are operated by government, business, academe, and independent institutes. The success of their parent organizations is closely tied to the success of these R&D organizations. In this report, organizations refers to an organization that performs research and/or development activities (often a laboratory), and parent refers to the superordinate organization of which the R&D organization is a part. When the organization under discussion is formally labeled a laboratory, it is referred to as such. The question arises: How does one know whether an organization and its programs are achieving excellence in the best interests of its parent? Does the organization have an appropriate research staff, facilities, and equipment? Is it doing the right things at high levels of quality, relevance, and timeliness? Does it lead to successful new concepts, products, or processes that support the interests of its parent?

This report offers assessment guidelines for senior management of organizations and of their parents. The report lists the major principles of assessment, noting that details will vary from one organization to another. It provides sufficient information to inform the design of assessments, but it does not prescribe precisely how to perform them, because different techniques are needed for different types of organizations.

Best Practices in Assessment of Research and Development Organizations covers three key factors that underpin the success of an R&D organization: (1) the mission of the organization and its alignment with that of the parents; (2) the relevance and impact of the organization's work; and (3) the resources provided to the organization, beginning with a high-quality staff and management.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!