National Academies Press: OpenBook
« Previous: Summary
Suggested Citation:"1 Introduction." National Research Council. 2011. 2009-2010 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/13137.
×

1
Introduction

This introductory chapter describes the biennial assessment process conducted by the National Research Council’s (NRC’s) Army Research Laboratory Technical Assessment Board (ARLTAB). It then describes the preparation and organization of the report, the assessment criteria, and the approach taken during the report preparation.

THE BIENNIAL ASSESSMENT PROCESS

The charge of ARLTAB is to provide biennial assessments of the scientific and technical quality of the Army Research Laboratory (ARL). These assessments include the development of findings and recommendations related to the quality of ARL’s research, development, and analysis programs. The Board is charged to review the work of ARL’s six directorates but not to review two key elements of the ARL organization that manage and support basic research: the Army Research Office (ARO) and the Collaborative Technology Alliances (CTAs). Although the primary role of the Board is to provide peer assessment, it may also offer advice on related matters when requested to do so by the ARL Director; such advice focuses on technical rather than programmatic considerations. The Board is assisted by six NRC panels that focus on particular portions of the ARL program. The Board’s assessments are commissioned by ARL itself rather than by one of its parent organizations.

For this assessment, ARLTAB consisted of seven leading scientists and engineers whose experience collectively spans the major topics within the scope of ARL. Six panels, one for each of ARL’s directorates,1 report to the Board. Six of the Board members serve as panel chairs. The panels range in

1

The six ARL directorates are the Computational and Information Sciences Directorate (CISD), Human Research and Engineering Directorate (HRED), Sensors and Electron Devices Directorate (SEDD), Survivability and Lethality Analysis Directorate (SLAD), Vehicle Technology Directorate (VTD), and Weapons and Materials Research Directorate (WMRD). The Board

Suggested Citation:"1 Introduction." National Research Council. 2011. 2009-2010 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/13137.
×

size from 10 to 20 members, whose expertise is carefully matched to the technical fields covered by the directorate(s) that they review. In total, 96 experts participated, without compensation, in the process that led to this report.

The Board and panels are appointed by the National Research Council with an eye to assembling balanced slates of experts without conflicts of interest and with balanced perspectives. The 96 experts include current and former executives and research staff from industrial research and development (R&D) laboratories, leading academic researchers, and staff from Department of Energy national laboratories and federally funded R&D centers. Twenty-eight of them are members of the National Academy of Engineering, 4 are members of the National Academy of Sciences, and 1 is a member of the Institute of Medicine. A number have been leaders in relevant professional societies, and several are past members of organizations such as the Army Science Board and the Defense Science Board. The Board and its panels are supported by NRC staff, who interact with ARL on a continuing basis to ensure that the Board and panels receive the information that they need to carry out their assessments. Board and panel members serve for finite terms, generally 4 to 6 years, staggered so that there is regular turnover and a refreshing of viewpoints.

Biographical information on the Board members appears in Appendix B, along with a list of the members of each panel.

PREPARATION AND ORGANIZATION OF THIS REPORT

The current report is the sixth biennial report of ARLTAB. Its first biennial report was issued in 2000; annual reviews by the Board had been issued in 1996, 1997, and 1998. As with the earlier reviews, this report contains the Board’s judgments about the quality of ARL’s work (Chapters 2 through 7 focus on the individual directorates, and Chapter 8 provides a crosscutting overview of ARL). The rest of this chapter explains the rich set of interactions that support those judgments.

The amount of information that is funneled to the Board, including the consensus evaluations of the recognized experts who make up the Board’s panels, provides a solid foundation for a thorough peer review. This review is based on a large amount of information received from ARL and on panel interactions with ARL staff. Most of the information exchange occurs during the annual meetings convened by the respective panels at the appropriate ARL sites. Both at scheduled meetings and in less formal interactions, ARL evinces a very healthy level of information exchange and acceptance of external comments. The assessment panels engaged in many constructive interactions with ARL staff during their annual site visits in 2009 and 2010. The dates of the panel site visits are included in the introductory section of Chapters 2 through 7 on the individual directorates. In addition, useful collegial exchanges took place between panel members and individual ARL investigators outside of scheduled meetings as ARL staff members sought additional clarification about panel comments or questions and drew on panel members’ contacts and sources of information.

Each panel meeting lasted 2½ days, during which time the panel members received a combination of overview briefings by ARL management and technical briefings by ARL staff. Prior to the meetings, panels received extensive materials for review, including selected staff publications.

The overview briefings brought the panels up to date on ARL’s long-range planning. This context-building step is needed because the panels are purposely composed mostly of people who—while experts

does not have a panel specifically devoted to the Army Research Office, which is another unit of ARL, but all Board panels examine how well the in-house research and development of ARO and ARL are coordinated. Appendix A provides information summarizing the organization and resources of ARL and its directorates.

Suggested Citation:"1 Introduction." National Research Council. 2011. 2009-2010 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/13137.
×

in the technical fields covered by the directorates(s) that they review—are not engaged in work focused on Army matters. Technical briefings for the panels focused on the R&D goals, strategies, methodologies, and results of selected projects at the laboratory. Briefings were targeted toward coverage of a representative sample of each directorate’s work over the 2-year assessment cycle. Briefings included poster sessions that allowed direct interaction of the panelists with staff of other projects that either were not covered in the briefings or had been covered in prior years.

Ample time during both overview and technical briefings was devoted to discussion, both to clarify the relevant panel’s understanding and to convey the immediate observations and understandings of individual panel members to ARL’s scientists and engineers. The panels also devoted sufficient time to closed-session deliberations, during which they developed consensus findings and identified important questions or gaps in panel understanding. Those questions or gaps were discussed during follow-up sessions with ARL staff so that the panel was confident of the accuracy and completeness of its assessments. Panel members continued to refine their findings, conclusions, and recommendations during written exchanges and teleconferences among themselves after the meetings.

In addition to the insights that they gained from the panel meetings, Board members received exposure to ARL and its staff at Board meetings each winter. The 2010 Board meeting focused on the assessment process, and discussions with the ARL management team led to improvements in the assessment process in terms of the panels receiving timely read-ahead materials for their meetings, the development of the panel meeting agendas, and more attentiveness to the assessment process time lines. Also, some Board members attended the annual ARL Program Formulation Workshop in 2009 and 2010; at these workshops the ARL directorates discussed their programs with the directorates’ customers and stakeholders.

ASSESSMENT CRITERIA

During the assessment, the Board and its panels considered the following questions posed by the ARL Director:

  1. Is the scientific quality of the research of comparable technical quality to that executed in leading federal, university, and/or industrial laboratories both nationally and internationally?

  2. Does the research program reflect a broad understanding of the underlying science and research conducted elsewhere?

  3. Does the research employ the appropriate laboratory equipment and/or numerical models?

  4. Are the qualifications of the research team compatible with the research challenge?

  5. Are the facilities and laboratory equipment state of the art?

  6. Does the research reflect an understanding of the Army’s requirement for the research or the analysis?

  7. Are programs crafted to employ the appropriate mix of theory, computation, and experimentation?

  8. Is the work sufficiently unique and appropriate to the ARL niche?

  9. Are there especially promising projects that, with application of adequate resources, could produce outstanding results that could be transitioned ultimately to the field?

Within the general framework described above, the Board also developed and the panels applied detailed assessment criteria organized in the following six categories (Appendix C presents the complete set of assessment criteria):

Suggested Citation:"1 Introduction." National Research Council. 2011. 2009-2010 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/13137.
×
  1. Effectiveness of interaction with the scientific and technical community—criteria in this category relate to cognizance of and contributions to the scientific and technical community whose activities are relevant to the work performed at ARL;

  2. Impact on customers—criteria in this category relate to cognizance of and contributions in response to the needs of the Army customers who fund and benefit from ARL R&D;

  3. Formulation of projects’ goals and plans—criteria in this category relate to the extent to which projects address ARL strategic goals and are planned effectively to achieve stated objectives;

  4. R&D methodology—criteria in this category address the appropriateness of the hypotheses that drive the research, of the tools and methods applied to the collection and analysis of data, and of the judgments about future directions of the research;

  5. Capabilities and resources—criteria in this category relate to whether current and projected equipment, facilities, and human resources are appropriate to achieve success of the projects; and

  6. Responsiveness to the Board’s recommendations—with respect to this criterion, the Board does not consider itself to be an oversight committee. The Board has consistently found ARL to be extremely responsive to its advice, so the criterion of responsiveness encourages discussion of the variables and contextual factors that affect ARL’s implementation of responses to recommendations rather than an accounting of responses to the Board’s recommendations.

APPROACH TAKEN DURING THE REPORT PREPARATION

This report represents the Board’s consensus findings and recommendations, developed through deliberations that included consideration of the notes prepared by the panel members summarizing their assessments. The Board’s aim with this report is to provide guidance to the ARL Director that will help ARL sustain its process of continuous improvement. To that end, the Board examined its extensive and detailed notes from the many Board, panel, and individual interactions with ARL over the 2009-2010 period. From those notes it distilled a shorter list of the main trends, opportunities, and challenges that merit attention at the level of the ARL Director. The Board used that list as the basis for this report. Specific ARL projects are used to illustrate these points in the following chapters when it is helpful to do so, but the Board did not aim to present the Director with a detailed account of 2 years’ worth of interactions with bench scientists. The draft of this report was subsequently honed and reviewed according to NRC procedures before being released.

The approach to the assessment by the Board and its panels relied on the experience, technical knowledge, and expertise of its members, whose backgrounds were carefully matched to the technical areas within which the ARL activities are conducted. The Board and its panels reviewed selected examples of the standards and measurements activities and the technological research presented by ARL; it was not possible to review all ARL programs and projects exhaustively. The Board’s goal was to identify and report salient examples of accomplishments and opportunities for further improvement with respect to the technical merit of the ARL work, its perceived relevance to ARL’s definition of its mission, and apparent specific elements of ARL’s resource infrastructure that is intended to support the technical work. Collectively, these highlighted examples for each ARL directorate are intended to portray an overall impression of the laboratory while preserving useful mention of suggestions specific to projects and programs that the Board considered to be of special note within the set of those examined. The Board applied a largely qualitative rather than quantitative approach to the assessment; it is possible that future assessments will be informed by further consideration of various analytical methods that can be applied. The assessment panels’ site visits are currently scheduled to be repeated annually and the assessment report to be issued biennially.

Suggested Citation:"1 Introduction." National Research Council. 2011. 2009-2010 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/13137.
×

REPORT CONTENT

This chapter discusses the biennial assessment process used by ARLTAB and its six panels. Chapters 2 through 7 provide a detailed assessment of each of the six ARL directorates. Chapter 8 presents an overview focused on crosscutting issues across all of ARL. The appendixes provide the ARL organizational chart and staffing profile, biographical information on the members of ARLTAB and a list of the membership of its panels, the assessment criteria used by ARLTAB and its panels, and a list of acronyms found in the report.

Suggested Citation:"1 Introduction." National Research Council. 2011. 2009-2010 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/13137.
×
Page 12
Suggested Citation:"1 Introduction." National Research Council. 2011. 2009-2010 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/13137.
×
Page 13
Suggested Citation:"1 Introduction." National Research Council. 2011. 2009-2010 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/13137.
×
Page 14
Suggested Citation:"1 Introduction." National Research Council. 2011. 2009-2010 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/13137.
×
Page 15
Suggested Citation:"1 Introduction." National Research Council. 2011. 2009-2010 Assessment of the Army Research Laboratory. Washington, DC: The National Academies Press. doi: 10.17226/13137.
×
Page 16
Next: 2 Computational and Information Sciences Directorate »
2009-2010 Assessment of the Army Research Laboratory Get This Book
×
Buy Paperback | $34.00 Buy Ebook | $27.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The charge of the Army Research Laboratory Technical Assessment Board (ARLTAB) is to provide biannual assessments of the scientific and technical quality of the research, development, and analysis programs at the Army Research Laboratory (ARL). The advice provided in this report focuses on technical rather than programmatic considerations.

The Board is assisted by six National Research Council (NRC) panels, each of which focuses on the portion of the ARL program conducted by one of ARL's six directorates. When requested to do so by ARL, the Board also examines work that cuts across the directorates. The Board has been performing assessments of ARL since 1996. The current report summarizes its finding for the 2009-2010 period, during which 96 volunteer experts in fields of science and engineering participated in the following activities: visiting ARL annually, receiving formal presentations of technical work, examining facilities, engaging in technical discussions with ARL staff, and reviewing ARL technical materials.

The Board continues to be impressed by the overall quality of ARL's technical staff and their work and applauds ARL for its clear, passionate concern for the end user of its technology--the soldier in the field--and for ARL's demonstrated mindfulness of the importance of transitioning technology to support immediate and longer-term Army needs. ARL staff also continue to expand their involvement with the wider scientific and engineering community. In general, ARL is working very well within an appropriate research and development (R&D) niche and has been demonstrating significant accomplishments.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!