Appendix

Possible Metrics for Assessment of Program and Project Accomplishments and Plans

PROGRAM AND PROJECT ACCOMPLISHMENTS

• Did the accomplishments represent significant scientific advances?
• What are the major consequences for the science if the project succeeds?
• Is the potential, long-term Army application of the research significant?
• Is the research novel, leading the field in an important area, and does it have the appropriate level of risk and payoff?
• Was related research being sponsored by other major players adequately summarized in terms of approach and goals? Were there areas of duplication?
• When comparisons are made, are the comparison groups well defined?
• How do the accomplishments map to the stated program goals and the Army Research Office (ARO) strategic plans?
• Do the accomplishments reflect productivity and ingenuity on the part of the performers?
• What portion of the accomplishments are attributable to the ARO funding?
• What is the network of contacts involved in the project?
• Were there appropriate examples of significant transitions, or anticipated transitions, of research to follow-on applied research or exploratory development either within industry or within an Army or a Department of Defense laboratory?
• Are there any high-priority missed opportunities/areas relating to the program or project?
• Was the speed of knowledge acceleration or transfer appropriate?
• What are the number, type, and caliber of awards and recognition related to the program or project?
• Has follow-up funding been awarded for the project or program?

PROGRAM PLANNING

• How, specifically, does the project address one or more critical challenges that the Army of the Future will face?
• Should research funding align topically with the current or the anticipated future focus of a division/program?
• Is there a clear and cogent strategy regarding how each of the program managers’ major objectives are likely to make substantial and unique progress in advancing the scientific frontiers of their discipline?
• Is there some reasonable basis (e.g., incipient breakthrough, new understanding, or novel theory) to believe that the scientific objectives might be met?

• Have the highest priority objectives been selected? If not, what is the potential that the accomplishments will lead to significant scientific advances?
• What mixture of collaboration (within ARO and with outside teams/agencies) and competition is best? What percentage of projects should be unique to ARO?
• If higher-priority areas are identified, what projects or programs should be reduced or eliminated to accommodate the new areas?
• Are anticipated transitions within the Army likely to be greater than transitions to the commercial sector or other government agencies (e.g., Naval Research Laboratory or Office of Naval Research)?
• Should funding portfolios be homogeneous or heterogeneous? Should funding of divergent or convergent works be preferred?
• What diversified portfolio of short-/medium-/long-term; low-/high-risk; and collaborative versus ARO-only projects is best, and why?
• What project type (e.g., preliminary exploration, longer-term research, workshop, or Multidisciplinary University Research Initiative) is best, and why?
• What ratio of faculty, staff, and students is best for a given topic and type of project, and why?
• What principal investigator (PI) turnover rate is best?
• How closely should program managers work with PIs? Are research collaborations between both required for a given project?
• What diversity ratios (e.g., gender, age, ethnicity) are best for a given project or program, and why?
• What percentage of a given project or program should be undertaken by postdoctoral researchers and graduate students?
• Are the most qualified applicants selected, independent of geospatial distribution?
• What composition of researchers, practitioners, and entrepreneurs is best to ensure high-quality research results that can be translated into products?
• What are the difficult aspects of a proposed project, and what are the major technical risks involved?