Appendix
Possible Metrics for Assessment of Program and Project Accomplishments and Plans
PROGRAM AND PROJECT ACCOMPLISHMENTS
- Did the accomplishments represent significant scientific advances?
- What are the major consequences for the science if the project succeeds?
- Is the potential, long-term Army application of the research significant?
- Is the research novel, leading the field in an important area, and does it have the appropriate level of risk and payoff?
- Was related research being sponsored by other major players adequately summarized in terms of approach and goals? Were there areas of duplication?
- When comparisons are made, are the comparison groups well defined?
- How do the accomplishments map to the stated program goals and the Army Research Office (ARO) strategic plans?
- Do the accomplishments reflect productivity and ingenuity on the part of the performers?
- What portion of the accomplishments are attributable to the ARO funding?
- What is the network of contacts involved in the project?
- Were there appropriate examples of significant transitions, or anticipated transitions, of research to follow-on applied research or exploratory development either within industry or within an Army or a Department of Defense laboratory?
- Are there any high-priority missed opportunities/areas relating to the program or project?
- Was the speed of knowledge acceleration or transfer appropriate?
- What are the number, type, and caliber of awards and recognition related to the program or project?
- Has follow-up funding been awarded for the project or program?
PROGRAM PLANNING
- How, specifically, does the project address one or more critical challenges that the Army of the Future will face?
- Should research funding align topically with the current or the anticipated future focus of a division/program?
- Is there a clear and cogent strategy regarding how each of the program managers’ major objectives are likely to make substantial and unique progress in advancing the scientific frontiers of their discipline?
- Is there some reasonable basis (e.g., incipient breakthrough, new understanding, or novel theory) to believe that the scientific objectives might be met?
- Have the highest priority objectives been selected? If not, what is the potential that the accomplishments will lead to significant scientific advances?
- What mixture of collaboration (within ARO and with outside teams/agencies) and competition is best? What percentage of projects should be unique to ARO?
- If higher-priority areas are identified, what projects or programs should be reduced or eliminated to accommodate the new areas?
- Are anticipated transitions within the Army likely to be greater than transitions to the commercial sector or other government agencies (e.g., Naval Research Laboratory or Office of Naval Research)?
- Should funding portfolios be homogeneous or heterogeneous? Should funding of divergent or convergent works be preferred?
- What diversified portfolio of short-/medium-/long-term; low-/high-risk; and collaborative versus ARO-only projects is best, and why?
- What project type (e.g., preliminary exploration, longer-term research, workshop, or Multidisciplinary University Research Initiative) is best, and why?
- What ratio of faculty, staff, and students is best for a given topic and type of project, and why?
- What principal investigator (PI) turnover rate is best?
- How closely should program managers work with PIs? Are research collaborations between both required for a given project?
- What diversity ratios (e.g., gender, age, ethnicity) are best for a given project or program, and why?
- What percentage of a given project or program should be undertaken by postdoctoral researchers and graduate students?
- Are the most qualified applicants selected, independent of geospatial distribution?
- What composition of researchers, practitioners, and entrepreneurs is best to ensure high-quality research results that can be translated into products?
- What are the difficult aspects of a proposed project, and what are the major technical risks involved?