• Are there mechanisms in place to cut stagnant and unnecessary programs in order to prevent the dilution of the quality of more important programs?
• How should the impact of programs in basic research be assessed?
• How should innovation be assessed?
• How should spin-offs and transitions be measured?
• How are publication citations and patents—measures of the transition of knowledge—assessed?
• Are assessment metrics focused on outcomes as well as on activities?
• Are both historical impact and predictions of future success considered?
J. Stephen Rottler emphasized that there has been a need within his organization to shift from quantitative to qualitative assessment informed by data. Workshop discussants identified the following questions:
• Is there external oversight of the assessment, even if the assessment is not being conducted by an external review board?
• Is there a strong internal review to ensure that the product is not trivial before being submitted to external review? Is this applied to publications (especially for scientific publications) as well as to programs?
• To allow for candor without worry about giving offense or meeting with reprisal, especially in small scientific communities, do external reviews include processes to preserve anonymity?
• Are the terms of external review board members appropriate (generally between 3 and 5 years)?
• Does the review team have a balance of expertise and backgrounds?
• Do the review team members have good community reputations?
• Does the chair of the review team show good judgment?
• Has a clear tasking charge been provided for the assessment team?
• Has the audience for assessment reports been identified?
• Have mechanisms for both formal and informal communication of assessment findings been established?
• Is benchmarking included in the assessment as a useful means for assessing process factors—that is, how things are accomplished within the organization?
Each of the six presenters examined elements of technical management that affect the quality of the R&D scientific and technical work. Workshop discussants identified the following questions: