- An assessment of even a narrow field requires taking an average from disparate processes and systems, which can cause such assessments to be overly broad. For example, the number of patents granted within a particular field may be important, but individual researchers should not be judged by how many patents each one has generated.
- If all past research had been required to justify its value in terms of practical benefits, advances that have led to massive practical benefits would not have occurred.
- The knowledge generated by fundamental research has an intrinsic value regardless of its application. Without it, applied work would stagnate.
- Policymakers and the public in general agree on the value of research. Could research that fails to identify many of the benefits of science undermine that consensus and therefore be harmful?
PROGRESS IN UNDERSTANDING THE ISSUES
- The ever-growing power of the Web and the information sharing it enables will facilitate the analysis of research outputs. Natural language processing, machine learning technologies, and crowd sourcing will increasingly glean many reasonably accurate metrics from publications, patents, social networks, blogs, and so forth, and this capability will increase over time. Furthermore, this approach will be less costly and provide more information than government-mandated reporting. However, government agencies will need to create new tools to use these data to help fulfill their missions.
- The benefits of research results, both in terms of new knowledge and trained students, are vastly different from discipline to discipline and even from subdiscipline to subdiscipline. Thus, the determination of impacts requires very detailed analysis that is highly sector specific. For example, the evaluation of physics is different than the evaluation of computer science, and the evaluation of theoretical computer science is different than the evaluation of research in parallel computation.