It was apparent to the committee that all the agencies interviewed have made good-faith efforts to comply with the requirements of GPRA. During the focus groups, they described in detail the evolution of their approaches, their frequent midcourse corrections, and their expenses in time and effort. The act forbade the use of outside consultants or additional hiring to design and execute GPRA responses, and for most agency officials the demands of GPRA produced an increased workload that promises to continue for some time. (For more details, see the agencies' responses, summarized in Appendix C.)
All agencies use expert review panels to evaluate their research programs. However, in response to GPRA, each of the agencies addresses the issue of expert review in a different way. Further, while some have well-established procedures that they are just refining, others are still at the very early stages of development.
Both NSF and NIH use advisory groups who produce evaluations via an alternative format approved by OMB. Using this method, there is no attempt to quantify a goal or the degree to which it has been met. Rather, goals are successfully met or substantially exceeded in NIH's case or successful or minimally effective in NSF's case, as determined by a single (in NIH's case) or multiple (in NSF's case) expert review panels. At NIH, a single overarching panel evaluates all NIH's research programs at one time. At NSF, numerous committees of visitors review individual research programs on a rolling 3-year basis. The results of those evaluations are then provided to several advisory committees whose membership represent several disciplines.
DOD uses a process called Technology Area Reviews and Assessments (TARA) to evaluate science and technology programs through expert peer reviews. In the DOD process, basic research is not isolated from applied research and advanced technology