When the benchmarking studies were proposed, it was not clear whether a panel of experts in a particular field could analyze that field in an objective fashion. By "objective", the committee means "a reasonably balanced view of US research compared with that of the rest of the world." In spite of that concern, COSEPUP has concluded that each panel has been able to produce a reasonably objective report. An important goal of the benchmarking program was to conduct the studies at modest expense within a short period. Given the large number of fields in science and engineering, benchmarking would not be practical if it were expensive or time-consuming. The three experiments were completed in 6-8 months at a cost of $50,000 each. (A key factor in the low cost is that all panelists agreed to be pro bono contributors.)
The studies succeeded in identifying key factors that influenced the status of fields. For example, in mathematics, human resources— particularly the reliance on foreign talent—was identified as a key issue. The panel noted that current US leadership depends substantially on temporary waves of immigrants from Europe (notably from the former Soviet Union) and Asia that cannot be counted on to continue. In addition, the panel warned that the quality of US research could be affected in the future by the observed falloff in numbers of American students pursuing graduate-level mathematics.
Using information from the Department of Energy and the National Institute of Standards and Technology, the materials science and engineering panel was able to identify facilities and infrastructure as the keys to research leadership. In the United States, some materials research facilities, many of which were built in the 1960s, are deteriorating. In Europe and Japan, facilities tend to be more modern.
In immunology, the three tools for evaluating US research (reputation survey, citation analysis, and journal-publication analysis) were distinct and had different strengths and flaws, but they led to basically the same conclusion: the names of US researchers appeared between 2-3 times as often as the names of non-US researchers. The immunology panel was able to identify an important concern that arises from shifts in the US health-care system. The new emphasis on managed care means that fewer patients are available to academic institutions for clinical trials in immunology. The US health-care system differs from that of many European countries, where the centralized medical system provides an abundance of such patients.
COSEPUP found that each panel developed its conclusions by using a similar set of methods, with some variation to match field-specific