National Academies Press: OpenBook

An Assessment of the National Science Foundation's Science and Technology Centers Program (1996)

Chapter: HOW WELL HAS THE STC PROGRAM BEEN MANAGED AND EVALUATED?

« Previous: HOW WELL HAS THE DESIGN OF THE STC PROGRAM WORKED?
Suggested Citation:"HOW WELL HAS THE STC PROGRAM BEEN MANAGED AND EVALUATED?." National Academy of Sciences, National Academy of Engineering, Institute of Medicine, and National Research Council. 1996. An Assessment of the National Science Foundation's Science and Technology Centers Program. Washington, DC: The National Academies Press. doi: 10.17226/5401.
×

4

How Well Has the STC Program Been Managed and Evaluated?

Management of the STC program nationally is split between NSF's Office of Science, Technology, and Infrastructure (OSTI) and the directorates. OSTI provides administrative management, and the individual directorates provide scientific management. Visiting peer-review committees provide recurring scientific and managerial review of the individual centers. Center directors are, of course, responsible for ensuring that the research, education, and knowledge-transfer activities funded under their proposals are carried out effectively and efficiently. This chapter explores how well this management and evaluation process has worked relative to the overall goals of the STC program.

OBSERVATIONS AND COMMENTARY

The panel relied on several sources of information in evaluating the management of the STC program. Of particular importance were the site-visit reports, discussions with several Center directors, and responses to the surveys of the STC directors conducted by Abt. In addition, the panel reviewed a report of a National Academy of Public Administration (NAPA) study that was conducted in response to a Congressional request (NAPA 1995). The recommendations from that report are summarized in Box 4-1.

MANAGEMENT ISSUES ENCOUNTERED BY NSF AND INDIVIDUAL CENTERS

The panel has identified a number of general management issues, some of which are manifested at the NSF level, some at the center level, and some at both levels.

Suggested Citation:"HOW WELL HAS THE STC PROGRAM BEEN MANAGED AND EVALUATED?." National Academy of Sciences, National Academy of Engineering, Institute of Medicine, and National Research Council. 1996. An Assessment of the National Science Foundation's Science and Technology Centers Program. Washington, DC: The National Academies Press. doi: 10.17226/5401.
×

BOX 4-1

NAPA Study Elements and Recommendations

The NAPA study focused on four elements:

  • The degree to which the centers exhibited interdisciplinary collaboration, a university base, knowledge transfer to industry, and educational outreach.

  • The value of the center concept of management in science and technology.

  • The management approaches taken by each center.

  • The NSF's approach to the management of the STC program.

The study evaluated five centers to develop its conclusions and recommendations. It recommended that

  • NSF, Congress, and the administration recognize the good return on federal government expenditures through the STC program as being in the national interest and worthy of continued support.

  • NSF continue the current matrix method of managing the centers, with funding through the responsible directorates and Office of Polar Programs and coordination through the Office of Science and Technology Infrastructure.

  • The National Science Board (NSB) form a subcommittee, including (as appropriate) representatives of industry and other stakeholders, to

    • — Act as a monitoring body and inform Congress and other stakeholders of the goals, activities, and value of the STC program.

    • — Advise NSF in establishing processes and criteria to assess management of the STCs without adding to the current oversight burden.

    • — Advise NSF in establishing criteria to replace weak center directors or directors who leave.

  • The success of centers depends critically on the degree of scientific and administrative abilities of their directors. In the few cases where problems occurred at individual centers, the source of the problem turned out to be the executive leadership of the director.

  • Because centers vary widely in their scope, objectives, research foci, appropriate institutional linkages, and so forth, effective control of organizations and programs presents problems for both center directors and NSF program managers.

  • Once funded, centers must adhere to the goals and principles that their original proposals espoused. Ensuring that review and monitoring processes are effective can be a problem for NSF managers.

Suggested Citation:"HOW WELL HAS THE STC PROGRAM BEEN MANAGED AND EVALUATED?." National Academy of Sciences, National Academy of Engineering, Institute of Medicine, and National Research Council. 1996. An Assessment of the National Science Foundation's Science and Technology Centers Program. Washington, DC: The National Academies Press. doi: 10.17226/5401.
×

Both NSF and the individual centers have used this accumulated experience to improve their management of the centers.

Consistent High-Quality Management and Review

Center directors must ensure that their centers are not just groups of independent scientists, but rather embody real collaboration. That has not always been the case. For example, site visitors to one center questioned how closely center personnel work together; in another, there was some question as to whether the center funds 35 separate entities or is a true center. Similar concerns have been raised about some parts of two other centers.

Another challenge to leadership is maintaining focus. One center's work potentially affects research on sustainable systems, global warming, hazardous-waste site remediation, and biodiversity. The research problem is important and highly multidisciplinary, but a more-focused activity could result in greater influence on the research community than the research now being conducted, which has resulted in few publications that are merely descriptive.

In another case, a renowned scientist and pioneer in the field first requested that the 3rd-year review be delayed and then in year 4 requested that NSF support be phased out—an unfortunate end for a center whose objectives and approach were ideally suited to the intent of the STC program. The failure was not one of research objectives but rather of a central research tool (a superconducting magnet). It might be that neither the center management nor NSF management fully understood the imperatives of such a major engineering and fabrication challenge. The task of providing the magnet was given to a nonindustrial organization that could not deliver it. Poor management by the STC, NSF, or both appears to have caused the problem. Such mistakes in judgment are unfortunate but sometimes occur at this level of risk-taking. Termination of the center was appropriate.

NSF sometimes provides mixed messages to STCs through its site visits. For example, one center was reviewed three times by NSF; the visits resulted in the appointment of a new director and a vigorous outreach program, particularly in K-12 education, for which it developed software. The NSF staff was particularly constructive in trying to help the center to solve its problems and fulfill its aims. However, although several years ago the NSF reviews said that there was not enough outreach and education, a followup visit recently indicated that there was not enough research. In response to the first site-visit report, almost all the time of the postdoctoral scientists and almost all the resources of the center in the summer months are devoted to K-12 and outreach programs.

One of the centers is in a leadership role in its field. But when this center began, it had some difficulty in managing interinstitutional research, which resulted in warnings from its own advisory committee and a highly critical NSF site review. After a year of provisional funding, the problems were corrected. The same center received conflicting advice on the question of applied research. The

Suggested Citation:"HOW WELL HAS THE STC PROGRAM BEEN MANAGED AND EVALUATED?." National Academy of Sciences, National Academy of Engineering, Institute of Medicine, and National Research Council. 1996. An Assessment of the National Science Foundation's Science and Technology Centers Program. Washington, DC: The National Academies Press. doi: 10.17226/5401.
×

center was clearly made up of research scientists who worked on basic materials problems. It was unlikely that they would move into applied research even though the site committee was advising it. At another center, the advisory committee called for in the initial proposal has been inactive, meeting only once in 5 years. From such considerations, we conclude that the overall management of the STCs was very good but can be improved by recognizing the previous pitfalls mentioned and by proper use of the review process.

NSF's Administrative and Scientific Management of the STC Program

When the STC program began, staff from the various NSF directorates was not enthusiastic about the center concept. A separate office within OSTI was created by NSF Director Erich Bloch to oversee the centers. This office designed the program and solicitation and made the original funding decisions. Once the centers came into existence, their management of the program was split between the research directorates and OSTI.

OSTI, with the assistance of the assistant directors and the head of Office of Policy and Planning, functions as the policy-making group. They also design the procedures and formats for proposals, recordkeeping, and so on, but funding and technical management are handled by the various directorates. People from the program directorates rotate through OSTI to learn about the centers and the program.

Although there have been suggestions to shift management to OSTI or to the program directorates, the current arrangement seems preferable if the role of each participant is well understood. Several elements are needed for the STC program to thrive and meet its objectives. First, the research done in each center must be first-rate; this is best ensured by attention of the directorates, who have the best access to high-quality scientific and technical expertise. Second, there needs to be consistency in the management of the program across centers; undoubtedly, differences would emerge if each directorate managed its own centers without reference to those in other directorates. Third, and most important, the budgets for center support must retain a separate identity so that funds can flow to the most promising science independently of directorates. This is particularly important with respect to large grants. The STC program must keep a separate and distinct budgetary identity because the tradeoff of this program with other NSF activities (individual investigator, facilities, etc.) needs to be at the foundation level, with the NSF director responsible for that budgetary decision. The program also needs an advocate outside the individual directorates. For all these reasons, we believe that OSTI or its equivalent must continue to play a strong role.

STC Program Review

The STC program has been reviewed at both the programmatic level and the level of individual centers. At each site visit (annually for years 1-3, in competi-

Suggested Citation:"HOW WELL HAS THE STC PROGRAM BEEN MANAGED AND EVALUATED?." National Academy of Sciences, National Academy of Engineering, Institute of Medicine, and National Research Council. 1996. An Assessment of the National Science Foundation's Science and Technology Centers Program. Washington, DC: The National Academies Press. doi: 10.17226/5401.
×

tion for renewal at years 3 and 6, and every 18 months between years 3-6 and after the 6th year), the NSF-selected site-visit team reviews the entire center operation, including the educational and outreach activities. The panel found these reports to be a valuable source of information on the effectiveness of the centers. In several cases where management problems occurred, the NSF periodic site-review process worked well in sorting out the problems and assisted in bringing about solutions. Thus, we believe that NSF site review is important, particularly in the first few years. The value of the peer-review committees is in their expertise in the fields related to the research foci of the centers. However, such groups are less well suited to assess center accomplishments in K-12 outreach programs, because they usually comprise research scientists, not education experts. However, persons from the NSF Education and Human Resources Directorate have participated in some of these visits. In addition, the education expertise of potential site visitors is a factor in selecting teams. In November 1995, two of the STCs made a formal presentation on their educational activities to the Education and Human Resources Advisory Committee.

Center directors express considerable concern that they have been over-reviewed. Several additional reviews (at least of the original 11 STCs) have occurred in recent years, some of which were largely beyond NSF's control. First, the NSF inspector general decided to audit a number of Centers. Second, Congress mandated the study of the STC program by the NAPA. Third, NSF selected the STC program as a pilot project under the Government Performance and Results Act (GPRA), which led to the Abt evaluation and additional site visits. Finally, each STC's external advisory committee (a requirement of the cooperative agreement between each STC's university and NSF) periodically reviews it and advises its directors.

Interviews with center directors reported in the Abt study and this panel's discussions with STC directors support the argument that review is excessive. Moreover, directors point out that many of the reviews ask for the same information, causing them to wonder how much of the information is used or even readily available for use.

The committee does not endorse the recommendations by NAPA for deeper NSB involvement in the review process. NSB—through its membership, committee structure, and procedures—is well equipped to review the broad goals, strategies, and priorities of programs, but it is not the appropriate forum in which to assess how well each activity is performing. Thus, a review of our assessment of the entire STC program is an appropriate subject for NSB attention, but its direct involvement in the reviews of individual centers is inappropriate, except to hear and use such reviews.

THE STC PROGRAM VIEWED AS AN INVESTMENT BY NSF

The total fiscal year 1996 budget for all of NSF's modes of support is $3.2 billion; a breakdown is shown in Table 4-1. Of the $200 million devoted to all

Suggested Citation:"HOW WELL HAS THE STC PROGRAM BEEN MANAGED AND EVALUATED?." National Academy of Sciences, National Academy of Engineering, Institute of Medicine, and National Research Council. 1996. An Assessment of the National Science Foundation's Science and Technology Centers Program. Washington, DC: The National Academies Press. doi: 10.17226/5401.
×

centers at NSF, $60 million goes to the STC program. Thus, less than 2% of NSF's research budget (which does not include any overhead funds for NSF) goes to the STCs. That is a very small investment, compared with the other modes of support shown in Table 4-1 and Figure 4-1.

TABLE 4-1 NSF 1996 Budget by Modes of Support (in millions)

 

Amount

Proportion of Research Budget

Research projects (individual-investigator and group grants)

$1,700

53%

Facilities (large multiuser facilities)

$700

22%

Education & Training (K-12, undergraduate, graduate, postdoctorate, and underrepresented groups, such as women and minorities)

$600

19%

Centers (ERCs, earthquake, ecology, materials, minority, industry/university cooperatives, mass spectrometry)

$140

4%

STCs

$60

2%

The panel considers the STCs as examples of a mode of support that allows particular types of research problems to be addressed that otherwise would not be. If research problems are regarded as arrayed along a spectrum, with some problems well-suited to individual-investigator modes of inquiry, others to a center mode, and others to a facility mode, STCs emerge as one mode of support that helps balance the NSF portfolio of funding instruments. STCs constitute an experimental beginning of an effort to achieve balance. No evidence suggests that the limits of research problems best suited to center-like modes of support have been reached.

How Well Does the STC Program Fit With NSF's Strategic Plan?

NSF's current strategic plan puts forth a mission, NSF's vision, several long-range goals, and core strategies. The long-range goals, in abbreviated form, are

  • To enable the United States to uphold a position of world leadership in all aspects of science, mathematics, and engineering.

  • To promote the discovery, integration, dissemination, and employment of new knowledge in service to society.

  • To achieve excellence in US science, mathematics, engineering, and technology education at all levels.

Suggested Citation:"HOW WELL HAS THE STC PROGRAM BEEN MANAGED AND EVALUATED?." National Academy of Sciences, National Academy of Engineering, Institute of Medicine, and National Research Council. 1996. An Assessment of the National Science Foundation's Science and Technology Centers Program. Washington, DC: The National Academies Press. doi: 10.17226/5401.
×

FIGURE 4-1 NSF research budget by modes of support, 1990-1997.

The core strategies, abbreviated, are

  • To develop intellectual capital.

  • To strengthen the physical infrastructure.

  • To integrate research and education.

  • To promote partnerships.

Clearly, the goals and strategies demand a variety of approaches. The individual-investigator mode is certainly an important and proven mode in meeting some of the goals and strategies, but other modes are needed to meet goals that require multidisciplinary and multi-investigator efforts. Science and engineering research and education have thrived in a variety of venues, and NSF must offer a rich array of modes for the best results. In fact, the NSF strategic plan states explicitly that NSF “encourages flexibility in the methods used to promote the progress of science and its benefits to society.” Thus, the STC program fits NSF's strategic plan well.

Suggested Citation:"HOW WELL HAS THE STC PROGRAM BEEN MANAGED AND EVALUATED?." National Academy of Sciences, National Academy of Engineering, Institute of Medicine, and National Research Council. 1996. An Assessment of the National Science Foundation's Science and Technology Centers Program. Washington, DC: The National Academies Press. doi: 10.17226/5401.
×

The STC program was based on the idea that subjects to be addressed through the center mode of support should be identified by the research community. This panel endorses that approach. The role of NSF management is to state the general program goals clearly. The panel urges that any changes in goals or in priorities among them be clearly articulated and communicated to the centers and be reflected in new solicitations.

FINDINGS

The panel's key observations as to management and evaluation in the STC program are as follows:

  1. The success of individual centers depends critically on having strong scientific and administrative leadership.

  2. The nation and NSF are making a good investment in the STC program, gaining considerable leverage in research with a relatively small part of the total NSF budget. The STCs constitute an effort by NSF to achieve a balanced approach to research problems amenable to different modes of support. The panel considers the center approach to be a valuable and necessary tool in NSF's portfolio of support mechanisms. The limitations of the center approach are defined by the existence of research problems that are most amenable to attack by research teams with the combination of resources that defines centers.

  3. The STC program needs two kinds of management: administrative and scientific. Centralizing the program in one NSF directorate is difficult because no single office has sufficient scientific expertise. The original program announcement and selection needs to come from a central office to open up the process fully and allow funding to move to the best centers across all fields of science.

  4. The reviews of centers formally required by NSF are frequent but not excessive. However, the burden placed on some centers is excessive because of additional reviews–such as those of the NSF inspector general, NAPA, and the GPRA pilot program. The normal review process facilitated the solution to several management problems when they occurred in specific centers; it plays a valuable role. To the extent that multiple layers of review seek the same information from centers, NSF should try to coordinate reviews so as to avoid redundant data collection and to make data previously collected available to all who have good reason to be interested.

  5. Because NSF has most recently appeared to place equal weight on research, education, and knowledge exchange goals, some center directors have as well.

  6. There is some concern that resources devoted to K-12 activities compete with the time, energy, and funds devoted to research and that, if centers desire to pursue these activities, they not overwhelm the core mission of centers—research and education at the university level.

  7. The STC program fits in well with NSF's overall strategic plan.

Suggested Citation:"HOW WELL HAS THE STC PROGRAM BEEN MANAGED AND EVALUATED?." National Academy of Sciences, National Academy of Engineering, Institute of Medicine, and National Research Council. 1996. An Assessment of the National Science Foundation's Science and Technology Centers Program. Washington, DC: The National Academies Press. doi: 10.17226/5401.
×

CONCLUSIONS

The STC program is making a major contribution to the nation's scientific and engineering research for a relatively small investment. Management of individual STCs and NSF's management of the overall program are critical for their success. NSF's mechanism for managing the program seems to be working well, although attention should be paid to balancing centralized administrative (budgetary) management against decentralized scientific management. Furthermore, program goals and priorities should be clearly articulated and communicated and closely linked to overall NSF priorities. Of particular concern are the equal emphasis that NSF now places on the three program goals, with the inclusion of K-12 education as an addition to these program goals, and the evolutionary manner in which these priorities have shifted. At the individual center level, success requires a combination of a high degree of scientific leadership and enthusiasm and unusual administrative skills.

Suggested Citation:"HOW WELL HAS THE STC PROGRAM BEEN MANAGED AND EVALUATED?." National Academy of Sciences, National Academy of Engineering, Institute of Medicine, and National Research Council. 1996. An Assessment of the National Science Foundation's Science and Technology Centers Program. Washington, DC: The National Academies Press. doi: 10.17226/5401.
×
Page 26
Suggested Citation:"HOW WELL HAS THE STC PROGRAM BEEN MANAGED AND EVALUATED?." National Academy of Sciences, National Academy of Engineering, Institute of Medicine, and National Research Council. 1996. An Assessment of the National Science Foundation's Science and Technology Centers Program. Washington, DC: The National Academies Press. doi: 10.17226/5401.
×
Page 27
Suggested Citation:"HOW WELL HAS THE STC PROGRAM BEEN MANAGED AND EVALUATED?." National Academy of Sciences, National Academy of Engineering, Institute of Medicine, and National Research Council. 1996. An Assessment of the National Science Foundation's Science and Technology Centers Program. Washington, DC: The National Academies Press. doi: 10.17226/5401.
×
Page 28
Suggested Citation:"HOW WELL HAS THE STC PROGRAM BEEN MANAGED AND EVALUATED?." National Academy of Sciences, National Academy of Engineering, Institute of Medicine, and National Research Council. 1996. An Assessment of the National Science Foundation's Science and Technology Centers Program. Washington, DC: The National Academies Press. doi: 10.17226/5401.
×
Page 29
Suggested Citation:"HOW WELL HAS THE STC PROGRAM BEEN MANAGED AND EVALUATED?." National Academy of Sciences, National Academy of Engineering, Institute of Medicine, and National Research Council. 1996. An Assessment of the National Science Foundation's Science and Technology Centers Program. Washington, DC: The National Academies Press. doi: 10.17226/5401.
×
Page 30
Suggested Citation:"HOW WELL HAS THE STC PROGRAM BEEN MANAGED AND EVALUATED?." National Academy of Sciences, National Academy of Engineering, Institute of Medicine, and National Research Council. 1996. An Assessment of the National Science Foundation's Science and Technology Centers Program. Washington, DC: The National Academies Press. doi: 10.17226/5401.
×
Page 31
Suggested Citation:"HOW WELL HAS THE STC PROGRAM BEEN MANAGED AND EVALUATED?." National Academy of Sciences, National Academy of Engineering, Institute of Medicine, and National Research Council. 1996. An Assessment of the National Science Foundation's Science and Technology Centers Program. Washington, DC: The National Academies Press. doi: 10.17226/5401.
×
Page 32
Suggested Citation:"HOW WELL HAS THE STC PROGRAM BEEN MANAGED AND EVALUATED?." National Academy of Sciences, National Academy of Engineering, Institute of Medicine, and National Research Council. 1996. An Assessment of the National Science Foundation's Science and Technology Centers Program. Washington, DC: The National Academies Press. doi: 10.17226/5401.
×
Page 33
Suggested Citation:"HOW WELL HAS THE STC PROGRAM BEEN MANAGED AND EVALUATED?." National Academy of Sciences, National Academy of Engineering, Institute of Medicine, and National Research Council. 1996. An Assessment of the National Science Foundation's Science and Technology Centers Program. Washington, DC: The National Academies Press. doi: 10.17226/5401.
×
Page 34
Next: RECOMMENDATIONS »
An Assessment of the National Science Foundation's Science and Technology Centers Program Get This Book
×
Buy Paperback | $14.00 Buy Ebook | $11.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The National Science Foundation requested that the Committee on Science, Engineering, and Public Policy of the NAS, the NAE, and the IOM form a panel to evaluate the accomplishments of the NSF Science and Technology Centers program (not individual centers) against its goals in research, education, and knowledge transfer. This report is the result of the work of the panel charged with that effort, and provides recommendations for moving forward.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!