Evaluation of NSF's Program of Grants for Vertical Integration of Research and Education in the Mathematical Sciences (VIGRE) (2009)

The committee identified nine recommendations on the basis of its review of the National Science Foundation’s (NSF’s) program of Grants for Vertical Integration of Research and Education in the Mathematical Sciences (VIGRE) and its accomplishments.

As discussed in Chapter 2, the VIGRE program was begun in 1998 in an effort to revitalize mathematics education at the undergraduate and graduate levels. In the committee’s judgment, the impact of the VIGRE program has been mitigated by a variety of factors, including the following:

• The lack of a consistent statement of program goals and of an explicit expectation of scientific quality in the award process,

• A fluctuating level of funding inconsistent with the magnitude of the reforms desired,

• Overly restrictive programmatic requirements,

• Limitations on participation,

• An undefined assessment process, and

• The absence of a systematic means for disseminating successful innovations to the broader mathematical sciences community.

Nevertheless, the underlying need for a program like the VIGRE program still exists. Moreover, there are impressive examples of the meaningful impact that the VIGRE program has had on the educational programs of some departments, leading to the kind of systemic change called for when the VIGRE program was conceived. Examples of these changes can be found in Cozzens (2008). The committee believes that if the recommendations presented below are incorporated into the VIGRE program, it can

better serve its intended purpose and have a wider impact, consistent with the original design of the program.

As discussed in Chapter 2, the committee found considerable confusion over the goals of the VIGRE program, in part because they have shifted over time. Although initially it was not seen as a workforce program, in later requests for proposals (RFPs) it appears to have become one. In order to maximize the impact of the program, the committee believes that a clear, concise, consistent set of goals should be established.

While the goals of the VIGRE program have evolved over time, the following four seem to have been part of the program since its inception:

• Vertical integration of mathematics education and research,

• Greater breadth in the mathematical education of students,

• Improved communication skills for graduates in the mathematical sciences, and

• Increased exposure of students in mathematical sciences to disciplines that require mathematics.

The committee believes that these goals were responsive to the studies that led up to the creation of the VIGRE program and that they are still relevant for the mathematical sciences community. Whether these are the goals that capture the expectations of the National Science Foundation for its program at this point is not for the committee to decide. However, the committee believes that NSF should establish a clear set of goals for the program and emphasize them in future publications and RFP solicitations.

Finally, the committee could find no specific reference to the scientific quality of the proposed activity as a criterion for selecting VIGRE awardees. In the committee’s view, this serious omission should be corrected. The goals of the VIGRE program will best be met in the future if funds are granted to individuals and departments that set a high standard for quality in their disciplines.

Although it is a worthy aspiration for VIGRE program RFPs to call simultaneously for vertical integration from undergraduate education to postdoctoral research, for department-wide change across all subdisciplines, and for simultaneous and significant change in a department’s undergraduate, graduate, and postdoctoral programs, this should not be seen by NSF as the only path to achieving the goals of the program or to realizing the recommendations of the national panels referred to in Chapter 2. The committee has seen many examples of benefits to education, breadth of experience, and culture from interactions across some vertical divisions, such as postdoctorals mentoring graduate students or graduate students mentoring undergraduates. The experience of the committee members is that there are benefits to connectivity; but no evidence has been presented that all of those elements of vertical integration need to be present in a department in order to see any benefits. NSF’s Division of Mathematical Sciences (DMS) has moved recently in the direction of making its workforce programs, such as Research Training Groups and Mentoring Through Critical Transition Points, broader and more flexible. DMS

has even issued an open call for workforce proposals without VIGRE’s predefined structures. Within the broad goals established for the VIGRE program, awardees should enjoy increased flexibility and should be encouraged to employ additional local initiative.

The committee is very supportive of the departmental nature of the grants, but it believes that the grants should not necessarily require engagement by a major portion of the department. Proposals should be entertained that, for example, build on the particular strengths of a department or that help broaden its impact. Further, the committee believes that the VIGRE program should allow for awards that do not necessarily include vertical integration across the full breadth of a department’s educational programs or across all educational levels from the undergraduate to that of postdoctoral fellows. Moreover, proposals involving fewer faculty, with greater release time and staff support, and emphasizing particular elements of the VIGRE programs in the departments should be permitted.

In sum, the committee believes that program requirements should contain fewer “ands” and more “ors.” Not all proposals need to satisfy all changes at once. This would encourage a broader range of institutions to apply and contribute.

The committee was dismayed to learn that, too often, changes accomplished by individual VIGRE grants have not been sustained. Indeed, responses to the committee’s survey of departments of mathematical sciences indicated that many of the achievements at individual programs have not been or will not be continued after the expiration of the VIGRE grant. Because NSF is such a dominant source of support in the mathematical sciences, sustainability of reform efforts in mathematics requires special consideration by the Foundation. In order for the changes envisioned by the VIGRE program to persist, the committee urges NSF to make longer-term awards that require a commitment from the host universities to provide the necessary support to sustain successful new initiatives resulting from VIGRE as funding from NSF phases down and terminates.

Some of this sustainability of reform efforts can be accomplished within the VIGRE program framework. Offering larger awards and a return to the longer wind-down periods of the “centers of excellence”¹ grants of the 1960s would provide incentives for institutions to make continuing commitments to their mathematical science programs (and give them the time to build the required continuing resource base). The centers-of-excellence grants funded new faculty positions in various fields for up to 10 years, with the expectation that institutions would pick up funding for those salaries at the end of that period. Such a model could address the serious challenge, particularly for many public and small private universities, of sustaining various components of a VIGRE grant.

In this spirit, the committee urges that initial VIGRE awards should be for a 5-year period, with a review at the end of the 3rd year. Based on a satisfactory review and an institutional commitment of funds to sustain the VIGRE grant, departments would be granted a second 5 years for a total of 10 years of funding. Departments deemed not to be making satisfactory progress would have their grants terminated, with a 1-year “phase-down” grant, perhaps at a lower level than the initial grant, to accommodate special situations such as support for graduate students and postdoctoral fellows that began in year 4. The committee is not recommending that the second 5-year renewal period necessarily be funded at the same level as the first. The second 5-year award might be smaller, augmented by a requirement of insti-

tutional funding, and could be more focused on the successful aspects of its preceding award. VIGRE awards should be viewed as seed money for change, especially for the first 5 years of funding, with the expectation of sustainability if awards are extended for a full 10 years.

To encourage the submission of more proposals, the committee urges that NSF institute a preproposal stage for VIGRE awards. Preproposals would be designed to be less detailed and to require less preparation than full proposals. The preparation of proposals for VIGRE projects requires large amounts of faculty and administrative time. The committee’s survey indicated that some departments did not apply for VIGRE grants because of the work involved in the application process. The declining number of proposals for VIGRE grants over the years since the program’s inception is further evidence that the magnitude of the effort to develop proposals of this scale is not deemed a cost-effective use of time. Requiring the early engagement of a proposing institution’s higher administration could lead to stronger institutional commitments to sustain VIGRE-induced change. A carefully constructed preproposal process, in which NSF commented on the strengths and weaknesses of an institution’s preproposal, would enable institutions to test ideas with NSF to see if they hold promise without the institution’s incurring the enormous expenditures of resources to develop a full-blown proposal. The committee believes that the guidance which departments would gain during a preproposal process would also strengthen the quality of the proposals submitted.

The committee believes that a program designed to increase departmental interaction, communication, and cooperation is ill-served when large portions of the graduate student and postdoctoral population, namely foreign nationals, are excluded. Although foreign students and postdoctoral fellows may participate in VIGRE program activities, they may not receive support from the VIGRE program. In some instances this enhances a tendency for them to be isolated within the department to the detriment of themselves and the detriment of domestic students. Inclusion of such talented students in the VIGRE program would aid their acculturation and English language skills and thus enhance the chance of their remaining permanently in the United States and contributing to the mathematical sciences workforce. Indeed, one aspect of a VIGRE grant might be to encourage activities among foreign students and postdoctoral fellows that improve their English language and teaching skills.

This recommendation is consistent with language in the National Academies’ (2005) report Policy Implications of International Graduate Students and Postdoctoral Scholars in the United States, which says:

This report goes on, in its recommendations, to further justify the inclusion of foreign graduate students and postdoctoral fellows in the programs of U.S. universities. The findings and recommendations cited below are unambiguous concerning the benefits of greater international participation with little or no negative consequences to the United States.

Chapter 2 identifies the need for well-prepared master’s- or doctoral-level professionals who can use sophisticated mathematics in nonacademic settings. Examples include professions involving financial mathematics, biostatistics, K-12 education, and a range of areas at the interfaces of the computational sciences, including computational mathematics and, more recently, advanced analytics. The experience and impact of the Professional Science Masters program,² developed by the Sloan Foundation, offer a good example of how such programs can be vertically integrated into a department’s degree programs.

The committee urges that the VIGRE program’s scope be expanded to allow support for such efforts in doctorate-granting departments. These would not require large funding, because many of these programs, as with the master of business administration, typically are job-market driven and do not require significant student stipend support.

The VIGRE program was launched with no apparent plan for assessing the effectiveness of the program. No quantifiable goals were set, and no predetermined data elements to measure performance were defined. If the program is to be continued, this deficiency must be addressed. As noted in Chapter 5, it is difficult at this point to disentangle the quantitative and qualitative changes in departments that can be attributed to a VIGRE grant from those that occurred as a result of other factors.

Assessment needs to occur both at the macro level for the program as a whole and at the level of individual program grantees. As part of the assessment strategy, NSF needs to develop a limited set of supporting data elements, aligned with the goals of the program, that it will systematically collect from VIGRE-supported departments. The committee suggests that the data collection specifically include the following:

• A small, carefully chosen set of metrics, conforming as much as possible to data collected more broadly by NSF and the American Mathematical Society or submitted to the Department of Education, that can be collected with relative ease by all projects and required of all projects annually;

• A database to track students and postdoctoral fellows supported by participating VIGRE institutions beyond the expiration of the grants;

• Ten years of benchmarking data from grantees as opposed to the current 5-year requirement; and

• Survey instruments to collect data from VIGRE program participants (e.g., students, postdoctoral fellows, faculty), to measure the impact of the program on individuals supported by it.

Further, the committee believes that NSF’s expectations of awardees must be clarified. NSF should develop a consistent evaluation strategy for VIGRE grantees and, as noted above, a limited set of metrics that must be collected by all projects and reported annually. The evaluation strategy must be carefully constructed so as not to work against the need for simplification and flexibility that the committee has noted in Recommendation 3. Overall, the system of 3rd-year review by means of self-assessment and site visit is working well. The evaluation process should be transparent, and the results of the evaluation need to be used in renewal reviews.

From the beginning, the VIGRE program should have been conceived of as a national program, offering the possibility of fundamental change in mathematics education through the dissemination of successful programs and best practices. A number of strategies could be taken to improve the dissemination of the practices and accomplishments of the VIGRE program.

First, for example, awardees should be expected to engage more fully in efforts to disseminate their activities and outcomes, with dissemination plans included in the proposals. Each awardee should maintain a VIGRE Web site, and access to Web sites should be maintained by the institution after the funding ends, to provide a record of the program. Departments should be encouraged to disseminate examples of their VIGRE activities by, for example, developing resources that could be picked up by other departments. Examples of such resources would include Webcasting lectures or symposia and providing syllabi.

Second, NSF should take the lead in developing a framework and infrastructure for information and communication mechanisms that would encompass all workforce program grants, which could include the following:

• Encouraging textbook development and other ways of capturing curriculum reform;

• Sponsoring the creation and promotion of a “portal” to all VIGRE Web sites. This portal would enable any mathematics or statistics department to identify and implement activities pioneered at the VIGRE-award institutions that might work for their own department. A model is the NSF ADVANCE (Advancement of Women in Academic Science and Engineering Careers) program portal at http://research.cs.vt.edu/advance/index.htm (because the VIGRE program is now one component in a suite of workforce programs, the focus of a “what works” portal should not be limited to the VIGRE program but should encompass the full range of associated programs; strategies that work should not be artificially constrained by program boundaries);

• Sponsoring meetings, sessions at meetings of the mathematics and statistics professional societies, and workshops; and

• Encouraging structured engagement of the NSF Mathematical Science Research Institutes.

Third, the committee suggests that NSF develop at least a pilot program of “adaptive implementation grants.” The RFP for these grants would invite proposers to base their proposals on the replication or adaptation of successful VIGRE activities, such as those in the “what works” portal, to their own institutions (as is now happening with the UTeach program at the University of Texas at Austin). A good example of what the committee envisions is the ADVANCE program’s Partnerships for Adaptation, Implementation, and Dissemination (PAID) awards that support analysis, adaptation, dissemination and use of existing innovative materials and practices that have been demonstrated to be effective.

This page intentionally left blank.