Academic Careers for Experimental Computer Scientists and Engineers (1994)

The requirements for good research in experimental computer science and engineering (ECSE) are different from those of many other academic disciplines and require attention. Evaluation of such research likewise demands special attention because it differs from standard academic practice. Most significantly, the computational artifact is the medium of research in ECSE. The creation of an artifact often embodies a substantial portion of the intellectual contribution of experimental research and represents a significant intellectual effort. It is therefore important that the evaluation of ECSE research take proper account of the implications of the artifact as medium, as discussed in Chapters 1 and 2.

The importance of artifacts in demonstrating proofs of existence, concept, and performance in ECSE means that the development and implementation of computing artifacts with wide impact are comparable to the publication of papers with wide impact. In addition, the rapid pace of the ECSE field puts a high premium on timely communications. Because conferences are the vehicle of choice in ECSE for the dissemination of research, well-refereed conference proceedings (as well as work published in refereed private journals) should be given as much weight as archival journal articles in evaluating a candidate's research portfolio for promotion or tenure.

The infrastructure requirements of ECSE faculty are closer to those of the laboratory-based science and engineering disciplines than to

those of the more theoretically oriented disciplines such as mathematics or statistics. Indeed, the committee believes that the lack of experimental infrastructure and/or a supportive research environment, including collaborators and mentors, may have greatly hampered or even prevented many talented experimentalists from producing significant research and thus led to their not receiving appropriate academic recognition. Reasonable expectations for research output should be scaled to match the resources available to an ECSE faculty member or team; this is especially important in light of the tighter funding picture for the foreseeable future.

The teaching dimension is problematic for many ECSE faculty, although differences among institutions of higher education obscure it to a certain degree. The very high student-faculty ratio in computer science and engineering (CS&E); the grading of complex student laboratory or project work in ECSE; the installation, maintenance, and upgrading of student ECSE laboratories; and keeping ECSE courses current in the face of rapidly changing technological underpinnings—all present extraordinarily time-intensive demands on ECSE faculty that should be recognized in making teaching assignments. The service dimension presents less of a problem, except for the rather frequent demand on ECSE faculty time to provide computer-related advice to the rest of the institution.

The focus on artifacts in ECSE, and other differences between the experimental and analytical methodologies, have led to tension between theoreticians and experimentalists. The manifestations of this tension vary from none at all in some departments to the perception, and perhaps the fact, in others that even very good junior experimental faculty members are being evaluated by criteria that are not appropriate for their research areas.

The committee makes the following recommendations to improve the academic environment for ECSE.

The importance of a supportive research environment for ECSE faculty cannot be overstated; indeed, it is so important that its absence may well defeat the most talented and gifted faculty member. Departments can help ECSE faculty, and especially new assistant professors, by

• Providing adequate "start-up" packages to ensure that resources are available to begin research immediately;

• Providing mentoring and advocacy, as described in Chapter 5;¹

• Providing opportunities to teach advanced seminars in which graduate students can receive needed training in preparation for joining a research project;

• Considering mechanisms by which an assistant professor can move "off the tenure track" temporarily if research difficulties arise (while the tenure clock is ticking, assistant professors and their departments must be ever sensitive to the productive use of time);

• Explicitly encouraging collaborative work with like-minded colleagues wherever they may be found;

• Providing adequate teaching assistant support for time-intensive laboratory-based courses (teaching loads may also be adjusted when developing such courses); and

• Resolving matters related to potential intellectual property arrangements with industrially supported research or research undertaken jointly with industry.

In addition, departments must understand that high-quality ECSE research with great impact often demands a substantial commitment of resources. Departments that wish to maintain high-quality ECSE research programs must pay careful attention to the following:

• Equipment and equipment upgrades (which for state-of-the-art systems may be necessary as often as yearly);

• Equipment maintenance;

• Laboratory space;

• Technical staff, to keep the computing environment current;

• Software resources such as computer-aided design (CAD) tools; and

• Hardware resources such as networking and servers.

The six-year probationary period before most tenure decisions are made is short enough that even if a junior ECSE faculty member has structured his or her research so that significant intermediate results have been reported, the record is still likely to differ from

those of other academics, including theoretical computer scientists. This record may well contain fewer publications, fewer publications in archival journals, and more alternative forms of publication such as distributed software or other demonstrated artifacts.

The committee recognizes a wide range of approaches to evaluating candidates for tenure and promotion, and it does not wish to intrude on institutional prerogatives in determining how best to evaluate candidates. At the same time, the committee believes that evaluators should use standards and criteria that normally characterize productive work in the ECSE discipline, rather than standards that may be better aligned with more traditional academic disciplines, taking care not to exclude meaningful evidence of achievement simply because it is nonstandard (as discussed in Chapter 5.)

Evaluating artifacts is difficult, although certain data such as the number of users of a given artifact may provide some insight into the extent of its impact. Perhaps the best way to document impact, as well as other aspects of a person's research track record and potential, is to obtain informative letters of reference. Of course, the central question then becomes, Who should write letters for a candidate?

The committee believes that the primary criteria in selecting a potential letter writer should be his or her stature in the field and familiarity with the candidate's work. Other factors, such as the letter writer's institutional location or status as collaborator of the candidate, should not be cause for excluding letters from such individuals. In particular, because views from industry may be important for judging the impact of ECSE work, letters from individuals in industry or government should not be arbitrarily limited and should carry equal weight to those of similarly qualified and reputable individuals in academia.

Given the strong dependence of experimental computer scientists and engineers in academia on federal funding, it is obvious that federal funding policy can have a major impact on the field. Federal agencies that fund ECSE research may wish to take into account the following considerations:

• A variety of funding structures are needed to support ECSE research. These run the gamut from small, relatively short term grants or contracts that focus primarily on the elaboration of a concept, to

large, relatively long term grants or contracts associated with deliverable computing artifacts. A good model of the latter is the National Science Foundation's Microelectronics Information Processing Systems (MIPS) program. Research initiation awards should continue. The committee recognizes the tightness of research budgets but points out that excessive trimming in the size or duration of individual research initiation awards will increase the risk that any given award will not lead to a significant ECSE research project.

• Computer science departments are major beneficiaries of tax policies that encourage computer manufacturers to donate computer equipment to universities. However, because the deductibility of such contributions is determined on the basis of the cost to manufacture the equipment, these same policies discourage the donation of software (because the ''manufacturing" cost of software is not much more than the cost of copying a few tapes or disks). The committee does not have the expertise to comment fully on the ramifications of tax policy but points out that the manufacturing costs of software do not reflect R&D costs and understate the value of software from both technical and business perspectives.

• Given the long time scales necessary to establish reputations in ECSE, postdoctoral support for new Ph.D.s in ECSE would be especially beneficial. Two or three years of postdoctoral support in which new Ph.D.s could begin to develop a research program and style would enable them to "hit the ground running" upon taking an assistant professorship. More importantly, the artifacts on which their reputations are based would have additional time to propagate into the community.

The computer and software industries in the United States have a direct and vested interest in the health of both the research and the educational dimensions of ECSE in academia. In addition, the spread of computer technology throughout business and industry, and the increasing sophistication of applications therein, suggest that firms in noncomputer industries—especially those that engage in significant applications development—also have a stake in ECSE. Thus, these industries may wish to take into account the following considerations:

• Academic ECSE research and education can be greatly enriched across the board by intellectual contact with industry. However, less well recognized schools find it especially difficult to estab

lish collaborative work arrangements with industry. The committee points out that exposure of local computer science and engineering departments to the problems and needs of industry may result not only in meaningful collaborative work but also in students who are better informed about those problems. Such students graduating from less well recognized universities may be more likely to work for computer, software, or other computing-intensive companies near these universities.

• Academic ECSE research has benefited greatly from industrial donations of equipment. However, maintenance costs are often substantial, and university funds to cover such costs are in short supply. In some cases, donated technical support and maintenance may be worth as much to a university as a donated machine.

• The source code for a software system is essential for most meaningful experimental research on that system. An academic researcher's access to a needed source code will certainly reduce the time required for him or her to complete an experimental software system and may result in an improved system of direct interest to the owner of the source code. Of course, the researcher and the industrial provider of the source code will have to reach agreements that guard the interests of both (the researcher in being able to publish or present results of the research and the company in maintaining its competitive advantages from that software).