Appendix B Comparing Journal and Conference Publication

This appendix compares several different quantitative measures of the acceptance and refereeing process (i.e., timeliness and selectivity) to determine how well the different publication media achieve the goals that experimental computer science and engineers desire, how these publication media might be improved, and what these publications tell us about the success of various academic researchers.

TIME TO PUBLICATION

As mentioned earlier, long delays to publication are a major hindrance to progress in experimental computer science and engineering (ECSE).1 Long publication delays also affect recognition and can be a problem in considering tenure and promotion cases. Journal editors try their best to minimize delay but are often stymied in their efforts by referees who do not return reviews in a timely fashion. The committee examined the time from submission to acceptance and publication for a variety of leading conferences and journals.

Table B.1 presents the times from submission to acceptance to publication for some major conferences; by comparison, the times for

1  

The circulation of preprints in ECSE is extensive (assisted in rapid distribution by connections to the Internet) and testifies to a problem with timeliness.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 135
Academic Careers for Experimental Computer Scientists and Engineers Appendix B Comparing Journal and Conference Publication This appendix compares several different quantitative measures of the acceptance and refereeing process (i.e., timeliness and selectivity) to determine how well the different publication media achieve the goals that experimental computer science and engineers desire, how these publication media might be improved, and what these publications tell us about the success of various academic researchers. TIME TO PUBLICATION As mentioned earlier, long delays to publication are a major hindrance to progress in experimental computer science and engineering (ECSE).1 Long publication delays also affect recognition and can be a problem in considering tenure and promotion cases. Journal editors try their best to minimize delay but are often stymied in their efforts by referees who do not return reviews in a timely fashion. The committee examined the time from submission to acceptance and publication for a variety of leading conferences and journals. Table B.1 presents the times from submission to acceptance to publication for some major conferences; by comparison, the times for 1   The circulation of preprints in ECSE is extensive (assisted in rapid distribution by connections to the Internet) and testifies to a problem with timeliness.

OCR for page 135
Academic Careers for Experimental Computer Scientists and Engineers TABLE B.1 Time from Submission to Acceptance and Publication of Papers for Selected Conferences and Journals   Time to Acceptance (months) Time to Publication (months) Conferencea PLDI 2.3 7.3 ISCA 2.6 6.2 ASPLOS 2.5 7.0 SOSP 3.0 7.8 OOPSLA 2.6 7.2 SIGGRAPH 2.3 6.8 Average 2.5 7.0 Journalb TOPLAS 24.0 32.4 TOCS 17.0 21.3 IEEE TOC N/A 32.2 IEEE TSE N/A 29.2 a PLDI, conference on Programming Language Design and Implementation; ISCA, International Symposium on Computer Architecture; ASPLOS, conference on Architectural Support for Programming Languages and Operating Systems; SOSP, Symposium on Operating System Principles; OOPSLA, conference on Object-Oriented Programming Systems, Languages, and Applications; SIGGRAPH, conference of the ACM Special Interest Group on Graphics. b TOPLAS, Transactions on Programming Languages and Systems (published by ACM); TOCS, Transactions on Computer Systems (published by ACM); TOC, Transactions on Computers (published by IEEE; data for 1988 to 1992, special issues omitted); and TSE, Transactions on Software Engineering (published by IEEE; data for 1988 to 1992, special issues omitted). journal submission and publication are much longer. Table B.1 also indicates the comparable times for two of the primary Association of Computing Machinery (ACM) journals that publish ECSE research. Notice that the conferences listed, except for ASPLOS, are annual and the journals quarterly. For Transactions on Programming Languages and Systems (TOPLAS), it takes approximately 10 times longer to acceptance and 5 times longer to publication than conferences require, whereas Transactions on Computer Systems (TOCS) requires about 6 times longer for acceptance and 2.7 times longer for publication than conferences. (For TOCS the average was significantly reduced by the use of special issues that offered fast-track publication to selected, top-rated conference papers.) Multiple sequences of refereeing and revision account for a large

OCR for page 135
Academic Careers for Experimental Computer Scientists and Engineers TABLE B.2 Publication Time and Manuscript Revision History for TOPLAS Revisions Required Fraction of Submissions (%) Time to Acceptance (months) Time to Publication (months) 1 50 18.0 26.4 2 45 30.0 37.2 3 5 37.2 46.8 TABLE B.3 Age of References in Journal and Conference Publications Publicationa Fraction of References to Papers Less Than Two Years Old (%) Median Age of Reference (years) ASPLOS (Conference) 32 3 ISCA (Conference) 35 3 PLDI (Conference) 26 4 TOCS (Journal) 20 4.5 TOPLAS (Journal) 12 5 a ASPLOS, conference on Architectural Support for Programming Languages and Operating Systems; ISCA, International Symposium on Computer Architecture; PLDI, conference on Programming Language Design and Implementation; TOCS, Transactions on Computer Systems (published by ACM); TOPLAS, Transactions on Programming Languages and Systems (published by ACM). part of the delay in journal publication. For example, Table B.2 presents data from three years of TOPLAS. Even if only the 50 percent of submissions that require merely one revision are considered, journal acceptance requires about 6 times longer, and publication 3.7 times longer, than the average conference, which confirms that conferences offer a substantial advantage in timeliness. With the rapid advances in experimental computer science, timeliness is critical. One way of seeing how important timeliness is and how it is affected by publication time is to look at the citations in published papers. Table B.3 contains data computed from looking at all the references in an entire year for TOCS and TOPLAS plus all the references in one year's ASPLOS, ISCA, and PLDI conferences. From these data, it is clear that the longer selection and publication process of journals means that they cannot be as current as conference publica

OCR for page 135
Academic Careers for Experimental Computer Scientists and Engineers tions. Rather than having about 30 percent of their references to recent papers (last two years), journals have on average only about 16 percent of their references to papers this recent. The average age of a reference is also older. One of the reasons often given (by researchers as well as university administrators) for preferring journal over conference publication is the more critical reviewing and more permanent record offered by the former. If the journal papers were more critically reviewed and therefore of higher quality, it might be expected that they would be cited significantly more frequently than conference papers. However, this is not borne out by the data. For example, among conferences the citations to other conference papers outnumber the citations to journals by 1.6 to 1 (45 percent versus 28 percent). Although the citations among journal papers are evenly divided, this appears to reflect the much older median age of a citation. For example, among the conference papers with citations older than four years, the citations to journal and conference papers are also evenly balanced. Thus, it appears that many researchers do not perceive a significant difference in quality or importance between conference and journal publication. It is worth noting that a positive feedback loop may have occurred in ECSE. In particular, the need for rapid publication has driven many academic experimentalists away from journal publication, perhaps resulting in a reduction of the pressure on journals to publish.2 By contrast, other fast-moving disciplines make use of journals with rapid publication times; an informal survey of materials science and biotechnology journals indicates mean publication times on the order of several months, rather than the years that characterize CS&E journals. SELECTIVITY Among the smaller fraction of experimental computer science researchers who favored journal publication over conference publication (approximately one-quarter of the respondents to the survey in Appendix A), nearly half listed the more critical reviewing of journals as their motivation. Although the journal reviewing process may be more thorough, 2   This is not to say that journal editors are satisfied with the speed with which journal articles can be refereed, revised, and ultimately published—quite the contrary is true.

OCR for page 135
Academic Careers for Experimental Computer Scientists and Engineers TABLE B.4 Acceptance Rates for Selected Conferences (%) Conferencea Year 1 Year 2 Year 3 Average SIGGRAPH 21 19 20 20 ASPLOS 24 30 16 23 ISCA 14 19 23 19 PLDI 17 17 17 18 a SIGGRAPH, conference of the ACM Special Interest Group on Graphics; ASPLOS, conference on Architectural Support for Programming Languages and Operating Systems; ISCA, International Symposium on Computer Architecture; PLDI, conference on Programming Language Design and Implementation. the prestigious conferences are highly selective. Table B.4 shows the acceptance rates for the last three occurrences of several top experimental conferences (ASPLOS is biennial, whereas the others are annual). It is much harder to determine the acceptance rates for journals, because comparable statistics are not kept and the long reviewing process makes it difficult to determine the final outcome of a paper. The committee did examine the submission and publication rates for TOPLAS and found that the average acceptance rate over a five-year period was 27 percent of the papers submitted (this includes revised papers). An informal survey of editors of other major ACM and Institute of Electrical and Electronics Engineers (IEEE) journals found acceptance rates that vary from about 25 to 35 percent. The private journals tend to be in the same range, although at the higher end.