Appendix B Comparing Journal and Conference Publication
This appendix compares several different quantitative measures of the acceptance and refereeing process (i.e., timeliness and selectivity) to determine how well the different publication media achieve the goals that experimental computer science and engineers desire, how these publication media might be improved, and what these publications tell us about the success of various academic researchers.
TIME TO PUBLICATION
As mentioned earlier, long delays to publication are a major hindrance to progress in experimental computer science and engineering (ECSE).1 Long publication delays also affect recognition and can be a problem in considering tenure and promotion cases. Journal editors try their best to minimize delay but are often stymied in their efforts by referees who do not return reviews in a timely fashion. The committee examined the time from submission to acceptance and publication for a variety of leading conferences and journals.
Table B.1 presents the times from submission to acceptance to publication for some major conferences; by comparison, the times for
TABLE B.1 Time from Submission to Acceptance and Publication of Papers for Selected Conferences and Journals
|
Time to Acceptance (months) |
Time to Publication (months) |
Conferencea |
||
PLDI |
2.3 |
7.3 |
ISCA |
2.6 |
6.2 |
ASPLOS |
2.5 |
7.0 |
SOSP |
3.0 |
7.8 |
OOPSLA |
2.6 |
7.2 |
SIGGRAPH |
2.3 |
6.8 |
Average |
2.5 |
7.0 |
Journalb |
||
TOPLAS |
24.0 |
32.4 |
TOCS |
17.0 |
21.3 |
IEEE TOC |
N/A |
32.2 |
IEEE TSE |
N/A |
29.2 |
a PLDI, conference on Programming Language Design and Implementation; ISCA, International Symposium on Computer Architecture; ASPLOS, conference on Architectural Support for Programming Languages and Operating Systems; SOSP, Symposium on Operating System Principles; OOPSLA, conference on Object-Oriented Programming Systems, Languages, and Applications; SIGGRAPH, conference of the ACM Special Interest Group on Graphics. b TOPLAS, Transactions on Programming Languages and Systems (published by ACM); TOCS, Transactions on Computer Systems (published by ACM); TOC, Transactions on Computers (published by IEEE; data for 1988 to 1992, special issues omitted); and TSE, Transactions on Software Engineering (published by IEEE; data for 1988 to 1992, special issues omitted). |
journal submission and publication are much longer. Table B.1 also indicates the comparable times for two of the primary Association of Computing Machinery (ACM) journals that publish ECSE research. Notice that the conferences listed, except for ASPLOS, are annual and the journals quarterly.
For Transactions on Programming Languages and Systems (TOPLAS), it takes approximately 10 times longer to acceptance and 5 times longer to publication than conferences require, whereas Transactions on Computer Systems (TOCS) requires about 6 times longer for acceptance and 2.7 times longer for publication than conferences. (For TOCS the average was significantly reduced by the use of special issues that offered fast-track publication to selected, top-rated conference papers.)
Multiple sequences of refereeing and revision account for a large
TABLE B.2 Publication Time and Manuscript Revision History for TOPLAS
Revisions Required |
Fraction of Submissions (%) |
Time to Acceptance (months) |
Time to Publication (months) |
1 |
50 |
18.0 |
26.4 |
2 |
45 |
30.0 |
37.2 |
3 |
5 |
37.2 |
46.8 |
TABLE B.3 Age of References in Journal and Conference Publications
Publicationa |
Fraction of References to Papers Less Than Two Years Old (%) |
Median Age of Reference (years) |
ASPLOS (Conference) |
32 |
3 |
ISCA (Conference) |
35 |
3 |
PLDI (Conference) |
26 |
4 |
TOCS (Journal) |
20 |
4.5 |
TOPLAS (Journal) |
12 |
5 |
a ASPLOS, conference on Architectural Support for Programming Languages and Operating Systems; ISCA, International Symposium on Computer Architecture; PLDI, conference on Programming Language Design and Implementation; TOCS, Transactions on Computer Systems (published by ACM); TOPLAS, Transactions on Programming Languages and Systems (published by ACM). |
part of the delay in journal publication. For example, Table B.2 presents data from three years of TOPLAS. Even if only the 50 percent of submissions that require merely one revision are considered, journal acceptance requires about 6 times longer, and publication 3.7 times longer, than the average conference, which confirms that conferences offer a substantial advantage in timeliness.
With the rapid advances in experimental computer science, timeliness is critical. One way of seeing how important timeliness is and how it is affected by publication time is to look at the citations in published papers.
Table B.3 contains data computed from looking at all the references in an entire year for TOCS and TOPLAS plus all the references in one year's ASPLOS, ISCA, and PLDI conferences. From these data, it is clear that the longer selection and publication process of journals means that they cannot be as current as conference publica
tions. Rather than having about 30 percent of their references to recent papers (last two years), journals have on average only about 16 percent of their references to papers this recent. The average age of a reference is also older.
One of the reasons often given (by researchers as well as university administrators) for preferring journal over conference publication is the more critical reviewing and more permanent record offered by the former. If the journal papers were more critically reviewed and therefore of higher quality, it might be expected that they would be cited significantly more frequently than conference papers.
However, this is not borne out by the data. For example, among conferences the citations to other conference papers outnumber the citations to journals by 1.6 to 1 (45 percent versus 28 percent). Although the citations among journal papers are evenly divided, this appears to reflect the much older median age of a citation. For example, among the conference papers with citations older than four years, the citations to journal and conference papers are also evenly balanced. Thus, it appears that many researchers do not perceive a significant difference in quality or importance between conference and journal publication.
It is worth noting that a positive feedback loop may have occurred in ECSE. In particular, the need for rapid publication has driven many academic experimentalists away from journal publication, perhaps resulting in a reduction of the pressure on journals to publish.2 By contrast, other fast-moving disciplines make use of journals with rapid publication times; an informal survey of materials science and biotechnology journals indicates mean publication times on the order of several months, rather than the years that characterize CS&E journals.
SELECTIVITY
Among the smaller fraction of experimental computer science researchers who favored journal publication over conference publication (approximately one-quarter of the respondents to the survey in Appendix A), nearly half listed the more critical reviewing of journals as their motivation.
Although the journal reviewing process may be more thorough,
TABLE B.4 Acceptance Rates for Selected Conferences (%)
Conferencea |
Year 1 |
Year 2 |
Year 3 |
Average |
SIGGRAPH |
21 |
19 |
20 |
20 |
ASPLOS |
24 |
30 |
16 |
23 |
ISCA |
14 |
19 |
23 |
19 |
PLDI |
17 |
17 |
17 |
18 |
a SIGGRAPH, conference of the ACM Special Interest Group on Graphics; ASPLOS, conference on Architectural Support for Programming Languages and Operating Systems; ISCA, International Symposium on Computer Architecture; PLDI, conference on Programming Language Design and Implementation. |
the prestigious conferences are highly selective. Table B.4 shows the acceptance rates for the last three occurrences of several top experimental conferences (ASPLOS is biennial, whereas the others are annual).
It is much harder to determine the acceptance rates for journals, because comparable statistics are not kept and the long reviewing process makes it difficult to determine the final outcome of a paper. The committee did examine the submission and publication rates for TOPLAS and found that the average acceptance rate over a five-year period was 27 percent of the papers submitted (this includes revised papers). An informal survey of editors of other major ACM and Institute of Electrical and Electronics Engineers (IEEE) journals found acceptance rates that vary from about 25 to 35 percent. The private journals tend to be in the same range, although at the higher end.