The committee found it challenging to identify reliable and robust nontraditional indicators for assessing a nation’s technological research capabilities specific to the computing performance challenges outlined in Chapter 1. After significant methodological consideration and debate, the committee conducted a pilot study to determine whether a bibliometric analysis of papers at select prestigious conferences in advanced computing could provide a useful snapshot of a nation’s capabilities in specific technology areas critical to the study’s charge.
The committee noted the strengths and challenges of a methodology that is both objective (e.g., conference publication data) and subjective (e.g., using committee expertise to identify specific conferences for analysis). By excluding traditional journal publications—as well as papers from conferences not considered to be representative of the most relevant and leading research—some relevant research may have been excluded; alternative samplings of conferences could also yield different results. On the other hand, a selective sampling of conferences may better support a more focused assessment of research efforts across specific technology areas (i.e., semiconductors and nanoscale devices and circuits, architecture, programming systems, and applications).
An ideal analysis would include all papers relevant to the computing performance challenges outlined in Chapter 1, whether published at conferences or in traditional journals, weighted by citations and impact factors, as well as expert judgment.1 Given limitations in time and resources, however, this was simply not feasible. Further, the committee’s analysis represents only one of likely many different and potentially useful measures of a nation’s research capabilities and innovativeness. These factors considered, the committee’s methodological approach and insights from its assessment are presented here as a pilot effort that will, no doubt, benefit from deeper, subsequent exploration by others.
F.1 Methodological Overview of Conference Paper Authorship Analysis
To assess a nations’ technology-specific research capabilities, the committee analyzed paper authorship—specifically, the geographical locations of authors—in many of the top technical conferences in four research areas most closely related to the technological challenges outlined in Chapter 1. These relate to the computing performance challenge and the shift to multicore processors: semiconductor devices and circuits, computer architecture, programming systems, and applications.
F.1.1 Rationale for Conferences as a Preferred Venue
In the computing community—unlike many other science and engineering disciplines—conference papers are often the publication venue of choice rather than journals.2 In fact, highly selective computer science
1Council of Canadian Academies, 2012, Informing Research Choices: Indicators and Judgment – The Report of the Expert Panel on Science Performance and Research Funding (available at http://www.scienceadvice.ca/uploads/eng/assessments%20and%20publications%20and%20news%20releases/Science%20performance/SciencePerformance_FullReport_EN_Web.pdf).
2David Patterson, Lawrence Snyder, and Jeffrey Ullman, 1999, “Best Practices Memo: Evaluating Computer Scientists and Engineers for Promotion and Tenure” in Computing Research News, June. Available at http://www.cra.org/uploads/documents/resources/bpmemos/tenure_review.pdf. Last accessed on August 15, 2012.
conferences often have higher citation indices and greater venue impact compared to related computing journals.3,4 The importance of conference publications within this community is consistent with a 2011 NRC report, A Data-Based Assessment of Research-Doctorate Programs in the United States,5 which indicates that “for the field of computer science, refereed conference papers are an important form of scholarship.”
In this report, conferences are the preferred venue over journals for the computer science-focused areas described in Sections F.1.3–F.1.6 for several reasons. First, these conferences tend to have a much shorter time from submission to publication than journals in the area, resulting in the most recent, significant innovations appearing at the conferences first.6 Second, conferences provide a larger sample size than journals of highly recognized recent top-quality research. Third, the conferences identified by the committee often have more focused research interests compared to journals that would publish related (albeit less recent and possibly less regarded), but broader works.7
For example, in advanced architecture research, relevant to the challenges described in Chapter 1, architecture papers appear in a wider range of journals that include more than just computer architecture (e.g., IEEE Transactions on Computers, ACM Transactions on Computer Systems, and IEEE Transactions on Parallel and Distributed Systems). Achieving the same level of topical focus would require disaggregation of the journal data on a paper-by-paper basis. This is simply not feasible for the wide range of conferences selected and papers analyzed for this report. Thus, analysis of conference papers allows a more focused assessment of the research areas identified by the committee as critical for sustaining computing performance and the shift to multicore processors, as opposed to computer science generally.
Similar to prestigious journals, premier conferences in each of the targeted hardware, architecture, and software research areas are highly competitive and conference submissions are rigorously peer-reviewed. Conferences are also competitive publishing venues because representation is professionally beneficial. Conferences provide an opportunity for researchers to share new research, to learn from others, and to gain exposure to recent and significant research efforts.8,9
F.1.2 Determination of a Nation’s Paper Contributions at Conferences
As a proxy indicator for a nation’s technology-specific research capabilities, the committee analyzed the weighted distribution of authors for research papers given at the conferences listed and described in Sections F.1.3–F.1.6. To do this, the committee noted the home nation for each author (defined as the geographic location of that author’s affiliation listed on each paper’s title page) and computed each nation’s weighted authorship contribution to each conference paper. It is important to note that this analysis does not distinguish between a U.S.-based author who is a U.S. citizen and a U.S.-based author who is not a U.S. citizen (and may eventually return to his or her home country), which may, in some cases, diminish a nation’s research representation at the sampled conferences.10
3It is worth noting that industry participation in these conferences may be limited by the demise of central research labs, less emphasis on outside presentations and publications, and reluctance to report on the most important research that companies are performing.
4As one example, CiteSeer, which keeps statistics about computer science publications, reports a venue impact of 0.14, 0.08, and 0.6 for three architecture research conferences analyzed in this report (High-Performance Computer Architecture, International Symposium on Computer Architecture, and the International Symposium on Microarchitecture, respectively) compared to 0.02 and 0.01 for two related architecture research journals: IEEE Transactions on Computers and IEEE Transactions on Parallel and Distributed Systems, respectively. See http://citeseer.ist.psu.edu/stats/venues. Last accessed on August 15, 2012.
6This is particularly relevant to the analysis of recent conference papers in Appendix F.2.
7One exception of a journal that is equally regarded with a similarly themed premier conference is the IEEE Journal of Solid-State Circuits (JSSC). JSSC publishes approximately 200 papers per year, with the papers typically being 10-15 pages long. By comparison, the International Solid-State Circuits Conference (ISSCC) typically publishes the same number of papers, but each paper is 3 pages long. As both venues are highly regarded and may represent a similar sample of annual papers, a more in-depth analysis of contributions in this area would benefit by addressing both ISSCC and JSSC.
8David Patterson, Lawrence Snyder, and Jeffrey Ullman, 1999, “Best Practices Memo: Evaluating Computer Scientists and Engineers for Promotion and Tenure” in Computing Research News, June. Available at http://www.cra.org/uploads/documents/resources/bpmemos/tenure_review.pdf. Last accessed on August 15, 2012.
102010 National Science Foundation data show that the share of non-U.S. citizens receiving U.S. doctoral degrees in natural sci-
A nation’s weighted11 contribution to each conference paper is merely the number of authors geographically located in (or professionally affiliated with) that particular nation divided by the total number of authors of the paper. Thus, the weighted contributions for a given paper always sum to one. To compute the weighted percentage of papers contributed by a nation at a given conference, each nation’s weighted contributions to each conference paper are summed and then divided by the total number of papers given at that conference. The committee believes this measurement—as opposed to total papers or total authors—better reflects a nation’s authorship contribution.12 This measure also has the distinct advantage that it is public information13 and can be extended and reproduced by others.
Based on the technical challenges outlined in Chapter 1, the committee identified four research areas critical to addressing the computing performance challenge and the shift to multicore processors: semiconductor devices and circuits, computer architecture, programming systems, and applications. To compute the weighted percentage of papers contributed by a nation in each of these research areas, each nation’s weighted contributions to each conference paper are summed across all conferences assigned to a particular research area (discussed in Sections F.1.3–F.1.6) and then divided by the total number of papers given at those conferences.14 All but three of the conferences analyzed by the committee are sponsored or published by either (or both) the Institute of Electrical and Electronics Engineers (IEEE) or the Association for Computing Machinery (ACM), the two preeminent international technical societies in electrical engineering and computing. Based on committee members’ opinions and knowledge of the fields, the following sections identify a limited number of top conferences that make available new and interesting research germane to the study’s charge.
F.1.3 Semiconductor Devices and Circuits Conferences
As described in Chapter 1, the end of Dennard scaling has placed greater pressure on innovative devices and circuits to deliver more energy-efficient technologies for building microprocessors. To explore the research capabilities in these areas, the committee analyzed papers from three conferences—two in semiconductor and nanoscale devices and one in semiconductor circuits following the methodology described earlier in Section F.1.2. These conferences are described below.
- International Electron Devices Meeting (IEDM). As stated on the conference Web site, IEDM is “the world’s pre-eminent forum for reporting technological breakthroughs in the areas of semiconductor and electronic device technology, design, manufacturing, physics, and modeling. IEDM is the flagship conference for nanometer-scale CMOS (complementary-symmetry metal-oxide-semiconductor) transistor technology, advanced memory, displays, sensors, MEMS (microelectromechanical systems) devices, novel quantum and nano-scale devices and phenomenology, optoelectronics, devices for power and energy harvesting, high-speed devices, as well as process technology and device modeling and simulation. The conference scope not only encompasses devices in silicon, compound and organic semiconductors, but also in emerging material systems. IEDM is truly an international conference, with strong representation from speakers from around the globe.”15 In 2011, IEDM included 36 sessions encompassing more than 200 papers. IEDM is sponsored by the IEEE.
ences and engineering is high and increasing at a higher rate than for U.S. citizens (available at http://www.nsf.gov/statistics/seind10/pdf/overview.pdf; last accessed on September 2, 2012). Additionally, a 2007 report by the Oak Ridge Institute for Science and Education, Stay Rates of Foreign Doctorate Recipients from U.S. Universities (available at http://orise.orau.gov/files/sep/stayrates-foreign-doctorate-recipients-2007.pdf) found that two-thirds of foreign citizens who received science or engineering doctorates from U.S. universities in 2005 continued to live in the United States in 2007.
11In an unweighted analysis, two countries will receive 50 percent of a particular paper’s contribution to the conference, even if nine coauthors are located in one country and one coauthor is located in the other country.
12Examination of the country of origin of members of conference programming committees may be a useful future activity and provide additional insight for assessing a nation’s interest and capabilities in a particular technological field.
13While bibliometric databases, such as SciVerse Scopus and Web of Science, provide some conference publication and citation data, comprehensive and consistent data for each conference across the time periods analyzed in the report do not exist. For example, Scopus includes only limited or no coverage of ECOOP, Eurographics, OSDI, SC, SOSP, VLDB, and WWW, and large gaps in annual coverage exist for ISCA, MICRO, POPL, and PPoPP.
14This allows all amassed papers in each research area to be weighted equally. In contrast, by first calculating a nation’s weighted percentage of contributed papers for each conference (as reported in Appendix G) and then averaging across all conferences assigned to a particular research area, a bias is introduced that could skew the overall average in favor of those conferences with larger numbers of presented papers.
- International Conference on Nanotechnology (NANO). From the conference Web site: “NANO is the flagship IEEE conference in Nanotechnology, which makes it a must for students, educators, researchers, scientists and engineers alike, working at the interface of nanotechnology and the many fields of electronic materials, photonics, bio- and medical devices, alternative energy, environmental protection, and multiple areas of current and future electrical and electronic applications. In each of these areas, NANO is the conference where practitioners will see nanotechnologies at work in both their own and related fields, from basic research and theory to industrial applications.”16 In 2011, NANO included more than 400 papers. NANO is sponsored by the IEEE.
- International Solid-State Circuits Conference (ISSCC). As stated on the conference Web site, ISSCC is “the premier forum for the presentation of advances in solid-state circuits and systems-on-a-chip.”17 ISSCC topics include advanced memory circuits, low-power circuits, high-speed signaling, and microprocessors, among many others. In 2011, ISSCC included 28 sessions encompassing more than 200 papers. ISSCC is sponsored by the IEEE.
F.1.4 Computer Architecture Conferences
Computer architecture includes the design and study of computer hardware implementations and computer design at the hardware-software boundary. Computer architects seek to make computers faster, lower power, cheaper, more reliable, and easier to program. Many computer architecture researchers focus on parallel and multicore systems. The committee analyzed conference papers from four top-flight conferences, described below.
- International Symposium on Architectural Support for Programming Languages and Operating Systems (ASPLOS). ASPLOS “is the premier forum for multidisciplinary systems research, spanning hardware, computer architecture, compilers, languages, operating systems, networking, and applications,” 18 and includes papers on parallel hardware and software. In 2011, ASPLOS included 14 sessions with 32 papers. ASPLOS is sponsored by the ACM.
- International Symposium on High Performance Computer Architecture (HPCA). HPCA covers many of the same topics as ISCA and MICRO.19 In 2011, HPCA included 14 sessions and 46 papers. HPCA is sponsored by the IEEE.
- International Symposium on Computer Architecture (ISCA). As stated on the conference Web site, ISCA is “the premier forum for new ideas and experimental results in computer architecture,”20 including parallel architecture and multicore systems. In 2011, ISCA included 14 sessions with 40 papers. ISCA is sponsored by ACM and the IEEE.
- International Symposium on Microarchitecture (MICRO). As stated on the conference Web site, MICRO “brings together researchers in fields related to microarchitecture, compilers, chips, and systems for technical exchange on traditional microarchitecture topics and emerging research areas.”21 In 2011, MICRO included 13 sessions with 44 papers. MICRO is sponsored by ACM and the IEEE.
Roughly, the same community of researchers publishes in and attends the conferences described above, although ASPLOS includes additional research areas on the boundary between computer architecture, programming languages, and operating systems.
F.1.5 Programming Systems Conferences
In this report, the committee focuses on (1) programming systems that encompass programming language design and implementation, and (2) programming tools, including programming models, languages, compilers, runtime systems, and virtual machines for parallel systems that are necessary to enable applications to exploit emerging silicon trends and chip architectures. The following five top conferences were analyzed, which collectively cover a range of programming system technologies:
- European Conference on Object-Oriented Programming (ECOOP). ECOOP covers “topics on object-oriented technologies, software development, systems, languages and
applications.”22 ECOOP was established in 1987. ECOOP and OOPSLA are peer conferences, were established within 1 year of each other, and have followed the same historical trends on topics. Whereas many of the other conferences the committee sampled are often hosted in the United States or Canada, ECOOP has only ventured outside of Europe twice (1990, 2012). In 2011, ECOOP included 9 sessions and 26 papers. Since 2007, ECOOP has been sponsored by ACM.
- Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA). OOPSLA “embraces all aspects of software construction and delivery” and is “a premier forum for software innovation.”23 OOPSLA started in 1986 when object-oriented programming systems were emerging to be a forum for researchers and practitioners to explore this new paradigm. Object-oriented programming subsequently became a dominant paradigm. Now OOPSLA is much broader and covers the same topics as PLDI and POPL. In 2011, OOPSLA included 17 sessions with 61 papers (the most in its history). OOPSLA is sponsored by the ACM.
- Programming Language Design and Implementation (PLDI). PLDI focuses “on the design, implementation, development, and use of programming languages. [It] emphasizes innovative and creative approaches to compile-time and runtime technology; novel language designs and features; and results from implementations.”24 Parallel programming systems are a significant component of PLDI. In 2011, PLDI included 20 sessions with 55 papers. PLDI is sponsored by the ACM.
- Symposium on Principles of Programming Languages (POPL). POPL is the leading “forum for the discussion of all aspects of programming languages and systems, with emphasis on how principles underpin practice.”25 POPL includes research on the principles of parallel programming systems. In 2011, POPL included 16 sessions with 49 papers. POPL is sponsored by the ACM.
- Symposium on Principles and Practice of Parallel Programming (PPoPP). “PPoPP is a forum for leading work on all aspects of parallel programming, including foundational and theoretical aspects, techniques, tools, and practical experiences.”26 Conference topics include work on concurrent and parallel (e.g., multicore, heterogeneous, and distributed) systems. In 2011, PPoPP included 8 sessions with 26 papers. PPoPP is sponsored by the ACM.
F.1.6 Applications Conferences
Computer applications are a tremendously broad area encompassing topics such as scientific computing, security, distributed and cloud computing, databases, and artificial intelligence. Representing the full range of these areas is beyond the scope of this report. However, since parallel application development is central to the success of multicore systems, the committee sampled seven conferences that have a strong focus on computational application needs and historically have depended on parallel and multicore systems, described below.
- Annual Conference of the European Association for Computer Graphics (Eurographics). Eurographics is a “Europe-wide professional Computer Graphics Association…that supports its members in advancing the state of the art in Computer Graphics and related fields such as Multimedia, Scientific Visualization, and Human Computer Interfaces.”27 In 2011, Eurographics included 14 sessions and 35 papers.
- Symposium on Operating Systems Design and Implementation (OSDI). OSDI “brings together professionals from academic and industrial backgrounds…[to discuss] the design, implementation, and implications of systems software.”28 In 2010, OSDI included 11 sessions with 32 papers. OSDI is sponsored by USENIX, the Advanced Computing Systems Association.
- International Conference on Computer Graphics and Interactive Techniques (SIGGRAPH). SIGGRAPH is the “premier international forum for disseminating new scholarly work in computer graphics and interactive techniques.”29 Graphics has a huge computational demand that has long been satisfied by parallel hardware, including both
graphics processors and multicore processors. In 2011, SIGGRAPH included 28 sessions with 115 papers. SIGGRAPH is sponsored by the ACM.
- International Conference for High Performance Computing, Networking, Storage, and Analysis (SC). The SC conference engages “the international community in high performance computing, networking, storage, and analysis.”30 SC is the premier conference on supercomputing applications and systems, and has been a leading venue focusing on parallel systems ranging from traditional supercomputers to many-cabinet machines to multicore to systems built from multicore hardware. In 2011, SC included 74 papers. SC is sponsored by the ACM and the IEEE.
- Symposium on Operating Systems Principles (SOSP). The SOSP conference focuses on “research related to the design, implementation, analysis, evaluation, and deployment of computer systems software…[taking] a broad view of the systems area and solicits contributions from many fields of systems practice, including, but not limited to, operating systems, file and storage systems, distributed systems, mobility, security, embedded systems, dependability, system management, peer-to-peer systems, and virtualization.”31 In 2011, SOSP included 9 sessions and 28 papers. SOSP is sponsored by the ACM.
- International Conference on Very Large Databases (VLDB). The VLDB conference covers “current issues in data management, database and information systems research.”32 Database applications are particularly instructive in this setting, because the database community has developed and matured parallel algorithms and technologies that exploit parallel hardware. In 2011, VLDB included 30 sessions and 104 papers (18.1 percent acceptance rate). VLDB is sponsored by the nonprofit organization Very Large Data Base Endowment Inc.
- International World Wide Web Conference (WWW). The WWW conference “aims to provide the world a premier forum for discussion and debate about the evolution of the Web, the standardization of its associated technologies, and the impact of those technologies on society and culture.”33 The explosive growth in the numbers of these applications and their scale and parallelism make them well suited to this study. In 2011, WWW included 27 sessions and 81 papers (12.5 percent acceptance rate). Conference proceedings are published by the ACM.
In total, the analyses of conference data presented in Sections F.2 and F.3 represent aggregated results from 19 conference series (i.e., ASPLOS, ECOOP, Eurographics, HPCA, IEDM, ISCA, ISSCC, MICRO, NANO, OOPSLA, OSDI, PLDI, POPL, PPoPP, SIGGRAPH, SC, SOSP, VLDB, and WWW). Four time points (1996, 2001, 2006, and 2011)34 were analyzed for each conference series, with exception to NANO that had only one time point, resulting in a total of 73 individual conferences comprising 4,719 papers and 23,859 authors.
F.1.7 Global Reach and Rationale for Conferences and Other Methodological Considerations
U.S. scientists did dominate many of the early innovations, creating the international technical societies and initiating most of the publication venues in the technology areas described in Sections F.1.3–F.1.6. The following data provide a starting point to examine the relationship between this historical U.S. advantage, as well as the location of many of the conference sites in the United States, and international research activity in these areas.
While proceedings of the selected conferences are all published in English and the conferences themselves are often held in the United States (with the exception of ECOOP, Eurographics, and WWW which were always held outside the United States for all years analyzed), more than one-third of the 73 specific conference venues analyzed by the committee were held outside the United States. As with any conference, whether held in the United States, Europe, or Asia, the location of the meeting has the potential to introduce travel biases for U.S.- and non-U.S.-based researchers. For example, insufficient travel funds or restrictive government policies could prevent qualified researchers from participating in conferences in other countries.
As one means of exploring potential travel biases due to the location of a top technical conference, the
34In cases where conference proceedings authorship data was not available or where conferences were held in alternating years and thus not available for the committee’s selected time points, conferences held in adjacent years were analyzed as specified in Appendix G.
committee compared regional researcher participation at 2008–2011 SIGGRAPH conferences (held in the United States) and SIGGRAPH Asia conferences, based on publication data using the methodology described in F.1.2 (see Figure F-1). While Asian participation was higher at the SIGGRAPH Asia conference than at the SIGGRAPH conference in the United States (except in 2010, when Asian participation rates were similar at both conferences), U.S. and European participation at both conferences was similar, regardless of whether the conference was held in the United States or Asia.
As an additional data point, the committee investigated the relationship between conference location and the average fraction of conference papers contributed by the United States. For all years combined, the United States contributed ~ 69 percent of papers at conferences located in the United States compared to ~62 percent when located elsewhere, as shown in Table F-1. This table also shows the U.S. share of conference papers for each of the time points analyzed in Sections F.2 and F.3, as well as the number of U.S.-located and non-U.S.-located conferences included in the committee’s analysis. Table F-2 compares the average U.S. share of conference papers at U.S.- and non-U.S.located conferences on a conference-by-conference basis. As shown in the table, U.S. paper contributions at HPCA, ISCA, MICRO, OSDI, SOSP, and VLDB, were similar or higher when the conference was located outside the United States.
These data, along with the SIGGRAPH and SIGGRAPH Asia results, suggest that for the analyses presented in this report, conference location does not induce a significant travel bias for U.S. researchers. However, as the full impact of all potential travel biases for every nation across all conferences and location cannot be measured, the above findings should be taken into consideration when interpreting the conference analyses presented in Sections F.2 and F.3.
Assessing objectively the influence of a conference is difficult because of the lack of easily comparable criteria. One starting point for this information is the Microsoft Academic Search database,35 a publicly available Web resource that indexes publication data for some of the conferences the committee considered and mines publication and venue citations for publications and authors. 36 Table F-3 shows publication statistics, including the number of papers published and number of citations to those published papers for all of the conferences the committee selected that appear in the Microsoft database, along with a selection of European and Asian conferences that might be considered competitors. The purpose of these data is not to justify the subset of conferences identified by the committee in Sections F.1.3–F.1.6, but rather to provide an objective measure of venue impact.
In general, the papers in the international conferences the committee selected have significantly higher citations per paper than the regional conferences found in the Microsoft database.37 The two exceptions to this are for NANO and for ECOOP. According to the Microsoft data, ECOOP has a relatively high citation rate compared with other programming system conferences, such as OOPSLA and PPoPP. The Microsoft data for OOPSLA does not reflect the expert opinion of the committee, and more careful examination of the data revealed that the number of papers reported for OOPSLA was more than twice the hand-counted technical papers.38 This bias was corrected in the Table. CiteSeer39 provides another objective view of programming conference impact, ranking the following venues from highest to lowest: POPL, 0.45; PLDI, 0.4; OOPSLA, 0.16; and ECOOP, 0.14. Regardless, the Microsoft data shows all the programming system venues as highly cited and, along with CiteSeer, are consistent with the committee’s selection of leading technical conferences in the four research areas described in Sections F.1.3–F.1.6.
As an additional measure, attendance at the three circuits and devices conferences selected by the committee (ISSCC, IEDM, and NANO) can be compared with three notable regional conferences in these areas [IEEE Asian Solid-State Circuits Conference (A-SSCC), IEEE European Solid-State Devices
36In considering conference citation analysis, it is important to recognize that citation counts are influenced by several different factors, including differences in database coverage, differences in citation practices among research fields, and the age distribution of the (cited) articles.
37For example, the IEEE “architecture”’ (ISCA, MICRO, HPCA, and ASPLOS) conferences selected for analysis report 25.6–34.9 citations per paper, compared with the non-IEEE or non-ACM European Conference on High Performance and Embedded Architecture and Compilation (HiPEAC) (4.2 citations per paper) and Asia-Pacific Computer Systems Architecture Conference (ASCAC) (2.2 citations per paper).
38The automated system count of publications appears to have included all non-research track papers, such as poster abstracts and workshop papers, which are rarely cited.
FIGURE F-1 Comparison of researcher participation (via publication authorship) at two conference venues: SIGGRAPH (U.S.based) and SIGGRAPH Asia (non-U.S.-based). Data compiled from SIGGRAPH and SIGGRAPH Asia between 2008 and 2011 (~600 papers and ~1300 authors).
TABLE F-1 Comparison of U.S. Paper Contributions at U.S.-located and Non-U.S.-located Conferences
|Average U.S. % Share of Conference Papers|
|U.S.-located conferences (64% of total conferences analyzed)||71||66.3||75.3||64.4||69.2|
no. of conferences analyzed:
|Non-U.S.-located conferences (36% of total conferences analyzed)||58.6||66.8||53.4||69.6||62.1|
no. of conferences analyzed:
TABLE F-2 Comparison of U.S. Paper Contributions at U.S.-located and Non-U.S.-located Conferences (By Individual Conferences)
|U.S. % Share of Conference Papers|
|HPCA||ISCA||MICRO||OSDI||POPL||PLDI||SIGGRAPH||SOSP||VLDB||Average U.S. % Share of Papers|
|When located elsewhere||83.7||93.1||86.5||86.8||44.2||79.0||52.7||87.7||58.3||74.7|
* Conferences located in the United States for all years analyzed include: ASPLOS, IEDM, ISSCC, NANO, OOPSLA, PPoPP, and SC. Conferences located outside the United States for all years analyzed by the committee include: ECOOP, Eurographics, and WWW
TABLE F-3 Conference Citation Analysis from Microsoft Scholar, June 2012
|Semiconductor Devices and Circuits||Papers||Citations||Citations/paper|
* OOPSLA citation data was corrected by hand counting technical papers that exclude non-research track papers, which are rarely cited. Data compiled from Microsoft Scholar June 2012.
Research Conference (ESSDERC), and IEEE European Solid-State Circuits Conference (ESSCIRC)]. In 2011, ISSCC, IEDM, and NANO had significant historical attendance (according to the IEEE): 3,000, 1,500, and 400, respectfully. In contrast, 2011 attendance at A-SSCC, ESSDERC, and ESSCIRC was between 300 and 350. While conference attendance statistics do not directly correlate with quality or influence, they indicate the level of interest in the technologies and ideas found in the conference.
The committee expects that an analysis including too many (and thus a higher proportion of lower quality)
conferences would underestimate the quality of leading research efforts and obscure authorship trends for leading research papers. On the other hand, too limited a sampling that overprescribes measures of conference quality and impact would also skew the assessment. Lastly, the breadth of research topics covered by a particular conference (and the same for journals that are oftentimes broader) should also be considered to avoid too broad a sweep of the technological field. In balancing these factors, the committee has identified a limited number of top technical conferences across four technology-specific research areas that, based on its expertise and deep domain knowledge of the field, are most critical to addressing the computing performance challenge described in Chapter 1.
F.1.8 Methodological Summary
In summary, the committee believes that high quality conferences attract leading researchers and showcase significant, recent research contributions to the field. In Sections F.2 and F.3, geographic distributions of conference authorship are quantified to provide a technology-specific assessment of national and regional research capabilities.40,41
The committee’s analysis is not intended to be representative of all scientific outputs across the four specified research areas; for example, it does not presume that all relevant technologies presented in journal publications are implicitly represented in the selected conferences. The committee also recognizes the significant and ongoing progress in bibliometric and scientometric approaches to assess the quantity, quality, and impact of scientific output. The assessment provided in subsequent sections is provided as an important first step toward new approaches to assess the global research landscape in specific advanced computing technologies.
F.2 Current National and Regional Advanced Research Capabilities in Four Key Technology Areas
In this section, conference representation is used as a proxy indicator of a nation’s current (2011) advanced research capabilities in each of the following key technology areas: semiconductor devices and circuits, computer architecture, programming systems, and applications. For each technology area, all nations with at least 1 percent conference representation are ranked based on their weighted authorship contributions (following the methodology described in Section F.1.2) in the targeted conferences previously described in Sections F.1.3–F.1.6. In addition, regional research capabilities are also provided for comparison.
As previously discussed, conference papers tend to have a much shorter time from submission to publication than computing journals in related areas, resulting in the most recent, significant innovation appearing at conferences first. Thus, an assessment of current conference research efforts is particularly relevant given the increasing rate at which scientific discoveries are made and then disseminated via the Web. Time series analyses of national and regional research capabilities are shown in Section F.3.
F.2.1 Advanced Semiconductor Devices and Circuit Research
Table F-4 shows national capabilities in advanced semiconductor and nanoscale devices (IEDM and NANO) research, as well as semiconductor circuits (ISSCC) research in 2011. The table shows all countries with at least 1 percent representation. The United States has a strong competitive position in both of these areas (50 percent in devices and 36 percent in circuits) followed by Japan, Taiwan, and Korea. China has only token representation in these conferences at this time. Figures F-2 and F-3 show the same data broken down by region.
In semiconductor devices, the United States has the highest representation with half of the papers, followed by Asia with less than one-third of the papers and Europe with even fewer.42 In circuits, the United States, Asia, and Europe all share approximately one third of the papers. NANO represents research that typically is targeted further in the future than those published in IEDM. At NANO the United States represents over 60
40This approach is consistent with a 2010 National Research Council (NRC) report, S&T Strategies of Six Countries: Implications for the United States, in which conference publication analysis was described as a “technology-specific indicator [that] gives a relatively accurate picture of [national] S&T standing.” Available at http://www.nap.edu/catalog.php?record_id=12920.
41A related analysis was also conducted in a 2000 NRC report, Experiments in International Benchmarking of U.S. Research Fields, which used U.S. contributions of papers at the annual Conference on Magnetism and Magnetic Materials as a measure of U.S. participation in magnetic materials research. See http://www.nap.edu/catalog.php?record_id=9784.
42U.S. representation in semiconductor devices is reduced from ~50 percent to ~38 percent when only IEDM publications are considered (see Figure F-7 and Table F-8). Despite this reduction, the United States remains well ahead of Japan (~17 percent).
TABLE F-4 Current (2011) National Capabilities in Advanced Semiconductor and Nanoscale Devices and Circuits Research (Measured by Percent Share of Conference Papers)
|Advanced Semiconductor Circuits Research|
|Advanced Semiconductor and Nanoscale Devices Research|
Data compiled from IEDM and NANO (semiconductor and nanoscale devices) and ISSCC (semiconductor circuits).
FIGURE F-2 Current (2011) regional capabilities in advanced semiconductor and nanoscale devices research (measured by percent share of conference papers). Data compiled from IEDM and NANO.
FIGURE F-3 Current (2011) regional capabilities in advanced semiconductor circuits research (measured by percent share of conference papers). Data compiled from ISSCC.
percent of all papers, while Asia and Europe represent less than 20 percent and 10 percent respectively.
F.2.2 Advanced Architecture Research
Table F-5 shows national and regional capabilities in advanced architecture research based on aggregated data from ASPLOS, HPCA, ISCA, and MICRO in 2011. This table includes all countries with at least 1 percent representation. Figure F-4 shows the same data broken down by region. U.S. research dominates that of any other nation or region and has a weighted average representation of more than 85 percent of the papers. No other nation or region contributes more than 3 percent or
7 percent of the papers, respectively. This result is perhaps not surprising, given the U.S. historical dominance in commercial microprocessors, including Intel, AMD, and IBM, as well as former commercial microprocessors from DEC, HP, and others. That Japan is not represented on this list suggests that Japanese universities and industry research institutions are not focused on mainstream computer architectures. While Japan has activity and expertise in the area, notably the custom processors from Fujitsu that are in the K supercomputer, their national research focus generally lies elsewhere.
TABLE F-5 Current (2011) National Capabilities in Advanced Architecture Research (Measured by Percent Share of Conference Papers)
|Advanced Architecture Research|
Data compiled from ASPLOS, HPCA, ISCA, and MICRO.
FIGURE F-4 Current (2011) regional capabilities in advanced architecture research (measured by percent share of conference papers). Data compiled from ASPLOS, HPCA, ISCA, and Micro.
F.2.3 Advanced Programming Systems Research
Table F-6 shows national capabilities in advanced programming systems research based on aggregated data from ECOOP, OOPSLA, PLDI, POPL, and PPoPP. This table includes all countries with at least 1 percent representation. Figure F-5 shows the same data broken down by region. In programming systems research, the United States dominates with approximately 60 percent of the papers. The national breakdown in Table F-6 shows that aside from the United States, the national distribution of papers is diverse.
Figure F-5 shows that in programming systems, Europe is a distant second to the United States with about 25 percent representation, while all of Asia accounts for less than 10 percent. Disaggregating the data (see Appendix G), the United States has an even stronger position in PPoPP and PLDI, accounting for about 75 percent representation in each of them. These conferences are practical in nature, with papers presenting prototype software systems and applications running on real hardware platforms. In POPL, a more theoretical conference, the United States has a little less than 45 percent of the papers, on par with European presentation. Asia has less than 10 percent of the papers, with greater representation in applications than programming systems. In ECOOP, the United States accounted for about 30 percent representation, putting it 15 percentage points behind Europe.
TABLE F-6 Current (2011) National Capabilities in Advanced Programming Systems Research (Measured by Percent Share of Conference Papers)
|Advanced Programming Systems Research|
Data compiled from ECOOP, OOPSLA, PLDI, POPL, and PPoPP.
FIGURE F-5 Current (2011) regional capabilities in advanced programming systems research (measured by percent share of conference papers). Data compiled from ECOOP, OOPSLA, PLDI, POPL, and PPoPP.
2.2.4 Advanced Applications Research
Table F-7 shows national capabilities in advanced applications research based on aggregated data from Eurographics, OSDI, SIGGRAPH, SC, SOSP, VLDB, and WWW in 2011. Figure F-6 shows this data broken down into regions. In applications research, the U.S. accounted for almost 64 percent of papers, followed by Germany, Canada, and China, with 5-6 percent representation each.
Disaggregated data show that the United States is also the lead paper contributor in SOSP (~92 percent compared with ~4 percent in Asia), SC (~80 percent compared with 9 percent for both Asia and Europe), and WWW (~72 percent compared with ~13 percent in Europe and ~8 percent in Asia) and very strong leads in SIGGRAPH and VLDB (~53 percent, putting the United States ahead of Europe by approximately 30 points in both conferences), suggesting the United States maintains strong core competencies in parallel applications.
TABLE F-7 Current (2011) National Capabilities in Advanced Applications Research (Measured by Percent Share of Conference Papers)
|Advanced Applications Research|
Data compiled from Eurographics, OSDI, SIGGRAPH, SC, SOSP, VLDB, and WWW.
FIGURE F-6 Current (2011) regional capabilities in advanced applications research (measured by percent share of conference papers). Data compiled from Eurographics, OSDI, SIGGRAPH, SC, SOSP, VLDB, and WWW.
F.2.5 Summary of Current National Technological Leadership
In each of the four key technical areas, the United States holds the lead, with particularly strong representation in architecture research ~85 percent (compared to ~5 and 7 percent representation by Asia and Europe, respectfully). In programming systems research, the United States maintains the lead at ~36 percent followed by Europe at ~29 percent and Asia at ~8 percent. The strongest European paper contributors were Germany and the UK at ~12 and 10 percent, respectfully. In applications research, the United States leads paper contributions at ~64 percent followed by Europe at ~17 percent and Asia at ~11 percent. This data suggests that the U.S. is particularly strong in design and systems engineering. While the United States maintains a strong lead in semiconductor and nanoscale devices (~50 percent) compared to Asia (~28 percent) and Europe (~16 percent), its contributions in semiconductor circuits research are comparable to Europe and Asia (each separated by ~7 percentage points).
F.3 Longitudinal Changes in the Global Research Landscape
To assess how the competitive research landscape has changed over time, the committee examined the competitive metrics (described in Section F.1) for the same set of conferences in the previous section in 5-year increments over a 15-year span from 1996 to 2011.
Tables F-8 through F-11 show the relative representation of different nations in each of the technical areas of advanced computing research from 1996 to 2011 in 5-year increments, ranked by position in 2011. Each of the tables includes only those nations that have at least a 1 percent representation for at least one of the years. Countries that are not represented in 2011, but have at least 1 percent representation for any increment, are marked in italicized text. Figures F-7 through F-11 show the aggregate regional representation for each area of advanced research over the same time span. For the time-series data, the semiconductor devices area includes only IEDM, because historical data for NANO was not easily available.
In semiconductor devices research, U.S. representation has remained relatively stable with the largest gains made by Taiwan and Belgium (~ 5-6 percentage point increases each). In semiconductor circuits research, the United States shows a moderate decline, in tandem with an overall broadening in
TABLE F-8 (1996–) National Capabilities in Advanced Semiconductor and Devices and Circuits Research (Measured by Percent Share of Conference Papers)
|Semiconductors & Nanoscale Devices|
Data compiled from IEDM (semiconductor and nanoscale devices) and ISSCC (semiconductor circuits).
international representation. In this area largest leaps were made by Korea, Taiwan, and the Netherlands with 5–6 percentage point increases each. At the same time, Japan has dropped significantly in both semiconductor devices and in semiconductor circuits (~17 and 15 percent, respectfully).
In architecture research, the United States has maintained a significant lead, with no major advances by any other nation or region. In programming systems, the U.S. lead has been challenged somewhat by increases in Europe by small but steady gains by Israel, Switzerland, and the UK (as well as China, India, and Korea to a lesser degree).
In the application areas, U.S. representation has retained a stable lead over the 15-year period with no other nation ever contributing more than 8 percent (France, which contributed ~8 percent in 1996, has since dropped to ~2 percent in 2011). While representing only a small percentage of applications papers, China made a notable move from no representation in 1996 to ~4 percent in 2011.
TABLE F-9 (1996–2011) National Capabilities in Advanced Architecture Research (Measured by Percent Share of Conference Papers)
Data compiled from ASPLOS, HPCA, ISCA, and Micro.
TABLE F-10 (1996–2011) National Capabilities in Advanced Programming Systems Research (Measured by Percent Share of Conference Papers)
Data compiled from ECOOP, OOPSLA, PLDI, POPL, and PPoPP.
TABLE F-11 (1996–2011) National Capabilities in Advanced Applications Systems Research (Measured by Percent Share of Conference Papers)
Data compiled from Eurographics, OSDI, SIGGRAPH, SC, SOSP, VLDB, and WWW.
FIGURE F-7 (1996–2011) Regional capabilities in advanced semiconductor and nanoscale devices research (measured by percent share of conference papers). Data compiled from IEDM.
FIGURE F-8 (1996–2011) Regional capabilities in advanced semiconductor circuits research (measured by percent share of conference papers). Data compiled from ISSCC.
FIGURE F-9 (1996–2011) Regional capabilities in advanced architecture research (measured by percent share of conference papers). Data compiled from ASPLOS, HPCA, ISCA, and MICRO.
FIGURE F-10 (1996–2011) Regional capabilities in advanced programming systems research (measured by percent share of conference papers). Data compiled from ECOOP, OOPSLA, PLDI, POPL, and PPoPP.
FIGURE F-11 (1996–2011) Regional capabilities in advanced applications research (measured by percent share of conference papers). Data compiled from Eurographics, OSDI, SC, SIGGRAPH, SOSP, VLDB, and WWW.