Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 71
F
Pilot Study of Papers at Top Technical Conferences in
Advanced Computing
T he committee found it challenging to identify
reliable and robust nontraditional indicators for
assessing a nation's technological research
capabilities specific to the computing performance
challenges outlined in Chapter 1. After significant
time and resources, however, this was simply not
feasible. Further, the committee's analysis represents
only one of likely many different and potentially useful
measures of a nation's research capabilities and
innovativeness. These factors considered, the
methodological consideration and debate, the committee committee's methodological approach and insights from
conducted a pilot study to determine whether a its assessment are presented here as a pilot effort that
bibliometric analysis of papers at select prestigious will, no doubt, benefit from deeper, subsequent
conferences in advanced computing could provide a exploration by others.
useful snapshot of a nation's capabilities in specific
technology areas critical to the study's charge. F.1 Methodological Overview of Conference
The committee noted the strengths and challenges of Paper Authorship Analysis
a methodology that is both objective (e.g., conference
publication data) and subjective (e.g., using committee To assess a nations' technology-specific research
expertise to identify specific conferences for analysis). capabilities, the committee analyzed paper authorship--
By excluding traditional journal publications--as well as specifically, the geographical locations of authors--in
papers from conferences not considered to be many of the top technical conferences in four research
representative of the most relevant and leading areas most closely related to the technological challenges
research--some relevant research may have been outlined in Chapter 1. These relate to the computing
excluded; alternative samplings of conferences could performance challenge and the shift to multicore
also yield different results. On the other hand, a selective processors: semiconductor devices and circuits, computer
sampling of conferences may better support a more architecture, programming systems, and applications.
focused assessment of research efforts across specific
technology areas (i.e., semiconductors and nanoscale F.1.1 Rationale for Conferences as a Preferred Venue
devices and circuits, architecture, programming systems,
and applications). In the computing community--unlike many other
An ideal analysis would include all papers relevant science and engineering disciplines--conference papers
to the computing performance challenges outlined in are often the publication venue of choice rather than
Chapter 1, whether published at conferences or in journals.2 In fact, highly selective computer science
traditional journals, weighted by citations and impact
factors, as well as expert judgment.1 Given limitations in 20and%20publications%20and%20news%20releases/Science%20
performance/SciencePerformance_FullReport_EN_Web.pdf).
2
David Patterson, Lawrence Snyder, and Jeffrey Ullman, 1999,
1
Council of Canadian Academies, 2012, Informing Research "Best Practices Memo: Evaluating Computer Scientists and Engi-
Choices: Indicators and Judgment The Report of the Expert neers for Promotion and Tenure" in Computing Research News,
Panel on Science Performance and Research Funding (available June. Available at http://www.cra.org/uploads/documents/resourc-
at http://www.scienceadvice.ca/uploads/eng/assessments% es/bpmemos/tenure_review.pdf. Last accessed on August 15, 2012.
71
OCR for page 72
72 THE GLOBAL ECOSYSTEM IN ADVANCED COMPUTING
conferences often have higher citation indices and greater ACM Transactions on Computer Systems, and IEEE
venue impact compared to related computing journals.3,4 Transactions on Parallel and Distributed Systems).
The importance of conference publications within this Achieving the same level of topical focus would
community is consistent with a 2011 NRC report, A require disaggregation of the journal data on a
Data-Based Assessment of Research-Doctorate paper-by-paper basis. This is simply not feasible for
Programs in the United States,5 which indicates that "for the wide range of conferences selected and papers
the field of computer science, refereed conference papers analyzed for this report. Thus, analysis of conference
are an important form of scholarship." papers allows a more focused assessment of the
In this report, conferences are the preferred venue research areas identified by the committee as critical
over journals for the computer science-focused areas for sustaining computing performance and the shift
described in Sections F.1.3F.1.6 for several reasons. to multicore processors, as opposed to computer
First, these conferences tend to have a much shorter time science generally.
from submission to publication than journals in the area, Similar to prestigious journals, premier
resulting in the most recent, significant innovations conferences in each of the targeted hardware,
appearing at the conferences first.6 Second, conferences architecture, and software research areas are highly
provide a larger sample size than journals of highly competitive and conference submissions are
recognized recent top-quality research. Third, the rigorously peer-reviewed. Conferences are also
conferences identified by the committee often have more competitive publishing venues because
focused research interests compared to journals that representation is professionally beneficial.
would publish related (albeit less recent and possibly less Conferences provide an opportunity for researchers
regarded), but broader works.7 to share new research, to learn from others, and to
For example, in advanced architecture research, gain exposure to recent and significant research
relevant to the challenges described in Chapter 1, efforts.8,9
architecture papers appear in a wider range of
journals that include more than just computer F.1.2 Determination of a Nation's Paper Contributions
architecture (e.g., IEEE Transactions on Computers, at Conferences
3
It is worth noting that industry participation in these confer-
As a proxy indicator for a nation's technology-
ences may be limited by the demise of central research labs, less specific research capabilities, the committee analyzed the
emphasis on outside presentations and publications, and reluctance weighted distribution of authors for research papers given
to report on the most important research that companies are per- at the conferences listed and described in Sections F.1.3
forming. F.1.6. To do this, the committee noted the home nation
4
As one example, CiteSeer, which keeps statistics about com- for each author (defined as the geographic location of
puter science publications, reports a venue impact of 0.14, 0.08,
and 0.6 for three architecture research conferences analyzed in this
that author's affiliation listed on each paper's title page)
report (High-Performance Computer Architecture, International and computed each nation's weighted authorship
Symposium on Computer Architecture, and the International Sym- contribution to each conference paper. It is important to
posium on Microarchitecture, respectively) compared to 0.02 and note that this analysis does not distinguish between a
0.01 for two related architecture research journals: IEEE Transac- U.S.-based author who is a U.S. citizen and a U.S.-based
tions on Computers and IEEE Transactions on Parallel and Dis- author who is not a U.S. citizen (and may eventually
tributed Systems, respectively. See http://citeseer.ist.psu.edu/stats/
venues. Last accessed on August 15, 2012.
return to his or her home country), which may, in some
5
NRC, 2011, A Data-Based Assessment of Research-Doctorate cases, diminish a nation's research representation at the
Programs in the United States, Washington, D.C.: The National sampled conferences.10
Academies Press (available online at https://download.nap.edu/
rdp/index.html?)
6 8
This is particularly relevant to the analysis of recent conference David Patterson, Lawrence Snyder, and Jeffrey Ullman, 1999,
papers in Appendix F.2. "Best Practices Memo: Evaluating Computer Scientists and Engi-
7
One exception of a journal that is equally regarded with a simi- neers for Promotion and Tenure" in Computing Research News,
larly themed premier conference is the IEEE Journal of Solid-State June. Available at http://www.cra.org/uploads/documents/
Circuits (JSSC). JSSC publishes approximately 200 papers per resources/bpmemos/tenure_review.pdf. Last accessed on August
year, with the papers typically being 10-15 pages long. By com- 15, 2012.
9
parison, the International Solid-State Circuits Conference (ISSCC) NRC, 1994, Academic Careers for Experimental Computer
typically publishes the same number of papers, but each paper is 3 Scientists and Engineers, Washington, D.C.: The National
pages long. As both venues are highly regarded and may represent Academies Press (available online at http://www.nap.edu/open
a similar sample of annual papers, a more in-depth analysis of book.php?record_id=2236).
10
contributions in this area would benefit by addressing both ISSCC 2010 National Science Foundation data show that the share of
and JSSC. non-U.S. citizens receiving U.S. doctoral degrees in natural sci-
OCR for page 73
PILOT STUDY OF PAPERS AT TOP TECHNICAL CONFERENCES 73
A nation's weighted11 contribution to each by the committee are sponsored or published by either
conference paper is merely the number of authors (or both) the Institute of Electrical and Electronics
geographically located in (or professionally affiliated Engineers (IEEE) or the Association for Computing
with) that particular nation divided by the total number Machinery (ACM), the two preeminent international
of authors of the paper. Thus, the weighted contributions technical societies in electrical engineering and
for a given paper always sum to one. To compute the computing. Based on committee members' opinions and
weighted percentage of papers contributed by a nation at knowledge of the fields, the following sections identify a
a given conference, each nation's weighted contributions limited number of top conferences that make available
to each conference paper are summed and then divided new and interesting research germane to the study's
by the total number of papers given at that conference. charge.
The committee believes this measurement--as opposed
to total papers or total authors--better reflects a nation's F.1.3 Semiconductor Devices and Circuits Conferences
authorship contribution.12 This measure also has the
distinct advantage that it is public information13 and can As described in Chapter 1, the end of Dennard
be extended and reproduced by others. scaling has placed greater pressure on innovative devices
Based on the technical challenges outlined in and circuits to deliver more energy-efficient technologies
Chapter 1, the committee identified four research areas for building microprocessors. To explore the research
critical to addressing the computing performance capabilities in these areas, the committee analyzed
challenge and the shift to multicore processors: papers from three conferences--two in semiconductor
semiconductor devices and circuits, computer and nanoscale devices and one in semiconductor circuits
architecture, programming systems, and applications. To following the methodology described earlier in Section
compute the weighted percentage of papers contributed F.1.2. These conferences are described below.
by a nation in each of these research areas, each nation's
· International Electron Devices Meeting (IEDM).
weighted contributions to each conference paper are
As stated on the conference Web site, IEDM is
summed across all conferences assigned to a particular
"the world's pre-eminent forum for reporting
research area (discussed in Sections F.1.3F.1.6) and
technological breakthroughs in the areas of
then divided by the total number of papers given at those
semiconductor and electronic device technology,
conferences.14 All but three of the conferences analyzed
design, manufacturing, physics, and modeling.
IEDM is the flagship conference for nanometer-
ences and engineering is high and increasing at a higher rate than scale CMOS (complementary-symmetry metal-
for U.S. citizens (available at http://www.nsf.gov/statistics/
seind10/pdf/overview.pdf; last accessed on September 2, 2012). oxide-semiconductor) transistor technology,
Additionally, a 2007 report by the Oak Ridge Institute for Science advanced memory, displays, sensors, MEMS
and Education, Stay Rates of Foreign Doctorate Recipients from (microelectromechanical systems) devices, novel
U.S. Universities (available at http://orise.orau.gov/files/sep/stay- quantum and nano-scale devices and
rates-foreign-doctorate-recipients-2007.pdf) found that two-thirds phenomenology, optoelectronics, devices for
of foreign citizens who received science or engineering doctorates power and energy harvesting, high-speed devices,
from U.S. universities in 2005 continued to live in the United
States in 2007. as well as process technology and device
11
In an unweighted analysis, two countries will receive 50 per- modeling and simulation. The conference scope
cent of a particular paper's contribution to the conference, even if not only encompasses devices in silicon,
nine coauthors are located in one country and one coauthor is lo- compound and organic semiconductors, but also in
cated in the other country. emerging material systems. IEDM is truly an
12
Examination of the country of origin of members of confer- international conference, with strong
ence programming committees may be a useful future activity and
provide additional insight for assessing a nation's interest and representation from speakers from around the
capabilities in a particular technological field. globe."15 In 2011, IEDM included 36 sessions
13
While bibliometric databases, such as SciVerse Scopus and encompassing more than 200 papers. IEDM is
Web of Science, provide some conference publication and citation sponsored by the IEEE.
data, comprehensive and consistent data for each conference
across the time periods analyzed in the report do not exist. For
example, Scopus includes only limited or no coverage of ECOOP,
Eurographics, OSDI, SC, SOSP, VLDB, and WWW, and large reported in Appendix G) and then averaging across all conferences
gaps in annual coverage exist for ISCA, MICRO, POPL, and assigned to a particular research area, a bias is introduced that
PPoPP. could skew the overall average in favor of those conferences with
14
This allows all amassed papers in each research area to be larger numbers of presented papers.
15
weighted equally. In contrast, by first calculating a nation's See http://www.his.com/~iedm/. Last accessed on January 9,
weighted percentage of contributed papers for each conference (as 2012.
OCR for page 74
74 THE GLOBAL ECOSYSTEM IN ADVANCED COMPUTING
· International Conference on Nanotechnology ASPLOS included 14 sessions with 32 papers.
(NANO). From the conference Web site: "NANO ASPLOS is sponsored by the ACM.
is the flagship IEEE conference in · International Symposium on High Performance
Nanotechnology, which makes it a must for Computer Architecture (HPCA). HPCA covers
students, educators, researchers, scientists and many of the same topics as ISCA and MICRO.19
engineers alike, working at the interface of In 2011, HPCA included 14 sessions and 46
nanotechnology and the many fields of electronic papers. HPCA is sponsored by the IEEE.
materials, photonics, bio- and medical devices, · International Symposium on Computer
alternative energy, environmental protection, and Architecture (ISCA). As stated on the conference
multiple areas of current and future electrical and Web site, ISCA is "the premier forum for new
electronic applications. In each of these areas, ideas and experimental results in computer
NANO is the conference where practitioners will architecture,"20 including parallel architecture and
see nanotechnologies at work in both their own multicore systems. In 2011, ISCA included 14
and related fields, from basic research and theory sessions with 40 papers. ISCA is sponsored by
to industrial applications."16 In 2011, NANO ACM and the IEEE.
included more than 400 papers. NANO is · International Symposium on Microarchitecture
sponsored by the IEEE. (MICRO). As stated on the conference Web site,
· International Solid-State Circuits Conference MICRO "brings together researchers in fields
(ISSCC). As stated on the conference Web site, related to microarchitecture, compilers, chips, and
ISSCC is "the premier forum for the presentation systems for technical exchange on traditional
of advances in solid-state circuits and systems-on- microarchitecture topics and emerging research
a-chip."17 ISSCC topics include advanced memory areas."21 In 2011, MICRO included 13 sessions
circuits, low-power circuits, high-speed signaling, with 44 papers. MICRO is sponsored by ACM and
and microprocessors, among many others. In the IEEE.
2011, ISSCC included 28 sessions encompassing
more than 200 papers. ISSCC is sponsored by the Roughly, the same community of researchers
IEEE. publishes in and attends the conferences described
above, although ASPLOS includes additional research
F.1.4 Computer Architecture Conferences areas on the boundary between computer architecture,
programming languages, and operating systems.
Computer architecture includes the design and study
of computer hardware implementations and computer F.1.5 Programming Systems Conferences
design at the hardware-software boundary. Computer
architects seek to make computers faster, lower power, In this report, the committee focuses on (1)
cheaper, more reliable, and easier to program. Many programming systems that encompass programming
computer architecture researchers focus on parallel and language design and implementation, and (2)
multicore systems. The committee analyzed conference programming tools, including programming models,
papers from four top-flight conferences, described languages, compilers, runtime systems, and virtual
below. machines for parallel systems that are necessary to
· International Symposium on Architectural Support enable applications to exploit emerging silicon trends
for Programming Languages and Operating and chip architectures. The following five top
Systems (ASPLOS). ASPLOS "is the premier conferences were analyzed, which collectively cover a
forum for multidisciplinary systems research, range of programming system technologies:
spanning hardware, computer architecture,
compilers, languages, operating systems, · European Conference on Object-Oriented
networking, and applications," 18 and includes Programming (ECOOP). ECOOP covers "topics
papers on parallel hardware and software. In 2011, on object-oriented technologies, software
development, systems, languages and
16 19
See http://ieeenano2011.org/. Last accessed on January 9, See www.ece.lsu.edu/hpca-18/. Last accessed on January 9,
2012. 2012.
17 20
See http://isscc.org/. Last accessed on January 9, 2012. See Isca2012.ittc.ku.edu. Last accessed on January 9, 2012.
18 21
See research.microsoft.com/en-us/um/Cambridge/events/ See www.microarch.org/micro44/. Last accessed on January
asplos_2012. Last accessed on January 9, 2012. 9, 2012.
OCR for page 75
PILOT STUDY OF PAPERS AT TOP TECHNICAL CONFERENCES 75
applications."22 ECOOP was established in 1987. programming, including foundational and
ECOOP and OOPSLA are peer conferences, were theoretical aspects, techniques, tools, and practical
established within 1 year of each other, and have experiences."26 Conference topics include work on
followed the same historical trends on topics. concurrent and parallel (e.g., multicore,
Whereas many of the other conferences the heterogeneous, and distributed) systems. In 2011,
committee sampled are often hosted in the United PPoPP included 8 sessions with 26 papers. PPoPP
States or Canada, ECOOP has only ventured is sponsored by the ACM.
outside of Europe twice (1990, 2012). In 2011,
ECOOP included 9 sessions and 26 papers. Since F.1.6 Applications Conferences
2007, ECOOP has been sponsored by ACM.
· Object-Oriented Programming, Systems, Computer applications are a tremendously broad
Languages, and Applications (OOPSLA). area encompassing topics such as scientific computing,
OOPSLA "embraces all aspects of software security, distributed and cloud computing, databases, and
construction and delivery" and is "a premier artificial intelligence. Representing the full range of
forum for software innovation."23 OOPSLA these areas is beyond the scope of this report. However,
started in 1986 when object-oriented programming since parallel application development is central to the
systems were emerging to be a forum for success of multicore systems, the committee sampled
researchers and practitioners to explore this new seven conferences that have a strong focus on
paradigm. Object-oriented programming computational application needs and historically have
subsequently became a dominant paradigm. Now depended on parallel and multicore systems, described
OOPSLA is much broader and covers the same below.
topics as PLDI and POPL. In 2011, OOPSLA · Annual Conference of the European Association
included 17 sessions with 61 papers (the most in for Computer Graphics (Eurographics).
its history). OOPSLA is sponsored by the ACM. Eurographics is a "Europe-wide professional
· Programming Language Design and Computer Graphics Association. . . that supports
Implementation (PLDI). PLDI focuses "on the its members in advancing the state of the art in
design, implementation, development, and use of Computer Graphics and related fields such as
programming languages. [It] emphasizes Multimedia, Scientific Visualization, and Human
innovative and creative approaches to compile- Computer Interfaces."27 In 2011, Eurographics
time and runtime technology; novel language included 14 sessions and 35 papers.
designs and features; and results from · Symposium on Operating Systems Design and
implementations."24 Parallel programming Implementation (OSDI). OSDI "brings together
systems are a significant component of PLDI. In professionals from academic and industrial
2011, PLDI included 20 sessions with 55 papers. backgrounds. . . [to discuss] the design,
PLDI is sponsored by the ACM. implementation, and implications of systems
· Symposium on Principles of Programming software."28 In 2010, OSDI included 11 sessions
Languages (POPL). POPL is the leading "forum with 32 papers. OSDI is sponsored by USENIX,
for the discussion of all aspects of programming the Advanced Computing Systems Association.
languages and systems, with emphasis on how · International Conference on Computer Graphics
principles underpin practice."25 POPL includes and Interactive Techniques (SIGGRAPH).
research on the principles of parallel programming SIGGRAPH is the "premier international forum
systems. In 2011, POPL included 16 sessions with for disseminating new scholarly work in computer
49 papers. POPL is sponsored by the ACM. graphics and interactive techniques."29 Graphics
· Symposium on Principles and Practice of Parallel has a huge computational demand that has long
Programming (PPoPP). "PPoPP is a forum for been satisfied by parallel hardware, including both
leading work on all aspects of parallel
26
See Dynopt.org/ppopp-2012/. Last accessed on January 9,
22
See http://ecoop12.cs.purdue.edu/. Last accessed on January 2012.
27
12, 2012. See http://www.eg.org/index.php/about-eg/about-eg. Last ac-
23
See http://researchr.org/conference/oopsla-2012. Last acces- cessed on June 29, 2012.
28
sed on January 12, 2012. See http://static.usenix.org/event/osdi10/cfp/. Last accessed on
24
See pldi12.cs.purdue.edu. Last accessed on January 9, 2012. June 29, 2012.
25 29
See www.cse.psu.edu/popl/12. Last accessed on January 9, See www.siggraph.org/s2011/. Last accessed on January 9,
2012. 2012.
OCR for page 76
76 THE GLOBAL ECOSYSTEM IN ADVANCED COMPUTING
graphics processors and multicore processors. In culture."33 The explosive growth in the numbers of
2011, SIGGRAPH included 28 sessions with 115 these applications and their scale and parallelism
papers. SIGGRAPH is sponsored by the ACM. make them well suited to this study. In 2011,
· International Conference for High Performance WWW included 27 sessions and 81 papers (12.5
Computing, Networking, Storage, and Analysis percent acceptance rate). Conference proceedings
(SC). The SC conference engages "the are published by the ACM.
international community in high performance
computing, networking, storage, and analysis."30 In total, the analyses of conference data presented in
SC is the premier conference on supercomputing Sections F.2 and F.3 represent aggregated results from 19
applications and systems, and has been a leading conference series (i.e., ASPLOS, ECOOP, Eurographics,
venue focusing on parallel systems ranging from HPCA, IEDM, ISCA, ISSCC, MICRO, NANO,
traditional supercomputers to many-cabinet OOPSLA, OSDI, PLDI, POPL, PPoPP, SIGGRAPH, SC,
machines to multicore to systems built from SOSP, VLDB, and WWW). Four time points (1996,
multicore hardware. In 2011, SC included 74 2001, 2006, and 2011)34 were analyzed for each
papers. SC is sponsored by the ACM and the conference series, with exception to NANO that had only
IEEE. one time point, resulting in a total of 73 individual
· Symposium on Operating Systems Principles conferences comprising 4,719 papers and 23,859 authors.
(SOSP). The SOSP conference focuses on
"research related to the design, implementation, F.1.7 Global Reach and Rationale for Conferences
analysis, evaluation, and deployment of computer and Other Methodological Considerations
systems software. . . [taking] a broad view of the
systems area and solicits contributions from many U.S. scientists did dominate many of the early
fields of systems practice, including, but not innovations, creating the international technical societies
limited to, operating systems, file and storage and initiating most of the publication venues in the
systems, distributed systems, mobility, security, technology areas described in Sections F.1.3F.1.6. The
embedded systems, dependability, system following data provide a starting point to examine the
management, peer-to-peer systems, and relationship between this historical U.S. advantage, as
virtualization."31 In 2011, SOSP included 9 well as the location of many of the conference sites in
sessions and 28 papers. SOSP is sponsored by the the United States, and international research activity in
ACM. these areas.
· International Conference on Very Large While proceedings of the selected conferences are
Databases (VLDB). The VLDB conference covers all published in English and the conferences themselves
"current issues in data management, database and are often held in the United States (with the exception of
information systems research."32 Database ECOOP, Eurographics, and WWW which were always
applications are particularly instructive in this held outside the United States for all years analyzed),
setting, because the database community has more than one-third of the 73 specific conference venues
developed and matured parallel algorithms and analyzed by the committee were held outside the United
technologies that exploit parallel hardware. In States. As with any conference, whether held in the
2011, VLDB included 30 sessions and 104 papers United States, Europe, or Asia, the location of the
(18.1 percent acceptance rate). VLDB is meeting has the potential to introduce travel biases for
sponsored by the nonprofit organization Very U.S.- and non-U.S.-based researchers. For example,
Large Data Base Endowment Inc. insufficient travel funds or restrictive government
· International World Wide Web Conference policies could prevent qualified researchers from
(WWW). The WWW conference "aims to provide participating in conferences in other countries.
the world a premier forum for discussion and As one means of exploring potential travel biases
debate about the evolution of the Web, the due to the location of a top technical conference, the
standardization of its associated technologies, and
the impact of those technologies on society and 33
See www.www.2011india.com. Last accessed on June 19,
2012.
30 34
See www.Sc11.supercomputing.org. Last accessed on January In cases where conference proceedings authorship data was
9, 2012. not available or where conferences were held in alternating years
31
See www.sosp.org and www.sigops.org/sosp/sosp11/current. and thus not available for the committee's selected time points,
Last accessed on June 19, 2012. conferences held in adjacent years were analyzed as specified in
32
See www.vldb.org/2011. Last accessed on June 19, 2012. Appendix G.
OCR for page 77
PILOT STUDY OF PAPERS AT TOP TECHNICAL CONFERENCES 77
committee compared regional researcher participation at including the number of papers published and number of
20082011 SIGGRAPH conferences (held in the United citations to those published papers for all of the
States) and SIGGRAPH Asia conferences, based on conferences the committee selected that appear in the
publication data using the methodology described in Microsoft database, along with a selection of European
F.1.2 (see Figure F-1). While Asian participation was and Asian conferences that might be considered
higher at the SIGGRAPH Asia conference than at the competitors. The purpose of these data is not to justify
SIGGRAPH conference in the United States (except in the subset of conferences identified by the committee in
2010, when Asian participation rates were similar at both Sections F.1.3F.1.6, but rather to provide an objective
conferences), U.S. and European participation at both measure of venue impact.
conferences was similar, regardless of whether the In general, the papers in the international
conference was held in the United States or Asia. conferences the committee selected have significantly
As an additional data point, the committee higher citations per paper than the regional conferences
investigated the relationship between conference location found in the Microsoft database.37 The two exceptions to
and the average fraction of conference papers this are for NANO and for ECOOP. According to the
contributed by the United States. For all years combined, Microsoft data, ECOOP has a relatively high citation rate
the United States contributed ~ 69 percent of papers at compared with other programming system conferences,
conferences located in the United States compared to such as OOPSLA and PPoPP. The Microsoft data for
~62 percent when located elsewhere, as shown in Table OOPSLA does not reflect the expert opinion of the
F-1. This table also shows the U.S. share of conference committee, and more careful examination of the data
papers for each of the time points analyzed in Sections revealed that the number of papers reported for
F.2 and F.3, as well as the number of U.S.-located and OOPSLA was more than twice the hand-counted
non-U.S.-located conferences included in the technical papers.38 This bias was corrected in the Table.
committee's analysis. Table F-2 compares the average CiteSeer39 provides another objective view of
U.S. share of conference papers at U.S.- and non-U.S.- programming conference impact, ranking the following
located conferences on a conference-by-conference venues from highest to lowest: POPL, 0.45; PLDI, 0.4;
basis. As shown in the table, U.S. paper contributions at OOPSLA, 0.16; and ECOOP, 0.14. Regardless, the
HPCA, ISCA, MICRO, OSDI, SOSP, and VLDB, were Microsoft data shows all the programming system
similar or higher when the conference was located venues as highly cited and, along with CiteSeer, are
outside the United States. consistent with the committee's selection of leading
These data, along with the SIGGRAPH and technical conferences in the four research areas
SIGGRAPH Asia results, suggest that for the analyses described in Sections F.1.3F.1.6.
presented in this report, conference location does not As an additional measure, attendance at the three
induce a significant travel bias for U.S. researchers. circuits and devices conferences selected by the
However, as the full impact of all potential travel biases committee (ISSCC, IEDM, and NANO) can be
for every nation across all conferences and location compared with three notable regional conferences in
cannot be measured, the above findings should be taken these areas [IEEE Asian Solid-State Circuits Conference
into consideration when interpreting the conference (A-SSCC), IEEE European Solid-State Devices
analyses presented in Sections F.2 and F.3.
Assessing objectively the influence of a conference
is difficult because of the lack of easily comparable
criteria. One starting point for this information is the
Microsoft Academic Search database,35 a publicly
available Web resource that indexes publication data for
some of the conferences the committee considered and 37
mines publication and venue citations for publications For example, the IEEE "architecture"' (ISCA, MICRO,
HPCA, and ASPLOS) conferences selected for analysis report
and authors. 36 Table F-3 shows publication statistics, 25.634.9 citations per paper, compared with the non-IEEE or
non-ACM European Conference on High Performance and Em-
bedded Architecture and Compilation (HiPEAC) (4.2 citations per
35
See http://academic.research.microsoft.com/. Last accessed on paper) and Asia-Pacific Computer Systems Architecture Confer-
June 26, 2012. ence (ASCAC) (2.2 citations per paper).
36 38
In considering conference citation analysis, it is important to The automated system count of publications appears to have
recognize that citation counts are influenced by several different included all non-research track papers, such as poster abstracts and
factors, including differences in database coverage, differences in workshop papers, which are rarely cited.
39
citation practices among research fields, and the age distribution http://citeseer.ist.psu.edu/stats/venues. Last accessed on June
of the (cited) articles. 26, 2012.
OCR for page 78
78 THE GLOBAL ECOSYSTEM IN ADVANCED COMPUTING
Figure F-1 Comparison of researcher participation (via publication authorship) at two conference venues: SIGGRAPH (U.S.-
based) and SIGGRAPH Asia (non-U.S.-based). Data compiled from SIGGRAPH and SIGGRAPH Asia between 2008 and
2011 (~600 papers and ~1300 authors).
TABLE F-1 Comparison of U.S. Paper Contributions at U.S.-located and Non-U.S.-located Conferences
Average U.S. % Share of Conference Papers
1996 2001 2006 2011 All years
U.S.-located conferences
71 66.3 75.3 64.4 69.2
(64% of total conferences analyzed)
no. of conferences analyzed: 12 10 13 12
Non-U.S.-located conferences
58.6 66.8 53.4 69.6 62.1
(36% of total conferences analyzed)
no. of conferences analyzed: 6 8 5 7
TABLE F-2 Comparison of U.S. Paper Contributions at U.S.-located and Non-U.S.-located Conferences (By Individual
Conferences)
U.S. % Share of Conference Papers
SIGGRAPH
MICRO
VLDB
HPCA
POPL
SOSP
OSDI
ISCA
PLDI
Average U.S. %
Share of Papers
When U.S.-located 81.8 88.3 87.3 90.2 52.1 63.1 71.4 96.2 53.8 76.0
When located elsewhere 83.7 93.1 86.5 86.8 44.2 79.0 52.7 87.7 58.3 74.7
* Conferences located in the United States for all years analyzed include: ASPLOS, IEDM, ISSCC, NANO,
OOPSLA, PPoPP, and SC. Conferences located outside the United States for all years analyzed by the
committee include: ECOOP, Eurographics, and WWW
78
OCR for page 79
PILOT STUDY OF PAPERS AT TOP TECHNICAL CONFERENCES 79
TABLE F-3 Conference Citation Analysis from Microsoft Scholar, June 2012
Semiconductor Devices and Circuits Papers Citations Citations/paper
ISSCC 7,271 40,221 5.5
IEDM 9,886 47,925 4.8
ESSCIRC 1,498 5,090 3.4
A-SSCC 676 1,324 2.0
ESSDERC 1,553 2,511 1.6
NANO 2,028 2,266 1.1
Architectures
Micro 905 31,607 34.9
ASPLOS 355 11,255 31.7
HPCA 661 17,891 27.1
ISCA 1,334 34,094 25.6
HiPEAC (Europe) 137 570 4.2
ACSAC (Asia) 339 750 2.2
Programming Systems
POPL 1,267 68,945 54.4
PLDI 519 18,550 35.7
ECOOP (Europe) 783 26,054 33.3
*
OOPSLA 835 14,712 17.9
PPoPP 319 5,641 17.7
Applications
OSDI 255 20,373 79.9
SOSP 349 23,845 68.3
VLDB 2,739 12,2095 44.6
WWW 2,927 60,153 20.6
SIGGRAPH 3,492 100,567 28.8
Eurographics (Europe) 228 3,823 16.8
Supercomputing (SC) 2,994 36,868 12.3
PARLE (Europe) 416 3,804 9.1
* OOPSLA citation data was corrected by hand counting technical papers that exclude non-research track
papers, which are rarely cited. Data compiled from Microsoft Scholar June 2012.
Research Conference (ESSDERC), and IEEE European 350. While conference attendance statistics do not
Solid-State Circuits Conference (ESSCIRC)]. In 2011, directly correlate with quality or influence, they indicate
ISSCC, IEDM, and NANO had significant historical the level of interest in the technologies and ideas found
attendance (according to the IEEE): 3,000, 1,500, and in the conference.
400, respectfully. In contrast, 2011 attendance at A- The committee expects that an analysis including too
SSCC, ESSDERC, and ESSCIRC was between 300 and many (and thus a higher proportion of lower quality)
OCR for page 80
80 THE GLOBAL ECOSYSTEM IN ADVANCED COMPUTING
conferences would underestimate the quality of leading F.2 Current National and Regional Advanced
research efforts and obscure authorship trends for Research Capabilities in Four Key Technology
leading research papers. On the other hand, too limited a Areas
sampling that overprescribes measures of conference
quality and impact would also skew the assessment. In this section, conference representation is used as a
Lastly, the breadth of research topics covered by a proxy indicator of a nation's current (2011) advanced
particular conference (and the same for journals that are research capabilities in each of the following key
oftentimes broader) should also be considered to avoid technology areas: semiconductor devices and circuits,
too broad a sweep of the technological field. In computer architecture, programming systems, and
balancing these factors, the committee has identified a applications. For each technology area, all nations with
limited number of top technical conferences across four at least 1 percent conference representation are ranked
technology-specific research areas that, based on its based on their weighted authorship contributions
expertise and deep domain knowledge of the field, are (following the methodology described in Section F.1.2)
most critical to addressing the computing performance in the targeted conferences previously described in
challenge described in Chapter 1. Sections F.1.3F.1.6. In addition, regional research
capabilities are also provided for comparison.
F.1.8 Methodological Summary As previously discussed, conference papers tend to
have a much shorter time from submission to publication
In summary, the committee believes that high than computing journals in related areas, resulting in the
quality conferences attract leading researchers and most recent, significant innovation appearing at
showcase significant, recent research contributions to the conferences first. Thus, an assessment of current
field. In Sections F.2 and F.3, geographic distributions of conference research efforts is particularly relevant given
conference authorship are quantified to provide a the increasing rate at which scientific discoveries are
technology-specific assessment of national and regional made and then disseminated via the Web. Time series
research capabilities.40,41 analyses of national and regional research capabilities
The committee's analysis is not intended to be are shown in Section F.3.
representative of all scientific outputs across the four
specified research areas; for example, it does not F.2.1 Advanced Semiconductor Devices and Circuit
presume that all relevant technologies presented in Research
journal publications are implicitly represented in the
selected conferences. The committee also recognizes the Table F-4 shows national capabilities in advanced
significant and ongoing progress in bibliometric and semiconductor and nanoscale devices (IEDM and
scientometric approaches to assess the quantity, quality, NANO) research, as well as semiconductor circuits
and impact of scientific output. The assessment provided (ISSCC) research in 2011. The table shows all countries
in subsequent sections is provided as an important first with at least 1 percent representation. The United States
step toward new approaches to assess the global research has a strong competitive position in both of these areas
landscape in specific advanced computing technologies. (50 percent in devices and 36 percent in circuits)
followed by Japan, Taiwan, and Korea. China has only
token representation in these conferences at this time.
Figures F-2 and F-3 show the same data broken down by
region.
In semiconductor devices, the United States has the
highest representation with half of the papers, followed
40 by Asia with less than one-third of the papers and
This approach is consistent with a 2010 National Research
Council (NRC) report, S&T Strategies of Six Countries: Implica-
Europe with even fewer.42 In circuits, the United States,
tions for the United States, in which conference publication analy- Asia, and Europe all share approximately one third of the
sis was described as a "technology-specific indicator [that] gives a papers. NANO represents research that typically is
relatively accurate picture of [national] S&T standing." Available targeted further in the future than those published in
at http://www.nap.edu/catalog.php?record_id=12920. IEDM. At NANO the United States represents over 60
41
A related analysis was also conducted in a 2000 NRC report,
Experiments in International Benchmarking of U.S. Research
42
Fields, which used U.S. contributions of papers at the annual Con- U.S. representation in semiconductor devices is reduced from
ference on Magnetism and Magnetic Materials as a measure of ~50 percent to ~38 percent when only IEDM publications are con-
U.S. participation in magnetic materials research. See sidered (see Figure F-7 and Table F-8). Despite this reduction, the
http://www.nap.edu/catalog.php?record_id=9784. United States remains well ahead of Japan (~17 percent).
OCR for page 81
PILOT STUDY OF PAPERS AT TOP TECHNICAL CONFERENCES 81
TABLE F-4 Current (2011) National Capabilities in
Advanced Semiconductor and Nanoscale Devices and
Circuits Research (Measured by Percent Share of
Conference Papers)
Advanced Semiconductor Circuits Research
1 USA 36.4
2 Japan 12.4
3 Korea 10.4
4 Netherlands 7.3
5 Taiwan 6.7
6 Germany 5.3
7 Belgium 3.5
8 France 3.4
9 Italy 2.9 FIGURE F-2 Current (2011) regional capabilities in advanced
semiconductor and nanoscale devices research (measured by
10 UK 2.3 percent share of conference papers). Data compiled from
11 China 2.0 IEDM and NANO.
12 Canada 1.8
13 Switzerland 1.6
14 Austria 1.0
Advanced Semiconductor and
Nanoscale Devices Research
1 USA 50.2
2 Japan 10.4
3 Taiwan 6.8
4 Korea 5.7
5 UK 3.1
6 Germany 2.9
7 Canada 2.7
8 France 2.5
9 Belgium 2.2
10 Italy 2.1
11 China 1.9 FIGURE F-3 Current (2011) regional capabilities in advanced
12 India 1.8 semiconductor circuits research (measured by percent share of
13 Singapore 1.1 conference papers). Data compiled from ISSCC.
14 UAE 1.1
Data compiled from IEDM and NANO (semiconductor and
nanoscale devices) and ISSCC (semiconductor circuits).
percent of all papers, while Asia and Europe represent from ASPLOS, HPCA, ISCA, and MICRO in 2011. This
less than 20 percent and 10 percent respectively. table includes all countries with at least 1 percent
representation. Figure F-4 shows the same data broken
F.2.2 Advanced Architecture Research down by region. U.S. research dominates that of any
other nation or region and has a weighted average
Table F-5 shows national and regional capabilities in representation of more than 85 percent of the papers. No
advanced architecture research based on aggregated data other nation or region contributes more than 3 percent or
OCR for page 82
82 THE GLOBAL ECOSYSTEM IN ADVANCED COMPUTING
7 percent of the papers, respectively. This result is F.2.3 Advanced Programming Systems Research
perhaps not surprising, given the U.S. historical
dominance in commercial microprocessors, including Table F-6 shows national capabilities in advanced
Intel, AMD, and IBM, as well as former commercial programming systems research based on aggregated data
microprocessors from DEC, HP, and others. That Japan from ECOOP, OOPSLA, PLDI, POPL, and PPoPP. This
is not represented on this list suggests that Japanese table includes all countries with at least 1 percent
universities and industry research institutions are not representation. Figure F-5 shows the same data broken
focused on mainstream computer architectures. While down by region. In programming systems research, the
Japan has activity and expertise in the area, notably the United States dominates with approximately 60 percent
custom processors from Fujitsu that are in the K of the papers. The national breakdown in Table F-6
supercomputer, their national research focus generally shows that aside from the United States, the national
lies elsewhere. distribution of papers is diverse.
Figure F-5 shows that in programming systems,
Europe is a distant second to the United States with
TABLE F-5 Current (2011) National Capabilities in about 25 percent representation, while all of Asia
Advanced Architecture Research (Measured by Percent accounts for less than 10 percent. Disaggregating the
Share of Conference Papers) data (see Appendix G), the United States has an even
stronger position in PPoPP and PLDI, accounting for
Advanced Architecture Research about 75 percent representation in each of them. These
1 USA 85.7 conferences are practical in nature, with papers
2 Korea 2.5 presenting prototype software systems and applications
3 France 2.2 running on real hardware platforms. In POPL, a more
theoretical conference, the United States has a little less
4 China 1.7
than 45 percent of the papers, on par with European
5 Canada 1.6 presentation. Asia has less than 10 percent of the papers,
6 Switzerland 1.4 with greater representation in applications than
7 Australia 1.0 programming systems. In ECOOP, the United States
Data compiled from ASPLOS, HPCA, ISCA, and MICRO. accounted for about 30 percent representation, putting it
15 percentage points behind Europe.
TABLE F-6 Current (2011) National Capabilities in
Advanced Programming Systems Research (Measured
by Percent Share of Conference Papers)
Advanced Programming Systems Research
1 USA 36.4
2 Germany 12.4
3 UK 10.4
4 Switzerland 7.3
5 Israel 6.7
6 France 5.3
7 Canada 3.5
8 India 3.4
9 Japan 2.9
FIGURE F-4 Current (2011) regional capabilities in advanced 10 China 2.3
architecture research (measured by percent share of conference 11 Denmark 2.0
papers). Data compiled from ASPLOS, HPCA, ISCA, and
Micro. 12 Chile 1.8
13 Korea 1.6
Data compiled from ECOOP, OOPSLA, PLDI, POPL, and
PPoPP.
OCR for page 83
PILOT STUDY OF PAPERS AT TOP TECHNICAL CONFERENCES 83
TABLE F-7 Current (2011) National Capabilities in
Advanced Applications Research (Measured by Percent
Share of Conference Papers)
Advanced Applications Research
1 USA 63.9
2 Germany 6.0
3 Canada 4.7
4 China 4.3
5 France 2.4
6 Israel 2.3
7 UK 2.2
8 Hong Kong 1.9
9 Italy 1.5
FIGURE F-5 Current (2011) regional capabilities in advanced 10 Switzerland 1.5
programming systems research (measured by percent share of
conference papers). Data compiled from ECOOP, OOPSLA, 11 Japan 1.2
PLDI, POPL, and PPoPP. 12 Korea 1.0
Data compiled from Eurographics, OSDI, SIGGRAPH, SC,
SOSP, VLDB, and WWW.
2.2.4 Advanced Applications Research
Table F-7 shows national capabilities in
advanced applications research based on aggregated data
from Eurographics, OSDI, SIGGRAPH, SC, SOSP,
VLDB, and WWW in 2011. Figure F-6 shows this data
broken down into regions. In applications research, the
U.S. accounted for almost 64 percent of papers, followed
by Germany, Canada, and China, with 5-6 percent
representation each.
Disaggregated data show that the United States is
also the lead paper contributor in SOSP (~92 percent
compared with ~4 percent in Asia), SC (~80 percent
compared with 9 percent for both Asia and Europe), and
WWW (~72 percent compared with ~13 percent in
Europe and ~8 percent in Asia) and very strong leads in
SIGGRAPH and VLDB (~53 percent, putting the United
States ahead of Europe by approximately 30 points in FIGURE F-6 Current (2011) regional capabilities in advanced
both conferences), suggesting the United States applications research (measured by percent share of conference
maintains strong core competencies in parallel papers). Data compiled from Eurographics, OSDI,
applications. SIGGRAPH, SC, SOSP, VLDB, and WWW.
OCR for page 84
84 THE GLOBAL ECOSYSTEM IN ADVANCED COMPUTING
F.2.5 Summary of Current National Technological TABLE F-8 (19962011) National Capabilities in
Leadership Advanced Semiconductor and Devices and Circuits
Research (Measured by Percent Share of Conference
In each of the four key technical areas, the United Papers)
States holds the lead, with particularly strong Semiconductors & Nanoscale Devices
representation in architecture research ~85 percent
(compared to ~5 and 7 percent representation by Asia 1996 2001 2006 2011
and Europe, respectfully). In programming systems USA 39.3 40.6 34.2 37.9
research, the United States maintains the lead at ~36 Japan 34.4 24.5 21.1 16.9
percent followed by Europe at ~29 percent and Asia at Taiwan 2.3 5.0 4.5 8.2
~8 percent. The strongest European paper contributors
were Germany and the UK at ~12 and 10 percent, Korea 6.3 8.1 9.4 7.5
respectfully. In applications research, the United States Belgium 1.5 3.0 6.4 6.4
leads paper contributions at ~64 percent followed by France 3.4 2.3 4.4 6.0
Europe at ~17 percent and Asia at ~11 percent. This data Italy 3.9 3.4 2.8 3.7
suggests that the U.S. is particularly strong in design and Singapore - 1.7 4.5 3.0
systems engineering. While the United States maintains
China 0.5 0.5 0.6 2.6
a strong lead in semiconductor and nanoscale devices
(~50 percent) compared to Asia (~28 percent) and UK - - 1.7 1.9
Europe (~16 percent), its contributions in semiconductor Germany 3.0 5.4 3.7 1.7
circuits research are comparable to Europe and Asia Austria - 0.4 0.6 1.1
(each separated by ~7 percentage points). Canada 0.5 1.0 0.4 0.1
Netherlands 1.9 1.7 1.7 0.7
F.3 Longitudinal Changes in the Global Research
Landscape Switzerland 2.1 0.8 2.2 0.6
Semiconductor Circuits
To assess how the competitive research landscape
has changed over time, the committee examined the 1996 2001 2006 2011
competitive metrics (described in Section F.1) for the USA 46.6 51.1 45.2 36.4
same set of conferences in the previous section in 5-year Japan 27.7 18.6 15.8 12.4
increments over a 15-year span from 1996 to 2011. Korea 3.9 4.7 6.1 10.4
Tables F-8 through F-11 show the relative Netherlands 2.0 4.0 3.1 7.3
representation of different nations in each of the
Taiwan 0.6 0.1 7.0 6.7
technical areas of advanced computing research from
1996 to 2011 in 5-year increments, ranked by position in Germany 4.2 4.4 5.3 5.3
2011. Each of the tables includes only those nations that Belgium 1.0 5.5 1.5 3.5
have at least a 1 percent representation for at least one of France 0.9 0.8 1.0 3.4
the years. Countries that are not represented in 2011, but Italy 2.9 - 3.4 2.9
have at least 1 percent representation for any increment,
UK 0.8 0.8 0.6 2.3
are marked in italicized text. Figures F-7 through F-11
show the aggregate regional representation for each area China - - 1.5 2.0
of advanced research over the same time span. For the Canada 3.4 2.2 1.4 1.8
time-series data, the semiconductor devices area includes Switzerland 4.3 2.7 1.9 1.6
only IEDM, because historical data for NANO was not Austria - - 2.3 1.0
easily available.
Finland - 1.2 0.1 -
In semiconductor devices research, U.S.
representation has remained relatively stable with the Hong Kong - 1.2 - -
largest gains made by Taiwan and Belgium (~ 5-6 Sweden 0.7 - 1.0 0.4
percentage point increases each). In semiconductor Data compiled from IEDM (semiconductor and nanoscale
circuits research, the United States shows a moderate devices) and ISSCC (semiconductor circuits).
decline, in tandem with an overall broadening in
OCR for page 85
PILOT STUDY OF PAPERS AT TOP TECHNICAL CONFERENCES 85
international representation. In this area largest leaps TABLE F-9 (19962011) National Capabilities in
were made by Korea, Taiwan, and the Netherlands with Advanced Architecture Research (Measured by Percent
56 percentage point increases each. At the same time, Share of Conference Papers)
Japan has dropped significantly in both semiconductor
devices and in semiconductor circuits (~17 and 15 1996 2001 2006 2011
percent, respectfully). USA 79.9 89.2 90.7 85.7
In architecture research, the United States has Korea - 0.2 - 2.5
maintained a significant lead, with no major advances by
France 3.1 1.0 0.7 2.2
any other nation or region. In programming systems, the
U.S. lead has been challenged somewhat by increases in China - - - 1.7
Europe by small but steady gains by Israel, Switzerland, Canada 5.5 1.2 1.9 1.6
and the UK (as well as China, India, and Korea to a Switzerland - - - 1.4
lesser degree). Australia 0.4 - - 1.0
In the application areas, U.S. representation has
Belgium - 1.0 0.3 -
retained a stable lead over the 15-year period with no
other nation ever contributing more than 8 percent India - - 1.9 0.6
(France, which contributed ~8 percent in 1996, has since Israel - - 1.2 -
dropped to ~2 percent in 2011). While representing only Japan 2.1 2.0 1.2 -
a small percentage of applications papers, China made a Spain 2.4 5.5 0.8 0.6
notable move from no representation in 1996 to ~4
Sweden 1.8 - - -
percent in 2011.
UK 2.0 - 0.3 0.7
Data compiled from ASPLOS, HPCA, ISCA,
and Micro.
TABLE F-10 (19962011) National Capabilities in TABLE F-11 (19962011) National Capabilities in
Advanced Programming Systems Research (Measured Advanced Applications Systems Research
by Percent Share of Conference Papers) (Measured by Percent Share of Conference Papers)
1996 2001 2006 2011 1996 2001 2006 2011
USA 62.2 63.8 67.1 56.5 USA 64.3 57.5 64.0 63.9
Germany 8.0 3.0 2.3 7.9 Germany 4.2 7.3 7.9 6.0
UK 5.2 4.3 7.1 7.2 Canada 3.5 3.7 3.5 4.7
Switzerland 1.9 2.9 1.8 4.5 China - 0.5 3.0 4.3
Israel 1.7 5.1 2.0 3.2 France 8.3 3.5 1.9 2.4
France 7.5 4.8 3.1 3.0 Israel 1.5 2.1 1.4 2.3
India - - - 2.1 UK 1.8 3.0 3.9 2.2
Japan 2.7 5.2 2.3 2.0 Hong Kong - 0.3 1.7 1.9
Canada 2.4 1.0 1.7 2.2 Italy 1.8 1.7 1.3 1.5
China - - 1.2 1.4 Switzerland 1.4 1.9 0.7 1.5
Denmark 2.4 2.3 0.9 1.4 Japan 3.7 5.6 3.0 1.2
Chile - - 0.2 1.1 Korea 0.4 1.1 0.7 1.0
Korea - - 0.4 1.0 Australia 0.7 1.0 0.4 0.4
Australia 0.4 0.6 1.8 0.7 Austria 1.3 1.5 0.5 0.7
Italy 1.1 2.5 0.2 0.9 India 0.7 0.8 1.7 0.6
Netherlands 2.3 2.1 0.8 0.9 Singapore 0.3 1.4 0.6 0.9
Sweden 0.5 - 1.7 0.2 Spain 1.0 1.5 0.5 0.4
Data compiled from ECOOP, OOPSLA, PLDI, Taiwan 1.0 0.9 0.8 0.6
POPL, and PPoPP. Data compiled from Eurographics, OSDI, SIGGRAPH, SC,
SOSP, VLDB, and WWW.
OCR for page 86
86 THE GLOBAL ECOSYSTEM IN ADVANCED COMPUTING
FIGURE F-7 (19962011) Regional capabilities in advanced semiconductor and nanoscale devices
research (measured by percent share of conference papers). Data compiled from IEDM.
FIGURE F-8 (19962011) Regional capabilities in advanced semiconductor circuits research (measured
by percent share of conference papers). Data compiled from ISSCC.
FIGURE F-9 (19962011) Regional capabilities in advanced architecture research (measured by percent
share of conference papers). Data compiled from ASPLOS, HPCA, ISCA, and MICRO.
OCR for page 87
PILOT STUDY OF PAPERS AT TOP TECHNICAL CONFERENCES 87
FIGURE F-10 (19962011) Regional capabilities in advanced programming systems research (measured
by percent share of conference papers). Data compiled from ECOOP, OOPSLA, PLDI, POPL, and PPoPP.
FIGURE F-11 (19962011) Regional capabilities in advanced applications research (measured by percent
share of conference papers). Data compiled from Eurographics, OSDI, SC, SIGGRAPH, SOSP, VLDB,
and WWW.
OCR for page 88
