Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 71
F Pilot Study of Papers at Top Technical Conferences in Advanced Computing T he committee found it challenging to identify reliable and robust nontraditional indicators for assessing a nation's technological research capabilities specific to the computing performance challenges outlined in Chapter 1. After significant time and resources, however, this was simply not feasible. Further, the committee's analysis represents only one of likely many different and potentially useful measures of a nation's research capabilities and innovativeness. These factors considered, the methodological consideration and debate, the committee committee's methodological approach and insights from conducted a pilot study to determine whether a its assessment are presented here as a pilot effort that bibliometric analysis of papers at select prestigious will, no doubt, benefit from deeper, subsequent conferences in advanced computing could provide a exploration by others. useful snapshot of a nation's capabilities in specific technology areas critical to the study's charge. F.1 Methodological Overview of Conference The committee noted the strengths and challenges of Paper Authorship Analysis a methodology that is both objective (e.g., conference publication data) and subjective (e.g., using committee To assess a nations' technology-specific research expertise to identify specific conferences for analysis). capabilities, the committee analyzed paper authorship-- By excluding traditional journal publications--as well as specifically, the geographical locations of authors--in papers from conferences not considered to be many of the top technical conferences in four research representative of the most relevant and leading areas most closely related to the technological challenges research--some relevant research may have been outlined in Chapter 1. These relate to the computing excluded; alternative samplings of conferences could performance challenge and the shift to multicore also yield different results. On the other hand, a selective processors: semiconductor devices and circuits, computer sampling of conferences may better support a more architecture, programming systems, and applications. focused assessment of research efforts across specific technology areas (i.e., semiconductors and nanoscale F.1.1 Rationale for Conferences as a Preferred Venue devices and circuits, architecture, programming systems, and applications). In the computing community--unlike many other An ideal analysis would include all papers relevant science and engineering disciplines--conference papers to the computing performance challenges outlined in are often the publication venue of choice rather than Chapter 1, whether published at conferences or in journals.2 In fact, highly selective computer science traditional journals, weighted by citations and impact factors, as well as expert judgment.1 Given limitations in 20and%20publications%20and%20news%20releases/Science%20 performance/SciencePerformance_FullReport_EN_Web.pdf). 2 David Patterson, Lawrence Snyder, and Jeffrey Ullman, 1999, 1 Council of Canadian Academies, 2012, Informing Research "Best Practices Memo: Evaluating Computer Scientists and Engi- Choices: Indicators and Judgment The Report of the Expert neers for Promotion and Tenure" in Computing Research News, Panel on Science Performance and Research Funding (available June. Available at http://www.cra.org/uploads/documents/resourc- at http://www.scienceadvice.ca/uploads/eng/assessments% es/bpmemos/tenure_review.pdf. Last accessed on August 15, 2012. 71
OCR for page 72
72 THE GLOBAL ECOSYSTEM IN ADVANCED COMPUTING conferences often have higher citation indices and greater ACM Transactions on Computer Systems, and IEEE venue impact compared to related computing journals.3,4 Transactions on Parallel and Distributed Systems). The importance of conference publications within this Achieving the same level of topical focus would community is consistent with a 2011 NRC report, A require disaggregation of the journal data on a Data-Based Assessment of Research-Doctorate paper-by-paper basis. This is simply not feasible for Programs in the United States,5 which indicates that "for the wide range of conferences selected and papers the field of computer science, refereed conference papers analyzed for this report. Thus, analysis of conference are an important form of scholarship." papers allows a more focused assessment of the In this report, conferences are the preferred venue research areas identified by the committee as critical over journals for the computer science-focused areas for sustaining computing performance and the shift described in Sections F.1.3F.1.6 for several reasons. to multicore processors, as opposed to computer First, these conferences tend to have a much shorter time science generally. from submission to publication than journals in the area, Similar to prestigious journals, premier resulting in the most recent, significant innovations conferences in each of the targeted hardware, appearing at the conferences first.6 Second, conferences architecture, and software research areas are highly provide a larger sample size than journals of highly competitive and conference submissions are recognized recent top-quality research. Third, the rigorously peer-reviewed. Conferences are also conferences identified by the committee often have more competitive publishing venues because focused research interests compared to journals that representation is professionally beneficial. would publish related (albeit less recent and possibly less Conferences provide an opportunity for researchers regarded), but broader works.7 to share new research, to learn from others, and to For example, in advanced architecture research, gain exposure to recent and significant research relevant to the challenges described in Chapter 1, efforts.8,9 architecture papers appear in a wider range of journals that include more than just computer F.1.2 Determination of a Nation's Paper Contributions architecture (e.g., IEEE Transactions on Computers, at Conferences 3 It is worth noting that industry participation in these confer- As a proxy indicator for a nation's technology- ences may be limited by the demise of central research labs, less specific research capabilities, the committee analyzed the emphasis on outside presentations and publications, and reluctance weighted distribution of authors for research papers given to report on the most important research that companies are per- at the conferences listed and described in Sections F.1.3 forming. F.1.6. To do this, the committee noted the home nation 4 As one example, CiteSeer, which keeps statistics about com- for each author (defined as the geographic location of puter science publications, reports a venue impact of 0.14, 0.08, and 0.6 for three architecture research conferences analyzed in this that author's affiliation listed on each paper's title page) report (High-Performance Computer Architecture, International and computed each nation's weighted authorship Symposium on Computer Architecture, and the International Sym- contribution to each conference paper. It is important to posium on Microarchitecture, respectively) compared to 0.02 and note that this analysis does not distinguish between a 0.01 for two related architecture research journals: IEEE Transac- U.S.-based author who is a U.S. citizen and a U.S.-based tions on Computers and IEEE Transactions on Parallel and Dis- author who is not a U.S. citizen (and may eventually tributed Systems, respectively. See http://citeseer.ist.psu.edu/stats/ venues. Last accessed on August 15, 2012. return to his or her home country), which may, in some 5 NRC, 2011, A Data-Based Assessment of Research-Doctorate cases, diminish a nation's research representation at the Programs in the United States, Washington, D.C.: The National sampled conferences.10 Academies Press (available online at https://download.nap.edu/ rdp/index.html?) 6 8 This is particularly relevant to the analysis of recent conference David Patterson, Lawrence Snyder, and Jeffrey Ullman, 1999, papers in Appendix F.2. "Best Practices Memo: Evaluating Computer Scientists and Engi- 7 One exception of a journal that is equally regarded with a simi- neers for Promotion and Tenure" in Computing Research News, larly themed premier conference is the IEEE Journal of Solid-State June. Available at http://www.cra.org/uploads/documents/ Circuits (JSSC). JSSC publishes approximately 200 papers per resources/bpmemos/tenure_review.pdf. Last accessed on August year, with the papers typically being 10-15 pages long. By com- 15, 2012. 9 parison, the International Solid-State Circuits Conference (ISSCC) NRC, 1994, Academic Careers for Experimental Computer typically publishes the same number of papers, but each paper is 3 Scientists and Engineers, Washington, D.C.: The National pages long. As both venues are highly regarded and may represent Academies Press (available online at http://www.nap.edu/open a similar sample of annual papers, a more in-depth analysis of book.php?record_id=2236). 10 contributions in this area would benefit by addressing both ISSCC 2010 National Science Foundation data show that the share of and JSSC. non-U.S. citizens receiving U.S. doctoral degrees in natural sci-
OCR for page 73
PILOT STUDY OF PAPERS AT TOP TECHNICAL CONFERENCES 73 A nation's weighted11 contribution to each by the committee are sponsored or published by either conference paper is merely the number of authors (or both) the Institute of Electrical and Electronics geographically located in (or professionally affiliated Engineers (IEEE) or the Association for Computing with) that particular nation divided by the total number Machinery (ACM), the two preeminent international of authors of the paper. Thus, the weighted contributions technical societies in electrical engineering and for a given paper always sum to one. To compute the computing. Based on committee members' opinions and weighted percentage of papers contributed by a nation at knowledge of the fields, the following sections identify a a given conference, each nation's weighted contributions limited number of top conferences that make available to each conference paper are summed and then divided new and interesting research germane to the study's by the total number of papers given at that conference. charge. The committee believes this measurement--as opposed to total papers or total authors--better reflects a nation's F.1.3 Semiconductor Devices and Circuits Conferences authorship contribution.12 This measure also has the distinct advantage that it is public information13 and can As described in Chapter 1, the end of Dennard be extended and reproduced by others. scaling has placed greater pressure on innovative devices Based on the technical challenges outlined in and circuits to deliver more energy-efficient technologies Chapter 1, the committee identified four research areas for building microprocessors. To explore the research critical to addressing the computing performance capabilities in these areas, the committee analyzed challenge and the shift to multicore processors: papers from three conferences--two in semiconductor semiconductor devices and circuits, computer and nanoscale devices and one in semiconductor circuits architecture, programming systems, and applications. To following the methodology described earlier in Section compute the weighted percentage of papers contributed F.1.2. These conferences are described below. by a nation in each of these research areas, each nation's · International Electron Devices Meeting (IEDM). weighted contributions to each conference paper are As stated on the conference Web site, IEDM is summed across all conferences assigned to a particular "the world's pre-eminent forum for reporting research area (discussed in Sections F.1.3F.1.6) and technological breakthroughs in the areas of then divided by the total number of papers given at those semiconductor and electronic device technology, conferences.14 All but three of the conferences analyzed design, manufacturing, physics, and modeling. IEDM is the flagship conference for nanometer- ences and engineering is high and increasing at a higher rate than scale CMOS (complementary-symmetry metal- for U.S. citizens (available at http://www.nsf.gov/statistics/ seind10/pdf/overview.pdf; last accessed on September 2, 2012). oxide-semiconductor) transistor technology, Additionally, a 2007 report by the Oak Ridge Institute for Science advanced memory, displays, sensors, MEMS and Education, Stay Rates of Foreign Doctorate Recipients from (microelectromechanical systems) devices, novel U.S. Universities (available at http://orise.orau.gov/files/sep/stay- quantum and nano-scale devices and rates-foreign-doctorate-recipients-2007.pdf) found that two-thirds phenomenology, optoelectronics, devices for of foreign citizens who received science or engineering doctorates power and energy harvesting, high-speed devices, from U.S. universities in 2005 continued to live in the United States in 2007. as well as process technology and device 11 In an unweighted analysis, two countries will receive 50 per- modeling and simulation. The conference scope cent of a particular paper's contribution to the conference, even if not only encompasses devices in silicon, nine coauthors are located in one country and one coauthor is lo- compound and organic semiconductors, but also in cated in the other country. emerging material systems. IEDM is truly an 12 Examination of the country of origin of members of confer- international conference, with strong ence programming committees may be a useful future activity and provide additional insight for assessing a nation's interest and representation from speakers from around the capabilities in a particular technological field. globe."15 In 2011, IEDM included 36 sessions 13 While bibliometric databases, such as SciVerse Scopus and encompassing more than 200 papers. IEDM is Web of Science, provide some conference publication and citation sponsored by the IEEE. data, comprehensive and consistent data for each conference across the time periods analyzed in the report do not exist. For example, Scopus includes only limited or no coverage of ECOOP, Eurographics, OSDI, SC, SOSP, VLDB, and WWW, and large reported in Appendix G) and then averaging across all conferences gaps in annual coverage exist for ISCA, MICRO, POPL, and assigned to a particular research area, a bias is introduced that PPoPP. could skew the overall average in favor of those conferences with 14 This allows all amassed papers in each research area to be larger numbers of presented papers. 15 weighted equally. In contrast, by first calculating a nation's See http://www.his.com/~iedm/. Last accessed on January 9, weighted percentage of contributed papers for each conference (as 2012.
OCR for page 74
74 THE GLOBAL ECOSYSTEM IN ADVANCED COMPUTING · International Conference on Nanotechnology ASPLOS included 14 sessions with 32 papers. (NANO). From the conference Web site: "NANO ASPLOS is sponsored by the ACM. is the flagship IEEE conference in · International Symposium on High Performance Nanotechnology, which makes it a must for Computer Architecture (HPCA). HPCA covers students, educators, researchers, scientists and many of the same topics as ISCA and MICRO.19 engineers alike, working at the interface of In 2011, HPCA included 14 sessions and 46 nanotechnology and the many fields of electronic papers. HPCA is sponsored by the IEEE. materials, photonics, bio- and medical devices, · International Symposium on Computer alternative energy, environmental protection, and Architecture (ISCA). As stated on the conference multiple areas of current and future electrical and Web site, ISCA is "the premier forum for new electronic applications. In each of these areas, ideas and experimental results in computer NANO is the conference where practitioners will architecture,"20 including parallel architecture and see nanotechnologies at work in both their own multicore systems. In 2011, ISCA included 14 and related fields, from basic research and theory sessions with 40 papers. ISCA is sponsored by to industrial applications."16 In 2011, NANO ACM and the IEEE. included more than 400 papers. NANO is · International Symposium on Microarchitecture sponsored by the IEEE. (MICRO). As stated on the conference Web site, · International Solid-State Circuits Conference MICRO "brings together researchers in fields (ISSCC). As stated on the conference Web site, related to microarchitecture, compilers, chips, and ISSCC is "the premier forum for the presentation systems for technical exchange on traditional of advances in solid-state circuits and systems-on- microarchitecture topics and emerging research a-chip."17 ISSCC topics include advanced memory areas."21 In 2011, MICRO included 13 sessions circuits, low-power circuits, high-speed signaling, with 44 papers. MICRO is sponsored by ACM and and microprocessors, among many others. In the IEEE. 2011, ISSCC included 28 sessions encompassing more than 200 papers. ISSCC is sponsored by the Roughly, the same community of researchers IEEE. publishes in and attends the conferences described above, although ASPLOS includes additional research F.1.4 Computer Architecture Conferences areas on the boundary between computer architecture, programming languages, and operating systems. Computer architecture includes the design and study of computer hardware implementations and computer F.1.5 Programming Systems Conferences design at the hardware-software boundary. Computer architects seek to make computers faster, lower power, In this report, the committee focuses on (1) cheaper, more reliable, and easier to program. Many programming systems that encompass programming computer architecture researchers focus on parallel and language design and implementation, and (2) multicore systems. The committee analyzed conference programming tools, including programming models, papers from four top-flight conferences, described languages, compilers, runtime systems, and virtual below. machines for parallel systems that are necessary to · International Symposium on Architectural Support enable applications to exploit emerging silicon trends for Programming Languages and Operating and chip architectures. The following five top Systems (ASPLOS). ASPLOS "is the premier conferences were analyzed, which collectively cover a forum for multidisciplinary systems research, range of programming system technologies: spanning hardware, computer architecture, compilers, languages, operating systems, · European Conference on Object-Oriented networking, and applications," 18 and includes Programming (ECOOP). ECOOP covers "topics papers on parallel hardware and software. In 2011, on object-oriented technologies, software development, systems, languages and 16 19 See http://ieeenano2011.org/. Last accessed on January 9, See www.ece.lsu.edu/hpca-18/. Last accessed on January 9, 2012. 2012. 17 20 See http://isscc.org/. Last accessed on January 9, 2012. See Isca2012.ittc.ku.edu. Last accessed on January 9, 2012. 18 21 See research.microsoft.com/en-us/um/Cambridge/events/ See www.microarch.org/micro44/. Last accessed on January asplos_2012. Last accessed on January 9, 2012. 9, 2012.
OCR for page 75
PILOT STUDY OF PAPERS AT TOP TECHNICAL CONFERENCES 75 applications."22 ECOOP was established in 1987. programming, including foundational and ECOOP and OOPSLA are peer conferences, were theoretical aspects, techniques, tools, and practical established within 1 year of each other, and have experiences."26 Conference topics include work on followed the same historical trends on topics. concurrent and parallel (e.g., multicore, Whereas many of the other conferences the heterogeneous, and distributed) systems. In 2011, committee sampled are often hosted in the United PPoPP included 8 sessions with 26 papers. PPoPP States or Canada, ECOOP has only ventured is sponsored by the ACM. outside of Europe twice (1990, 2012). In 2011, ECOOP included 9 sessions and 26 papers. Since F.1.6 Applications Conferences 2007, ECOOP has been sponsored by ACM. · Object-Oriented Programming, Systems, Computer applications are a tremendously broad Languages, and Applications (OOPSLA). area encompassing topics such as scientific computing, OOPSLA "embraces all aspects of software security, distributed and cloud computing, databases, and construction and delivery" and is "a premier artificial intelligence. Representing the full range of forum for software innovation."23 OOPSLA these areas is beyond the scope of this report. However, started in 1986 when object-oriented programming since parallel application development is central to the systems were emerging to be a forum for success of multicore systems, the committee sampled researchers and practitioners to explore this new seven conferences that have a strong focus on paradigm. Object-oriented programming computational application needs and historically have subsequently became a dominant paradigm. Now depended on parallel and multicore systems, described OOPSLA is much broader and covers the same below. topics as PLDI and POPL. In 2011, OOPSLA · Annual Conference of the European Association included 17 sessions with 61 papers (the most in for Computer Graphics (Eurographics). its history). OOPSLA is sponsored by the ACM. Eurographics is a "Europe-wide professional · Programming Language Design and Computer Graphics Association. . . that supports Implementation (PLDI). PLDI focuses "on the its members in advancing the state of the art in design, implementation, development, and use of Computer Graphics and related fields such as programming languages. [It] emphasizes Multimedia, Scientific Visualization, and Human innovative and creative approaches to compile- Computer Interfaces."27 In 2011, Eurographics time and runtime technology; novel language included 14 sessions and 35 papers. designs and features; and results from · Symposium on Operating Systems Design and implementations."24 Parallel programming Implementation (OSDI). OSDI "brings together systems are a significant component of PLDI. In professionals from academic and industrial 2011, PLDI included 20 sessions with 55 papers. backgrounds. . . [to discuss] the design, PLDI is sponsored by the ACM. implementation, and implications of systems · Symposium on Principles of Programming software."28 In 2010, OSDI included 11 sessions Languages (POPL). POPL is the leading "forum with 32 papers. OSDI is sponsored by USENIX, for the discussion of all aspects of programming the Advanced Computing Systems Association. languages and systems, with emphasis on how · International Conference on Computer Graphics principles underpin practice."25 POPL includes and Interactive Techniques (SIGGRAPH). research on the principles of parallel programming SIGGRAPH is the "premier international forum systems. In 2011, POPL included 16 sessions with for disseminating new scholarly work in computer 49 papers. POPL is sponsored by the ACM. graphics and interactive techniques."29 Graphics · Symposium on Principles and Practice of Parallel has a huge computational demand that has long Programming (PPoPP). "PPoPP is a forum for been satisfied by parallel hardware, including both leading work on all aspects of parallel 26 See Dynopt.org/ppopp-2012/. Last accessed on January 9, 22 See http://ecoop12.cs.purdue.edu/. Last accessed on January 2012. 27 12, 2012. See http://www.eg.org/index.php/about-eg/about-eg. Last ac- 23 See http://researchr.org/conference/oopsla-2012. Last acces- cessed on June 29, 2012. 28 sed on January 12, 2012. See http://static.usenix.org/event/osdi10/cfp/. Last accessed on 24 See pldi12.cs.purdue.edu. Last accessed on January 9, 2012. June 29, 2012. 25 29 See www.cse.psu.edu/popl/12. Last accessed on January 9, See www.siggraph.org/s2011/. Last accessed on January 9, 2012. 2012.
OCR for page 76
76 THE GLOBAL ECOSYSTEM IN ADVANCED COMPUTING graphics processors and multicore processors. In culture."33 The explosive growth in the numbers of 2011, SIGGRAPH included 28 sessions with 115 these applications and their scale and parallelism papers. SIGGRAPH is sponsored by the ACM. make them well suited to this study. In 2011, · International Conference for High Performance WWW included 27 sessions and 81 papers (12.5 Computing, Networking, Storage, and Analysis percent acceptance rate). Conference proceedings (SC). The SC conference engages "the are published by the ACM. international community in high performance computing, networking, storage, and analysis."30 In total, the analyses of conference data presented in SC is the premier conference on supercomputing Sections F.2 and F.3 represent aggregated results from 19 applications and systems, and has been a leading conference series (i.e., ASPLOS, ECOOP, Eurographics, venue focusing on parallel systems ranging from HPCA, IEDM, ISCA, ISSCC, MICRO, NANO, traditional supercomputers to many-cabinet OOPSLA, OSDI, PLDI, POPL, PPoPP, SIGGRAPH, SC, machines to multicore to systems built from SOSP, VLDB, and WWW). Four time points (1996, multicore hardware. In 2011, SC included 74 2001, 2006, and 2011)34 were analyzed for each papers. SC is sponsored by the ACM and the conference series, with exception to NANO that had only IEEE. one time point, resulting in a total of 73 individual · Symposium on Operating Systems Principles conferences comprising 4,719 papers and 23,859 authors. (SOSP). The SOSP conference focuses on "research related to the design, implementation, F.1.7 Global Reach and Rationale for Conferences analysis, evaluation, and deployment of computer and Other Methodological Considerations systems software. . . [taking] a broad view of the systems area and solicits contributions from many U.S. scientists did dominate many of the early fields of systems practice, including, but not innovations, creating the international technical societies limited to, operating systems, file and storage and initiating most of the publication venues in the systems, distributed systems, mobility, security, technology areas described in Sections F.1.3F.1.6. The embedded systems, dependability, system following data provide a starting point to examine the management, peer-to-peer systems, and relationship between this historical U.S. advantage, as virtualization."31 In 2011, SOSP included 9 well as the location of many of the conference sites in sessions and 28 papers. SOSP is sponsored by the the United States, and international research activity in ACM. these areas. · International Conference on Very Large While proceedings of the selected conferences are Databases (VLDB). The VLDB conference covers all published in English and the conferences themselves "current issues in data management, database and are often held in the United States (with the exception of information systems research."32 Database ECOOP, Eurographics, and WWW which were always applications are particularly instructive in this held outside the United States for all years analyzed), setting, because the database community has more than one-third of the 73 specific conference venues developed and matured parallel algorithms and analyzed by the committee were held outside the United technologies that exploit parallel hardware. In States. As with any conference, whether held in the 2011, VLDB included 30 sessions and 104 papers United States, Europe, or Asia, the location of the (18.1 percent acceptance rate). VLDB is meeting has the potential to introduce travel biases for sponsored by the nonprofit organization Very U.S.- and non-U.S.-based researchers. For example, Large Data Base Endowment Inc. insufficient travel funds or restrictive government · International World Wide Web Conference policies could prevent qualified researchers from (WWW). The WWW conference "aims to provide participating in conferences in other countries. the world a premier forum for discussion and As one means of exploring potential travel biases debate about the evolution of the Web, the due to the location of a top technical conference, the standardization of its associated technologies, and the impact of those technologies on society and 33 See www.www.2011india.com. Last accessed on June 19, 2012. 30 34 See www.Sc11.supercomputing.org. Last accessed on January In cases where conference proceedings authorship data was 9, 2012. not available or where conferences were held in alternating years 31 See www.sosp.org and www.sigops.org/sosp/sosp11/current. and thus not available for the committee's selected time points, Last accessed on June 19, 2012. conferences held in adjacent years were analyzed as specified in 32 See www.vldb.org/2011. Last accessed on June 19, 2012. Appendix G.
OCR for page 77
PILOT STUDY OF PAPERS AT TOP TECHNICAL CONFERENCES 77 committee compared regional researcher participation at including the number of papers published and number of 20082011 SIGGRAPH conferences (held in the United citations to those published papers for all of the States) and SIGGRAPH Asia conferences, based on conferences the committee selected that appear in the publication data using the methodology described in Microsoft database, along with a selection of European F.1.2 (see Figure F-1). While Asian participation was and Asian conferences that might be considered higher at the SIGGRAPH Asia conference than at the competitors. The purpose of these data is not to justify SIGGRAPH conference in the United States (except in the subset of conferences identified by the committee in 2010, when Asian participation rates were similar at both Sections F.1.3F.1.6, but rather to provide an objective conferences), U.S. and European participation at both measure of venue impact. conferences was similar, regardless of whether the In general, the papers in the international conference was held in the United States or Asia. conferences the committee selected have significantly As an additional data point, the committee higher citations per paper than the regional conferences investigated the relationship between conference location found in the Microsoft database.37 The two exceptions to and the average fraction of conference papers this are for NANO and for ECOOP. According to the contributed by the United States. For all years combined, Microsoft data, ECOOP has a relatively high citation rate the United States contributed ~ 69 percent of papers at compared with other programming system conferences, conferences located in the United States compared to such as OOPSLA and PPoPP. The Microsoft data for ~62 percent when located elsewhere, as shown in Table OOPSLA does not reflect the expert opinion of the F-1. This table also shows the U.S. share of conference committee, and more careful examination of the data papers for each of the time points analyzed in Sections revealed that the number of papers reported for F.2 and F.3, as well as the number of U.S.-located and OOPSLA was more than twice the hand-counted non-U.S.-located conferences included in the technical papers.38 This bias was corrected in the Table. committee's analysis. Table F-2 compares the average CiteSeer39 provides another objective view of U.S. share of conference papers at U.S.- and non-U.S.- programming conference impact, ranking the following located conferences on a conference-by-conference venues from highest to lowest: POPL, 0.45; PLDI, 0.4; basis. As shown in the table, U.S. paper contributions at OOPSLA, 0.16; and ECOOP, 0.14. Regardless, the HPCA, ISCA, MICRO, OSDI, SOSP, and VLDB, were Microsoft data shows all the programming system similar or higher when the conference was located venues as highly cited and, along with CiteSeer, are outside the United States. consistent with the committee's selection of leading These data, along with the SIGGRAPH and technical conferences in the four research areas SIGGRAPH Asia results, suggest that for the analyses described in Sections F.1.3F.1.6. presented in this report, conference location does not As an additional measure, attendance at the three induce a significant travel bias for U.S. researchers. circuits and devices conferences selected by the However, as the full impact of all potential travel biases committee (ISSCC, IEDM, and NANO) can be for every nation across all conferences and location compared with three notable regional conferences in cannot be measured, the above findings should be taken these areas [IEEE Asian Solid-State Circuits Conference into consideration when interpreting the conference (A-SSCC), IEEE European Solid-State Devices analyses presented in Sections F.2 and F.3. Assessing objectively the influence of a conference is difficult because of the lack of easily comparable criteria. One starting point for this information is the Microsoft Academic Search database,35 a publicly available Web resource that indexes publication data for some of the conferences the committee considered and 37 mines publication and venue citations for publications For example, the IEEE "architecture"' (ISCA, MICRO, HPCA, and ASPLOS) conferences selected for analysis report and authors. 36 Table F-3 shows publication statistics, 25.634.9 citations per paper, compared with the non-IEEE or non-ACM European Conference on High Performance and Em- bedded Architecture and Compilation (HiPEAC) (4.2 citations per 35 See http://academic.research.microsoft.com/. Last accessed on paper) and Asia-Pacific Computer Systems Architecture Confer- June 26, 2012. ence (ASCAC) (2.2 citations per paper). 36 38 In considering conference citation analysis, it is important to The automated system count of publications appears to have recognize that citation counts are influenced by several different included all non-research track papers, such as poster abstracts and factors, including differences in database coverage, differences in workshop papers, which are rarely cited. 39 citation practices among research fields, and the age distribution http://citeseer.ist.psu.edu/stats/venues. Last accessed on June of the (cited) articles. 26, 2012.
OCR for page 78
78 THE GLOBAL ECOSYSTEM IN ADVANCED COMPUTING Figure F-1 Comparison of researcher participation (via publication authorship) at two conference venues: SIGGRAPH (U.S.- based) and SIGGRAPH Asia (non-U.S.-based). Data compiled from SIGGRAPH and SIGGRAPH Asia between 2008 and 2011 (~600 papers and ~1300 authors). TABLE F-1 Comparison of U.S. Paper Contributions at U.S.-located and Non-U.S.-located Conferences Average U.S. % Share of Conference Papers 1996 2001 2006 2011 All years U.S.-located conferences 71 66.3 75.3 64.4 69.2 (64% of total conferences analyzed) no. of conferences analyzed: 12 10 13 12 Non-U.S.-located conferences 58.6 66.8 53.4 69.6 62.1 (36% of total conferences analyzed) no. of conferences analyzed: 6 8 5 7 TABLE F-2 Comparison of U.S. Paper Contributions at U.S.-located and Non-U.S.-located Conferences (By Individual Conferences) U.S. % Share of Conference Papers SIGGRAPH MICRO VLDB HPCA POPL SOSP OSDI ISCA PLDI Average U.S. % Share of Papers When U.S.-located 81.8 88.3 87.3 90.2 52.1 63.1 71.4 96.2 53.8 76.0 When located elsewhere 83.7 93.1 86.5 86.8 44.2 79.0 52.7 87.7 58.3 74.7 * Conferences located in the United States for all years analyzed include: ASPLOS, IEDM, ISSCC, NANO, OOPSLA, PPoPP, and SC. Conferences located outside the United States for all years analyzed by the committee include: ECOOP, Eurographics, and WWW 78
OCR for page 79
PILOT STUDY OF PAPERS AT TOP TECHNICAL CONFERENCES 79 TABLE F-3 Conference Citation Analysis from Microsoft Scholar, June 2012 Semiconductor Devices and Circuits Papers Citations Citations/paper ISSCC 7,271 40,221 5.5 IEDM 9,886 47,925 4.8 ESSCIRC 1,498 5,090 3.4 A-SSCC 676 1,324 2.0 ESSDERC 1,553 2,511 1.6 NANO 2,028 2,266 1.1 Architectures Micro 905 31,607 34.9 ASPLOS 355 11,255 31.7 HPCA 661 17,891 27.1 ISCA 1,334 34,094 25.6 HiPEAC (Europe) 137 570 4.2 ACSAC (Asia) 339 750 2.2 Programming Systems POPL 1,267 68,945 54.4 PLDI 519 18,550 35.7 ECOOP (Europe) 783 26,054 33.3 * OOPSLA 835 14,712 17.9 PPoPP 319 5,641 17.7 Applications OSDI 255 20,373 79.9 SOSP 349 23,845 68.3 VLDB 2,739 12,2095 44.6 WWW 2,927 60,153 20.6 SIGGRAPH 3,492 100,567 28.8 Eurographics (Europe) 228 3,823 16.8 Supercomputing (SC) 2,994 36,868 12.3 PARLE (Europe) 416 3,804 9.1 * OOPSLA citation data was corrected by hand counting technical papers that exclude non-research track papers, which are rarely cited. Data compiled from Microsoft Scholar June 2012. Research Conference (ESSDERC), and IEEE European 350. While conference attendance statistics do not Solid-State Circuits Conference (ESSCIRC)]. In 2011, directly correlate with quality or influence, they indicate ISSCC, IEDM, and NANO had significant historical the level of interest in the technologies and ideas found attendance (according to the IEEE): 3,000, 1,500, and in the conference. 400, respectfully. In contrast, 2011 attendance at A- The committee expects that an analysis including too SSCC, ESSDERC, and ESSCIRC was between 300 and many (and thus a higher proportion of lower quality)
OCR for page 80
80 THE GLOBAL ECOSYSTEM IN ADVANCED COMPUTING conferences would underestimate the quality of leading F.2 Current National and Regional Advanced research efforts and obscure authorship trends for Research Capabilities in Four Key Technology leading research papers. On the other hand, too limited a Areas sampling that overprescribes measures of conference quality and impact would also skew the assessment. In this section, conference representation is used as a Lastly, the breadth of research topics covered by a proxy indicator of a nation's current (2011) advanced particular conference (and the same for journals that are research capabilities in each of the following key oftentimes broader) should also be considered to avoid technology areas: semiconductor devices and circuits, too broad a sweep of the technological field. In computer architecture, programming systems, and balancing these factors, the committee has identified a applications. For each technology area, all nations with limited number of top technical conferences across four at least 1 percent conference representation are ranked technology-specific research areas that, based on its based on their weighted authorship contributions expertise and deep domain knowledge of the field, are (following the methodology described in Section F.1.2) most critical to addressing the computing performance in the targeted conferences previously described in challenge described in Chapter 1. Sections F.1.3F.1.6. In addition, regional research capabilities are also provided for comparison. F.1.8 Methodological Summary As previously discussed, conference papers tend to have a much shorter time from submission to publication In summary, the committee believes that high than computing journals in related areas, resulting in the quality conferences attract leading researchers and most recent, significant innovation appearing at showcase significant, recent research contributions to the conferences first. Thus, an assessment of current field. In Sections F.2 and F.3, geographic distributions of conference research efforts is particularly relevant given conference authorship are quantified to provide a the increasing rate at which scientific discoveries are technology-specific assessment of national and regional made and then disseminated via the Web. Time series research capabilities.40,41 analyses of national and regional research capabilities The committee's analysis is not intended to be are shown in Section F.3. representative of all scientific outputs across the four specified research areas; for example, it does not F.2.1 Advanced Semiconductor Devices and Circuit presume that all relevant technologies presented in Research journal publications are implicitly represented in the selected conferences. The committee also recognizes the Table F-4 shows national capabilities in advanced significant and ongoing progress in bibliometric and semiconductor and nanoscale devices (IEDM and scientometric approaches to assess the quantity, quality, NANO) research, as well as semiconductor circuits and impact of scientific output. The assessment provided (ISSCC) research in 2011. The table shows all countries in subsequent sections is provided as an important first with at least 1 percent representation. The United States step toward new approaches to assess the global research has a strong competitive position in both of these areas landscape in specific advanced computing technologies. (50 percent in devices and 36 percent in circuits) followed by Japan, Taiwan, and Korea. China has only token representation in these conferences at this time. Figures F-2 and F-3 show the same data broken down by region. In semiconductor devices, the United States has the highest representation with half of the papers, followed 40 by Asia with less than one-third of the papers and This approach is consistent with a 2010 National Research Council (NRC) report, S&T Strategies of Six Countries: Implica- Europe with even fewer.42 In circuits, the United States, tions for the United States, in which conference publication analy- Asia, and Europe all share approximately one third of the sis was described as a "technology-specific indicator [that] gives a papers. NANO represents research that typically is relatively accurate picture of [national] S&T standing." Available targeted further in the future than those published in at http://www.nap.edu/catalog.php?record_id=12920. IEDM. At NANO the United States represents over 60 41 A related analysis was also conducted in a 2000 NRC report, Experiments in International Benchmarking of U.S. Research 42 Fields, which used U.S. contributions of papers at the annual Con- U.S. representation in semiconductor devices is reduced from ference on Magnetism and Magnetic Materials as a measure of ~50 percent to ~38 percent when only IEDM publications are con- U.S. participation in magnetic materials research. See sidered (see Figure F-7 and Table F-8). Despite this reduction, the http://www.nap.edu/catalog.php?record_id=9784. United States remains well ahead of Japan (~17 percent).
OCR for page 81
PILOT STUDY OF PAPERS AT TOP TECHNICAL CONFERENCES 81 TABLE F-4 Current (2011) National Capabilities in Advanced Semiconductor and Nanoscale Devices and Circuits Research (Measured by Percent Share of Conference Papers) Advanced Semiconductor Circuits Research 1 USA 36.4 2 Japan 12.4 3 Korea 10.4 4 Netherlands 7.3 5 Taiwan 6.7 6 Germany 5.3 7 Belgium 3.5 8 France 3.4 9 Italy 2.9 FIGURE F-2 Current (2011) regional capabilities in advanced semiconductor and nanoscale devices research (measured by 10 UK 2.3 percent share of conference papers). Data compiled from 11 China 2.0 IEDM and NANO. 12 Canada 1.8 13 Switzerland 1.6 14 Austria 1.0 Advanced Semiconductor and Nanoscale Devices Research 1 USA 50.2 2 Japan 10.4 3 Taiwan 6.8 4 Korea 5.7 5 UK 3.1 6 Germany 2.9 7 Canada 2.7 8 France 2.5 9 Belgium 2.2 10 Italy 2.1 11 China 1.9 FIGURE F-3 Current (2011) regional capabilities in advanced 12 India 1.8 semiconductor circuits research (measured by percent share of 13 Singapore 1.1 conference papers). Data compiled from ISSCC. 14 UAE 1.1 Data compiled from IEDM and NANO (semiconductor and nanoscale devices) and ISSCC (semiconductor circuits). percent of all papers, while Asia and Europe represent from ASPLOS, HPCA, ISCA, and MICRO in 2011. This less than 20 percent and 10 percent respectively. table includes all countries with at least 1 percent representation. Figure F-4 shows the same data broken F.2.2 Advanced Architecture Research down by region. U.S. research dominates that of any other nation or region and has a weighted average Table F-5 shows national and regional capabilities in representation of more than 85 percent of the papers. No advanced architecture research based on aggregated data other nation or region contributes more than 3 percent or
OCR for page 82
82 THE GLOBAL ECOSYSTEM IN ADVANCED COMPUTING 7 percent of the papers, respectively. This result is F.2.3 Advanced Programming Systems Research perhaps not surprising, given the U.S. historical dominance in commercial microprocessors, including Table F-6 shows national capabilities in advanced Intel, AMD, and IBM, as well as former commercial programming systems research based on aggregated data microprocessors from DEC, HP, and others. That Japan from ECOOP, OOPSLA, PLDI, POPL, and PPoPP. This is not represented on this list suggests that Japanese table includes all countries with at least 1 percent universities and industry research institutions are not representation. Figure F-5 shows the same data broken focused on mainstream computer architectures. While down by region. In programming systems research, the Japan has activity and expertise in the area, notably the United States dominates with approximately 60 percent custom processors from Fujitsu that are in the K of the papers. The national breakdown in Table F-6 supercomputer, their national research focus generally shows that aside from the United States, the national lies elsewhere. distribution of papers is diverse. Figure F-5 shows that in programming systems, Europe is a distant second to the United States with TABLE F-5 Current (2011) National Capabilities in about 25 percent representation, while all of Asia Advanced Architecture Research (Measured by Percent accounts for less than 10 percent. Disaggregating the Share of Conference Papers) data (see Appendix G), the United States has an even stronger position in PPoPP and PLDI, accounting for Advanced Architecture Research about 75 percent representation in each of them. These 1 USA 85.7 conferences are practical in nature, with papers 2 Korea 2.5 presenting prototype software systems and applications 3 France 2.2 running on real hardware platforms. In POPL, a more theoretical conference, the United States has a little less 4 China 1.7 than 45 percent of the papers, on par with European 5 Canada 1.6 presentation. Asia has less than 10 percent of the papers, 6 Switzerland 1.4 with greater representation in applications than 7 Australia 1.0 programming systems. In ECOOP, the United States Data compiled from ASPLOS, HPCA, ISCA, and MICRO. accounted for about 30 percent representation, putting it 15 percentage points behind Europe. TABLE F-6 Current (2011) National Capabilities in Advanced Programming Systems Research (Measured by Percent Share of Conference Papers) Advanced Programming Systems Research 1 USA 36.4 2 Germany 12.4 3 UK 10.4 4 Switzerland 7.3 5 Israel 6.7 6 France 5.3 7 Canada 3.5 8 India 3.4 9 Japan 2.9 FIGURE F-4 Current (2011) regional capabilities in advanced 10 China 2.3 architecture research (measured by percent share of conference 11 Denmark 2.0 papers). Data compiled from ASPLOS, HPCA, ISCA, and Micro. 12 Chile 1.8 13 Korea 1.6 Data compiled from ECOOP, OOPSLA, PLDI, POPL, and PPoPP.
OCR for page 83
PILOT STUDY OF PAPERS AT TOP TECHNICAL CONFERENCES 83 TABLE F-7 Current (2011) National Capabilities in Advanced Applications Research (Measured by Percent Share of Conference Papers) Advanced Applications Research 1 USA 63.9 2 Germany 6.0 3 Canada 4.7 4 China 4.3 5 France 2.4 6 Israel 2.3 7 UK 2.2 8 Hong Kong 1.9 9 Italy 1.5 FIGURE F-5 Current (2011) regional capabilities in advanced 10 Switzerland 1.5 programming systems research (measured by percent share of conference papers). Data compiled from ECOOP, OOPSLA, 11 Japan 1.2 PLDI, POPL, and PPoPP. 12 Korea 1.0 Data compiled from Eurographics, OSDI, SIGGRAPH, SC, SOSP, VLDB, and WWW. 2.2.4 Advanced Applications Research Table F-7 shows national capabilities in advanced applications research based on aggregated data from Eurographics, OSDI, SIGGRAPH, SC, SOSP, VLDB, and WWW in 2011. Figure F-6 shows this data broken down into regions. In applications research, the U.S. accounted for almost 64 percent of papers, followed by Germany, Canada, and China, with 5-6 percent representation each. Disaggregated data show that the United States is also the lead paper contributor in SOSP (~92 percent compared with ~4 percent in Asia), SC (~80 percent compared with 9 percent for both Asia and Europe), and WWW (~72 percent compared with ~13 percent in Europe and ~8 percent in Asia) and very strong leads in SIGGRAPH and VLDB (~53 percent, putting the United States ahead of Europe by approximately 30 points in FIGURE F-6 Current (2011) regional capabilities in advanced both conferences), suggesting the United States applications research (measured by percent share of conference maintains strong core competencies in parallel papers). Data compiled from Eurographics, OSDI, applications. SIGGRAPH, SC, SOSP, VLDB, and WWW.
OCR for page 84
84 THE GLOBAL ECOSYSTEM IN ADVANCED COMPUTING F.2.5 Summary of Current National Technological TABLE F-8 (19962011) National Capabilities in Leadership Advanced Semiconductor and Devices and Circuits Research (Measured by Percent Share of Conference In each of the four key technical areas, the United Papers) States holds the lead, with particularly strong Semiconductors & Nanoscale Devices representation in architecture research ~85 percent (compared to ~5 and 7 percent representation by Asia 1996 2001 2006 2011 and Europe, respectfully). In programming systems USA 39.3 40.6 34.2 37.9 research, the United States maintains the lead at ~36 Japan 34.4 24.5 21.1 16.9 percent followed by Europe at ~29 percent and Asia at Taiwan 2.3 5.0 4.5 8.2 ~8 percent. The strongest European paper contributors were Germany and the UK at ~12 and 10 percent, Korea 6.3 8.1 9.4 7.5 respectfully. In applications research, the United States Belgium 1.5 3.0 6.4 6.4 leads paper contributions at ~64 percent followed by France 3.4 2.3 4.4 6.0 Europe at ~17 percent and Asia at ~11 percent. This data Italy 3.9 3.4 2.8 3.7 suggests that the U.S. is particularly strong in design and Singapore - 1.7 4.5 3.0 systems engineering. While the United States maintains China 0.5 0.5 0.6 2.6 a strong lead in semiconductor and nanoscale devices (~50 percent) compared to Asia (~28 percent) and UK - - 1.7 1.9 Europe (~16 percent), its contributions in semiconductor Germany 3.0 5.4 3.7 1.7 circuits research are comparable to Europe and Asia Austria - 0.4 0.6 1.1 (each separated by ~7 percentage points). Canada 0.5 1.0 0.4 0.1 Netherlands 1.9 1.7 1.7 0.7 F.3 Longitudinal Changes in the Global Research Landscape Switzerland 2.1 0.8 2.2 0.6 Semiconductor Circuits To assess how the competitive research landscape has changed over time, the committee examined the 1996 2001 2006 2011 competitive metrics (described in Section F.1) for the USA 46.6 51.1 45.2 36.4 same set of conferences in the previous section in 5-year Japan 27.7 18.6 15.8 12.4 increments over a 15-year span from 1996 to 2011. Korea 3.9 4.7 6.1 10.4 Tables F-8 through F-11 show the relative Netherlands 2.0 4.0 3.1 7.3 representation of different nations in each of the Taiwan 0.6 0.1 7.0 6.7 technical areas of advanced computing research from 1996 to 2011 in 5-year increments, ranked by position in Germany 4.2 4.4 5.3 5.3 2011. Each of the tables includes only those nations that Belgium 1.0 5.5 1.5 3.5 have at least a 1 percent representation for at least one of France 0.9 0.8 1.0 3.4 the years. Countries that are not represented in 2011, but Italy 2.9 - 3.4 2.9 have at least 1 percent representation for any increment, UK 0.8 0.8 0.6 2.3 are marked in italicized text. Figures F-7 through F-11 show the aggregate regional representation for each area China - - 1.5 2.0 of advanced research over the same time span. For the Canada 3.4 2.2 1.4 1.8 time-series data, the semiconductor devices area includes Switzerland 4.3 2.7 1.9 1.6 only IEDM, because historical data for NANO was not Austria - - 2.3 1.0 easily available. Finland - 1.2 0.1 - In semiconductor devices research, U.S. representation has remained relatively stable with the Hong Kong - 1.2 - - largest gains made by Taiwan and Belgium (~ 5-6 Sweden 0.7 - 1.0 0.4 percentage point increases each). In semiconductor Data compiled from IEDM (semiconductor and nanoscale circuits research, the United States shows a moderate devices) and ISSCC (semiconductor circuits). decline, in tandem with an overall broadening in
OCR for page 85
PILOT STUDY OF PAPERS AT TOP TECHNICAL CONFERENCES 85 international representation. In this area largest leaps TABLE F-9 (19962011) National Capabilities in were made by Korea, Taiwan, and the Netherlands with Advanced Architecture Research (Measured by Percent 56 percentage point increases each. At the same time, Share of Conference Papers) Japan has dropped significantly in both semiconductor devices and in semiconductor circuits (~17 and 15 1996 2001 2006 2011 percent, respectfully). USA 79.9 89.2 90.7 85.7 In architecture research, the United States has Korea - 0.2 - 2.5 maintained a significant lead, with no major advances by France 3.1 1.0 0.7 2.2 any other nation or region. In programming systems, the U.S. lead has been challenged somewhat by increases in China - - - 1.7 Europe by small but steady gains by Israel, Switzerland, Canada 5.5 1.2 1.9 1.6 and the UK (as well as China, India, and Korea to a Switzerland - - - 1.4 lesser degree). Australia 0.4 - - 1.0 In the application areas, U.S. representation has Belgium - 1.0 0.3 - retained a stable lead over the 15-year period with no other nation ever contributing more than 8 percent India - - 1.9 0.6 (France, which contributed ~8 percent in 1996, has since Israel - - 1.2 - dropped to ~2 percent in 2011). While representing only Japan 2.1 2.0 1.2 - a small percentage of applications papers, China made a Spain 2.4 5.5 0.8 0.6 notable move from no representation in 1996 to ~4 Sweden 1.8 - - - percent in 2011. UK 2.0 - 0.3 0.7 Data compiled from ASPLOS, HPCA, ISCA, and Micro. TABLE F-10 (19962011) National Capabilities in TABLE F-11 (19962011) National Capabilities in Advanced Programming Systems Research (Measured Advanced Applications Systems Research by Percent Share of Conference Papers) (Measured by Percent Share of Conference Papers) 1996 2001 2006 2011 1996 2001 2006 2011 USA 62.2 63.8 67.1 56.5 USA 64.3 57.5 64.0 63.9 Germany 8.0 3.0 2.3 7.9 Germany 4.2 7.3 7.9 6.0 UK 5.2 4.3 7.1 7.2 Canada 3.5 3.7 3.5 4.7 Switzerland 1.9 2.9 1.8 4.5 China - 0.5 3.0 4.3 Israel 1.7 5.1 2.0 3.2 France 8.3 3.5 1.9 2.4 France 7.5 4.8 3.1 3.0 Israel 1.5 2.1 1.4 2.3 India - - - 2.1 UK 1.8 3.0 3.9 2.2 Japan 2.7 5.2 2.3 2.0 Hong Kong - 0.3 1.7 1.9 Canada 2.4 1.0 1.7 2.2 Italy 1.8 1.7 1.3 1.5 China - - 1.2 1.4 Switzerland 1.4 1.9 0.7 1.5 Denmark 2.4 2.3 0.9 1.4 Japan 3.7 5.6 3.0 1.2 Chile - - 0.2 1.1 Korea 0.4 1.1 0.7 1.0 Korea - - 0.4 1.0 Australia 0.7 1.0 0.4 0.4 Australia 0.4 0.6 1.8 0.7 Austria 1.3 1.5 0.5 0.7 Italy 1.1 2.5 0.2 0.9 India 0.7 0.8 1.7 0.6 Netherlands 2.3 2.1 0.8 0.9 Singapore 0.3 1.4 0.6 0.9 Sweden 0.5 - 1.7 0.2 Spain 1.0 1.5 0.5 0.4 Data compiled from ECOOP, OOPSLA, PLDI, Taiwan 1.0 0.9 0.8 0.6 POPL, and PPoPP. Data compiled from Eurographics, OSDI, SIGGRAPH, SC, SOSP, VLDB, and WWW.
OCR for page 86
86 THE GLOBAL ECOSYSTEM IN ADVANCED COMPUTING FIGURE F-7 (19962011) Regional capabilities in advanced semiconductor and nanoscale devices research (measured by percent share of conference papers). Data compiled from IEDM. FIGURE F-8 (19962011) Regional capabilities in advanced semiconductor circuits research (measured by percent share of conference papers). Data compiled from ISSCC. FIGURE F-9 (19962011) Regional capabilities in advanced architecture research (measured by percent share of conference papers). Data compiled from ASPLOS, HPCA, ISCA, and MICRO.
OCR for page 87
PILOT STUDY OF PAPERS AT TOP TECHNICAL CONFERENCES 87 FIGURE F-10 (19962011) Regional capabilities in advanced programming systems research (measured by percent share of conference papers). Data compiled from ECOOP, OOPSLA, PLDI, POPL, and PPoPP. FIGURE F-11 (19962011) Regional capabilities in advanced applications research (measured by percent share of conference papers). Data compiled from Eurographics, OSDI, SC, SIGGRAPH, SOSP, VLDB, and WWW.
OCR for page 88