Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page R1
An Agenda for
Improved Evaluation
of Supercomputer Performance
A Report Prepared by the
Committee on Supercomputer Performance and Development
Energy Engineering Board
Commission on Engineering and Technical Systems
National Research Council
NATIONAL ACADEMY PRESS
Washington, D.C. 1986
OCR for page R2
NOTICE: The project that is the subject of this report was approved by
the Governing Board of the National Research Council, whose members are
drawn from the councils of the National Academy of Sciences, the National
Academy of Engineering, and the Institute of Medicine. The members of the
committee responsible for the report were chosen for their special
competences and with regard for appropriate balance.
This report has been reviewed by a group other than the authors
according to procedures approved by a Report Review Committee consisting
of members of the National Academy of Sciences, the National Academy of
Engineering, and the Institute of Medicine.
The National Research Council was established by the National Academy
of Sciences in 1916 to associate the broad community of science and
technology with the Academy's purposes of furthering knowledge and of
advising the federal government. The Council operates in accordance with
general policies determined by the Academy under the authority of its
congressional charter of 1863, which establishes the Academy as a private,
nonprofit, self-governing membership corporation. The Council has become
the principal operating agency of both the National Academy of Sciences
and the National Academy of Engineering in the conduct-of their services
to the government, the public, and the scientific and engineering
communities. It is administered jointly by both Academies and the
Institute of Medicine. The National Academy of Engineering and the
Institute of Medicine were established in 1964 and 1970, respectively,
under the charter of the National Academy of Sciences.
This is a report of work supported by Grant No. DE-FG05-85ER25008, from
the C.S. Department of Energy, and Grant No. N00014-85-G-0233, from the
Office of Naval Research, U.S. Department of the Navy, to the National
Academy of Sciences.
Copies available from:
Energy Engineering Board
Commission on Engineering and Technical Systems
National Research Council
2101 Constitution Avenue, N.W.
Washington, D.C. 20418
Printed in the United States of America
Second Printing
OCR for page R3
COMMITTEE ON SUPERCOMPUTER PERFORMANCE AND DEVELOPMENT
E. F. INFANTE (Chairman), University of Minnesota -
CLIFFORD N. ARNOLD, ETA Systems, Incorporated, St. Paul, Minnesota
ROBERT R. BORCHERS, Lawrence Livermore National Laboratory, Livermore,
California
JAMES C. BROWNE, Department of Physics and Computer Science, University
of Texas at Austin
BILL L. BUZBEE, Los Alamos National Laboratory, Los Alamos, New Mexico
ROBERT H. EWALD, Cray Research, Incorporated, Minneapolis, Minnesota
SIDNEY FERNBACH, Alamo, California
DUNCAN H. LAWRIE, Department of Computer Science, University of Illinois,
Urbana, Illinois
JOANNE L. MARTIN, International Business Machines Corporation, Yorktown
Heights, New York
VICTOR L. PETERSON, National Aeronautics and Space Administration, Moffett
Field, California
PAUL SCHNECK, Supercomputing Research Center, Lanham, Maryland
JACK WORLTON, Los Alamos, New Mexico
Liaison with Energy Engineering Board
THELMA ESTRIN, School of Engineering and Applied Science, University of
California at Los Angeles
Staff
DENNIS F. MILLER, Executive Director, Energy Engineering Board
JOHN M. RICHARDSON, Study Director, Committee on Supercompute~
Performance and Development
ROBERT COHEN, Senior Staff Officer, Committee on Supercomputer Performance
and Development
HELEN D. JOHNSON, Staff Associate, Energy Engineering Board
CARLITA M. PERRY, Administrative Secretary, Committee on Supercomputer
Performance and Development
. . .
OCR for page R4
OCR for page R5
ENERGY ENGINEERING BOARD
*HERBERT H. WOODSON, (Chairman), The University of Texas at Austin
WILLIAM R. GOULD, (Chairman), Southern California Edison Company,
Rosemead, California
ALLAN J. BARD, The University of Texas at Austin
*ROBERT J. BUDNITZ, Future Resources Associates, Inc., Berkeley,
California
THELMA ESTRIN, University of California at Los Angeles
CHARLES F. GAY, Arco Solar, Chatsworth, California
S. WILLIAM GOUSE, Mitre Corporation, McLean, Virginia
NICHOLAS J. GRANT, Massachusetts Institute of Technology, Cambridge
*BRUCE H. HANNON, University of Illinois at Urbana-Champaign, Urbana
GARY H. HEICHEL, University of Minnesota, St. Paul
JOSEPH M. HENDRIE, Brookhaven National Laboratory, Upton, New York
WILLIAM W. HOGAN, Harvard University, Cambridge, Massachusetts
BAINE P. KERR, Pennzoil Company, Houston, Texas
HENRY R. LINDEN, Gas Research Institute, Chicago, Illinois
EDWARD A. MASON, Amoco Research Center, Naperville, Illinois
ALAN D. PASTERNAK, Energy Consultant, Sacramento, California
THOMAS H. PIGFORD, University of California, Berkeley
ADEL F. SAROFIM, Massachusetts Institute of Technology,
MAXINE L. SAVITZ. The Garrett Corcoration. Los Anaeles.
Cambridge
, ~ California
WESTON M. STACEY, JR., Georgia Institute of Technology, Atlanta
RICHARD STEIN, The Stein Partnership, New York, New York
THOMAS E. STELSON, Georgia Institute of Technology, Atlanta
LEON STOCK, University of Chicago, Illinois
GEORGE S. TOLLEY, University of Chicago, Illinois
DAVID C. WHITE, Massachusetts Institute of Technology, Cambridge
RICHARD WILSON, Harvard University, Cambridge, Massachusetts
Technical Advisory Panel
HAROLD M. AGNEW, GA Technologies, Inc., Solana Beach, California
FLOYD L. CULLER, JR., Electric Power Research Institute, Palo Alto,
California
CHAUNCEY STARR, Electric Power Research Institute, Palo Alto,
California
ALBERT R.C. WESTWOOD, Martin Marietta Labs, Martin Marietta
Corporation, Baltimore, Maryland
Staff
Dennis F. Miller, Executive Director, Energy Engineering Board
Helen D. Johnson, Staff Associate
Cheryl A. Winter, Staff Assistant
*Term expired June 30, 1986
OCR for page R6
OCR for page R7
PREFACE
This report presents the results of a brief study to assess current
methods of evaluating supercomputer performance, identify opportunities
for improvement, and recommend the outlines of a research agenda to
realize these opportunities.
A substantial scientific community and body of literature are devoted
to computer performance measurement and evaluation. Impressive successes
have been achieved in theory, prediction, and experimentation for systems
that have sequential architecture and applications that involve data
handling and manipulation. Less successful and mature are the methods and
measurements for evaluation of system performance when scientific problems
are run on supercomputers of diverse architectural characteristics.
Performance evaluation is especially important for supercomputers, not
merely because they are a costly resource but, more fundamentally, because
they must often work at the limits of their performance to produce the
results wanted. The complexity of these systems, together with the
increasing diversity of computing architectures becoming available, makes
evaluation of supercomputer performance more problematical, yet more
important, than for the more common midrange computer systems. Moreover,
what is learned about the performance of operational systems can be
expected to guide the design of future systems.
The U.S. Department of Energy and the Office of Naval Research, in
response to these needs, arranged with the National Research Council to
assess the improvements attainable in performance evaluation of
large-scale scientific computers.
On the basis of its own experience and its contacts with members of
research and development community, the committee established for the
purpose concluded that current performance measures and methods for
supercomputer evaluation do indeed need significant improvement.
Moreover, the committee judged that no well developed scientific
foundation exists for supercomputer performance evaluation.
In consequence, this report paints in broad strokes the outlines of
this topic, its problems, and its possibilities. Filling in the details
vii
OCR for page R8
of the panorama requires the successful accomplishment of research outlined
in this report. In addition, it requires the help of a formal mechanism of
the research community to track, assess, and disseminate research results
with a view to bringing about commonly accepted methodologies of performance
evaluation.
The report is directed to persons in the federal agencies and elsewhere
who fund research, and to the industrial and academic researchers who
perform it.
On behalf of the committee members I wish to voice our gratitude
interest and support of Donald M. Austin, of the U.S. Department
and Charles J. Holland, of the Office of Naval Researcn.
. , ,
research results
of
for the
Energy,
Carl Died, of Cray
Research; Troy L. Wilson, of International Business Machines Corporation;
and the members of the Parallel Processing Group of the National Bureau of
Standards generously shared their knowledge with members of the committee
In the course of his review of the report manuscript, Peter J. Denning, of
the Research Institute for Advanced Computer Science, provided the committee
with valuable contributions to the topics presented here. The task was
initiated by Dennis F. Miller, Executive Director of the Energy Engineering
Board. The contributions to the committee's work by John M. Richardson and
of the board's staff, are most gratefully acknowledged.
Carlita M. Perry handled the manifold administrative tasks with grace, tact,
and competence.
Finally, I wish to thank the members of the committee for their extensive
contributions of time and effort, and for their selfless sharing of
knowledge and experience.
I acknowledge all these contributions with sincere gratitude.
Robert Cohen, of the board's
.
E. F. Infante
viii
Chairman
committee on Supercomputer
Performance and Development
OCR for page R9
CONTENTS
SUMMARY 1
INTRODUCTION.
The Problem / 9
Approach to the Study ,
/ 11
PRINCIPLES FOR THE EVALUATION OF SUPERCOMPUTERS.
The Context for Evaluation / 13
Criteria for Evaluation / 14
Matching Computational and Application Characteristics
The Cycle of Performance Evaluation ~ 17
/ 17
13
THE CURRENT STATE OF THE ART IN EVALUATION OF SUPERCOMPUTERS.. 21
Lessons from the Past
/ 21
Performance Metrics / 23
Test Programs for Performance Measurement / 24
Analytical Approach to Performance Evaluation
Summary / 28
References / 29
/ 27
4 IMPROVEMENTS ATTAINABLE IN PERFORMANCE EVALUATION 31
Models and Classification Schemes / 32
Requirements for Produc ing Measurements
Five Stages in Performance Evaluation
System Performance Issues / 41
Conclusion / 42
References / 43
1X
/ 37
/ 38
OCR for page R10
5 AN AGENDA FOR RESEARCH.
Opportunities / 5.1
Research Program Specifications
Conduct of Research Program / 5.7
APPENDIX A STATEMENT OF TASK.
APPENDIX B ANNOTATED SHORT BIBLIOGRAPHY OF ALGORITHM- OR.
APPLICATION-SPECIALIZED COMPUTER SYSTEMS
x
/ 5.2
53
