Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 1
SUMMARY
During the past ten years a substantial literature on methods for the
evaluation of the performance of computer systems was developed by the
community of industrial and academic scientists and engineers.
Noteworthy successes were achieved in the evaluation of performance of
sequential architecture machines on applications comprising data
reduction and manipulation. For such systems and applications,
theoretical models and performance measures were developed with good
predictive power. In essence, these models enjoy a sound scientific
underpinning. Much less success and maturity, however, characterize the
methods and measures of performance evaluation of computer systems that
must execute scientific applications whose widely varied computational
structures lead to diverse and poorly characterizable workloads.
Performance evaluation is especially difficult and problematical for
supercomputers--general purpose systems devoted to large-scale
scientific computations--because of their complexity, diversity of
architectures, and the demanding nature of the applications they
undertake. The qualitative nature of the performance evaluation for
supercomputers departs from an emphasis on throughput and balanced
maximal utilization of all components of the system. The new emphasis
is primarily on the computational speed of the processors and
secondarily on assuring that specific demands by the computational
structure of the application on a particular component of the system do
not impede the potential performance of the whole system.
Thus, in spite of progress, methods and measures for the evaluation
of supercomputer performance need further improvement. Metrics, such as
millions of floating point operations per second (MFLOPS) and millions
of instructions per second (MIPS), and certain test program benchmarks
are repeatedly used. However, the interpretation of these quantities
and their use can be contradictory and misleading. There is ? at this
time, no commonly accepted methodology for the evaluation of the
performance of supercomputer systems and no standard set of metrics for
the representation of performance. More powerful evaluation methods are
essential for comparing competing systems, allocating available machine
resources, matching specific computing tasks to specific hardware, and
1
OCR for page 2
2
designing new systems. To be useful, evaluation methods should
encompass the complexity of supercomputer systems, their variety, and
their diverse applications. These are complex and novel expectations of
the evaluation process. Progress is being made, but there are strong
indications from the current debate in the literature and technical
journals, as well as in the trade press, that performance evaluation of
supercomputer systems has not developed to the desired level of
sophistication and coherence.
The importance of performance evaluation grows as the capability and
cost of the system grow. Supercomputers, vital components in the
industry and defense of the nation, are used precisely because they
provide the highest performance; and they are acquired at high cost.
Use of the full capability of such resources is an obvious aim.
The evaluation of a supercomputer should be based on more than its
speed, throughput, and cost. The issues of obsolescence, compatibility,
and adherence to accepted standards must be considered in both
acquisition and design. Consequently, these important criteria should
be an integral part of a comprehensive evaluation methodology .
New research programs are emphasizing the design and development of
more powerful supercomputers and of techniques for their utilization.
An inability to assess and measure accurately the performance of new
architectural designs, which often use many processors in parallel,
would inhibit effective development and invite the possibility of
overlooking fruitful ideas in architecture and algorithm design. In a
real sense, research in new computer architectural concepts drives the
need for performance evaluation soundly based in theory.
In recognition of this problem, this committee was to review the
state of the art in supercomputer performance evaluation, to identify
reasonable expectations for improvement, and to sketch an agenda for
research that would lead to the desired improvements. This report is an
initial step, which should be followed by support of research by the
funding agencies, conduct of research by the academic and industrial
communities, and consolidation of progress by a group of qualified
experts organized under some formal mechanism.
The remainder of this summary is organized around the conclusions and
recommendations of the report, drawing for support on key points from
the several chapters.
CONCLUS IONS
The conclusions of the study are stated below, each one followed by a
brief discussion.
OCR for page 3
3
Conclusion One
Some of the methods. and corresponding metrics, used in the past are
not well suited for the performance evaluation of current and future
supercomputer systems. ~
The questions central to the evaluation of supercomputer performance are
different in nature from those addressed by the established methods of
computer performance analysis; for this reason established methods have
not been successfully applied. For example, many current performance
models for supercomputers emphasize processor speed in terms of hardware
features (instructions, floating point operations, logical inferences,
and so forth), ignoring the performance of the other components of the
system, such as the operating system, compilers, and input-output
devices. Supercomputers have complex structures that make it hard to
apply these simple performance models. Today it is axiomatic that
supercomputer system performance is a function of problems,
implementation, and algorithms. The recent advent of diverse
architectures, radically different from each other and from the
sequential von Neumann type, has further invalidated a simple approach
and made it even more evident that there is an urgent need for the
development of methods, models, and metrics capable of describing the
performance of the overall system, both hardware and software. The
great variety of supercomputer structures in use or being proposed today
has precluded even a satisfactory taxonomy of systems, let alone an
accurate performance model.
Conclusion Two
Some performance measures and metrics in current use for the
evaluation of supercomputers, such as rates of hardware operation and
simple kernel execution are often misleading and misused.
The lack of good methods of performance evaluation for supercomputers
has led to extensive use of certain specific measures and benchmarks as
indicators of a system's performance. These can be helpful, but they
are hard to extrapolate accurately across a variety of machines and
algorithms. Several popular metrics (such as peak MFLOPS) and
benchmarks exist today; the performance of machines is often quoted in
terms of these. The existence of these metrics and benchmarks must be
regarded favorably, but their use raises serious questions of
oversimplification. They are not meaningful for a wide range of
circumstances and applications, and they do not represent the complex
relationships between the diverse hardware and software components of a
supercomputer. Often, metrics that measure merely hardware features are
poorly correlated with the performance of an entire system on a real
OCR for page 4
4
application. There is, at this time, no agreed upon methodology for the
use of benchmarks or their interpretation. There is considerable
mistrust of simplistic performance claims throughout the entire
supercomputer community.
Conclusion Three
No well developed scientific foundation exists for supercomputer
Performance evaluation.
Performance evaluation of supercomputers, especially those with parallel
architecture, has lagged behind other developments in computer science
and engineering. A scientific approach, embrac ing the diversity and
complexity of the systems and providing a clear methodology for
evaluating them, would offer better promise of success. A central issue
in understanding performance measurements on supercomputers involves the
pairing of architectures and applications. Recent work on the study of
architectural and algorithmic taxonomies has identified differences that
are critical to performance; further work could lead to the successful
development of useful and scientific performance models and more
systematic, comprehensive benchmarks.
The body of the report outlines a five-stage approach to performance
evaluation models: (1) determine the major supercomputer application
areas and the predominant mathematical solution techniques, (2) select a
collection of representative programs covering scientific disciplines
and solution techniques, (3) define the appropriate parameters of the
applications and the architectures that will allow models to be
developed, (4) define the metrics necessary to understand the
performance of the models relative to those parameters, and (5) assess
the relationship between the computational and architectural models.
The research agenda in the body of the report, outlined later in our
recommendations, should establish the scientific basis that is now
insufficiently developed for comprehensive evaluation methods.
Conclusion Four
Performance evaluation of supercomputers is an emerging area of
significant interest and importance. There are numerous ongoing
efforts by industrial. governmental, and academic laboratories: but
more effective mechanisms for teaching. assessing. and disseminating
progress in this area are desirable.
There is considerable activity by both vendors and buyers in
supercomputer system performance definition. There is increasing
research at academic, industrial, and national laboratories supported by
OCR for page 5
5
significant initiatives by the U.S. Department of Energy; the National
Science Foundation; and the U.S. Department of Defense, including the
Office of Naval Research and the Defense Advanced Research Projects
Agency. These activities reflect the increased importance, and need for
progress, in supercomputer performance evaluation. There do not seem to
be highly effective mechanisms for the discussion and dissemination of
research and of its transfer into methodology. In particular, the
committee sees no effective organization to evaluate, track, and
translate as rapidly as possible research and experience in this area
into commonly accepted criteria for performance evaluation.
RECOMMENDATIONS
On the basis of the foregoing conclusions, the committee offers the
following recommendations, each one stated first and then briefly
discussed.
Recommendation One
Performance evaluation methods for supercomputer performance, beyond
those already available should be developed and used: and these
methods should be based on sufficiently general and unifying concepts
to address the whole computer system including multiple processors
memory input-output subsystems, and software.
The current unsatisfactory situation In supercomputer performance
evaluation can be overcome because better performance models and more
systematic and comprehensive test programs are possible. The body of
this report outlines a reasonable approach to this end, which should
result in methods and measures having validity and usefulness comparable
to those currently in use by the general purpose computer performance
evaluation community.
The performance evaluation process is cyclic in nature, requiring
several iterations for a satisfactory result. We begin with the systems
of interest, either existing or proposed supercomputers, and generate
either empirical or analytical models of them. We experiment by
measuring performance, if the sys tem is an exis tiny computer, or by
simulating performance, if the system is a proposed one. The data from
the experiments are then combined to give several metrics of
performance. The system or the model of it, is modified and the cycle
is repeated.
OCR for page 6
6
Recommendation Two
Supercomputer systems should be provided with hardware and software
for the collection of performance evaluation data.
Some systems come equipped by the vendor with sophisticated hardware and
software for collecting performance data. This feature gives the
researcher and user the tools for experimentation and observation that
are essential to scientific understanding of the key issues underlying
performance evaluation.
Recommendation Three
Funding agencies should support more emphasis on supercomputer
performance evaluation methods in existing research and development
programs and should initate the support of basic research in the
science of supercomputer performance evaluation.
The body of the report suggests an agenda, or framework, for research,
the outcome of which is expected to provide the scientific underpinning
for a sound methodology of computer performance evaluation. The
research framework centers on the following topics:
1. Hierarchical characterization of applications and algorithms in
terms of their fundamental units of computation at several levels of
resolution.
2. Procedures for synthesizing fundamental arithmetic and logical
operations on arrangements of data into powerful computational
procedures (algorithms).
3. Procedures for synthesizing fundamental machine functions into
application-specific architectures that meet specific performance goals.
4. Methods for modeling and evaluating the performance of
supercomputers .
fundamental arithmetic and logica
into powerful computational
Recommendation Four
The U.S. Department of Energy and the National Science Foundation
should undertake a leadership role in establishing a formal mechanism
to track. assess and disseminate research results with a view to
bringing about commonly accepted methodologies of supercomputer
performance evaluation.
A number of centers supported by the Defense Advanced Research
Projects Agency and the Office of Naval Research are actively evaluating
OCR for page 7
novel architectures. The National Bureau of Standards, with support
from the Defense Advanced Research Projects Agency, has initiated a
research program on the evaluation of performance of diverse
architectures. The National Aeronautics and Space Administration both
conducts and supports much computer research. The U.S. Department of
Energy supports significant research in supercomputer performance
evaluation, both in its own laboratories and at universities. The
National Science Foundation provides research support in this area to
the academic community, and its Directorate for Computer and Information
Science and Engineering has a direct interest in supercomputing. Major
U.S. vendors have active research and development groups in
supercomputer evaluation methods.
The manifold activities taking place in this area would benefit by
increased communication and leveraging of effort. Although the
development of standards is certainly not appropriate now, the
development and distribution of a coherent set of measurement criteria
Is .
The committee believes it would be desirable for one or two of the
federal funding agencies, perhaps in cooperation with others, to
establish a group of experts who would, over a period of time, collect
and evaluate research results and experience in the area of
supercomputer performance evaluation and attempt to translate it into a
broadly usable collection of methods and measures.
The experience in
this field of the U.S. Department of Energy and the recent emphasis on
supercomputing at the National Science Foundation suggest that these are
the appropriate lead agencies for this task.
OCR for page 8
Representative terms from entire chapter:
supercomputer performance
