Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
SUMMARY During the past ten years a substantial literature on methods for the evaluation of the performance of computer systems was developed by the community of industrial and academic scientists and engineers. Noteworthy successes were achieved in the evaluation of performance of sequential architecture machines on applications comprising data reduction and manipulation. For such systems and applications, theoretical models and performance measures were developed with good predictive power. In essence, these models enjoy a sound scientific underpinning. Much less success and maturity, however, characterize the methods and measures of performance evaluation of computer systems that must execute scientific applications whose widely varied computational structures lead to diverse and poorly characterizable workloads. Performance evaluation is especially difficult and problematical for supercomputers--general purpose systems devoted to large-scale scientific computations--because of their complexity, diversity of architectures, and the demanding nature of the applications they undertake. The qualitative nature of the performance evaluation for supercomputers departs from an emphasis on throughput and balanced maximal utilization of all components of the system. The new emphasis is primarily on the computational speed of the processors and secondarily on assuring that specific demands by the computational structure of the application on a particular component of the system do not impede the potential performance of the whole system. Thus, in spite of progress, methods and measures for the evaluation of supercomputer performance need further improvement. Metrics, such as millions of floating point operations per second (MFLOPS) and millions of instructions per second (MIPS), and certain test program benchmarks are repeatedly used. However, the interpretation of these quantities and their use can be contradictory and misleading. There is ? at this time, no commonly accepted methodology for the evaluation of the performance of supercomputer systems and no standard set of metrics for the representation of performance. More powerful evaluation methods are essential for comparing competing systems, allocating available machine resources, matching specific computing tasks to specific hardware, and 1
2 designing new systems. To be useful, evaluation methods should encompass the complexity of supercomputer systems, their variety, and their diverse applications. These are complex and novel expectations of the evaluation process. Progress is being made, but there are strong indications from the current debate in the literature and technical journals, as well as in the trade press, that performance evaluation of supercomputer systems has not developed to the desired level of sophistication and coherence. The importance of performance evaluation grows as the capability and cost of the system grow. Supercomputers, vital components in the industry and defense of the nation, are used precisely because they provide the highest performance; and they are acquired at high cost. Use of the full capability of such resources is an obvious aim. The evaluation of a supercomputer should be based on more than its speed, throughput, and cost. The issues of obsolescence, compatibility, and adherence to accepted standards must be considered in both acquisition and design. Consequently, these important criteria should be an integral part of a comprehensive evaluation methodology . New research programs are emphasizing the design and development of more powerful supercomputers and of techniques for their utilization. An inability to assess and measure accurately the performance of new architectural designs, which often use many processors in parallel, would inhibit effective development and invite the possibility of overlooking fruitful ideas in architecture and algorithm design. In a real sense, research in new computer architectural concepts drives the need for performance evaluation soundly based in theory. In recognition of this problem, this committee was to review the state of the art in supercomputer performance evaluation, to identify reasonable expectations for improvement, and to sketch an agenda for research that would lead to the desired improvements. This report is an initial step, which should be followed by support of research by the funding agencies, conduct of research by the academic and industrial communities, and consolidation of progress by a group of qualified experts organized under some formal mechanism. The remainder of this summary is organized around the conclusions and recommendations of the report, drawing for support on key points from the several chapters. CONCLUS IONS The conclusions of the study are stated below, each one followed by a brief discussion.
3 Conclusion One Some of the methods. and corresponding metrics, used in the past are not well suited for the performance evaluation of current and future supercomputer systems. ~ The questions central to the evaluation of supercomputer performance are different in nature from those addressed by the established methods of computer performance analysis; for this reason established methods have not been successfully applied. For example, many current performance models for supercomputers emphasize processor speed in terms of hardware features (instructions, floating point operations, logical inferences, and so forth), ignoring the performance of the other components of the system, such as the operating system, compilers, and input-output devices. Supercomputers have complex structures that make it hard to apply these simple performance models. Today it is axiomatic that supercomputer system performance is a function of problems, implementation, and algorithms. The recent advent of diverse architectures, radically different from each other and from the sequential von Neumann type, has further invalidated a simple approach and made it even more evident that there is an urgent need for the development of methods, models, and metrics capable of describing the performance of the overall system, both hardware and software. The great variety of supercomputer structures in use or being proposed today has precluded even a satisfactory taxonomy of systems, let alone an accurate performance model. Conclusion Two Some performance measures and metrics in current use for the evaluation of supercomputers, such as rates of hardware operation and simple kernel execution are often misleading and misused. The lack of good methods of performance evaluation for supercomputers has led to extensive use of certain specific measures and benchmarks as indicators of a system's performance. These can be helpful, but they are hard to extrapolate accurately across a variety of machines and algorithms. Several popular metrics (such as peak MFLOPS) and benchmarks exist today; the performance of machines is often quoted in terms of these. The existence of these metrics and benchmarks must be regarded favorably, but their use raises serious questions of oversimplification. They are not meaningful for a wide range of circumstances and applications, and they do not represent the complex relationships between the diverse hardware and software components of a supercomputer. Often, metrics that measure merely hardware features are poorly correlated with the performance of an entire system on a real
4 application. There is, at this time, no agreed upon methodology for the use of benchmarks or their interpretation. There is considerable mistrust of simplistic performance claims throughout the entire supercomputer community. Conclusion Three No well developed scientific foundation exists for supercomputer Performance evaluation. Performance evaluation of supercomputers, especially those with parallel architecture, has lagged behind other developments in computer science and engineering. A scientific approach, embrac ing the diversity and complexity of the systems and providing a clear methodology for evaluating them, would offer better promise of success. A central issue in understanding performance measurements on supercomputers involves the pairing of architectures and applications. Recent work on the study of architectural and algorithmic taxonomies has identified differences that are critical to performance; further work could lead to the successful development of useful and scientific performance models and more systematic, comprehensive benchmarks. The body of the report outlines a five-stage approach to performance evaluation models: (1) determine the major supercomputer application areas and the predominant mathematical solution techniques, (2) select a collection of representative programs covering scientific disciplines and solution techniques, (3) define the appropriate parameters of the applications and the architectures that will allow models to be developed, (4) define the metrics necessary to understand the performance of the models relative to those parameters, and (5) assess the relationship between the computational and architectural models. The research agenda in the body of the report, outlined later in our recommendations, should establish the scientific basis that is now insufficiently developed for comprehensive evaluation methods. Conclusion Four Performance evaluation of supercomputers is an emerging area of significant interest and importance. There are numerous ongoing efforts by industrial. governmental, and academic laboratories: but more effective mechanisms for teaching. assessing. and disseminating progress in this area are desirable. There is considerable activity by both vendors and buyers in supercomputer system performance definition. There is increasing research at academic, industrial, and national laboratories supported by
5 significant initiatives by the U.S. Department of Energy; the National Science Foundation; and the U.S. Department of Defense, including the Office of Naval Research and the Defense Advanced Research Projects Agency. These activities reflect the increased importance, and need for progress, in supercomputer performance evaluation. There do not seem to be highly effective mechanisms for the discussion and dissemination of research and of its transfer into methodology. In particular, the committee sees no effective organization to evaluate, track, and translate as rapidly as possible research and experience in this area into commonly accepted criteria for performance evaluation. RECOMMENDATIONS On the basis of the foregoing conclusions, the committee offers the following recommendations, each one stated first and then briefly discussed. Recommendation One Performance evaluation methods for supercomputer performance, beyond those already available should be developed and used: and these methods should be based on sufficiently general and unifying concepts to address the whole computer system including multiple processors memory input-output subsystems, and software. The current unsatisfactory situation In supercomputer performance evaluation can be overcome because better performance models and more systematic and comprehensive test programs are possible. The body of this report outlines a reasonable approach to this end, which should result in methods and measures having validity and usefulness comparable to those currently in use by the general purpose computer performance evaluation community. The performance evaluation process is cyclic in nature, requiring several iterations for a satisfactory result. We begin with the systems of interest, either existing or proposed supercomputers, and generate either empirical or analytical models of them. We experiment by measuring performance, if the sys tem is an exis tiny computer, or by simulating performance, if the system is a proposed one. The data from the experiments are then combined to give several metrics of performance. The system or the model of it, is modified and the cycle is repeated.
6 Recommendation Two Supercomputer systems should be provided with hardware and software for the collection of performance evaluation data. Some systems come equipped by the vendor with sophisticated hardware and software for collecting performance data. This feature gives the researcher and user the tools for experimentation and observation that are essential to scientific understanding of the key issues underlying performance evaluation. Recommendation Three Funding agencies should support more emphasis on supercomputer performance evaluation methods in existing research and development programs and should initate the support of basic research in the science of supercomputer performance evaluation. The body of the report suggests an agenda, or framework, for research, the outcome of which is expected to provide the scientific underpinning for a sound methodology of computer performance evaluation. The research framework centers on the following topics: 1. Hierarchical characterization of applications and algorithms in terms of their fundamental units of computation at several levels of resolution. 2. Procedures for synthesizing fundamental arithmetic and logical operations on arrangements of data into powerful computational procedures (algorithms). 3. Procedures for synthesizing fundamental machine functions into application-specific architectures that meet specific performance goals. 4. Methods for modeling and evaluating the performance of supercomputers . fundamental arithmetic and logica into powerful computational Recommendation Four The U.S. Department of Energy and the National Science Foundation should undertake a leadership role in establishing a formal mechanism to track. assess and disseminate research results with a view to bringing about commonly accepted methodologies of supercomputer performance evaluation. A number of centers supported by the Defense Advanced Research Projects Agency and the Office of Naval Research are actively evaluating
novel architectures. The National Bureau of Standards, with support from the Defense Advanced Research Projects Agency, has initiated a research program on the evaluation of performance of diverse architectures. The National Aeronautics and Space Administration both conducts and supports much computer research. The U.S. Department of Energy supports significant research in supercomputer performance evaluation, both in its own laboratories and at universities. The National Science Foundation provides research support in this area to the academic community, and its Directorate for Computer and Information Science and Engineering has a direct interest in supercomputing. Major U.S. vendors have active research and development groups in supercomputer evaluation methods. The manifold activities taking place in this area would benefit by increased communication and leveraging of effort. Although the development of standards is certainly not appropriate now, the development and distribution of a coherent set of measurement criteria Is . The committee believes it would be desirable for one or two of the federal funding agencies, perhaps in cooperation with others, to establish a group of experts who would, over a period of time, collect and evaluate research results and experience in the area of supercomputer performance evaluation and attempt to translate it into a broadly usable collection of methods and measures. The experience in this field of the U.S. Department of Energy and the recent emphasis on supercomputing at the National Science Foundation suggest that these are the appropriate lead agencies for this task.
