electrical, or optical engineering. High-performance computing has become necessary to make full use of many observations. For example, powerful algorithms have greatly enhanced imaging telescopes that operate in the radio and in the optical domains (Figure 5.1; see also Plate 4.3). The VLA that astronomers use today is much more powerful and flexible than the one that was originally designed, without any major modification of the telescope itself, because of more powerful processing at the VLA and at national supercomputer centers.
The volume of data produced by some astronomical instruments is already large and is growing rapidly. Data rates of 10 gigabytes per day are common, and 100 gigabytes per day may soon be exceeded. Detector arrays will soon produce two-dimensional images with up to 2,048 × 2,048 pixels (elements), and some instruments will add spectral, temporal, or polarization channels to generate even larger datasets. The data flow from the VLA can exceed 72 gigabytes per day. The current VLA computers cannot handle this maximum data flow, so that subsets of the data must be selected for transmission and analysis. This under-utilization could become even more dramatic for the next generation of optical and infrared telescopes that will use arrays consisting of millions of individual detectors. Computers capable of processing 30 gigabytes per night will be as essential as improved detectors for the optical and infrared sky surveys planned for the 1990s. Without adequate computational and data storage capabilities, astronomers will not be able to push current or planned ground-based telescopes, which cost tens of millions of dollars each, to their limits.
Intensive data processing is often required to convert observations to understanding. The type of data processing required varies widely depending on the telescope and the purpose of the observations. Large user facilities with stable instrumentation and operating conditions like the VLA lend themselves to processing in which the typical user has little involvement until the final analysis stages. Similarly, specialized survey instruments like the Infrared Astronomical Satellite (IRAS) need production software to process large volumes of raw data into scientifically useful catalogs and images. Software development for these instruments is often best handled by professional software engineers working in close conjunction with astronomers who understand the technical problems and scientific goals and who are familiar with details of the instruments.
The problem of making general-purpose data-reduction software is a difficult one. The National Optical Astronomy Observatories developed the Image Reduction and Analysis Facility (IRAF) package as a community program for reducing, calibrating, and analyzing images or two-dimensional spectra from optical and infrared telescopes. Similarly, the National Radio Astronomy Observatory created the Astronomical Image Processing System (AIPS) for