Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Partitioning Visual Processes WALTER MAKOUS IDENTIFYING AND SEQUENCING OPERATIONS Method and Rationale The work reported here is based on the idea that vision consists of the transformation of spatiotemporal patterns of light into behavior and experience. It follows that a visual scientists goal is to understand the operations that accomplish this transformation and the paths that signals take through those operations. In human vision the pursuit of this goal is complicated by the fact that the operations and their organization usually must be inferred from the relationship between input and output (i.e., from psychophysics). If the visual system were entirely linear, any given relationship between input and output could be produced by many different combinations of operations and choosing among them would be either guesswork or a matter of taste. So nonlinear processes are what allow us to pare down the alternative representations of the system and, when they have been pared down enough, to know what goes on inside. The way nonlinearities constrain the alternatives can be seen from a simple example. Consider a system with To inputs, a and b, that are combined to determine a single output, c. If there is a nonlinearity in the system, such that c or log a and c or log b, one can establish whether the inputs sum before the log transformation ti.e., c or log (`a + b)] or after it [i.e., c or (log a) + (log b)] by varying both a and b while holding their sum constant (a ~ b = k). If the inputs sum before the log transformation, the variations of a and b have no effect on the output; if they sum after the log transformation, variations of a and b do affect the output This is true of 78
PARTITIONING VISUAL PROCESSES 79 any nonlinear transformation, not just the log transformation used in this example. The method my co-workers and I used is directly analogous to this example. The two inputs, a and b, correspond to the illumination of rows of adjacent cones. Under some conditions, differential illumination of adjacent cones, such as that produced by a 60 c deg-i grating, cannot be discriminated from a homogeneous field; that is, the input from adja- cent cones is summed to produce a single-valued output, c. 1b determine whether this summation occurs before or after any nonlinear transforma- tions, we systematically vary the relative illumination of adjacent cones so that in some regions of the retina adjacent cones are equally illuminated (a = by, and in other regions they receive very different illumination (a b); but the sum is always held constant (a + b = k). The necessary conditions of illumination are produced by two laser interferometers. The first was invented by Dave Williams. As its design has been published (Williams, 1985a), I shall not describe it here except to say that it allows precise and facile control over the contrast, orientation, and spatial frequency of gratings created by optical interference, which can exceed 200 c degas. As the changes of contrast are rapid and afford no extraneous cues, the instrument supports forced choice observations. The second interferometer, of simpler design (MacLeod et al., 1985), allows superimposition of two separate interference gratings on the retina, such as was done first by Burton (1973~. The result of such superposition is illustrated in Figure 1. The first two lines represent the profiles of the two sine wave gratings created by optical interference. When they are superimposed, the retinal illuminances at each point add, producing the profile of "beats" shown in the third line, where modulation is very high in some regions and nearly absent in others. If the period of the gratings is about twice the diameter of a cone, adjacent cones receive very different illumination (a ~ b) in the places where modulation of the beat pattern is high and very nearly the same illumination (a ~ by in regions where modulation is low, but the sum of the illumination of adjacent stripes is always the same (a + b = k). If the excitation at each point is a nonlinear function of illumination, such as is produced by a rectifer, clipper, or abrupt saturation (as illustrated by the function graphed in the inset between the third and fourth lines in Figure 1), the excitation has the profile shown in the fourth line. The top of the curve is clipped off, and of course the local space average tends to be more negative in certain regions than in others. These variations of negativity show up in the Fourier spectrum as a new component (shown by the smooth curve) that has been introduced by the nonlinear distortion; its frequency is equal to the difference between the frequencies of the two
80 LL o CL Cl) 2 2 WALTER MUCOUS ID MU o L I I I ~ ~ A ~ ~1 \ /, \, \, , \, \ , \ , \ , \ , \ , J I ~ I ~ ~ \ \ / ~ ~ \ ~ \ ~ I_ ~ 1 1 1) 1~ 1 ~ / \ 1_ . - ~ Vow - - ~ ~ 1 11 0 200 400 600 BOO tOOD SPACE FIGURE 1 Formation of a difference frequency distortion product. The third curve is the sum of corresponding points in the fimt two. The bottom curve is this sum, transformed by the nonlinearity depicted by the graph between the two: vertical fluctuations of the third curve are the input, on the horizontal axis of the graph; and output, on the vertical axis of the graph, determines the vertical fluctuations of the last curve. Hence, variation of the input beyond the modpoint produces no variation in the output. The smooth curse superimposed on the output shows the frequency and phase of the distortion product that is a Fourier component of this output curve, on which it is superimposed.
PARTITIONING VISUAL PROCESSES 81 superimposed gratings. Any nonlinear distortion produces this difference frequency; different nonlinearities simply produce different amplitudes of the difference grating. Figure 2 shows how we use this distortion product. It depicts the information in two gratings of 60 and 70 c/de" entering a nonlinear stage. Coming out of the nonlinear stage are signals of the original frequencies, plus those of many other frequencies that are the products of the nonlinear distortion. In the human visual system these signals next encounter a low- pass filter that blocks everything but that of the lowest frequency, 10 c/de". Subjectively, the observer sees only the low-frequency distortion product without seeing the high-frequency gratings that produced it. The essence of the technique lies in the fact that signals of only two input frequencies exist before the nonlinear stage and only the low-frequency distortion product exists after the nonlinear stage. If one increases or decreases the frequencies of the two gratings while keeping the difference between their frequencies and everything else constant, the distortion product must remain unchanged. If this shift of frequencies does change the amplitude of the distortion product, it can only be due to the changes of the amplitudes of the signals elicited by the gratings at the site of the nonlinear stage. So the detectability of the distortion product can be used to measure the attenuation of different frequencies by whatever precedes the nonlinear stage. Conversely, by changing the difference between the frequencies of input gratings without changing their mean frequency, the frequency of the distortion product can be varied with little effect on the signal passing through the stages that precede the nonlinearity. As the input pattern changes little, as there is little change in the signal reaching the nonlinear stage, and as the distortion product exists only after the nonlinearity, any effect on detectability of the distortion product can be caused only by attenuation of the signal after the nonlinear stage. In practice, instead of using gratings of different spatial frequencies, we use gratings of equal spatial frequencies and create a distortion product by rotating the gratings slightly in opposite directions. This produces a pattern similar to a moire pattern that has a distortion product corresponding to a sinusoidal grating oriented nearly perpendicular to the gratings producing it. The spatial frequency of the distortion product is changed by slightly changing the relative orientations of the gratings. Thus, there is no change of spatial frequency preceding the nonlinear process, and if the stages preceding the nonlinear process are reasonably isotropic, this procedure affects nothing except those processes that depend on the spatial frequency of the distortion product.
82 ~ o o CD 0 ~4 4~ at 0 on WALTER MAKOUS WWW: 'We o to to WVWVV~- ~A ~NVVVW WWW: Non 1 in e an it y Lo w Pa ss Fi 1 te r FIGURE 2 Frequencies present at successive stages of the visual system. Two sine waves enter, and no other signals are present preceding the nonlinear stage. So processes preceding the nonlinear stage are tested only at these input frequencies. The nonlinearity creates a vanet~,r of distortion products, one of which is the difference Sequence. A subsequent low-pass filter blocks the input signals as well as all the distortion products except the lowest frequency, the difference frequency. The entering sine waves create the signal that is detected (the difference frequency) at the stage of the nonlinearity, and whatever happens to them subsequently is irrelevant. Only the difference frequency can be detected at the end of the system. The difference frequency exists only following the nonlinear stage, and so cannot be affected By what precedes the nonlinear stage (except indirectly through the effects on the input signals). So the processes that precede the nonlinearity are tested at the frequencies of the input signals, and the processes that follow the nonlinearity are tested at the frequency of the difference between the input frequencies. Illustrative Results Basic Partitions Figure 3 shows schematically how this interferometric technique parti- tions visual processes. Interference gratings are unaffected by aberrations of the eye's optics and for many purposes must be treated as though they originate on the retina. This allows one to separate the effects of the eye's optics from other influences on the spatial information in visual stimili. Campbell and Green (1965) used this to measure the losses of contrast attributable to the eye's optics. This they did by comparing the contrast
PARTITIONING VISUAL PROCESSES S , 1 1 1 R 83 Optics First Spatial Filter (Receptor Aperture) Nonlinearity Second Spatial Filter FIGURE 3 The four compartments into which the visual system is partitioned by the interferometer and the nonlinearity.
84 WALTER MAXOUS sensitivity of directly viewed gratings, which are degraded by the optics of the eye, to the sensitivity to interference fringes, which are not. The procedures used here allow further separation of the causes of losses of contrastthat is, separation of those between the optics and the nonlinear stage (labeled here the first filter or the receptor aperture) from those that follow the nonlinear stage (here called the second filter). Losses Preceding the Nonlinear Stage When distortion products of fixed frequency are created by superim- posed interference gratings of variable frequency, the detectability of the distortion products decreases with increasing spatial frequency of the grat- ings (MacLeod et al., 1985; Chen et al., 1988~. Data from one observer are shown by the filled triangles in Figure 4 (MacLeod et al., 1985~. Therefore, a stage of spatial filtering, separate from aberrations of the eye's optics, precedes the nonlinear stage. However, detectability of the distortions does not decrease greatly until the spatial frequencies of the gratings approach very high values. This means that spatial filtering preceding the nonlinear stage is not great. Information in retinal images must enter the visual system through a set of individual receptors that for many purposes can be considered identical parallel channels. The optical apertures of these receptors constitute low- pass filters. The attenuations of contrast inferred from the detections are just what would be produced by filtering the gratings through an optical aperture of 24" of arc (mean of four observers, with a standard deviation of 6"~. This is close to 80 percent of the anatomical diameter of foveal cones (Curcio et al., 1987), as predicted by Miller and Bernard (1983~. This allows for no additional filtering. Since all the spatial information entering the visual system must pass through the aperture of a receptor, and since filtering accounts for all that is measured, we take it as the average optical aperture for the foveal cones of these observers. Consequently, the box in Figure 3 representing the spatial filtering following the optics of the eye and preceding the nonlinear stage is labeled receptor aperture. The psychologically measured aperture 3.8 deg from the fovea is 52.4" of arc (mean of four observers, with a standard deviation of 11") (MacLeod et al., 1985; Chen et al., 1988~. The difference in the psychophysically measured apertures is in proportion to the difference in cone sizes (Curcio et al., 1987~. Losses Following the Nonlinear Stage The octagons in Figure 4 show the attenuations of varying spatial fre- quencies by those processes that follow the nonlinear stage (Chen et al.,
PARTITIONING VISUAL PR~ESSES ~ Ll-) ._ . .> - ._ in c a) cn _. in o (3 LO C) 85 In cur O Overall CSF · Combined Fillers O - ~1 WM E: ~ I 0 _^ ~6, ~1 ' AL First Filter Second Filter': I_ lo 0 ~ 1 1 1 1 1 1 0 20 40 60 80 100 120 140 Spatial Frequency FIGURE 4 Log attenuation of signals of varying frequency (e deg- 1 ) measured by sensitivity to the contrast of interference fringes. Unfilled squares show the negative log contrast of interference fringes at threshold when the fringes are presented alone. The octagons show the negative log contrast of a 60-c deg 1 interference fringe superimposed on another 60-e deg 1 fringe of 100 percent contrast. The angle between the two fringes was adjusted to produce distortion products at the frequencies shown on the horizontal axis. Only the distortion product determined threshold. The triangles represent half the negative log contrast of an interference fringe superimposed on another fringe of 100 percent contrast, where the frequency of the distortion product was 10 e deg 1, and the frequency of the fringes is represented by the positron of the points on the horizontal axis. Only the distortion product determined threshold. (Division by 2 is necessary to allow for the feet that both fringes are attenuated.) The filled diamonds are the sum of the octagons and tnangles, displaced vertically as a set to fit the unfilled squares. 1988). This combines in a single curve all the influences on transmission of spatial information following the nonlinear stage, whether a direct con- sequence of the receptive fields of visual neurons or such things as spatial uncertainty (Pelli, 1985), probability summation (Graham, 1977), or the consequences of photon noise (Banks et al., 1987~. The low-pass filtering performed by these processes reduces the width of the band of spatial frequencies passed to about a third of that reaching the nonlinear process.
86 WALTER MAK:OUS S ~2 1 4 R Ocular Media Receptor Apert. Small Choke Spat. Antag. Nonlinearity Big Choke Spat. Antag. Neural Blur FIGURE 5 Elaboration of Figure 3 to show the spatial antagonism and temporal filters that form parts of the first and second filters. The octagons in Figure 4 also show that the signals elicited by gratings of 5 c/de" are attenuated more than those elicited by gratings of 10 c/d ea. Although this is but a single point, the difference is highly reliable and both observers tested show it. An increasing attenuation of signals elicited by gratings of decreasing spatial frequency requires an antagonistic interaction between signals originating in different receptors. This is a refinement not represented by the four operations of Figure 3. Consequently, in Figure 5 the representation of the filters preceding and following the nonlinear stage has been expanded lo more detail. The last stage of Figure 3, labeled second filter, now consists of a low-pass filter, labeled neural blur, and a junction where antagonistic signals from different receptors interact. The negative sign represents mutual antagonism without specifically denoting algebraic subtraction. A Test for Serial Filter Model According to the logic expressed by Figure 3, the signals elicited by the gratings are attenuated by two serial filters, one before the nonlinear stage and one after. Then the contrast sensitivity to interference fringes (the total attenuation of the signals by the visual system) should be the
PARTITIONING VISUAL PROCESSES 87 product of the attenuations by these two filters, that is, the sum of the triangles and octagons of Figure 4 (which have logarithmic ordinates). The filled diamonds in Figure 4 represent that sum, and the squares represent the contrast sensitivity to interference fringes (thus avoiding attenuation by the optics of the eye). As we cannot determine the absolute size of the distortion products, the set of diamonds has been adjusted vertically for the best fit. Insofar as the shapes of the curves defined by the two sets of points are similar over the range where the test can be made, the validity of partitioning these attenuations is supported. (Control experiments also show that the properties of the second spatial filter do not depend on the spatial frequencies of the fringes used to produce the distortion gratings.) Spatial Antagonism Preceding the Nonlinearity Our technique does not allow direct measurement of the attenuation preceding the nonlinear stage at low spatial frequencies. However, if the sum of the effects represented by triangles and octagons equals the total effects determining the shape of the contrast sensitivity curve represented by the squares, the difference between the squares and octagons should resect the shape of the curve represented by the triangles. This provides a means of estimating where the triangles might lie if we could measure the function at low spatial frequencies. When this is done, the results show a progressive loss of sensitivity as spatial frequency decreases, so that 5-c/de" gratings are attenuated more than twice as much as 30-c/de" gratings. As stated above, this requires spatially antagonistic interactions between signals from different cones, and so in Figure 5 the connections representing such lateral interactions are introduced before the nonlinear stage. Spatiotemporal Filters Both antagonistic interactions (but only the antagonistic interactions) are reduced by decreasing the duration of the test grating from 500 to 50 msec (Chen et al., 1988~; so the spatially antagonistic signals pass through low-pass temporal filters. These also have been introduced in Figure 5 in the pathway of signals entering into the spatial antagonism, where they have been given the admittedly passe label (to save space) choke. Decreasing the duration of the test grating affects the low-frequency drop following the nonlinear stage more than that preceding the nonlin- earity. Thus, the size or effectiveness of the two chokes differs, and this is represented by calling one big and the other small.
88 Cumulative Spatial Filtering WALTER MAKOUS The results above show that spatial filtering by the visual system can be separated into the serial stages of Figure 3, and the unfilled squares in Figure 4 show the accumulated attenuations by two of these stages. However, the attenuations attributable to the optics are not shown, and the transfer functions of Figure 4 do not serve intuition as well as other representations, such as spread functions. Consequently, the cumulative spreads of signal from a point source following each of the three stages are shown in Figure 6. The solid line is the optical point-spread function computed from Westheimer's formula (1986), which is based on Campbell and Gubisch's data (1966~. The dashed line is the convolution of the optical spread function with the spread function of the first filter, computed from triangles of Figure 4. Finally, the dotted line is the convolution of the dashed line with the spread function of the second filter, computed from the octagons of Figure 4. As the last filter follows a nonlinearity, this is not a true spread function. Nevertheless, it gives an intuitive grasp of the relative losses of fine detail as the information in an image passes through the visual system, confirming the general belief that the optics of the eye account for most but certainly not all of it. The figure also puts the psychophysical contribution of center-surround antagonism into perspective. Summary of Results The preceding section shows how the interferometer and nonlinear process can be used. Figure 7 summarizes the results this approach has produced so far. This diagram is not intended as a model but as a way to summarize the results, organize them, and show how they relate to one another. With the exceptions specified below, all the features of the diagram are dictated by the data. As a consequence, the figure shows only those operations required by these data, and some operations in the diagram represent a multitude of processes that the experiments do not separate. Each stage, then, represents an operation the visual system must, according to our observations, perform on the information in its stimuli, and the arrows depict the flow of information through these operations. The labels are chosen simply to carry the most descriptive information in the space available. The solid lines and bold labels denote the stages where we have quantified the relationship between what goes into the designated operation and what comes out.
-9 ~ ~ to lo 89 ; : /' ~ // / // /// By/ ~ / it \\ \\ `\ ~ \\ \\ \ \ ~ ~ -3 -2 -1 0 1 2 Minutes FIGURE 6 Poinl~pread ranchos aher lbc in~aUon has pawed 1b~ugb 1be eyed opUo Amid Hne), lb~ugb the blledng preceding 1be nonlinear stage (dashed Une), and 1broupb the enlists visual Islam (dolled 11nc).
9o WALTER MAKOUS S 2 3 it' 5 8 l ..... 1 ~3 _. . == . . ~ i.! Ilk 7 i) r- 1 1 1 1 el05~> L . .- i J I ~ 1 1'''''' ........ , .............. 1 9 10 11 .> 14 ,s 16 18 19 20 .. . 21 22 23 1 ~1 '.~ 1 Am, ! R ,............. Ocular Media Speckle Noise Cone Aperture Pigment Screen Escaped Light Pigment Screen Recaptured Light Stiles-Crfd. Eff. Pigment Screen Spect. Absorpt. Poisson Noise Choke Small Choke Spat. Antag. Adaptation Gain Control Choke Nonlinearity Neural blur Big Choke Spat. Antag. Sn~all choke Late Vision FIGURE 7 Schematic diagram of the processes identified by use of the interferometer and nonlinearity. Operations where such experiments have quantified the relationship between input and output are emboldened.
PARTITIONING VISUAL PROCESSES Optics (Stages 1 to 3) 91 The first operation performed on the entering light is spatial filtering by the optics of the eye (stage 1~. This we have experimentally bypassed so far, and so we have not measured it quantitatively, although Campbell and Green (1965) have. As the coherent light of lasers passes through the ocular media, small inhomogeneities cause local interference patterns referred to as laser speckle. This is a source of spatial noise that masks the gratings (Williams, 1985b). Although this is not normally part of vision, it affects most of the observations made in these experiments and thus is included in the sequence of operations (stage 2~. The magnitude of masking caused by laser speckle is evaluated by mixing incoherent light, which is not subject to such speckle, with the coherent light from the laser. The elevation of threshold it produces can approach a factor of 10, but it varies with spatial frequency and observer (Williams, 1985b; Makous et al., 1985; Chen et al., 1988) but not the level of adaptation (Makous et al., 1985~. Light that enters the cone is spatially filtered by the receptor aperture (stage 3), as discussed above. Phenomena Related to Light Absorption (Stages 4 to 11) The Stiles-Crawford effect (1933) shows that more light is needed to have a given effect if it falls on the cones at an oblique angle than if it is incident axially. This is generally attributed to the escape of obliquely incident light from the cones. Some of the light that escapes the cones is ultimately absorbed by inert pigments in the pigment epithelium and is entirely lost to the visual system. This is part of the escaped light that is reflected by the Stiles-Crawford effect. The amount of light that must be added to obliquely incident light so as to have the same effect as axially incident light must equal the amount lost. However, if some light that escapes one cone re-enters and is absorbed by another cone (i.e., Is recaptured by the other cone), it is not lost but has the same effect on most measures of the Stiles-Crawford effect as if it had been captured by the first cone it struck. Such light that escapes from one cone only to be recaptured by its neighbor must decrease the contrast of interference fringes cast on the retina by oblique light. The loss of contrast sensitivity to such fringes, then, provides a way to measure the recaptured light. We have so measured the lost light and the recaptured light in two observers (Chen and Makous, 1988) and introduced symbols in Figure 7 to represent them. The total amount of light that escapes the cones (stage 5) is the sum of the light lost (stage 8), measured by the Stiles-Crawford effect,
92 WALTER MAKOUS and the recaptured light (stage 7), measured by comparison of contrast sensitivity to interference fringes at oblique incidence with that at normal incidence. Some 70 to 90 percent of the light absorbed by the cones when it passes through the center of the pupil (axial incidence) is lost (stage 8) to the visual system if it passes through the margin of the dilated pupil; over half of what is ultimately absorbed (at stage 10) is recaptured light (stage 7~. However, escaped light spreads little more one than cone before recapture. This recaptured light, then, causes optical cross talk among receptors. Self-screening. Sensitivity to interference fringes is reduced by oblique incidence even when the stripes run parallel to the plane of incidence. This is taken as evidence that the incident light is temporarily captured by the first cone it encounters, before escaping. As light passes through the pigment contained by the cones, its spectrum changes, owing to preferential absorption of certain wavelengths by the photopigment (pigment screening). So there may be some self-screening of the light before it escapes. To recognize this possibility, stage 4 is introduced in Figure 7. Recaptured light may also undergo some self-screening after recapture and before absorption; stage 9 is introduced to recognize this possibility. Finally, the light captured by the first cone it strikes and ultimately absorbed by the same cone may pass through some pigment that neither escaped light nor recaptured light sees; stage 6 is introduced to cover that possibility. None of these individual stages of self-screening is definitely estab- lished, but the phenomenon of self-screening in cones is generally accepted, and present evidence is consistent with its occurrence at any or all of these stages. Action speculum. As self-screening is the complement of the action spectrum, absorption of course occurs wherever pigment screening occurs. However, absorption and self-screening are shown as separate stages in Figure 7 to indicate that, although we have no quantitative information on these separate components of self-screening (stages 4, 6, and 9), we have identified the absorption spectrum (stage 10~. The Michelson contrast of interference fringes (Born and Wolf, 1970) is: C = 2(I:I2~/2/(I~ + Id. In the case where the light is filtered through a spectrally tuned filter (i.e., a visual photopigment), I is the retinal irradiance times the spectral absorption of the pigment at the wavelength used. Note that if the sum of I: and I2 is held constant the equation defines a parabola with a peak at the point where the adsorptions of the two wavelengths are equal. We have complete data on one observer and preliminary data on another showing that the threshold for distortions produced by superimposing 40-c/de" fringes of 632.8 and 514.5 nm (forming
PARTITIONING VISUAL PROCESSES 93 heterochromatic beats) follows the equation above (Chen et al., 1986 unpublished observations) when I is weighted according to a single pigment with the spectral absorption a Smith and Pokorny fundamental (Porkorny and Smith, 1986~. Which fundamental it is depends on the adaptive state of the eye. Poisson noise. After the effects of the speckle noise (stage 2) are taken into account, sensitivity varies inversely with the square root of the mean retinal illuminance (Makous et al., 1985~. This is attributed to a source of noise that increases proportionally; that source is introduced in Figure 7. Since some unknown part of this must be shot noise associated with the quantal adsorptions, and because the sources of any other such noise can be localized only between the site of absorption and that of the nonlinear process, representation of all such sources of noise is positioned (stage 11) close to the spectral filter (stage 10), which also is associated with absorption of light. These two sources of noise, laser speckle and Poisson noise, account for 98 percent of the variance of thresholds measured at varying levels of adaptation. Neural Processes Preceding the Nonlinear Process Low-pass temporal filter Stage 124. Since there is a limit to the rate of flicker that can be detected, the visual system has a temporal filter. However, when two homogeneous superimposed fields flicker at slightly different rates, no Dicker at the difference frequency can be detected unless the flicker of the fields also is detectable (Makous and Mandler, unpublished observations, 1984~. This means that temporal filtering after the nonlinear stage does not attenuate the flicker of the superimposed fields below that of the distortion product. This is true even when the superimposed fields are flickered at rates that are much harder to detect than that of the difference frequency. This would not occur if all or even most of the attenuation of temporal fluctuations followed the nonlinear process. Hence, at least some of the attenuation of temporal frequencies must precede the nonlinear process, and so stage 12 (called a choke to save space) is introduced to represent it in Figure 7. (No significance should be attached to its position here relative to stages 13 to 17, for our present observations do not allow us to localize it in the sequence of these processes.) We cannot say whether additional temporal filtering follows the nonlinear stage, but at present we have no evidence of it. To provide for that possibility, however, stage 22 is introduced. Surround antagonism (stages 13 and 149. This is discussed above.
94 wALTER MAKOUS Aftereffects of light exposure (adaptation) (stage 15). If one of the two interference fringes used to create these distortion products is presented as an intense 10-msec flash, subsequent presentation of the other fringe alone produces distortions similar to those observed when the two are presented simultaneously (MacLeod et al., 1985~. Since the contrast sensitivity and hence the spatial spread of the aftereffect of the flashed grating are the same as those measured by steadily presented gratings, the residual physiological change of the adaptation produced by and outlasting the flashed grating is localized in the pathway between the receptor aperture and the site of the nonlinear stage (i.e., before the site of summation of signals from multiple cones). We have no conclusive evidence on whether it is subject to spatial antagonism (lies before or after stage 14~; nor do we know whether the aftereffect sums with the effects of the steadily presented grating or attenuates it, say, by changing the gain of stage 16 or some other gain control. However, Hayhoe's experiments on sensitivity to test flashes against backgrounds of varying size (Hayhoe, 1979; Buss et al., 1982) lead us to place it tentatively at stage 15, after spatial antagonism (stage 14) and before the gain control mechanism (stage 16~. Adaptive gain control with temporal filter. Contrast sensitivity for the distortion products is constant as mean intensity varies from 240 to 10,000 Td, and if the ehects of noise (stages 2 and 11) are taken into account, the sizes of the distortion products are inferred to be constant down to 1.7 Td (Makous et al., 1985~. This invariance of contrast sensitivity we attribute to a gain control mechanism that keeps the time-average input to the nonlinear stage constant, and stage 16 is introduced to represent it in Figure 7. The gain control adjustment cannot be instantaneous, or no signal would be passed on the rest of the system. Hence, the low-pass temporal filter is introduced at stage 17. The gain control mechanism is tentatively positioned following spatial antagonism because of Hayhoe's demonstration (Hayhoe, 1979; Buss et al., 1982) of spatial antagonism preceding a gain of control mechanism that might be the same as that responsible for the effects observed here. The Nonlinearity As discussed above, the distortions would not be seen if there were no nonlinear process (Burton, 1973), which is designated as stage 18 (Figure 7~. The form of the nonlinearity is described by a third-order polynomial, the coefficients of which are determined from three measurements (Makous et al., 1985~: sensitivity to single gratings, which almost exclusively determines the linear parameter (a, in the equation below); sensitivity to distortions produced by two superimposed gratings of the same frequency, which
PARTITIONING VISUAL PROCESSES o o Cat ° to - 95 - - - 04 0.6 08 10 1 1 2 1 4 1 6 FIGURE 8 The fond of the nonlineanty. I is scaled so that the mean intensity (1000 Td) is unity. R is scaled in arbitrary units, with zero response set at the mean intensity. 0: WM. determines the value of the quadratic parameter (b below); and sensitivity to distortions produced by two superimposed gratings, one with a frequency twice that of the other, which determines the cubic component (c below). As no higher-order distortion products can be detected, no higher-order terms are required in the polynomial. Computations confirm that this polynomial produces distortions of equal size under the three threshold conditions listed above. This function is shown in Figure 8, the equation being: R = aI ~ bI2 + cI3, where a _ 65.0, b = -57.0, and c = 82.2. The third-order polynomial is the only function we have found that does not produce the distortions we observe. Its shape is independent of spatial frequency where we (Chen and Makous, unpublished observations, 1987) have been able to test it (above 30 c deg-~), and it is independent of mean intensity above lOOO Td. Control tests show that the amplitudes of the distortions are proportional to the product of the contrasts of the superimposed beams, as required by theory (MacLeod et al., 1985~.
96 Processes Following the Nonlinear Stage WALTER MAKOUS The stages (19 to 23) following the nonlinear stage are in arbitrary sequence, and as each of these stages almost certainly represents many different processes, their sequences may be interlaced, parallel, and/or recursive. The first three have been discussed above. The last is taken up briefly in the next paragraph. Generalizing the Approach The results summarized by Figure 7 suggest that use of the Rochester interferometer to exploit a nonlinear process has been rewarding, and working out the remaining unknowns in this system promises continuing rewards, but these successes also stimulate an interest in testing the power of analogous techniques elsewhere in the visual system. This would require a method for bypassing the present nonlinear process (stage 18) and iden- tification of a different nonlinear process susceptible to such exploitation. Neither obstacle seems insurmountable. Relations to Anatomy and Physiology Inferences about the organization underlying behavior gain explana- tory and predictive power if the inferences can be related to anatomical structures and physiological processes. Although Figure 7 is essentially a way to organize a set of psychophysical observations, it also aids the effort to relate these observations to anatomy and physiology. The physical referents for stages 1 and 2, the locus of temporal filtering (stages 12 and 22), and the stages following the nonlinearity are discussed above. Stages 3 through 11 patently describe the properties of a large population of cones. Although there are three classes of cones within the retinal region tested, the experiments represented by stage 10 show that in these experiments all processes preceding the nonlinear stage represent the properties of the L cones. Horizontal cells mediate spatial antagonism (Baylor et al., 1971; Naka, 1971), such as is represented by stages 13 and 14, and the spatial extent of the antagonism observed here is close to the size of HI horizontal cells (Boycott et al., 1987~. However, an additional contribution from more proximal cells, such as amacrine cells, cannot be excluded at present. Little can be said about the locus of bleaching adaptation (stage 15) except that it precedes the nonlinear stage (stage 18~. A gain change of 5000 to 1 (stages 16 and 17) also precedes the nonlinear stage. This exceeds by more than 100-fold that observed in the membrane potentials (Valeton and Van Norren, 1983) and membrane currents (Schnapf, 1988) of primate
PARTITIONING VISUAL PROCESSES 97 cones. Whatever accounts for this gain change, it occurs too far proximal to affect the electrical properties of cone membranes. However, as the release neurotransmitter from a cone can be influenced without affecting the electrical properties of the cone (Sarantis et al., 1988), one cannot exclude the site on cones where neurotransmitter is released from the locus of the gain change. Locus of the Nonlinearity The nonlinear process is the point of reference in the sequence of op- erations, and identification of its anatomical locus is necessary to determine the locus of many of the other processes. Once this locus is determined, correspondences between the electrophysiologically observed properties of neurons and psychophysically observed operations can be established, and the information present or lost in the two domains can be compared. If the two correspond closely enough, the functional organization of the human visual system can be determined by comparisons across species, and param- eters specific to the human species can be determined by psychophysics. The locus of the nonlinear process is not yet narrowly constrained. As the amplitude of every physiological process is limited, all must enter a nonlinear stage at least when driven hard. This makes it hard to conceive that the signal passes very far without encountering a nonlinear stage. As discussed above, the particular nonlinearity producing the distor- tions we observe under these conditions (stage 18) is preceded by spatial antagonism (stage 14), adaptation, temporal filtering, and gain adjustment, but this excludes only sites distal to cone neurotransmitter release. The small aperture preceding the nonlinear stage, corresponding to that of a single cone (stage 3), means that no pooling of signals from other cones precedes the nonlinearity. However, this is true of the midget or pa~vocellular system of the fovea at least as far as the lateral geniculate nucleus and probably at the cortex as well. So if these observations can be mediated by the parvocellular system, this hardly constrains the locus of the nonlinear process at all. Although the small aperture (stage 3) fails to constrain the locus of the nonlinear process, it does limit applicability of observations on processes preceding the nonlinear stage to individual cones and the parvocellular system. Even the parvocellular system, however, is heterogeneous, includ- ing, for example, both on and off pathways. If both on and off systems contribute to observations of processes preceding the nonlinearity, their respective contributions are not known.
98 Form of the Nonlinear Process WALTER MAKOUS As the distortions observed psychophysically presumably depend On a physiological process with nonlinear properties, the simplest assumptions suggest that the very distortions observed should be produced by trans- forming each point in the stimulus pattern according to the mathematical function that describes the physiological process. However, none of the functions that have been used to describe physiological processes so serve. Evidently, the assumptions underlying the expectation that they would serve are too simple. Complications arise (1) from the multiplicity of dissimilar but interrelated nonlinear physiological processes; (2) from the differences between the response of individual members of a population and the sta- tistical properties of the population as a whole; and (3) from subsequent processing, such as those that transform continuous physiological processes into discrete behavioral responses. Multiplicity of processes. Each cell has several observable properties that contain visual information. Nearly all such processes enter a nonlinear phase as they approach saturating limits, and some have nonlinearities near threshold, but no two need necessarily be simply related to one another. The input to a bipolar cell, for example, is a change of concentration of a neurochemical, but there may be several such neurochemicals in the outer plexiform layer that affect the cell. The concentration of any one may be a nonlinear function of stimulus intensity, though not necessarily the same function. Each neurochemical affects the properties of ionic channels in the postsynaptic membrane, and perhaps the properties of the presynaptic membrane as well, but different transmitters have different effects on different channels, and a given channel has different effects on the con- ductance of different ions. In aggregate these ionic conductances affect membrane current and membrane potential at the presynaptic membrane of the bipolar cell in the inner plexiform layer. This presynaptic potential is determined by a weighted average of currents through different types of channels, in different regions of the cell, over different courses of time; by electrical coupling among cells; and by all the influences described for the outer plexiform layer. So the concentrations of neurochemicals and the conductance, current, and potential of the synaptic membrane each play their respective roles in carrying the visual information. Although these are all interrelated, their interrelations are seldom linear, and so each typically is related to stimulus intensity by a different function. It seems probable that each of these participate in carrying the visual information through the system and that none is related to it in a simple way.
PARTITIONING VISUAL PROCESSES 99 Population statics. An individual distortion grating is created by the concerted action of many individual nonlinear elements. Unless each fol- lows the identical nonlinear function, the amplitude and detectability of the grating may be different from the contribution of any individual element. For example, if each element had an all-or-none response but different thresholds, the average response would be graded. So the individual phys- iological functions that create the distortions can be quite different from their collective effect. Subsequent processing If there were a particular physiological pro- cess related to the psychophysically observed nonlinearity, it would in any case be a continuous, time-varying signal. But psychophysical thresholds are discrete: yes or no; alternative 1 or 2. Then what aspect of the time-varying signal determines the psychophysical response? Whatever is chosen whether the favorite, peak response, or something else, such as integrated response subsequent temporal filtering has different effects on responses that do not have the same time course. Normann and Werblin (1974) reported, for example, that the exponent of the Naka-Rushton equa- tion (Naka and Rushton, 1966) describing peak responses of mudpuppy receptors is 0.7 for long flashes and 1.0 for short flashes. So responses to stimuli that are both at threshold may differ, according to any measure, before the final stage of temporal filtering (cf. Zacks, 1970), and one cannot know how to relate physiological responses to psychophysics without knowing how the signals are processed between the site observed and the site where the response is determined. It is worth adding here that this has nothing to do with psychophysical linking laws. However valuable the concept may have proved elsewhere, here it is a red herring. When one associates the squeeze of the trigger of a rifle with the emission of hot gases from the barrel, it is not necessary to invoke chemophysical linking laws; it is satisfactory to describe the process as a causal sequence. The same is true of the experiments discussed here. ON METAPHOR IN VISION An essay such as this perhaps allows one to indulge in broader com- mentary than might otherwise be permissible, and I would like to use this platform to comment on the metaphors that visual scientists use. Those I have used here are borrowed from electronics and communication theory. I think they are the best we have for this work, and they serve because the visual system and the artificial systems for which they were developed are subject to similar demands. Yet natural and artificial systems do differ, and so in some ways the metaphors are inapt.
100 WALTER MAKOUS For example, functional diagrams such as Figure 7, and others that visual scientists use to represent neural circuits, show discrete elements interrelated by discrete connections. I wonder if they do justice to the variety of cells represented and the complexity and variations in the inter- connections among them and the systems they form. To some extent the schematic diagrams we use arise from the idea that individual neurons are the appropriate units of analysis in the visual system. However, the ubiquitous gap junctions spread electrical potentials through the barriers between cells that guard their independence, so that the cooperative actions of these extensively interconnected neurons generate fields that tend to vary continuously in space, perhaps more in accordance with the rules of field theory than those of the neuron doctrine. When 90 percent of the signal issuing from the individual rod originates in other rods (Fain, 1975), the idea of individual cells as independent elements is strained. On the other hand, metaphors derived entirely from field theory go too far toward ignoring the discontinuities produced by the membranes that are there. Block diagrams such as Figure 7 are also clumsy in representing multiple local interactions and feedback They do not adroitly treat the cooperative action of large interacting populations of diverse cells. This is not to say that these things cannot or are not being done; it is to say that they are done neither adroitly nor with grace; and when they are done, the concepts are adopted from other sciences, not born of the peculiarities of the phenomena we study. I end this essay with comments on an altogether different kind of metaphor, such as Pugh's handsome and informative portrait of the trans- duction process inside photoreceptors (Pugh, this volume). It deftly encap- sulates a complex and dynamic set of interrelationships. Yet these are, and for the present must remain, static representations of a dynamic system. And, for once, I think they make the possibilities of electronic journals attractive, for in a journal viewed through a computer, the processes and cycles now imitated by arrows could be set into motion. When such illus- trations incorporate the models themselves, the possibilities enlarge. We shall be able to play with the variables and watch gears turn as the model grinds out the results. Perhaps then models will be fun for readers as well as modelers, and the distinction between the two may blur. ACKNOWLEDGMENTS Preparation of this paper was supported by Public Health Service grants EY~885 and EY-1319.
PARTITIONING VISUAL PROCESSES 101 REFERENCES Banks, M.S., W.S. Geisler, and P.J. Bennett 1987 The physical limits of grating visibility. Vision Research 27:1915-1924. Baylor, D.A., M.G.F. Fuortes, and P.M. O'Bryan 1971 Receptive fields of cones in the retina of the turtle. Joumal of Physiology (London) 214:265-294. Born, M., and E. Wolf 1970 Principles of Optics, 4th edition. New York: Pergamon Press. Boycott, B.B., J.M. Hopkins, and H.G. Sperling 1987 Cone connections of the horizontal cells of the rhesus monkey's retina. Proceedings of the Royal Society of London B 229:345-379. Burton, G.J. 1973 Evidence for non-linear response processes in the human visual system from measurements on the threshold of spatial beat frequencies. Vision Research 13:1211-1225. Buss, C.M., M.M. Hayhoe, and C.F. Stromeye III 1982 Lateral interactions in the control of visual sensitivity. Vision Research 22:693-709. Campbell, F.W., and D.G. Green 1965 Optical and retinal factors affecting visual resolution. Journal of Physiology 181:576-593. Campbell, F.W., and R.W. Gubisch 1966 Optical quality of the human eye. Joumal of Physiology 186:55~578. Chen, B., and W. Makous 1987 Retinal losses of contrast with oblique incidence. Investigative Ophthabnology and lingual Science 28(Suppl.~:357. 1988 Light capture by cones. Joumal of Physiology, submitted. Chen, B., W. Makous, and D.R. Williams 1988 Serial spatial filters in vision. InvestigativeOphthalmology and Visual Science 29(Suppl.~:58. Curcio, C.A., K.R. Sloan, Jr., O. Packer, A.E. Hendrickson, and R.E. Kalina 1987 Distribution of cones in human and monkey retina: individual and radial asymmetry. Science 236:57~582. Fain, G.L. 1975 Quantum sensitivity of rods in the toad retina. Science 187:838~4l. Graham, N. 1977 Visual detection of aperiodic spatial stimuli by probability summation among narrowband channels. Vision Research 17:637~52. Hayhoe, M. 1979 After-effects of small adapting fields. Joumal of Physiology 296:141-158. MacLeod, D.A I., D.R. Williams, and W. Makous 1985 Difference frequency gratings above the resolution limit. Investigative Oph- thalmology and Visual Science 26(Suppl.~:11. Makous, W., D.R. Williams, and D.A.I. MaeLeod 1985 Nonlinear transformation in human vision. Joumal of the Optical Society of America A 2:P80. Miller, W.H., and G. Bernard 1983 Averaging over the foveal receptor aperture curtails aliasing. Incision Research 23:1365-1369. Naka, HI. 1971 Receptive field mechanism in the vertebrate retina. Science 171:691093.
102 WALTER MAKOUS Naka, KI., and W.A.H. Rushton 1966 S-potentials from colour units in the retina of fish (Cyprinidae). Joumal of Physiology (London) 185:536 555. Normann, R.A., and F.S. Werblin 1974 Control of retinal sensitivity: I. Light and dark adaptation of vertebrate rods and cones. Joumal of General Physiology 63:37~1. Pelli, D.G. 1985 Uncertainty explains many aspects of visual contrast detection and discrimi- nation. Joumal of the Optical Society of America A 2:150~1532. Pokorny, J., and Smith, V.C. 1986 Colorimet~y and color discrimination. Pp. 8-1 to 8-51 in Handbook of Perception and Human Performance, K.R. Boff, Lo Kaufman, and J.P. Thomas, eds. New York: Wiley. Sarantis, M., K. Everett, and D. Attwell 1988 A presynaptic action of glytamate at the cone output synapse. Nature 332:451~53. Stiles, W.S., and B.H. Crawford 1933 The luminous efficiency of rays entering the pupil at different points. Proceedings of the Royal Society (London) B112:42~50. Valeton, J.M., and D. Van Norron 1983 Light adaptation of primate cones: an analysis based on extracellular data. lesion Research 23:1539-1547. Westheimer, G. 1986 The eye as an optical instrument. Pp. 4-1 to 4-20 in Handbook of Perception and Human Performance, KR. Boff, L. Kaufman, and J.P. Thomas, eds. New York: Wiley. Williams, D.R. 1985a Aliasing in human foveal vision. Vision Research 25:195-205. 1985b Visibility of interference fringes near the resolution limit. Joumal of the Optical Society of America A 2:1087-1093. Zacks, J.L 1970 Temporal summation phenomena at absolute threshold: their relation to visual mechanisms. Science 170:197-199.
