Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 78
Partitioning Visual Processes
WALTER MAKOUS
IDENTIFYING AND SEQUENCING OPERATIONS
Method and Rationale
The work reported here is based on the idea that vision consists of
the transformation of spatiotemporal patterns of light into behavior and
experience. It follows that a visual scientists goal is to understand the
operations that accomplish this transformation and the paths that signals
take through those operations. In human vision the pursuit of this goal is
complicated by the fact that the operations and their organization usually
must be inferred from the relationship between input and output (i.e.,
from psychophysics). If the visual system were entirely linear, any given
relationship between input and output could be produced by many different
combinations of operations and choosing among them would be either
guesswork or a matter of taste. So nonlinear processes are what allow us
to pare down the alternative representations of the system and, when they
have been pared down enough, to know what goes on inside.
The way nonlinearities constrain the alternatives can be seen from
a simple example. Consider a system with To inputs, a and b, that are
combined to determine a single output, c. If there is a nonlinearity in the
system, such that c or log a and c or log b, one can establish whether the
inputs sum before the log transformation ti.e., c or log (`a + b)] or after it
[i.e., c or (log a) + (log b)] by varying both a and b while holding their sum
constant (a ~ b = k). If the inputs sum before the log transformation, the
variations of a and b have no effect on the output; if they sum after the log
transformation, variations of a and b do affect the output This is true of
78
OCR for page 79
PARTITIONING VISUAL PROCESSES
79
any nonlinear transformation, not just the log transformation used in this
example.
The method my co-workers and I used is directly analogous to this
example. The two inputs, a and b, correspond to the illumination of
rows of adjacent cones. Under some conditions, differential illumination
of adjacent cones, such as that produced by a 60 c deg-i grating, cannot
be discriminated from a homogeneous field; that is, the input from adja-
cent cones is summed to produce a single-valued output, c. 1b determine
whether this summation occurs before or after any nonlinear transforma-
tions, we systematically vary the relative illumination of adjacent cones so
that in some regions of the retina adjacent cones are equally illuminated
(a = by, and in other regions they receive very different illumination (a
b); but the sum is always held constant (a + b = k).
The necessary conditions of illumination are produced by two laser
interferometers. The first was invented by Dave Williams. As its design
has been published (Williams, 1985a), I shall not describe it here except to
say that it allows precise and facile control over the contrast, orientation,
and spatial frequency of gratings created by optical interference, which can
exceed 200 c degas. As the changes of contrast are rapid and afford no
extraneous cues, the instrument supports forced choice observations.
The second interferometer, of simpler design (MacLeod et al., 1985),
allows superimposition of two separate interference gratings on the retina,
such as was done first by Burton (1973~. The result of such superposition
is illustrated in Figure 1.
The first two lines represent the profiles of the two sine wave gratings
created by optical interference. When they are superimposed, the retinal
illuminances at each point add, producing the profile of "beats" shown in
the third line, where modulation is very high in some regions and nearly
absent in others.
If the period of the gratings is about twice the diameter of a cone,
adjacent cones receive very different illumination (a ~ b) in the places
where modulation of the beat pattern is high and very nearly the same
illumination (a ~ by in regions where modulation is low, but the sum of
the illumination of adjacent stripes is always the same (a + b = k).
If the excitation at each point is a nonlinear function of illumination,
such as is produced by a rectifer, clipper, or abrupt saturation (as illustrated
by the function graphed in the inset between the third and fourth lines in
Figure 1), the excitation has the profile shown in the fourth line. The top
of the curve is clipped off, and of course the local space average tends
to be more negative in certain regions than in others. These variations of
negativity show up in the Fourier spectrum as a new component (shown
by the smooth curve) that has been introduced by the nonlinear distortion;
its frequency is equal to the difference between the frequencies of the two
OCR for page 80
80
LL
o
CL
Cl)
2
2
WALTER MUCOUS
ID
MU
o L I I I
~ ~ A ~ ~1
\
/, \, \, , \,
\ , \ , \ , \ , \ ,
J
I ~ I ~ ~ \
\ /
~ ~ \ ~ \
~ I_ ~
1 1
1) 1~
1 ~ / \
1_ . - ~ Vow - - ~ ~
1 11
0 200 400 600 BOO tOOD
SPACE
FIGURE 1 Formation of a difference frequency distortion product. The third curve is the
sum of corresponding points in the fimt two. The bottom curve is this sum, transformed
by the nonlinearity depicted by the graph between the two: vertical fluctuations of the
third curve are the input, on the horizontal axis of the graph; and output, on the vertical
axis of the graph, determines the vertical fluctuations of the last curve. Hence, variation
of the input beyond the modpoint produces no variation in the output. The smooth curse
superimposed on the output shows the frequency and phase of the distortion product that
is a Fourier component of this output curve, on which it is superimposed.
OCR for page 81
PARTITIONING VISUAL PROCESSES
81
superimposed gratings. Any nonlinear distortion produces this difference
frequency; different nonlinearities simply produce different amplitudes of
the difference grating.
Figure 2 shows how we use this distortion product. It depicts the
information in two gratings of 60 and 70 c/de" entering a nonlinear stage.
Coming out of the nonlinear stage are signals of the original frequencies,
plus those of many other frequencies that are the products of the nonlinear
distortion. In the human visual system these signals next encounter a low-
pass filter that blocks everything but that of the lowest frequency, 10 c/de".
Subjectively, the observer sees only the low-frequency distortion product
without seeing the high-frequency gratings that produced it. The essence of
the technique lies in the fact that signals of only two input frequencies exist
before the nonlinear stage and only the low-frequency distortion product
exists after the nonlinear stage.
If one increases or decreases the frequencies of the two gratings
while keeping the difference between their frequencies and everything else
constant, the distortion product must remain unchanged. If this shift of
frequencies does change the amplitude of the distortion product, it can
only be due to the changes of the amplitudes of the signals elicited by
the gratings at the site of the nonlinear stage. So the detectability of the
distortion product can be used to measure the attenuation of different
frequencies by whatever precedes the nonlinear stage.
Conversely, by changing the difference between the frequencies of
input gratings without changing their mean frequency, the frequency of
the distortion product can be varied with little effect on the signal passing
through the stages that precede the nonlinearity. As the input pattern
changes little, as there is little change in the signal reaching the nonlinear
stage, and as the distortion product exists only after the nonlinearity, any
effect on detectability of the distortion product can be caused only by
attenuation of the signal after the nonlinear stage.
In practice, instead of using gratings of different spatial frequencies, we
use gratings of equal spatial frequencies and create a distortion product by
rotating the gratings slightly in opposite directions. This produces a pattern
similar to a moire pattern that has a distortion product corresponding to a
sinusoidal grating oriented nearly perpendicular to the gratings producing
it. The spatial frequency of the distortion product is changed by slightly
changing the relative orientations of the gratings. Thus, there is no change
of spatial frequency preceding the nonlinear process, and if the stages
preceding the nonlinear process are reasonably isotropic, this procedure
affects nothing except those processes that depend on the spatial frequency
of the distortion product.
OCR for page 82
82
~ o
o
CD
0
~4
4~
at 0
on
WALTER MAKOUS
WWW:
'We
o
to
to
WVWVV~-
~A
~NVVVW
WWW:
Non 1 in e an it y Lo w Pa ss Fi 1 te r
FIGURE 2 Frequencies present at successive stages of the visual system. Two sine waves
enter, and no other signals are present preceding the nonlinear stage. So processes preceding
the nonlinear stage are tested only at these input frequencies. The nonlinearity creates
a vanet~,r of distortion products, one of which is the difference Sequence. A subsequent
low-pass filter blocks the input signals as well as all the distortion products except the
lowest frequency, the difference frequency. The entering sine waves create the signal that is
detected (the difference frequency) at the stage of the nonlinearity, and whatever happens
to them subsequently is irrelevant. Only the difference frequency can be detected at the
end of the system. The difference frequency exists only following the nonlinear stage, and
so cannot be affected By what precedes the nonlinear stage (except indirectly through the
effects on the input signals). So the processes that precede the nonlinearity are tested
at the frequencies of the input signals, and the processes that follow the nonlinearity are
tested at the frequency of the difference between the input frequencies.
Illustrative Results
Basic Partitions
Figure 3 shows schematically how this interferometric technique parti-
tions visual processes. Interference gratings are unaffected by aberrations
of the eye's optics and for many purposes must be treated as though they
originate on the retina. This allows one to separate the effects of the eye's
optics from other influences on the spatial information in visual stimili.
Campbell and Green (1965) used this to measure the losses of contrast
attributable to the eye's optics. This they did by comparing the contrast
OCR for page 83
PARTITIONING VISUAL PROCESSES
S
, 1
1
1
R
83
Optics
First Spatial Filter
(Receptor Aperture)
Nonlinearity
Second Spatial Filter
FIGURE 3 The four compartments into which the visual system is partitioned by the
interferometer and the nonlinearity.
OCR for page 84
84
WALTER MAXOUS
sensitivity of directly viewed gratings, which are degraded by the optics
of the eye, to the sensitivity to interference fringes, which are not. The
procedures used here allow further separation of the causes of losses of
contrast—that is, separation of those between the optics and the nonlinear
stage (labeled here the first filter or the receptor aperture) from those that
follow the nonlinear stage (here called the second filter).
Losses Preceding the Nonlinear Stage
When distortion products of fixed frequency are created by superim-
posed interference gratings of variable frequency, the detectability of the
distortion products decreases with increasing spatial frequency of the grat-
ings (MacLeod et al., 1985; Chen et al., 1988~. Data from one observer are
shown by the filled triangles in Figure 4 (MacLeod et al., 1985~. Therefore,
a stage of spatial filtering, separate from aberrations of the eye's optics,
precedes the nonlinear stage. However, detectability of the distortions does
not decrease greatly until the spatial frequencies of the gratings approach
very high values. This means that spatial filtering preceding the nonlinear
stage is not great.
Information in retinal images must enter the visual system through a set
of individual receptors that for many purposes can be considered identical
parallel channels. The optical apertures of these receptors constitute low-
pass filters. The attenuations of contrast inferred from the detections are
just what would be produced by filtering the gratings through an optical
aperture of 24" of arc (mean of four observers, with a standard deviation
of 6"~. This is close to 80 percent of the anatomical diameter of foveal
cones (Curcio et al., 1987), as predicted by Miller and Bernard (1983~. This
allows for no additional filtering. Since all the spatial information entering
the visual system must pass through the aperture of a receptor, and since
filtering accounts for all that is measured, we take it as the average optical
aperture for the foveal cones of these observers. Consequently, the box in
Figure 3 representing the spatial filtering following the optics of the eye
and preceding the nonlinear stage is labeled receptor aperture.
The psychologically measured aperture 3.8 deg from the fovea is 52.4"
of arc (mean of four observers, with a standard deviation of 11") (MacLeod
et al., 1985; Chen et al., 1988~. The difference in the psychophysically
measured apertures is in proportion to the difference in cone sizes (Curcio
et al., 1987~.
Losses Following the Nonlinear Stage
The octagons in Figure 4 show the attenuations of varying spatial fre-
quencies by those processes that follow the nonlinear stage (Chen et al.,
OCR for page 85
PARTITIONING VISUAL PR~ESSES
~ Ll-)
._ .
.>
-
._
in
c
a)
cn
_.
in
o
(3 LO
C)
85
In
cur O Overall CSF
· Combined Fillers
O - ~1 WM
E: ~
I
0 _^
~6,
~1
' AL—
First Filter
Second Filter': I_
lo
0 ~ 1 1 1 1 1 1
0 20 40 60 80 100 120 140
Spatial Frequency
FIGURE 4 Log attenuation of signals of varying frequency (e deg- 1 ) measured by
sensitivity to the contrast of interference fringes. Unfilled squares show the negative log
contrast of interference fringes at threshold when the fringes are presented alone. The
octagons show the negative log contrast of a 60-c deg 1 interference fringe superimposed
on another 60-e deg 1 fringe of 100 percent contrast. The angle between the two fringes
was adjusted to produce distortion products at the frequencies shown on the horizontal
axis. Only the distortion product determined threshold. The triangles represent half the
negative log contrast of an interference fringe superimposed on another fringe of 100
percent contrast, where the frequency of the distortion product was 10 e deg 1, and the
frequency of the fringes is represented by the positron of the points on the horizontal axis.
Only the distortion product determined threshold. (Division by 2 is necessary to allow for
the feet that both fringes are attenuated.) The filled diamonds are the sum of the octagons
and tnangles, displaced vertically as a set to fit the unfilled squares.
1988). This combines in a single curve all the influences on transmission
of spatial information following the nonlinear stage, whether a direct con-
sequence of the receptive fields of visual neurons or such things as spatial
uncertainty (Pelli, 1985), probability summation (Graham, 1977), or the
consequences of photon noise (Banks et al., 1987~. The low-pass filtering
performed by these processes reduces the width of the band of spatial
frequencies passed to about a third of that reaching the nonlinear process.
OCR for page 86
86
WALTER MAK:OUS
S
~2
1
4
R
Ocular Media
Receptor Apert.
Small Choke
Spat. Antag.
Nonlinearity
Big Choke
Spat. Antag.
Neural Blur
FIGURE 5 Elaboration of Figure 3 to show the spatial antagonism and temporal filters
that form parts of the first and second filters.
The octagons in Figure 4 also show that the signals elicited by gratings
of 5 c/de" are attenuated more than those elicited by gratings of 10 c/d ea.
Although this is but a single point, the difference is highly reliable and both
observers tested show it. An increasing attenuation of signals elicited by
gratings of decreasing spatial frequency requires an antagonistic interaction
between signals originating in different receptors. This is a refinement not
represented by the four operations of Figure 3. Consequently, in Figure
5 the representation of the filters preceding and following the nonlinear
stage has been expanded lo more detail. The last stage of Figure 3, labeled
second filter, now consists of a low-pass filter, labeled neural blur, and a
junction where antagonistic signals from different receptors interact. The
negative sign represents mutual antagonism without specifically denoting
algebraic subtraction.
A Test for Serial Filter Model
According to the logic expressed by Figure 3, the signals elicited by
the gratings are attenuated by two serial filters, one before the nonlinear
stage and one after. Then the contrast sensitivity to interference fringes
(the total attenuation of the signals by the visual system) should be the
OCR for page 87
PARTITIONING VISUAL PROCESSES
87
product of the attenuations by these two filters, that is, the sum of the
triangles and octagons of Figure 4 (which have logarithmic ordinates). The
filled diamonds in Figure 4 represent that sum, and the squares represent
the contrast sensitivity to interference fringes (thus avoiding attenuation by
the optics of the eye). As we cannot determine the absolute size of the
distortion products, the set of diamonds has been adjusted vertically for
the best fit. Insofar as the shapes of the curves defined by the two sets of
points are similar over the range where the test can be made, the validity
of partitioning these attenuations is supported. (Control experiments also
show that the properties of the second spatial filter do not depend on the
spatial frequencies of the fringes used to produce the distortion gratings.)
Spatial Antagonism Preceding the Nonlinearity
Our technique does not allow direct measurement of the attenuation
preceding the nonlinear stage at low spatial frequencies. However, if the
sum of the effects represented by triangles and octagons equals the total
effects determining the shape of the contrast sensitivity curve represented
by the squares, the difference between the squares and octagons should
resect the shape of the curve represented by the triangles. This provides
a means of estimating where the triangles might lie if we could measure
the function at low spatial frequencies. When this is done, the results
show a progressive loss of sensitivity as spatial frequency decreases, so
that 5-c/de" gratings are attenuated more than twice as much as 30-c/de"
gratings. As stated above, this requires spatially antagonistic interactions
between signals from different cones, and so in Figure 5 the connections
representing such lateral interactions are introduced before the nonlinear
stage.
Spatiotemporal Filters
Both antagonistic interactions (but only the antagonistic interactions)
are reduced by decreasing the duration of the test grating from 500 to 50
msec (Chen et al., 1988~; so the spatially antagonistic signals pass through
low-pass temporal filters. These also have been introduced in Figure 5 in
the pathway of signals entering into the spatial antagonism, where they
have been given the admittedly passe label (to save space) choke.
Decreasing the duration of the test grating affects the low-frequency
drop following the nonlinear stage more than that preceding the nonlin-
earity. Thus, the size or effectiveness of the two chokes differs, and this is
represented by calling one big and the other small.
OCR for page 88
88
Cumulative Spatial Filtering
WALTER MAKOUS
The results above show that spatial filtering by the visual system can
be separated into the serial stages of Figure 3, and the unfilled squares
in Figure 4 show the accumulated attenuations by two of these stages.
However, the attenuations attributable to the optics are not shown, and
the transfer functions of Figure 4 do not serve intuition as well as other
representations, such as spread functions. Consequently, the cumulative
spreads of signal from a point source following each of the three stages
are shown in Figure 6. The solid line is the optical point-spread function
computed from Westheimer's formula (1986), which is based on Campbell
and Gubisch's data (1966~. The dashed line is the convolution of the optical
spread function with the spread function of the first filter, computed from
triangles of Figure 4. Finally, the dotted line is the convolution of the
dashed line with the spread function of the second filter, computed from
the octagons of Figure 4. As the last filter follows a nonlinearity, this
is not a true spread function. Nevertheless, it gives an intuitive grasp of
the relative losses of fine detail as the information in an image passes
through the visual system, confirming the general belief that the optics
of the eye account for most but certainly not all of it. The figure also
puts the psychophysical contribution of center-surround antagonism into
perspective.
Summary of Results
The preceding section shows how the interferometer and nonlinear
process can be used. Figure 7 summarizes the results this approach has
produced so far.
This diagram is not intended as a model but as a way to summarize
the results, organize them, and show how they relate to one another. With
the exceptions specified below, all the features of the diagram are dictated
by the data. As a consequence, the figure shows only those operations
required by these data, and some operations in the diagram represent a
multitude of processes that the experiments do not separate.
Each stage, then, represents an operation the visual system must,
according to our observations, perform on the information in its stimuli,
and the arrows depict the flow of information through these operations.
The labels are chosen simply to carry the most descriptive information in
the space available. The solid lines and bold labels denote the stages where
we have quantified the relationship between what goes into the designated
operation and what comes out.
OCR for page 92
92
WALTER MAKOUS
and the recaptured light (stage 7), measured by comparison of contrast
sensitivity to interference fringes at oblique incidence with that at normal
incidence. Some 70 to 90 percent of the light absorbed by the cones when
it passes through the center of the pupil (axial incidence) is lost (stage 8)
to the visual system if it passes through the margin of the dilated pupil;
over half of what is ultimately absorbed (at stage 10) is recaptured light
(stage 7~. However, escaped light spreads little more one than cone before
recapture. This recaptured light, then, causes optical cross talk among
receptors.
Self-screening. Sensitivity to interference fringes is reduced by oblique
incidence even when the stripes run parallel to the plane of incidence. This
is taken as evidence that the incident light is temporarily captured by
the first cone it encounters, before escaping. As light passes through the
pigment contained by the cones, its spectrum changes, owing to preferential
absorption of certain wavelengths by the photopigment (pigment screening).
So there may be some self-screening of the light before it escapes. To
recognize this possibility, stage 4 is introduced in Figure 7. Recaptured
light may also undergo some self-screening after recapture and before
absorption; stage 9 is introduced to recognize this possibility. Finally, the
light captured by the first cone it strikes and ultimately absorbed by the
same cone may pass through some pigment that neither escaped light nor
recaptured light sees; stage 6 is introduced to cover that possibility.
None of these individual stages of self-screening is definitely estab-
lished, but the phenomenon of self-screening in cones is generally accepted,
and present evidence is consistent with its occurrence at any or all of these
stages.
Action speculum.
As self-screening is the complement of the action
spectrum, absorption of course occurs wherever pigment screening occurs.
However, absorption and self-screening are shown as separate stages in
Figure 7 to indicate that, although we have no quantitative information on
these separate components of self-screening (stages 4, 6, and 9), we have
identified the absorption spectrum (stage 10~.
The Michelson contrast of interference fringes (Born and Wolf, 1970)
is: C = 2(I:I2~/2/(I~ + Id. In the case where the light is filtered
through a spectrally tuned filter (i.e., a visual photopigment), I is the
retinal irradiance times the spectral absorption of the pigment at the
wavelength used. Note that if the sum of I: and I2 is held constant the
equation defines a parabola with a peak at the point where the adsorptions
of the two wavelengths are equal. We have complete data on one observer
and preliminary data on another showing that the threshold for distortions
produced by superimposing 40-c/de" fringes of 632.8 and 514.5 nm (forming
OCR for page 93
PARTITIONING VISUAL PROCESSES
93
heterochromatic beats) follows the equation above (Chen et al., 1986
unpublished observations) when I is weighted according to a single pigment
with the spectral absorption a Smith and Pokorny fundamental (Porkorny
and Smith, 1986~. Which fundamental it is depends on the adaptive state
of the eye.
Poisson noise. After the effects of the speckle noise (stage 2) are
taken into account, sensitivity varies inversely with the square root of the
mean retinal illuminance (Makous et al., 1985~. This is attributed to a
source of noise that increases proportionally; that source is introduced in
Figure 7. Since some unknown part of this must be shot noise associated
with the quantal adsorptions, and because the sources of any other such
noise can be localized only between the site of absorption and that of the
nonlinear process, representation of all such sources of noise is positioned
(stage 11) close to the spectral filter (stage 10), which also is associated with
absorption of light. These two sources of noise, laser speckle and Poisson
noise, account for 98 percent of the variance of thresholds measured at
varying levels of adaptation.
Neural Processes Preceding the Nonlinear Process
Low-pass temporal filter Stage 124. Since there is a limit to the rate
of flicker that can be detected, the visual system has a temporal filter.
However, when two homogeneous superimposed fields flicker at slightly
different rates, no Dicker at the difference frequency can be detected unless
the flicker of the fields also is detectable (Makous and Mandler, unpublished
observations, 1984~. This means that temporal filtering after the nonlinear
stage does not attenuate the flicker of the superimposed fields below that of
the distortion product. This is true even when the superimposed fields are
flickered at rates that are much harder to detect than that of the difference
frequency. This would not occur if all or even most of the attenuation of
temporal fluctuations followed the nonlinear process. Hence, at least some
of the attenuation of temporal frequencies must precede the nonlinear
process, and so stage 12 (called a choke to save space) is introduced to
represent it in Figure 7. (No significance should be attached to its position
here relative to stages 13 to 17, for our present observations do not allow
us to localize it in the sequence of these processes.) We cannot say whether
additional temporal filtering follows the nonlinear stage, but at present we
have no evidence of it. To provide for that possibility, however, stage 22 is
introduced.
Surround antagonism (stages 13 and 149. This is discussed above.
OCR for page 94
94
wALTER MAKOUS
Aftereffects of light exposure (adaptation) (stage 15). If one of the two
interference fringes used to create these distortion products is presented as
an intense 10-msec flash, subsequent presentation of the other fringe alone
produces distortions similar to those observed when the two are presented
simultaneously (MacLeod et al., 1985~. Since the contrast sensitivity and
hence the spatial spread of the aftereffect of the flashed grating are the same
as those measured by steadily presented gratings, the residual physiological
change of the adaptation produced by and outlasting the flashed grating
is localized in the pathway between the receptor aperture and the site
of the nonlinear stage (i.e., before the site of summation of signals from
multiple cones). We have no conclusive evidence on whether it is subject
to spatial antagonism (lies before or after stage 14~; nor do we know
whether the aftereffect sums with the effects of the steadily presented
grating or attenuates it, say, by changing the gain of stage 16 or some other
gain control. However, Hayhoe's experiments on sensitivity to test flashes
against backgrounds of varying size (Hayhoe, 1979; Buss et al., 1982) lead
us to place it tentatively at stage 15, after spatial antagonism (stage 14) and
before the gain control mechanism (stage 16~.
Adaptive gain control with temporal filter. Contrast sensitivity for the
distortion products is constant as mean intensity varies from 240 to 10,000
Td, and if the ehects of noise (stages 2 and 11) are taken into account,
the sizes of the distortion products are inferred to be constant down to
1.7 Td (Makous et al., 1985~. This invariance of contrast sensitivity we
attribute to a gain control mechanism that keeps the time-average input to
the nonlinear stage constant, and stage 16 is introduced to represent it in
Figure 7. The gain control adjustment cannot be instantaneous, or no signal
would be passed on the rest of the system. Hence, the low-pass temporal
filter is introduced at stage 17. The gain control mechanism is tentatively
positioned following spatial antagonism because of Hayhoe's demonstration
(Hayhoe, 1979; Buss et al., 1982) of spatial antagonism preceding a gain
of control mechanism that might be the same as that responsible for the
effects observed here.
The Nonlinearity
As discussed above, the distortions would not be seen if there were no
nonlinear process (Burton, 1973), which is designated as stage 18 (Figure 7~.
The form of the nonlinearity is described by a third-order polynomial, the
coefficients of which are determined from three measurements (Makous et
al., 1985~: sensitivity to single gratings, which almost exclusively determines
the linear parameter (a, in the equation below); sensitivity to distortions
produced by two superimposed gratings of the same frequency, which
OCR for page 95
PARTITIONING VISUAL PROCESSES
o
o
Cat
°
to
-
95
-
-
-
04 0.6 08 10
1
1 2 1 4 1 6
FIGURE 8 The fond of the nonlineanty. I is scaled so that the mean intensity (1000
Td) is unity. R is scaled in arbitrary units, with zero response set at the mean intensity. 0:
WM.
determines the value of the quadratic parameter (b below); and sensitivity
to distortions produced by two superimposed gratings, one with a frequency
twice that of the other, which determines the cubic component (c below).
As no higher-order distortion products can be detected, no higher-order
terms are required in the polynomial. Computations confirm that this
polynomial produces distortions of equal size under the three threshold
conditions listed above. This function is shown in Figure 8, the equation
being: R = aI ~ bI2 + cI3, where a _ 65.0, b = -57.0, and c = 82.2.
The third-order polynomial is the only function we have found that
does not produce the distortions we observe. Its shape is independent of
spatial frequency where we (Chen and Makous, unpublished observations,
1987) have been able to test it (above 30 c deg-~), and it is independent
of mean intensity above lOOO Td. Control tests show that the amplitudes
of the distortions are proportional to the product of the contrasts of the
superimposed beams, as required by theory (MacLeod et al., 1985~.
OCR for page 96
96
Processes Following the Nonlinear Stage
WALTER MAKOUS
The stages (19 to 23) following the nonlinear stage are in arbitrary
sequence, and as each of these stages almost certainly represents many
different processes, their sequences may be interlaced, parallel, and/or
recursive. The first three have been discussed above. The last is taken up
briefly in the next paragraph.
Generalizing the Approach
The results summarized by Figure 7 suggest that use of the Rochester
interferometer to exploit a nonlinear process has been rewarding, and
working out the remaining unknowns in this system promises continuing
rewards, but these successes also stimulate an interest in testing the power
of analogous techniques elsewhere in the visual system. This would require
a method for bypassing the present nonlinear process (stage 18) and iden-
tification of a different nonlinear process susceptible to such exploitation.
Neither obstacle seems insurmountable.
Relations to Anatomy and Physiology
Inferences about the organization underlying behavior gain explana-
tory and predictive power if the inferences can be related to anatomical
structures and physiological processes. Although Figure 7 is essentially a
way to organize a set of psychophysical observations, it also aids the effort
to relate these observations to anatomy and physiology.
The physical referents for stages 1 and 2, the locus of temporal filtering
(stages 12 and 22), and the stages following the nonlinearity are discussed
above. Stages 3 through 11 patently describe the properties of a large
population of cones. Although there are three classes of cones within the
retinal region tested, the experiments represented by stage 10 show that
in these experiments all processes preceding the nonlinear stage represent
the properties of the L cones.
Horizontal cells mediate spatial antagonism (Baylor et al., 1971; Naka,
1971), such as is represented by stages 13 and 14, and the spatial extent
of the antagonism observed here is close to the size of HI horizontal cells
(Boycott et al., 1987~. However, an additional contribution from more
proximal cells, such as amacrine cells, cannot be excluded at present.
Little can be said about the locus of bleaching adaptation (stage 15)
except that it precedes the nonlinear stage (stage 18~. A gain change of
5000 to 1 (stages 16 and 17) also precedes the nonlinear stage. This exceeds
by more than 100-fold that observed in the membrane potentials (Valeton
and Van Norren, 1983) and membrane currents (Schnapf, 1988) of primate
OCR for page 97
PARTITIONING VISUAL PROCESSES
97
cones. Whatever accounts for this gain change, it occurs too far proximal
to affect the electrical properties of cone membranes. However, as the
release neurotransmitter from a cone can be influenced without affecting
the electrical properties of the cone (Sarantis et al., 1988), one cannot
exclude the site on cones where neurotransmitter is released from the locus
of the gain change.
Locus of the Nonlinearity
The nonlinear process is the point of reference in the sequence of op-
erations, and identification of its anatomical locus is necessary to determine
the locus of many of the other processes. Once this locus is determined,
correspondences between the electrophysiologically observed properties of
neurons and psychophysically observed operations can be established, and
the information present or lost in the two domains can be compared. If the
two correspond closely enough, the functional organization of the human
visual system can be determined by comparisons across species, and param-
eters specific to the human species can be determined by psychophysics.
The locus of the nonlinear process is not yet narrowly constrained.
As the amplitude of every physiological process is limited, all must
enter a nonlinear stage at least when driven hard. This makes it hard to
conceive that the signal passes very far without encountering a nonlinear
stage. As discussed above, the particular nonlinearity producing the distor-
tions we observe under these conditions (stage 18) is preceded by spatial
antagonism (stage 14), adaptation, temporal filtering, and gain adjustment,
but this excludes only sites distal to cone neurotransmitter release.
The small aperture preceding the nonlinear stage, corresponding to
that of a single cone (stage 3), means that no pooling of signals from other
cones precedes the nonlinearity. However, this is true of the midget or
pa~vocellular system of the fovea at least as far as the lateral geniculate
nucleus and probably at the cortex as well. So if these observations can be
mediated by the parvocellular system, this hardly constrains the locus of
the nonlinear process at all.
Although the small aperture (stage 3) fails to constrain the locus of the
nonlinear process, it does limit applicability of observations on processes
preceding the nonlinear stage to individual cones and the parvocellular
system. Even the parvocellular system, however, is heterogeneous, includ-
ing, for example, both on and off pathways. If both on and off systems
contribute to observations of processes preceding the nonlinearity, their
respective contributions are not known.
OCR for page 98
98
Form of the Nonlinear Process
WALTER MAKOUS
As the distortions observed psychophysically presumably depend On a
physiological process with nonlinear properties, the simplest assumptions
suggest that the very distortions observed should be produced by trans-
forming each point in the stimulus pattern according to the mathematical
function that describes the physiological process. However, none of the
functions that have been used to describe physiological processes so serve.
Evidently, the assumptions underlying the expectation that they would serve
are too simple. Complications arise (1) from the multiplicity of dissimilar
but interrelated nonlinear physiological processes; (2) from the differences
between the response of individual members of a population and the sta-
tistical properties of the population as a whole; and (3) from subsequent
processing, such as those that transform continuous physiological processes
into discrete behavioral responses.
Multiplicity of processes. Each cell has several observable properties
that contain visual information. Nearly all such processes enter a nonlinear
phase as they approach saturating limits, and some have nonlinearities near
threshold, but no two need necessarily be simply related to one another.
The input to a bipolar cell, for example, is a change of concentration
of a neurochemical, but there may be several such neurochemicals in the
outer plexiform layer that affect the cell. The concentration of any one
may be a nonlinear function of stimulus intensity, though not necessarily
the same function.
Each neurochemical affects the properties of ionic channels in the
postsynaptic membrane, and perhaps the properties of the presynaptic
membrane as well, but different transmitters have different effects on
different channels, and a given channel has different effects on the con-
ductance of different ions. In aggregate these ionic conductances affect
membrane current and membrane potential at the presynaptic membrane
of the bipolar cell in the inner plexiform layer. This presynaptic potential
is determined by a weighted average of currents through different types of
channels, in different regions of the cell, over different courses of time; by
electrical coupling among cells; and by all the influences described for the
outer plexiform layer. So the concentrations of neurochemicals and the
conductance, current, and potential of the synaptic membrane each play
their respective roles in carrying the visual information. Although these are
all interrelated, their interrelations are seldom linear, and so each typically
is related to stimulus intensity by a different function. It seems probable
that each of these participate in carrying the visual information through the
system and that none is related to it in a simple way.
OCR for page 99
PARTITIONING VISUAL PROCESSES
99
Population statics. An individual distortion grating is created by the
concerted action of many individual nonlinear elements. Unless each fol-
lows the identical nonlinear function, the amplitude and detectability of the
grating may be different from the contribution of any individual element.
For example, if each element had an all-or-none response but different
thresholds, the average response would be graded. So the individual phys-
iological functions that create the distortions can be quite different from
their collective effect.
Subsequent processing If there were a particular physiological pro-
cess related to the psychophysically observed nonlinearity, it would in any
case be a continuous, time-varying signal. But psychophysical thresholds
are discrete: yes or no; alternative 1 or 2. Then what aspect of the
time-varying signal determines the psychophysical response? Whatever is
chosen whether the favorite, peak response, or something else, such as
integrated response subsequent temporal filtering has different effects on
responses that do not have the same time course. Normann and Werblin
(1974) reported, for example, that the exponent of the Naka-Rushton equa-
tion (Naka and Rushton, 1966) describing peak responses of mudpuppy
receptors is 0.7 for long flashes and 1.0 for short flashes.
So responses to stimuli that are both at threshold may differ, according
to any measure, before the final stage of temporal filtering (cf. Zacks, 1970),
and one cannot know how to relate physiological responses to psychophysics
without knowing how the signals are processed between the site observed
and the site where the response is determined.
It is worth adding here that this has nothing to do with psychophysical
linking laws. However valuable the concept may have proved elsewhere,
here it is a red herring. When one associates the squeeze of the trigger of
a rifle with the emission of hot gases from the barrel, it is not necessary to
invoke chemophysical linking laws; it is satisfactory to describe the process
as a causal sequence. The same is true of the experiments discussed here.
ON METAPHOR IN VISION
An essay such as this perhaps allows one to indulge in broader com-
mentary than might otherwise be permissible, and I would like to use this
platform to comment on the metaphors that visual scientists use. Those I
have used here are borrowed from electronics and communication theory.
I think they are the best we have for this work, and they serve because the
visual system and the artificial systems for which they were developed are
subject to similar demands. Yet natural and artificial systems do differ, and
so in some ways the metaphors are inapt.
OCR for page 100
100
WALTER MAKOUS
For example, functional diagrams such as Figure 7, and others that
visual scientists use to represent neural circuits, show discrete elements
interrelated by discrete connections. I wonder if they do justice to the
variety of cells represented and the complexity and variations in the inter-
connections among them and the systems they form.
To some extent the schematic diagrams we use arise from the idea
that individual neurons are the appropriate units of analysis in the visual
system. However, the ubiquitous gap junctions spread electrical potentials
through the barriers between cells that guard their independence, so that the
cooperative actions of these extensively interconnected neurons generate
fields that tend to vary continuously in space, perhaps more in accordance
with the rules of field theory than those of the neuron doctrine. When
90 percent of the signal issuing from the individual rod originates in other
rods (Fain, 1975), the idea of individual cells as independent elements is
strained. On the other hand, metaphors derived entirely from field theory
go too far toward ignoring the discontinuities produced by the membranes
that are there.
Block diagrams such as Figure 7 are also clumsy in representing
multiple local interactions and feedback They do not adroitly treat the
cooperative action of large interacting populations of diverse cells.
This is not to say that these things cannot or are not being done; it is
to say that they are done neither adroitly nor with grace; and when they
are done, the concepts are adopted from other sciences, not born of the
peculiarities of the phenomena we study.
I end this essay with comments on an altogether different kind of
metaphor, such as Pugh's handsome and informative portrait of the trans-
duction process inside photoreceptors (Pugh, this volume). It deftly encap-
sulates a complex and dynamic set of interrelationships. Yet these are, and
for the present must remain, static representations of a dynamic system.
And, for once, I think they make the possibilities of electronic journals
attractive, for in a journal viewed through a computer, the processes and
cycles now imitated by arrows could be set into motion. When such illus-
trations incorporate the models themselves, the possibilities enlarge. We
shall be able to play with the variables and watch gears turn as the model
grinds out the results. Perhaps then models will be fun for readers as well
as modelers, and the distinction between the two may blur.
ACKNOWLEDGMENTS
Preparation of this paper was supported by Public Health Service grants
EY~885 and EY-1319.
OCR for page 101
PARTITIONING VISUAL PROCESSES
101
REFERENCES
Banks, M.S., W.S. Geisler, and P.J. Bennett
1987 The physical limits of grating visibility. Vision Research 27:1915-1924.
Baylor, D.A., M.G.F. Fuortes, and P.M. O'Bryan
1971 Receptive fields of cones in the retina of the turtle. Joumal of Physiology
(London) 214:265-294.
Born, M., and E. Wolf
1970 Principles of Optics, 4th edition. New York: Pergamon Press.
Boycott, B.B., J.M. Hopkins, and H.G. Sperling
1987 Cone connections of the horizontal cells of the rhesus monkey's retina.
Proceedings of the Royal Society of London B 229:345-379.
Burton, G.J.
1973 Evidence for non-linear response processes in the human visual system from
measurements on the threshold of spatial beat frequencies. Vision Research
13:1211-1225.
Buss, C.M., M.M. Hayhoe, and C.F. Stromeye III
1982 Lateral interactions in the control of visual sensitivity. Vision Research
22:693-709.
Campbell, F.W., and D.G. Green
1965 Optical and retinal factors affecting visual resolution. Journal of Physiology
181:576-593.
Campbell, F.W., and R.W. Gubisch
1966 Optical quality of the human eye. Joumal of Physiology 186:55~578.
Chen, B., and W. Makous
1987 Retinal losses of contrast with oblique incidence. Investigative Ophthabnology
and lingual Science 28(Suppl.~:357.
1988 Light capture by cones. Joumal of Physiology, submitted.
Chen, B., W. Makous, and D.R. Williams
1988 Serial spatial filters in vision. InvestigativeOphthalmology and Visual Science
29(Suppl.~:58.
Curcio, C.A., K.R. Sloan, Jr., O. Packer, A.E. Hendrickson, and R.E. Kalina
1987 Distribution of cones in human and monkey retina: individual and radial
asymmetry. Science 236:57~582.
Fain, G.L.
1975 Quantum sensitivity of rods in the toad retina. Science 187:838~4l.
Graham, N.
1977 Visual detection of aperiodic spatial stimuli by probability summation among
narrowband channels. Vision Research 17:637~52.
Hayhoe, M.
1979 After-effects of small adapting fields. Joumal of Physiology 296:141-158.
MacLeod, D.A I., D.R. Williams, and W. Makous
1985 Difference frequency gratings above the resolution limit. Investigative Oph-
thalmology and Visual Science 26(Suppl.~:11.
Makous, W., D.R. Williams, and D.A.I. MaeLeod
1985 Nonlinear transformation in human vision. Joumal of the Optical Society of
America A 2:P80.
Miller, W.H., and G. Bernard
1983 Averaging over the foveal receptor aperture curtails aliasing. Incision Research
23:1365-1369.
Naka, HI.
1971 Receptive field mechanism in the vertebrate retina. Science 171:691093.
OCR for page 102
102
WALTER MAKOUS
Naka, KI., and W.A.H. Rushton
1966 S-potentials from colour units in the retina of fish (Cyprinidae). Joumal of
Physiology (London) 185:536 555.
Normann, R.A., and F.S. Werblin
1974 Control of retinal sensitivity: I. Light and dark adaptation of vertebrate
rods and cones. Joumal of General Physiology 63:37~1.
Pelli, D.G.
1985 Uncertainty explains many aspects of visual contrast detection and discrimi-
nation. Joumal of the Optical Society of America A 2:150~1532.
Pokorny, J., and Smith, V.C.
1986 Colorimet~y and color discrimination. Pp. 8-1 to 8-51 in Handbook of
Perception and Human Performance, K.R. Boff, Lo Kaufman, and J.P. Thomas,
eds. New York: Wiley.
Sarantis, M., K. Everett, and D. Attwell
1988 A presynaptic action of glytamate at the cone output synapse. Nature
332:451~53.
Stiles, W.S., and B.H. Crawford
1933 The luminous efficiency of rays entering the pupil at different points.
Proceedings of the Royal Society (London) B112:42~50.
Valeton, J.M., and D. Van Norron
1983 Light adaptation of primate cones: an analysis based on extracellular data.
lesion Research 23:1539-1547.
Westheimer, G.
1986 The eye as an optical instrument. Pp. 4-1 to 4-20 in Handbook of Perception
and Human Performance, KR. Boff, L. Kaufman, and J.P. Thomas, eds.
New York: Wiley.
Williams, D.R.
1985a Aliasing in human foveal vision. Vision Research 25:195-205.
1985b Visibility of interference fringes near the resolution limit. Joumal of the
Optical Society of America A 2:1087-1093.
Zacks, J.L
1970
Temporal summation phenomena at absolute threshold: their relation to
visual mechanisms. Science 170:197-199.
Representative terms from entire chapter:
distortion product
