Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
2 The Field of Cognitive Psychophysiology In this chapter we define cognitive psychophysiology in terms of its two parts, cognitive science and psychophysiology. WHAT IS coGNITWE SCIENCE:? The Oxford Dictionary of the English Language defines cogni- tion as "the action or faculty of knowing taken in its widest sense, including sensation, perception, conception, as distinguished from feeling and volition; also, more specifically, the action of cognizing an object in perception proper." Thus, cognitive science is the body of scientific knowledge pertaining to cognition, defined to include all forms of knowing. Cognitive science focuses on questions about how information must be stored internally and processed in order for an organism to recognize objects, learn, use language, reason, or navigate. Theories are tested in part by attempting to build computer programs that mimic human performance (the so-called computational approach) and in part by using the experimental methods of cognitive psychol- ogy. The computational approach characterizes the nature of infor- mation processing at two levels of analysis. At one level, theorists decompose the processing system into sets of "processing modules," each of which performs some part of the processing used to accom- plish a task. Modules are "black boxes," specifying how specific types of input are transformed to produce appropriate output. Sternberg 7
8 BRAIN AND COGNITION: SOME NEW TECHNOLOGIES (1969), for example, postulated one module that compares an input stimulus in short-term memory to a set of items on a list. At another level, theorists attempt to Recover the way in which processing is actuaBy accomplished within the modules. In some cases processing is characterized by step-by-step sequential manip- ulation of stored symbols, as is done in conventional computers, whereas in other cases, processing corresponds to the formation of patterns of activation in a network of interconnected nodes, as is done in parallel distributed processing systems (Rumelhart and Mc- Cleliand, 1986~. For example, the list-compar~son module posited by Sternberg could operate either by storing the items in memory as symbols in a list and then comparing an input symbol against each stored symbol, or by establishing a pattern of weights distributed through a neural network. In this latter case, comparison of input to stored items is accomplished simply by discovering whether the network settles into a specific state when a given input is presented. In either case, the computational approach leads one to posit a set of modules and to characterize how they serve to transform information. Cognitive psychology has contributed to cognitive science so- phisticated methodologies, a rich data base on characteristics of human performance, and techniques for modeling such data. The methodologies of cognitive psychology are based on observing rela- tive response times, error rates, or types of judgments. For example, cognitive psychologists have developed techniques for inferring prop- erties of processing by analyzing trade-offs between speed and accu- racy (i.e., the inverse relationship between times and errors, which reflects how careful a subject is when responding); they have used signal detection theory in the analysis of errors to determine what is stored. They have also developed numerous methods for obtaining judgments of perceived similarity among stimuli. These judgments in turn can be submitted to multidimensional scaling and cluster analy- ses, allowing one to draw inferences about the processing underlying the judgments. Hypotheses derived from theories that embody dif- ferent modular structures or types of processing are tested against data. For example, if there is a discrete module that compares input to lists stored in short-term memory, then it should be possible to find brain-damaged patients with focal lesions who have lost this specific ability. In short, then, the result of the alliance between computational theorizing and cognitive psychology is the develop- ment of detailed theories of information processing that are not only consistent with the available data about human performance, but
COGNITIVE PSYCHOPHYSIOLOGY 9 that also make empirically testable predictions (see Anderson, 1983; Kosslyn, 1980; Rumelhart and McClelland, 1986~. WHAT IS PSYCHOPHYSIOLOGY? Cognitive psychophysiology refers to the study and use of mea- sures of physiological functions for the purpose of elucidating pro- cesses and mechanisms that underlie cognition. The physiological processes studied include both central nervous system (CNS) and au- tonom~c nervous system (ANS) activities. Traditionally, psychophys- iologists interested in the ANS measure such variables as changes in heart rate or sweat gland activity (Coles, Donchin, and Forges, 1986~. Studies of the CNS have been dominant in cognitive psychophysi- ology and are based on more widely developed technologies than are studies of ANS activity related to cognition. For that reason, the report concentrates on those activities designed to clarify CNS mechanisms involved in human cognition. The human brain ~ largely inaccessible to the sort of fine-grained analysis other organisms can be subjected to in the pursuit of knowl- edge about how neuronal activity relates to psychological processes and states. We already know enough about brain and behavioral processes to see that a full understanding of another organism is insufficient to allow a complete appreciation of how the human brain carries out its appointed tasks. While it is imperative for the student of human behavior to keep an eye on the developments in under- standing brain and behavioral processes in nonhuman species, it is also becoming clear that an understanding of human psychological processes will require studying human brains at work. This is an ambitious goal and one not easy to achieve. THE INTERFACE BETWEEN coGNITWE SCIENCE AND PSYCHOPHYSIOLOGY Three fields are currently engaged in the empirical study of men- tal activity: computational theorists attempt to understand seeing, remembering, reasoning, and so on by building virtual machines that mimic such processes. Cognitive psychologists conduct experiments to measure differences in behavior under different circumstances and attempt to fit models to account for response times, error rates, or various types of decisions. Psychophysiologists try to gain insight into the mind by observing the activity of its neural substrate.
10 BRAIN AND COGNITION: SOME NEW TECHNOLOGIES In addition, it should be noted that scholars in anthropology, linguistics, and philosophy also address issues about the mind, and some aspects of cognitive science draw heavily on these fields. How- ever, this work currently is difficult to connect to psychophysiology; hence we do not consider these facets of cognitive science further here or in the sections to follow. Each of the three fields listed above has virtues as well as lim- itations. Cognitive science has to a large extent grown out of an alliance between the computational approach and cognitive psychol- ogy. The weaknesses of each field taken in isolation are to a large degree corrected for by the strengths of the others. It seems likely that psychophysiology has much to gain from interactions with this new amalgam, and vice versa. In this section we first treat the virtues and limitations of each of the major fields, taken singly, to the study of cognition. We then propose an alliance among them, which would take advantage of each one's unique strengthsempirical, technological, and theoretical- while compensating for the limitations inherent in each single field. The section concludes with a discussion of the advantages of com- bining them, leading to the suggestion, made in Chapter 6, for an enlarged study of the interface between the disciplines. [~ tations and Virtues of a Psychophysiological Approach Psychophysiological data may be especially useful for identifying the structure of information processing in the brain. But to be max- imally useful, they must be used in conjunction with sophisticated theories and methodologies that are capable of discriminating among such theories. Attempts to program computers to behave with the intelligence of even a field mouse have been of limited success. One thing we have learned from such efforts is just how complicated cognitive processing is. Even the simplest task, such as deciding whether a dot is inside or outside a closed boundary, requires sophisticated processing (Uldman, 1984~. If we are to understand the neural ba- sis of cognition, we must be prepared to formulate rather complex theories. Until very recently, however, this has not been done in psy- chophysiology. For example, "localizing oneself in space" is typically considered a single function in the psychophysiological literature, whereas a computationally-oriented theorist would be inclined to de- compose this process into many disparate encoding, representational,
COGNITIVE PSYCHOPHYSIOLOGY 11 and retrieval operations. Similarly, visual agnosia ("m~ndblindness") is described and the underlying causes of the deficit are explained by reference to damage of anatomical areas and their connectionsbut exactly what is done by these areas is never clearly specified. Thus, to expand the contribution of a psychophysiological ap- proach it is of interest to consider what the two major strands of cognitive science, computational theorizing and cognitive psychol- ogy, can offer. Limitation and Virtues of the Computational Approach Although the brain clearly is not a standard digital computer, brain activity can be conceptualized as the carrying out of computa- tions. Computational modeling of brain activity occurs at multiple levels of analysis. The most appropriate level for present purposes focuses on the decomposition of processing into modules, each of which may correspond to a distinct neural network. Any given task presumably recruits many such modules to work together, and the ways in which modules interact determines task performance. Al- though specifying the precise operation of the individual modules is of course critical for a theory of information processing, at the current level of technology we are unlikely to be able to use methods of as- sessing brain activity to directly test theories at this level of analysis. The main contribution of the computational approach to cognitive psychophysiology will therefore probably be to offer guidelines for how one formulates theories of processing modules. An example is the work of Marr (1982~: according to Marr, the most important task is to formulate the "theory of the computation," a theory of what is computed by a processing module. Marr argues that the information available and the purpose of a computation often virtually dictate what the computation must be. This sort of theory can be likened to a solution to a mathematics problem, arising through logical analysis of the nature of the problem to be solved and of the input available to solve it. That is, if the task is very well defined and the input is highly restricted, a specific computation may almost be logically necessary. Furthermore, Marr claims that once a computation is defined, the task of characterizing the representations and processes used in carrying out the step-by- step processing itself is now highly constrained: the representation of the input and the output must make explicit the information
12 BRAIN AND COGNITION: SOME NEW TECHNOLOGIES necessary for the computation to serve its purpose (e.g., picking out likely locations of edges), and the representations must be sensitive to the necessary distinctions, be stable over irrelevant distinctions, and have a number of other properties (see Marr, 1982, Chapter 5~. Marr's strong claims about the importance of the theory of the computation do seem appropriate for some of the problems of low- leve! vision, but only because there are such severe constraints on the input posed by the nature of the world and the geometry of surfaces and because the purpose of a computation is so well defined (e.g., to detect places where intensity changes rapidly, to derive depth from disparities in the images striking each eye, to recover structure from information about changes on a surface as an object moves). In broader areas of cognition, the situation is different. First, the basic abilities in need of explanation, analogous to our ability to see edges or to see depth, must be discovered. For example, with the advent of new methodologies, our picture of what can be accomplished in mental imagery has changed drastically (e.g., see Shepard and Cooper, 1982~. Second, the input to a "mental" computation is often not obvious, not necessarily being constrained by some easily observed property of the stimulus. One must have a theory of what is represented before one can even begin to specify the input to the computations. Third, the optimal computation will depend in part on the kinds of processing operations that are available and the type of representation used. For example, if a parallel-distributed processing network is used, computing the degree to which an input is similar to stored information should be relatively easy, whereas serial search through a list will be more difflcult-and vice versa if symbols are stored as discrete elements in lists that are operated on by distinct processes. The consequences of these difficulties are illustrated by problems with some of Marr's own work on "higher-levein vision. Marr posits that shapes must be stored using "object-centeredn descriptions, as opposed to "viewer-centeredn descriptions. In an object-centered description, an object is described relative to itself, not from a partic- ular point of view. Thus, terms such as dorsal and ventral would be used in an object-centered description, rather than top and bottom, which would be used in a viewer-centered description. Marr argues that because objects are seen from so many different points of view, it would be difficult to recognize an object by matching viewer-centered descriptions to stored representations. However, this argument rests on assumptions about the kinds of processing operations that are
COGNITIVE PSYCHOPHYSIOLOGY 13 available. If there is an Orientation normalization" preprocessor, for example, the argument is obviated: in this case, a viewer-centered description could be normalized (e.g., so the longest axis is always vertical) before matching to stored representations. And in fact, we do mentally rotate objects to a standard orientation when subtle judgments must be made (see Shepard and Cooper, 1982~. The fact that we do seem to normalize the represented orientation, at least in some cases, casts doubt on the power or generality of object-centered representations. In fact, when the matter was put to empirical test, Jolicoeur and Kosslyn (1983) found that people can use both viewer- centered and object-centerec} coordinate systems in storing ~nforma- tion, and they seem to encode a viewer-centered one even when they also encode an object-centered one, but not vice versa. The point is that a logical analysis of the computation is not enough. At least for high-level cognitive functions, the specifics of a computation will depend to some extent on what types of processing operations are available in the system. One can only discover the actual state of affairs empirically, by studying the way the brain works. Although the computational approach is not sufficient in itself to lead one to formulate a correct theory of information processing, it does have a lot to contribute to the enterprise. Analyzing how one could build a computer program to emulate a human function is a very useful way of enumerating alternative processing modules and algorithms. Not only does this approach raise alternatives that may not have otherwise been considered, but it also eliminates others by forcing one to work them out concretely enough to reveal their flaws (the Guzman approach to vision is a good example; see Winston, 1978~. [nnitatione and Virtues of the Cognitive Psythology Approach The predominant approach in cognitive psychology is solidly empirical: researchers have developed methodologies that make use of response times, error rates, and various judgments and have at- tempted to develop models that account for these data. The method- ologies used have become very sophisticated and powerful, allowing researchers to observe quite subtle regularities in processing. As we saw in the previous section, such data place strong constraints on theories of processing: since processing takes place in real time,
14 BRAIN AND COGNITION: SOME NEW TECHNOLOGIES there will always be measurable consequences of any given sequence of activity. Although cognitive psychologists occasionally focus on the na- ture of the step-by-step process a subject ~ using to carry out an entire task (e.g., see Simon and Simon, 1978), more typically they are interested in studying how information is represented and processed within a single stage of processing. However, it has proven difficult to draw firm conclusions about the representations or processes used in even one stage of processing because of two general problems: structure/process trade-offs and task demand artifacts. Anderson (1978) demonstrated that structure/process trade-offs are in principle always possible, so that, given any set of data, more than one theory can be formulated to account for the data. That is, what are, In one theory, properties of a given representation operated on by a specific process are, in another theory, properties of a different representation operated on by a different process. For example, consider the memory scanning results described by Sternberg (1969~. He asked subjects to hold lists of digits in mind, with lists varying from 1 to 6 in length. Shortly thereafter, a probe digit was presented, and subjects were to decide as quickly as possible whether the probe was a member of the list. The tone to make this decision increased linearly with increasing set size (by about 39 ms per additional item). One theory of this result posits that the list of digits (the structure) is held ~ nd and then scanned serially (the process) when the probe arrives. Alternatively, one could posit an unordered collection (the structure) with each item being compared simultaneously with the probe (the process). In this case, all one needs to do is assume that the comparison process slows down as more things need to be compared, and the two theories wall mimic each other. More time is required when more items are on the memorized list to be compared with the probe. In this example, the two theories seem to account for the data equally well but they were created entirely ad hoc simply to account for the data. Constraints on the theories are required, a source of motivation for selection of the specific representations and processes. Why should information be represented as an ordered list or as an unordered collection? Why is more time required if one compares more items simultaneously? Computational considerations are one possible source of constraint. However, we saw in the previous section that computational constraints in themselves are not sufficient, and
COGNITIVE PSYCHOPHYSIOLOGY 15 in fact the observation of how the system functions puts constraints on computational theories themselves. Anderson (1978) drew some very pessimistic conclusions from the possibility of structure/process trade-offs, but others such as Hayes- Roth (1979) and Pylyshyn (1979) were less gloomy. The upshot of the debate seems to be that while it is possible to derive inferences about processing mechanisms from behavioral data, it is very difficult to do so. One argument to be developed here is that psychophysiological data are powerful supplements to the usual behavioral data, and would greatly constrain the use of structure/process trade-offs to develop alternative theories. Another problem in interpreting behavioral data is the possibility of distorting behavior because of perceived task demands. That is, subjects may respond in a manner congruent with their beliefs concerning acceptable behavior to the task and the situation. If they do so, then data from many studies of, for example, mental imagery may say nothing about the nature of the underlying mechanisms, but may only reflect the subjects' understanding of tasks, knowledge of physics and perception, and ability to regulate their response times. Although the problem of task demands h" been brought to the attention of cognitive psychophysiologists primarily in the literature of mental imagery, it is applicable to many domains in cognitive psy- chology and, indeed, in other areas of psychology. There is no way to ensure that subjects are not unconsciously producing data in accor- dance with their tacit knowledge about perception and cognition and their understanding of what the task requires them to do. In con- trast, not only do neurological maladies produce behavioral deficits of various types, but often the patients are not aware of the nature of the deficits. Thus, psychophysiological data might profitably supple- ment the usual cognitive data, if for no reason other than to rule out task demand as a source of explanation. And such data are useful for other purposes, as discussed in the following section. The Strength of a Co~bmed Approach Psychophysiological approaches can be used to circumvent some of the difficulties inherent in the traditional measures user] by cog- nitive psychologists, which are based strictly on the observation of overt responses. First, structure/process trade-offs are greatly min- imized if neurophysiological data are used. By relating processing to anatomical areas, many of the degrees of freedom are removed
1 16 BRAIN AND COGNITION: SOME NEW TECHNOLOGIES from cognitive science theories: in all cases in which a given area is active or damaged, the consequences must be the same. When one has fixed the properties of some area, those properties cannot be changed at whim by a theorist in order to account for new data. Second, difficulties due to task demands are virtually eliminated if brain activation measures are used, because subjects cannot respond to explicit task demands by directly altering the activity of specific regions of the brain. Whereas a person can regulate the time taken to press a button, it is not so easy to regulate intentionally the activity of the right parietal lobe, for example. In addition, psychophysi- ological measures can be used to monitor on-line and in real time the activity of processing entities that are not directly manifested by overt behavior. The computational approach, by contrast, especially as con- stra~ned by data from cognitive psychology, is useful for generating hypotheses about processing mechanisms. Analyzing the require- ments of the task at hand and how one would need to program a computer to perform it is a good way to generate alternative possi- bilities. In addition, this approach provides a way of testing complex theories by actually building a computer program that emulates cog- nitive processing (see Newell and Simon, 1972~. Precise theories of on-line brain functioning may be so complex that many of a theory's implications will be derives} only by using simulation models. Furthermore, once there are prior reasons for positing a spe- cific modular composition of the system, the standard techniques of cognitive psychology become more powerful. When a module is de- fined, the number of degrees of freedom is reduced for possible struc- ture/process trade-offs. That is, without modularity constraints, any part of the system can be invoked in combination with any other part to explain a specific result; but if a result can be shown to rest on the operation of a specific module, the explanation of the result is limited to fewer alternatives. When well-specified classes of alternative theo- ries are defined, cognitive psychologists will be better able to specify which phenomena will distinguish among competing accounts (for an example see the mental rotation case noted above in Kosslyn, 1980: Ch.8~. One example of progress following from such a combined ap- proach began with computational analyses suggesting that spatial localization should be decomposed into at least two types of pro- cesses. On one hand, if one were to build a machine to recognize semirigid objects (e.g., a human form) it would be desirable to include
COGNITIVE PSYCHOPHYSIOLOGY 17 a module to encode representations of rather broad categories of spa- tial relations among parts. Such representations would be constant over different contortions of the object. For example, the forearm and upper arm remain connected (a categorical relation) no matter how they are configured. On the other hand, if the machine is also intended to navigate and reach for objects, it is desirable to include a module to encode representations of the specific metric coordinates between parts or objects. For these purposes, a broad category of relations (e.g., one object ~ cleft of" another) is not useful; one needs to know precise positions. The possible Extinction between these types of representation has been investigated by noting that categorical representations are I~guage-like (all can be easily named by a word or two) and hence might be processed more effectively in the left cerebral hemisphere. In contrast, coordinate representations are critical for navigation, which appears to draw in large part on right hemisphere processes. And in fact, it has been found that cat- egorical spatial relations are apprehended more effectively ~ the left hemisphere, whereas coordinate relations are apprehended more em fectively in the right hemisphere (Kosslyn, 1987' 1988~; this inference is based in part on work using some of the technologies discussed in this report. This dissociation provides evidence for the existence of distinct processes underlying the two types of spatial representation, which was not obvious until computational analyses led to the dim tinction between the two and specific brain-based hypotheses were tested. In summary, psychophysiological data offer constraints both on theories of processing modules and theories of the algorithms used. The logic of dissociations and associations in deficits or patterns of brain activation ~ a powerful way of developing and testing compu- tational theories, particularly so if it is supplemented by the method- ologies and analytic techniques of cognitive psychology. The method- ologies developed by the cognitive psychologists for the most part can be adapted for use in psychophysiological studies.