|
BOX 2.4
Mode or modality refers primarily to the human senses used to process incoming information (e.g., vision, audition, taction, olfaction), not mode in the sense of purpose (e.g., word processing mode versus spread sheet mode). Additionally, in its conventional definition, medium refers to both the material object (e.g., paper, video) and the means by which information is conveyed (e.g., a sheet of paper with text on it). These definitions could include the possibility of layering, so that a natural language mode might use written text or speech as media even though those media themselves rely on other modes. Media and mode are related nontrivially. First, a single medium may support several modalities. For example, a piece of paper may support both language and graphics, just as a visual display may support text, images, and video. Likewise, a single modality may be supported by many media. For example, the language modality can be supported visually (i.e., written language) and aurally (i.e., spoken language). In fact, spoken language can have a visual component (e.g., lip reading). Just as a single medium may support several modalities and a single modality may be supported by many media, many media may support many modalities, and likewise. For example, a multimedia document that includes text, graphics, speech, and video affects several modalities (e.g., visual and auditory perception of natural language, visual perception of images (still and moving), and auditory perception of sounds). Finally, this multimedia and multimodal interaction occurs over time. Therefore, it is necessary to account for the processing of discourse, context shifts, and changes in agent states over time.
SOURCE: Maybury (1994). |