Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 61
Beyond Productivity: Information Technology, Innovation, and Creativity 3 Advancing Creative Practices Through Information Technology How can computer scientists support new artistic and design practices? How is computer science (CS) research and development being stimulated and altered by emerging practices at the intersection of information technology (IT) and the arts and design? What are the prospects for new research directions that are interesting and useful for both computer science and the arts and design? Answering these questions requires a close look at the relationship between IT and the arts and design. That examination begins with this chapter, which focuses on the design, applications, and implications of the tools of IT from the perspective of the arts and design worlds, and it continues in Chapter 4. STRANGE BEDFELLOWS? One of the more obvious ways in which IT and the arts and design interact is in the use of technology to extend the expressive range of and modes of access to existing genres of the arts and design: Examples include Web-based art and hypertext, opera staging using new sensing and video technologies, musical compositions that feature both newly created instruments and interaction styles, and textile design and production based on digital weaving techniques. Given experimentation to date, it is clear that new tools developed by computer scientists can be immediately applied by artists and other creative practitioners within a wide array of contexts. But there are further important implications of information technology and creative practices (ITCP) for computer science research and development. Box 3.1 provides context on the nature of that research for those who may be unfamiliar with it. Rather than using computational technology as black boxes for arts and design applica
OCR for page 62
Beyond Productivity: Information Technology, Innovation, and Creativity BOX 3.1 Computing, Computer Science, and Research Computing is rooted in the discipline of computer science,1 which studies information and computational processes—including the representation, implementation, manipulation, and communication of information.2 There are relatively few inherent natural limitations in computer science, as compared with other science and engineering fields such as physics, chemistry, and mechanical engineering. This means that the definitions—and, by extension, the capabilities—of computing and IT often do not have an obvious finite upper bound for expansion. There are, however, practical constraints on the available capabilities of computer hardware and software. In addition, the growing reach and complexity of computers and networks (such as the Internet) have heightened the risk and impact of system failures and created formidable challenges in areas such as security and the management of intellectual property. Efforts to improve computing capabilities are central to IT research. The key intellectual themes in computer science and engineering are algorithmic thinking (i.e., about rules for processing information), representation of information, and computer programs.3 Some IT research lays out principles or constraints that apply to all computing and communications systems; other studies focus on specific IT systems, such as user interfaces. 1 The history of computing (which involves other fields such as electrical engineering) has been widely documented; see, for example, Computer Science and Telecommunications Board, National Research Council, 1999, Funding a Revolution: Government Support for Computing Research, National Academy Press, Washington, D.C.; and the public television documentary Triumph of the Nerds, transcript available at <http://www.pbs.org/nerds>. 2 These aspects of computer science will be discussed in the forthcoming report of the Committee on the Fundamentals of Computer Science—Challenges and Opportunities, Computer Science and Telecommunications Board, National Research Council. 3 See Computer Science and Telecommunications Board, National Research Council, 1992, Computing the Future: A Broader Agenda for Computer Science and Engineering, Juris Hartmanis and Herbert Lin, eds., National Academy Press, Washington, D.C. tions, some are engaging in IT research as a form of art and design practice itself. This activity is less like historical arts research and more like computer science research, although it asks radically different kinds of questions and introduces a variety of methodologies generally unfamiliar to computer scientists. In certain areas of research, particularly human-computer interaction and artificial intelligence, there may be a convergence developing between new trends within CS research and the work of “outsiders” who bring in fresh perspectives. Unlike the use of computers for particular applications, this intersection of art and design and IT research leads to some deep and fundamental rethinking of CS research and what it is about in the first place. The intended outcomes go beyond making new tools for art and design practice, though that may be one outcome, to arrive at a fundamentally new way to do research—a true hybrid. Both of these perspectives on interaction are important for the future of ITCP. In practice, they are intermingled. On the one hand, developing tools for new kinds of practices can lead to fundamental insights into the tool-development process. On the other, artists and designers who get their hands dirty in fundamental CS research are
OCR for page 63
Beyond Productivity: Information Technology, Innovation, and Creativity able to build new tools and applications that can be useful for the arts and other creative domains. The implications of this new medium for artistic and design practice run deeper than simple application. In previous communications revolutions, a new medium that was widely adopted not only added new possibilities for artistic and design expression, but also changed the way older media were used.1 Communication media influence the relationship between sense and bodily skill, and they alter the way in which artists and designers reason or feel about time and space. If one were to think of such a shift as a radical break, in which a developing medium brings an entirely new art form into existence with little relationship to historical precursors, the naïve response for computer science researchers would be simply to generate as many new-media forms as possible as a way of advancing the opportunities for art and design practice. One technically oriented current in the contemporary avant-garde has indeed sought to push technology forward, in order to discover new expressive possibilities and genres that can be conceived only with the help of advanced technologies. Technological advances exert a strong pull, challenging artists and designers to conceive new expressive forms that can take full advantage of ever-increasing processing speeds and bandwidth rates. Both this push on and pull of new technologies focus on the new possibilities created by those technologies, rather than on the needs and perspectives of art and design practices using “old” (and by implication out-of-date) media. It is a mistake to overemphasize the entirely new digital worlds that are uniquely possible using computers, as though the adaptation of already existing content or art forms to the new medium were only a lesser, transitional stage on the way toward the more significant discovery of purportedly new, essentially digital art forms.2 What tends to be overlooked, both by the modernist artist’s and designer’s technology push and the information technologist’s pull for advanced content, are the subtle and by no means trivial processes of change occurring in the traditional art and design forms as they adjust to and begin to find their own responses to information technologies. See Box 3.2 for an overview of how technology has influenced music while traditional activities have persisted. Precisely because these developments are less visible, whether judged by criteria of radical artistic and design novelty or technical 1 The rise of broadband continues that phenomenon. For example, one recent study found that, because of Internet use, 37 percent of broadband users watched less television, 31 percent spent less time shopping, and 18 percent spent less time reading newspapers; nearly 90 percent said that the Internet had improved their ability to learn. See <http://www.pewinternet.org/reports/toc.asp?Report=63>. 2 See J.D. Bolter and R. Grusin, 1999, Remediation, MIT Press, Cambridge, Mass., pp. 48-50. Note that this situation may differentiate the arts from various practical activities (e.g., office and factory work), where early mechanization was, indeed, a step toward broad reconceptualization of different activities. Tradition and continuity are recognized across the arts. The naïve response for computer science researchers would be to generate as many new-media forms as possible for art and design practice.
OCR for page 64
Beyond Productivity: Information Technology, Innovation, and Creativity BOX 3.2 Technology, Music, and the Evolution of Expectations Consider the history of music. Early music was not written—there was only the performance. The evolution of the musical score, a standardized interface between the composer and the performer, made it possible for the composer to achieve wider visibility by having his music performed numerous times by different groups. One tradeoff for this wider distribution was limited scope—only music that corresponded to this standard score could be written down (excluding all but Western music). Another was that the performance was still transient. It is said that Bach and Mozart were wonderful performers, but records are preserved mainly in listeners’ accounts, and in the occasional attempts by both masters to notate transcripts of their more spontaneous fantasy. With the advent of recording, the performance itself could be captured. And the relationship between the composer and performer shifted. More improvisational styles such as jazz, blues, and rock can now be captured as a performance, not as a score, and the performer of current popular music is expected to perform his own music, not “cover” the music of others. We are back to the era of Bach or Mozart, but with global scale—the benefits of wider distribution now extend to other forms of music that cannot be written down. In turn, recording technology also enabled a new form of music—music that was created in whole or in part with synthesized, recorded, electronic signals, defying both transcription and performance. Thus, the introduction of recording raised new questions about music as a creative process and the definition of a live performance. There is something incongruous about attempting to trigger all the pleasure and excitement of a live gathering of an audience to look at a computer and a bunch of speakers up on stage. To some extent, of course, this is because we know that there will be no spontaneity in the performance— that creativity has already happened. This is what recording did to the creative aspect of music. The computer is now doing much more. First, the computer can now be the performer. Digital technology allows composers to craft their own instruments and to create a performance using only the computer. Computer-driven pianos now replay a performance with many of the nuances of the original pianist’s keystrokes. Interactive computer programs can participate in the performance, producing and manipulating sounds in response to a performer’s actions. Digital technology, through interactivity, helped bring spontaneity back into the performance of electronic music. What seems to be evolving is a remodularization of the creative process, the resolution of which is a creative act in its own right, and one that cannot escape the feedback and interaction with the listener. The role of the listener, the audience, illustrates how advances are gated by a social process. Secondly, the computer can capture a performance and make it permanent. What the recording industry did for music, the computer can in principle do for a more dynamic and multimedia creation. Consider also the emergence of the disc jockey who does a live remix of pre-existing music (including electronic music): This seems to be a reassertion of the persistent value of the spontaneous process of performance, involving real-time feedback between the performer and the audience. Here, again, questions arise about how to view the “dilemma” of the computer as a part of the creative process. progress, they receive less attention than more obviously glamorous showcases do. But these subtle developments are no less important for the long-term ecology of digital culture, suggesting limitations of, and different possibilities for, the development of technology as a medium. For example, tacit knowledge—unformalized and probably unformalizable knowledge such as design methodologies or embodied skills such as drawing or dancing—has always played an impor
OCR for page 65
Beyond Productivity: Information Technology, Innovation, and Creativity tant role in the arts. Yet IT has by its nature had severe difficulty codifying the most subtle and refined of such artistic practices. The ineffable human feel can be simulated acceptably through clever tricks, but the danger all too often is that consumers of technology, including artists and designers, will accommodate themselves to the reduced expressive bandwidth afforded in easy-to-use interfaces, as discussed below.3 The emerging research paradigm for embodied interaction in human-computer interaction (HCI) is one opportunity for a different style of interface perhaps more compatible with highly skilled art practice.4 TOOLS NEEDED TO SUPPORT CREATIVE WORK: HARDWARE AND SOFTWARE Computer and communications hardware and software are the tools of ITCP and the means by which almost all digital media are created and manipulated. These tools can do many things,5 each and all of which may be embraced in ITCP. A simple list of gross capabilities would include: Automation of processes such as drawing, composing, editing, and so on (assorted software); Handling, representing, and displaying or performing information (databases, browsers, displays and speakers, printers, projection systems, and so on); Analysis of information and phenomena (visualization and sonification, modeling and simulation, artificial intelligence and learning systems); 3 For sociological analysis of creative effects of the commoditization of musical instruments in digital forms, see P. Théberge, 1997, Any Sound You Can Imagine: Making Music/Consuming Technology, University Press of New England, Hanover, New Hampshire. 4 Embodied interaction has been a theme in digital arts at least since Myron Kreuger’s VideoPlace of the early 1970s. See P. Dourish, 2001, Where the Action Is: The Foundations of Embodied Interaction, MIT Press, Cambridge, Mass.; and P. Dourish, 1999, “Embodied Interaction: Exploring the Foundations of a New Approach to HCI” (see <http://www.ics.uci.edu/~jpd/publications>). An example of a new body-centered interface for art is described in Steven Schkolne, Michael Pruett, and Peter Schröder, 2001, “Surface Drawing: Creating Organic 3D Shapes with the Hand and Tangible Tools,” pp. 261-268 in Proceedings of CHI 2001, ACM Press, New York. 5 Broad interpretations include not only components and artifacts but also their theoretical underpinnings and digital content. The Computing Research Repository, for example, lists 34 subject areas; see <http://xxx.lanl.gov/new/cs.html>. The Computing Research Repository is an online archive of computer science research results that uses the Internet to allow access to technical reports, conference papers, and other work on a near-real-time basis.
OCR for page 66
Beyond Productivity: Information Technology, Innovation, and Creativity BOX 3.3 One Set of ITCP Technologies One perspective on technologies relevant to ITCP work is provided by Carnegie Mellon University’s Entertainment Technology Center (ETC), which offers a master’s degree jointly conferred by the College of Fine Arts and the School of Computer Science. Entertainment technology “requires a fluid definition, necessitated in large part by advances in technology that are making possible ever-new entertainment experiences and venues. What was meant by the phrase entertainment technology as recently as a year ago requires redefinition in light of recent developments in both technology and entertainment.”1 Nevertheless, the following listing is provided: Networked and free-standing interactive computer games Avatar creation and utilization Massive multiplayer online games Digital entertainment Specialty venues such as theme parks Motion-based rides Console and PC interactive game design Creation of unique input devices Virtual reality utilizing head-mounted displays Other forms of virtual reality technology Wearable computing for entertainment purposes Massive immersive display environments Interactive robot animatronics Synthetic interview technology Speech recognition Augmented reality Telepresence for entertainment and education purposes Digital production and postproduction Sound synthesis, surround sound, three-dimensional sound, and streaming audio Development of haptic devices (e.g., force feedback) Entertainment robotics. 1 See <http://www.etc.cmu.edu/about.html>. Connection to the physical world (sensors, microphones, digital cameras, actuators, robots, human interfaces, systems for interactivity); and Communications (telephony, television, the Internet, and assorted underlying capabilities, from wireless to broadband connectivity). Ultimately, IT acts on information typically associated with products of the human mind (pictures, music, and ideas), although IT must also address data and information in less processed, intermediate states.6 6 This concept is derived from Computer Science and Telecommunications Board, National Research Council, 1992, Computing the Future: A Broader Agenda for Computer Science and Engineering, Juris Hartmanis and Herbert Lin, eds., National Academy Press, Washington, D.C.
OCR for page 67
Beyond Productivity: Information Technology, Innovation, and Creativity Human-computer interaction specialist Ben Shneiderman argues that IT for creativity support falls into eight categories: searching, visualizing, consulting, thinking, exploring, composing, reviewing, and disseminating.7 Because the currency of IT is bits, IT tools can handle any mode or medium, and they can integrate any combination of media— although sophistication has costs in complexity and dollars, and for technical and/or economic reasons, not all desired effects are possible (see “Economic Realities” below). For a concrete illustration of a tool set inspired by entertainment applications, see Box 3.3. Lists like these show the broad range of IT that may be linked to some aspect of the arts or design through ITCP; no list conveys all the creative possibilities inherent in the use of IT. By definition, tools are supposed to be helpful, but like other tools, IT tools have shortcomings. The insights into the nature of IT that collectively constitute fluency, as described in Chapter 2, include an understanding of the limitations of the tools and the constrained nature of the typical software design. The pervasive hype about IT in the mass media may, by contrast, feed unrealistic expectations. Although artists and designers share frustrations with other user groups, their perspectives, like those of other users, may help illuminate new paths for IT research and development. This situation was recognized in a recent special issue of a leading computer science journal: Many significant advances in research on human creativity have occurred, yet today’s tools often contain interface elements that stymie creative efforts. A discontinuity exists between technology tools and our ability to interact with them in natural, beneficial, and most importantly, for this discussion, creative ways.8 Tools vary in terms of the computing and programming skills required to use them. As long as the tools required to produce computer-mediated work are programming tools, the result will be programmer-created design. That is not a bad thing—and in some cases the result(s) can be wonderful. But it does mean that an investment must be made in learning, which is somewhat like the requirement to master other tools used by artists, but also different, because of factors such as the range of features and capabilities available from software and the relatively rapid change in technology. There is a great distance from the paintbrush or piano to programming in C++. As emphasized in Chapter 2, fluency can provide a middle ground between simple acceptance of a tool and expertise in programming.9 Although the 7 See Ben Shneiderman, 2002, “Creativity Support Tools,” Communications of the ACM 45(10): 116-120. 8 Winslow Burleson and Ted Selker, 2002, “Introduction (Special Issue: Creativity and Interface),” Communications of the ACM 45(10): 88-90. 9 Of course, new approaches to programming that simplify it may be helpful— assuming that the results do not present the same concerns that software packages do about built-in constraints. There is a great distance from the paintbrush or piano to programming in C++.
OCR for page 68
Beyond Productivity: Information Technology, Innovation, and Creativity discussion in this chapter focuses on software and to a lesser extent hardware for creating artifacts and performances, it should be noted that other tools consist of content (images, sounds, text, and so on) repositories. There the concerns center on the accessibility of the content—involving indexing, permission to use (or ease of obtaining same), and so on. Yet still other tools support public access.10 HARDWARE AND SOFTWARE TOOLS: A MIXED BLESSING Information technology has obviously proved useful and accessible enough to give rise to ITCP in many guises. In the process, a number of observations have emerged about the nature of IT and the adaptations that artists and designers make in using it. The concerns gleaned from artists and designers help to explain why it takes a long time to integrate a new technology into the making of non-trivial art or design work. Because all computers are universal machines, more advanced designs can replace simpler ones, providing improved performance without changing the basic functionality of computation. Software is even more mutable in that it has almost no physical constraints. Because of rapid changes in hardware capabilities, software rarely matures to a stable configuration. On the positive side, this pace of change provides frequent opportunities to incorporate improved support for creativity in new systems. But for most users, especially in relatively resource-poor areas such as the arts and humanities, this constantly changing tool set is difficult to master. For well over a decade, it has been the lament of artists and designers that the intellectual and financial demands of constantly updating tools and playing technological catch-up results in low-quality work and burnout.11 Smart artists, observed a reviewer, resign from the Moore’s law rat race. These conditions suggest that it is reasonable to expect a wide range of willingness and ability among artists and designers to retrain and to upgrade their tools; how this will affect ITCP remains to be seen. Developers of software tools that can support creative practices have a number of variables to consider, all of which may affect the 10 See tool characteristics in Sharon L. Greene, 2002, “Characteristics of Applications That Support Creativity,” Communications of the ACM 45(10): 100-104. 11 An anecdote shared by a reviewer featured a senior colleague who reported that when he discovered the Amiga computer, he felt sure he had found the tool with which he would make his magnum opus. After several years of learning about and constantly upgrading his Amiga tools, he had become an Amiga expert but as yet had produced no work. Much to his chagrin, the machine then became obsolete. More generally, in the mid-1990s a committee of media arts faculty prepared a document, colloquially referred to as the “‘burnout” document, that outlined the new and unrecognized loads on media arts faculty. It was endorsed by the College Art Association, Inter-Society for the Electronic Arts, and Special Interest Group on Computer Graphics and Interactive Techniques. It takes a long time to integrate a new technology into the making of non-trivial art or design work.
OCR for page 69
Beyond Productivity: Information Technology, Innovation, and Creativity ways in which users interact with the tools—and through the tools, their own work—and the ease with which users can produce original or even groundbreaking works with those tools. To name just a few, tool developers dictate the user conceptual model (or metaphors) exposed by their tools, the amount of structure supported or imposed on the work process and product, the number and kinds of different presentations and representations of content supported, the kinds of manipulations directly implemented, the openness and extensibility of the tool at various levels, and the levels of abstraction afforded. The developers’ decisions often reflect attempts to make the system easy to use,12 or simplifying assumptions about what users want, but they can also greatly affect what users produce.13 Very little research has explored the relationships between these design decisions and the fitness of the tool for various kinds of ITCP work—what follows are some observations made by the committee. In an effort to get work done, it is only natural to follow the path of least resistance established by the tools that are available, so these tools play a very important part in how users conceptualize their work and assess their options. The tools may also leave traces: Architects may look at a building and detect which design tools were used, and musicians may hear new pieces and detect which composition tools were used.14 Thus, IT may act as a flywheel in at least some contexts of its use. Artists with IT fluency recognize these practical realities and their roots in the values and procedures of computer science; they work with or around the limitations of the tools. In the popular style known as object-oriented programming, a programmer creates a new kind of object (concretely, a data structure that describes something) and defines the operations that other programmers can perform on it. Other programmers that use the object cannot get at the data structure itself, but only at the operations that the object-creator defined. This gives the object-creator the ability to come back later and change the actual data structure (perhaps to make it more efficient) without having to coordinate with any other pro 12 An early illustration in a number of (non-artistic) fields was statistics; statistical software (and later spreadsheets) popularized and helped to disseminate a variety of quantitative methods used in different applications. More recently, software packages have begun to implement neural networks, face recognition, data acquisition, or image display. Because these functions can be technically difficult to create, careful packaging can allow others to use advanced technology with less effort. 13 For example, the literary theorist has written, with reference to hypertext authoring environments: “The strength of metaprograms is that they take away most of the pain involved in programming an application from scratch; . . . [their] weakness is that they limit the programmer by presenting a predefined range of operations that the programmer must use. . . . This may be compared to pre-modern modes of authorship, in which the author could use predefined paradigms to produce a genre text, without much creative effort.” See Espen J. Aarseth, 1997, Cybertext: Perspectives on Ergodic Literature, Johns Hopkins University Press, Baltimore, Maryland, p. 173. 14 In other contexts, such as education, people have noted how the presentation software PowerPoint shapes and constrains their choice of content and organization for public speaking. Tool developers dictate the user conceptual model (or metaphors) exposed by their tools.
OCR for page 70
Beyond Productivity: Information Technology, Innovation, and Creativity It is much easier to design a tool for a well-defined task than for non-deterministic, creative work. grammers. The drawback is that the user can perform only the operations the object-creator thought of. If the user wants to add a new operation, he or she cannot, if doing so requires getting at the base data representation. It is much easier to design a tool for a well-defined task than for non-deterministic, creative work, which cannot be reduced to a set of tasks. As one example of how these design decisions can affect the utility of tools, consider an image-editing program that provides a “blur” feature with a capability for “more” and “less” control. Technically, this is a simple mathematical convolution on the image, but the tool designers chose not to expose the numeric parameters for this operation. The benefit is that an operation assumed to be common—blurring an image—is simplified. The tradeoff is that other convolutions— which might prove interesting or useful to a user—are impossible without some other software mechanism. One design point that provides good support for explorations of ITCP is to use a small number of concepts in a general and flexible way combined with openness and interoperability with other tools. A good example is the interactive music language “Max,” a full visual programming environment that has developed a community with at least three ranks of creative users. The least technical of these would be a performer who develops skills as an interactor with a particular set of patches (configurations), much like a musical performer would practice her instrument. Users may then learn to make their own programs, by patching together already existing objects in unique ways. Such patches then become instruments that permit a high degree of virtuosity for use within particular genres (like techno-music or interactive dance). Creating a good tool requires an understanding of the problem area and often experience at the cutting edges of artistic and design and technical disciplines, just to understand the underlying concepts and representations that the tool will address. Tool making also can benefit from combining technical expertise in designing and implementing software with a healthy dose of common sense to arrive at something general enough to be useful but simple enough to be used.15 Problems sometimes arise when tool design assumes too much separation of idea from expression.16 15 People familiar with computer science might consider Donald Knuth as an early explorer of ITCP. Venerated for his understanding of software, Knuth has done work on fonts and electronic composition that is recognized broadly. 16 Wright points out that the separation between idea and expression in software— which can degenerate to the software supplying all the ideas—is one example of a more general problem with creative software tools. These tools most commonly “aim to mirror internal creative process by organizing it into an external data process or structure. The software is a system of menu commands and options which seeks [is designed] to match an internal model of creativity as a process of decision making that seeks to approximate an ideal artistic goal.” But artists and designers “do not know in advance what they want before they start—the creative process is actually a process of
OCR for page 71
Beyond Productivity: Information Technology, Innovation, and Creativity A tool may offer just the right functionality, but not at the right component level. For example, an image can be manipulated through the use of a comprehensive interactive application such as PhotoShop, or by typing a command to run a specific program such as one of the Unix ppm utilities,17 or by calling a function in an image processing library. A software library is of little use to a non-programmer, whereas a full-blown application is of little use in creating a custom software program. Sometimes, a full-blown application can be made to perform like a custom software program through scripting, and sometimes software libraries or other components can be integrated with an application via a plug-in capability (see below). More could be accomplished with tools designed to suit a range of different tasks. Ensuring this flexibility calls for either a variety of tools or a hierarchical structure that allows access to tools at various levels of abstraction, permitting a migration path from intelligent enhancement at the novice level to customization for experts (as illustrated by the language “Max” described above). Because creativity is associated with novelty, comprehensive tools for creative work will be neither possible nor necessary to develop, any more than it is necessary for a pencil to include all functions for drawing. It will always be the task of the innovator to create new tools from components, to create new applications, and to create new artifacts by using tools in unanticipated ways. In this respect, ITCP will draw from both artistic and design and computer science traditions with experimentation. Small professional communities that share experiences and talent often develop effective tools—this phenomenon is evident in the growth of computational science and the development of the Web by and for physicists, and it has been observed in humanities fields as well.18 Many small tools are developed by individuals or small teams that are able to keep their tools focused and coherent. Design choices related to tool extensibility may be particularly important for broadening participation in ITCP and tool development. Programming language extensions (also called scripting languages) allow end users to customize applications. For example, computer music and animation programs often use text-based scripts or scores to describe music and images. Users can create scripts either by editing text directly or by writing programs to generate scripts. Another way to customize tools is with a plug-in, which is a software compo playing and ‘visual thinking’ that leads to a variety of interim ‘solutions’ and modifications of the original ‘problem’ . . . .” See Richard Wright, 1999, “Programming with a Paintbrush: A Study in the Production Culture of the Moving Image,” Filmwaves, Issue 12. 17 These programs perform a wide range of image-processing tasks including format conversion, scaling, filtering, and color adjustment. 18 See American Council of Learned Societies, 1998, Computing and the Humanities: Summary of a Roundtable Meeting, Occasional Paper No. 41, available online at <http://www.acls.org/op41-toc.htm>. Comprehensive tools for creative work will be neither possible nor necessary.
OCR for page 85
Beyond Productivity: Information Technology, Innovation, and Creativity problems, and solutions through electronic bulletin boards. Most of the contributors work without monetary compensation. In both examples, it is important to have a relatively small group to coordinate and integrate various contributions in order to maintain a coherent product. Open-Source Tools for Learning Python, based on early Lisp-like languages, is excellent for art and design education because of its simple and powerful syntax. An example of looking ahead to new models for languages is the Design by Numbers project at the Massachusetts Institute of Technology,4 which includes a simple language for graphical expression (called DBN), distributed freely over the Web, as a means to teach programming to visual artists and designers. This system is currently in use in more than 25 schools worldwide, with a sequel called the Proce55ing project that introduces two-dimensional and three-dimensional graphics concepts to visual designers.5 Individual tools available online include Alice,6 a three-dimensional (3D) interactive graphics programming environment for Windows built as a public-service project by a research group at Carnegie Mellon University. The intent was to make it easy for novices to develop interesting 3D environments and to explore the new medium of interactive 3D graphics. Alice is primarily a scripting and prototyping environment for 3D object behavior, not a 3D modeler, but it does read many common 3D file formats. By writing simple scripts, Alice users can control object appearance and behavior, and while the scripts are executing, objects respond to user input via mouse and keyboard. The current version of the Alice authoring tool is free to everyone and runs on computers that are commonly available for reasonable prices. Worlds created in Alice can be viewed and interacted with inside a standard Web browser once the Alice plug-in has been installed. The Alice core distribution includes a large library of textured models. The Alice plug-in does not allow authoring, but it does allow viewing of any Alice world. 4 Led by John Maeda at the MIT Media Lab. See <http://dbn.media.mit.edu>. 5 Proce55ing, an electronic sketchbook for developing ideas and a context for learning fundamentals of computer programming within the context of the electronic arts, was initiated by Ben Fry and Casey Reas at the MIT Media Lab and by the Interaction Design Institute Ivrea. As of November 2002, the software is in a pre-release stage, but bug fixes are being made in preparation for a more complete 1.0 release. Proce55ing will be free to download and use. See <http://proce55ing.net/>. 6 See <http://www.alice.org>. more obvious in IT, standards are common in arts and design arenas, from the dimensions of a violin to the Pantone color system. Standards can emerge from a standards-setting body or de facto from a leading supplier (as a consequence of competition within the marketplace). A good example for ITCP is digital images and associated software. Digital images are basically just arrays of pixels, yet they can be produced, stored, and manipulated in many ways. With standard representations for digital images, many programs and devices can interoperate, including cameras, digital editors, Web browsers, optical character reading (or other scanning) software, and graphical interface tools. But what should be standard? An image can exist in several forms—a set of high-level instructions for rendering, a set of polygons, or an array of pixels. A pixel representation may be a universal standard, but it forces the loss of high-level information. A high-level standard may be more complex to define, and to agree to, but it can
OCR for page 86
Beyond Productivity: Information Technology, Innovation, and Creativity preserve the ability to perform high-level operations. Another example is the musical instrument digital interface (MIDI) standard. MIDI was created to connect music keyboards to synthesizers, but it was soon realized that MIDI could allow computers to control synthesizers. The availability of low-cost synthesizers that are more or less interchangeable has been critical to any number of creative computer music systems. The rise of ITCP raises questions about the standards-setting process and the opportunities for artists and designers, as users, to have input. It is not uncommon for user groups to be on the sidelines of vendor efforts to set standards; artists and designers are not being singled out. And as recent attempts by non-profit organizations to explore how to inject “public interest” voices into IT standards-setting demonstrate, it is not easy for people with less technical depth, fewer financial resources, and different conversational norms to participate effectively in various IT standards-setting cultures.45 That said, a constructive step would be for specific communities of practice within the arts-design/ITCP world to try to determine for themselves a consensus view of the kinds of features or qualities they would like to see. A consensus statement would carry more weight with standards setters than would input from lone individuals or representatives of small or niche groups. As suggested above, research prototypes could aid in fostering consensus by demonstrating what a standard may lead to in practice. SELECTED AREAS FOR THE DEVELOPMENT OF HARDWARE AND SOFTWARE THAT WOULD PROMOTE CREATIVE WORK Precisely what the future holds is uncertain, but based on current expectations and trends,46 the capabilities available for work in ITCP will become far more powerful and diverse in the coming years. The timing of the present report is therefore propitious. Among notable research trends, nanotechnology is becoming a reality.47 Computers, 45 For example, the advocacy group, the Center for Democracy and Technology, has tried an experiment in the last couple of years to see how it might participate in Internet Engineering Task Force (IETF) standards-setting activities (John Morris, a lawyer, was the group’s participant). The Ford Foundation also fostered discussion of standards setting in its Digital Media Forum. Of course, since the early days of computing there have been various user groups that tried to influence the product designs of major vendors. 46 Raw computational power has been rising exponentially for years. A popular yardstick is Moore’s law: For the last 40 years, computing capability per dollar has doubled every 18 to 24 months, equivalent to a 100-fold improvement every 10 to 13 years, reflected in both rapidly increasing performance and declining price. Internet traffic has been growing by a factor of about two annually, outstripping the growth in computing speed. 47 Nanotechnologies are generally defined as having structures in the size range of 1 to 100 nanometers (billionths of a meter). Technologies of this size often exhibit novel properties. See, for example, information on the National Nanotechnology Initiative at <http://www.nano.gov/>. Research prototypes could aid in fostering consensus by demonstrating what a standard may lead to in practice.
OCR for page 87
Beyond Productivity: Information Technology, Innovation, and Creativity sensors, and other devices are becoming ever smaller. New sensors for sound and light combined with faster and cheaper digital signal processors will make large-scale system sensing increasingly practical.48 The synthesis of materials to meet defined requirements means that the manufacture of devices will be simplified. In the near future, a number of specific changes are envisioned. Computing speed will become so great (10 gigahertz in a handheld device) that operations as complex as real-time voice recognition without a learning curve could be commonplace. There is also likely to be enhanced machine intelligence and capability for seeing, hearing, and speaking.49 And many devices will likely be considered disposable. Thus, although some excellent tools for ITCP work are available, there are many opportunities for improvement. The committee identified a handful of areas that could exert considerable leverage in promoting ITCP, because such tools often enable the widespread adoption of new technologies. Distributed Control One area in need of support is distributed control. Software that coordinates multiple computers is among the most difficult of programs to write. This is true even for commercial business systems, but the problem is more acute in creative contexts because programming resources tend to be much more limited there. Tools can provide simplified protocols for communication between networked computers, including wireless devices. The essentially sequential nature of software systems has made it difficult to envision a plausible distributed control paradigm. Much research in the related area of parallel computation was curtailed in the 1990s due to the evolution of the conventional microprocessor from the megahertz to gigahertz speed range in only a decade—there was no need to make computer programs run faster when the computer itself had become faster in a matter of months. Distributed control has taken center stage once again, not as a potential means to increase the speed of computation but as a way to handle the difficult task of coordinating computation across multiple computing units. There is a simple impediment to advancing the state of the art in this area; namely, programming methodologies for asynchronous computing are not in place, even in high-end systems. Software system designs inherit a legacy similar to that of the automobile industry in that only incremental, evolutionary changes are adopted instead of the kind of revolutionary advancements needed to write more sophis 48 From Computer Science and Telecommunications Board, National Research Council, 2001, Embedded, Everywhere: A Research Agenda for Networked Systems of Embedded Computers, National Academy Press, Washington, D.C. 49 See, for example, John Markoff, 2002, “Technology Gives Sight to Machines, Inexpensively,” New York Times, June 17, available online at <http://www.nytimes.com/2002/06/17/technology/17VISI.html>.
OCR for page 88
Beyond Productivity: Information Technology, Innovation, and Creativity ticated programs that take advantage of multiprocessor and multisystem interactions. Sensors and Actuators Another area deserving attention is sensors and actuators. Artists and designers and scientists alike often want to collect data in the real world and to control mechanical devices using computers. Advanced sensing technologies and actuators exist, but without tools to simplify their use, these technologies remain out of reach of most individuals who are not specialists. Sensors and actuators that can be used by non-experts can promote creative endeavors (see, for example, Box 3.8). Note that the area of motion control and robotics is experiencing a renaissance in computer science: Advances in component technologies and artificial intelligence moved the field along considerably in the 1990s, and research in the area is expected to thrive. A subtle reason for recent advancements in this area can be associated with the development of the Web. Once-difficult-to-obtain catalogs or obscure parts listings that were available only to hacker communities are now easily found on the Web with a quick search. Interfacing with these parts was once also difficult, requiring access to the relevant technical community. Now most of this knowledge is published and easily accessed on the Web. Interfacing information, sample projects, and even advanced research can easily be found by the curious builder. In addition to the challenge of using sensor and actuator technologies, there is the problem of interfacing them to computers, especially if they are non-standard. This situation might be alleviated by the design and distribution of software and hardware that take advantage of serial interfaces such as the universal serial bus (USB) and the IEEE-1394 multimedia standard.50 There have been several attempts at developing general-purpose sensor systems and input/output boxes in the arts, such as the iCube system and notably, Michael Rodemer’s EZIO board, designed at the Art and Technology Department of the Art Institute of Chicago in the mid-1990s and now commercially available. The success of the Parallax Inc. Basic Stamp microcontroller as a low-cost platform for educational experimentation has changed the public’s access to embedded computing. Ample documentation, complete technical information, and well-documented sample projects have 50 One interesting effort is the development of “phidgets,” or physical widgets, at the University of Calgary. Phidgets are building blocks that assist in the development of physical user interfaces. Phidgets consist of a hardware device, software architecture for communication and connection management, a well-defined software application programming interface (API) for device programming, a simulation capability, and an optional component for interacting with the device, as described in “An Overview of Phidgets,” available online at <http://www.cpsc.ucalgary.ca/grouplab/phidgets/>.
OCR for page 89
Beyond Productivity: Information Technology, Innovation, and Creativity made this the platform of choice for many budding “physical computing” courses nationwide. Once one is versed in the Basic Stamp microcontroller, it is easy to move to more economical systems by going one level deeper into the inner workings of the Basic Stamp to its central controller, the “PIC” chip. A wide variety of PIC chips, and competitor single-chip computers such as AVR Inc.’s line, have made it possible for anyone to easily build a sub-$10 computer for embedded control. Any art education program should find it attractive that platforms priced at less than $10 can be budgeted for as easily as any other art supply like paint, canvas, or a brush. With respect to interfacing the various devices that are emerging today, one challenge is the speed and reliability of data transfer. Recently buses such as the CAN bus have emerged as high-speed standards for the robotics community, and the consumer-class USB and IEEE-1394 technologies are continually giving rise to faster versions like USB-2 and Firewire-2.51 Yet at some point the rapid progress in these standards can lead to confusion in the general population. What economists call the cost of switching—having to relearn a technology that one had finally felt comfortable with but then learned had suddenly become passé—can deter or slow adoption of technology, as well as influence standards-setting tactics. Video and Audio Extensive editing and processing tools exist for time-based media (including video and audio), but these tools assume a particular style of working—namely, that the product will be a linear video or audio recording. In addition, most tools do not support real-time processing. These limitations hamper artists who want to work with video and audio media in interactive performances, take other approaches to interactivity, stream material over the Internet, or use scripts to process or generate video. Standards continually emerge and compete in this arena such as Apple QuickTime, Macromedia Flash, and Real Video. A common difficulty for all of these standards is to maintain compatibility across the multiple platforms they support. Often the same version of any system, such as Flash, will display a file properly on a PC but not on a Macintosh. These inconsistencies are rarely addressed, and content creators must simply work around these known problems. The standards are constantly upgraded, solving some problems yet introducing others. For an emerging field, like digital media or ITCP generally, standards that lack reliability are a serious problem. It is as if the art of film were emerging in an era when the film you might have shown just once might suddenly dissolve before your very eyes. 51 Buses enable data transfer among the different components of a computer.
OCR for page 90
Beyond Productivity: Information Technology, Innovation, and Creativity BOX 3.8 Ghostcatching The fruit of a collaboration between multimedia artists Paul Kaiser and Shelley Eshkar and dancer/ choreographer Bill T. Jones, Ghostcatching1 is a virtual dance performance that makes use of motion-capture technology. Envisioning a blend of performance, filmmaking, drawing, and computer composition, the artists used light-sensitive sensors attached to 22 key points of Jones’s body and eight cameras to capture his movements. Roughly 40 sequences of Jones’s movement were recorded digitally. The resulting “capture data”—which represented a record of only the movements of the sensors and not of Jones’s body per se— were then used as the raw information for Kaiser and Eshkar’s creative application. See Figure 3.8.1. Jones’s “movements were then manipulated electronically and re-choreographed on a computer screen to make an original virtual performance.”2 The results include an 8-minute digital projection, 13 still images taken from the dance, photographs that describe the artists’ process, and a soundtrack consisting of Jones’s own sounds: “chanting, humming, singing, talking, grunting, and more.”3 Commissioned by the Cooper Union for the Advancement of Science and Art,4 the installation opened at the Cooper Union’s Arthur A. Houghton Jr. Gallery in New York City and ran from January 6 to February 13, 1999. 1. Improvising 2. Wearing motion-capture markers 1 For more information, see <http://www.cooper.edu/art/ghostcatching/ghost/exhibit.html>. 2 Zoe Ingalls, 1999, “Using New Technology to Create ‘Virtual Dance’,” Chronicle of Higher Education 45(21): A29-A30. 3 Ingalls, 1999, “Using New Technology to Create ‘Virtual Dance’.”
OCR for page 91
Beyond Productivity: Information Technology, Innovation, and Creativity 3. Markers optically recorded and converted to digital three-dimensional files 4. Motion files applied to kinematic model of the body 5. “Hand-drawn” lines modeled as mathematical curves 6. Sampled charcoaled strokes applied and rendered as a final drawn body FIGURE 3.8.1 Ghostcatching. Images courtesy of Bill T. Jones, Paul Kaiser, and Shelley Eshkar; available for download at <ftp://ftp.eshkar.com/ghostcatching/>.
OCR for page 92
Beyond Productivity: Information Technology, Innovation, and Creativity Generative Processes From both a computational and an artistic and design standpoint, an attractive feature of computers is that they enable generative processes. Rather then creating each note or brush stroke by hand, an artist or designer can design a process that generates material automatically. Of particular interest are fractals, chaotic systems, and systems exhibiting non-linear dynamics. These have strong connections to natural processes, can be explored only by using computers, and often create interesting and unpredicted output. A programmer could set up a generative process relatively easily, but tools are needed that can encapsulate these processes to make them more accessible to artists and designers. Generative processes are interesting to contemplate for a few reasons. First, the associated computer models depend on assumptions—and to ensure that the models do not present the kinds of constraints criticized above, it will be important for artists and designers to have some understanding of the models’ inner workings, which in turn depends on comfort with quantitative analyses. Second, some of the earliest explorations of ITCP were generative processes. A notable instance is Harold Cohen’s work with the artificial intelligence drawing-generating program, Aaron.52 More contemporary simulation models, some of which explore “artificial life,” have been used by biologists and other scientists to understand evolution, yielding a new understanding of its pacing and realization. The Santa Fe Institute led the development and use of these models, providing training in modeling methods, and it also has reached out to the arts community. The rise of computer games that feature artificial worlds suggests but one of the more obvious potential ITCP connections. Artificial-world development is also being explored outside the game context.53 When computing was young, the most natural form of creative activity on the computer was programming. This led to some landmark work by researchers at Bell Labs, such as Ken Knowlton, who were versed in programming but in addition had a penchant for visual art. The idea that forms can emerge from programming echoes complex processes in the physical world where the result is something you cannot expect. More recently with the rise and proliferation of digital tools, programmatic approaches to form have become less popular. For ITCP to advance, a fundamental shift is needed from computation or mathematics based on numbers to that based on symbols. 52 See <http://www.umcs.maine.edu/~larry/latour/aaron.html>. 53 See the World Generator/The Engine of Desire by Bill Seaman with Gideon May. The work enables the construction of virtual worlds in real time in an interactive environment. Although the work is artistic, it has been featured in a number of IT studies and has potential for many design and interactive tool applications. For further information, see <http://faculty.risd.edu/faculty/bseamanweb/web/texts.html> and <http://www.nada.kth.se/erena/pdf/D1_2.pdf>.
OCR for page 93
Beyond Productivity: Information Technology, Innovation, and Creativity Reliable, Low-latency Communication over the Internet Artists and designers have common cause with a wide range of users in their interest in reliable, low-latency communication over the Internet.54 Higher bandwidth and/or quality-of-service guarantees could enable new levels of interaction, distributed concerts, two-way video, and other creative activities that necessitate specific levels of network performance. As some observed to the committee, were artists able to stream their own bandwidth-demanding work out, it would yield greater diversity of voices and stimulation to creativity. This is the crux of the broadband dilemma: Broadband users are twice as likely as dial-up users to have their own Web sites, with nearly 60 percent becoming creators and managers of content in this manner, according to a report by the Pew Internet & American Life Project.55 But pervasive broadband deployment in the United States will require billions of dollars in investment;56 subscriber levels, while growing, fall far short of the potential, given existing deployment, and therefore widespread use is not expected soon. That said, in the academic world, the Internet2 project and complementary federal research infrastructure programs provide high-bandwidth service on and between many campuses because of its potential to support cutting-edge application research. Music performance over the Canadian equivalent, the CANARIE network, illustrates the general proposition that these facilities could be valuable ITCP vehicles. Recent advances in low-latency ultra-videoconferencing have been carried out by musicianresearchers at McGill and Stanford Universities.57 As in other communities, opinions in the arts world about the value of broadband differ. To some, a benefit of ever-increasing processor speeds has been the development of media-rich simulations and other works, stimulating demand for broadband services. Yet others find such digital spectacles less compelling than new social models for peer-to-peer cultural production. From this standpoint, advanced content should be focused not on creating volumes of traffic 54 The Internet is based on best-effort policies that make no guarantees for the timely delivery of information. Reliability is achieved by re-transmitting data when a loss is detected, but this recovery introduces delays. See Computer Science and Telecommunications Board, National Research Council, 2001, The Internet’s Coming of Age, National Academy Press, Washington, D.C. 55 Mike Snider, 2002, “Faster Connection Allows Users to Do More,” USA Today, June 23, available online at <http://www.usatoday.com/life/cyber/tech/2002/06/24/broadband.htm>. See also The Broadband Difference: How Online Americans’ Behavior Changes with High-speed Internet Connections at Home, available online <http://www.pewinternet.org/reports/toc.asp?Report=63>. 56 See Computer Science and Telecommunications Board, 2001, The Internet’s Coming of Age, p. 45. 57 Jeremy Cooperstock, an engineer at McGill University, and Chris Chafe, a composer at Stanford University; see <http://www.cim.mcgill.ca/~jer/research/rtnm/>. Internet2 and CANARIE are discussed further in Chapters 5 and 8.
OCR for page 94
Beyond Productivity: Information Technology, Innovation, and Creativity that can fill up fat data pipes, but on infrastructural advances that would allow a wide public access to and control over digital objects, although with the recognition that there are certainly legitimate and productive needs for greater capacity.58 This perspective shifts the focus to thinking of how to increase the range of interpersonal connection and personal expression distributed across society. One issue relating to broadband is that the quantitative increase in bit traffic makes possible qualitative change in digital expression. However, this potential is confounded by a tendency for capabilities to fluctuate with IT innovations: After the initial emergence of digital media serving smaller amounts of kilobytes, higher amounts of data could be conveyed through the CD-ROM media, only to go back to smaller amounts of kilobytes over the (narrowband or dial-up) Web, and then up to large capacity again via DVD-ROM and now to broadband. Looking across this series of innovations, they have not been associated with an intrinsic change in content for the better—perhaps because of uncertainty associated with the fluctuations, or perhaps because experience with any one technology has been too abbreviated. Tool Design and Human-Computer Interaction There is a great deal of work to be done in understanding the ramifications of the tradeoffs that occur every day in the design of tools. The discussion above includes examples of how the tool designers’ choices in areas such as abstraction, extensibility, and metaphor can affect the practices within a field and the results obtained through the use of the tool. Insufficient work has gone into understanding how to leverage this relationship to better support creative practices. This goes far beyond the usability studies once typically associated with human-computer interaction research. Questions to be addressed include these: When does embodying domain knowledge in a tool help? What is the impact of supporting multiple metaphors in one tool? When are constraints good inspirations, and when do they cause excessive perspiration? When can implicit assumptions help, and when can they be hindrances? How much and what kind of openness yields opportunity, and what yields frustration? In other words, what is the relationship among information, its representation, presentation, and manipulation, and the range of work encouraged? Researchers and developers have produced a small number of data points—more work in understanding this area can have deep ramifications for encouraging ITCP. 58 A Canadian task force concluded a series of consultations with the arts and cultural community with a report entitled Filling the Pipe: Stimulating Canada’s Broadband Content Industry Through R&D, available online at <http://www.canarie.ca/press/publications/broadband_report.pdf>. A quantitative increase in bit traffic makes possible qualitative change in digital expression.
OCR for page 95
Beyond Productivity: Information Technology, Innovation, and Creativity Programming Languages Finally, some areas of great interest to artists are not well served by programming languages, which tend to be driven by science, engineering, the Web, and more commercial application areas. Some of the topics discussed above, especially distributed control, video and audio processing, and generative processes, could be supported better by languages. Languages are particularly important for specifying interactive behavior and generative processes, an area not well supported by commercial software. Because a great deal of creative activity involves combining existing concepts in new ways, and because programming languages provide the glue for assembling software tools and libraries into applications, languages are critical to innovation. Progress has been achieved in making programming languages simpler for novice programmers, but this work needs to be adapted to support more creative work.
Representative terms from entire chapter: