What is Cinema?

Digital Cinema and the History of a Moving Image

by Lev Manovich

It is useful to think about the relations between cinema and new media in terms of two main vectors. The first vector goes from cinema to new media, and it constitutes the backbone of this book. Chapters I - V used history and theory of cinema to map out the logic which drives the technical and stylistic development of new media. I also traced the key role which cinematic language is placing in new media interfaces - both traditional HCI (interface of the operating system and software applications) and what I called "cultural interfaces" - the interfaces between the human user and cultural data.  

The second vector goes in the opposite direction: from computers to cinema. How does computerization affects our very concept of a moving images? Does if offer new possibilities for film language? Did it led to the development of totally new forms of cinema? This last chapter is devoted to these questions. In part I already started dealing them in "Compositing" section and in "Illusion" chapter. Since the main part of this chapter focused on the new identity of a still computer generated image, it is logical that we now extend our inquiry to include moving images.

Before proceeding I would like to offer two lists. My first list of the summary of how (at the time of writing - 1999) I think about the effects of computerization on cinema proper:

1.    Use of computer techniques in traditional filmmaking:

1. 1.1. 3D computer animation / digital composing. Example:

1."Titanic" (James Cameron, 1997); ""The City of Lost Children" (Marc Caro and J.P. Jeunet, 1995).

        1.2. Digital painting. Example: "Forest Gump" (Robert Zemeckis, 1994).

        1.3. Virtual Sets. Example: "Ada" (Lynn Hershman,1997).

        1.4. Virtual Actors / Motion capture. Example: "Titanic."

2. New forms of computer-based cinema

2.1. Motion rides / location-based entertainment. Example: rides produced by

Douglas Trumball.

        2.2. "Typographic cinema": film + graphic design + typography. Examples: film title sequences.

        2.3. Net.cinema: films designed exclusively for Internet distribution. Example: New Venue, one of the first onlines sites devoted to showcasing short digital films. In 1998 it accepted only QuickTime files under 5 MG.

        2.4. Hypermedia interfaces to a film which allows non-linear access at different scales. Examples: "WaxWeb" (David Blair, 1994-1999); Stephen Mamber's database interface to Hitchock's "Psycho" (Mamber, 1996-).

2.5. Interactive movies and games which are structured around film-like sequences. These sequences can be created using traditional film techniques (example: "Jonny Mnemonic" game) or computer animation (example: "Blade Runner" game). (The pioneer of interactive cinema is experimental filmmaker Graham Weinbren whose laserdisks Sonata and The Erl King are the true classics of this new form.) Note that it is hard to draw a strict line between such interactive movies and many other games which may not use traditional film sequences yet follow many other conventions of film language in their structure. From this perspective, the majority of 1990s computer games can be actually considered interactive movies.

2.6. Animated, filmed, simulated or hybrid sequences which follow film language, and appear in HCI, Web sites, computer games and other areas of new media. Examples: transitions and QuickTime movies in Myst; FMV (full motion video) opening in Tomb Rider and many other games.

The first section of this chapter, "Digital Cinema and the History of a Moving Image," will focus on 1.1 - 1.3. The second section, "New Language of Cinema," will use examples drawn from 2.3 - 2.6.^[i][i]

Note that this list does not include such new production technologies as DV (digital video) or new distribution technologies such as digital film projection or network film distribution which by 1999 was already used in Hollywood on a experimental basis; nor do I mention growing number of Web sites devoted to distribution of films.^[ii][ii] Although all these developments will undoubtedly have important effect on the economics of film production and distribution, they do not appear to have a direct effect of film language, which is my main concern here.

My second, and a  highly tentative list, summarizes some of the distinct qualities of a computer-based image. This list pulls together arguments presented throughout the book so far. As I already noted in Chapter 1, I feel that it is important to pay attention not only to the new properties of a computer image which can be logically deduced from its new "material" status, but also to how images are actually used in computer culture. Therefore the number of properties on this list reflect the typical usage of images, rather some "essential" properties it may have because of its digital status. It is also legitimate to think of some of these qualities as particular consequences of the oppositions which define a concept of representation, summarized in the Introduction:

1.    Computer-based image is discrete, since it is broken into pixels. This makes it more like a human language (but not in the semiotic sense of having distinct units of meaning).

2.    Computer-based image is modular, since it typically consists from a number of layers whose contents often correspond to meaningful parts of the image.

3.    Computer-based image consists from two levels, a surface appearance and the underlying code (which may be the pixel values, a mathematical function or HTML code). In terms of its "surface," an image participates in the dialog with other cultural objects. In terms of its code, an image exist on the same conceptual plane as other computer objects. (The surface-code pain can be related to signifier - signified, base - superstructure, unconscious - conscious pairs. So, just as a signifier exists in a structure with other signifiers of a language, a "surface" of an image, i.e. its "contents" enters in dialog with all other images in a culture.)   

4.    Computer-based images are typically compressed using lossy compression techniques, such as JPEG. Therefore, presence of noise (in a sense of undesirable artifacts and loss of original information) is its essential, rather than accidental, quality.

5.    An image acquires the new role of an interface (for instance, imagemaps on the Web, or the image of a desktop as a whole in GUI ). Thus image becomes image-interface. In this role it functions as a portal into another world, like an icon in Middle Ages or a mirror in modern literature and cinema. Rather than staying on its surface, we expect to go "into" the image. In effect, every computer user becomes Carrol's Alice. Image can function as an interface because it can be "wired" to programming code; thus clicking on the image activates a computer program (or its part).

6.    The new role of an image as image-interface competes with is older role as representation. Therefore, conceptually, a computer image is situated between two opposing poles: an illusionistic window into a fictional universe and a tool for computer control. The task of new media design and art is learn how to combine these two competing roles of an image.

7.    Visually, this conceptual opposition translates into the opposition between a depth and surface, between a window into a fictional universe and a control panel.

8.    Along with functioning as image-interfaces, computer images also functions as image-instruments. If image-interface controls a computer, an image-instrument allows the user to remotely affect physical reality in real time. This ability not just to act but to "teleact" distinguishes new computer-based image-instrument from old image-instruments. Additionally, if before image-instruments such as maps were clearly distinguished from illusionistic images, such as paintings (although recall Alpers's argument that classical Dutch painting combines both concepts), computer images often combine both functions.

9.    A computer image is frequently hyperlinked to other images, texts, and other media elements. Rather than being a self-enclosed entity it points, leads to, directs the user outside of itself towards something else. A moving image may also include hyperlinks (for instance, in QuickTime format.) We can say that a hyperlinked image, and hypermedia in general, "externalizes" Pierce's idea of infinite semiosis and Derrida's concept of infinite deferral of meaning - although this does not mean that this "externalization" automatically legitimizes these concepts. Rather than celebrating "the convergence of technology and critical theory," we should use new media technology as an opportunity to question our accepted critical concepts and models.

10.           Variability and automation, these general principles of new media, also apply to images. For example, using a computer program a designer can automatically generate infinite versions of the same image which can vary in size, resolution, colors, composition and so on.  

11.           From a single image which represented the "cultural unit" of a previous period we move to a database of images. Thus if the hero of Antonioni's Blow-up (1966) was looking for truth within a single photographic image, the equivalent of this operation in a computer age is to work with a whole database of many images, searching and comparing them with each other. (Although many contemporary films include scenes of image search, none of them makes it a subject of a film the way Blow-up focuses on zooming into a photograph. From this perspective, it is interesting that fifteen years later Blade Runner still applies "old" cinematic logic in relation to a computer-based image. In a well-known scene the hero uses voice commands to direct a futuristic computer device to pan and zoom into an image. In reality already since the 1950s military used different computer techniques for image analysis to automatically identify objects represented in an image, detect changes in images over time, etc. which relied on databases of images.^[iii][iii]) Any unique image you may desire probably already exists on the Internet or in some database. As I already noted, today the problem is no longer how to create the right image, but how to find already existing one.  

Since a computer-based moving image, just as its analog predecessor, is simply a sequence of still images, all these properties apply to it as well. To delineate the new qualities of a computer-based still image I compared it with other types of modern images commonly used before it - drawing, a map, a painting and most importantly, a still photograph. It would be logical to begin discussion of the computer-based moving image by also relating it to two most common types of moving images it replaces in its turn - the film image and an animated image. The first section, "Digital Cinema and the History of a Moving Image" does precisely this. It asks how the shift to computer-based representation and production processes redefines the identity of a moving image and the relationship between cinema and animation. This section also invokes the question of computer-based illusionism, considering it in relation to animation, analog cinema and digital cinema. The following section "The New Language of Cinema" presents the examples of some of the new directions for film language - or, more generally, the language of moving images - opened up by computerization. My examples come from different areas where computer-based moving image are used: digital films, net.films, self-contained hypermedia, and Web sites.

 

Cinema, the Art of the Index

Most discussions of cinema in the computer age have focused on the possibilities of interactive narrative. It is not hard to understand why: since the majority of viewers and critics equate cinema with storytelling, computer media is understood as something which will let cinema tell its stories in a new way. Yet as exciting as the ideas of a viewer participating in a story, choosing different paths through the narrative space and interacting with characters may be, they only address one aspect of cinema which is neither unique nor, as many will argue, essential to it: narrative.  

        The challenge which computer media poses to cinema extends far beyond the issue of narrative. Computer media redefines the very identity of cinema. In a symposium which took place in Hollywood in the Spring of 1996, one of the participants provocatively referred to movies as "flatties" and to human actors as "organics" and "soft fuzzies."^[iv][iv] As these terms accurately suggest, what used to be cinema's defining characteristics have become just the default options, with many others available. When one can "enter" a virtual three-dimensional space, to view flat images projected on the screen is hardly the only option. When, given enough time and money, almost everything can be simulated in a computer, to film physical reality is just one possibility.

        This "crisis" of cinema's identity also affects the terms and the categories used to theorize cinema's past. French film theorist Christian Metz wrote in the 1970s that "Most films shot today, good or bad, original or not, 'commercial' or not, have as a common characteristic that they tell a story; in this measure they all belong to one and the same genre, which is, rather, a sort of 'super-genre' ['sur-genre']."^[v][v] In identifying fictional films as a "super-genre' of twentieth century cinema, Metz did not bother to mention another characteristic of this genre because at that time it was too obvious: fictional films are live action films, i.e. they largely consist of unmodified photographic recordings of real events which took place in real physical space. Today, in the age of photorealistic 3D computer animation and digital compositing, invoking this characteristic becomes crucial in defining the specificity of twentieth century cinema. From the perspective of a future historian of visual culture, the differences between classical Hollywood films, European art films and avant-garde films (apart from abstract ones) may appear less significance than this common feature: that they  relied on lens-based recordings of reality. This section is concerned with the effect of computerization on cinema as defined by its "super genre" as fictional live action film.^[vi][vi]

        During cinema's history, a whole repertoire of techniques (lighting, art direction, the use of different film stocks and lens, etc.) was developed to modify the basic record obtained by a film apparatus. And yet behind even the most stylized cinematic images we can discern the bluntness, the sterility, the banality of early nineteenth century photographs. No matter how complex its stylistic innovations, the cinema has found its base in these deposits of reality, these samples obtained by a methodical and prosaic process. Cinema emerged out of the same impulse which engendered naturalism, court stenography and wax museums. Cinema is the art of the index; it is an attempt to make art out of a footprint.

        Even for director Andrey Tarkovsky, film-painter par excellence, cinema's identity lay in its ability to record reality. Once, during a public discussion in Moscow sometime in the 1970s he was asked the question as to whether he was interested in making abstract films. He replied that there can be no such thing. Cinema's most basic gesture is to open the shutter and to start the film rolling, recording whatever happens to be in front of the lens. For Tarkovsky, an abstract cinema is thus impossible.

        But what happens to cinema's indexical identity if it is now possible to generate photorealistic scenes entirely in a computer using 3D computer animation; to modify individual frames or whole scenes with the help a digital paint program; to cut, bend, stretch and stitch digitized film images into something which has perfect photographic credibility, although it was never actually filmed?     

        This section will address the meaning of these changes in the filmmaking process from the point of view of the larger cultural history of the moving image.  Seen in this context, the manual construction of images in digital cinema represents a return to nineteenth century pre-cinematic practices, when images were hand-painted and hand-animated. At the turn of the twentieth century, cinema was to delegate these manual techniques to animation and define itself as a recording medium. As cinema enters the digital age, these techniques are again becoming the commonplace in the filmmaking process. Consequently, cinema can no longer be clearly distinguished from animation. It is no longer an indexical media technology but, rather, a sub-genre of painting.  

        This argument will be developed in two stages.  I will first follow a historical trajectory from nineteenth century techniques for creating moving images to twentieth-century cinema and animation. Next I will arrive at a definition of digital cinema by abstracting the common features and interface metaphors of a variety of computer software and hardware which are currently replacing traditional film technology. Seen together, these features and metaphors suggest a distinct logic of a digital moving image. This logic subordinates the photographic and the cinematic to the painterly and the graphic, destroying cinema's identity as a media art. In the beginning of the next section "New Language of Cinema" I will examine different production contexts which already use digital moving images - Hollywood films, music videos, CD-ROM-based games and other stand-alone hypermedia - in order to see if and how this logic has begun to manifest itself.               

A Brief Archeology of Moving Pictures

As testified by its original names (kinetoscope, cinematograph, moving pictures), cinema was understood, from its birth, as the art of motion, the art which finally succeeded in creating a convincing illusion of dynamic reality. If we approach cinema in this way (rather than the art of audio-visual narrative, or the art of a projected image, or the art of collective spectatorship, etc.), we can see it superseding previous techniques for creating and displaying moving images.

        These earlier techniques shared a number of common characteristics. First, they all relied on hand-painted or hand-drawn images. The magic lantern slides were painted at least until the 1850s; so were the images used in the Phenakistiscope, the Thaumatrope, the Zootrope, the Praxinoscope, the Choreutoscope and numerous other nineteenth century pro-cinematic devices. Even Muybridge's celebrated Zoopraxiscope lectures of the 1880s featured not actual photographs but colored drawings painted after the photographs.^[vii][vii]  

        Not only were the images created manually, they were also manually animated. In Robertson's Phantasmagoria, which premiered in 1799, magic lantern operators moved behind the screen in order to make projected images appear to advance and withdraw.^[viii][viii] More often, an exhibitor used only his hands, rather than his whole body, to put the images into motion. One animation technique involved using mechanical slides consisting of a number of layers. An exhibitor would slide the layers to animate the image.^[ix][ix] Another technique was to slowly move a long slide containing separate images in front of a magic lantern lens. Nineteenth century optical toys enjoyed in private homes also required manual action to create movement - twirling the strings of the Thaumatrope, rotating the Zootrope's cylinder, turning the Viviscope's handle.   

        It was not until the last decade of the nineteenth century that the automatic generation of images and their automatic projection were finally combined. A mechanical eye became coupled with a mechanical heart; photography met the motor. As a result, cinema - a very particular regime of the visible - was born. Irregularity, non-uniformity, the accident and other traces of the human body, which previously inevitably accompanied moving image exhibitions, were replaced by the uniformity of machine vision.^[x][x] A machine, which like a conveyer belt, was now spitting out images, all sharing the same appearance, all the same size, all moving at the same speed, like a line of marching soldiers.

        Cinema also eliminated the discrete character of both space and movement in moving images. Before cinema, the moving element was visually separated from the static background as with a mechanical slide show or Reynaud's Praxinoscope Theater (1892).^[xi][xi] The movement itself was limited in range and affected only a clearly defined figure rather than the whole image. Thus, typical actions would include a bouncing ball, a raised hand or eyes, a butterfly moving back and forth over the heads of fascinated children - simple vectors charted across still fields.   

        Cinema's most immediate predecessors share something else. As the nineteenth-century obsession with movement intensified, devices which could animate more than just a few images became increasingly popular. All of them - the Zootrope, the Phonoscope, the Tachyscope, the Kinetoscope - were based on loops, sequences of images featuring complete actions which can be played repeatedly. The Thaumatrope (1825), in which a disk with two different images painted on each face was rapidly rotated by twirling a strings attached to it, was in its essence a loop in its most minimal form: two elements replacing one another in succession. In the Zootrope (1867)  and its numerous variations, approximately a dozen images were arranged around the perimeter of a circle.^[xii][xii] The Mutoscope, popular in America throughout the 1890s, increased the duration of the loop by placing a larger number of images radially on an axle.^[xiii][xiii] Even Edison's Kinetoscope (1892-1896), the first modern cinematic machine to employ film, continued to arrange images in a loop.^[xiv][xiv] 50 feet of film translated to an approximately 20 second long presentation - a genre whose potential development was cut short when cinema adopted a much longer narrative form.   

From Animation to Cinema

Once the cinema was stabilized as a technology, it cut all references to its origins in artifice. Everything which characterized moving pictures before the twentieth century - the manual construction of images, loop actions, the discrete nature of space and movement - all of this was delegated to cinema's bastard relative, its supplement, its shadow - animation. Twentieth century animation became a depository for nineteenth century moving image techniques left behind by cinema.

        The opposition between the styles of animation and cinema defined the culture of the moving image in the twentieth century. Animation foregrounds its artificial character, openly admitting that its images are mere representations. Its visual language is more aligned to the graphic than to the photographic. It is discrete and self-consciously discontinuous: crudely rendered characters moving against a stationary and detailed background; sparsely and irregularly sampled motion (in contrast to the uniform sampling of motion by a film camera - recall Jean-Luc Godard's definition of cinema as "truth 24 frames per second"), and finally space constructed from separate image layers.

        In contrast, cinema works hard to erase any traces of its own production process, including any indication that the images which we see could have been constructed rather than recorded. It denies that the reality it shows often does not exist outside of the film image, the image which was arrived at by photographing an already impossible space, itself put together with the use of models, mirrors, and matte paintings, and which was then combined with other images through optical printing. It pretends to be a simple recording of an already existing reality - both to a viewer and to itself.^[xv][xv] Cinema's public image stressed the aura of reality "captured" on film, thus implying that cinema was about photographing what existed before the camera, rather than "creating the 'never-was'" of special effects.^[xvi][xvi] Rear projection and blue screen photography, matte paintings and glass shots, mirrors and miniatures, push development, optical effects and other techniques which allowed filmmakers to construct and alter the moving images, and thus could reveal that cinema was not really different from animation, were pushed to cinema's periphery by its practitioners, historians and critics.^[xvii][xvii]  

        In the 1990s, with the shift to computer media, these marginalized techniques moved to the center.

Cinema Redefined

A visible sign of this shift is the new role which computer generated special effects have come to play in Hollywood industry in the 1990s. Many blockbusters have been driven by special effects; feeding on their popularity. Hollywood has even created a new-mini genre of "The Making of..." videos and books which reveal how special effects are created.

        I will use special effects from 1990s Hollywood films for illustrations of some of the possibilities of digital filmmaking. Until recently, Hollywood studios were the only ones who had the money to pay for digital tools and for the labor involved in producing digital effects. However, the shift to digital media affects not just Hollywood, but filmmaking as a whole. As traditional film technology is universally being replaced by digital technology, the logic of  the filmmaking process is being redefined. What I describe below are the new principles of digital filmmaking which are equally valid for individual or collective film productions, regardless of whether they are using the most expensive professional hardware and software or its amateur equivalents.      

        Consider, then, the following principles of digital filmmaking:

1.    Rather than filming physical reality it is now possible to generate film-like scenes directly in a computer with the help of 3D computer animation. Therefore, live action footage is displaced from its role as the only possible material from which the finished film is constructed.    

2.    Once live action footage is digitized (or directly recorded in a digital format), it loses its privileged indexical relationship to pro-filmic reality. The computer does not distinguish between an image obtained through the photographic lens, an image created in a paint program or an image synthesized in a 3D graphics package, since they are made from the same material - pixels. And pixels, regardless of their origin, can be easily altered, substituted one for another, and so on. Live action footage is reduced to be just another graphic, no different than images which were created manually.^{[xviii][xviii]}  

3.    If live action footage was left intact in traditional filmmaking, now it functions as raw material for further compositing, animating and morphing. As a result, while retaining visual realism unique to the photographic process, film obtains the plasticity which was previously only possible in painting or animation. To use the suggestive title of a popular morphing software, digital filmmakers work with "elastic reality." For example, the opening shot of Forest Gump (Robert Zemeckis, Paramount Pictures, 1994; special effects by Industrial Light and Magic) tracks an unusually long and extremely intricate flight of a feather. To create the shot, the real feather was filmed against a blue background in different positions; this material was then animated and composited against shots of a landscape.^[xix][xix] The result: a new kind of realism, which can be described as "something which looks is intended to look exactly as if it could have happened, although it really could not."   

4.    Previously, editing and special effects were strictly separate activities. An editor worked on ordering sequences of images together; any intervention within an image was handled by special effects specialists. The computer collapses this distinction. The manipulation of individual images via a paint program or algorithmic image processing becomes as easy as arranging sequences of images in time. Both simply involve "cut and paste." As this basic computer command exemplifies, modification of digital images (or other digitized data) is not sensitive to distinctions of time and space or of differences of scale. So, re-ordering sequences of images in time, compositing them together in space, modifying parts of an individual image, and changing individual pixels become the same operation, conceptually and practically.  

Given the preceding principles, we can define digital film in this way:    

digital film = live action material + painting + image processing +   

compositing + 2D computer animation + 3D computer animation

Live action material can either be recorded on film or video or directly in a digital format.^[xx][xx] Painting, image processing and computer animation refer to the processes of modifying already existent images as well as creating new ones. In fact, the very distinction between creation and modification, so clear in film-based media (shooting versus darkroom processes in photography, production versus post-production in cinema) no longer applies to digital cinema, since each image, regardless of its origin, goes through a number of programs before making it to the final film.^[xxi][xxi]  

        Let us summarize these principles. Live action footage is now only raw material to be manipulated by hand: animated, combined with 3D computer generated scenes and painted over. The final images are constructed manually from different elements; and all the elements are either created entirely from scratch or modified by hand. Now we can finally answer the question "what is digital cinema?" Digital cinema is a particular case of animation which uses live action footage as one of its many elements.  

        This can be re-read in view of the history of the moving image sketched earlier. Manual construction and animation of images gave birth to cinema and slipped into the margins...only to re-appear as the foundation of digital cinema. The history of the moving image thus makes a full circle. Born from animation, cinema pushed animation to its boundary, only to become one particular case of animation in the end.

        The relationship between "normal" filmmaking and special effects is similarly reversed. Special effects, which involved human intervention into machine recorded footage and which were therefore delegated to cinema's periphery throughout its history, become the norm of digital filmmaking.

        The same logic applies for the relationship between production and post-production. Cinema traditionally involved arranging physical reality to be filmed though the use of sets, models, art direction, cinematography, etc. Occasional manipulation of recorded film (for instance, through optical printing) was negligible compared to the extensive manipulation of reality in front of a camera. In digital filmmaking, shot footage is no longer the final point but just raw material to be manipulated in a computer where the real construction of a scene will take place. In short, the production becomes just the first stage of post-production.

        The following example illustrates this new relationship between different stages of the filmmaking process . Traditional on-set filming for Stars Wars: Episode 1 - The Phantom Menace (George Lucas, 1999) was done in just 65 days. The post-production, however, stretched over two years, since ninety-five percent of the film (approximately 2,000 shots out of the total 2,200) was constructed on a computer.^[xxii][xxii]

        Here are two more examples to further illustrate the shift from re-arranging reality to re-arranging its images. From the analog era: for a scene in Zabriskie Point (1970), Michaelangelo Antonioni, trying to achieve a particularly saturated color, ordered a field of grass to be painted. From the digital era: to create the launch sequence in Apollo 13  (Universal Studious, 1995; special effects by Digital Domain), the crew shot footage at the original location of the launch at Cape Canaveral. The artists at Digital Domain scanned the film and altered it on computer workstations, removing recent building construction, adding grass to the launch pad and painting the skies to make them more dramatic. This altered film was then mapped onto 3D planes to create a virtual set which was animated to match a 180-degree dolly movement of a camera following a rising rocket.^{[xxiii][xxiii]}     

        The last example brings us to another conceptualization of digital cinema - as painting. In his book-length study of digital photography, William J. Mitchell focuses our attention on what he calls the inherent mutability of a digital image: "The essential characteristic of digital information is that it can be manipulated easily and very rapidly by computer. It is simply a matter of substituting new digits for old... Computational tools for transforming, combining, altering, and analyzing images are as essential to the digital artist as brushes and pigments to a painter."^[xxiv][xxiv] As Mitchell points out, this inherent mutability erases the difference between a photograph and a painting. Since a film is a series of photographs, it is appropriate to extend Mitchell's argument to digital film. With an artist being able to easily manipulate digitized footage either as a whole or frame by frame, a film in a general sense becomes a series of paintings.^[xxv][xxv]

        Hand-painting digitized film frames, made possible by a computer, is probably the most dramatic example of the new status of cinema. No longer strictly locked in the photographic, it opens itself towards the painterly. It is also the most obvious example of the return of cinema to its nineteenth century origins - in this case, to hand-crafted images of magic lantern slides, the Phenakistiscope, the Zootrope.

        We usually think of computerization as automation, but here the result is the reverse: what was previously automatically recorded by a camera now has to be painted one frame at a time. But not just a dozen images, as in the nineteenth century, but thousands and thousands. We can draw another parallel with the practice, common in the early days of silent cinema, of manually tinting film frames in different colors according to a scene's mood.^[xxvi][xxvi] Today, some of the most visually sophisticated digital effects are often achieved using the same simple method: painstakingly altering by hand thousands of frames. The frames are painted over either to create mattes ("hand drawn matte extraction") or to directly change the images, as, for instance, in Forest Gump, where President Kennedy was made to speak new sentences by altering the shape of his lips, one frame at a time.^{[xxvii][xxvii]} In principle, given enough time and money, one can create what will be the ultimate digital film: 90 minutes, i.e., 129600 frames completely painted by hand from scratch, but indistinguishable in appearance from live photography.

        The concept of digital cinema as painting can be also developed in a different way. I would like to compare the shift from analog to digital filmmaking to the shift from fresco and tempera to oil painting in early Renaissance. A painter making fresco has limited time before the paint dries; and once it is dried, no further changes to the image are possible. Similarly, a traditional filmmaker has limited means to modify images once they are recorded on film. In the case of Medieval tempera painting, this can be compared to the practice of special effects during the analog period of cinema. A painter working with tempera could modify and re-work the image, but the process was quite painstaking and slow. Medieval and early Renaissance masters would spend up to six months on a painting a few inches tall. The switch to oils greatly liberated painters by allowing them to quickly create much larger compositions (think, for instance, of the works by Veronese and Tician) as well as to modify them as long as necessary. This change in painting technology led the Renaissance painters to create new kinds of compositions, new pictorial space and even narratives. Similarly, by allowing a filmmaker to treat a film image as an oil painting, digital technology redefines what can be done with cinema.

        If digital compositing and digital painting can be thought of as an extension of the cell animation techniques (since composited images are stacked in depth parallel to each other, as cells on a animation stand), the newer method of computer-based post-production, makes filmmaking a subset of animation in a different way. In this method the live action, photographic stills and/or graphic elements are positioned in a 3D virtual space. This gives the director the ability to freely move the virtual camera through this space, dolling and panning. Thus cinematography is subordinated to 3D computer animation. We may think of this method as an extension of multiplane animation camera. However, if the camera mounted over a multiplane stand could only move perpendicular to the images, now it can move in a arbitrary trajectory. The example of a commercial film which rely on this newer method which one day may become the standard of filmmaking (because it gives the director most flexibility) is Disney's Alladin; the example of an independent work which fully explores the new aesthetic possibilities of this method without subordinating it to the traditional cinematic realism is The Forest by Tamas Waliczky (1994).  

In discussing digital compositing in "Compositing" section I pointed out that it can be thought off as an intermediary step from 2D images to 3D computer representation. The newer post-production method represents the next logical step towards %100 3D computer generated scenes. Instead of 2D space of "traditional" composite, we now have the layers of moving images positioned in a virtual 3D space.  

        The reader who followed my analysis of the new possibilities of digital cinema may wonder why I have stressed the parallels between digital cinema and the pre-cinematic techniques of the nineteenth century but did not mention twentieth century avant-garde filmmaking. Did not the avant-garde filmmakers already explore many of these new possibilities? To take the notion of cinema as painting, Len Lye, one of the pioneers of abstract animation, was painting directly on film as early as 1935; he was followed by Norman McLaren and Stan Brackage, the later extensively covering shot footage with dots, scratches, splattered paint, smears and lines in an attempt to turn his films into equivalents of Abstract Expressionst painting. More generally, one of the major impulses in all of avant-garde filmmaking, from Leger to Godard, was to combine the cinematic, the painterly and the graphic - by using live action footage and animation within one film or even a single frame, by altering this footage in a variety of ways, or by juxtaposing printed texts and filmed images.

         When the avant-garde filmmakers collaged multiple images within a single frame, or painted and scratched film, or revolted against the indexical identity of cinema in other ways, they were working against "normal" filmmaking procedures and the intended uses of film technology. (Film stock was not be designed to be painted on). Thus they operated on the periphery of commercial cinema not only aesthetically but also technically.

        One general effect of the digital revolution is that avant-garde aesthetic strategies became embedded in the commands and interface metaphors of computer software.^{[xxviii][xxviii]} In short, the avant-garde became materialized in a computer. Digital cinema technology is a case in point. The avant-garde strategy of collage reemerged as a "cut and paste" command, the most basic operation one can perform on digital data. The idea of painting on film became embedded in paint functions of film editing software. The avant-garde move to combine animation, printed texts and live action footage is repeated in the convergence of animation, title generation, paint, compositing and editing systems into single all-in-one packages. Finally, another move to combine a number of film images together within one frame (for instance, in Leger's 1924 Ballet Mechanique or in A Man with a Movie Camera) also become legitimized by technology, since all editing software, including Photoshop, Premiere, After Effects, Flame, and Cineon, by default assumes that a digital image consists of a number of separate image layers. All in all, what used to be exceptions for traditional cinema became the normal, intended techniques of digital filmmaking, embedded in technology design itself.^[xxix][xxix]

From Kino-Eye to Kino-Brush

In the twentieth century, cinema has played two roles at once. As a media technology, cinema's role was to capture and to store visible reality. The difficulty of modifying images once they were recorded was exactly what gave cinema its value as a document, assuring its authenticity. The same rigidity of the film image has defined the limits of cinema as I defined it earlier, i.e. the super-genre  of live action narrative. Although it includes within itself  a variety of styles - the result of the efforts of many directors, designers and cinematographers - these styles share a strong family resemblance. They are all children of the recording process which uses lens, regular sampling of time and photographic media. They are all children of a machine vision.

        The mutability of digital data impairs the value of cinema recordings as a documents of reality. In retrospect, we can see that twentieth century cinema's regime of visual realism, the result of automatically recording visual reality, was only an exception, an isolated accident in the history of visual representation which has always involved, and now again involves the manual construction of images. Cinema becomes a particular branch of painting - painting in time. No longer a kino-eye, but a kino-brush.^[xxx][xxx]         

        The privileged role played by the manual construction of images in digital cinema is one example of a larger trend: the return of pre-cinematic moving images techniques. Marginalized by the twentieth century institution of live action narrative cinema which relegated them to the realms of animation and special effects, these techniques reemerge as the foundation of digital filmmaking. What was supplemental to cinema becomes its norm; what was at its boundaries comes into the center. Computer media returns to us the repressed of the cinema.

        As the examples discussed in this section suggest, the directions which were closed off at the turn of the century when cinema came to dominate the modern moving image culture are now again beginning to be explored. Moving image culture is being redefined once again; the cinematic realism is being displaced from being its dominant mode to become only one option among many.

New Language of Cinema

Cinematic and Graphic: Cinegratography

3D animation, compositing, mapping, paint retouching: in commercial cinema, these radical new techniques are mostly used to solve technical problems while traditional cinematic language is preserved unchanged. Frames are hand-painted to remove wires which supported an actor during shooting; a flock of birds is added to a landscape; a city street is filled with crowds of simulated extras. Although most Hollywood releases now involve digitally manipulated scenes, the use of computers is always carefully hidden.^[xxxi][xxxi]  

Appropriately, in Hollywood the practice of simulating traditional film language received a name "invisible effects," defined as "computer-enchanced scenes that fool the audience into believing the sots were produced with live actors on location, but are really composed of a mélange of digital and live action footage."^{[xxxii][xxxii]}

Commercial narrative cinema still continues to hold on to the classical realist style where images function as un-retouched photographic records of some events which took place in front of the camera. So when Hollywood cinema uses computers to create fantastic, impossible reality, this is done through the introduction of various non-human characters such as aliens, mutants and robots. We never notice the pure arbitrariness of their colorful and mutating bodies, of the beams of energy emulating from their eyes, of the whirlpools of particles emulating from their wings, because they are made perceptually consistent with the set, i.e. they look like something which could have existed in a three-dimensional space and therefore could have been photographed.

But how do filmmakers motivate turning familiar reality such as a human body or a landscape into something phsically impossible in our world? Such transformations are motivated by the movie's narrative. The shiny metallic body of Terminator in Terminator 2 is possible because the Terminator is a cyborg send from the future; the rubber-like body of Jim Carrey in The Mask (Russell, 1994) is possible because his character wears a mask with magical powers. Similarly, in What Dreams May Come (PolyGram Filmed Entertainment, Ward, special effects by Mass.Illusions and others, 1998) the fantastic landscape made of swirling brushstrokes where the main hero is transported after his death is motivated by the unique status of this location.

While embracing computers as a productivity tool, cinema refuses to give up its unique cinema-effect, an effect which, according to film theorist Christian Metz's penetrating analysis made in the 1970s, depends upon narrative form, the reality effect and cinema's architectural arrangement all working together.^{[xxxiii][xxxiii]} Towards the end of his essay, Metz wonders whether in the future non-narrative films may become more numerous; if this happens, he suggests that cinema will no longer need to manufacture its reality effect. Electronic and digital media have already brought about this transformation. Beginning in the 1980s, new cinematic forms have emerged which are not linear narratives, which are exhibited on a television or a computer screen, rather than in a movie theater - and which simultaneously give up cinematic realism.

        What are these forms? First of all, there is the music video. Probably not by accident, the genre of music video came into existence exactly at the time when electronic video effects devices were entering editing studios. Importantly, just as music videos often incorporate narratives within them, but are not linear narratives from start to finish, they rely on film (or video) images, but change them beyond the norms of traditional cinematic realism. The manipulation of images through hand-painting and image processing, hidden in Hollywood cinema, is brought into the open on a television screen. Similarly, the construction of an image from heterogeneous sources is not subordinated to the goal of photorealism but functions as a aesthetic strategy. The genre of music video has been a laboratory for exploring numerous new possibilities of manipulating photographic images made possible by computers - the numerous points which exist in the space between the 2D and the 3D, cinematography and painting, photographic realism and collage. In short, it is a living and constantly expanding textbook for digital cinema.

         A detailed analysis of the evolution of music video imagery (or, more generally, broadcast graphics in the electronic age) deserves a separate treatment and I will not try to take it up here. Instead, I will discuss another new cinematic non-narrative form, CD-ROM-based games, which, in contrast to music video, relied on the computer for storage and distribution from the very beginning. And, unlike music video designers who were consciously pushing traditional film or video images into something new, the designers of CD-ROMs arrived at a new visual language unintentionally while attempting to emulate traditional cinema.            

        In the late 1980s, Apple began to promote the concept of computer multimedia; and in 1991 it released QuickTime software to enable an ordinary personal computer to play movies. However, for the next few years the computer did not perform its new role very well. First, CD-ROMs could not hold anything close to the length of a standard theatrical film. Secondly, the computer would not smoothly play a movie larger than the size of a stamp. Finally, the movies had to be compressed, degrading their visual appearance. Only in the case of still images was the computer able to display photographic-like detail at full screen size.    

        Because of these particular hardware limitations, the designers of CD-ROMs had to invent a different kind of cinematic language in which a range of strategies, such as discrete motion, loops, and superimposition, previously used in nineteenth century moving image presentations, in twentieth century animation, and in the avant-garde tradition of graphic cinema, were applied to photographic or synthetic images. This language synthesized cinematic illusionism and the aesthetics of graphic collage, with its characteristic heterogeneity and discontinuity. The photographic and the graphic, divorced when cinema and animation went their separate ways, met again on a computer screen.

        The graphic also met the cinematic. The designers of CD-ROMs were aware of the techniques of twentieth century cinematography and film editing, but they had to adopt these techniques both to an interactive format and to hardware limitations. As a result, the techniques of modern cinema and of nineteenth century moving image have merged in a new hybrid language which can be called "cinegratography.".  

        We can trace the development of this language by analyzing a few well-known CD-ROM titles. The best selling game Myst (Broderbund, 1993) unfolds its narrative strictly through still images, a practice which takes us back to magic lantern shows (and to Chris Marker's La Jetée).^{[xxxiv][xxxiv]} But in other ways Myst relies on the techniques of twentieth century cinema. For instance, the CD-ROM uses simulated camera turns to switch from one image to the next. It also employs the basic technique of film editing to subjectively speed up or slow down time. In the course of the game, the user moves around a fictional island by clicking on a mouse. Each click advances a virtual camera forward, revealing a new view of a 3D environment. When the user begins to descend into the underground chambers, the spatial distance between the points of view of each two consecutive views sharply decreases. If before the user was able to cross a whole island with just a few clicks, now it takes a dozen clicks to get to the bottom of the stairs! In other words, just as in traditional cinema, Myst slows down time to create suspense and tension.

        In Myst, miniature animations are sometimes embedded within the still images. In the next best-selling CD-ROM 7th Guest (Virgin Games, 1993), the user is presented with video clips of live actors superimposed over static backgrounds created with 3D computer graphics. The clips are looped, and the moving human figures clearly stand out against the backgrounds. Both of these features connect the visual language of 7th Guest to nineteenth century pro-cinematic devices and twentieth century cartoons rather than to cinematic verisimilitude. But like Myst, 7th Guest also evokes distinctly modern cinematic codes. The environment where all action takes place (an interior of a house) is rendered using a wide angle lens; to move from one view to the next a camera follows a complex curve, as though mounted on a virtual dolly.          

        Next, consider the CD-ROM Johnny Mnemonic (Sony Imagesoft, 1995). Produced to complement the fiction film of the same title, marketed not as a "game" but as an "interactive movie," and featuring full screen video throughout, it comes closer to cinematic realism than the previous CD-ROMs - yet it is still quite distinct from it. With all action shot against a green screen and then composited with graphic backgrounds, its visual style exists within a space between cinema and collage.

        It would be not entirely inappropriate to read this short history of the digital moving image as a teleological development which replays the emergence of cinema a hundred years earlier. Indeed, as computers' speed keeps increasing, the CD-ROM designers have been able to go from a slide show format to the superimposition of small moving elements over static backgrounds and finally to full-frame moving images. This evolution repeats the nineteenth century progression: from sequences of still images (magic lantern slides presentations) to moving characters over static backgrounds (for instance, in Reynaud's Praxinoscope Theater) to full motion (the Lumieres' cinematograph). Moreover, the introduction of QuickTime in 1991 can be compared to the introduction of the Kinetoscope in 1892: both were used to present short loops, both featured the images approximately two by three inches in size, both called for private viewing rather than collective exhibition. The two technologies appear to play the similar cultural role. If in the early 1890s the public patronized Kinetoscope parlors where peep-hole machines presented them with the latest marvel - tiny moving photographs arranged in short loops; exactly  a hundred years later, computer users were equally fascinated with tiny QuickTime Movies which turned a computer in a film projector, however imperfect.^[xxxv][xxxv] Finally, the Lumieres' first film screenings of 1895 which shocked their audiences with huge moving images found their parallel in 1995 CD-ROM titles where the moving image finally fills the entire computer screen (for instance, in Jonny Mnemonic.) Thus, exactly a hundred years after cinema was officially "born," it was reinvented on a computer screen.

        But this is only one reading. We no longer think of the history of cinema as a linear march towards only one possible language, or as a progression towards more and more accurate verisimilitude. Rather, we have come to see its history as a succession of distinct and equally expressive languages, each with its own aesthetic variables, each new language closing off some of the possibilities of the previous one - a cultural logic not dissimilar to Kuhn's analysis of scientific paradigms.^{[xxxvi][xxxvi]} Similarly, instead of dismissing visual strategies of early multimedia titles as a result of technological limitations, we may want to think of them as an alternative to traditional cinematic illusionism, as a beginning of digital cinema's new language.        

        For the computer / entertainment industries, these strategies represent only a temporary limitation, an annoying drawback that needs to be overcome. This is one important difference between the situation at the end of the nineteenth and the end of the twentieth centuries: if cinema was developing towards the still open horizon of many possibilities, the development of commercial multimedia, and of corresponding computer hardware (compression boards, storage formats such as DVD), is driven by a clearly defined goal: the exact duplication of cinematic realism. So if a computer screen, more and more, emulates cinema's screen, this not an accident but a result of conscious planning by the computer and entertainment industry. But this drive to turn new media into a simulation of  classical film language, which paralles the encoding of cinema's techniques in software interfaces and hardare itself, described in "Cultural Interfaces" section,  is just one direction for new media dvelopment among numerous others. I will next examine a number of new media and old media objects which point towards other possible trajectories.

New Temporality: Loop as a Narrative Engine

One of the underlying assumptions of this book is that by looking at the history of visual culture and media, and in particular cinema, we can find many strategies and techniques relevant to new media design. Put differently, in order to develop new aesthetics of new media we should pay as much attention to the cultural history as to computer's new unique possibilities to generate, organize, manipulate and distribute data.  

        As we scan through cultural history (which includes the history of new media up until the time of research), three kinds of situations will be particularly relevant for us:

·        when an earlier interesting strategy or technique was abandoned or forced into "underground" without fully developing its potential;

·        when an earlier strategy can be understood as a response to the technological constrains (I am using this more technical term on purpose instead of more ideologically loaded "limitations")  similar to the constrains of new media;

·        when an earlier strategy was used in a situation similar to a particular situation faced by new media designers. For instance, montage was a strategy to deal with modularity of a film (how do you join separate shots?) as well as with a problem of coordinating diffirent media types such as images and sound. Both of these simutaions are being faced once again today by new media designers.

I already used these principles in discussing the parallels between nineteenth century pro-cinematic techniques and the language of new media; they also guided me in thinking about animation (the "underground" of 20^th century cinema) as the basis for digital cinema new language. I will now use a particular parallel between early cinematic and new media technology to highlight another older technique useful to new media: a loop. Characterically, many new media products, be it cultural objects (such as games) or software (various media players such as QuickTime Player) use loops in their design while treating them as temporary technological limitations. I, however, want to think about it as a source of new possibilities for new media.^{[xxxvii][xxxvii]}

        As already mentioned in the previous section, all nineteenth century pro-cinematic devices, up to Edison's Kinetoscope, were based on short loops. As "the seventh art" began to mature, it banished the loop to the low-art realms of the instructional film, the pornographic peep-show and the animated cartoon. In contrast, narrative cinema has avoided repetitions; as modern Western fictional forms in general, it put forward a notion of human existence as a linear progression through numerous unique events.

        Cinema's birth from a loop form was reenacted at least once during its history. In one of the sequences of  A Man with a Movie Camera, Vertov shows us a cameraman standing in the back of a moving automobile. As he is being carried forward by an automobile, he cranks the handle of his camera. A loop, a repetition, created by the circular movement of the handle, gives birth to a progression of events - a very basic narrative which is also quintessentially modern: a camera moving through space recording whatever is in its way. In what seems to be a reference to cinema's primal scene, these shots are intercut with the shots of a moving train. Vertov even re-stages the terror which Lumieres's film supposedly provoked in its audience; he positions his camera right along the train track so the train runs over our point of view a number of times, crushing us again and again[LM1] [LM1].

        Early digital movies shared the same limitations of storage as nineteenth century pro-cinematic devices. This is probably why the loop playback function was built into QuickTime interface, thus giving it the same weight as the VCR-style "play forward" function. So, in contrast to films and videotapes, QuickTime movies were supposed to be played forward, backward or looped. Computer games also heavily relied on loops. Since it was not possible to animate in real time every character, the designers stored short loops of character's motion - for instance, an enemy soldier or a monster walking back and forth - which would be recalled at the appropriate times in the game. Internet pornography also heavily relied on loops. Many sites featured numerous "channels" which were supposed to stream either feature length feature films or "live feeds"; in reality they would usually play short loops (a minute or so) over and over. Sometimes a few films will be cut into a number of short loops which would become the content of 100, 500 or 1000 channels.^{[xxxviii][xxxviii]}  

The history of new media tells us that the hardware limitations never go away: they disappear in one area only to come back in another. One example of this which I already noted is the hardware limitations of the 1980s in the area of 3D computer animation.  In the 1990s they returned in the new area: Internet-based real-time virtual worlds. What used to be the slow speed of CPUs became the slow bandwidth. As a result the 1990s VRML worlds look like the pre-rendered animations done ten years earlier.

The similar logic applies to loops. Earlier QuickTime movies and computer games heavily relied on loops. As the CPU speed increased and larger storage media such as CD-ROM and DVD became available, the use of loops in stand-alone hypermedia declined. However, online virtual worlds such as Active Worlds came to use loops extensively, as it provides a cheap (in terms of bandwidth and computation) way of adding some signs of "life" to their geometric-looking environments.^{[xxxix][xxxix]} Similarly, we may expect that when digital videos will appear on small displays in our cellular phones, personal managers such as Palm Pilot or other wireless communication devices, they will once again will be arranged in short loops because of bandwidth, storage, or CPU limitations.   

        Can the loop be a new narrative form appropriate for the computer age?^[xl][xl] It is relevant to recall that the loop gave birth not only to cinema but also to computer programming. Programming involves altering the linear flow of data through control structures, such as "if/then" and "repeat/while"; the loop is the most elementary of these control structures. Most computer programs are based on repetitions of a set number of steps; this repetition is controlled by the program's main loop. So if we strip the computer from its usual interface and follow the execution of a typical computer program, the computer will reveal itself to be another version of Ford's factory, with a loop as its conveyer belt.

As the practice of computer programming illustrates, the loop and the sequential progression do not have to be thought as being mutually exclusive. A computer program progresses from start to end by executing a series of loops. Another illustration of how these two temporal forms can work together is Möbius House by Dutch team UN Studio/Van Berkel & Bos.^[xli][xli] In this house a number of functionally different areas are arranged one after another in the form of a Möbius strip, thus forming a loop. As the narrative of the day progresses from one activity to the next, the inhabitants move from area to area.

        Traditional cell animation similarly combines a narrative and a loop. In order to save labor, animators arrange many actions, such as movements of characters' legs, eyes and arms, into short loops and repeat them over and over. Thus, as already discussed in the previous section, in a typical twentieth century cartoon a large proportion of motions involves loops. This principle is taken to the extreme in Rybczynski's Tango. Subjecting live action footage to the logic of animation, Rybczynski arranges the trajectory of every character through space as a loop. These loops are further composited together resulting in a complex and intricate time-based structure. At the same time, the overall "shape" of this structure is governed by a number of narratives. The film begins in an empty room; next the loops of character's trajectories through this room are added, one by one. The end of the film mirrors its beginning as the loops are "deleted" in a reverse order, also one by one. This metaphor for a progression of a human life (we are born alone, gradually forms relations with other humans, and eventually die alone) is also supported by another narrative: the first character to appear in the room is a young boy, the last one is an old woman.  

The concept of a loop as an "engine" which puts the narrative in motion becomes a foundation of a brilliant interactive TV program Akvaario (aquarium) by a number of graduate students at Helsinki's University of Art and Design (Professor and Media Lab coordinator: Minna Tarrka).^[xlii][xlii] In contrast to many new media objects which combine the conventions of cinema, print and HCI, Akvaario aims to preserve the continuos flow of traditional cinema, while adding interactivity to it. Along with an earlier game Jonny Mnemonic (SONY, 1995), as well as the pioneering interactive laserdisk computer installations by Graham Weinbren done in the 1980s, this project is a rare example of a new media narrative which does not rely on the oscillation between non-interactive and interactive segments (see "Illusion, Narrative and Interactivity" section for the analysis of this temporal ossicilation.)

Using the already familiar convention of such games such as Tamagotchi (1996-), the program asks TV viewers to "take charge" of a fictional human character.^{[xliii][xliii]} Most shots which we see show this character engaged in different activities in his apartment: eating dinner, reading a book, starring into space. The shots replace each other following standard conventions of film and TV editing. The result is something which looks at first like a conventional, although very long, movie (the program was projected to run for three hours every day over the course of a few months), even though the shots are selected in real time by a computer porgram from a database of a few hundreds diffirent shots.

By choosing one of the four buttons which are always present on the bottom of the screen, the viewers control character's motivation. When a button is pressed, a computer program selects a sequence of particular shots to follow the shot which plays currently. Because of visual, spatial and referential discontinuity between shots typical of standard editing, the result is something which the viewer interprets as a conventional narrative. A film or television viewer viewer does not expect that any two shots which follow one another have to display the same space or subsequent moments of time. Therefore in Akvaario a computer program can "weave" an endless narrative by choosing from a database of different shots. What gives the resulting "narrative: a suficient continuity is that almost all shots show the same character.

Akvaario is one of the first examples of what in previous chapter I called a "database narrative." It is, in other words, a narrative which fully utilizes many features of database organization of data. It relies on our abilities to classify database records according to different dimensions, to sort through records, to quickly retrieve any record, as well as to "stream" a number of different records continuously one after another.

        In Akvaario the loop becomes the way to bridge linear narrative and interactive control. When the program begins, a few shots keep following each other in a loop. After users choose character's motivation by pressing a button, this loop becomes a narrative. Shots stop repeating and a sequence of new shots is displayed. If no button pressed again, the narrative turns back into a loop, i.e. a few shots start repeating over and over. In Akvaario a narrative is born from a loop and it returns back to a loop. The historical birth of modern fictional cinema out of the loop returns as a condition of cinema's rebirth as an interactive form. Rather than being an archaic leftover, a reject from cinema's evolution, the use of loop in Akvaario suggests a new temporal aesthetics for computer-based cinema.

        Jean-Louis Boissier's Flora petrinsularis realizes some of the possibilities contained in the loop form in a diffirent way.^[xliv][xliv] This CD-ROM is based on Rousseau's Confessions. It opens with a white screen, containing a numbered list. Clicking on each item leads us to a screen containing two windows, positioned side by side. Both windows show the same video loop made from a few diffirent shots. The two loops are offset from each other in time. Thus, the images appearing in the left window reappear in a moment on the right and vice versa, as though an invisible wave is running through the screen. This wave soon becomes materialized: when we click inside the windows we are taken to a new screen which also contains two windows, each showing loop of a rhythmically vibrating water surface. The loops of water surfaces can be thought of as two sign waves offset in phase. This structure, then, functions as a "meta-text" of a structure in the first screen. In other words, the loops of water surface act as a diagram of the loop structure which controls the correlations between shots in the first screen, similar to how Marey and the Gibsons diagrammed human motion in their film studies in the beginning of the twentieth century.

As each mouse click reveals another loop, the viewer becomes an editor, but not in a traditional sense. Rather than constructing a singular narrative sequence and discarding material which is not used, here the viewer brings to the forefront, one by one, numerous layers of looped actions which seem to be taking place all at once, a multitude of separate but co-existing temporalities. The viewer is not cutting but re-shuffling. In a reversal of Vertov's sequence where a loop generated a narrative, viewer's attempt to create a story in Flora petrinsularis leads to a loop.

It is useful to analyze the loop structure of Flora petrinsularis using montage theory. From this perspective, the repetition of images in two adjoint windows can be interpreted as an example of what Eisenstein called rhythmical montage. At the same time, Boissier takes montage apart, so to speak. The shots which in traditional temporal montage would follow each in time here appear next to each other in space. In addition, rather than being "hard-wired" by an editor in only one possible structure, here the shots can appear in different combinations since they are activated by a user moving a mouse across the windows.

        At the same time, it is possible to find more traditional temporal montage in this work as well - for instance, the move from first screen which shows close-up of a woman to a second screen which shows water surfaces and back to the first screen. This move can be interpreted as a traditional parallel editing. In cinema parallel editing involves alternating between two subjects. For instance, a chase sequence may go back and forth between the images of two cars, one pursuing another. However in our case the water images are always present "underneath" the first set of images. So the logic here is again one of co-existence rather than that of replacement, typical of cinema (see my discussion of spatial montage below).

        The loop which structures Flora petrinsularis on a number of levels becomes a metaphor for human desire which can never achieve resolution. It can be also read as a comment on cinematic realism. What are the minimal conditions necessary to create the impression of reality? As Boissier demonstrates, in the case of a field of grass, a close-up of a plant or a stream, just a few looped frames become sufficient to produce the illusion of life and of linear time.

        Steven Neale describes how early film demonstrated its authenticity by representing moving nature: "What was lacking [in photographs] was the wind, the very index of real, natural movement. Hence the obsessive contemporary fascination, not just with movement, not just with scale, but also with waves and sea spray, with smoke and spray."^[xlv][xlv] What for early cinema was its biggest pride and achievement - a faithful documentation of nature's movement - becomes for Boissier a subject of ironic and melancholic simulation. As the few frames are looped over and over, we see blades of grades shifting slightly back and forth, rhythmically responding to the blow of non-existent wind which is almost approximated by the noise of a computer reading data from a CD-ROM.

        Something else is being simulated here as well, perhaps unintentionally. As you watch the CD-ROM, the computer periodically staggers, unable to maintain consistent data rate. As a result, the images on the screen move in uneven bursts, slowing and speeding up with human-like irregularity. It is as though they are brought to life not by a digital machine but by a human operator, cranking the handle of the Zootrope a century and a half ago...

 

 

 

Spatial Montage

Along with taking on a loop, Flora petrinsularis can also be seen as a step towards what I will call a spatial montage. Instead of a traditional singular frame of cinema, Boissier uses two images at once, positioned side by side. This can be thought of a simplest case of a spatial montage. In general, spatial montage would involve a number of images, potentially of different sizes and proportions, appearing on the screen at the same time. This by itself of course does not result in montage; it up to the filmmaker to construct a logic which drives which images appear together, when they appear and what kind of relationships they enter with each other.

        Spatial montage represents an alternative to traditional cinematic temporal montage, replacing its traditional sequential mode with a spatial one. Ford's assembly line relied on the separation of the production process into a set of repetitive, sequential, and simple activities. The same principle made computer programming possible: a computer program breaks a tasks into a series of elemental operations to be executed one at a time. Cinema followed this logic of industrial production as well. It replaced all other modes of narration with a sequential narrative, an assembly line of shots which appear on the screen one at a time. A sequential narrative turned out to be particularly incompatible with a spatial narrative which played a prominent role in European visual culture for centuries. From Giotto's fresco cycle at Capella degli Scrovegni in Padua to Courbet's A Burial at Ornans, artists presented a multitude of separate events within a single space, be it the fictional space of a painting or the physical space which can be taken by the viewer all in once. In the case of Giotto's fresco cycle and many other fresco and icon cycles, each narrative event is framed separately but all of them can be viewed together in a single glance. In other cases, different events are represented as taking place within a single pictorial space. Sometimes, events which formed one narrative but they separated by time were depicted within a single painting. More often, the painting's subject became an excuse to show a number of separate "micro-narratives" (for instance, works by Hiëronymous Bosch and Peter Bruegel). All in all, in contrast to cinema's sequential narrative, in spatial narrative all the "shots" were accessible to a viewer at one. Like nineteenth century animation, spatial narrative did not disappear completely in the 20^th century; but just as animation, it came to be delegated to a minor form of Western culture - comics.

It is not accidental that the marginalization of spatial narrative and the privileging of sequential mode of narration coincided with the rise of historical paradigm in human sciences. Cultural geographer Edward Soja has argued that the rise of history in the second half of the nineteenth century coincided with the decline in spatial imagination and the spatial mode of social analysis.^[xlvi][xlvi] According to Soja, it is only in the last decades of the twentieth century that this mode made a powerful comeback, as exemplified by the growing importance of such concepts as "geopolitics" and "globalisation" as well as by the key role analysis of space played in theories of post-modernism. Indeed, although some of the best thinkers of the twentieth century such as Freud, Panofsky and Foucault were able to combine historical and spatial mode of analysis in their theories, they probably represent an exemption rather than the norm. The same holds for film theory, which, from Eisenstein in the 1920s to Deleuse in the 1980s, focused on temporal rather than spatial structures of film.

        Twentieth century film practice has elaborated complex techniques of montage between different images replacing each other in time; but the possibility of what can be called "spatial montage" between simultaneously co-exiting images was not explored as systematically. (Thus cinema also given to historical imagination at the expense of spatial one.) The notable exemptions include the use of split screen by Hans Abel in Napoléon in the 1920s and also by the American experimental filmmaker Stan Van der Beek in the 1960s; also some other works, or rather, events, of the 1960s "expanded cinema" movement, and, last but not least, the legendary multi-image multimedia presentation shown in the Chech Pavilion at the1967 World Expo. Emil Radok's Diaolyektan consisted from 112 separate cubes. One hundred and sixty different images could be projected onto each cube. Radok was able to "direct" each cube separately. To the best of my knowledge, since this project nobody tried again to create a spatial montage of this complexity in any technology.

Traditional film and video technology were designed to completely fill a screen with a single image; thus to explore spatial montage a filmmaker had to work "against" the technology. This in part explains why so few tried to do this. But when, in the 1970s, the screen became a bit-mapped computer display, with individual pixels corresponding to memory locations which can be dynamically updated by a computer program, one image/ one screen logic was broken. Since the Xerox Park Alto workstation, GUI used multiple windows. It would be logical to expect that cultural forms based on moving images will eventually adopt similar conventions. In the 1990s some computer games such as Golden Eye (Nintendo/Rare, 1997) already used multiple windows to present the same action simultaneously from different viewpoints. We may expect that computer-based cinema will eventually have to follow the same direction - especially when the limitations of communication bandwidth will disappear, while the resolution of displays will significantly increase, from the typical 1-2K in 2000 to 4K, 8K or beyond. I believe that the next generation of cinema - broadband cinema - will add multiple windows to its language. When this happen, the tradition of spatial narrative which twentieth century cinema suppressed will re-emerge one again.

Looking back at visual culture and art of the previous centuries gives many ideas for how spatial narrative can be further developed in a computer; but what about spatial montage? In other words, what will happen if we combine two different cultural traditions: informationally dense visual narratives of Renaissance and Baroque painters with "attention demanding" shot juxtapositions of twentieth century film directors? "My boyfriend came back from war!," a Web-based work by the young Moscow artist  Olga Lialina, can be read as an exploration of this direction.^{[xlvii][xlvii]} Using the capability of HTML to create frames within frames, Lialina leads us through a narrative which begins with an single screen. This screen becomes progressively divided into more and more frames as we follow different links. Throughout, an image of a human couple and of a constantly blinking window remain on the left part of screen. These two images enter into new combinations with texts and images on the right part which keep changing as the user interacts with the work. As the narrative activates different parts of the screen, montage in time gives way to montage in space. Put differently, we can say that montage acquires a new spatial dimension. In addition to montage dimensions already explored by cinema (differences in images' content, composition, movement) we now have a new dimension: the position of the images in space in relation to each other. In addition, as images do not replace each other (as in cinema) but remain on the screen throughout the movie, each new image is juxtaposed not just with one image which preceded it, but with all the other images present on the screen.   

        The logic of replacement, characteristic of cinema, gives way to the logic of addition and co-existence. Time becomes spatialized, distributed over the surface of the screen. In spatial montage, nothing is potentially forgotten, nothing is erased. Just as we use computers to accumulate endless texts, messages, notes and data, and just as a person, going through life, accumulates more and more memories, with the past slowly acquiring more weight than the future, spatial montage can accumulate events and images as it progresses through its narrative. In contrast to cinema's screen, which primarily functioned as a record of perception, here computer screen functions as a record of memory.

        As I already noted, spatial montage can also be seen as an aesthetics appropriate for the user experience of muli-tasking and multiple windows of GUI. In the text of his lecture "Of other spaces" Michel Foucault writes: "We are now in the epoch of simultaneity: we are in epoch of juxtaposition, the epoch of near and far, of the side-by-side, of the dispersed.our experience of the world is less of a long life developing through time that that of a network that connects points and intersects with its own skein."^{[xlviii][xlviii]} Writing this in the early 1970s, Foucault appears to prefigure not only the network society, exemplified by the Internet ("a network which connects points") but also GUI ("epoch of simultaneity.of the side-by-side). GUI allows the users to run a number of software applications at the same time; and it uses the convention of multiple overlapping windows to present both data and controls. The construct of  the desktop with presents the user with multiple icons which are all simultaneously and continuously "active" (since they all can be clicked at any time) follows the same logic of "simultaneity" and of "side-by-side." On the level of computer programming, this logic corresponds to object-oriented programming. Instead of a single program which, like Ford's assembly line, is executed one statement at a time, in object-oriented paradigm a number of objects send messages to each other. These objects are all active simultaneously. Object-oriented paradigm and multiple windows of GUI work together; object-oriented approach was in fact used to program the original Macintosh GUI which substituted the "one command at a time" logic of DOS with the logic of simultaneity of multiple windows and icons.  

The spatial montage of "My boyfriend came back from war!" follows this logic of simultaneity of modern GUI. Multiple and simultaneously active icons and windows of GUI become the multiple and simultaneously active frames and hyperlinks of this Web artwork. Just as the GUI user can click on any icon at any time, changing the overall "state" of the computer environment, the user of Lialina's site can activate different hyperlinks which are all simultaneously present. Each action either changes the contents of a single frame or creates new frame(s). In either case, the "state" of the screen as a whole is affected. The result is a new cinema where syncronic dimension is no longer  privileged to the diacronic dimension, space is no longer privileged to time, the simultaneity is no longer privileged to sequence, montage within a shot is no longer privileged to montage in time.

Cinema as an Information Space

As we saw in "Cultural Interfaces" section, cinema language which originally was an interface to narrative taking place in 3D space is now becoming an interface to all types of computer data and media. I discussed how such elements of this language as rectangular framing, mobile camera, image transitions, montage in time and montage within an image reappear in general purpose HCI, in interfaces of software applications and in cultural interfaces.

        Yet another way to think about new media interfaces in relation to cinema is to interpret the later as information space. If HCI is an interface  to computer data, and a book is interface to text, cinema can be thought of an interface to events taking place in 3D space. Just as painting before it, cinema presented us with familiar images of visible reality - interiors, landscapes, human characters - arranged within a rectangular frame. The aesthetics of these arrangements ranges from extreme scarcity to extreme density. The examples of the former are paintings by Morandi and shots in Late Spring (Yasujiro Ozu, 1949); the examples of the later are paintings by Bosch and Bruegel (and much of Northern Renaissance painting in general), and many shots in  A Man with a Movie Camera.^[xlix][xlix] It would be only a small leap to relate this density of "pictorial displays" to the density of contemporary information displays such as Web portals which may contain a few dozen hyperlinked elements; or the interfaces of popular software packages which similarly present the user with dozens commands at once. Can the contemporary information designers learn from information displays of the past - particular films, paintings and other visual forms which follow the aesthetics of density[LM2] [LM2]?   

        In making such a connection I rely on work of art historian Svetlana Alpers who claimed that in contrast to Italian Renaissance painting primarily concerned with narration, Dutch painting of the Seventeenth century is focused on description.^[l][l] While the Italians subordinated details to the narrative action, creating clear hierarchy of viewer's attention, in Dutch paintings particular details and, consequently, viewer's attention, are more evenly distributed throughout the whole image. While functioning as a window into an illusionary space, the Dutch painting also is a loving catalog of numerous objects, different material surfaces and light effects painted in minute detail (works by Vermeer, for instance.) The dense surfaces of these paintings can be easily related to contemporary interfaces; in addition, they can be also related to the future aesthetics of a moving image, when the digital displays will move much beyond the resolution of analog television and film.

        The trilogy of computer films by Paris-based filmmaker Christian Boustani, develops such an aesthetics of density. Taking his inspiration from Renaissance Dutch painting as well as from classical Japanese art, Boustani uses digital compositing to achieve unprecedented. for film, information density. While this density was typical for old art he draws on, it was never before achieved in cinema. In Brugge (1995) Boustani recreates the images typical of winter landscape scenes in Dutch seventeenth century painting. His next film A Viagem (The Voyage, 1998) achieves even higher information density; some shots of the film use as many as 1600 separate layers.

This new cinematic aesthetics of density seems to be highly appropriate for out age. If, from a city street to a Web page, we are surrounded by highly dense information surfaces, it is appropriate to expect from cinema similar logic. (In a same fashion, we may think of spatial montage as reflecting another contemporary daily experience: working with a number of different applications at once on a computer. If we are now used to distribute and rapidly switch our attention from one program to another, from one set of windows and command to another set, we may find multiple streams of audio-visual information presented simultaneously more satisfying than a single stream of traditional cinema.)

It is appropriate that some of the most dense shots of A Viagem recreates a Renaissance marketplace, this symbol of emerging capitalism which was probably responsible for the new density of Renaissance painting (think, for instance, of Dutch still-lives which function as a kind of store display window aiming to overwhelm the viewer and seduce her into making a purchase). In the same way, in the 1990s the commercialization of the Internet was responsible for the new density of Web pages. By the end of the decade all home pages of  big companies and Internet portals became indexes containing dozens of entries in a small type. If every small area of the screen can potentially contain a lucrative add or a link to a page with one, this leaves no place for the aesthetics of emptiness and minimalism. Thus it is not surprising that commercialized Web joined the same aesthetics of information density and competing signs and images which characterizes visual culture in a capitalist society in general.

        If Lialina's spatial montage relies on HTML frames and actions of the user to activate images appearing in these frames, Boustani's spatial montage is more purely cinematic and painterly. He combines mobility of camera and movement of objects characteristic of cinema which the "hyper-realism" of old Dutch painting which presented everything "in focus." In analog cinema, the inevitable "depth of field" artifact acted as a limit to the information density of an image. The achievement of Boustani is to create images where every detail is in focus and yet the overall image is easily readable. This could only be done through digital compositing. By reducing visible reality to numbers the computer makes possible for us to literally see in a new way. If, according  to Benjamin, early twentieth century cinema used close-up "to bring things 'closer' spatially and humanly," "to get hold of an object at very close range," and, as a result, destroyed their aura, digital composites of Boustani can be said to bring objects close to a viewer without "extracting" them away from their places in the word. (Of course also an opposite interpretation is possible: we can say that Boustani's digital eye is super-human. Similar to the argument in "Synthetic Image and its subject" section, his vision can be interpreted as the gaze of a cyborg or computer vison system which can see things equally well at any distance.)

        Scrutinizing the prototypical perceptual spaces of modernity - the factory, the movie theater, the shopping arcade - Walter Benjamin insisted on the contiguity between the perceptual experiences in the workplace and outside of it:

Whereas Poe's passers-by cast glances in all directions which still appeared to be aimless, today's pedestrians are obliged to do so in order to keep abreast of traffic signals. Thus technology has subjected the human sensorium to a complex kind of training. There came a day when a new and urgent need for stimuli was met by the film. In a film, perception in the form of shocks was established as a formal principle. That which determines the rhythm of production on a conveyer belt is the basis of the rhythm of reception in the film.^[li][li]

For Benjamin, the modern regime of perceptual labor, where the eye is constantly asked to process stimuli, equally manifests itself in work and leisure. The eye is trained to keep pace with the rhythm of industrial production at the factory and to navigate through the complex visual semiosphere beyond the factory gates. It is appropriate to expect that the computer age will follow the same logic, presenting the users with similarly structured perceptual experiences at work and at home, on a computer screen and outside of it. Indeed, as I already noted, we now use the same interfaces for work and for leisure, the condition exemplified most dramatically by Web browsers. Another example is the use of the same interfaces in flight and military simulators, in computer games modeled after these simulators, and in the actual controls of planes and other vehicles (recall the popular perception of Gulf War as "video game war.") But if Benjamin appears to regret that the subjects of an industrial lost pre-modern freedom of perception, now regimented by factory, modern city and film, we may instead think of information density of our own workspaces as a new aesthetic challenge, something to explore rather than to condemn. Similarly, we should explore the aesthetic possibilities of all aspects of user's experience with a computer, this key experience of modern life: dynamic windows of GUI, multi-tasking, search engines, databases, navigable space, and others.

Cinema as a Code

When radically new cultural forms appropriate for the age of wireless telecommunication, multitasking operating systems and information appliances will arrive, what will they look like? How would we even know they are here? Would future films look like a "data shower" from the movie "Matrix"?  Is the famous fountain at Xerox Park in which the strength of the water stream reflects the behavior of the stock market, with stock data arriving in real time over Internet, represents the future of public sculpture?

We don't yet know the answers to these questions. However, what artists and critics can do is point out the radically new nature of new media by staging - as opposed to hiding - its new properties. As my last example, I will discuss Vuk Cosic's ASCII films, which effectively stage one characteristic of computer-based moving images - their identity as a computer code.^[lii][lii]

It is worthwhile to relate Cosic's films to both Zuse's "found footage movies" from the 1930s, which I invoke in the beginning of this book, and to the first all-digital feature length movie made sixty years later - Lucas's Stars Wars: Episode 1, The Phantom Menace.^[liii][liii] Zuse superimposes digital code over the film images. Lucas follows the opposite logic: in his film, digital code "lies under" his images. That is, given that most images in the film were put together on computer workstations, during the post-production process they were pure digital data. The frames were made up from numbers rather than bodies, faces, and landscapes. The Phantom Menace is, therefore, can be called the first feature-length commercial abstract film: two hours worth of frames made up from matrix of numbers. But this is hidden from the audience.

What Lucas hides, Cosic reveals. His ASCII films "perform" the new status of media as digital data. The ASCII code that results when an image is digitized is displayed on the screen. The result is as satisfying poetically as it is conceptually - for what we get is a double image: a recognizable film image and an abstract code together. Both are visible at once. Thus, rather than erasing the image in favor of the code as in Zuse's film, or hiding the code from us as in Lucas's film, in ASCII films the code and the image coexist.

Like VinylVideo project by Gebhard Sengmüller which records TV programs and films on old vinyl disks,^[liv][liv] Cosic's ASCII initiative^[lv][lv] is a systematic program of translating media content from one obsolete format into another. These projects remind us that since at least the 1960s the operation of media translation has been at the core of our culture. Films transferred to video; video transferred from one video format to another; video transferred to digital data; digital data transferred from one format to another: from floppy disks to Jaz drives, from CD-ROMs to DVDs;  and so on, indefinitely. The artists noticed this new logic of culture early on: in the 1960s, Roy Lichtenstein and Andy Warhol already made media translation the basis of their art. Sengmuller and Cosic understand that the only way to deal with built-in media obsolescence of a modern society is by ironically resurrecting dead media. Sengmuller translates old TV programs into vinyl disks; Cosic translates old films into ASCII images.^[lvi][lvi]

Why do I call ASCII images an obsolete media format? Before the printers capable of outputting raster digital images became widely available toward the end of the 1980s, it was commonplace to make printouts of images on dot matrix printers by converting the images into ASCII code.  I was surprised that in 1999 I still was able to find the appropriate program on my UNIX system. Called simply "toascii," the command, according to the UNIX system manual page for the program, "prints textual characters that represent the black and white image used as input."

The reference to early days of computing is not unique to Cosic but shared by other net.artists. Jodi.org, the famous net.art project created by the artistic team of Joan Heemskerk and Dirk Paesmans, often evokes DOS commands and the characteristic green color of computer terminals from the 1980s^[lvii][lvii]; a Russian net.artist Alexei Shulgin has performed music in the late 1990s using old 386PC.^{[lviii][lviii]}  But in the case of ASCII code, its use evokes not only a peculiar episode in the history of computer culture but a number of earlier forms of media and communication technologies as well. ASCII is an abbreviation of American Standard Code for Information Interchange. The code was originally developed for teleprinters and was only later adopted for computers in the 1960s. A teleprinter was a twentieth-century telegraph system that translated the input from a typewriter keyboard into a series of coded electric impulses, which were then transmitted over communications lines to a receiving system, which decoded the pulses and printed the message onto a paper tape or other medium. Teleprinters were introduced in the 1920s and were widely used until the 1980s (Telex being the most popular system), when they were gradually replaced by fax and computer networks.^[lix][lix]

ASCII code was itself an extension of an earlier code invented by Jean-Maurice-Emile Baudot in 1874. In Baudot code, each letter of an alphabet is represented by a five-unit combination of current-on or current-off  signals of equal duration. ASCII code extends Baudot code by using eight-unit combinations (that is, eight "bits" or one "byte") to represent 256 different symbols. Baudot code itself was an improvement over the Morse code invented for early electric telegraph systems in the 1830s. And so on.

The history of ASCII code compresses a number of technological and conceptual developments which lead to (but I am sure will not stop at) a modern digital computers: cryptography, real-time communication, communication network technology, coding systems. By juxtaposing ASCII code with the history of cinema, Cosic accomplishes what can be called an artistic compression. That is, along with staging the new status of moving images as a computer code, he also "encodes" in these images many key issues of computer culture and new media art.

As this book has argued, in computer age, cinema, along with other established cultural forms, indeed becomes precisely a code. It is now used to communicate all types of data and experiences; and its language is encoded in interfaces and defaults of software programs and hardware itself. Yet, while on the one hand new media strengthens existing cultural forms and languages, including the language of cinema, it simultaneously "opens" them up for redefinition. The elements of their interfaces become separated from the types of data they were traditionally connected to. Further, what was previously in the background, on the margins, comes into the center. For instance, animation comes to challenge live cinema; spatial montage comes to challenge temporal montage, database comes to challenge narrative; search engine comes to challenge encyclopedia; and, last but not least, online distribution of culture challenges traditional "off-line" formats. To use a metaphor from computer culture, new media turns all culture and cultural theory into "open source." This "opening up" of all cultural techniques, conventions, forms and concepts is ultimately the most positive cultural effect of computerization - the opportunity to see the world and the human being anew, in ways which were not available to A Man with a Movie Camera.    

---