Framing, Re-framing, and Un-framing Cinema

Digital artists Paul Kaiser and Marc Downie of OpenEndedGroup collaborate with Tom Gunning (University of Chicago, Art History and Cinema & Media Studies) on a project that seeks to study and intervene in the current redefinition of the moving image as it shifts from the frame of classical cinema to the immersive framelessness and interactivity of virtual reality. 

Framing, Re-framing, and Un-framing Cinema — inventing methods for measuring, investigating, and creating film in the emerging age of virtual reality

Overview

We find ourselves at a historic moment as the moving image shifts from the frame of classical cinema to the immersive framelessness and interactivity of virtual reality (VR).  This constitutes perhaps the most radical transformation of the moving image since the very beginnings of cinema.  Our goal is not only to study but also to intervene in the current redefinition of the moving image before it becomes too rigidly determined by the powerful industrial forces now propelling it forward.

Our approach is two-fold. First, we will make self-contained VR films that are unlike any others, either those of the classic cinema or those likely to emerge from the VR mainstream. Second, we will forge new methods by which to measure and to study earlier forms of cinema as it developed.

Our key strategy will be to advance both goals by conceptualizing and programming virtual viewing (and hearing) machines that will operate inside the boundless space of virtual reality. These machines will serve as the common frameworks for both the creation of our new films and our novel studies of older films by means of visual measurement and analysis.

The anticipated outcomes will manifest themselves in cinemas and galleries, on the one hand, and in the classroom, on the other — specifically, in a film studies course to be co- taught in fall 2016.  It’s an ambitious project we’re proposing, for our aim is to create new forms both of cinema and of film pedagogy.

Background

Like others, we sense that our time uncannily echoes the turn of the 20th century, when a similar profusion of forms and possibilities of the moving image emerged. Careful historical comparison illuminates both eras, for it’s the same sort of fleeting historical moment we face now that Muybridge, Marey, Melies, the Lumieres, and Griffith faced more than a century before us. The future they were inventing was as much up for grabs for them as ours is for us right now.

As Tom Gunning has argued in his essay “Re-Newing Old Technologies: Astonishment, Second Nature, and the Uncanny in Technology from the Previous Turn-of -the-Century” (in Rethinking Media Change: The Aesthetics of Transition), such times don’t last long, because utopian ambitions are partly based on the unfamiliar nature of new media. The windows of possibility are fleeting during such phases of novelty and transition, before practice become codified and routine, and this makes the timing of our proposal urgent.

What struck us most forcefully in both the discussions and demos there was that before the emergent properties of VR will have had a chance to be explored, the commercial VR field has already settled on a few premature orthodoxies. The new conventional wisdom is that for VR to succeed in the marketplace, the concept of gameplay must be made paramount. In effect this means that mainstream VR is now drawing its lessons from video game forms rather than from cinematic forms.  Following this line of reasoning, the mainstream has decided that VR is to be frameless and interactive:

Frameless — VR is to plunge the viewer into a boundless three-dimensional space, where the continuous borderless expanse of a single wraparound image vastly exceeds the visual field of the eye. It’s assumed that you’ll see and navigate VR worlds in the same way as you do a first-person-shooter video game; you’ll move your virtual body through the virtual space as if through continuous physical space.

Interactive — VR is to interact with the user by responding continuously to his or her actions (and decisions). Here, though, we must distinguish between two forms of interactivity — one is an essential property of the new medium while the other is a premature overlay. The essential interactivity of VR is perceptual: VR responds with extraordinary precision to your every movement, so that even the slightest movement of your head will shift both what you see and what you hear in a minutely calibrated three-dimensional space. Everything exists and is perceived in relation to how you position your body at any given moment. This is new and it’s exciting to us. However, we regard as premature the assumptions being made about narrative interactivity. It’s presumed that viewers (or rather, game players) will interact with VR stories in the style of choose-your-own-adventure games, where the player’s decisions take them down fixed branches of a story to variable (but finite) endings. This is thought to insure replayability, another key consideration imported from the gaming world.

The emerging VR orthodoxy thus rules out a huge range of possibilities. For example, a cut (that is, the traditional cinematic edit) in VR is said to be impossible (or at least highly problematic) — and with that pronouncement, all new possibilities of montage are discarded before they’re even attempted. Indeed, the new VR proponents, though they pay lip service to “movie production values” and would gladly recruit both Hollywood’s stars and marketing budgets to their cause, largely do away with the methods and precedents of cinema.

We’re not willing to do that. Very early forms of the moving image have long fascinated all four of us in very different ways, and we’ve shared the belief that by going back to beginnings — to divergent forms of pre- and early cinema — we can not only study mainstream cinema within a much broader frame of reference and of theoretical possibility, but can also find and choose paths not taken by the prevailing forces to discover new kinds of visual experience.

A useful but admittedly overly simple way of looking at film history is to see how narrative film first refined but then eventually rigidified film form, especially as the medium was subordinated to tasks of narrative or conveying information or propaganda. Working at its edges, experimental filmmakers have served the vital function of keeping all the possibilities of film in play; and at times these possibilities fed back into mainstream cinema, keeping it alive. We propose to serve the same experimental function, making films in VR that build on possibilities already closed off by the mainstream. Given the state of the field, we are confident that we are uniquely able to provide the conceptual, historical, and aesthetic expertise to embark on this course — not to mention the vital software engineering abilities and relevant art and science experience needed to pull this off.

The plan in rough outline

At this early stage of the project, we can only suggest some of the film and cinemetric [analytical] devices we’ll be building, for part of the project will be devoted to our figuring this out.

We do know that we will work by means of rapid iterative loops, in which an initial idea is immediately implemented at least in rough form in 3D so that it can be studied and judged — perhaps rejected outright, or perhaps refined into something final. OpenEndedGroup are almost unique amongst digital artists in that they own their own development environment for making artworks. This code platform was first developed for OpenEndedGroup’s collaborations with choreographers — for rehearsal situations in costly theaters where the artists had to be able to change their visual projections as quickly as, say, choreographer Trisha Brown shifted her dancers on stage. It’s that kind of rapidity that will allow all project collaborators to be actively engaged through all key stages of the project. To allow art scholarship and art practice to be in deep dialog with each other, we must make the art-making move at the speed of discussion.

Our plan is to reconstruct the methods of film within the new medium of virtual reality. By way of illustration, a first move might be simply to place a flat virtual film “screen” within the 3D space of VR, and then to conduct a visual examination and analysis of early montage. Perhaps we’ll project one of Griffith’s first Biograph films onto the virtual screen as we examine the origins of narrative montage. We might then subject the film’s cutting to an automated analysis, with computer vision algorithms that can find shot length, shot type (close, medium, far, and tracking), and linkage type. This data could then be used to multiply the screens across lateral space, with a shot per screen and a visual notation of the montage analysis. Viewers would be able to come in close to the virtualized film to examine the mechanism of a particular cut or they could back away to see the over-all shape of the montage across the whole film. We begin with this simple unit — the projected film on a flat rectangle in virtual space — and expand outwards: by multiplication, by arrangement, and by choreographing the movement, appearance and dissolution of these units. With this fundamental building block embedded within the 3D space of VR, we can build anything from virtual cinemas to elaborate pre-cinematic devices to densely layered digital abstractions to massive visualizations.

Here a word about data visualization, as currently deployed across the field of digital humanities, might be useful. Typically, data visualizations offer only the grand summarizing view of a given set of data. For instance, the popular Google N-gram viewer has allowed many scholars to track word usage across a huge corpus of literature, with the findings typically represented in large graphs summarizing word frequency over time. But this approach confines you to the large view alone; you can’t zero in to see particular instances, which might well reveal both subtle and substantial shifts in word usage as well as in context. This limitation typifies too many data visualizations, pushing you away from a closer reading of the subject, forever disconnecting “data” from material, demanding a “distant reading” unrelatable to both actual reading and almost all critical practice. This is equally true even of visualizations of intrinsically visual things.  For example, most cinemetric studies summarize their measurement of films only in numbers and/or graphs; and while these analyses are a useful first step, our project, exemplified in our hypothetical example, is aimed at taking the next step, which is to see the data fully, both from afar and very close up.

A new VR film that might arise from a mechanism related to the montage visualization, though considerably more sophisticated. OpenEndedGroup has a work-in-progress entitled 1237 Stabs at Blade Runner, which is generated by a precise analysis and uncanny simulation of the movement found in every shot of that famous science fiction film. It works like this: An artificial intelligence agent uses computer vision to detect edges, colors, and motions within each shot, but with the limitation that, like most robots in the outside world, it can’t tell figure from ground, actor from set, or action from camera movement. As a further (intended) source of confusion, the computer hallucinates depth in the image, misreading the 2D film space as three dimensional. The agent attempts to configure a virtual body out of the movement it perceives on the screen, generating a series of abstract moving figures that nonetheless manage to embody an essence of the shot they derive from. In the VR version, each framed shot and analysis would be laid out as if on a very long study table, and again viewers could move close to inspect the abstract AI figures at full human (or android) scale, or they could back off a bit to compare the abstracted shot with the original, or they could step back further to see the shape of the whole.

Another set of cinema simulations would examine sound/image counterpoint. The precision of 3D sound in relation to the viewer’s movement is perhaps the most striking aspect of the actual VR experience. Sounds are placed so precisely in your personal hearing space that it seems as if you can reach out and touch their sources. One set of virtual viewing mechanisms would simulate possibilities of sound/image montage, while another would investigate methods of visualizing sound abstractly. This latter set would lead to a completely immersive VR film in which the sound of field recordings of an urban site or sites would be transformed into a vast and changing landscape through which the viewers travel. Here we push our planar projection unit to the point of complete dissolution. Note that this version sticks most closely to the new VR orthodoxy that mandates continuous virtual space and point-of-view navigation, but with the key difference that the continuous visual world will be an entirely abstract representation of sound. It will also lead to an intensely synesthetic experience not only because sound and image will be fused and focused, but also because the 3D image will seem as much tactile as visual — the viewers’ eyes will probe the imagery floating in the space before them almost as if touching its light and tracing its contours.

One other type of mechanism is worth describing here. Not only does VR allow the viewer to look and listen in any direction and at any distance desired, it also allows that path of attention to be tracked and recorded. So, for example, when we construct the abstract sound space, we can record Jacobs’s passage through it; it will almost seem as if he will virtually “film” the created space with the shifting direction of his gaze.

Aside from its intrinsic interest in VR, this also allows for the VR films and mechanisms to be translated back out into more convention framed 3D. This has its practical virtues, for the films can be examined as a group in the classroom and projected in regular film theaters.

Risks & Experimentation

The risks of the project derive entirely from its highly experimental nature. No aspect of this VR project will be created by the traditional means of film or animation — we would build our own camera and recording arrays; extend our own codebase; and collaborate within a real- time “performative” editing environment, with tools and techniques that open up the widest space for experimentation.