There is an urgent need for filmmakers, immersive theatre practitioners, and game designers to work together. 360 video and virtual reality offer promise, but there is still uncertainty about how to fulfill those promises. Many of the filmmakers stepping into the VR space are already anticipating the other side of the hype cycle: “it only has such an impact now because it is novel” said multiple creators at Sheffield Doc/Fest earlier this year. So what happens when this is no longer fresh and new?

Myron Krueger (1972) Videoplace

The imaginary of VR is heavily figured on the idea of immersion. In fact, while immersive art as an aesthetic quality goes all the way back to the 18th-century panorama, the technological notion of immersion that has proliferated in videogames originated in early VR research. Although VR pioneers such as Myron Krueger did not use the word “immersive” as they began building installations in the 1970s, by the time VR developers were meeting for annual conferences in the early 1990s, immersion was a common by-word accompanying discussions about how to make users feel a sense of “presence”.

Immersion for these VR researchers was a product of sensory information: higher-resolution displays, faster framerates, more elaborate surround sound, etc. Games developers added user agency to this list; the more you feel able to impact the world and/or your place within it, the more immersive it is. More immersion meant more presence. More presence meant that perhaps VR could go beyond spectacle, and grow into the dream of cyberspace.

This dream is a distraction. In VR just as in games, “immersion” turned out to be a cloudy idea with little immediate use for refining the work of developers. The promise of “immersion” in VR does not provide much in the way of creative benchmarks or even a meaningful goal. Any VR project can claim to be “immersive”, without stating anything about the intent or creative strategies at work — it’s something we attribute to the technology.

In the past few years, empathy has become another shallow selling point. A number of VR and 360 projects chronicle the experiences of other people, in the hope that the illusion of immersion or presence in a different place will create a stronger sense of being in somebody else’s situation. Like immersion, because empathy means a few different things at once, VR developers can claim to have provoked it simply by virtue of the technical facilities of telepresence. When empathy is not triggered by VR, it’s the tech that’s blamed for not yet being immersive enough, with the empathic tech horizon always one more feature upgrade away.

In fact, the very interactivity that is often thought to increase immersion can get in the way of empathy, as one person found when user-testing a (non-VR) game about poverty: player agency can give the impression that negative situations are a result of personal decisions. Even when empathy is triggered, the results can be counterproductive, leading either to an antisocial desire for retribution on somebody else’s behalf, or to that overwhelming sense of empathic distress, and in turn to a desire to remind oneself that this is all just Somebody Else’s Problem.

Documentary filmmakers working in VR and 360 seemed a little lost at sea at Doc/Fest in Sheffield earlier this summer. Many of the cinematic techniques that they have grown accustomed to using are not workable when the user is experiencing telepresence. Move the camera, and you risk inducing motion sickness. Frame a shot around one object, and the user might still choose to look the other way. And that’s before you even begin to get into room-scale VR such as the HTC Vive, where the user might choose to walk away from the scene entirely. Finally, the viewer may use any interactive abilities they have to disrupt the action deliberately.

Interventions by architects into film could offer a model for filmmakers moving into spatial media such as VR. Instead of concerning oneself with the frame and the scene, architects such as Steven Jacobs in his study of Alfred Hitchcock films point to an approach that centres on charged objects, familiar spaces made unfamiliar by the uncanniness of a medium, and the juxtaposition of structures arranged near one another.

In My Shoes installed at Union Street coworking space in Sheffield. Photo by Ellie Robinson

Those from the immersive theatre world know a lot more about how to do this: In My Shoes is currently one of the most effective 360 video projects out there, and that’s because it is presented as part of a bigger installation that recreates the tactile and even olfactory sensations experienced by the protagonist of this first-person film. Created by Jane Gauntlett, an artist with experience in immersive theatre, the video is organised not around the framing of shots, but the entrance and exit of characters. In My Shoes makes deliberate use of the minimally-interactive quality of 360 video and the disorienting feeling of wearing a VR headset to foreground a sense of helplessness. You feel that you are in a space that is both familiar and unfamiliar, your body is heavy, and you are running on autopilot.

Documentaries rooted in film traditions, such as Clouds Over Sidra, take a different approach, placing a 360 camera at the site of the documentary and then giving viewers the illusion of being there as a passive observer. The feeling alternates between a dreamlike sense of being present but unseen, and suddenly being aware that a passer-by is staring at you — no, not you, the ball of cameras that had been suddenly brought into their environment.

These projects reflect different understandings of what empathy is and how it is achieved. In My Shoes, as the title implies, is about experiencing a slice of somebody else’s life. Clouds Over Sidra is about temporarily feeling that you are there with someone, as they narrate to you what their life is like. In both, I was left with a sense of powerlessness; In My Shoes implied that the powerlessness belonged to the narrator, but in Clouds Over Sidra this feeling is more troubling: I know that an unprecedented number of people are braving life in refugee camps, and I have increased admiration for the resilience of Syrian refugees, but I don’t know how this knowledge will change my behaviour or political activities.

Attempts to insert some kind of agency through interactive elements are limited, and are complicated by the social politics surrounding documentary and even theatre. Nobody wants to be accused of making a videogame: some filmmakers speaking on panels at Doc/Fest even headed the accusation off before it reached them. “People asked me if I was going to make a videogame and I said of course not, haha”. Any form of interactivity in documentary media has to balance meaningful agency against the need to display the formal and stylistic trappings of a informative media. I get the impression that if one was to step too far into playfulness one risks losing status as a true documentarian.

A concrete example of this is currently touring the UK. Notes on Blindness: Into Darkness is an interactive 360 experience that accompanies the excellent movie Notes on Blindness. Both use audio from the diaries and memoirs of the late theologian John Hull to explore questions about how we experience the world around us and how we cope with the distance between us and the people who we love. The directors have said that their intention with the interactive element was to give people the opportunity to experience more of the diaries than they were able to fit into the film, and I imagined being able to spatially explore a representation of a world without sight, being surprised by the audio recordings that I find along the way. The experience is actually much more static; a predetermined set of recordings are played as digital animations are displayed in various locations, and sometimes I am instructed to stare intently at one of the animations in order for the scene to continue. Imagine a point-and-click adventure, but with only one clickable object on screen at any given time, and no ability to move from room to room. I had no sense that I was unearthing knowledge independently: it was being presented to me, and I was being scolded if I didn’t look at it hard enough.

To make the documentary interactive might require leaning into the feeling of exploring a space as an outsider, rather than always aiming for the up-close personal encounter with a single person’s experience. Owen Vince has offered a model for this in an essay for this publication, forcing game spaces to “disintegrate” in order to better understand them. I can imagine documentaries that give users the ability to see their impact on the environment, and feel their own responsibility for it — rather than simply being an inert camera-head as in Clouds Over Sidra. If giving users the ability to impact an environment through their interactions seems contrary to the need for documentary to present things as they are, consider that any system of interaction is a simulation that necessarily makes an argument about how a space comes to be. In the same way that a highly-interactive simulation of town planning reifies the power of the planner over the agency of local people, a 360 video that makes you feel present in Syria but erases your impact on the local area can actually downplay our global interconnectedness, by reifying the position of the neutral observer.

It would be reductive to say that the answer to all of VR documentary’s problems is game design. Goodness knows, videogames have their own problems to contend with, within and without the VR space. But when filmmakers ask questions such as “how do we guide the viewer to look in one place in particular without having a fixed frame of view?” or “how do we allow people to explore a wealth of information at their own pace?” then the architectural techniques developed for games seem to offer some solutions. The Doc/Fest alternate realities exhibition featured a fantastic example in Tracy Fullerton’s Walden, a 3D videogame in which the player explores an environment to unearth fragments of poetry. Walden is not VR, but it is an immersive experience that recreates how nature is known to those who have dedicated their life to giving it their full attention. It combines the active participation of game design with a sense of peacefully witnessing a part of literary history.

More people in games could take on the challenge undertaken in Walden of helping players to navigate documentary sources. Likewise, more people working in VR documentary could learn spatial storytelling from videogames.