Virtual Reality in Real Time: A Conversation

Shari Frilot and Homay King

from Film Quarterly Fall 2017, Volume 71, Number 1

Editor’s Note

Film Quarterly has been interested for some time in establishing a critical approach to works made in Virtual Reality (VR). Homay King had begun conducting interviews with Shari Frilot to that same end. FQ then invited them to make that dialogue public with a conversation on stage at UC Santa Cruz on the implications of the VR platform to share with FQ readers.

On May 2, 2017, at the University of California at Santa Cruz, Shari Frilot, Sundance Film Festival’s Senior Programmer and Chief Curator for New Frontier, spoke with Homay King, Professor of History of Art and Film Studies at Bryn Mawr College. They talked about the state of Virtual Reality today and discussed the VR and immersive media works that Frilot had curated for the 2017 edition of Sundance, as well as the platform’s implications for the future of storytelling.

Homay King: I want to begin by finding out what brought you to Sundance in the first place, first as a festival curator and then as the head of its New Frontier section.

Shari Frilot: My history is actually very organic. I started at Sundance in 1998, coming there from experimental film at the MIX festival I used to run. They gave me the Frontier section because I came there from experimental film. I was always interested in overlaying disciplines: film, art, digital technology, and cinematic expression through digital technology. We inaugurated New Frontier in 2007 and that’s when I started bringing in people from the art world, film people who were interested experimentation, and digital creative technologists, into a showcase that was socially oriented and had to do with the cinematic image. I tried to create a showcase that came from all these different disciplines but would actually be legible in a film environment.

Around 2009, as I was tracking artists for this showcase, I came across a journalist, Nonny de la Peña, who was making documentaries in Second Life.¹ In Gone Gitmo [2010], she talked about Guantanamo Bay inside Second Life; inside that virtual world that you could navigate on your home computer, she would show documentaries on the walls. That’s how she crossed my radar originally. Later, she invited me to her studio, which was in Mar Vista at the University of Southern California (USC), a DOD [Department of Defense] studio, and fitted me with a giant helmet with a big position tracker tree. The room was full of sensors. I had a giant backpack and three people following me around. This was 2011. Virtual reality had been around for a while, but this was the first time I had ever experienced it––and it was a revelation.

The piece was Hunger in Los Angeles [2012], and it was made up of her eyewitness audio accounts of a “hunger line” in Los Angeles. If I had read the story in words, I would have thought, “Wow, that happened, that’s really sad.” But I had experienced it, so I remember it as something that happened to me. I asked Nonny, “How are you going to fulfill your duty as a journalist?” and she said, “I don’t know, but I know that this is the way I want to report the news.” She had such passion.

I brought her to New Frontier in 2012. She had a seventeen-year-old intern named Palmer Luckey at the time. She was working with a $50,000 headset that had evolved over the 1990s. And USC was horrified: “You’re not going to take our $50,000 headset to Sundance.” So, she worked with Palmer. They showed up with a headset that was more or less duct-taped together with a cell phone inside. It was essentially the prototype for the Oculus Rift. I watched it happen. I watched it affect the audiences there, and saw that it was powerful, and I felt an obligation to continue not only to nurture, but also to take responsibility for, this medium that had such a powerful effect on audiences.

King: It’s fascinating that it was a journalist who first introduced you to this work. I’m thinking of the recent piece that de la Peña did, Kiya [2015]. I showed it in one of my classes at Bryn Mawr, and one of the students said, “Oh, this is so interesting. It eliminates the problem of the male gaze.” That was a way to put it that hadn’t quite occurred to me before. It’s because there’s no single encoded camera point of view.

The other feature of Kiya is that it’s a computer-generated reenactment of a 911-phone call in a case of domestic violence. So generically, we’re in the realm of documentary reenactment, but using original archival audio. And yet when you look at it, there’s another layer. I don’t know if I want to call it buffer or something else—because they’re computer-generated characters. This is in addition to the ability to look or not look, to turn and look for the police car coming or to look at the person who’s on the phone. I wonder if you have further thoughts about that and how it relates to her earlier work.

 

Frilot: You reminded me of when I first splashed into Hunger in Los Angeles. Nonny de la Peña was using Unity at the time—this was 2011, early Unity.² By film standards, it just looks like bad animation and the people looked really uncanny valley, curved teeth and stuff like that; and you, the viewer, are one of those people. But it didn’t take more than about a minute before I became a denizen of the uncanny valley.³ It was just my world, and that trumped any judgmental gaze I had on the aesthetics. The interaction I had with people, the authenticity of the audio recordings, and the situation itself were so powerful that it became inconsequential that the characters were fictional, because I was a fictional character too. It became normal for me. Not only normal: it was not only my image, but my environment. I was inside of it.

King: The question of aesthetics leads to something important: what we call “VR” is quite diverse. Some, like Kiya, can be viewed with a seven-dollar Google Cardboard on a cell phone app, some pieces require more sophisticated technology. I experienced this range for the first time in an installation by an artist named Florian Meisenberg, who created an imaginary museum using the HTC Vive, which is fully immersive, with simulcast video. It has haptic controllers so you can manipulate objects in the environment and there’s vibrational feedback. What I haven’t tried yet is social VR, like Life of Us [2017] by Chris Milk and Aaron Copeland, where there are people in the installation together. Could you say more about that?

Frilot: Some folks may not know the difference between 360º and positionally tracked VR. With 360º technology, if my fingers and I are in the same environment, then when I move, the finger moves. But in positionally tracked VR, if I move toward the finger, I get closer to the finger; it deepens the [user] buy-in. Life of Us is made for the Vive, which is positionally tracked and room-scale, and it’s social. It’s a seven-minute experience of the evolution of life on Earth. You inhabit the character, so you start out as an amoeba, you turn into a tadpole, you start to crawl on land, and then you turn into a raptor, and go all the way to the end: becoming a robot at a disco party with all your earlier manifestations. That’s your experience. But you do it with another person. So, we’re both in Vive, we’re looking at each other going through this evolution, and we can talk to each other.

The voice is mutated; it’s a part of the playfulness of it. So, you can actually meet somebody inside of VR, which is a tremendous feeling. You’re in an awe-inspiring environment and interacting: you may be pulling monkeys off the next person. It’s a joy-making machine and like a Disney ride as well.

 

King: That leads nicely to another Chris Milk piece that you included in New Frontier: Evolution of Verse [2015]. This is a VR piece that starts off in a placid lake setting. You can turn, it’s 360º, and then slowly you see a train coming in the distance. At a certain point, it turns and begins to come across the surface of the lake straight toward you. Of course, this is a reference to the Lumiére Brothers’ Arrival of a Train [1896], which some say is the first projected film; it’s often mistakenly credited that way. There’s the mythological scene that Tom Gunning and others have written about: audience members fleeing the theater at the sight of this photo-realistic train coming toward them, credulous spectators believing the image to be real. It’s probably an apocryphal story, but telling about the way we relate to images as if they were live perceptions. In Evolution of Verse, there’s also a kind of ribbon-like roller coaster, and we end up in something like the end of 2001: A Space Odyssey [Stanley Kubrick, 1968] with an infant floating in a void. The piece is interesting to me in part because it encapsulates cinema from the turn of the twentieth century, at the dawn of the medium. There is a reference to Lumiére and documentary. There is a reference to the “cinema of attractions” and trick films that show off the magic of the technology and maybe give you bodily sensations, a physical sense of vertigo. And finally, there’s sci-fi and fantasy, which is Méliès. That classic triumvirate: Lumiére, Edison, Méliès. I’m wondering if we could use that as a segue to think about the genres of VR that artists are exploring.

 

Frilot: It’s as various as you find in film, ranging from horror, told really effectively in VR, to documentary, which is interesting because you as the user are often one of the subjects of the documentary, to performance artists. Oscar Raby did a piece about his father who witnessed a terrible slaughter during Pinochet. In the VR, you are him. He’s trying to come closer to his father; there was a performative element to it. Rose Troche brings her indie cinema style to the medium and sometimes doesn’t open up the gaze all the way to 360º, but really uses the 270º or the 180º, which is a gaze that we have from cinema and naturally evolved from survival, and from the fact that our eyes are in the front of our heads. You have musicians bringing a new language to this medium, like Reggie Watts. For me, his work so gets to the wondrous possibilities of VR that go beyond the cinematic. He uses the medium to go down the rabbit hole of a sensorial journey.

King: There’s a piece in this year’s Whitney Biennial by Jordan Wolfson, Real Violence [2017]. It’s been getting a lot of buzz as a VR piece that involves witnessing a beating. It’s actually a mannequin with some post-production effects on it. But what was startling is to have that be the only VR piece in the show, because it made it seem as if that’s what VR is for. It’s so viscerally violent that there’s a kind of trigger warning before you agree to view the piece. I saw a lot of people in their headsets actually turning their heads away. To me, it is, in a way, a horror piece.

 

Frilot: It’s probably the most horrific thing that I’ve experienced in VR up to now. The way it’s constructed is actually quite conceptual and informed by art practice; I can understand why it’s in the Whitney. As in some VR, it makes you feel disoriented in the place you’re actually standing. This piece starts upside down.

King: You’re looking at the sky.

Frilot: It really ruptures you, right? And they have you holding onto a rail in this experience so you don’t fall. I would have fallen down if I hadn’t been holding onto the rail. And then it moves you into this orientation you described, the artist taking a baseball bat and brutally beating this “guy.” It feels very real. It’s on the street in New York on a curb, and there are cars in the street and no one’s stopping.

King: And you hear a prayer being sung in Hebrew. I see how that piece works with the whole Biennial, the themes and content. But to me it’s a little unfortunate that it will be a lot of people’s first experience with really high-definition VR.

Frilot: Yeah, it was on the CV1, one of the best headsets out there right now. It certainly will communicate how visceral and effective this medium is at making you feel not only present but even violated, as if you’re in danger. You’re implicated. This is one of the things that I find fascinating about VR in general and what I found interesting in seeing Real Violence. I had to watch myself respond to this piece at close range. It wasn’t separated from me. I was actually in the space and it was really disturbing. When I chose to continue on, when I chose to look away, it was telling me something about who and how I am in the world, and it was hard to face.

King: And there’s a huge value to that. My hands flew up to my eyes as if I were at a horror film, but of course with a VR headset, that doesn’t block it. You have to turn or close your eyes. What you just said now—about how it viscerally implicates us, makes us aware and more cognizant and mindful of our own ethical implication in the situation that we’re witnessing, makes us feel bodily at risk or maybe in a position to intervene—is a point I find really salient.

It’s something that I think also applies to some of the more journalistic pieces, such as Gabo Arora’s Waves of Grace [2015], a documentary piece about an area in Liberia stricken with the Ebola virus. The main character is a woman who survived Ebola, so she has immunity, which means she can work as a caregiver to those who are ill. We follow her through these spaces where we truly feel in the middle of it, not observing from a safe distance. The possibility for any kind of masterful gaze genuinely feels somewhat undone by the fact that we’re in a 360º space.

 

Frilot: That’s interesting, to use the term “masterful gaze.” Mastery is something that’s valued in this culture, and VR really breaks that down. You are not the master of your experience. You’re an explorer, a discoverer. A lot of people talk about how VR brings out the childlike in you. It’s got to be due to this: you lose all claim to mastery inside of these experiences. It’s not as if you discovered buffalo the first time you saw Bison Herd [Danfung Dennis, 2016], but you did discover what it feels like to be next to a moist buffalo, maybe even smell it. It makes you feel like you didn’t know anything before, even though it’s something you’ve seen on TV many, many times.

 

King: And to emphasize what is different about VR, compare and contrast that to Planet Earth or any traditional style of nature documentary. There’s still the sense of wonder, there’s still an intense proximity to the animals, but because you’re sitting in your chair over here, and you’re seeing these wonderful sweeping vistas over there, it feels very different than being in the middle of a herd of these large animals, becoming one of them, in a way.

Frilot: With books and photographs and movies, the domain of mastery is always there because you can put the book down. You can close your eyes in VR, actually. But we’re not quite there yet; people don’t choose to do that, unless they’re sick. If you sign up for it, you’re signing up for letting go. So what does that mean in terms of how stories will come to construct the real?

King: I want to return to something you said the first time we ever talked about VR: you said reality is being reconstructed. I asked you if subjectivity itself is being reorganized. Is there something happening that is potentially on the level of the invention of photography, the invention of cinema, the invention of snapshots and portable cameras? All of these things created shifts in our perception of the world, our sense of what we have access to, and, in a way, our political consciousness. We’re just barely on the cusp of it with this particular medium, but do you have thoughts about directions it could go, good and bad? Do you think it’s on a par with the invention of cinema, or is that going too far?

Frilot: It’s there. It’s like the printing press a little bit.

King: That was a big one!

Frilot: And cinema too. It reorganized society. I think the medium of VR has the power to do that. It’s not there yet because of the points of access that make it too expensive right now. Google Cardboard is a sort of gateway to VR; not the greatest, but it’s out there. As I’m watching this field evolve, it takes a quantum leap every three months. Artists are taking it on in different ways. And it’s not just in the art world, it’s come into medicine, too. Medical use is 25 percent of the VR industry, which is interesting for me. How do medicine and storytelling overlap? How much of medicine is storytelling? This shift has the potential to affect not only medicine, but our basic understanding of ourselves, including our anatomy and physiology.

King: And what it is to be a vital living creature? This is fascinating. It reminds me of one of the pieces I wanted to talk about with you, Notes on Blindness [2016]. It’s a VR adaptation of the journals and audio recordings of John Hall, a man who kept a diary of his experience of losing his vision. He talks about certain ways in which the world shut down for him and he would find himself in a state of panic, but also other ways in which the world actually opened up for him: how he came to love it when it rained because it created a whole topography around him where he could hear things near and far. In the VR piece, you enter that world through binaural audio, so you’ll hear something and turn toward it, and then a loose outline of an object comes into view, mirroring that process of Hall’s hearing where things were. In a way, you’re almost seeing through audio. That piece fascinates me because the visuals are extremely sparse, but it makes really good use of directional audio. I thought it was an innovative way to use the medium.

 

Frilot: Over time this man found a way to see through his ears, out of necessity and hardship. And in just one piece, you’re able to understand what it’s like to see through your ears. And it is made possible in VR through this new convergence of a language that speaks not only to your eyes and ears, but to where your body is at.

King: Exactly. Your proprioceptive self is included. You could use it as a way to train people to live with disability or to train their other senses. Notes on Blindness isn’t built that way, as it’s a memoir, but I could see an application for that.

Frilot: Essentially it’s a shortcut to learn how…

King: To see with your ears.

Frilot: To see in sonic holograms. That really points to the density of information this medium can communicate. You know, I showed a short documentary [Scientists Have Found a Way to Make Paraplegics Move Again, Michael Tabb and Ananya Bhattacharya, 2016] in this year’s festival because I wanted to root the show in other things happening with VR, to point to what in this medium goes beyond entertainment. It was a documentary about some researchers from Duke University based in Brazil working with paraplegics. They created an exoskeleton, a machine into which they put the paraplegics along with a VR. Through the VR, the patients could control the exoskeleton. And an unexpected thing happened: after about a month of training, the paraplegics started to move their legs again. Every single person who went through this training was later reclassified as not entirely paraplegic anymore. They gained some control over their legs, even if they couldn’t necessarily walk. This is profound.

King: It’s miraculous. It’s genuinely miraculous that the mind-neurons-body connection could literally be rewired by images coordinated with certain kinds of movement. But I also worry about the dystopic potentials of that rewiring. It can easily lead to: “Let’s use this to train soldiers not to feel anything.”

Frilot: It speaks to the mental plasticity that we all have. And it hasn’t been touched only by VR. Games have already been used by the military to train military.

King: And recruit.

Frilot: There’s an assumption that this could also happen with VR, but keep in mind this idea about mastery. I really love that you brought this up, because things can go awry when you package what you learn with a sense of mastery. In VR, you have no sense of mastery; you have an enormous amount of vulnerability. Is that a bad thing, a scary thing, or is that what turns the tide? Is it an empowering thing, in terms of re-arming ourselves with our own humanity?

King: This is related to something else you said earlier—and I think we were talking about VR in general—you said there was a moment where you observe yourself being scared, and you see how you’re scared. That brief pause in experience where a part of you knows this isn’t real, and it gives you a little cushion of space to be mindful, have awareness, to observe your own, in some cases, nearly instant reflexive reactions. And then of course I can relate that to the whole genre of guided meditation VRs, some of which are on the New York Times app.

Frilot: It’s a genre right now: retraining mind and body in a way.

King: Exactly. So, stepping back a bit, here’s a story that gets to this question of mastery in another way. I’ve written about how the term “virtual reality” first entered the lexicon.⁴ It was in 1987 in an essay in the Whole Earth Review.⁵ It was written by Yaakov Garb, a UC Berkeley doctoral student in mathematics. It’s a Roland Barthes–style critique of advertisements for computers. He says, look at these godlike figures whisking numbers through gridded space: it’s an image of the computer as control. The slogans are things like, “The Power of the World at Your Fingertips.” It’s all about being the grand wizard; the computer allows you to become the master puppeteer of the world. He critiques these ads, saying they contribute to the fantasy that computers allow us to be omnipotent. That fantasy occludes the realities of how the hardware is made, where the energy’s coming from, and so on. He thinks it’s a dangerous fantasy.

But if you go back to 1398, which is when the word “virtual” first appears in English, you get an almost opposite meaning.⁶ It means, “Possessed of certain physical virtues or capacities” and “effective in respect of inherent natural qualities or powers,” even “capable of exerting influence by means of potentials.” That was the meaning that I wanted to recover, and see if there are works of digital art, video, etc., that are doing this.

Frilot: Well, you know, both are terms that implicate values: the values that you extol or that you bring to the technology. I have a hard time thinking that a technology is bad or good on its own. You have to use the technology and your values will determine how you use it. The later definition, the one in the 1980s, very much describes our relationship to our cell phones, how we use the cell phone to master our world, to magnify our presence. We’re always the top, very rarely the bottom, in that relationship with the cell phone. The volume of information, of course, can be overwhelming. But we never throw in the towel. No, it’s always a fight to stay on top of things and to stay the master of it.

In the end, I think technology is an extension of humanity. Technology has formed how our minds and our bodies have evolved ever since technology began. The arrow helped us kill animals and fire to cook it, and our brains started to get better and denser. And there’s a feedback loop: the technology and the animus form a continuum. I think we get into trouble when we think that technology is separate from us. That affects the kinds of choices we make. It’s my third arm or my left hand. Would I endanger my left hand? What situations would I put my left hand into? If you truly have that visceral relationship to the technology, you start to use it in a different way. It becomes an agent of your vulnerability and presence. And you get a different design.

King: The design affects not only our sense of ourselves, but also how we perceive the world around us, what our relationship to it will be.

Frilot: Notes on Blindness is a great example of that, or The Click Effect [Sandy Smolan and James Nestor, 2016], which is about how dolphins and whales communicate to each other in sonic holograms. They send clicks, and they don’t hear through their ears; they receive a higher sonic hologram in the gel underneath their chins and they can see in 360 degrees what is being communicated.

It would be hard to explain that or know what it feels like in a “flattie.” That’s what traditional media (film and television) is called these days—a “flattie.” But in VR, the ability to explain what it feels like behind and in front of you takes the opportunity for storytelling to the next level. That’s why we need cinema studies involved. We desperately need critical thinking, deep thinking around what this medium is and what it can do.

Header Image: Hunger in Los Angeles (Nonny de la Peña, 2012).

Notes
1. Editor’s Note: Second Life is an online digital world that was launched in 2003 by Linden Lab. See secondlife.com.
2. Editor’s Note: Unity is a cross-platform game engine that was originally developed by Unity Technologies and first made available in 2005.
3. Editor’s Note: The uncanny valley is a term coined by Masahiro Mori in 1970 to describe the sense of eeriness and revulsion elicited by human replicas that appear almost, but not quite, like living human beings.
4. Homay King, Virtual Memory: Time-Based Art and the Dream of Digitality (Durham, NC: Duke University Press, 2015).
5. Yaakov Garb, “Virtual Reality,” Whole Earth Review 57 (Winter 1987): 118–20.
6. Definition, Oxford English Dictionary.