Visitors watch Jesus VR: The Story of Christ at the 73rd Venice Film Festival in 2016. Andreas Lenz/Getty Images Hide caption
Toggle caption Andreas Lenz/Getty Images
It’s hard to describe Marvel Studios’ What If…? — An Immersive Story. Combining elements of interactive gaming, narrative-driven film and 3D comic books, it puts the viewer – the player – at the center of a new tale that charts the destinies of Superheroes and villains from the Marvel Cinematic Universe.
To experience What If…?, you’ll need to wear an Apple Vision Pro headset. It’s considered “mixed reality,” meaning it incorporates both virtual reality (VR), which transports you to another world, and augmented reality (AR), which overlays a video on top of your real room. The story alternates between AR scenes, where characters appear in the user’s real room, and a series of dizzying VR landscapes from the multiverse.
Some scenes are passive, such as when the supervillain Thanos is put on trial for theft, but there are also many interactive scenes, such as one in which a character explains how to use hand gestures, such as making a fist, to defend oneself against enemies or cast spells.
Screenshot from What If…? – An Immersive Story. Marvel Studios/ILM Immersive Hide caption
Toggle Caption Marvel Studios/ILM Immersive
It’s a completely different experience from traditional TV or movie watching, and industry insiders believe it will change the face of entertainment.
“It’s like we’re creating a new canvas,” said Shereef Fatou of ILM Immersive, executive producer on the immersive version of What If…?
Although interactivity is central to the experience, viewers can also opt out of it.
“There’s a big audience of old-school gamers who really want to shoot stuff,” Fatou said, “and then there’s also the people who don’t want to play the games and just want to see a great story.”
Catering to a wide range of tastes
Gamers have been using VR systems for decades, but over the past decade, new headsets with more powerful graphics and motion tracking technology have begun to attract a wider audience.
The entertainment on offer today caters to a wide range of tastes: Meta headset users, for example, can watch NBA basketball games with friends using the Xtadium app, explore a haunted Irish castle in the VR live-action horror series “The Faceless Lady,” or enjoy pop star Sabrina Carpenter’s recent immersive VR concert.
Trailer for VR horror TV series “The Faceless Lady” YouTube
“Sitting courtside at your favorite basketball game or watching your favorite superhero is a totally different experience. [in virtual reality]”That’s according to Jason Thompson, creator and host of The Construct, a YouTube channel focused on consumer VR. “It just doesn’t compare to watching it on a flat screen.”
Thompson said he uses apps like Big Screen to watch traditional TV shows and movies on a headset. It can also be a social experience, if he so chooses: Users can chat with others in a watch party or mute them if there’s too much talking. It can also change the environment, so your living room couch turns into a fancy movie theater, complete with virtual popcorn. Thompson said he sometimes watches on the Big Screen bedroom setting while lying in bed.
“The screen is actually on the ceiling,” Thompson said, “and you have to lean back to see the screen.”
Lying down to watch VR content isn’t just a whim. It has a practical purpose: Most of today’s headsets are heavy and uncomfortable to wear, and Thompson said lying down reduces strain on the head and neck.
“For VR to be great, you need to be comfortable with it,” he said.
An industry gaining a foothold
Technology companies are working on this.
NPR correspondent Chloe Beltman experiences “What If…? An Immersive Story” at ILM Immersive headquarters in San Francisco. Chloe Beltman/NPR Hide caption
Switch caption Chloe Veltman/NPR
“As the years go by, as the technology gets better, easier to set up, and more seamless with other device interactions, more and more people are adopting it,” says Sarah Malkin, director of immersive entertainment at Meta, now the market leader in consumer VR. “We knew we were essentially investing in the future of computing, and that it would take time.”
Malkin said Apple’s entry into the market, with the launch of its Apple Vision Pro in February, is a good sign that headsets are becoming more mainstream, even though prices for headsets range from $300 to more than $3,000. But Apple and Meta haven’t released specific sales figures, so it’s hard to know for sure how the market is developing.
“The key to this market is consumer adoption,” said Ben Arnold, a consumer technology analyst at market research firm Circana, “because that makes app development more attractive.”
Adoption Depends on Both Technology and Content
Filmmakers say that to tell better stories in VR, the technology needs to be easy for creators to use.
“The basic ways of interacting have yet to be figured out,” said Eugene Chang, a VR film writer and director at Penrose Studios who has worked on several VR films, including “Arden’s Wake,” a post-apocalyptic ocean adventure that won the inaugural Lion Award for Best VR at the Venice Film Festival in 2017. “It should feel as natural as using an iPhone, and we’re nowhere near there.”
Penrose – Arden’s Wake – Trailer YouTube
Chung said that with many current VR services for TV and movies, people can easily become frustrated because they don’t know where to focus their attention in a particular scene, or they often can’t fully interact with the characters even when they want to.
“You see things happening, but things don’t react the way you want them to,” Chong says. “You can’t go up to a character and talk to them about Shakespeare, for example, or ask them what they had for lunch.”
But he’s excited to continue exploring the creative possibilities of this new medium, especially with so many of today’s young people growing up as VR natives.
“I have no doubt in my mind that this is the future of all entertainment and really all computing,” he said.
Jennifer Vanasco edited the audio and digital versions of this article. Isabella Gomez Sarmiento mixed the audio version.
Thanks to Will Mitchell and James Mastromarino