What is virtual LED production technology, why is it changing Hollywood, and how can filmmakers best shoot on a next-generation LED stage?
Find out in this interview!
Imagine stepping onto a movie set that lets you capture a golden hour that lasts all day, change the weather, move photorealistic mountains in the background, blockbuster-quality VFX creature interacting with actors, or looking for locations around the world Being able to search the world without going into space.
Now imagine you are doing this completely in real time.
This is what happens when you use the virtual LED production technology. And the good news is that it’s no longer just an invention of our collective imagination or even a demo at trade shows.
Thanks to its successful use on projects like The MandalorianThis technology is real, on its way to mainstream, and is already changing Hollywood.
But what is going on under the hood to bring this cutting-edge technology to life, and what should filmmakers expect when shooting on a virtual LED production stage?
We speak to one of the technology pioneers at Disguise to see how it all comes together Find out more below!
What is LED-based virtual production?
Basically, virtual LED production is what happens when filmmakers can use a combination of technologies to replace their green screens with walls of LED panels. Using a game engine, these LED walls display real-time backgrounds and visual effects right on set.
“This creates a far richer experience for the entire film crew than using green screens,” said Ed Plowman, CTO of Disguise.
Over the past three years, he has developed an LED-friendly virtual production workflow that has enabled companies like Framestore, Lux Machina, and High Res to deliver films, including Solo: A Star Wars Story and live-action commercials for Nissan and Hyundai.
“By using a virtual LED production phase, filmmakers can see and capture all of the final pixels of a shot in the camera on set,” said Plowman. “All without waiting for the post.”
According to Plowman, some of the other benefits of using LED steps include:
- Post production teams no longer have to spend time typing on green screens.
- Post production teams automatically have correct reflections and contact lighting on actors or physical set pieces, and no more green spills.
- If a director wants to change the angle of the light so that the shadow on the actor goes in a different direction, the director no longer needs to move real lights. All you have to do is move the sun in the virtual LED environment.
- Actors no longer need to just visualize the final movie with a green screen for reference. Your eye lines will be more accurate when looking at Computer Generated (CG) elements.
- With LED floors, virtual notices can be placed on the set for everyone.
- Directors and cameramen can see, change, and draw background positions and visual effects in preproduction (even with virtual site scouts) so fewer re-shots or iterations are required in the post.
- There is no need to travel. Every world can be rotated from one place.
- Large-scale set extensions can be created that blend seamlessly with the LED background in the post, just in case the LED wall doesn’t fill the entire camera view.
- Dynamic augmented reality elements can be added seamlessly for LED assets that need to move in front of your actors.
How does LED-based virtual production redefine the filmmaking process?
Virtual LED production is revolutionizing the possibilities on set – but it is also redefining our production pipelines.
Work normally associated with post production needs to be moved to preproduction to create images for the LED screens. This requires a massive cultural change. Instead of fixing it in the mail, the graphics need to be ironed out before you even go on the first day of shooting.
For Plowman, however, this simply goes back to the way moving image pipelines were originally created.
For example Disney’s multiplane camera rig which was used for in 1937 Snow White and the Seven DwarfsIt was a camera pointed down at several layers of glass with background graphics painted on it.
These panels would then be manually moved past the camera individually at different speeds and distances to create a realistic feeling of parallax or depth. Forest sequences would feel more real, as the trees would move past the camera faster than the moon, for example.
“In early films, all of these effects would occur in-camera, on-set through tricks of perspective,” Plowman said. “That’s exactly what we’re doing – moving effects creation back to film sets. Only now are we applying technology to them.
“With an LED stage structure, you still have your rear and front planes. But instead of pre-drawn, two-dimensional glass surfaces like the one used by Disney’s multiplane camera, planes will consist of layers of real-time content from a game engine and the physical actors and set pieces in the real world. ”
Thanks to real-time tracking data, today’s advanced virtual production technology also understands the spatial layout of the stage and the position of the camera in relation to the screen.
“This means your images are constantly being recalibrated according to the real-time position of your camera so they have the correct perspective distortion to look accurate for the final shot in the camera,” said Plowman. “In other words, you don’t have to worry about the panels looking like a simple flat panel display. It will always feel like a real three-dimensional space that you can zoom, pan, tilt and venture through. ”
And only if you slide pictures directly onto the LED wall and then record them in the camera.
“You can also add augmented reality elements to the scene as extra planes sitting in front of your actors,” Plowman said. “You can create a 3D kite that flies through the scene, seamlessly switching from the LED background to flying in front of the actor and then back to the LED again. And you can use the dragon’s LEDs to light that actor, or even get the dragon to do some basic interaction with the actor. ”
Breakdown of an LED Virtual Production Setup
There are several technologies that can form a state-of-the-art virtual LED production setup. Some of the basic elements are:
LED panels. These can be bright up to 6000 nits. In the early days of virtual LED production, many sets looked like an open cube with LED floors and walls.
“It turns out this is the worst configuration you can have,” Plowman said. “There are two lights fighting each other in the corners. So if you look at it from the wrong angle, you will see a seam.”
Modern LED levels – or volume levels – now usually have one Mandalorianscurved screen. These wraparound layouts are known as the “cave” and can come in large sizes.
“I’ve seen stages big enough to drive a car between the two sides of the curved LED screen,” Plowman said. “Fortunately, the car was on a track so the LEDs couldn’t be damaged!”
The quality of the LED panel normally depends on what is known as the pixel pitch, which describes the density of pixels (LED clusters) on an LED display that correlates with the resolution.
The real cameraon which a tracking device such as Ncam, Stype RedSpy or Mo-Sys StarTracker is located.
Generative or real-time content from a game engine like Unreal. This content can be entirely CG or a real location scanned with LiDAR or photogrammetry. This real-time content is then fed into the LED screens.
Content mapping software and hardwaresuch as the rx and vx hardware units from disguise and the r18 software. These help render the graphics from the game engine and display them accurately on the stage with minimal latency. The system ensures that the LED graphics are always adapted to the place where the camera is located in three-dimensional space. Content can also be fed in via an ACES pipeline.
Tips for recording on a virtual LED production stage
As with any disruptive technology, the workflows have to change in order to take photos on a virtual LED production stage.
“There are cultural barriers to overcome between people on set,” said Plowman. “Usually one person takes care of the traditional lighting and another takes care of the media server and LEDs, and then the cameraman. If the cameraman wants to set up a stage in a certain way and the person in charge of the media server / LEDs tells them it can’t because the LED doesn’t work, this is a problem. ”
According to Plowman, the most important thing to remember is that recording on a virtual LED production stage isn’t like showing up to record something physical – and it’s not an all-in-one solution for every single species of recording.
The best way to know what features it can offer is to take a day in advance to test what a virtual production volume can do before starting a project.
“There are many sequences that still need to be captured in the field,” said Plowman, “but there are sequences – like those that would include a green screen – where using LEDs can completely change your workflow for the better. The only thing to remember is that for the LED shots, you need to do the work in advance. There is less post-processing to do, so you need to prepare to do a lot of pre-visualization in advance.
“If you do that right, you’ll have more creative freedom that day than if you tried to fix everything in the mail. You can see the last shot there and then, combining both CG and the real world. If we talk about film magic, that’s what I would imagine. That immediacy and impact of seeing the whole picture on set instead of waiting weeks to see it on screen. ”
What’s next? Learn more about virtual productions
Learn how to prepare to work as a cameraman in the band. Check out the options for LED backdrops (and here’s another one). Find out more about how this technology was used The Mandalorian.