by James Delhauer
Virtual production has changed the way many jobs are done on set. In the 1930s, the “process shot,” a filmmaking technique where actors performed in front of a pre-filmed background, was accomplished by hanging a curtain and using a rear projector to create the illusion that they were standing on location instead of in a studio. Today, the same effect is achieved using LED walls to create more immersive illusions.
Dmitry Kovalev has worked for Stargate Studios for the past three years and has been a member of Local 695 for the last year. Since late 2022, he has also taught a virtual production playback course for Local 695 to share knowledge of LED playback with fellow members. During this time, the majority of his work has been playback for LED driving process shot scenes —scenes in which actors in cars or other vehicles are shot on a sound stage. By positioning the car in front of a large LED wall and then placing OLED television screens around it in strategic places, Production can realistically simulate the effect of driving through a city or the countryside or anywhere else imaginable, complete with reflected stoplights, sidewalks, and pedestrians in the car’s windshield. This saves production the trouble of hitching a car to another driving vehicle and operating on an open road. It is cheaper, safer, and offers more creative control to the creative team.
The first version of Stargate Studios’ current car rig configuration was born during the prep for White Noise back in 2021. Over the next three years, they have refined their setup and have done LED driving shots for shows like Your Honor, The Lincoln Lawyer, The Consultant, and many others. To ensure the best quality of the final product, they prefer to control as much of the process that leads to the LED driving shoot as possible, and it starts with shooting the plates.
Stargate Studios has an in-house team dedicated to acquiring and creating plates for virtual production, led by Stargate CEO and Local 600 member Sam Nicholson. When Stargate began developing its shooting workflows for plates, it was always with the idea of making life on set easier for Dmitry as the Playback Operator. The team worked in collaboration with Sony to develop their CTRL+CAM application that allows an operator to control multiple 8K cameras simultaneously. With this app, the team can make global setting changes (aperture, ISO, shutter speed, card formatting, etc.) on all units to ensure continuity across all plates, and all media is perfectly synchronized by timecode. The app has also integrated GPS data into the camera report it generates, which draws the map for each plate. Sometimes clients only need generic city-driving footage, but more often, they request a specific route. Having a map for each take gives a quick-and-easy overview of what was shot, helps navigate through available plates on set, and saves a lot of time in the reviewal process. All of this gives the on-set playback operator the most flexibility possible to do their job.
On set is where the magic happens. Virtual production is a mixture of different skills and crafts, and the playback operator sits at the heart of that mixture. To do this job, Dmitry has to understand how his work interacts with the work of the on-set camera and lighting departments, and how to communicate his needs to them while being receptive to theirs.
When the big rolling LED screens surround a vehicle and multiple cameras frame up for the next shot, he starts aligning the correct plate for each display and each camera. It may sound trivial, but in this case, the devil is in the details. What makes or breaks the final image is the position of the horizon in the plate, the alignment of the points of convergence in the plate and in the camera, and even the rotation angle of the background. This is similar to compositing in traditional visual effect workflows, but unlike traditional VFX, there is a very limited amount of time to align it on set.
This is what a big part of Dmitry’s class for Local 695 is devoted to: properly positioning the background in the frame. To create the illusion of the car driving on the street, they need to match the field of view of the lens and properly position the plate in the frame in a way that matches the horizon and the vanishing points.
Another important aspect of virtual production is color. Nowadays, it’s common to find video walls that display colors accurately enough for cinema cameras to capture them without issue, but an operator needs a fundamental understanding of color engines, color spaces, and color theory to do their job effectively. If there is a misalignment in the color space of the footage and that of the display, the plates may not look correct on the day of shooting. Sometimes there’s just no helping the properties of the display itself or they are working with multiple displays that all have different color properties simultaneously. When this happens, it’s essential that an operator understands the basics of color-correction so that all media can be made to match, regardless of what it is displayed on.
But color can interact directly with lighting. An often-overlooked issue is the black level of the footage. Dmitry told me that many operators that he has trained will notice that the shadows on an LED wall are milky and washed out, leading them to “crush” the blacks to create a cleaner image, but while this results in an increase to the contrast of the image in the mid-to-low spectrum, the issue persists. That’s because the issue was never a color issue to begin with, but rather resulted from light spill of nearby lighting fixtures. Conversely, large LED displays do output a fair amount of illumination—so much so that a playback operator may need to brighten or darken the image so as not to negatively impact the exposure of the camera that is capturing the scene.
This is part of why an LED Playback Operator needs to develop a working relationship with the Gaffer and on-set lighting team. Additionally, a scene inside a moving vehicle needs kinetic light; shadows from moving trees or city lights at night or anything else that should affect the environment inside a vehicle surrounded by windows. There are a lot of ways to achieve the illusion, but unless you’re driving through a barren landscape with no trees, clouds, or turns, the light has to move.
This is so fundamental to the process that another member of his team, Jon Craig, has joined Local 728 in order to work on set. His job involves working with Dmitry to synchronize the video playback with Stargate’s proprietary Kinetic lights. This creates the appearance of moving lights on the windshield and the performer’s face to help sell the illusion that the car or vehicle is in motion.
With the technology evolving and a lot of groups out there figuring out economical ways to use virtual production for films and TV shows, it’s exciting to be in the midst of the development of these approaches and participating in setting quality standards. They’re at an intersection of technology, visual effects, color science, playback and lighting, and to do this right, it’s very important to understand how all of this works together.
Dmitry, Stargate Studios, and Local 695 will continue to work together to train new generations of LED Playback Operators. As virtual production continues to expand and become the norm, these skills will continue to grow in demand. Anyone interested in pursuing this work is encouraged to brush up on platforms like DaVinci Resolve, the Unreal Game Engine, Disguise Media Servers, and any other emerging virtual production tools.
Local 695 members who are interested in Dmitry’s Stargate Studios LED playback training are encouraged to contact the Local’s Director of Education & Training, Casey Weiss, at CaseyW@local695.com.