Virtual production relies on a diverse stack of technologies. Bringing them together is always a challenge of integration, but there’s almost no part of that stack which is as fundamental as the LED video wall itself. It defines the working space for actors and crew as well as the limits of picture quality that set the limits of camera placement and lens choices. The rest of the installation will often be designed around the display, so that the type of video wall panels used, their capabilities and placement, will have a huge influence on the entire facility.

Defining The Requirement

Spaces using a green screen, with real time visual effects preview, have been called virtual production studios. That approach lacks the benefits of in-camera visual effects - the realistic integration of transparent, reflective and out-of-focus subjects. The solution to that necessarily involves a video wall, although the accompanying technology might vary depending on the scenes to be shot, how the camera will move, and other factors.

For instance, some of the best-known applications of virtual productions used LED video walls, but did not use camera tracking. Instead, they relied on limited camera movement to keep the illusion realistic within any one shot. In that situation, the background image might be rendered in real time, but it might also be based on live-action photography. For common requirements, such as driving scenes, libraries already exist to provide background plates, especially for congested cities where uninterrupted driving plates may be difficult to shoot.

With a more mobile camera, the background image must change to create the proper perspective based on the camera positioning. In that case, the camera position will be established by one of several real time tracking systems, requiring extra equipment around the stage. The image will be rendered in real time by computer software such as Unreal Engine, which might run on several high-powered computers in a nearby server room. The design work is similar to a conventional three-dimensional environment build for visual effects - in fact, the two might indirectly share assets. Camera movement is limited only by the size of the space and the design of the virtual world.

A hybrid approach, with 2D live-action material used inside a three-dimensional virtual world alongside true 3D objects, provides some of the benefits of both.

Because of this flexibility, production facilities can be built to cover a huge range of scenes - from sit-down interviews to vast action scenes. Some of the busiest have been modestly-sized setups configured for in-car conversation scenes, perhaps requiring nothing more than a rectangular display or several displays each smaller than many cinema screens. Compact configurations have even made transportable for use on location or at an existing studio.

At the other end of the scale, virtual production studios built to depict big, expansive areas often have large video walls built in cove shapes, to cover a wide angle of view. Some configurations might have overhead or floor displays, which might have different performance from the video wall itself. Some ceiling and floor panels might be suitable for use in vision; others might be intended more to cast light on the action. Other facilities might have wild (moveable) sections of video wall to accommodate special setups, plus spare panels to be placed as lighting or reflection sources.

Essential Equipment

A virtual production facility might be large or small, used for broadcast or single-camera production, or both. All of them will use a wide variety of technologies alongside the video wall to create a closed-loop display system. A scene will be generated, displayed, photographed by a camera, and displayed again on a monitor for critical evaluation. The choice of display interacts with all those processes.

Cameras may be of a known type, in the case of a broadcast facility.  A rental stage might see many different types. Either way, there are considerations about resolution as well as moire patterning from the tiny repetitive elements of the display, which will affect how the screen can be photographed in terms of focus, field of view and distance. Larger sensors, making it easier to keep the screen out of focus, are sometimes preferred.

Lighting must interact believably with the virtual environment. Some modern video wall panels can produce light of high color quality which can be a source of illumination which integrates perfectly with the virtual scene. Even so, lighting directors and cinematographers invariably augment that with traditional film and television lighting, creating a demand for space and access that can conflict with screen design.

Monitoring will be crucial for evaluating the match between the screen image, the real world, and any additional lighting.

Tracking Equipment can operate in one of two main ways. An inside out system uses a witness camera mounted on the taking camera to photograph markers placed around the stage. An outside in system places cameras around the stage which photograph markers placed on the camera, like a system intended to capture actors’ motion. In either case, there will be markers and cameras which must see them, perhaps influencing the layout of the facility.

Servers might supply live-action footage or render 3D content. For any significant display they will occupy several full-height racks of equipment, with attendant power supply requirements, plus a nearby workstation for a technical administrator. Generally, the video wall will be broken down into sections not much larger than a conventional video display, often around 4,000 pixels on the longest size, with the servers precisely synchronized.

Other servers might deal with tracking data, with the interaction between rendering and tracking depending on the specifics of the software involved.

Processors are responsible for converting the output of the servers into signals which the LED video wall panels can understand. Often, processors will be racked alongside servers, and connected to the display panels via a significant cableway.

Receivers in each video wall panel are often supplied by the same manufacturer as the processors. The capabilities of receivers vary, but might include features such as storage of calibration data, ensuring each panel matches to create an invisible join with those nearby.

Video wall panels are the basic display element, often a few hundred millimetres on a side. Each panel is often fairly simple, electronically, relying on the intelligence of the processor and receiver to be useful. The electronics, physical arrangement and optical characteristics of the LED emitters, however, determines some of the most crucial aspects of a virtual production facility’s capabilities, including brightness, dynamic range, resolution and frame rate.

Panel Fundamentals: Resolution

Both financially and practically, and more than any other piece of equipment, the video wall panels determine what a virtual production facility can do. Often, they will be selected for enough resolution to satisfy a given range of camera positions, bearing in mind the available space and intended scenic layout.

LED panel resolution is specified as the distance in millimetres between pixel centres. It’s perhaps the most important characteristic, and it affects more than simply the ability to display a sharp image. The emitters, arranged in a rectilinear grid, are a potential cause of moire patterning. Some cameras are more susceptible to this than others, though often the wall will be rendered out-of-focus, which can avoid the problem. In broadcast television, conversely, the display may include graphics which must appear sharp.

Sometimes, the image displayed on the wall itself will be a blurred, out-of-focus representation of a scene. This allows the video wall to accurately simulate objects which are more distant from the taking camera than the screen itself, and which in reality would be more defocused than the screen surface.

Because of these and related situations, it is normal for the image displayed on the video wall to be at a lower resolution than the panels themselves could display. The closely-spaced emitters reduce the susceptibility to moiré and make the display easier to use in a wider variety of setups, especially at high f/ numbers and long focal lengths, or when focusing near to the screen.

Panel Fundamentals: Brightness & Contrast

LED displays have high brightness and contrast, a large part of the reason they’re so suitable as displays for in-camera visual effects. Virtual production can be seen as a development of back projection, which used a white projection screen. No matter what might appear in the rest of the scene, that white projection screen risked being illuminated by foreground lighting, compromising contrast to the point of highlighting the artifice. As a result, the technique was only really effective in low light scenes.

LED video walls, conversely, are mostly black. The emitters may involve optical plastics, which are often the most reflective part of the panel. While no panel is entirely non-reflective, better panels use finishes designed to minimize problems when interacting with other light sources. That helps to maximize dynamic range, the difference between maximum white and minimum black levels. Modern cinema cameras invariably have more dynamic range than a video wall panel, so that the maximum brightness of the video wall may not be enough to drive the camera to its own peak white. Because of this, camera teams will generally configure their monitoring to approximate a final grade which allows the video wall to achieve what looks like a clean peak white.

Nonetheless, compared to historic display technologies, LED video walls are very powerful. The display technology used for virtual production is derived from designs made for outdoor applications which must compete with sunlight. Very often, a video wall at full power is too bright to balance well with the sort of lighting habitually used in modern single-camera drama production. This consideration interacts with the other critical characteristic of LED panels: sheer speed.

Panel Fundamentals: Refresh Rate

The LED emitters on a panel are not dimmed by driving them with lower currents. Instead, they use pulse-width modulation, switching on and off rapidly to control average output. 

Fine brightness control is therefore affected by how fast individual LEDs can be switched on and off, which is expressed as the panel’s refresh rate, often a number such as 3,840 or 7,680Hz (these values are only coincidentally the same as the pixel dimensions of a UHD or 8K video frame).

This means that as frame rate increases, there are less pulses per frame, and the fineness of brightness control decreases. The right refresh rate specification - higher is better - will therefore be important depending on the intended application. High frame rates, or techniques involving hidden tracking data or multiple simultaneous camera angles (discussed in more detail below) will require higher refresh rates.

For reasons of electronic practicality, LED video wall panels are also multiplexed. This means that not all of the LEDs are illuminated at once; instead, they illuminate in sequence, sometimes as one row of four, eight or sixteen. The higher the number, the fewer LEDs are illuminated at any time. Lower numbers are better here, potentially increasing brightness and the ability of the display to avoid flicker or banding at high frame rates or shutter speeds.

Panel Fundamentals: The Physical Build

Finally, a virtual production display is a large, complex construction demanding proper structural and safety engineering. Different panels may have different approaches to mounting and alignment, and different means for to connecting and distributing power and data. Where there are wild or independent wall sections, or particularly for a portable configuration, the ability to quickly rig and de-rig panels, or replace them, may be important.