Amazingly enough, semantics are important on this matter, as many authors and vendors are including concepts that often only describe partial aspects of VR. So, here’s our two cents to shed some light on the concepts, the possibilities these technologies provide for virtual production, and the pros and cons of each approach.

We will not consider interactive content consumption (via VR glasses, for instance) in this article, as it adds an additional layer of complexity that may require a whole document for itself.

The Virtual Reality... What Is XR?

All the synthetic or virtual “realities” mentioned in this document are the result of combining 3D computer-generated (CG) imagery with real feeds, and not just standard chroma keying over images or video. By using 3D CG imagery, the resulting camera views display a fully immersive world where cameras move freely in 3D space, compared to the relatively constrained camera movements of typical video compositing.

So, we can start by defining Virtual Reality (VR) as the method that displays computer-generated images along with real feeds. Therefore, by extension, we could say that traditional Chroma Keying could represent a form of Virtual Reality, as the resulting image is obtained by compositing different images, but we will not consider it VR as we already stated the requirement of using 3D environments.

The Virtuality Continuum concept, as defined by Paul Migram and Fumio Kishino in 1994, is defined as a continuous scale ranging between the completely real (reality) and the completely virtual (virtuality). If we apply this concept to the audio-visual content creation, we can then consider VR as the “container” for all the rest of the synthetic realities, from virtual sets to AR, ER, XR or IMR, which we detail and classify below based on the mentioned virtuality continuum concept.

So, Extended Reality (XR) can be assimilated to Mixed Reality (MR) which we can define as the combination of real and virtual combined environments. Currently, for broadcast and film content creation, XR is understood as the combination of virtual environments projected in a LED wall, and sometimes LED floors, in combination with real characters on stage, sometimes including other graphic elements in a sort of IMR. However, LEDs are just a display method, and there is no technical reason why the LEDs can’t be replaced by a Chroma set for creating XR content.

VR/AR/XR Workflows And Setups

Virtual Set

A virtual set is a 3D digital background on which we integrate a real talent captured from camera in a chroma set, and then keyed out. Virtual sets can be trackless, TrackFree™ or tracked, and using one or more cameras, and one or several workstations to render the scenes, depending on its complexity. But in all cases a camera captures the feed of the talent(s) on a chroma set and sends it to a computer, which will render the keyed feed in a 3D virtual set.

Augmented Reality (AR)

AR integrates in-context information, graphics, or 3D virtual elements on top of real footage. Camera tracking is mandatory to ensure the added graphics are perfectly integrated in the original footage, both in movement, perspective, and look. The camera captures a real scene, and a workstation adds the synthetic elements on top.

Immersive Mixed Reality (IMR): Combining Virtual Sets And AR

Sometimes the real environment is not suitable for our content requirements, so we must enhance the real scene with virtual backgrounds, along with the AR elements. This enhancement can mean just the inclusion of a virtual set for the talent to be placed in, or a variety of virtual elements placed on top of the real world environment. The basic setup is similar to the AR one, but increasingly complex scenes and objects may require, in most cases, a render per camera approach.

Extended Reality (XR) Using LED Walls

When using LED walls/floors for an XR project, users can integrate a talent with a virtual environment without a chroma screen, as the background image is directly rendered and displayed on the LEDs, and the camera captures the “composite” scene of the LEDs plus the real talents. The background render is previously sent to the LEDs based on the camera tracking information, so it will display a correct perspective based on the camera position and parameters.

LED-Based XR: What Can We Do With It?

Regardless of using LED screens or chroma sets, the available XR applications are quite diverse, however if we focus on LED-based XR the basic scenery we can create are virtual sets, set extensions or in-screen AR, which can be further developed by using additional AR graphics on top of the real footage.

There are many possible LED setups, from single, flat videowalls of any size, composed walls with different flat screens placed at angles, corners or LED floors, or even shaped and curved screens. The typical workflow involves one camera, as multi-camera live productions with LEDs have additional issues to deal with to perfectly match the cameras with the background rendering.

As each camera requires its own render for the LED screens based on its point of view, this render needs to be displayed on the LEDs prior to be captured by the camera. This can’t be immediate as renders imply processing delays, so multi-camera productions will require switcher macros, or LEDs displaying the content at a multiple of the camera frame rate, so the different cameras can be adjusted to see the signal designated for each camera, by using shutter and phase adjustments.

LEDs proved their benefits with advertising, drama and movie productions avoiding the need for chroma sets, and they significantly improved the impact of live shows such as concerts and other mass events with live audiences.

In addition to displaying a virtual background on the LEDs, this can be further expanded to include AR graphics on top of the scene or even adding Set Extensions to the real set, by virtually expanding the background outside of the LEDs. However, this can only be seen by remote audiences and not live on set. 

LED vs Chroma: Choose Wisely

As said before, there is no technical reason why the results obtained for XR applications using LED screens or chroma sets can’t be similar. However, although we can reach the same result by applying any of these methods, both have their own advantages and disadvantages we must consider when choosing one or the other prior to prepare an XR production.

LEDs work better for a live production, as the audience can see the “composed” result live, while using a chroma set will require additional infrastructure for the live audience to see the final content (additional screens, content sent to smartphones, etc). However, content sent to a remote audience can use both approaches, as they will only see the composed result.

One of the key advantages of using LED-based XR is to avoid chroma keying by creating an “immersive” environment where color, reflections and refraction of real objects work as in the real world. This technique, then, resembles the traditional filmmaking that used projections or painted walls, and the scene is captured “as is”, not requiring further compositing.

However, this also implies some disadvantages, as later adjustments can be time consuming, as they would be with real shooting – one of the reasons why chroma keying was developed! Also, LEDs are better for studio production or controlled events, as it is impractical taking a LED wall outdoors for remote shooting, while we can use chroma keying anywhere.

Also, LED walls bring in additional issues such as moiré due to the pixel pitch, focusing issues, incremental delays when adding AR elements, and, most importantly, shadows. LEDs are light emitters, so it is difficult to drop real shadows on them, which is quite noticeable when using LED floors.

At the end of the day, both technologies have their own pros and cons, and it is the nature of the production we are planning which will define which technology fits best with what we want to achieve. Of course, both technologies do not exclude each other, and can be combined in the same production. However, when using live production for events LED-based XR seems to fit better as the audience in the event will also see the production as it was designed, and not only the audience at home as it would be the case with chroma key-based production.

https://www.brainstorm3d.com