Other articles in this series:

In today’s highly competitive media landscape, the on-screen presentation of a media company’s programming can make all the difference between ratings success and failure. That’s why many have embraced virtual sets as a cost-effective way of improving their on-air look by either replacing an existing physical studio set or expanding the studio space without physically building any new walls.

While the addition of a virtual background can place studio presenters in any location in the world, augmented reality (AR) graphics are rendered in front of the on-air talent and help liven up the set in ways physical set pieces cannot. When used in combination with each other, as many media companies are now doing, the on-air effect can be captivating. AR systems are used to place virtual 3D graphic elements in the physical world.

These elements are arranged three dimensionally and attached to a live video signal, then output as a combination of live video and graphics. The technology takes the idea of virtual graphics from virtual sets but places them in the front of the presenter or in the foreground of a studio. A big benefit is that these graphical elements can then be used in tandem with a physical set, adding an extra layer of depth to the studio, or extending the studio space, without the need for a chromakey or green screen wall. The technology can even be used for outdoor studio environments, such as during live sporting events.

Increasing Demand

Virtual sets and virtual technology in general is possibly in the best shape ever, as virtual sets now provide photo-realistic visual quality (thanks to improved processing engines and it’s a mature technology in which most of the processes are known and controlled). In addition, the flexibility virtual sets provide for broadcast and content creation in terms of cost savings (actors or talent’s time, travel, props, etc.) mean virtual sets continue to have a significant and positive affect for today’s demanding content creation.

“To be fair, virtual sets and LED walls are not technologies that exclude each other, on the contrary, they can complement each other by providing additional options for content providers,” said Miguel Churruca, Marketing and Communications Director at Brainstorm. “And we should not forget that LED video walls can also display a virtual set.”

For one thing, virtual sets have gotten easier to set up and control. And they are not subject to visual artifacts like loss of focus, moiré’ effects and pitch control that can cause negative effects on LED displays. With virtual sets you have to configure a chromakey while seamlessly integrating the talent into the green screen background.

With a virtual set, it’s also easier to create an immersive experience for the viewer, whereby the talent appears within the scene being described and can interact with floating charts and maps to tell a more comprehensive story. Many news organizations in Europe are deploying big cycloramas (curved backwalls) that create immersive environments and viewers are reacting positively.

“You just need a camera shooting against a green set and you can have floors, reflections, and 3D maps,” said Churruca. “These are things that are very difficult to obtain in the physical world with LEDs. We can create an entire environment that is totally believable. I saw a set in Spain that recreated Beijing for the recent Winter Olympics and people online said they thought the reporter was working in China.”

The Technology Is Mature

Virtual sets used to be complex to set up. The technology is now mature enough that people know how to work with them and they understand the issues involved in setting up a virtual environment. You need to look at the lighting of the set, the camera tracking, and the composition of the shot.

Users of virtual sets can also precisely control the lighting, both the physical fixtures (with a DMX system) as well as the colors of the set (using special software features built into the VS system). This combination can be really powerful in changing the mood of the set of a specific story and then changing it back to a main pre-set and move on to the next story. The latest VS systems can also add shadows in interesting ways that can’t be done with an LED monitor wall. It can be complicated and takes a bit of coding to do on a virtual set, but it really makes the look special.

"Chroma keying and lighting are key elements within virtual sets," said Thierry Gonzalo, Senior Product Manager of AR/VR products at Vizrt. "The biggest challenge is combining the real world, that has a green background and talent, with the virtual world."

Addressing ease of use, Vizrt has introduced XR Set Express, a fully-equipped, virtual studio-in-a-box. It includes pre-configured and pre-calibrated features enabling quick setup and instantaneous use.

"Virtual sets have now become a mission-critical tool for many broadcasters, regardless of size, that is used every day," said Gonzalo. "With the introduction of Vizrt XR Set Express, for example, those that want to leverage the benefits of an XR set can do so, faster, and simpler than ever before."

The system leverages Vizrt’s Virtual Studio capabilities alongside the IP-connected workflows of NDI, developed by NewTek (and now owned by Vizrt). NDI networking is used to enable video, tracking data and full control of the Panasonic UE100 PTZ camera through a single cable connection. Viz Arc provides control for studio setup and configuration. Positioning of graphics, studio customization, auto-setting of the keyer, presetting desired camera positions and much more is now accomplished within a few clicks.

For the Winter Games this year, BBC Sport constructed a stunning virtual set with Viz Engine 4 and Unreal Engine to create video content for all coverage coming from Beijing. Designed with seven different views, the actual studio space was only 84 sq. meters (about 904 sq. ft.). BBC Sport was able to create a seamless virtual studio environment with easy operator control to bring fans unprecedented access to the Winter Games and it's been incredibly popular as well. BBC Sport has also used the virtual studio for the Australian Open, Golf, and various other sports shows.

Camera Tracking

Camera tracking is another aspect of virtual sets that has made a huge difference in judging the perspective of the on-air presenter and making the set move accordingly. There are now a number of camera tracking systems on the market—from companies like Mo-Sys, OptiTrack, Stype, and others—that work well with virtual sets.

Virtual sets rely on camera movement data that must be carefully tracked with a real-time rendering and compositing system to create these highly realistic environments. Camera tracking is critical in order to convey an accurate on-air impression of a presenter walking around and interacting with a virtual world. The tracking data from the camera and the rendering of the graphics are composited and output as a final image. Keyed graphics can be included on top if necessary.

Currently, there are three types of camera tracking systems used with virtual sets: Mechanical, Optical, and Image-based.

1. Mechanical Tracking
Mechanical Camera Tracking Systems consist of specialized hardware, typically a camera tracking head and a lens encoder that can be installed on cameras and on pedestals, in the studio or in the field. These mechanical tracking systems work in tandem with a compositing server for each camera (which is controlled through the video production switcher in the control room or OB van). Mechanical tracking systems offer the best accuracy and can cope with any camera movement as long as the camera is put on a tripod, a crane or rails.

2. Optical Tracking
Another type of motion tracking system uses Optical Tracking technology to capture the camera data and the location of the talent within the virtual set. This technology comes from the motion capture capabilities originally developed for the film and gaming industry. A typical mo-cap system uses a series of specialized illumination units to capture reflective, moving markers with a set of cameras. The technology tends to be more expensive than mechanical tracking technology, but it also provides a lot of flexibility to add extra cameras and tracked objects to the set. In addition, it allows for greater freedom of movement for the on-air talent within the virtual set. Therefore, multiple cameras and objects are trackable simultaneously using the same system.

3. Image-Based Tracking
The third, and most recent entry into the virtual space, is Image-based tracking. With an image-based tracking system, there are no tracking heads on the camera, and no additional gear is required at the venue or on the camera. The technology allows for tracking a camera using the television camera image (pixels) alone to calculate the movement of the camera. A computer application does the tracking calculations based on the recorded or live TV footage. The camera can move around in the studio or around the sports field.

The usage of image-based tracking can be limited by a number of environmental factors. These include extreme light conditions, inclement weather or even all green environments like sports fields where no contrasting features can be detected.

So Many Options

“We now have more options now in terms of camera tracking systems,” said Churruca. “Some are mounted on the camera and use a hanging grid with small stickers to track the camera’s movements in relation to what’s going on in the virtual set. In general, camera tracking is less complex than it used to be. You just need to be careful to deliver all of the data calibrated in the proper way.

“With a virtual set you can move the camera angles easily,” he said. “The whole thing with tracking is to be able to know where the camera is in space and apply the rendering to that position to create the full effect. Because we’re looking at the set from the view of the camera.”

Brainstorm’s latest virtual set is the Infinity set, which is now being offered as a package with its eStudio render engine and Aston real-time motion graphics solution.  It’s also able to work with Unreal Engine, a real-time processor that is used to boost the performance of real-time graphics display. The target is XR content, that is, extended reality images rendered in a 3D environment.