All 12 articles in this series are now available in our free 54 page eBook ‘Virtual Production For Broadcast’ – download it HERE.

All articles are also available individually:

The key benefit of virtual production is that it can allow a crew to operate very much as it would in a real location, with the benefits of reliable virtual weather and a short commute between locations that might be continents apart in reality - or non-existent. The trade-off is that virtual production involves some new technology, and moves quite a lot of what would have been conventional visual effects work into pre-production. Realising all those benefits while keeping the technology out of the way of the creativity is mainly a matter of preparation.

Defining The Production

Virtual production can be used to create huge, spectacular science-fiction or fantasy worlds. It could also be used to shoot - for instance - a dozen couch commercials in a dozen different lounges in less time than ever. The only thing that those productions might have in common is the desire to photograph in locations that don’t exist (or don’t conveniently exist) and for the in-camera results of virtual production. Given such a wide variety of applications, there’s necessarily a lot of variation in the planning process.

As with any production technique, the starting point will be the preferred approach of senior creatives. Some directors work closely from storyboards; others prefer to be more reactive, walking onto a location and finding setups in the moment. Virtual production is often more capable of handling that kind of spontaneity than many visual effects techniques, if only because there are always real-time, in-camera previews of the finished shot. Nonetheless, even the most free form production will benefit from at least a broad plan for blocking, staging and camera work.

Many of those fundamentals arise from where cameras, people and objects are - the geometry of the scene. Whether that’s the real world of the foreground, or the virtual world in the video display, things must be arranged to create a useful composition, reserve space for the foreground action to take place, and ensure enough LED wall coverage to occupy the proposed shots. Equally, the camera department must avoid photographing the LED wall in such a way that its individual pixels become visible, which may mean avoiding very deep depth of field, long lenses, close proximity to the wall or some combination of those factors.

Communication

Virtual production specialists consulted in the preparation of this piece agreed on many things, but particularly emphasised the need for communication. Virtual production is a new field, and technologies, capabilities and approaches vary widely between facilities. The people involved are very aware of the concerns of camera teams, and equally keen to engage in consultancy before a production has made a firm decision about what to do.  Facilities will sometimes donate time and consultancy for testing and planning, and are generally very happy to discuss the right approach in order to avoid future problems.

Blocking & Framing

It’s easy to think of the LED wall as a background in the same way as a traditional back-projection screen, and at the most basic level that can be perfectly valid. If a production chooses to use entirely two-dimensional background plates, that’s exactly how it will work, albeit with the benefit of the high contrast and brightness of LED display technology. Where there’s a three-dimensional world beyond the LED wall, though, it’s more accurate to think of the wall as a window onto that world. From a distance, that might mean almost the same thing as a two-dimensional background plate, but as the taking camera approaches the video wall, more and more of the world beyond will be revealed.

In extremis, the taking camera could reveal almost a 360-degree view of the virtual environment by moving close to the LED wall and viewing it at extreme angles, just as someone might approach a normal window for a wider view of the outside world. Close range and extreme angles create the sort of setup that risks revealing the pixels of the LED wall, but it should be clear that in principle it’s possible to reveal a huge amount of the virtual world outside that virtual window. What’s crucial about this is that it generally won’t be necessary - or practical - to define the entire virtual world in fine detail. Establishing approximate blocking and framing lets productions choose what to put in the virtual world and where to concentrate the efforts of the virtual art department, which we’ll discuss in more detail soon.

Where the geometry of the scene is very large, virtual production also makes it possible to move through that scene over much longer distances than are physically available. Techniques involving treadmills are an advanced topic, but can make it possible to create long, continuous walking or running shots over distances limited only by the size of the virtual world. More commonly, it’s possible for a production to move an actor across a virtual production stage, cut, return to the start position, and adjust the apparent position of the virtual world to give the impression of moving through a huge world in several edits. That, and related techniques, can make it easier to avoid having subjects too close to the LED wall.

Keeping The Wall Soft

Many virtual production setups rely on the LED wall being slightly out of focus. Traditional front- or back-projection might have more immunity to that as gaps between pixels are typically much smaller. As the resolution of an LED wall increases, there is less and less need to keep it out of focus. The specifics vary with different setups, but where a particularly huge, particularly high resolution LED wall is used, particularly with a wide lens, the need for it to be defocused will be minimised. Still, LED walls sometimes need to be out of focus to avoid the visibility of pixels and interference patterns.

That’s easier to achieve if the geometry of the scene has been designed with that in mind. Story-relevant real world subjects, including people, which we want to see in sharp focus must be far enough away from the LED wall that when they are sharp, the wall is still soft. Wide lenses might reveal the edges of the wall or increase depth of field enough to make pixels or moire visible, while particularly long lenses might, again, enlarge the wall to the point where pixels become visible.

Larger-sensor cameras, with their reduced depth of field for a given field of view, can make it easier to keep the wall soft, especially where there’s a demand for wide-angle lenses. The wall must be large enough to cover the shot, something that’s a particular concern with wide angles, unless there’s an intention to extend the shot in post (which will impose other requirements regarding characterisation of lenses, and perhaps other factors). Regardless, one of the challenges of blocking and staging for virtual production is to ensure enough physical separation between foreground objects and LED wall. The compromise is that the further a subject is from the wall, the less it receives the convincing interactive lighting the wall provides.

Test Sessions

Even before the design of the virtual world is finalised - or before background plates are shot, composited and prepared - a simple test shoot on a virtual production stage is likely to be helpful. Working with a stand-in, alongside the intended camera and lenses, setting up shots and evaluating the effects of focal length, camera position, framing and exposure on a good-quality display will help establish what’s possible and where the limits are. If the facility in question hasn’t worked with the intended camera package before, evaluating the camera and lenses for tracking, as well as colour matching and calibration, can also be addressed during a test session.

Where a production has created a new 3D virtual world, a test of that world, on the actual rendering servers to be used during the shoot, is a good idea. Lighting of the virtual world must match both the cinematographer’s intent, and must match the lighting used for the real world, although that will inevitably only be finalised on the day. Tests are also important where the world contains complicated lighting or geometry.  Often, simple changes to virtual worlds can massively increase rendering performance, but it’s clearly preferable to avoid having to make those fixes while a full production waits.

On The Day

Well-planned productions will pre-empt as much of the technical preparation as possible, although in extremis it is possible (though very much not recommended) to arrive on the morning of the shoot with a full cast and crew and a virtual world on a hard disk. The virtual world data may be many gigabytes in size, and must be copied to all of the rendering servers; those servers must then do preparatory work before the real time rendering can begin. Meanwhile, lenses can be characterised for their focus behaviour, and the camera tracking system set up, with tracking markers attached to the camera and calibrated for position throughout the working space.

With those tasks complete, it will be possible to move the camera around inside the virtual production stage and view the results, at which point the conventional work of finalising the blocking, staging and lighting can begin. Approaches to lighting for virtual production - whether that’s lighting of the virtual world, or interactive lighting in the real world - are a big subject for another time. Ideally, though, the well-planned virtual production makes most of the camera technique work in much the same way it always has, and that’s one of the greatest benefits of working this way.