The process of converting HDR material to SDR is not new. Color grading, vision engineers and features such as an auto knee function have been doing similar things for decades. The requirement is to take the huge contrast of the real world, as represented in the data sent from a camera, and convert it for the limited contrast capabilities of a broadcast system.

What is new is the need to create two output images, each having different dynamic range, from the same source material. The process of converting between those images is widely referred to as tone mapping. In some cases, it is done using a fixed conversion. Alternatively, some devices implement algorithms which analyze the image and act accordingly.

Also, in practice, most HDR pictures will use a wide color gamut, so most conversions from HDR to SDR will involve conversion from a wide color gamut (such as that described in the ITU’s Recommendation BT.2020) to a less capable one (such as that described in BT.709).

The Challenge

Early HDR broadcasts sometimes forced production teams to duplicate at least some parts of the vision engineering chain, with one part for HDR and one for SDR. If only for cost reasons, this could not be a practical solution for the long term.

Automating HDR to SDR conversion involves prioritizing regions of the image depending on their content, and how important that content is expected to be to the audience. This requires understanding of what the image represents and how it should reasonably look, which is something that only humans - and increasingly AI - can do. As an example, consider the behavior of a photographer’s light meter, which includes mathematics intended to approximate what humans like to see. A proper photographic exposure is dependent on a value called the meter’s calibration constant. That constant varies between meter manufacturers, having been established by asking a large number of people to look at a large number of photographs and give an opinion as to which were correctly exposed.

Proper image appearance, overall, is subjective. For example, in a downhill ski competition, viewers want to see the skier and the snow in the bright sunlight. Exposing the image to prioritize shadow detail in the dark forests in the background would not be appropriate. To know that requires an understanding of what is in the picture and why it is important.

Look Up Tables

Sometimes, automatic conversion from HDR to SDR is performed with lookup tables (LUTs). LUTs are a technology which is found all over modern film and television production, particularly where a monitor built to display images encoded according to Recommendation BT.709 is used to display images encoded using a different standard, particularly a manufacturer’s proprietary standard, as in single-camera production. A LUT might also embody a creative grading decision or some other technical standards conversion, such as a conversion from the sRGB standard used on computers to a standard such as BT.709.

A LUT is straightforward in concept, being simply a table of numbers. Notionally, for every possible pixel value in the input image, it contains a corresponding pixel value to be used in the output image.  In principle, a (three-dimensional) LUT can convert any color to any other color, and can therefore handle both color and brightness.

In practice, for ten-bit data, this would require an enormous list of over one billion values, since there are 30 bits per pixel and 230 is 1,073,741,824. So, mathematical techniques are used to store a lower number of control points and interpolate between them. Either way, because a LUT has three input values (often for R, G, and B, or Y, CR and CB data) it can be thought of as a cube, and is often referred to as such; a common format for storing them ends the file name with .cube. Two-dimensional LUTs can make brightness and contrast changes only, and therefore cannot handle both color and brightness-relevant conversions.

Viewing an HDR image directly on an SDR display results in a very low-contrast, milky image, since the HDR image expects the display to have much greater contrast range than it does. The job of the LUT, then, is - in effect - to add contrast to the image data so that it looks correct on an SDR monitor. The amount of contrast to be added is quite large and this naturally risks clipping bright areas to white, crushing dark areas to black, and leaving other areas with what might seem to be improper brightness.

Design of LUTs for HDR to SDR conversion is therefore an exacting task, usually done on a per-manufacturer or per-broadcaster basis. These LUTs may be considered proprietary. Success of this technique is very dependent on vision engineering performed with knowledge and experience of the LUT’s behavior, with accurate monitoring of both the HDR and SDR output to ensure that the conversion is producing an appropriate result in both cases.

A key advantage is that the LUT’s behavior is completely consistent and predictable, though satisfying both HDR and SDR outputs may occasionally involve minor compromises to either or both.

Active Conversion

Given that the best possible SDR representation of an HDR image is inherently subjective, it is reasonable to think that a fixed conversion can never be ideal in all circumstances. Because of this, manufacturers have created conversion devices which alter the way they treat the image based on that image’s content.

Despite the comparison we considered earlier, HDR to SDR conversion is not directly analogous to altering exposure. Perhaps more accurately (though still incompletely), this sort of down conversion is analogous to the auto knee function found in many broadcast cameras. The knee in question refers to the point at which bright image areas are considered to be nearing overexposure. Above that point, brightness is compressed - effectively reducing contrast in bright areas - to produce a gentler, more pleasing tail-off into overexposure. The concept is similar to applying an S-curve to an image using something like Photoshop’s curves filter, and the knee refers to the angle at the top right.

Cameras have had auto knee features for decades. Commercial HDR to SDR conversion devices are likely to implement much more advanced and complex algorithms than that, with intelligent handling of highlights, shadows, and a wide variety of other image content, and with some manual controls to allow users to implement a creative intent. The specifics are likely to be considered proprietary, although this is an environment in which AI might be able bring an unprecedentedly human-like understanding of what an image should look like.

Converting Color

In contrast to HDR-SDR conversions involving contrast and brightness, converting color spaces is a much more straightforward prospect. In principle, there is only one correct way to convert (say) a Rec. 2020 image to a Rec. 709 image, although in practice it is possible for naive conversions to create unpleasant image artefacts. In parts of a Rec. 2020 image where saturation exceeds that of which Rec. 709 is capable, that color is out of gamut.

Simply moving all out-of-gamut color to the nearest in-gamut color may create visible discontinuities in the image such as edges where there should be none, or distracting flat areas where saturation should vary organically. There are several complexities, not least of which is what “nearest” means. We are used to bright areas of the image clipping; when converting wide color gamut images to conventional images, we may encounter similarly ugly problems with highly saturated areas of the image clipping.

Making a color in-gamut might mean altering its hue, its saturation, or both, which will have different effects on the image (generally, though not always, maintaining hue while altering saturation is less distracting). As with converting HDR to SDR, gradual reduction of saturation as colors begin to approach maximum saturation (like auto knee, but for color) is likely to be appropriate, but again, proprietary approaches abound.

There is an argument that reducing the color gamut of images is a less trying task than reducing exposure, since the average color saturation of most real-world images is low anyway. Even so, specific subjects such as filtered or LED light sources, or particularly graphics, can achieve very high saturation. These are among the most likely subjects to create difficulties in conversion between color standards.

The Future

HDR is unusual in that improvement in displays is absolutely expected to create better - which means different - pictures over time. As such it is quite possible that manufacturers’ understandable need to keep selling televisions will create a constantly-improving home user experience that also represents a constantly moving target.

History offers us at least one cautionary tale. Spend long enough on the internet, and it is possible to find high definition video recorded in the early 1990s, which has similar numerical specifications to the HD video recorded now - but which probably would not be viewed as broadcast quality in a modern context. Whether these changes will be significant enough to create problems - even significant enough to obsolete equipment - remains to be seen.