Distributing multiple video resolutions via IP at adaptive bit rates is a major part of any entertainment-delivery operation today. The nature of adaptive-bit-rate (ABR) services means there will be downstream deliverables in multiple profiles derived from a single HD or UHD source — all of which require test and measurement to verify their quality. And the most accurate way to do that is through full-reference testing.

The basic idea behind full-reference testing is to take a short video clip, send it through a system to be tested, and then compare the system’s output to the original (See figure). If the system causes any differences in any of the video frames, those differences can be measured using a variety of objective metrics to yield numerical results.

Today most traditional television signals are maintained at specific resolutions and frame rates from production through transmission. This consistency makes reference-based testing relatively easy to do in the broadcast domain because you can do a straight comparison between the processed video and its source. However, in the IP domain, it’s not so simple.

Why It’s Challenging to Test ABR Content

The foundation of full-reference testing is that the source and processed sequences must be at the same resolution and frame rate. In other words, you must start out comparing apples to apples in order to accurately measure changes in quality between the source and processed video.

With IP content, you can’t simply compare the processed video to the source, because the source will very likely be at too high a resolution. That means you first have to reduce the resolution/frame rate of the source material in order to create an acceptable reference for each delivery profile. Then, as you create new tests for new downstream profiles, you must create similarly processed streams to use as references during each test.

Furthermore, network operators rely on ABR delivery in order to accommodate a range of network conditions and end devices, which means they must be prepared to deliver multiple profiles (resolutions, bit rates, frame rates) for every asset. ABR delivery makes the testing process even more complex because you must perform a test for every one of those profiles.

With these complicating factors, it’s easy to see why testing the quality of IP-based ABR video can be so troublesome. So what’s the best way to do it?

A New, More Accurate Process for Measuring ABR Video in the IP Domain

In order to get the most accurate results from full-reference testing, you want to start with the most pristine reference possible. When testing IP streams that are delivered at different resolutions than the source video, the best way to arrive at a pristine reference is to use the same encoder to create the reference as you do to create the downstream deliverables. Doing so minimizes artifacts and/or degradations of using different video scaling techniques that could distort the results.

Therefore, the new methodology relies first and foremost on creating the references and test sequences with the same encoder/transcoder. Then, once you’ve created the identically formatted streams, you use the same measurement system and scoring index throughout the entire testing process to ensure repeatable results. Finally, after the tests have run, an experienced video engineer considers the resulting score while visually comparing the reference and the processed streams side by side.

To apply this test method, follow these steps:

1. Generate a mezzanine-quality reference for each profile using the highest possible bit rate and optimal encoding parameters.
2. Generate each profile’s test signal using the application’s encoding parameters.
3. Calculate quality using the chosen algorithm and scale, comparing each test-profile signal to the appropriate reference-profile signal.
4. Analyze results against visual comparisons of source- and downstream-profile test segments.

Repeat this process for each stream that needs testing, creating a new version of the reference for each profile or device under test, and making incremental adjustments to the variables one at a time in order to test the effect on quality.

Because manually repeating the test for every variation and then visually comparing the results can be a daunting task, the methodology might call for an automated tool to manage the testing phase (steps 3 and 4). Such a tool would measure video quality in the many different combinations of bit rates, frame rates, and lower-than-broadcast resolutions using an algorithm that yields numerical results based on human-perception scales. At the same time, it would automatically create measurement charts and synchronized, side-by-side pictures so that video engineers can see the differences between the reference and the processed video.

The graph shows an estimated quality trend based on the averages of many sequence-scores. (In this case, a lower score indicates better picture quality). Blue and red lines show a potential difference in quality averages between two encoders. In general, quality will increase as bit rates increase, signified by downward-sloping curves on the graph.

For example, the figure above illustrates the testing process using Video Clarity’s ClearView video quality analyzer.

1. An HD/UHD video source feeds an uncompressed signal to the encoder's input.
2. The output of the transcoder goes into ClearView, where it is captured in real time and simultaneously decoded to form one sequence in a set of full-reference test sequences.
3. Additional outputs are then captured and added to the set by making incremental changes in the transcoder bit rate. More sets of full-reference test sequences can be generated by using different source content and repeating the process of capturing and storing the encoder output sequences.
4. In this example, ClearView performs several types of measurements on a complete set of full-reference test sequences. The test results create a single quality score per frame of video on any given sequence, and then presents results using the MS-SSIM measurement on the DMOS scale (or MS-SSIM scale). It can also test using the Sarnoff JND —Just Noticeable Differences, or MOVIE Temporal, MOVIE Spatial and overall MOVIE measurements.
5. The output is a set of automatically aligned the sequences, each with measured quality, comparison charts, and a set of the uncompressed reference and processed test streams side-by-side on HDTV or UHDTV monitors for visual inspection.

A Proven, Applied Method for ABR Streaming

This method is a proven, mathematically accurate means of objectively comparing different profiles and equipment. In fact, major broadcast and cable networks around the world use this method to develop and refine their OTT delivery quality. For example, a major U.S. television network relied on this method to perfect multiple ABR profiles for streaming a recent major sporting event.

When using ABR services to deliver low-resolution video over IP networks, this test method gives operators the truest possible picture of how their downstream IP deliverables match up to the source video. The result? An accurate way of deciding how to deliver ABR content and where to invest equipment dollars.