But these features rely on dynamic and complex IP-based video delivery platforms, which in turn makes quality assurance extremely challenging. What does quality assurance look like for OTT services, and how are network-side monitoring solutions supporting the continued drive towards excellent QoE (Quality of Experience)?

Network-Side

Before discussing quality, it is necessary to clarify the scope of “network-side”.

Network-side is inherently focused on service-level measurements and the impacts and actions that span multiple streams to multiple end customers simultaneously. This contrasts with client-side that is inherently focused on an individual customer experience (albeit data can be aggregated for an entire audience).

One of the primary challenges with OTT quality assurance is that different consumers can have very different service experiences for a wide range of reasons. The dynamic nature of networks and internet-based delivery is a general risk to streaming video quality. To manage quality, it is therefore imperative to understand network paths between origination points and the receiving devices (see Figure 1). And to proactively protect QoE, it is necessary to focus on network-side measures and network-side corrective actions.

But don’t think that “network side” means it is impossible to understand individual customer experience. On the contrary, from the network side it is possible – and very necessary – to understand quality of service (QoS) to all as well as quality of experience (QoE) to individuals (see below).

Because of the complex OTT delivery network reality (see Figure 1), OTT service providers have a tough job to deliver excellent customer service. Not only should they be able to monitor QoS and QoE, but they should also be able to pinpoint incidents, understand root cause, and assure quality, all in real-time. OTT service providers who are most sensitive to customer dissatisfaction from poor service should also be able to predict potential future incidents and avert them.

The good news is that lessons have been learned about how to approach this problem by IPTV and Cable TV operators. Like OTT service providers deploying Apps or websites, these operators deploy set-top-boxes. Over many years, it’s become clear that set-top-boxes can only tell you something has gone wrong. They cannot tell you exactly where the problem is in the delivery chain. Measuring at the OTT App is the equivalent of the set-top-box. But OTT is more complex in this regard because to know what went wrong requires monitoring and corrective action over multiple third-party network domains.

OTT service providers deploy client-side monitoring for good reason – to know the activities and experiences of their customers. The capabilities of client-side monitoring cannot be fully replaced by network-side monitoring. But network-side capabilities must take the lead in assuring quality of service and quality of experience at the consumer’s device. OTT service providers need to be able to see inside the delivery network and directly supervise QoS and QoE. 

Quality In OTT

Quality for this article is grouped into three distinct categories – quality of content, quality of delivery and quality of experience. Each can individually affect a consumer’s decision to continue watching an OTT service provider’s content.

Quality of content – including topics like image resolution, subtitles, audio loudness and more – is normally measured at the point of ingest to the delivery platform. In OTT, this typically means the file or live encoders. Today, as important VOD content is stored in various caching servers, and OTT service providers grapple with QoE subjects like stream start-up time, there is growing pressure for quality control on the content that is stored inside CDNs.

Quality of delivery is concerned with the actual delivery of content in the required bitrates, package-types and language versions into the locations where the OTT service is provided. This can be measured at an individual stream level or at an aggregate audience level. Of growing importance is the ability to predict and avoid quality-affecting situations, such as network congestion, whether they are under the OTT service provider’s direct control or not. Quality of delivery is important for both live streaming services and VOD services.

Quality of experience is the ultimate measure, reporting how individual consumers are experiencing the video playout. Measures here include bitrates played, bit-rate consistency/changes, rebuffering rates, and start-up times. Individual consumers can have a very different experience of the same OTT service for a wide range of reasons – device performance, LAN performance and network contention are some examples. The OTT service provider needs to manage this situation above all else to assure the end customer’s satisfaction.

QoS And QoE Measures

Network-wide measures are often referred to as QoS measures, simply because the network-side can see performance of the whole service – i.e., every stream that passes through the network. This compares with client-side measures which can be grouped under the heading of QoE, because QoE is an individual experience measure. But this loose categorization hides the fact that QoE is very measurable from the network-side.

While there are good examples of network-side and client-side integrations in the industry, they are few. Today, the most typical situation for OTT service providers to manage quality is to monitor stream quality at the delivery platform (i.e., the encoder or the origin) and to use client-side monitoring for understanding QoE and to balance traffic between CDNs. But as the cable and IPTV operators learned, to fix or prevent an incident you need to know exactly where the root cause is. Normally, in the multi-dimensional pull/push/cache system of OTT, the root cause is somewhere inside the complex set of network interconnections between the encoder and the device.

“In today’s ABR streaming world, monitoring vendors must focus on both enhancing solutions for multi-supplier OTT delivery and supporting the transition to new hybrid on-prem and cloud architectures”, states Joel Daly, VP Product Management at Telestream. “And with service deployment agility evolving rapidly through advances in automation and orchestration solutions, real-time data from monitoring solution is a critical feedback mechanism for delivering both live and on-demand OTT services.”

On the network-side, it is essential to measure performance of the network in a customer-centric manner. In other words, to measure the network using QoE metrics. It is already proven from integrations between network-side and client-side tools that it is possible to measure QoE from the network. It is also clear that this provides an early-warning tool for managing service quality. By interpreting network-side data correctly, OTT services can avert customer experience incidents. A classic example relates to the sustained bit-rate metric (Figure 2), which can be detected early by network-side tools.