That is but the beginning. Customers have to be satisfied that a service or product is affordable and represents good value, and in this respect the product has to stand comparison with competing offerings. It’s fairly easy to compare value in a field where there’s a lot of competition, for example in the purchase of a house or a car. And although there’s no way to road test a house before you buy it, you can certainly do that with a car – or you can read road tests and reviews, as you can with most other consumer goods. In the case of services rather than goods, there isn’t the same history of ‘road testing’ information, but increasingly customers make their experiences known on the internet, so the astute buyer is able to make a more informed choice than before.

Decision made

Once committed to a contract for a service such as phone, internet or digital media, a consumer is generally inclined to stick with the choice unless there are compelling reasons to change. The most likely reason to look elsewhere is the failure of the service to live up to expectations. Because a consumer chooses the service on the basis of the visible benefits – number of channels, type of content, bandwidth, and so on – any disappointment will probably arise from poor service quality.

Digital media businesses are highly completive in a market where the customer can switch quickly if not satisfied. This means providers need to know when their viewers feel they are getting a satisfactory service, and when they aren’t. In strict terms the only way to evaluate how customers feel about a service is by asking them, in much the way that market research does. Good market research measures a lot of factors that are influential on consumer opinion and these include economic pressures and social trends that have nothing to do with the quality of the product or service itself.

Importance of proper metrics

For those whose responsibility it is to ensure that the service itself is of unimpeachable quality, the idea of measuring ‘consumer satisfaction’ has come to be embodied in the concept of (QoE) monitoring. Conventional wisdom says that QoS (Quality of Service) monitoring can provide diagnostic information on the devices in the delivery chain and the transport streams. The QoE element is intended to provide a user-centric evaluation of the service, pointing to any faults that have an impact on the user’s experience.

The established concept of QoE was really rooted in the assumption that QoS alone is insufficient to guarantee good service quality at the consumer end. And when QoE first originated in the telecoms industry this was definitely true. This lead to the development of a new measurement methodology called MOS (Mean Opinion Scoring). It was deployed when telcos were developing new infrastructure or, more recently, services such as VoIP.

A panel of expert assessors would listen to the service and note their evaluation of quality while key phrases such as ‘You will have to be very quiet’, ‘There was nothing to be seen’, and ‘They worshipped wooden idols’ were transmitted over the line. The experts would record their scores, assessing any impairments on a scale of 1-5, from ‘imperceptible’ to ‘very annoying’. The scores would then be averaged and weighted using the kind of statistical manipulations common in social sciences and market research.

Tools for monitoring digital media

The same basic concept of QoE monitoring later took hold in digital media monitoring and MOS spawned ‘VideoMOS’ and ‘AudioMOS’ – criteria which brought opinion-scoring methodologies into the media arena. 

But media providers need to know what real users are experiencing, rather than panels of expert assessors. So MOS evaluation became ‘robotized’ and algorithms were developed to simulate those subjective reactions from ‘imperceptible’ to ‘very annoying’. The robots ‘watch’ the service and the data is fed into the algorithmic engine, which attempts to simulate the subjective reaction of the viewer, with scores for factors such as ‘jerkiness’, ‘blurriness’ and ‘blockiness’.

But subjectivity is complicated, and simulation is an unreliable way to assess service quality. A human viewer watching a top-quality 1080i transmission of the Super Bowl, followed on the same channel by the 1940s B&W movie Casablanca might be fully satisfied. Yet in a robotized QoE ‘subjective’ assessment based on MOS criteria, the Super Bowl would score high, while Casablanca would be marked way down for ‘blurriness’, scratches and other artefacts, because of its lack of resolution and color. Distorted by such results, the data from this kind of QoE system becomes more or less meaningless to the provider.

QoS can do more

We are now in an era where integrated end-to-end monitoring systems for QoS can do much more to ensure quality for the end user. By going far beyond mere verification that the equipment is functioning correctly, a proper solution can give the service provider total visibility and control over the integrity of the signal throughout the entire network—or almost the entire network.

This is because with OTT services the content delivery chain includes the internet. The ability to check the quality of the service at the point of reception by the consumer is still essential if we are to have confidence that the user experience is as good as it should be. 

QoS and QoE—part of same continuum

But the traditional concept of separate domains for QoS and QoE is now out of date. Most of today’s advanced monitoring systems consider QoS and QoE as part of the same continuum, both capable of delivering objective, empirical data without the artificial and spurious distraction of simulated ‘opinion’.

QoS and QoE data often come from different points in the delivery chain. QoE data arrives in large part from the point of consumption and is represented in different ways to the monitoring staff. But it’s the close correlation between QoS and QoE data within the same integrated environment that allows operators to reach and maintain a higher quality level.

If your monitoring system can correlate and compare data on the factors that affect quality throughout the whole delivery chain, from ingest at the head end to viewing on the smartphone, then you have a completely trustworthy objective set of data on which to assess – in real time – both quality of service and of experience.

And beyond that, if you want to measure subjective opinion, there’s always market research.