Broadcast infrastructures employing transport standards such as SDI and AES distribute video and audio synchronously. The principal reason for this is to maintain very low levels of latency, but when we use the IP domain to transfer video and audio, we need to embrace the concept of asynchronous processing.

Broadcast video has been synchronous since the founders of television synchronized CRT televisions with the scanning coils in the cameras to enable continuous and fluid motion on the screen. Although this method is largely outdated and we don’t really need to use it anymore, it’s still the prevalent methodology due to backwards compatibility that dominates how broadcast television operates today. Which is why, as an industry, we are so obsessed with synchronization.

Synchronization Principals

SDI and AES use a clock to synchronize the receiving device to the sending device. This solves two challenges: firstly, there is no under- or over-run of data, and secondly, the latency is kept very low, to the order of a few tens of nanoseconds. Both SDI and AES have, and continue to, serve the broadcast industry well. 

Asynchronous systems, regardless of the type of data they represent, suffer from two fundamental challenges. The sending device can send data faster or slower than the receiving device can ingest it, and the latency of data transfer is both potentially unpredictable and variable.

The synchronous nature of SDI and AES means that there is no data under- or over-run as the receiver clock synchronizes to the sender clock so that all the data is clocked out of the sender and into the receiver simultaneously. The same is not true of asynchronous systems. By their very nature, the sender and receiver clocks are not synchronized together.

Although synchronous systems have many benefits, in terms of data integrity and exceptionally low latency, they do suffer from complexity and lack of flexibility. Their complexity comes from an imposed timing constraint that is a function of all synchronized systems, and the lack of flexibility is due to the highly specified protocols that determine the intrinsic interconnectivity between the data and clocking system. With SDI for example, the data and transport layer are intrinsically entwined meaning that it is difficult to send SDI over a different type of network as its timing plane is tied directly to the data plane. However, asynchronous protocols such as IP differentiate the data layer from the underlying physical layer meaning the IP datagram can be easily sent over multiple networks such as WiFi, Ethernet and Fiber. And this is one of the major strengths of asynchronous networks.

Event Driven Architectures

It’s worth remembering that when we speak of asynchronous, we’re not just referring to the network, as asynchronous encapsulates many other aspects of a system. For example, a computer server is largely an asynchronous system. There are certainly some synchronous subsystems such as the memory access that often uses data bursting to read and write to the memory. But overall, the computer isn’t time constrained. Again, this is both a major strength and a major weakness.

Another way of thinking about asynchronous systems is to think in terms of event driven architectures. By this, we mean that a system responds to an event that may well be random in nature. For example, when typing a key on a keyboard, in the overall scheme of the computer server, the key-typing is an event, which causes other events to happen in the server, such as the operating system detecting the key stroke and then writing a character to a screen, and to memory, and then the disk storage. All these systems are asynchronous relative to each other and “do something” in response to an initial event.

The advantage of event-driven systems is that the resource of the computer server can be utilized with much greater efficiency but there will be some variable latency. As an example, contrast this to the real-time-operation of an aircraft electronic control system where the flightdeck controls are physically separated from the aircraft flight surfaces such as the ailerons. With such a system, the pilot will need continuous feedback and know that the control input to aileron movement is both predictable in time and displacement. The pilot doesn’t want to move the control stick to the left and then have the aileron move 10ms or 100ms or 500ms later depending on the resource demand at that time on the rest of the system, the response time must always be the same. To an extent, it doesn’t really matter how long the response time is, but it must be determinate.

Such systems often use time-division multiplexing (TDM), which is a form of synchronous data exchange. The data clocks may not necessarily be synchronized, but the data slot within the TDM frame must be determinate so that the receiver always knows where the channel of data resides within the frame. This way, it is time determinate resulting in low latency and high data integrity.

Synchronous IP

IP can, and does, have the option of being transferred over TDM type networks and one example of this is SDH (Synchronous Digital Hierarchy). This is a standard often used by telcos to connect network exchanges and other high speed, low latency, networks together.

To make a computer system operate synchronously would be a colossal undertaking and would require a redesign of not only most of the hardware, but also the operating systems and the application software. Video and audio processing does use synchronous processing, and these systems often use FPGAs. Although FPGAs are highly flexible and programmable, they start to take us down the “custom design route” again and negate many some of the advantages of using COTS type infrastructures. That said, some cloud providers such as AWS are supplying virtualized FPGAs, but the challenge still comes in making them work effectively as they require specialist tools for their development, and even more specialist design engineers to build applications. It is currently relatively common to utilize a hybrid compute resource infrastructure where FPGA’s are deployed as re-programmable blocks of synchronous processing alongside CPU processing in a primarily COTS server architecture. This achieves something of a middle ground where a software defined infrastructure is built around blocks of specialized compute resource that has some degree of flexibility of use.

As we move forward stand now, a potentially far better method of high-speed processing is using GPUs. These are readily available in cloud hyperscaler infrastructures as well as computer servers. They have many software developers working in other industries programming them and providing applications that have crossed to broadcasting.

Relaxing Timing Constraints

As asynchronous systems effectively remove the timing constraints from servers, storage, and networks, their operation becomes both simpler and more flexible. It’s simpler as the receiving equipment doesn’t have to synchronize to the sender, and more flexible as the data plane and physical link are only loosely associated giving much greater scope for other protocols to be used.

Although asynchronous systems do not need to be phase and frequency locked to the senders’ clock, they must, on average be able to receive all the data the sender is transmitting. Again, this can be part of a network, or a sub-function within the overall system. The average-rate is the devil in the detail that can cause so many challenges for broadcasters.

When we speak of an SDI data rate of 1.485Gb/s (for HD) then this is a measure of the average. There will be some deviation from the average, but it will be relatively small, and when the senders clock moves slightly in frequency, then the receiver’s clock will move with it to maintain synchronization. However, with an asynchronous system, the deviation from the average can vary greatly depending on many factors such as the resource utilization and network congestion. This means that if the data rate reaching the receiver deviates by a significant amount, which is likely, then data under- and over-run may be experienced, resulting in lost data.

One of the most frequently used methods of reducing the effects of data loss through deviation from the average datarate is the implementation of memory buffers, normally residing in the receiving equipment. If a series of IP packets are sent to a receiver at a significantly higher datarate than the average, then the buffers will temporarily store the packets until the compute resource can process them. This works well but if the compute resource doesn’t empty the buffer faster than the sending device sends them (on average) then the buffer will over-run, resulting in lost data. To alleviate this the receiver buffer can be made bigger, but in doing so, the latency greatly increases to the extent where the latency is proportional to the buffer size. Also, latency caused by buffers within a system concatenate resulting in not only increased latency, but also variable and non-deterministic latency.

Buffers exist all over asynchronous systems, even in the sub-functions such as the memory reading and writing to computer memory. They solve many problems and help keep asynchronous systems much less complex and much more flexible than synchronous systems. However, they do introduce non-determinate latency and can create many challenges in broadcast systems.

Balancing buffer latency against data loss is such an interesting subject that a whole branch of computer science called “queueing theory” is dedicated to understanding and making asynchronous systems more determinate.