All 18 articles in this series are now available in our free 84 page eBook ‘Designing IP Broadcast Systems’ – download it HERE.

All articles are also available individually:

Since the first tube cameras were connected to the first CRTs, broadcast engineers have been working in an environment that is constantly expanding and changing. The real-time nature of live news and sports has helped drive a culture of rapid problem solving, however, this practice is not always harmonious with the methods adopted by IT engineers.

The accelerated growth of IT infrastructure and network systems has facilitated the adoption of IP and datacenters by broadcasters for media distribution and processing. This adoption has provided some massive gains for broadcasters through software defined networking and infrastructure to deliver the ultimate in scalability and flexibility. As broadcast engineers, it helps to remember that IT, as an industry and profession, has been around for a good thirty years. Probably not as old as broadcasting, but datacenter, network, and IT engineers have been honing their skills for a long time.

For as long as anybody can remember broadcast engineers have been building custom boxes that interface one control system to another or convert between media signals. The ubiquitous GPIO with its ability to send and receive one bit of information is now only just showing signs of abating but even this has found its way into NMOS. These interface units were born out of necessity as off-the-shelf equipment wasn’t available at the time and new innovative solutions had to be found. The advantage of these initiatives is that the broadcaster got on-air and stayed there, the disadvantage is that broadcast facilities are even today plagued with boxes and legacy systems that are considered essential, but nobody can remember why.

The need to push technology to its limits to reach the signal bandwidths and network datarates television demands, especially with video, has resulted in esoteric standards such as SDI and AES. These are specific to television and form part of a small industry that is not seen in IT.

By comparison, IT had a relatively clean slate as they didn’t have to maintain archaic methods of backwards compatibility that has been at the heart of television since the 1930s. IT systems have certainly built on the shoulders of their predecessors, but the need to maintain systems that are completely backwards compatible as a nationwide and international system are largely unheard of. That said, there are areas within a service providers system that needs to maintain backwards compatibility such as in banking where it is almost impossible to switch one system off and another on without the users knowing. But these systems are largely isolated and service provider specific, unlike in national TV stations where all broadcasters within a country must comply with the same broadcasting specification.

There are many similarities between the working philosophies of IT and broadcasting. For example, they must both maintain high levels of availability and must both work in real-time. Medical IT is an example of meeting these requirements as medics need 24/7 access to patient records, and in modern systems this includes x-ray images, photographs, and a whole plethora of data. All this is needed to keep patients alive. Consequently, IT have not only developed infrastructures and networks that are highly reliable and available but have also designed working practices that maintain these requirements.

ITIL processes were seen as a massive help for IT departments maintaining high availability systems as they provided processes that stopped a system being inadvertently interrupted or broken. Processes such as change control meant that when a system was being upgraded then every stakeholder had to sign-off on the change. For example, if the telephone system was to be upgraded then meetings would take place between the IT department and the stakeholders so that the telephone system wasn’t switched off when it was most needed. Although this sounds like the perfect system, as it maintains high availability, it does so at the cost of flexibility. It’s quite easy to get into a situation where nothing happens as the users of the telephone system refuse to sign-off on the work as they constantly need the communications it provides. This leads to all kinds of upgrade and security issues as the software is never patched or updated.

Agile and by implication DevOps are relatively new to IT. It’s a bit of a moot point to suggest that IT and DevOps are the same, but as IT infrastructure processes develop, they are doing so in an agile manner. This has the potential to part from ITIL which may fill some IT engineers with horror as they worry that their systems availability or resilience may be compromised. But many IT systems, especially those found in broadcasting, constantly need to change, and so find a natural harmony with agile working practices.

It could be said that broadcasting has been working using agile methodologies even before anybody formalized the agile processes. When something needed to be done, a broadcast engineer would always be ready to make the necessary changes quickly and without hesitation as there is little alternative in a live television studio environment. If a production switcher went down then time was of the essence and well-rehearsed fault finding would be executed, usually before the studio crew were even aware the fault was being rectified.

Over the past ten years or so, broadcast equipment has become incredibly reliable. As an example, in the 1980s there were often multiple engineers assigned to each and every studio, but now, there are multiple studios assigned to one engineer. As equipment has become more reliable, then the need for a small army of engineers who can repair to component level has diminished. Another consequence of reliable equipment and increased demand is that playout systems have ballooned and are easily able to meet the demands of viewers watching multiple streams and channels. But with this increase in volume there has become a need to introduce more processes into broadcast attitudes for maintenance and repair.

This is leading to an interesting amalgamation between IT and broadcast; IT is becoming more agile, and broadcasting is becoming more process driven. That said, this doesn’t mean that IT will be completely agile, and that broadcasting will become completely process driven, but there is an interesting compromise that seems to be joining the two disciplines.

IT and broadcast have come from two different methods of operation.

IT tend to be more system specialists with little component level fault finding requirements, whereas broadcast engineers come from a heritage of low-level component and circuit board repair with less of an emphasis on vendor specific systems and more focus on workflows. The network links further demonstrate this as IT use industry standard IP, whereas broadcasters historically use baseband analogue coax and SDI. This in part is due to the historical nature of how the two have approached their fields of expertise.

Due to the COTS nature of IT hardware, the need to repair to component level has rarely been a requirement. With the proliferation of SLAs and higher demand for hardware across many seemingly unrelated industries, it is more efficient to swap out system components such as servers, with the option of swapping out circuit boards, power supplies, and disc drives where the physical size of the equipment means physical equipment changing is difficult. If this is contrasted to television where analog technology was driven to its limits at an early age, the broadcast engineers have had to learn component level fault finding as the economies of scale often meant that vendors couldn’t provide adequate SLAs and spare equipment, especially in the early years.

These two different approaches to maintaining systems has also caused some level of demarcation between the IT and broadcast engineering disciplines. But as broadcast equipment is becoming more reliable, the need to repair to component level has largely disappeared. 

Furthermore, the systems with which broadcasters now need to understand are significantly more complex than they were ten or twenty years ago, leading the broadcast engineer to focus more on support at a system level.

One other interesting approach other than process and agile, is that of chaos engineering. Here, DevOps type engineers will consider every possible eventuality that could lead to a system failure and then design for it. Randomly, equipment, networks and other infrastructure components will be switched off to stress test the system, and if the chaos engineering approach has worked, then the workflows will continue as the infrastructures self-heal.

As IT and broadcast infrastructures develop, the difference between how the respective engineers approach their professions is drawing closer together, which is reassuring for CEOs running broadcast facilities. Agile methodologies are key to flexibility, but the need to adopt processes is still important. A compromise is always required.