Other articles in this series:

Why do computer servers with their associated software contain the potential for security vulnerabilities?

The fundamental answer to this question is that the part of the CPUs memory that contains the executable software is read and write accessible. In other words, it can be changed and modified, and not necessarily by the person authorized to do so.

Read Only Computers

Executing software from read/write memory may seem obvious, but computers don’t have to operate this way. If the programs are executed from read-only memory, such as an EPROM, then it would be virtually impossible to remotely hack the computer. However, the problem with this solution is that the computer becomes completely inflexible; new software cannot be easily loaded into the computer as new physical ICs would have to be installed in the device. We can make a computer almost un-hackable but at the expense of complete inflexibility. Therefore, we must make a compromise.

Anybody who thinks that running a computer program from EPROM is a completely ridiculous idea should take five minutes to speak to one of their colleagues of more established years who will gladly tell you all kinds of historical anecdotes of how they replaced the EPROMs in the character generator with only ten minutes to air.

If we want flexibility, then the price we pay is that infrastructure security becomes more challenging. And if we don’t want massive flexibility, then just go back to SDI/AES infrastructures, these still have their place in broadcast facilities.

Downtime Inconvenience

Modern servers and operating systems are generally very good at updating software with the latest patches and security releases. But one of the challenges we have is that the servers have a nasty habit of executing a software update at the most inconvenient of times, generally because in 24/7 broadcasting there isn’t a convenient time. Consequently, the software upgrades are disabled and the servers become vulnerable.

The cybersecurity Common Vulnerabilities and Exposures website (CVE.org) shows a database of over 240,000 known security vulnerabilities shared by vendors across the globe. For example, RedHat provided notification CVE-2024-6409 which advises of a potential exploit in OpenSSH where a race off condition between signals can leave a server open to a remote attack. This was fixed by RedHat with a security patch released on 30^th July 2024 (although other patches will now have superseded this).

The OpenSSH security threat is just one of many exploits that are revealed monthly, and even weekly across multiple operating systems. Monitoring the CVE database is one method of keeping across the multitude of potential security breaches and fixes, but few broadcast and IT engineers have the time to do this. As security forms a fundamental part of an operating system vendors service, they will have entire teams of specialist researchers and engineers dedicated to finding, understanding, and repairing these vulnerabilities. Although it’s good practice to keep your knowledge of security vulnerabilities up to date, the best solution to keeping a broadcaster’s infrastructure as secure as possible is to have regularly scheduled maintenance for every server so that they can be rebooted and automatically load any new security patches.

Common Software Libraries

Another challenge with software-based systems is that the broadcaster doesn’t know which libraries their vendors are using. Most software vendors delivering to the broadcast industry will use open-source libraries within their designs. From a commercial and reliability viewpoint, there is absolutely no benefit to re-designing libraries that form the basis of computer applications. Instead of taking weeks and months to deliver features, the timeline would move back to years.

Open-source software has revolutionized how applications are built and continue to exponentially reduce delivery times. However, hostile actors also know this and spend time meticulously dissecting the software to find potential vulnerabilities, with the knowledge that these libraries are ubiquitous in the commercial world, and so if they find a vulnerability, then their nefarious rewards could be great. On the flip side of the coin, there are vendors that have specialist teams dedicated to understanding the exploits the hackers have found and then provide solutions to stop them.

This brings us to the benefits of building vendor relationships. Software designers in the broadcast industry are best placed to understanding security vulnerabilities as it forms the basis of their development cycles. Continuous testing disciplines software development teams to not only test regularly for functional reliability, but also to test for security exploits. For vendors that are well established in this process, software leaves their factory as safe and reliable as is physically possible thus providing the broadcaster with peace of mind.

Continuous Development Testing

During the procurement process the broadcaster would do well to investigate how effective the vendors continuous testing cycles are and understand how proactive a software vendor is in terms of identifying security vulnerabilities and communicating them. A well-established development team who takes security seriously will be monitoring web sites such as CVE and have proactive processes in place to deliver the necessary security patches to update their code.

Collaboration between the software vendor and broadcaster is a key component in the security mix and a high level of trust needs to be established between the two for system critical applications. This goes beyond the contractual SLA obligations so that the broadcaster can have trust in not only the delivery of the software on day one, but the ongoing support that the vendor is willing to put into the maintenance of the product. Although SLA’s may seem expensive, they are worth their weight in gold if they deliver on the continued maintenance and support of the software.

It's not all a one-way street, the broadcaster must do their bit and allow software patches and upgrades to be installed in their workflows. This may well require coordinated down times of certain parts of the infrastructure, but a well-designed system will not only be able to tolerate this, but have the necessary resilience built into the infrastructure from the very beginning of the design. There’s no point in a vendor rapidly providing multiple patches if the broadcaster fears rebooting their servers. If an emergency patch is needed to be installed, then the broadcaster needs to be able to facilitate the work quickly.

Regularly Server Maintenance

It's worth pointing out that the software vendors may have the ultimate responsibility for upgrading the broadcasters’ servers for security fixes, but the fact they need to do so is rarely their fault. Hostile international actors work diligently to find holes in software applications of all types, especially those used in other parts of industry, and the sad fact is that broadcasters can inadvertently become a victim to this.

If we take the attitude that servers will need regular patching and upgrading, then broadcasters stand a much better chance of being able to block security breaches quickly. Combined with help and collaboration from reputable and trustworthy software vendors, regular upgrading and patching is a fundamental requirement of any broadcast infrastructure.

A server containing a security breach looks just like any other server, capacitor C1 won’t have popped leaving a faint burning smell, and a dry joint won’t be making the picture break up, it just looks normal. And so just because a server doesn’t appear to be broke doesn’t meant that it is working perfectly, security breaches have a nasty habit of lying in wait to be triggered, and even if the server hasn’t been infiltrated, then a known vulnerability means it’s only a matter of time. So, if it ain’t broke, then it may still need to be fixed.