Migrating to IP requires an entirely different mindset to the traditional broadcast engineering approach. Although television technology has been developing over the past eighty years, its only recently that it has moved from being a cottage industry to approaching full maturity.

Broadcast engineers are renowned for finding innovative solutions to complex problems. Especially when new technologies emerge such as the introduction of SDI, HD, and MPEG. Hand crafted boxes of electronics, cables, and even software were used to interface new and old systems together, especially where manufacturers didn’t have an off-the-shelf solution.

Complex Workflows Emerged

Although this approach solved a specific problem and kept the station on air, with the passage of time the modification was forgotten, resulting in overly complex workflows and esoteric systems. A lack of formal change procedures and documentation updates compounded the complexities. But, at the time, the broadcast engineers were under a great deal of pressure to keep the stations broadcasting in the face of emerging and cutting-edge technology.

ITIL (Information Technology Infrastructure Library) is the process IT departments use when operating and maintaining data center facilities. It is a series of procedural systems, such as change management and incident management, that help IT managers run their facility with the maximum efficiency possible.

Such processes provide a deep audit trail for changes made to the infrastructure to allow engineers to track change and provide roll-back if something goes wrong. The impact on users is assessed and nobody is permitted to make the change until the authorized manager has signed off. This also prevents unauthorized changes and helps maintain high levels of security.

Ticketing systems help enforce the audit process. If a user wants a change to their system or has detected a problem with their service, then they will call the IT help-desk and raise a ticket number. Apart from being a vehicle to track the fix, managers use help-desk ticket data to quickly identify patterns with emerging issues and allocate appropriate resource to keep the system working reliably.

Processes Maintain Order

Generations of broadcast engineers have just fixed problems when they occur and learned very early on in their careers not to jeopardize a live transmission or advert. Multi-channel playout services have introduced formal change processes and broadcast engineers are beginning to see ITIL type processes more and more in television.

IT engineers have had less opportunity to fix computers and network switches at the component level due to the dense integration of high pin-count semiconductors. The maturity of the IT industry means there are no, or few, serviceable parts. Software configuration was originally only available through command line entry, often requiring a detailed knowledge and understanding of complex attributes and strings of esoteric characters to support them.

Up to about fifteen years ago, it was common for a service manual to be shipped with broadcast equipment such as video tape recorders, cameras, and sound consoles. Most broadcast facilities had maintenance departments and in part this was due to the analogue nature of broadcast electronics and their propensity to drift. Embedded software gave little opportunity for modifications to be made to parameters within the code.

Different Ways of Thinking

Consequently, broadcast and IT engineers have evolved with different thought processes and disciplines. They had to focus on the divergent aspects of hardware and software to make their respective systems work reliably and efficiently.

Developments in telecommunications systems such as 5G are demonstrating that the transfer of high capacity data rates with low latency is possible. In the space of just a few years, IT has moved from backroom office applications to full blown broadcast quality real-time signal processing. High speed hardware is becoming more and more difficult to design and build, and the demands on the scarce skillsets needed to achieve reliable design is increasing as servers play greater roles in our lives.

All this demonstrates that the knowledge, understanding, and thought processes of broadcast and IT engineers can at times, be worlds apart. This is not because of any group being better than the other but is instead due to the different business environments that both broadcast and IT have worked in.

Mutual Respect

For broadcast facilities to continue to prosper, both disciplines must gain a deep understanding of not just the detail of the technology of their counterparts, but also gain an appreciation of the way each group thinks and works.

Building relationships with trusted integrators and manufacturers is often key to bridging the gap and facilitate cohesive and harmonious working between broadcast and IT engineers. There is much information throughout the internet on how IT and broadcasting work but it’s often difficult to evaluate the accuracy of the data being presented. Again, partnering with a trusted vendor and respected publisher can be of paramount importance.

The key criteria to developing new skillsets is to keep an open and flexible mind. Adaptability is probably the most critical personal characteristic any person can possess. And as we progress through IP migration, this is truer now for broadcast and IT engineers than it ever has been. But this is an attitude that must be championed and driven by the CEO, and then all the way down through the management chain.

Deep Monitoring

Troubleshooting in IP requires both IT and broadcast engineers to develop new methods of working and thinking. TCP (Transmission Control Protocol) is an efficient, reliable, and proven method of distributing data around IP networks, especially for office applications. But we do not use it to distribute ST2022-6 or ST2110 signals as the latency TCP creates is generally unacceptable for television. Instead we use UDP (User Datagram Protocol) – a fire-and-forget transport method.

Because of its durability, TCP can mask underlying network issues that may not be easily detected by office users. Most people surfing the internet, operating their finance software, or using word processing, wouldn’t notice lost packets of data and so wouldn’t report a problem to IT. But dropped packets can cause critical quality of service issues for video and audio distribution.

Switches and routers support various monitoring solutions to report dropped packets or deterioration in signal quality. But facilities use a multitude of vendor products to provide greater resilience and reliability. A high-level monitoring system such as Paessler’s PRTG provides a helicopter-view of the entire facility. PRTG constantly polls switches, routers, and servers to acquire critical data reporting and provide alarm and monitoring information as required. However, standard IT diagnostics do not understand legacy tools for broadcast such as ember+, therefore, broadcast devices now need to support diagnostic output interfaces such as SNMP, sFlow, etc, to provide complete monitoring systems.

Recording Ethernet traffic to gain information around seemingly random events is a critical tool. But knowing what and when to record is even more important as a 100Gb/s Ethernet link will soon fill a dedicated storage device.

Monitoring data on a network using an analytics tool such as Wireshark is often unacceptable when diagnosing real-time issues. Wireshark generally runs on a server or laptop, which in turn may well be using an off-the-shelf NIC (Network Interface card). The NIC is the device that interfaces the Ethernet with the CPU and, due to the buffering employed, will destroy any time critical information.

NIC’s are available to maintain the timing relationship but care must be taken when choosing them to be sure they provide accurate time referencing.

Beware of Switch Monitoring

Routers and switches provide facilities for network administrators to route and transfer specific data packets to a specified port on the switch, thus providing a convenient method of monitoring packets. But care must be taken when using these facilities as critical timing information could be compromised as the switch extracts the packets for analysis and transfers them through the fabric to the monitoring port.

Few IT engineers working in broadcasting will have experienced PTP (Precision Time Protocol) and gained the understanding needed to make it work. Broadcast engineers fully understand the need for accurate timing but may not have the knowledge required to design a PTP-aware network. Again, this is an area where broadcast and IT engineers need to collaborate and even pull in the help of a manufacturer or integrator who has deep knowledge and experience in this field.

Migration is continuing to progress at an unprecedented rate and few would have predicted such fast integration. All disciplines within the broadcast and IT arenas are playing catchup and not only need to understand their respective technologies, but also have a deep appreciation for each other’s working practices. Trusted manufacturers, consultants, service providers, and publishers can go a long way to make IP migration reliable and as easy as it can possibly be.