Transitioning to IP is a major challenge for any broadcaster, especially when moving an existing facility. Greenfield sites have it difficult too, and learning how they successfully made their installations work will help us all.

BCE’s wish was to be format agnostic and ready for future technologies such as 1080P, 4K and HDR. Preliminary requirements dictated the system should be flexible, scalable, redundant and reliable. Aspirations for us all, but often difficult to achieve. Cable installation was identified as a major area for cost savings so fiber was used extensively. To de-risk reliance on a single supplier, multiple vendors and COTS equipment was used wherever possible.

Waiting for ST2110

During the design phase ST2110 had not been ratified so Costas and his team needed to decide on the specification of the system that was available at the time, but would allow for future upgrade to ST2110. SMPTE 2022-6/-7 was chosen for SDI video and AES67 for audio over IP, and DANTE was used for RTL Radio. 

To reduce the amount of copper cable required, fiber was used wherever possible, to the extent where all connections between buildings, floors and rooms were fiber, and the only cables carrying video that exist are intra connections within rooms and racks.

Fiber Challenges

And this is where the challenges began.

Fiber is an established technology and has been in active use since the 1970’s. However, the combination of interfacing it with SDI, and the variable quality of fiber available, caused Costas no end of issues. It soon became apparent that fiber and the associated connectors, SFP and QSFP’s vary wildly in quality and more attention had to be paid to the procurement stage. Systems could fail in a non-predictable manner.

During investigations into the issues Costas stumbled across another challenge, the differing attitudes and working practices of broadcast and IT engineers. Broadcast engineers are used to working in new, dynamic, fast paced environments where equipment often goes wrong and creative, out-of-the-box solutions must be found. 

Banking Influence

IT engineers have their training and influence in the well-ordered world of the banking industry. Many of the working practices IT engineers work to are well established in the finance market, and their ITIL processes exist for good reason as decimal points can make an enormous difference in the finance sector.

Information Technology Infrastructure Library (ITIL) is a set of detailed procedures used in the IT industry and are designed to bring order to a system to stop engineers jumping into systems and fixing them without proper authorization. This method of working is usually alien to most broadcast engineers, but in the IT world there must be an audit trail for every change or modification. As networks typically affect so many different services, one change on one part of the network could have unintended consequences on another. ITIL aims to stop these actions but the system only really works in an established working environment.

IP television is at the cutting edge of technology and driving IT to its limits. Engineers are finding obstacles nobody had predicted or expected. Consequently, when something goes wrong it must be fixed immediately, there is no time to raise a help desk ticket or have five meetings about the proposed fix.

Forward With ITIL

Broadcast television will eventually move to the ITIL model and we have already witnessed this in SDI playout facilities. The last thing anybody wants is an unscheduled software upgrade causing outage on fifty pay-per-view channels. But we must have working systems before these processes can be instigated.

ITIL is used to maintain, install and modify systems that are understood, well-ordered and defined. Broadcast IP is not there yet. But it’s fair to say that if we had stuck with SDI playout systems we would probably be using ITIL extensively now. However, to facilitate the new business models of flexibility, scalability and COTS, we must move to IP and we will continue to find solutions to complications we do not yet know we have.

Improve Equipment Monitoring

One of the areas Costas highlighted for equipment design improvement is monitoring the SDI-IP interface where television turns from copper coax to fiber. He also identified varying quality in SFP/QSFP light sources causing packet loss. Using light power meters from the telco industry, BCE’s engineers could prove that fiber channels were failing, but in a further twist this always wasn’t due to light transmitters. 

Procurement Specifications

Terminating fiber proved to be fraught with the same cost issues affecting fiber. Inferior quality connections don’t tend to just fail, they start by creating unpredictable erratic packet loss. Channels would work one minute but not the next, again the origin of the failure proved difficult to find. Was packet loss due to issues in the SDI domain or IP? A failing SDI connection can cause packet loss on an IP connection due to the amount of erratic data being created.

BCE solved the dilemma by assigning a small team of wiremen (fiber-men) to running and terminating or connecting fiber exclusively, a task requiring great skill, patience and experience, and giving highly specified fiber specifications to his procurement department ensuring the use of high quality fiber. Dirt is the fiber’s greatest enemy as even small amounts of debris can cause major headaches for video-IP installations running at the limits of their capabilities.

In article two of “BCE Going Deeper Part 2 – Choosing IP Routers” we continue our conversations with Costas and his colleagues.