At 419,000 sq. ft., the new building—located on the same property as its existing SDI-centric facility in Montreal—is about two-thirds smaller than its previous headquarters, yet when they move in later this year the CBC news and entertainment teams look to be more productive.

The building will be responsible for over 100 TV, radio and web channels while allowing the broadcaster’s production teams to seamlessly manage multiple audio and video formats and mix IP live inputs using back-to-back IP sources and file-based clips Adhering to an open standards-based IP approach by basing its workflows on the SMPTE ST 2110 standard allows the CBC to be fully format and resolution agnostic. This complex infrastructure also provides the needed flexibility and scalability to adapt its services to changing viewer demands.

Spine-And-Leaf Topology

François Legrand, Senior Director, Core Systems Engineering Solutions at CBC/Radio-Canada, said that the infrastructure is designed as an IT-centric spine-and-leaf topology, with two identical and redundant signal paths. However, a pure spine & leaf topology can’t be non-blocking for the type of multicast traffic at CBC, so they designed an infrastructure based on this architecture, but had to make two important modifications to make it suitable for broadcast applications.

“Ninety-nine percent of our real-time traffic is multicast, so this is a challenge,” Legrand said. “To overcome this we use a single modular spine switch per network area, which is itself internally built like a spine-leaf network. We have four networked areas, thus 4 spine switches and 4 segregated zones of failure (production red, production blue, presentation red, presentation blue), red and blue are the redundant paths.”

These Production spines are handled by Arista 7516R2 switches while the presentation spines are Arista 7508R2 switches.

“The second thing is that the random path selection done using IT traditional technique such as ECMP is based on hash compute on IP packet headers,” he said. “This yields the same result for all of the packets of a specific media flow. So to work correctly a very high number of flows per 100 Gb/s link is required. This works for audio but not for video, even in HD, and it’s worse in UHD. For this reason, the network is software defined using the Arista AMCS software. Equal distribution of flows between all uplinks is then guaranteed.”

The new building’s data center hosts the spine switch and wherever they need network connectivity they install a leaf switch. The link between the spine and the leaf features as many100 Gb/s bi-directional connectivity as needed, anywhere in the building. Between the leaf and each end device it is whatever the end device requires. So, 100 Gb/s gives them plenty of capacity to seamlessly share audio, video and data.

“The network doesn’t care about video resolution or audio stream size,” said Legrand, “It just needs to be able to recognize an end device and control it instantly. Unfortunately, every company offers a different way to talk to their devices. That’s a problem.”

The new building is multifunctional, supporting the CBC’s News and General TV production departments. There are three studios for general TV production and two control rooms. Any control room can be assigned to any (or all) of the studios.

“We want the entire building to be used as a giant production studio,” he said.

There are patch bays on the walls of each department, allowing journalists to report on-air from their desk in the newsroom or from an anchored news set. They can even plug in a camera on the roof of the building.

Software-Defined Architecture

The new IP technology installed—from companies like Arista, Canon, Cisco, Dell, Embrionix, Evertz, EVS, Grass Valley, Lawo, NetApp, Meinberg, Riedel Communications, Ross Video, Sony, Tag Video Systems, and Vizrt—enables smaller, more efficient use of the square footage available. In a greenfield IP build like this one, all of the equipment itself requires less space. For example, in the current building they have 550 equipment racks in the machine room. In the new building that number has shrunk to 146 racks. So, the building is smaller, but CBC will have the same number of employees as before creating and distributing content.

“We have fewer studios, but we expect to be producing the same amount if not more shows in the new building,” said Legrand. “In the past we had studios dedicated to a specific show. So one studio would do one show. We are changing that model so that our biggest studio in the new building will be able to produce four different shows a week. It is totally reconfigurable and reusable in any way we require. The new IP technology we’re installing is much more flexible than any technology we have used in the past.”

This flexibility comes from the combination of IP and software. 95% of its Playout/Master Control capabilities are now Intel x86 software based (the Grass Valley iTX system and TAG Video Systems’ multiviewer will also provide full disaster recovery between the broadcaster’s facilities in Toronto and Montreal. The Toronto piece will start to be implemented this summer), and 40 percent of its production tools are software-defined as well. This includes multiviewers, video servers, graphics systems, editing, etc. Legrand said he wants the later number to increase as fast as he can get manufacturers to support his ST 2110 architecture and software-defined approach. There are more than 150 Lawo V__matrix C100 software-defined processing blades, used for such applications as multiviewing, system monitoring and content steaming.

Moving From SDI To IP

Each of the new facility’s control rooms include a Ross Video Aquity video production switcher, which is used in tandem with the Ross OverDrive automated production system in several of its control rooms. The switcher is SDI based and therefore does not include an IP input or output. The workaround was to install Embrionix gateways to convert the SDI signal to IP. These gateways are not needed for equipment like the newly installed Sony HDC-3100 cameras (with Canon lenses) for the News department and HDC-3500 models (with a native 4K sensor) for the general TV production team. Both cameras natively support ST 2110 through 25 Gb/s network interfaces. SMPTE fiber is used to connect the cameras and CCUs, but from the CCU its native IP. Most CBC News production will be captured in the 1080i HD format, but the News department plans to move to 1080p and the entertainment department to UHD in the near future.

Tying It All Together

Legrand said one of the biggest challenges to making the facility run smoothly is getting all of the devices on the network to talk to one another. They have solved this problem by implementing a control layer on top of the 2110 network which supports a number of protocols in order to control different devices. That control layer is handed by Lawo’s VSM platform.

The virtualized VSM platform has been implemented as two clusters with three servers each. Throughout the facility there are 50 vsmPanels, more than 200 webPanels and over 2,000 control ports.

“The first components we chose for the new building were components that we knew we could easily control,” he said. “When we selected the VSM platform, then it was a matter of choosing components that generated the Lawo Ember+ protocol that VSM supported. When he started 18 months ago there were so few of them that it made our equipment choices easy. If they had implemented the Ember+ protocol or some other standardized protocol that was already supported by VSM, that’s what we implemented into our ecosystem.”

He said VSM has made a huge difference. Six months ago to install a new device on the network and control it would have taken two weeks of work. That’s if you have the correct code. If you have to write your own code, and the CBC team has had to do this on several occasions, it takes much longer (two months). Now, the VSM interface recognizes a new device in a much shorter timeframe.

The latest version of VSM supports the AMWA NMOS IS-O5 specification, which has been gaining traction among facilities looking for an open solution that is not tied to any one manufacturer. The spec describes a mechanism for handling connections between receivers and senders on an IP network. It’s like the IP version of patching SDI signals. Automatic discovery and registration requires IS-04, which is not supported by VSM at this time. In the CBC’s case, this would allow the Sony cameras and CCU to be automatically added to the network, no matter where in the building they are being used. So, thanks to the VSM system, the Sony cameras can be remotely virtually patched to any sender and receiver using the IS-O5 spec from the control room.

“The role of the VSM control system has become extremely important in the world of IP,” Legrand said. “and it’s crucial to the success of our project.”

Audio equipment includes two new Lawo mc²96 consoles and four mc²56 mixing desks for TV (both with a UHD core) and 17 Ruby radio consoles of varying sizes (12/16/24 faders) that support the AES67 networking standard used at the CBC. Legrand said that other manufacturers provide Dante networking interfaces, which are IP-compliant, but they don’t interface very well into a pure 2110 ecosystem. Finding enough audio equipment that does interface with 2110 equipment, he said, was another big challenge for the CBC’s facility design.

In IP, What’s An I/O Matrix?

When discussing infrastructure, in the world of SDI you talk about how many input and output ports you have and then what you want to do with them. It’s fixed and you don’t have a lot of flexibility. In IP you gain access to scalable bandwidth capacity. Control and orchestration of the two SMPTE 2110 spine-and-leaf networks is accomplished with 300 Arista switches, consisting of a 1RU box with 48 ports available for end-device use. Then they have installed four spine switches, which are much bigger, providing 450 Tb/s routing capability.

“As long as we don’t exceed that raw bandwidth, we can use it for radio, video (HD, 4K and 8K) and everything we need to do,” Legrand said.

So Legrand describes his facility’s capacity in terms of the number of multicast “flows” that are in the system. A typical TV signal includes four flows: One flow is used for picture data, one is the ancillary data (closed captioning and associated metadata), and two of them are reserved for audio (a 5.1 flow and stereo flow). The CBC’s new building will have more than 50,000 flows moving around at any one time during peak activity. Thanks to the flexibility of IP, one signal can include any number of flows, depending upon if it’s audio-only (radio), video (any resolution) or data-only (control and ancillary).

Opening Soon

The new building will open sometime this year, but due to the unfortunate pandemic the exact date is hard to pin down. The good news is that this IP infrastructure has allowed engineers staying at home to log into the network and monitor, diagnose and solve problems from afar. News operations employees are also remotely using the control room to prepare the next production season.

The first piece of wiring was installed in August 2018 and the finished building (without equipment and production studios) was turned over to his team on February 15, 2020. It’s taken over 18 months to get here, but Legrand said it’s been well worth it.

“To be honest we accomplished our original goal and went beyond that,” he said. “We knew we wanted to be an all-IP facility. We made the decision to go 2110 at a time when the standard was still being finalized and there were questions about other competing standards. But we didn’t want to be tied to any specific vendor. And we wanted to use COTs hardware as much as possible to reach maximum flexibility. So, we’re exactly where we thought we’d be 18 months ago.

“There’s no plug-and-play configuration in IP,” he said. “This is something the CBC is working closely with the vendors on. We need to automate the deployment and configuration of devices. We’ve tried to inspire ourselves by looking at what the cloud industry is doing. Because any device we add to the network is like a little server. Companies like Amazon and Facebook have millions of servers and they manage them in terms of the configuration requirements at the moment. We are currently using Ansible and other open source tools to get there. That’s what we’re doing with broadcast audio and video: getting any device [signal] to any person and any moment. When it works the way we designed it, it’s quite impressive to see it in action.”