The “new” building, a former Clairol hair shampoo factory, soon was home to NBC Sports, NBC Olympics, NBC Sports Digital, and NBC Sports Network. For the division, it was truly a time of expansion in both its linear and online channels and the new space provided the perfect blank canvas to design and deploy new types of collaborative production workflows for its staff and as well as fast and accurate delivery of content to all of its distribution platforms. The house format was 1080i HD.

As the sports landscape changed and new sports-related businesses (The Olympic Channel, Golf Channel and others) were launched or relocated there, Stamford’s internal infrastructure grew significantly in the amount of incoming feeds it could handle (from about 1,500 per month in 2012 to more than 6,000 per month during the Beijing Olympic Winter Games in Feb. 2022) and production staff it could accommodate.

HD To The Core

At the core it all started with a main router (an Evertz 1152 x 1152 baseband router, which is still in use today) and the building of four large studios and adjoining production control rooms. Once this feed capacity was exceeded, sometime in 2014, an NBC engineering team led by several experts in the media industry, including, Tim Canary, Senior Vice President of Engineering for the Stamford facility, began its first foray into IP with an Evertz EXE router, using IP gateway cards to accommodate baseband signals.

“Our initial move was to put a number of multiviewers and other things on an Evertz EXE IP router, in order to free up outputs of the EQX baseband router for our control rooms. But the [fifth, called “PCR8”] control room we were building at the time became larger than we first anticipated, so we said maybe it’s time to put that control room on the IP router with gateways,” he said. They had also deployed the Evertz ASPEN format, which provided separated video, audio and metadata IP workflows for NBC Sports’ HD productions.

Testing The IP Waters

Building the IP facility based on ASPEN gave NBC Sports greater flexibility and efficiencies over an SDI replacement. It also provided complete control over sending specific video and audio signals to particular destinations without the need for external embedding and de-embedding devices. ASPEN also maximized utilization on the 10GbE interfaces within the facility to ensure the production team was getting maximum throughput and efficiency in its IP core matrix.

Evertz’ Software Defined Video Networking (SDVN) platform—which included a EXE40-VSR router with 46Tb/s of switching capacity in a 40RU rack, 570IPG high density media gateways, and 3067VIP10G 10GbE multiviewers—was all controlled by Evertz MAGNUM SDVN orchestration software. What it did was give NBC Sports the ability to scale the facility to meet current and future needs.

However, those future demands soon skyrocketed, and the EXE router’s 10 GB/s capacity was exceeded once again. They now required 100-400 GB ports and the router’s 10 GB backplane would not allow for it.

“That router infrastructure was only provisioned for 1080i, so we couldn’t pass through any 3G signals without additional investment and infrastructure,” Canary said. “Also, at the time each gateway card could only be all 50 Hz or 60 Hz. And we were looking at the upcoming [2020 Summer] Olympics, where we needed to have 50 and 60 Hz mixed almost on an input by input basis.”

COTS-Base Signal Routing

The next crucial decision was to upgrade the infrastructure and migrate to hardware which offered more bandwidth and supporting ST-2110. A COTS-based system was chosen from Cisco/Grass Valley using GV Orbit as a SDN and control system. GV Orbit gave them a consolidated, overarching configuration, control and monitoring platform for orchestration of the facility’s broadcast media networks—which were now a collection of SDI, hybrid and pure IP islands. GV Orbit’s underlying architecture, however, is designed specifically for open standards-based IP systems and gave them the ability to configure and change systems and workflows on-the-fly, easily adding/removing devices or simply changing a name on the network.

The GV ORBIT was used for the Tokyo Olympics in 2021 and Beijing Winter Olympics in 2022. It allows the team to have mixed formats on every gateway and card in the router and add new cards as needed. Today the GV ORBIT system takes the place of the previously installed SDVN controller. The facility still uses Evertz MAGNUM to do all of the end user interface/control out of the EXE router as well as seamlessly communicating with GV Orbit. There are also a number of tie lines going between the Evertz and Grass Valley systems.

“To the end user, it was the familiar Evertz panel and interface, so they didn’t see anything different,” Canary said. “This was important to transition the facility smoothly.”

As with nearly all businesses, the pandemic reduced the number of people permitted in the building and put an emphasis on remote working. This caused them to have a team produce the Kentucky Derby and Notre Dame Football games from a production truck parked outside the building in Stamford.

(Owing to its “Herbal Essence” shampoo factory roots, the building conveniently features a multi-bay loading dock, which the team has outfitted with air conditioning systems to keep critical hardware cool during summer months.)

Networked Production Systems

Besides the infrastructure, many of the production tools have been upgraded with time. The facility now boasts a total of nine control rooms, including three large rooms, three mid-size rooms, and three micro control rooms. The large and mid-size rooms feature Sony switchers where the micro control rooms feature Simply Live and Ross All in One systems.

Some of the original Sony MVS 7000 HD switchers in the first four rooms have been replaced with XVS 9000 4K versions, to give them more in/out capacity. ­—the 7000 had 80 inputs and 40 assignable outputs, while the 9000 has up to 160 inputs, 80 outputs along with 16 baseband multiviewer outputs and supports 2110. There are also new SonyHDC-3500 and HDC-3200 2/3-inch 4K cameras, which can capture in native 1080p HDR,

“Even though we were experienced producing a portion of our Olympic content remotely, this move helped us enormously with the Tokyo Olympics because we had to produce so much more content back at home than we typically did for other Olympic Games,” said Canary. “So we needed that extra capacity for the larger control rooms. And even more so for the following Beijing Olympic Games.”

He added that for the Tokyo Games, most of the Stamford facility was still baseband with some IP gateway outputs feeding it.

“For Beijing, we didn’t have direct IP in, but we needed additional inputs to the switcher going over baseband through gateways.”

1080p HDR House Format

At this point, content from Beijing was being produced in 1080p HDR in addition to 1080i. For the NBC broadcast network’s Olympics coverage, all 4K content is produced in 1080P/HDR for the highest quality HD results in Stamford and then upconverted at the network’s two main distribution points.

Due to the flexibility of IP routing system, the staff was able to produce the Closing Ceremony of the Beijing Olympics until the late hours and then the next morning they were back pulling out baseband input cards on the Sony 9000 Switcher and putting in IP input cards to connect the switcher directly to the IP router so they could start producing the USFL games.

“So this growth pushed us to fill that switcher with IP cards and just grow to where we now had 4-5 IP cards in the frame,” said Canary. “In PCR 2, for example, all of the cards in the switcher have been pulled out again and it’s been populated with all IP cards and maybe one or two SDI cards in the switcher. But the entire switcher is fed off the GV IP router.”

There’s now well over 100 routable inputs into PCR 2’s switcher, which provides the capability for large-scale events from that room. This is their go-forward plan – 2110 signals directly into all the production switchers.

Remote Production And The Return Of Staff

While many sports productions that had been completely on site at an event were handled remotely during the pandemic, many of the lessons learned have carried over to NBC Sports’ current operations. And many more staff members are now working out of the facility again.

There’s been a lot of experimentation with what they call “shadow” production for remote sports production, whereby they’ll put cameras and some functions on site but also send “splits” back to Stamford to switch in a control room and complete the show. This work expands upon remote or “REMI” production techniques that NBC Sports has used since the 1996 Atlanta Games.

In addition, with the volume of remote events they produce, they have built out three slightly smaller production control rooms. Instead of a three-tier control room, it features two tiers for the director, TD, producer, three even smaller control rooms based around Simply Live and Ross Video All In One technologies—which include IP multiviewing, and Calrec Brio-12 audio consoles. There’s also a submix room with another Calrec Apollo console which utilizes the Calrec Impulse audio processing and routing engines which are interfaced with the legacy Hydra 2 systems.

As for utilizing the cloud, NBC Sports has taken a measured approach, starting with some S3 storage of archival material on the AWS Cloud. Most content is stored on premise and off site, on high-capacity LTO data tapes. Canary said it’s more efficient at present than storing this amount of data in the cloud. They are also looking at some cloud-based control rooms and other remote production technologies for the Paris Olympics. This includes evaluating Grass Valley’s AMPP as well as remote production workflows from Sony, Simply Live, and Ross Video.

Ever-Expanding Production

In 2022, yet another control room, PCR 12, was added with an additional Sony XVS 9000 switcher and Calrec Artemis console, to support even more content production. Teams from operations, engineering, finance and facilities worked together on the expansion project with much of it happening during the lead up to the 2022 Beijing Olympics and Super Bowl LVI. Construction began in March of 2022.

The other thing they did was to build 13 new smaller multi-purpose rooms to replicate a crew sitting in a production truck. These are called “Flex” rooms to help keep up with the volume of content they need to produce daily. During COVID temporary announcer booths were used extensively to socially distance people. The concept worked well, so it has continued on a permanent basis, albeit with sliding walls that allow one room to become two when required.

These Flex rooms can support everything from Chyron graphics and augmented reality content creation, EVS replay systems, Evertz multiviewers, golf ball tracking technology and RTS intercoms.

“We built 12 of these 2 seat rooms for flexible production use, each with sliding glass doors, in groups of four, and put dividers between them that can be opened to make double the space if we need more capacity for a production,” said Canary. He added that they also have one Super Flex room with eight seats in it and lots of monitoring, which was once the facility’s tape library. As tapes has been digitalized, they’ve been able to reuse the space for other things.

[Of Note: About 500 to 1,200 tapes per month are being digitized to LTO-9 cartridges and reside on the facility’s MAM system.]

Orchestrating The Expansion

The past decade has seen an amazing expansion and a lot more content coming through Stamford, as the facility can now do much more than it ever could. A whole lot more. And they can do much larger shows now because more of the switcher I/Os are routable.

While most of its internal scheduling of control rooms and other resources is performed manually with a ScheduALL system, including signal routing. This requires a technician to go in and set audio levels, QC and other production and distribution parameters. Canary said that they have found that doing it manually reduced some of the errors they’ve experienced with automated systems to date.

However, software orchestration does make sense in some areas which they will continue to leverage. The NBC Sports engineering team has begun implementing a new software-based orchestration layer from EVS called Cerebrum. The software is being tested within the Flex rooms now, but the plan is to use it facility-wide prior to the Paris Olympics in 2024.

“Cerebrum lets you do a number of neat things between routing and some of the things we really need to be doing moving forward,” said Canary. “It makes us more efficient. We know that as the volume of work increases, we need more efficient methods to perform the tasks.”

Feeds come into the plant over fiber. When a baseband feed comes in, Cerebrum will eventually route it to any frame sync in the building.

The facility had maybe 150 frame synchronizers ten years ago, now it requires more than 300, said Canary. They’ve all been converted to all IP frame syncs, so all of the baseband signals coming into the plant hit the frame syncs with a 2110 IP output to the router. Tie lines back to the coax router are used as needed, but almost all of the workflows are moving to IP based systems (some legacy frame syncs are still baseband).

Mission Accomplished?

While NBC Sports and its Stamford headquarters has been able to keep up with on-going demand—including the recent addition of the Peacock OTT service—and accommodate the needs of the network sports division, Canary said there’s always more that can be accomplished.

“As an engineer I don’t know if you ever get to where you’d ideally like to be, but we have refined the technology and workflows we need to produce the content we need,” he said. “There’s always a better way to do things. This facility has been flexible enough to allow us to do what we need to do and things we never even thought of ten years ago. I think we’ve squeezed a lot out of the building… which has been a home run for NBC Sports and allowed us to grow in a measured and economical way”