The flexibility of IP content handling has allowed the team—led by David Mazza, senior vice president and CTO for NBC Sports Group and NBC Olympics—to leverage as many as five different locations to process the live feeds coming from NBCU’s camera in Tokyo as well as nearly 100 4K UHD feeds from host broadcaster Outside Broadcasting Services (OBS) at the IBC. Signals will originate in Tokyo and be produced on site across dozens of Avid edit suites, or be sent back to either: the Sports Production Operations Center (SPOC) in Stamford, Conn., its main production facility; Telemundo, in Miami, Rockefeller Center in NYC; and Englewood Cliffs, NJ, home to CNBC—for inclusion into the main telecast as well as the network's myriad of digital platforms.

In addition, NBCU will utilize control rooms at the Sky facility in the UK to produce content, and station 250 people to work at a local hotel in Connecticut producing digital content. There’s dark fiber between the hotel and Stamford’s main facility.

So Much Content, So Few Staff On Site
In total, NBCU has said it will produce a whopping 7,000 hours of Olympics content by utilizing two broadcast networks, six cable networks, and multiple digital platforms, serving both English- and Spanish-language viewers.

For Mazza, preparations have been frenetic, trying to bring the NBCU eco-system together to support less people than usual on site, while also creating and distributing more content than any of the 16 Olympic telecasts Mazza has been involved with. They moved about 400 people home and then had to move people out of Stamford because it was too crowded. So NBCU will have about 500 less people than the Summer Games in 2016 in Rio de Janeiro, Brazil.

IP Requires Less Physical Equipment
Moving to IP has also meant having to send less equipment to Tokyo than ever. Typically the network sends the gear in preconfigured RIBS (Racks in a Box System) that are easy to set up on site. These RIBS are now all IP-complaint. Thanks to an on-site IP infrastructure, Mazza said they also saw a big benefit in reduction of coax cabling, which is extremely helpful for setup.

“It’s an interesting progression, because those RIBS went all the way back to 2000 in Sydney,” said Mazza. “We started with 13 of them. When we converted to HD in 2005 for the 2006 Torino games we dropped down to seven of them. And now in this IP conversion we’re down to two full and two half RIBS. If you plotted bandwidth against size, you could see how much more efficient the IP infrastructure allows us to be with a lot less hardware.”

Indeed, he said they now have much more capability with 3 RIBS than they did with 13. In Sydney the router was cutting edge a 320x320 SDI matrix. Now they are using a Grass Valley/Cisco IP router with about 1,000 x 1,500 I/O.

NBCU has sent over 100 Sony (HD and 4K) cameras to Tokyo and is augmenting that with 90 live feeds from the OBC broadcaster. The IBC 4K UHD feeds are a combination of all of the venue outputs (from roughly 38 venues), and a lot of other “beauty camera” positions that will be used throughout NBCU’s coverage.

For the last Olympic Games the IBC has been providing each host broadcaster with audio and video content stored in what the technical teams call the “VANDA+ Multi-lateral Clip Feed (MCF)”, which is “video and audio” clips of B-Roll material that doesn’t all make it to air (like unique POV and SSM shots).

“Almost every venue has an MCF feed that, in parallel to the competition, they are feeding all of this extra material that you might want to use in your tease the next day. For this Olympic Games, the amount of VANDA clips has doubled.

The network takes the output of its own trucks and flypacks on site into the IBC. So, they have about 160 incoming VANDA feeds daily and over 200 HD feeds go back to Stamford to support, all the at-home workflows that have been put in place and the Live Streaming. The number of feeds into Stamford is higher because some come “home” in multiple formats and at different levels of compression.

Most feeds are sent back to Stamford, where there will be eight control rooms and four mobile production trucks parked outside NBCSports headquarters. One is an extra truck that they call the “shift change” truck. This is for when shows are on the air for 24 hours/day, and they need a place to Air from while they are sanitizing the control room between crew shifts.

Venue Control Rooms In Different Locations
“When the pandemic hit, we realized we had to move more people home from Tokyo,” said Mazza. “It wasn’t feasible to send as many as we had the last Olympics. So, we moved most of the (Chyron) graphics operators and half of the editors to Stamford. Five of the venue control rooms in Tokyo—for indoor volleyball, basketball, beach volleyball, golf and diving—were all moved to Stamford. Each sport is dedicated to one of the trucks, some running at 1080p HDR, others 1080i HDR.

Sky is handling content creation for indoor volleyball. The feeds come from Tokyo Stamford on 8 large (10 Gb/s) AT&T fiber-optic circuits and then they are using 1 Gb/s Net Insight Nimbra circuits that exist between Sky (UK) and NBCSports (Stamford, Conn.). A Media Links system is being used at Sky to convert one of their control rooms over to 60 Hz, so the NBCU team does not have to convert to each clip that comes in as 50 Hz. The Media Links box outputs the 60 Hz feed to Sky, where the control room and EVS replay systems are flipped for the duration of the games.

OB Vans Handle Overflow In Stamford
The production trucks parked outside Stamford (running 1080p HDR) will work with a minimum of 12-16 feeds coming into each truck. Interestingly, many of the signals will not land in Stamford, because with so many venues to support, NBCU felt it had overloaded the system in Stamford. Literally, every technical asset located in Stamford will be used. That’s where the other production locations will come into play.

They have over 216 HD feeds coming home from Tokyo, across eight 10 Gb/s circuits coming back from Tokyo over fiber, satellite and the Internet, according to Mazza.

“We realized we were overloading Stamford, so we took three dayparts (the Olympic Channel, CNBC and NBCSN) and pushed those out of Stamford. The Olympic Channel was moved to Telemundo, in Miami (Tennis and Wrestling in 1080p60 HDR). NBCSN is host to a number of different events that are being controlled out of 30 Rockefeller Center in NYC. Olympics content appearing on CNBC is being produced at CNBC in Englewood Cliffs, N.J., which is using feeds from Stamford.

“There’s a Media Links frame that spews all of those venue feeds right into the trucks,” said Mazza, adding that all of the HD feeds going through the Media Links box includes hitless switching, providing 62 GB/s of hitless bandwidth.

“And then the trucks send finished programs back into Stamford. Sometimes it goes all the way back to Tokyo for the Primetime studio show, but otherwise feeds like the Golf Channel go to the Golf Channel master control in the NBC NOC at Englewood Cliffs, NJ.”

Resolving Latency A Concern
Sending content half way around the world takes dedication and compression to make it viable. That introduces approximately 180 milliseconds of roundtrip delay just on the network. For high-value, live content NBCU will use JPEG2000 circuits. Adding J2K introduces another 70 ms of delay. Working in MPEG, latency could be as high as 2 1/2 seconds.

“So the way we manage that is, if we’re doing venue production with camera cutting, and the feeds have to land in Stamford pretty much all at the same time, we’ll use J2K,” said Mazza. “If it’s a situation where the announcer is in Stamford and no one is on site in Tokyo, the announcer and producer don’t know that the video is late. So, in that case we’re using the IP VANDA from OBS—a 1.5 GB/s stream of all the OBS line cuts are muxed up and compressed, which we get as an MPEG-4 stream. We send it home and decode it. That could be up to three seconds late. But no one cares because there is no reference point.”

So, they are using a combination of the slower feeds, which are MPEG (where they are not worried about latency) and the faster J2K when they are worried about latency. Mazza said to the naked eye the picture quality looks pretty much equivalent.

“A layman could not tell the different, but there are some artifacts if you look closely at it,” he said. “It’s the latency issue we’re trying to solve.”

The team will also use a lot of Haivision MPEG-4 encoders for monitoring purposes, which takes up anywhere from 4-16 Mbps worth of bandwidth, are much faster than traditional long GOP MPEG-4. They’ll use those for program returns for the announcers at each venue. If they want to avoid burning up 200 Mb/s for a J2K circuit, they’ll use a 15 Mb/s Haivision box. In this way, the announcer for water polo in Tokyo can read the graphics coming from Stamford and not be behind by seconds.

“We have to get a program return to the commentators when they are at the venues,” Mazza said. “If they are in the control rooms in Stamford, both sides of the equation matter in terms of latency, otherwise they will be reading a graphic three seconds late.

4K Conversions At The IBC
The OBS is responsible for producing most of the 4K UHD content at this year’s games. They’ll be working with three flavors of video: 4K UHD, 1080p60 HDR, and 1080iSDR (that is still the house format in Stamford).

When a signal comes into the IBC, if it’s 4K UHD HDR, the NBCU team will downconvert to 1080p HDR immediately using XIP frame synchronizers from Grass Valley—which supports all of the different video formats in use— before it gets to Stamford.

“Sometimes they’ll be receiving two feeds (P and I) and sometimes they’ll take in P and derive the I from it,” Mazza said. “Sometimes we’ll be getting optical 4K and deriving the P.”

Logistical Limitations
Technology notwithstanding, Mazza said COVID-19 restrictions regarding the movement of staff and other personnel on site will be a big headache during the games. He and his team had to arrive in Tokyo 14 days prior to the Olympics to quarantine. Understandably, the movement of key staff is highly limited for everyone in Tokyo. (editors notes: these quarantine rules were streamlined on July 1^st when the IOC COVID playbook took effect).

“Our biggest challenge might be getting the right personnel in the right location at the right time,” said Mazza. “COVID restrictions are limiting much of the normal activity. To be safe, we will follow all the IOC/TOCOG rules and get tested repeatedly while we are here. I know we have a lot of technical challenges to overcome, but restriction of mobility will be the biggest challenge. All of this is hard during a normal Olympics. COVID restrictions make it doubly hard. But, we have an incredibly talented and passionate team of people working on this in Tokyo, Stamford, and all the other sites. And we are all super excited to get this going!”

###

Of note: NBC has held the American broadcasting rights to the Summer Olympic Games since the 1998 games in Seoul, South Korea, and the rights to the Winter Olympic Games since the 2002 games in Salt Lake City.