About the Live Sports Production series: 

This is the first of a 12 article series which uses a round table style format to explore the technology of live sports production with some of the industry’s leading system designers. It is a fascinating insight into what is being done every day around the world.

This series is based on a set of interviews with six individuals who are the senior system architects within three of the most respected organizations in broadcast: Broadcast Solutions, Diversified and NEP. We cannot thank these individuals enough for giving us their time and sharing their immense wisdom and experience in the delicate art of designing and deploying broadcast production infrastructure. These interviews spanned many aspects of live sports production and their comments will be serialized across this whole series.

One of the most fascinating aspects of discussing where the industry is in its technological and workflow evolution is that there is no one single answer to most of the key talking points – each of the organizations supports a diverse portfolio of clients across a range of geographical locations – and the approach taken is always carefully tailored to specific rights holder and broadcaster requirements.

As we get into our conversation, it is important to understand that we have three quite different businesses represented; Broadcast Solutions build OB Vehicles and do Systems Integration primarily in Europe and the Middle East, NEP are a global production Service Provider, Diversified is a Systems Integrator with the representative taking part here based in the USA.

Bear in mind the obvious differences between designing and building individual systems for others to use and NEP’s capacity to build systems at scale for their own teams to use.

We began our conversation with the rise of remote production workflows and the current balance between OB based and remote production happening across different sports and countries.

Antti Laurila. Chief Strategy Officer, Broadcast Solutions: “Of course we do both but neither is better. It depends on the use case and also the region. If you look at northern Europe the trend is definitely more towards remote than OB’s. But then the further south you go there may be more OB’s but it depends on the country and the connectivity that is available. It’s not black and white. In the Nordics we have built a couple of remote production centers and venue kits for different sports but they do still have some OB’s. Those companies who are doing remote are usually doing sports leagues, whether it’s football or ice hockey, and you have productions five days per week. Remote starts making sense when you have a lot of content repeating the same type of the productions.”

Rainer Kampe. CTO, Broadcast Solutions: “I think we need to highlight that remote production is mostly related to long term rights contracts because without having the rights for a certain period, it doesn’t make sense for the service provider. With remote production there is a certain investment in the beginning and it needs to pay off during the rights period or the period of the production contract. These long-term contracts are mostly related to second and third tier leagues. We don’t see the Premier Leagues going remote now, but maybe, for other reasons it is thinkable.”

John Guntenaar. CTO, NEP Europe: “More and more sports productions are moving towards remote production, but it’s really a case-by-case basis and we still see quite a lot of OB work. For high volume work, for instance in Norway, horse racing productions are completely remote, and a lot of second league and third league games are done remotely as well. In Sweden, tier one and tier two ice hockey is mainly produced remotely. Then if you look at soccer, in some countries where the travel distances are longer, it makes sense to use remote productions so that you have the ability to share crews and talent. In other locations where the distances are shorter, like Belgium and the Netherlands for instance, everything is drivable within a day and producing it remotely actually means sending more people. It does however take time for the client to financially justify the move towards the remote model. The more widespread in terms of travel distances is the geography presents, the more beneficial it is to do things remotely.”

Antti Laurila. Broadcast Solutions: “The main driver in my view for remote is more that the customers are seeking operational efficiency and cost savings. That’s the big thing for remote in general. If you want to do more with less and cheaper, with remote some calculations can achieve that by reducing the travel cost etc. Where you have a long term contract, then you can do it cheaper. There is clear evidence on the projects we have delivered that it’s cheaper compared to OB productions, because the rigging time is shorter and so on.”

“Also the quality control is better when everything is in the hub.  Another thing is that all the video refereeing, whether it’s a VAR in football or ice hockey, they are also doing that in a centralized way to get all the referees in the same building. The signals are coming anyhow to the hub, so then it starts making the sense to combine the production and refereeing together.”

Dan Turk. CTO, NEP Americas: “I would say the determining factor comes down to scale. With some live productions, doing things remotely provides cost savings. If we’re not traveling people, they’re able to work on two shows per week instead of one show. That all helps the network or client, and the bottom line. With the increase in rights fees in the U.S., everyone is looking for efficiencies. How do we produce the same level of show, of storytelling, of engineering, and not compromise, but do it more efficiently? If that means we can have a producer not spend three travel days, but be able to go to one place and do one show per day, that’s an efficiency. A lot of times you hear people are slower to adopt the remote workflows, but once they experience it, they can see the benefits. Sometimes it comes down to personal preference. There are still many that want to be onsite, to be close to the story, because we’re all here to tell the story. and engineering a path to do that is just one part of it.”

Patrick Daly. VP Media Innovations, Diversified: “There’s certainly a lot of demand still for OB trucks. There’s a comfortability with what has worked, and will continue to work. I’m sure much like the Diversified supply relationships with our clients, the truck companies have very strong relationships with many of these content producers and you don’t walk away from a partnership like that at the first sight of potentially increased margins. You figure out a way to work together and succeed together.”

Workflow Models

Our discussions moved on to the workflow of different approaches to live sports production. How are things being done?

John Guntenaar. NEP Europe: “Remote production comes in various forms. For us, there are three different remote models. There’s the signal acquisition model, a completely remote model, and the hybrid model. And then of course you have a traditional model with an OB truck.”

“We have some clients that are in-sourcing some of their production and they just want to acquire the feeds into their own facility, that’s the acquisition model. Which is what you see happening in the UK with Sky, for instance. With the completely remote model we essentially only send cameras to the venue. Maybe we have some local audio mixing if the latency is too high to do it from a central location, because video is easy, but audio becomes more difficult very quickly if you have higher latency. Basically, it’s unusable, especially something like intercom or talkback. If your intercom ring lives in a data center and you are 40 milliseconds away, when you are talking to your neighbor over talkback that is very challenging. So that’s why you need to have local audio processing. But either way, you can have all of the production people at the remote location.”

“There’s also the hybrid model where you would use the equipment in the data center, but you would have all of the control surfaces on premise. The main reason for the hybrid model is because the creative people want to be onsite, or as some people say, ‘I need to be able to smell the grass in the soccer stadium’. At the venue you would have something like a truck or a small truck that has all of the control surfaces, the vision mixer panel, and the screens and the instant replay remotes. But all of the stuff behind the curtain is actually in the data center.”

“With the hybrid model, we still have the ability to have a higher utilization of the equipment that’s housed in the data center. We have all of the benefits of being in the data center, including having redundant cooling, redundant power, and generators. We have all of those bells and whistles that you do not have at the venue location. This is especially important with IP. With baseband, if the truck loses power, they can re-boot and the truck would be back up in minutes. In the IP landscape, the booting of a switch takes 20 minutes or so before it is up. A power failure whilst you are on air is unfortunate regardless, but on an IP truck it is a much bigger deal than with a baseband truck.”

Patrick Daly. Diversified: “When we look at the capital expenditure to put one of these trucks together, to build the capabilities that you need, you want to utilize that at a high rate.”

“By placing those components into a data center environment, making them accessible 24 seven and really deploying them as a software defined set of capabilities, the same components can be used for game day, and maybe overnight can be used to run inference on your archive, to do object recognition, brand recognition, whatever the latest models may hold for us. There’s some value in that aspect for sure.”

“It’s not just the utilization of the components of the technology, but utilization of our workforce is also enhanced through some of these remote production capabilities. Yes I can fly my best operators to site and they can run the production and it will be great, but they’re not available for anything else that day and it takes away from their work life balance. At least in the US just because of the geographic sprawl, doing an event in LA if you’re a New York based person you’re chewing two days of travel.”

“But having these capabilities it’s really not about either or, we at Diversified keep coming back to the term decentralized. It’s less about moving to the cloud and more about moving the workloads to the appropriate infrastructure at the right time. Which may mean I’m a license holder for an application, and I’m responsible for orchestrating the movement of that application between on premises into the cloud, maybe at the edge, at the venue, on an OB truck or on some COTS layer at the venue or I’m subscribing to a SAS product. I’m just hooking my sources and my operational staff into that product to produce the event.”

“There’s the tier one events and we’ll probably always roll a truck for tier one events, if for nothing else but backup. But here in the United States high school football is a very big thing for communities. Millions of dollars a year goes into production equipment and events. It’s a thing but they’re way down the long tail of production, so I think remote is good for that.”

Dan Turk. NEP Americas: “A lot of times it’s about being able to utilize people better, because you can take your A level people and do more shows with them, because you’re not spending days of travel on either end of every single show. The networks can be more efficient, the people can do more work, you can get more content out of the same amount of time. That’s one of the hybrid models where we’re still seeing a pretty large onsite presence, and that’s proven to work really well. ESPN refers to them as REMCO’s, where the creative production is really onsite, the switcher is onsite, and audio is onsite, but we’re taking a few rooms to remote control things. REMCO is probably a little bit safer because your on-air signals are still master grade, and uncompressed. Obviously, some devices like the replay machines do some compression, but it’s mostly master grade onsite and normal transmission feeds being sent out.”

“Then the other approach is the true REMI model where your audio and video collection is onsite, maybe there’s some talent because you want to talk to coaches and walk the sidelines and kind of get the story, but the main elements of the production are back home. Production is back on the switcher, the replay devices and audio console, all of that is back in a central facility where they connect to the site to acquire signals. That’s a higher bandwidth model because you’re trying to send with lossless compression, all of the video and audio from the venue to wherever your REMI facility is. A lot of the networks in the U.S. have built up facilities in their buildings so that we can connect. We have five REMI trucks in the U.S. right now, meant for signal acquisition, a small audio console for IFB mixing, and some intercom. There isn’t a switcher or big audio console on site, instead, those elements are all back in the facility.”

John Guntenaar. NEP Europe: “The benefits of moving into the datacenter model is that we have the benefits of scale, where we can share equipment. For example, you can share a vision switcher. Many more spare parts are available. It also provides a setup where redundancy comes at no cost. The bigger your installation is, the easier it becomes to do things at scale and get the benefits from it. It makes it quite easy to light up a control room, and if the client wants the big space for a small production or a smaller space for a bigger production, or if they need some additional elements, like extra replay channels or an additional multi viewer or to have two extra graphics operators, then it becomes much easier to add those. With an OB truck designed to be running at 100%, if you need it to do 110%, then all bets are off and then you would need to improvise. It makes it way easier to manage and maintain and have it in a fixed installation. With a data center facility, we have more ability to spend a lot of time on things like monitoring and the provisioning and repurposing of equipment.

Preparation & Planning

Does remote production require more planning and preparation than OB?

John Guntenaar. NEP Europe: “I don’t think it does. When we are converting a client from using a traditional OB into remote production, we are already familiar with the job. We have so much experience with those clients and those projects that we know what the requirements are.”

“The way that we have designed our remote production facilities with all of the shared resources, is in such a way that we know we have capacity on our centralized production platform or connected production platform for two medium shows, and a large and an extra-large show.. We want the benefit of scale, similar to a truck where you drive a truck somewhere and produce a show, and then you drive it back again.”

Dan Turk. NEP Americas: “From the technical side it’s a bit more challenging. There’s a whole extra layer. Where you’re used to having one person able to touch everything right there within 53 feet, now you have to call someone on the other end to get them to look at it. It slows down your engineering . But overall, it’s a good, proven model. We all do it.”

Disaster Recovery

Engineers are paid to think the worst. Are there any key differences between approaches to disaster recovery in the different workflows?

Dan Turk. NEP Americas: “It’s ingrained in our fabric for how we do everything from an engineering perspective to ask, ‘what happens if something doesn’t work correctly’? With the onsite stuff, there’s a plan for everything.”

“When you go to the REMI model it involves a higher risk. You’re relying on connectivity and different countries have different levels of quality when it comes to connectivity. The U.S. is more disparate as there are a lot of different companies you have to cross. It’s more difficult to piece together. There’s a lot of breaks and you need that managed service to fix it when it does break. We have a good amount of connectivity in the U.S., but the amount of cases where there have been challenges is alarming . But that’s why there are ways around it. For example, routing and static dark fibers can work around it. We’ve been doing it for years now. When it’s all onsite you control the whole thing. With remote engineering it can be challenging and we have found that we have to add a few weeks of time to the testing and fail overs portion of the planning, because it takes longer to troubleshoot if it’s not okay.”

John Guntenaar. NEP Europe: “There are a number of ways to work around failure. In an OB, disaster recovery is built in with what we call the Red Rail, which probably has a different meaning in every country, but if the switcher fails on air you can bypass some of the things in the truck to a 1U unit, so you can still perform hard cuts. You won’t have the ability to use transitions, but you’re still able to get cameras on the air. What is most likely to happen in this case is to put camera one to air, and fix the problem. In the data center model we would have something similar. There is, of course, spare capacity, but, moving over to a completely separate vision switcher is not really feasible. In terms of disaster recovery, you have more tools in your tool chest to be able to work around the issues in the data center than you have in the truck. It’s not that the model is completely different.”

Connectivity

One simple thread seems to run through this discussion… connectivity is key… from commercial, operational and reliability perspectives.

John Guntenaar. NEP Europe: “The technology is mature enough to do remote production if we have connectivity in the country, but the connectivity is not always reliable in all countries. Sometimes the price of connectivity doesn’t make sense when weighed against the benefits. It differs from country to country. In the Nordics, it becomes beneficial quite easily, whereas in Belgium and the Netherlands, this isn’t always the case because the travel distances are very low. Throughout Europe the connectivity is relatively affordable and it’s super reliable compared to many other countries. In the U.S., because of the distances, it’s not super reliable because we have a lot of outages and fiber cuts so you must have extra measures in place to guard yourself against those issues. You can have stream redundancy with something like 2022-7 but if you have 3 or 4 fiber cuts a day, then even with dash seven you don’t really feel protected so you want extra measures in place for that.”

In the next article we take a moment to look at the creative workflow of live sports production and how much has or perhaps hasn’t changed.