DHCP Can Be IP Video Kryptonite

If you can see how a magic act is performed, POOF! the magic disappears. Live TV is a magic act because so much of the on-screen magic happens behind the curtain. When a random device malfunctions or fails during a live show, if the talent doesn’t blab the secret on-air, viewers probably won’t notice. As many great engineers and directors have said in so many ways, “They won’t know unless you tell ‘em.”

Creating exciting, expensive-looking TV that gets good ratings on a tiny budget requires some magic. The trick is to create content that appears to viewers to be a major sports network TV production without looking cheap. Radio station KRMS in Osage Beach MO does just that. Station co-owners Ken Kenzie and Dennis Klautzer are also broadcast engineers who dreamed of making world-class powerboat races at The Lake of the Ozarks and other venues into the live TV powerboat racing broadcast events they have become today. 

TV coverage started on the local public access cable channel as a radio broadcast with a fixed security camera looking out a dirty window. A bit over a decade later, it's grown to a 7 field-camera international sports TV event, produced for radio with plenty of wild sound, music beds and non-stop announcer chatter. The TV crew produces the graphics. The racing wild sound and the video is the star. The magic is how it is done. The quality and scope of the TV broadcasts would be unfeasible without IP video.

Recent 3-hour daily broadcast windows for the 2-day “Lake ShootOut” broadcast were uplinked unencrypted on Galaxy 17 with live captioning, and carried live on several local TV stations across two states and on cable systems over 6 states. The entire event was also continuously streamed live on the internet for 8 hours each day. MVPDs and stations that didn’t have a spare dish relied on the public stream, as did over 30,000 world-wide internet viewers. Typically the races get #1 or #2 ratings in their time slots on broadcast TV. We once beat the ratings of a local NFL preseason game airing at the same time in the same market.

I’ve been involved in live powerboat racing TV production for a decade. Our production dreams are big and funding is tight, not unlike most local TV stations in triple-digit DMAs. In fact, the powerboat racing productions are financed and produced by a couple of radio stations in a market so small it has no DMA or TMA designation. Over the years, inspiration and technology has made the dream of creating world-class TV content with a small-town radio station budget practical. The radio station connection is crucial because the radio station sells the ads that finance the TV broadcast. At the core, the productions are a radio sports broadcast with awesome natural sound and exciting live HD video of 200 MPH boats at full throttle. The program is simultaneously broadcast live on local AM, FM and TV, and the only source of live stats and info available to venue spectators.

The temporary ShootOut studio was the hub of IP-based production. Every source but the studio chroma key camera was IP.

The temporary ShootOut studio was the hub of IP-based production. Every source but the studio chroma key camera was IP.

The IP Solution

The first powerboat racing production challenge to overcome is the fact that you can’t run wires across a body of water or down a public beach. A well-financed, high-end, 53' OB sports truck live TV production would link the 7 live remote cameras the production uses with expensive one-way BAS microwave systems. That approach is well beyond our budget. We found that linking field production cameras to the studio via IP using 5.2 GHz Ubiquiti WiFi microwave systems on U-NII channels is reliable, transparent, and ridiculously less expensive than BAS. The bonus is that Wi-Fi links provide a full two-way connection because they're IP.

Not including the Matrox IP encoders and decoders, the two-way Ubiquiti links were less expensive than one-way BAS microwaves by a factor of 10-2, making on-hand spares a cheap investment. Having a broad selection of inexpensive electronics and antenna options on-hand has proven handy. The Wi-Fi links are dependable, but susceptible to all the usual 5 GHz RF issues such as the inability to penetrate trees or metal, and sensitive to RF shadows and multipath at a distance. Sometimes, a different antenna can resolve a SNR issue.

Thus far, other than deploying a back-up 900 MHz Wi-Fi system to burn through a tree in the middle of a one-mile Wi-Fi path, the production has never needed to replace a broken Ubiquiti with a spare because none have failed.

Typically, the widest-bandwidth Wi-Fi links use high-gain dish antennas to maximize SNR, but that’s not always an option. If a camera is on the move it must use an omnidirectional antenna. One example is an IP PTZ camera used on the anchored start boat where the starter waves a green flag to boats crossing the start line. Winds and currents can change the relative angle of the boat and antenna. An omni antenna on the boat was the solution.

A mobile hand truck provided reliable power and a Wi-Fi link for a roving reporter working in tight areas and crowds.

A mobile hand truck provided reliable power and a Wi-Fi link for a roving reporter working in tight areas and crowds.

Hand Truck TV

Another challenge was to design a live roving camera unit to provide AC power for the gear needed to feed content from around the event’s land venue, and it needed to be power self-sufficient all day. In addition to being used for the National Anthem, the roving camera is called the “DockCam” for live interviews with racers in their boat docks, and the “CutieCam” for babies and dog shots. It’s also used for ground-level beauty shots of vendor tents, pedestrian traffic, boats, swimmers and whatever attracts people’s attention. Nearly each on-air use of the roving camera unit is uniquely sponsored with an on-screen logo graphic. The logo sponsorships made the money available to build the hand truck solution using existing Matrox H.264 encoders and decoders.

The radio station owners like to invent solutions when the opportunity or need arises. Co-owner Ken Kenzie built a 2-wheel hand truck mobile unit to power and move the roving camera, encoder and Wi-Fi system around the venue. It proved to be a reliable continuous camera source for the 8 hours each day the 2-day show was on. There were plenty of interesting and colorful crowd and event shots only the roving camera could capture.

A weatherproof box on the hand truck contains power distribution and charging, and it protects electronic components not designed for outdoor use.

A weatherproof box on the hand truck contains power distribution and charging, and it protects electronic components not designed for outdoor use.

Secured to the ledge of the hand truck is a 12V car battery inside a heavy-duty plastic marine battery box. It connects to a weatherproof plastic box containing a 120 VAC inverter to power the encoder, the Ubiquiti Wi-Fi, and to the external HDMI-connected camera to keep it charged. The weatherproof plastic box also houses a 12V car battery charger.

The outdoor Wi-Fi unit and omni antenna are mounted on the hand control bar, above the height of most people. The receiving antenna for the production’s Wi-Fi systems is mounted on a 60’ Genie Lift with a boost from a 10’ section of Rohn tower to clear the tops of trees that grow every year. That antenna height above the venue made reception of the roving unit below remarkably stable regardless of people moving around and past it.

Kryptonite Management

The production relies almost entirely on remote cameras that can’t be wired-in directly, and local internet and cell phone service deteriorates significantly when the event takes place. This was first learned when the production switched to HD and brought in bonded cellular units for camera transport. Bonded cellular worked great until about 100,000 spectators showed up and service slowed to about Morse code speed..

Cell service at the recent racing production was typical. It all worked fine until the event began. Then it became nearly impossible to make or receive a phone call. Text messaging literally took minutes. Complaining spectators thought there was something wrong with their phones. At a previous race, cell service disappeared almost entirely for about an hour after a high-speed boat flipped on the course and spectators tried to simultaneously upload their videos of it.

Also, directing a show where the field cameras are about 5 seconds ahead of their video arriving in the studio is pure torture. Director: “Pan left!” Camera operator: “I already did.” There had to be something better.

Plan B was better. It was based on Matrox Maevex H.264 encoders and decoders for IP video transport via Wi-Fi, and the production began using IP transport in 2015. Wi-Fi IP video has proven to be nearly as reliable as copper wire, with delay measured in frames instead of seconds. But, the production has also learned of something simple that can wreck that feeling of IP stability in about a microsecond.

The IP production uses a private network, not in the usual 192.168.x.x range. All the video gear on the private network uses static IP addresses to avoid duplicate address conflicts. What wreaks havoc on the network is when someone or something logs on the network with dynamic host configuration protocol (DHCP) enabled. A device using DCHP will find an unused IP address within the preset range, claim it and log on. If that DHCP address happens to also be the static address of a production device that may not be turned on or plugged in at the moment, standby for some bizarre network behaviors when the static device tries to get on-line. It can cause problems across an IP video network that aren’t necessarily limited to the conflicting devices.

A rogue DHCP address on a private network with numerous static addresses can be live production Kryptonite. All it takes to upset the system is one unsuspecting person with a phone, tablet or laptop that has logged on the network before to walk in at the right time. Most phone and computer users have no idea how to disable their DHCP because they usually don't need to. Net address conflicts can be a huge troubleshooting whirlpool, particularly as DHCP-enabled mobile devices come and go during setup and production. Trust me, I've been there and done that just last week, and still have a mild headache from it.

The obvious solution would be to turn off DHCP on everyone's phone or have a unique Wi-Fi password for each show. The optimum DHCP defense is to not allow anyone to log anything into the network without an approved static IP address.  Either way, everyone will complain. The easiest solution that will draw the fewest complaints is to isolate separate spaces on the private IP network for gear and guests.

Our production environment on race days has a Gb internet fiber connection, virtually no cell phone service, and everyone involved around the studio expects Wi-Fi internet access. For the safety of the production, the network is set to reserve a specific address range for known static IP video production gear connected by Cat 5 or Wi-Fi. It also reserves a different address range for DHCP Wi-Fi logins that know the network and password but are not part of the known IP video system. That way, anyone in the studio wanting internet service on a personal device through the Wi-Fi network can do so with a DHCP address guaranteed not to conflict with the static production gear.

You might also like...

Designing IP Broadcast Systems - The Book

Designing IP Broadcast Systems is another massive body of research driven work - with over 27,000 words in 18 articles, in a free 84 page eBook. It provides extensive insight into the technology and engineering methodology required to create practical IP based broadcast…

Demands On Production With HDR & WCG

The adoption of HDR requires adjustments in workflow that place different requirements on both people and technology, especially when multiple formats are required simultaneously.

NDI For Broadcast: Part 3 – Bridging The Gap

This third and for now, final part of our mini-series exploring NDI and its place in broadcast infrastructure moves on to a trio of tools released with NDI 5.0 which are all aimed at facilitating remote and collaborative workflows; NDI Audio,…

Designing An LED Wall Display For Virtual Production - Part 2

We conclude our discussion of how the LED wall is far more than just a backdrop for the actors on a virtual production stage - it must be calibrated to work in harmony with camera, tracking and lighting systems in…

Microphones: Part 2 - Design Principles

Successful microphones have been built working on a number of different principles. Those ideas will be looked at here.