Live broadcast television production relies on people, technology and gear to all behave and communicate, from setup to teardown. A broadcast engineer’s live production duty is to make the signal look and sound flawlessly professional, beginning with pre-production all the way to sign-off without excuses. Given enough stability and experience, broadcast production and engineering can almost become a cookie-cutter routine. Throw in a couple of known unknowns and it's an adventure. Discover an unknown unknown during a live production and studio conditions can instantly escalate to DEFCON 2. Engineers strive to avoid this, but it happens.

That's exactly what happened on the first weekend in June at Lake Race 2015, a 2-day off-shore powerboat race and live television production from Missouri’s Lake of the Ozarks. I’m the EIC. This year’s Lake Race production became its own race-to-the-finish debut of several new technologies. With lots of help from friends and luck, we and our viewers won.

Offshore powerboat racers like the fresh water because it isn’t corrosive like salt water. They also like the estimated 50,000 spectators and the relatively calm waters. This year's event was broadcast live, Noon to 3 p.m. Saturday and Sunday on CW affiliates in Springfield and Columbia, MO. Some spectators report they came to watch the event because of what they saw on TV.

Viper Broadcasting has been producing the radio, cable, and later broadcast TV coverage for two annual world-class powerboat races at Lake of the Ozarks for many years. The first TV coverage was a NTSC security camera pointed out a window with radio station audio on the local cable access channel.

Television coverage grew to renting multi-camera TV production systems the week of the event for live weekend broadcasts on area TV stations. This year, Viper opted to buy its own new production gear, except for cameras and sticks. Ownership was when things started getting interesting, in a curious broadcast engineer sort of way.

NewTek aces pop quiz

The central element of the purchase was a NewTek TriCaster Mini HD4i with just-released Advanced Edition Software and a control surface. It was delivered on a Monday, two full work-weeks ahead of the race broadcast and appeared to work out of the box. But, a couple of unexplained things began to recur on day two and three. It would randomly drop any or all active HDMI inputs. It could be fixed by rebooting or simply replugging any HDMI input cable to the TriCaster. I gave all the input connections a strenuous workout but could not replicate the issue.

The system also disagreed with a sponsor logo from an imported Photoshop .PSD file. Live Text liked it. The TriCaster switcher did not. The logo and various new file versions of it all turned to video hash for no apparent reason. Dealer Wink Friesen at Wink AV did his best and knew when to send us on to NewTek tech support. The remote possibility of “one in very few” didn't make the pre-production timetable. Karma and time were not smiling on us.

When the drama gets thick, the broadcast engineer’s skin must be thicker. By Friday, after reloading everything without success and hours of research, trials, discussions and lingering questions, NewTek support tech Richard Evans looked inside the unit via Internet and declared a hardware failure. A RMA number was assigned and a new Mini over-nighted. The replacement was delivered 4 days before the show went to air. Evans was an accessible, reliable trooper and just the kind of support we needed at drama central.

Many Lake Race graphic elements and settings were already prepared. The unique files were copied to an external drive and reloaded in the replacement TriCaster at blazing USB 3.0 speed. It was an excellent new beginning that maintained altitude throughout the flight.

Not your father’s Toaster

To say the TriCaster Mini HD4i is an incredible device would be this year’s understatement. It’s a virtual Las Vegas casino buffet with a lunch-box handle and switchable screen, cascading with delicious switching, graphics, effects and branding goodies for all tastes and appetites. Watching the demo at NAB is one thing. Learning and using one for the first time to produce a high-action, multi-camera live broadcast production with huge expectations from sponsors, stations, owners and viewers was a broadcast engineer's dream.

There are many ways to learn all kinds of things about TriCasters on the Internet and little time. Advanced Edition was too new for much documentation. After some important initial hours of remote set-up and training from our dealer and watching a couple of task-specific training videos, I was ready to practice, rehearse and learn. The intuitive system is filled with alternatives and control of just about everything.

A virtual set wasn't necessary because the temporary studio at this race venue has a great view. The August race venue demands a virtual set. We can’t wait.

We didn’t push the Mini’s software boundaries, some of which appear to extend beyond the galaxy. We did turbocharge its four-HDMI input hardware limitation with a new Ensemble Designs Bright Eye NXT 410 HDMI router. The router not only lived up to expectations, it also converted two outside HD-SDI inputs to its HDMI output going to a single TriCaster input. It brought to the number of active video sources routed to one HDMI input to five for a grand total of eight TriCaster live sources plus graphics and internal DDRs. The NXT 410 switched between its various sources seamlessly on the air.

A separate TriCaster operator controlled the DDR replays, the Live Text laptop connection and the streaming video to Ustream. The two DDRs recorded all four inputs. We used it for replays. It wasn't a 3Play 4800, but close enough to give an experienced dedicated operator basic control. Similarly, the Live Text operator independently created and updated info off-line and inserted the graphics on-line through on an always-on downstream key.

Radio roots

Mixing as a radio remote allowed for something of a mix-minus scenario. An external manual audio mixer combined studio talent mics, a wild sound outdoor mic and playback from an independent computer for spots and GoPro clips, just like a radio mix. Output was fed to the line-level TriCaster Input 1 set to always-on. The TriCaster automatically added the highly attenuated audio-follow engine roar from field cameras as racers appeared on the screen. An audio op with ESP couldn't have kept up.

The full sound mix was in the TriCaster’s HDMI output. For backhaul to the TV stations, one TV affiliate provided a LiveU, the other a Dejero. Both units were fed through one HDMI DA. Both fed individual cable modems providing maximum Internet bandwidth for transport back to servers in their studios.

A 10 second delay was dialed in to the Dejero and Live U for accurate live-cuing and plenty of video buffer time. Video quality was outstanding. Return TV sound was the audio source for the Viper radio stations.

Good intentions gone bad

Viper purchased and loaded a copy of NewTek’s Live Text program on a laptop belonging to the Viper CFO for live graphics standings and driver info updates. The laptop and TriCaster worked together until the day before the broadcast. That day, they refused to shake hands. And refused. And refused. It took a long couple of hours to learn that the night before, the laptop owner’s spouse loaded a new anti-virus program on the laptop that included a new firewall. That was news we could have used two hours earlier!

The project was starting to look like one of those productions where the light at the end of the tunnel is an oncoming train. Bad hardware, a firewall surprise and a rain forest-like week leading to an outdoor event was just the beginning. We still needed on-location outdoor camera testing because that system's transmitters, receivers, encoders and decoders were all new and untested. More about that in Part 2.

Too early for this stuff

Friday afternoon before the race, we checked out and verified everything, agreed on approval and went to dinner. Saturday morning we were greeted by an unresponsive microwave camera receive site. Arriving crew members speculated someone had stolen our gear overnight. Would there be a ransom? I was off to the scene of the crime before my first coffee. An event volunteer had unplugged an old extension cord going to our receive site hidden above, to power her portable radio. All problems should be so easy. They weren’t.

What’s the very last thing anyone at a remote production wants to see on TV? Yep, about one minute into the live show, our cable TV return video feed became the dreaded Technical Difficulty graphic. The radio station was dead air. Cell phones started vibrating. We could hear audio in local production headsets, which were fed from the headphones jack out of the radio mixer. What?

We thought we had the audio all figured out and tests with master controls in both TV stations an hour before airtime were okay. When the show hit at Saturday Noon, both stations suddenly had no audio on their feeds. We were hearing audio in our headphones and thought everything was okay. Station operators checked their systems, found nothing wrong, punched up Technical Difficulty slides and called us. Murphy's law had stopped in for a visit.

Who? Me?

At this point, I’m just the video guy in a room full of radio people in full-panic mode. Amidst the chaos and denial, I happened to notice that hanging off the back of the TriCaster was an audio adaptor plug plugged into another adaptor plugged into another adaptor plugged into the cable from the radio mixer. It was the sole source of audio for the opening intro clip, music and studio mics. I didn’t know where those adaptors had been but they looked used and dirty. I carefully twisted the hodgepodge of Radio Shack's finest a bit and guess what.

A few minutes seemed like an hour. The adaptor plugs were cleaned, plugged back together and we had audio. It was live TV field production at its trench-level, gut-wrenching grittiest. Luck magically held the audio together electrically for the rest of the production. It wasn’t the best start nor the end of the learning experience adventure. Once we got past the opening Technical Difficulty meltdown, things really went... 

I'll cover the success and challenges in Part 2. Day one was just the warm-up. It was one of those memorable productions where you find out who your friends are, and who isn't.

In Part 2 we'll look at a plug-and-play fiber system, an amazing Wi-Fi Video over IP system, a pair of DJI drones lost and found and a stinging "Who sets the boundaries?" gotcha. Like most live TV production learning experiences, by the end of the show the broadcast engineer will either be the hero or the villain.