The IP change is a “YUGE” problem

While the industry argues over which standard or standards will rule the world (for at least 15 minutes), the people that need to make systems work on a daily basis, the engineering and tech support folks, are being ignored. And the more standards, protocols, API’s, middleware and “open” everything means such systems can be a nightmare to maintain for even the best trained personnel.

Let’s fantasize for a minute that we can actually build a working IP production system. Now what?

As a designer and project executive, I always say the building part is easy, it’s running these systems that is hard. I’m not diminishing the challenges, skills and knowledge that constructing a media infrastructure requires. However, when building a system, there is the luxury of being able to solve problems without worrying that those systems are live.

But once a system gets turned on and goes live or online, there is no turning it off. The integrators have completed their punch lists and moved on. Now its operations and engineering staff’s responsibility to assure everything works and stays that way.

Every time there is a major technology change, in comes the glue and black boxes that enable the transition to occur seamlessly. As the transition matures, the need for these little helpers diminishes because devices and systems begin to connect directly with each other.

The new MCR

The transmission or master control center is responsible for maintaining quality control over all inbound and outbound signals. This room is equipped with the tools to confirm the signals meet certain technical requirements. In the equipment rooms, there are dedicated stations that enable engineers to monitor the systems in a non-disruptive way. Devices and systems can be patched around, or if there are spares, taken to maintenance and/or to the engineering shop. The shop has the required tools and test equipment to get things working properly again.

Monitor, maintain and manage IP

However, the introduction of IP-centric software and hardware technology has blown up this entire model.

Just how does a staff monitor, maintain and manage these IP systems? Great question. Let’s examine how these tasks are performed in an IP environment.

First, monitoring needs to be done in a non-invasive and non-disruptive way. For the purposes of this article we are not going to discuss the type of the knowledge and skill sets needed. We will assume that properly trained video and IT staffs are available and work well together.

The program origination center, aka Master Control, now looks more like a data center, often called a Network Operations Center, or NOC. The operators here monitor multi-view displays, which are managed by an application. The displays show all program feeds and the playlist for each delivery platform (applications), which are controlled by automation (another application). All of these applications run on servers. These servers are connected via a network and they pull both content and playlists from shared storage.

Content quality control is handled by an application doing file or stream checking. Content contribution arrives as files over a dedicated network or open internet using cloud services and/or proprietary software. As the content become streams, the network needs additional types of monitoring and management.

New monitoring solutions required

IP and file-based technology has introduced a new level of management to broadcast and production systems. Devices are monitored using SNMP (Simple Network Management Protocol) traps. The servers are monitored with tools that display CPU utilization, running processes, memory and storage, and this happens before we even get to the application.

Many devices and systems have their own management tools. These might be accessible using a web browser. Does this mean a dedicated computer, or is this a good application for virtual machines (VM)? If content is playing from a server how is it monitored? From where are the servers, storage and network going to be monitored and managed? While KVM (Keyboard, Video Mouse) extenders or matrix might be a first thought, using KVM to remotely manage is highly disruptive of operating services. This is where sharing applications like VNC (Virtual Network Computing) are appropriate solutions.

IP system maintenance can be a Pandora’s Box

What about network bandwidth or congestion? This needs to be monitored. What about managing the storage itself? There are applications that manage storage retention policies. But, what happens when utilization peaks and it’s time to load balance? Or, consider a worst case scenario where the storage system fills to capacity.

Maintenance for IP systems opens a Pandora’s Box! First, both hardware and software come with Service Level Agreements (SLAs) that are typically a percentage or the purchase price. Before we even get to the dedicated application, regular updates must be applied to the operating system. Then there is threat protection (Anti-Virus, intrusion protection, security), which must be kept current. Also, what about firmware updates to firewalls and switches?

Checking for updates, determining whether they will create a problem or interfere with a running application are all part of maintenance. What happens when vendor A upgrades their software? Will those changes effect the middleware and API’s that are used by external systems? What if a new codec or IP standard comes out? Is the update considered a patch or does it require some custom code and who does the install?

Remote access

The shop is now a NOC and if there’s a problem, accessing the media systems can be done remotely. IP introduces remote access as a way to monitor, manage and maintain the systems. Monitoring systems can be configured to detect many errors. They can also send alerts and messages over email and text (SMS). Remote access is the ability to use either open internet or dedication access to connect into the network or troubled system. Once connected, the engineer has the ability to diagnose and correct the issue.

As I stated earlier, this access needs to be non-invasive and non-disruptive. Taking control away from an operator or automation system and then not having the ability to restore control is not a healthy way to solve a problem. It only creates more issues.

This is the first in a series that will look at the tools, methods and technologies to monitor, maintain and manage an IP facility. 

Some additional The Broadcast Bridge resources written by Gary Olson are listed below.