Instead, IP-enabled production offers the possibility of reduced movement of people and equipment, increased utilization of resources and more efficient deployment of the production team – bringing into play the tantalizing prospect of producing more content with fewer resources.

When any broadcaster or production facility makes a leap into future technology it is essential that they maintain a good understanding of how workflows and networks are functioning. To this extent, monitoring across the network is critical along with precise timing.

Historically, instruments such as spectrum analyzers and waveform monitors have lived inside media production and distribution facilities while network diagnostics and troubleshooting tools have lived in IT data centers and elsewhere. As broadcasters and content creators move from legacy SDI to hybrid IP, and more recently to SMPTE ST 2110 mezzanine or un-compressed video, audio and data, new and expanded audio-video quality assurance and monitoring tools are required. These guarantee that operational staff can take advantage of current and emerging options to enhance live production and make more efficient use of video and audio assets.

Maintaining IP-based systems at peak efficiency and reliability relies upon two fundamental activities: monitoring system health and then resolving any problems that have been found. Traditionally, these tasks have been assigned to different groups with overlapping responsibilities, including operations, maintenance and engineering.

With the growth of IP systems and hybrid SDI-over-IP configurations, these activities become tightly interwoven, requiring a new generation of tools that can cover the full range of technologies and systems. Furthermore, with the rapid growth of cloud-based media workflows, the tools required to monitor and troubleshoot these workflows must be able to handle the handoff points between private and public clouds/networks.

As the infrastructure backbone to produce a broadcast program migrates from traditional SDI to IP, many of the ways to produce that program do not change -- but the new technologies introduce the need to monitor and measure signals in different ways. We still move content from live events into production facilities and broadcast networks. We might also ingest material from program producers or commercial ad spots that come into our main broadcast network and are processed within the broadcast operation.

We can check the QC to ensure it meets required standards for audio and video levels, and we might edit that content and put together promo spots within our data center. And then we might finally broadcast that content from the Master Control Room to the viewer, either by streaming or transmission over broadcast networks.

At each of these phases we must ensure that levels are correct; video and audio are present; the metadata is there; and that we’ve got subtitles and closed captions within the stream. Traditionally, to achieve that we have monitored the presence of the signals to make sure they are correct, and that there are no pops or clicks on stereo or surround sound. Additionally we check that the video levels are in conformance with our video specifications, whether for SDR or HDR.

Within SMPTE ST 2110, the video, audio and data streams are carried as separate flows across the network. This allows for flexibility in manipulating the video, audio and data signals in a live production workflow. Precision Time Protocol (PTP, formerly named IEEE 1588) and the Real-time Transport Protocol (RTP) can be employed to provide the ability to synchronize these video, audio and data streams across the asynchronous IP network. Precision Time Protocol is the foundation for all of the timing and synchronization capabilities of an ST 2110 network.

In a live production scenario we’re still going to have to monitor at the sporting or other venue to ensure we have the correct audio and video levels – all those of those things as we shade the cameras and guarantee the quality of the content being sent to the broadcast network. Once content has arrived at master control we need to monitor again to ensure the levels are correct and that there are no problems with the video, audio and data sent from the venue.

Depending on how the content is moved within the facility, it could be ingested and stored to a server; sent to a QC operations area for evaluation; edited to create a program segment; or broadcast or streamed with the addition of logo bugs and captions. At each of these points it is important to ensure that the content is correct and there are no errors present – and if there is an error, we want to be alerted to the kind of problem that has occurred.

Within the main facility or Master Control Room, monitoring equipment can be directly attached to the IP media production/processing network. Having the ability to monitor all the elements at both the LAN and WAN hand-off points is essential.

In spine/leaf architectures, any spine location with access to the critical signal source and destination leaves can be used. However, to provide the best coverage, it may also make sense to monitor individual leaves within the network, so that problems caused by network glitches, overloads or configuration errors can be isolated from issues occurring at the signal sources.

Understanding what can go wrong is sometimes key to being able to isolate issues and quickly troubleshoot the problems. As mentioned, in SMPTE ST 2110 the video, audio and data streams are sent separately, so we need to make sure that the receiver subscribes to the correct streams with the right IP addresses. Synchronization of senders and receivers must be ensured, with PTP present throughout the network, and methods to monitor the PTP. In an IP media network, congestion can cause packet jitter and loss of packets, so the network must be reliably designed with methods to interpret the information for the various streams.

Typically, redundancy is deployed within the network architecture using SMPTE ST2022-7, which provides duplicate streams of the sender across two different networks. The receiver can then check on a packet-by-packet basis and continue to deliver a flawless stream, even if errors are appearing on both paths at varying points. Monitoring is essential here to isolate problem areas, and to ensure correct operation. These are areas an engineer may not have had to think about with legacy SDI-based operation. These new ways to monitor and ensure reliable transport will continue to evolve as content creators find new ways to gather and distribute content.

Always check the basics, such as the correct type of small form-factor pluggable (SFP) ports and transceivers; the correct type and connection of fiber cable; that the media node has valid IP and MAC addresses; and verification of the IP address, port value, payload type and TTL (Time to Live) value. These are important characteristics that identify your stream within the network; you need to ensure that each of these values is set correctly.

Once you have started to build out the network you will have multiple streams flowing across that network. You need to monitor the reliability of those streams at your various monitoring points. You could end up tracking a large number of streams. A good approach is to monitor by exception. This means not getting a whole catalog of errors, but looking at specific errors given by simple red/green indications.

Monitoring by exception allows you to follow events with red/green indicators that identify channel issues and make it easy to interpret specific problems with alerts on a channel-by-channel basis. Then you can dive a little deeper to troubleshoot any problems. The user information helps identify the type of problem that is present and pinpoints the root cause of the error.

Diagnostic systems permit more detailed investigation of issues that have been identified on IP-based media streams. One of the more useful tools is the PIT Histogram, which displays a graphic that shows Packet Interarrival Time. This can be handy for diagnosing problems caused by network buffer-induced jitter or by issues with senders that are not following the rules for traffic shaping. Audio, video and ancillary streams coming from a common source can also have their RTP timestamps checked to make sure that they are in sync.

Since many of today’s networks contain SDI-based, ST 2022-6 and ST 2110 equipment, any differences between the analog sync reference signals (including black burst and tri-level sync) and the PTP reference clock can cause skew between the different types of video signals. This can impact the ability of switchers and other devices that rely on accurately timed video signals.

Proper operation of a PTP network requires that the correct messages be sent at appropriate times. For example, sync messages need to be sent out regularly according to the selected PTP profile and there should not be any missing messages. Every device must be capable of receiving and sending PTP messages, and they must all be configured to be in the same PTP domain.

Download the white paper, IP Media Network Monitoring and Troubleshooting - A Combined Solution, that provides insight into maintaining IP-based systems at peak efficiency and reliability, delivering uninterrupted, high-quality video, audio, and caption/data streams to viewers.