Tale of Two Cities - The Evolution of Distributed Media Workflows
Ingest station at Televisa in use for Olympics coverage
Increased global connectivity is enabling broadcasters operating across geographically dispersed locations to reevaluate the way they do business. In this article we take a look at examples of distributed media workflows, where advanced integration connects separate broadcast sites to form a mesh network of operational locations.
The BBC is arguably a pioneer and trendsetter of advanced broadcast technology, and for our first example we travel to the BBC’s operation in the Channel Islands (a collection of UK Crown Dependencies) where BBC Jersey and BBC Guernsey are sharing and exchanging media assets seamlessly using a Sienna Distributed Media Cloud across the two island sites some 40km apart.
In the second example we travel to Mexico City and Televisa - the world’s largest Spanish language broadcaster - where we investigate the latest chapter in the biggest Sienna system in the world, and see how no-compromise fault tolerance and mirroring is achieved with a completely replicated infrastructure pair - and no single point of failure.
The two operations are quite different in scale, but each organisation is embracing advanced media asset management techniques and allowing autonomous islands of media to come together into a Distributed Media Cloud (DMC). In a DMC storage and control is not centralised, but is instead - distributed. This technique provides all of the benefits of a centralised system, but does not have any single point of failure - and there is no requirement to move all media across wide area networks (only media that is required at another site is moved, on demand). In a DMC, each node of the distributed media cloud is entirely autonomous, independent and resilient
BBC News
BBC Jersey became the second site in the world to adopt Sienna infrastructure in 2007. The system has continued to expand over its nine-year evolution, and with incremental software and hardware updates it remains state of the art. In the most recent round of updates, a decision was made to create a close integration between BBC Jersey and its neighbour BBC Guernsey, aimed at sharing media assets and speeding up remote workflow between the two islands. To achieve this, an additional Sienna system has been installed in Guernsey and the two sites have been linked into a Sienna Distributed Media Cloud (DMC).
The two sites each have their own Sienna Media Asset Management (MAM) system, local storage, and local edit and ingest clients. Jersey also has MOS-driven playout facilities. The integration between the sites links the two MAM systems in a Sienna DMC where the MAM at each site is aware of its peers, and users on either island can search the entire enterprise in a single federated search operation.
Results from all nodes in the DMC are presented in a single results page, allowing a user to instantly review and annotate a proxy of any asset from any location in the DMC in a single click. Furthermore, the system allows a user to select any asset from any DMC node, and send it directly to their local NLE video edit system. Sienna intelligently handles the demand for a remote asset by automatically localising it as a background operation then automatically adding it to the NLE project. A single mouse click is all that is required to get any asset from any node into any editor located at any site within a DMC.
MAM Operations
When media is localised for the requesting user, a new asset is created in the local site’s MAM - and that asset remains conjoined to the asset in the originating MAM. Changes to any instance of the conjoined asset at any location in the DMC (such as metadata updates) are reflected to all representations of the asset at all nodes in the DMC.
High speed media transfers between DMC nodes are achieved using Sienna MediaVortex MediaWAN accelerators. MediaVortex is a fault tolerant file transfer accelerator optimised for media workflows and growing files - delivering high speed media movement between the island sites.
At the end of the editorial process, users are able to deliver a completed package to one a of small number of pre-defined targets. Targets are configured to allow content to be delivered locally, or to additionally trigger automatic replication to the partner site. With this capability an editor in Guernsey can publish their finished story directly to the MAM in Jersey where it will be played to air.
The overall effect of the integration is to blur the geographic lines of sharing media, whilst maintaining autonomy at each site. Users are no longer concerned with manually transferring media before it can be edited, or by complex multi-stage manual delivery workflows for finished packages. The ability to search the entire enterprise also offers editors a greater palette of content and avoids unnecessary assignments (for example, to gather B roll footage of a location where that content may already exist at the other site).
With no single point of failure and no central orchestration overhead, the DMC scales smoothly to any number of nodes. Network connectivity between nodes can be designed according to requirements - as either a star or mesh - and even tunnelled over the public Internet. MediaVortex MediaWAN accelerators enable efficient and intelligent use of available bandwidth.
Ingest racks at Televisa
Televisa
First, a bit of history - Televisa Deportes originally selected Sienna for their coverage of the Beijing Olympics in 2008. This involved a massive shipment of technology to China, where Televisa’s Sienna was the second largest visiting media system at the games (second only to NBC Olympics huge setup). After the games, the entire system was sent to Mexico City where it has been used to operate the Deportes Sports channel ever since.
For the 2012 London Olympics, Televisa deployed an innovative remote workflow - and reaped significant operational and cost benefits. An additional and smaller Sienna system was purchased and installed in London at the games. The Sienna system in London was integrated with the Sienna system in Mexico using a Distributed Media Cloud (DMC) - allowing operators to control multiple ingest channels and consume and edit media that was being created thousands of miles away - over a 100mbit/s connection..
The key to the success of the Sienna DMC for Televisa at the games was the use of MediaVortex MediaWAN acceleration. MediaVortex is a content-aware intelligent WAN accelerator, and unlike other WAN acceleration tools, it understands media files and growing files created by live ingest. MediaVortex intelligently provides real time transfer of growing files across high latency WAN connections.
In addition, the workflow relied on the unique feature of the Sienna Distributed Media Cloud - ‘conjoined assets’ - where an asset exists across multiple locations in a clearly defined sibling relationship. The Media Asset Manager (MAM) at each side can include the high resolution, and/or proxy media representations, along with a shared and synchronised set of metadata and real time markers. Updates and changes to conjoined assets located anywhere in the DMC are propagated to all nodes.
At the games, this allowed Televisa to ingest 4000 hours of high definition video in London, and in real time propagate proxies of the in-progress recordings across the Atlantic. In Mexico, editors were able to sit at their own desks, and log and edit the media in real time, as was captured.
Editors working with proxy material in Mexico were able to complete just-in-time edits of material - and have the high resolution version conformed in London for very quick turn-around production in the fast moving games. The ingest operator in Mexico was even able to remotely control an entire rack of 24 HD Ingest channels in London.
Televisa linked London and Mexico City for London games
Save Money, Improve Flexibility
The objective of the remote workflow was to save money and improve flexibility and productivity - by reducing both the number of staff required to be located in London, and the amount of equipment needed on site for the games. Televisa were able to halve the on site head-count, and yet still delivered fantastic coverage of the Olympics, soundly beating their competition in ratings.
Televisa have now begun a fourth generation expansion and upgrade of their Sienna system. In this new expansion they are creating a totally mirrored, fault tolerant media infrastructure. All storage, ingest, MAM and playout systems are replicated in real time between a pair of tightly integrated (but independent) zones - creating the ultimate in hot failover, disaster recovery and business continuity. This delivers a 100% Recovery Point Objective (RPO) and a zero Recovery Time Objective (RTO), the ultimate targets of failover.
Sports news channel TDN is also connecting to the Sienna system, (operated by the Deportes sports channel) and TDN will have the same fault tolerant mirroring, right down to a fully synchronised hot-failover of their MOS news rundown, cued up and ready in the parallel automation system, controlling the independent replicated media.
Like the BBC, Televisa’s twin Sienna MAMs are integrated in a distributed media cloud, but in this case the task is different and the objective is to maintain a hot-live failover infrastructure. All assets (including baseband ingests currently in record) are live in the failover zone and ready to use in the event of any failure of the primary zone - including power, hardware, storage, network or software.
Redundant Operations
The task begins with replication of baseband ingests. Televisa runs a very large group of more than 60 Sienna PictureReady baseband-video ingest channels, and these are divided across the zones. Each zone’s channels are controlled by a number of Sienna IngestControl central management screens, and these IngestControl systems communicate across the zones to automate parallel operations.
Whenever a primary zone channel starts a record, a slaved secondary zone channel performs the same action - thereby providing a completely redundant operation, on independent hardware, software and storage.
Each ingest is checked into its respective MAM, but additionally the systems cross-reference their ingests, and *conjoin* the two assets so each one knows about its sibling. This conjoined asset approach has a number of applications, but the simplest is replication of metadata - when an operator logs the ingest in the primary zone, metadata changes such as real time markers are handed across the distributed media cloud and are also applied to the sibling asset, keeping both media and metadata in sync.
Also like the BBC, any packages or file based ingests published to the primary MAM are automatically replicated through a pair of Sienna MediaVortex MediaWAN accelerators, and are conjoined in the failover zone MAM.
A second application of the conjoined asset mechanism is used for TDN’s MOS based newsroom rundown. In the primary zone, Sienna AutomationX live-assist automation production control system drives a bank of Sienna VirtualVTR playout ports, playing media directly from the primary zone storage system. In parallel, and entirely automatically, the failover zone’s AutomationX is doing the same thing, but this time driving VirtualVTR ports playing the conjoined backup media assets from the failover zone storage.
In the event of any major failure in the primary zone failure, Televisa can quickly switch to the failover zone, and all assets, including still ingesting content is available, and TDN’s news bulletin playlist is already cued up in the backup AutomationX, playing the backup media.
Televisa’s vision and faith in the long-term development of Sienna have taken them from a state of the art tapeless workflow in Beijing 2008 through groundbreaking remote sports operations in London 2012 and onto a unique fully redundant distributed media workflow in 2016.
Mark Gilbert is CTO at Gallery SIENNA
You might also like...
Designing IP Broadcast Systems - The Book
Designing IP Broadcast Systems is another massive body of research driven work - with over 27,000 words in 18 articles, in a free 84 page eBook. It provides extensive insight into the technology and engineering methodology required to create practical IP based broadcast…
Demands On Production With HDR & WCG
The adoption of HDR requires adjustments in workflow that place different requirements on both people and technology, especially when multiple formats are required simultaneously.
If It Ain’t Broke Still Fix It: Part 2 - Security
The old broadcasting adage: ‘if it ain’t broke don’t fix it’ is no longer relevant and potentially highly dangerous, especially when we consider the security implications of not updating software and operating systems.
Standards: Part 21 - The MPEG, AES & Other Containers
Here we discuss how raw essence data needs to be serialized so it can be stored in media container files. We also describe the various media container file formats and their evolution.
NDI For Broadcast: Part 3 – Bridging The Gap
This third and for now, final part of our mini-series exploring NDI and its place in broadcast infrastructure moves on to a trio of tools released with NDI 5.0 which are all aimed at facilitating remote and collaborative workflows; NDI Audio,…