Disaster Recovery: What Every Broadcaster Should Know
Disasters can strike at any time and in many forms, leaving facilities with severe damage and stripping broadcast data centers of power for days. Though this can have catastrophic results, many broadcasters — through no fault of their own — surprisingly don’t have plans about how to truly protect content/playout since most of their time (and money, mind you) is spent creating, managing and delivering that content. But in the wake of several hurricanes in the US this year, as well as the horrific earthquake in Mexico, media companies are in fact putting greater emphasis on securing content and making sure their station’s main and backup feeds aren’t compromised during a disaster.
Eddie Ferraro, Managing Director at Globecast Americas
Content security takes many forms, with cyber security architecture and planning being critical to the design for anyone involved in content management whether it be playout, contribution, linear distribution or OTT.
While cyber security related to IP networks is currently getting a lot of attention, protection of the traditional linear distribution channels is critical since these channels still represent the overwhelming majority of revenue generation for television programmers. Therefore, providers are expanding services for protection of traditional linear channels that they service, with many solutions being extremely scalable. At the most basic level, they involve a single server with a loop of “evergreen” content. This type of simple solution is meant to protect the channel from just being in black, but cannot match the program guide or provide for scheduled interstitial materials (advertising and promos). This simple solution can scale up into servers with increasingly refreshed content, which can be managed by either the provider (such as Globecast) or the customer. At the opposite end of the spectrum, a full mirror of the primary master control operation can be developed where the program and interstitial elements, playlists, metadata and other required components are identical to the primary playout and operate in a mirrored environment. In addition to full disaster recovery this also allows the customer to switch between the two sites as needed to ensure both are providing the same level of service. In many cases, such designs are also used to accommodate upgrades and maintenance work at one of the facilities.
The next step following the choice of playout disaster recovery is the transmission protection options. In some cases, the output of the disaster recovery facility can be sent via fiber or IP to the primary uplink platform or directly to distributors, but often a separate protected distribution path is part of the overall disaster recovery design. A general requirement is that a backup satellite platform be able to utilize the same compression and encryption technology as the primary platform (e.g. Digicipher or PowerVu). Ideally, a backup path on the same satellite is required, but often another satellite with equivalent penetration into the distribution market is acceptable. At the high end, and with multiple channels under protection, a separate stand-alone mux on a dedicated antenna operating in a standby manner will serve as back up to the primary uplink facility. This design comes with some costs, however, as assets are assigned to this type of service whether or not it’s actually in operation. With major programmers operating channels that generate significant advertising and subscriber revenue, however, these costs are a very justified insurance policy. Often, the disaster recovery feed will be fed by fiber directly to one or more distributors to provide for a separate feed to the primary satellite delivered service. This provides for a completely dual and diverse feed of the channel(s) as an added layer of security for key distributors.
As media companies move toward a cloud and virtualized playout environment, facility-based backups will take on a different form. This is why it’s important to find a technology partner who can provide primary and disaster recovery services in a virtualized environment, allowing for geographic diversity for management and monitoring of services. This allows for a much more flexible approach to geographic diversity and allows for the management of international and regional networks across different facilities and regions. While cloud and virtualized playout and distribution for media allows for lower capex and greater flexibility, care must be taken to ensure that the network architecture takes into account disaster planning and cyber security. Accordingly, the entire content security ecosystem (including cyber security and disaster recovery) becomes more critical, given that there are many moving pieces, vendors, points where content is accessed and passed, and greater potential points of failure.
As companies embrace a cloud and virtualized architecture, it will become increasingly important to balance this new approach to proven and harden facility-based disaster recovery solutions. Indeed, a fully hybrid approach to managing content distribution globally will provide programmers with the best combination of cost effectiveness, efficiency, and peace-of-mind. Virtualized networks allow for a very robust approach to disaster recovery, provided service consistency is maintained in a cloud-based environment. The industry can take advantage of economies of scale in hardware deployments, and for some services, the physical or geographic presence of the primary and disaster recovery service is irrelevant. Some critical components of the program origination and distribution path need a physical presence at one or more locations (satellite uplinks, for example) but playout and program elements can exist in a virtual environment with inherent protections compared to a single facility.
The move to cloud-based virtualized systems is still in its infancy for primary, high-yielding networks. Some major programmers have announced a move in this direction, but a move to a fully virtualized environment will take some time (if it happens at all), given the value of knowing exactly where and how the content is being managed. Some customers are more comfortable than others, but for now a hybrid facility/virtual network design properly balances comfort, reliability, cost and flexibility as the industry becomes familiar with the risks and rewards of cloud-based services for primary distribution.
The Role of the Service Provider
Service providers give a valuable and independent service to customers for content security and disaster recovery. When looking at protecting services, key considerations are geographic, company, design and vendor diversity. Various things can go wrong that require the activation of a backup or recovery path. In the past, hardware or infrastructure (power, antenna) failures required the activation of a disaster recovery service but in the future major issues could be related to software viruses affecting a particular vendor or company or financial or management issues causing concerns over service quality. Striking a balance of technology and providers among a primary and disaster recovery plan is very important.
One example is media asset management. While it’s tempting (and easier) to demand the same underlying asset management systems for the primary and disaster recovery paths, it’s good to diversify and ensure cross compatibility with assets and the asset management systems in use by the primary and disaster recovery provider. This provides the added benefit of not getting locked in to a single vendor workflow and allows for easier replacement with the media asset management systems when required or desired.
Service disruptions of any kind are never acceptable to the broadcasting community. The evolution from strictly broadcast-style networks (dedicated via fiber and satellite links) to a hybrid IP environment requires a combined approach to business continuity and security, leveraging expertise in both broadcast networking and engineering and IP and IT networking. It is critical that customers and providers not move too far to one end of the technology spectrum, as there is too much at risk with high-value television networks to treat them like another IP stream.
There is no question and no debate that a migration to IP-based networks and services is the natural (and proper) evolution for broadcast networks, but a hybrid approach to the old and the new, and a thoughtful and well-architected approach to content security, cyber security and disaster recovery will ensure that uptime is maintained and channels are always available to distributors and viewers.
Do’s and Don’ts for Disaster Recovery:
DO:
- Ensure your disaster recovery plan is appropriate for you. DR is like insurance – make sure you have what you need but don’t overdo it.
- Plan for appropriate vendor and geographic diversity.
- Exercise your DR plan regularly. A backup that cannot be implemented based on the SLA isn’t useful.
- Evaluate your plan regularly to ensure it’s still appropriate.
- Talk to others about their experiences and what’s worked (and not worked) for them.
DON’T:
- Put all your eggs in one basket, with vendors, providers or geography.
- Ignore any potential DR requirements: Secondary networks, business and IT systems, advertising and promotional element all need to be considered.
- Implement and forget. Evaluate and test regularly. Review continuously.
- Be vague on SLA and procedural requirements. DR services need to be there when required!
- Wait until disaster strikes – work on a DR plan NOW! Start small if you need to – services are highly scalable and a simple disaster recovery plan is better than none.
You might also like...
Designing IP Broadcast Systems - The Book
Designing IP Broadcast Systems is another massive body of research driven work - with over 27,000 words in 18 articles, in a free 84 page eBook. It provides extensive insight into the technology and engineering methodology required to create practical IP based broadcast…
Demands On Production With HDR & WCG
The adoption of HDR requires adjustments in workflow that place different requirements on both people and technology, especially when multiple formats are required simultaneously.
If It Ain’t Broke Still Fix It: Part 2 - Security
The old broadcasting adage: ‘if it ain’t broke don’t fix it’ is no longer relevant and potentially highly dangerous, especially when we consider the security implications of not updating software and operating systems.
Standards: Part 21 - The MPEG, AES & Other Containers
Here we discuss how raw essence data needs to be serialized so it can be stored in media container files. We also describe the various media container file formats and their evolution.
NDI For Broadcast: Part 3 – Bridging The Gap
This third and for now, final part of our mini-series exploring NDI and its place in broadcast infrastructure moves on to a trio of tools released with NDI 5.0 which are all aimed at facilitating remote and collaborative workflows; NDI Audio,…