The transition from a mostly hardware-based solution to one that is software-based brings with it new requirements. One aspect that many technical managers will find unsettling is that changes come more rapidly. Just when the staff becomes comfortable with a process, vendors suggest an improved process.

In most cases that means software updates. Even if the engineering staff faithfully manages the regular software update process, at some point the vendor will stop supporting your version and suggest you move on to the next iteration. Then there are service agreements and licenses, all of which expire or change as products approach their end of life. These issues require proactive thinking if you are to prevent a system’s crash.

Change is easier for vendors

The broadcast and production industries were vastly different 10 or 20 years ago. There were a mere handful of very large companies that supplied almost everything to stations. A broadcaster or production house could buy virtually everything that might be needed from a single company. Today, the racks may be filled with a divergent range of vendor solutions. But it all still has to work together as a system.

Add to the multiplicity of available solutions is the vendor consolidation. While on the surface that appears to mean just one less vendor. It also often results in discontinued products and services.

The term, “support” for a device may now be measured in low, single-digit years. Software may even be updated several times per year. This instability make engineers’ and owners’ lives more complicated. The changes also have a direct impact on both capital and operating budgets.

Perhaps one of the most upsetting aspects of this new software world is the obsolescence of software “versions” and the inherent service contract that must accompany them.

Versioning conundrum

The interdependency and integration of systems has never been as meshed together and critical as it is today. In the past we would update a driver here and there in an editor or device controller. In the computer centric topology, applications communicate with each other directly with drivers, through API’s or with middleware, there’s software and firmware.

Let us consider a common scenario. Vendor A issues a new patch, but that vendor may never know how that change ripples down line to impact a myriad of other devices attached and communicating with it? Or more dramatically, when a new software version is installed, even when it’s the same vendors’ technology that’s attached, there is still an impact on the attached systems. This raises a question, do all the connected devices need similar updates to be applied at the same time? An even larger pothole could develop. Did vendor A check with the other equipment manufacturers and run tests to be sure the new software worked with their equipment?

Recent history is replete with examples of where an “update” was applied to a system, which responded to by promptly crashing. 

In May, Starbucks suffered crashes around the world with some of its payment systems following a routine technology update. The problems struck cafes in both North America and Europe, preventing an unspecified number of registers from working, according to company spokesman Reggie Borges. “We were doing a technology update for our store registers in the U.S. and Canada, and a limited number of those locations [were affected]…When it was time to get them back up, some of them didn’t take,” he said. One publication calculated that Starbucks lost about $3million dollars in revenue over the 45 minute outage because store cash registers were inoperable.

Closer to home, thousands of owners of high-end Samsung TVs complained after an August software update left their recently acquired $1,800 sets with blank, unusable screens. From a Guardian report:

Software updates, patches and refreshes may cause far more damage than any good they were supposed to do.

Back to broadcast. I have discovered that one vendor’s SLA states that it is the customer’s responsibility to check on versioning and install the provided update on time. If the customer does not maintain the proper software versions, the SLA states the company will no longer support any device that is more than two versions behind the current version.

Another vendor simply lists the latest version on their support site. It is your job to verify if you need that update. If you do, you must send an email to the company to get the update because the changes have to be installed by the company.

Changing directions

Here’s a fun one. As the media changes with mergers and acquisition, the newly formed consolidated companies sometimes turn in new directions. At the recent NAB NY convention, I participated in a roundtable discussion that was moderated by a recently consolidated vendor I will call company A. That company’s newly acquired products are part of a project on which I am working.

Company A sold my client a product from company B. Company A then acquired Company C, which had a competitive product to those from Company B. After the sale company A announced they were fully abandoning both Product A and the acquired Company B product line and going in a new direction. My client now anticipates losing product support.

Have a contingency plan

My articles for The Broadcast Bridge, have discussed the importance of using new types of engineering documentation to help engineers better understand the interdependency between applications and devices. It is important to first recognize these dependencies and second, to properly manage their versioning. For every system, device or application there is likely something upstream or downstream that was configured or developed based on one particular version of software. Manufacturers often specify which interfaces they have tested and recommend. The customer needs to ask if the vendor reached out to the original list of companies to retest and recertify the new software with whatever changes the other vendors may have made in their systems.

I would like to say there is an easy solution to most of these issues, but there isn’t. My current projects use new tracking system documentation to manage version management. The practice involves a routine maintenance program that includes checking for updates and educating staff about the importance of thinking upstream and downstream before applying any software update. Technicians need to ask themselves, “Do those systems also need to be updated?”

In the case of vendors that perform remote software updates, it is important to confirm in advance that all of a facility’s interfaces and services will not be affected during and after the update.

Software updates, patches and refreshes should be easy to perform and reliable in operation. The fact remains they are neither. Pause and consider all of the undesired possibilities before you click on the “Software Update” button. 

Editor’s Note: Gary Olson has a book on IP technology, “Planning and Designing the IP Broadcast Facility – A New Puzzle to Solve”, which is available at bookstores and online.