The answer is critical testing to ensure that these device controllers comply with the Advanced Media Workflow Association (AMWA)’s NMOS IS-04 and IS-05 protocols. This helps ensure that the controller is making the correct API calls to the network registry (04) and to the nodes to set up the correct connections between them (05).

Up until this year, such testing was conducted manually, which was expensive and time consuming. Thus, the NMOS Client Testing Work Group, within AMWA, was formed in May of this year to develop a suite of automated test (software) tools. The idea is to test every controller before it is deployed on a network to ensure it will work with other products on the network. These tests are vital to keeping an IP network up and running smoothly.

“Broadcasters want better products out on the market as quickly as possible,” said Rob Porter, Project Manager within the Advanced Technology Team at Sony. He’s based at Sony’s European Professional Engineering facility in Basingstoke, UK, and has led a team of AMWA members that included major broadcasters like the BBC and CBC in Canada as well as vendors of broadcast equipment, like EVS, Imagine Communications, Matrox Video and Pebble.

Porter, together with colleague Jonathan Thorpe, gave a presentation on the status of this “Automated NMOS Controller Testing” at the IP Oktoberfest 2021 virtual event last month, sponsored by the Alliance for IP Media Solutions (AIMS) and AMWA. AMWA has made this new test suite available for free at AMWA’s GitHub website (https://github.com/AMWA-TV) for anyone to review and implement.

“We need good testing to ensure these new products will interoperate correctly on the network,” he said. “Simply put, automated testing will make everyone’s life a bit easier, when it comes to testing, and could lead to facilities being designed and implemented faster and more efficiently.”

There is an existing test suite developed by AMWA over recent years, but it is only able to test the media nodes on the network (the cameras, servers, etc.) to make sure that they are conforming to the NMOS specifications and communicating over the APIs. It ensures that the devices register with the network registry correctly, for example. This test suite is also able to test the registry itself and make sure it’s able to process those API calls correctly, both by registering devices that attempt to register with the registry and also by telling a controller that queries the registry about what is available on the network.

What was missing from the first generation test tools was the ability to test each manufacturer’s controller. As with much of AMWA’s work, the specialized test software does the work.

“Testing controllers was something that was always done manually in the past,” said Porter. “Vendors were self-testing their controllers, which sometimes resulted in widely different results and interoperability issues. Now we have a new suite of automated tools that make the process easier and, I’d say, more genuine and fairer to everyone involved.”

As part of the automated test tools development process—which began in May of this year with the Working Group meeting every week for an hour over four months—the group studied the Joint Task Force on Networked Media (JT-NM) certification event from March of 2020, during which participating vendors connected their controller to a test VPN network set up at the CBC. Over a secure connection the broadcasters instructed each vendor to carry out various operations with their controller to discover what’s on the registry and tell them what they saw on their end. Or they had them try to make a connection between the sender and receiver in the correct way. They would then verify that those connections were indeed happening as they should.

It was a very time consuming (over several days) and expensive process for everyone involved. But they learned a lot and it all worked very well. The six vendors participating were then able to claim that their controllers were “JT-NM Tested”, which is something to brag about to customers.

“After an experience like this, it was clear that an automated test tool was required, that would basically replace that single engineer doing the testing at a broadcast facility with software tools and an automated process that could run in the background and perform the job quickly and accurately,” said Porter. “On the vendor side, we are also excited about speeding up the process of testing and releasing new products, and are committed to working on this area of interoperability in the long run.”

What the Working Group has come up with is a suite of tools that can test in either Semi-Automated or Fully Automated ways.

Semi-Automated still requires a human to drive the controller’s GUI according to specific instructions provided by the new testing tool. Because you are connecting your controller to a set of mocked up resources (mock nodes and mock registry) on a network, not only is the testing tool verifying that those API calls have been made correctly, but it is also using a test façade (with a webpage GUI) that gives the engineer instructions on what steps to follow.

For example, the operations engineer looks at his interface that’s connected to the mock registry and they see a list of “senders”, devices waiting to join the network. The engineer then selects from a list of multiple-choice answers that they can see on the testing façade. The test tool then verifies that they have correctly answered the questions, while at the same time checking it has received the correct API calls over the network from the controller.

So, it’s testing at two levels: one, that the operator is seeing the right device on their user interface and two, that the controller has made the right API calls.

The Fully Automated Testing mode is designed for the benefit of the vendor to speed up the process of product testing—ensuring the code is written correctly—in their labs. This method allows the user to perform a new set of testing using a Selenium robot framework to automatically operate the controller and run it against the AMWA test tool. The test tool is the same as the Semi-Automated one, but the way you operate the controller to test against it, is different and requires no human interaction, allowing it to be used as part of vendors’ automated continuous integration processes for software development.

The full test suite is now available in Beta for review and real-world testing on AMWA’s GitHub site. It performs the same level of testing that was performed at the JT-NM Tested event a year ago, but in an automated way, saving time and effort in the process. More development work and additional capability is sure to follow.

“Interoperability issues are expensive, not just for the end user but also for the system integrator and for all of the vendors that are involved in making these complex systems work,” said Porter. “If we can minimize those issues, we’ll all be in a good place.”