IP Monitoring & Diagnostics With Command Line Tools: Part 1 - The Command Line Interface
Being faced with an IP and file-based broadcast environment can be daunting if you are not already an experienced Information Technology systems administrator. This series of articles will help you acquire the skills one-by-one to monitor and diagnose problems from the UNIX command line. Then you can build more sophisticated tools of your own to manage and automate your systems.
All 12 articles in this series are now available in our free 80 page eBook ‘IP Monitoring & Diagnostics With Command Line Tools’ – download it HERE.
All articles are also available individually:
Introduction
When you build broadcast and media systems on a UNIX foundation, you also need Information Technology (IT) skills. Working under pressure with unfamiliar technology might be daunting at first. Don't panic! This series of articles will cover each topic in easy to digest pieces.
The Command Line Interface (CLI) in a UNIX system provides a lot of leverage to help you solve problems in an IP based infrastructure. That CLI is available in all of the operating systems you are likely to encounter. The Macintosh OS is based on a UNIX like foundation and Windows systems can be configured to look very similar. Most of the tools you need are based on open source projects which have been ported to all three platforms.
I will introduce some high-level topics to start with. Later articles will examine them in more detail:
In this article I introduce the Command Line Interface.
Next time, I cover connections between devices.
Then we look at implementing remote monitoring.
If you are new to this, some of the terminology may be opaque at first. Take it on faith for now. It will all become clear over the next few articles
Conceptually your entire system is simply a collection of distributed processes sending messages to each other.
The scenarios described in these articles have all happened in real-world situations. The solutions, advanced monitoring and diagnostic tools we will create later on have all been proved to work on live systems.
A problem-solving strategy
The diagnostic process is based on experience and the knowledge you acquire during the investigation. We will not have all the answers to start with. Just-in-time learning on the job is the key skill to develop. Look for familiar patterns that suggest probable causes. Apply your diagnostic methodology like this:
- Make plenty of notes during the entire process.
- Simplify the situation.
- Eliminate complexity.
- Reduce to the minimum required to reproduce the problem.
- Test for one thing at a time.
- Alter the configuration if necessary. Note how to change it back.
- When the test is complete, restore what you changed.
- Only change one thing at a time before testing again.
Documenting everything you do is vital. Your systems documentation should be complete and up to date and then accurately track any changes you apply to your systems.
When you discover a cure for the problem, publish a change note that describes the problem and how you solved it. Capture the details of any changes made to the system to fix it. Also describe how to undo the changes and remove the fix.
About the UNIX environment
UNIX is an architectural family in the same way that 'Windows' collectively describes the Microsoft operating systems.
Linux describes a sub-set, built on an open source foundation. There are many varieties of Linux. These systems are customised with configuration files or by rebuilding components from that open source code. Linux is a popular choice for deploying scalable infrastructures that are tailored to specific needs.
UNIX provides many powerful features for building your own tools:
- Command line tools are small, focussed and compact. Each is designed to accomplish a simple task efficiently.
- The input and output of any command line tool or process can be redirected to go to or come from a different place.
- Chain individual commands into pipelines to execute more complex tasks.
- Anything you type on the command line can be edited into a script to be called up again for repetitive tasks or to create new commands.
- Scripts can be scheduled for automatic execution on a regular basis.
- The file system manages the storage and makes the hardware devices and network available to your tools.
- Regular Expressions describe a syntax for manufacturing powerful text filters for searching, pattern matching, filtering and data validation.
- You have a great deal of control over how processes are executed.
- Helpful manual pages are available with the man command.
Terminal emulators
Before Graphical User Interfaces (GUI) were popular, the only way to interact with a computer was with a Command Line Interface using a terminal screen and keyboard. This was not hard, but required a large vocabulary of arcane commands. This is all still available but well-hidden on modern computers.
Start your journey into the world of Command Line Interfaces with one of these applications:
- Microsoft Windows Terminal app for keyboard and text-based interaction with the underlying operating system.
- Cygwin is also a useful addition to a Windows workstation because it creates a UNIX like environment to work in.
- Apple Macintosh systems have a Terminal app in the Utilities folder.
- Linux users have many alternative terminal emulators available. The default may be different depending on whether you have the Debian, Ubuntu, Suse or Red Hat (Fedora) version. The gnome-terminal app is often available. Install a different terminal app if you prefer it.
About command lines
When you start up your terminal emulator, it displays a small flashing cursor and waits for you to type something. You are now 'talking' to a shell on your own computer. The shell accepts your typed commands and echoes their output to the screen. Pressing the [RETURN] key calls the command to action.
On most systems, you are now in the UNIX environment. The exception is the Microsoft Windows Terminal app where you are still talking to a Windows OS. This is different to UNIX. Installing Cygwin to emulate UNIX instead might be helpful.
Some simple commands
Everything we build in this environment uses simple command line tools. Your skills will grow exponentially as you understand more of them. Start with these basic commands:
| Command | Description |
|---|---|
| man | Show the manual page for a command. |
| ls | List the contents of a directory. |
| cd | Change the current working directory. |
| pwd | Show the current working directory. |
| mkdir | Create a directory. |
| rmdir | Remove a directory. |
| cp | Duplicate a file to a new location. |
| mv | Move a file to a new location and/or rename it. |
| rm | Remove a file. Be careful, this may be dangerous. |
| uname | Describe the operating system. |
| hostname | Show the DNS name of this host. |
Use the online manual pages or find them on the web to read about each command before using it. In your command line type this:
man {command-name}
If you put "man ls" into a web search engine, it will list web pages about the ls command with the same help. The online versions may describe a different OS with options you cannot use. There is always a work-around with UNIX because it provides many different ways to accomplish the same thing.
The uname -a command tells you a lot about the version and build date of the operating system. Compare this against the other systems on your network to see if this machine is updated to the same revision as its siblings. Maintaining the same OS version across all your systems (if possible) will avoid version conflicts.
Connecting to another system
Use a Secure SHell (ssh) command to log in to a machine elsewhere in your broadcast infrastructure. Then you can run diagnostic tools directly on that computer. For now, we will assume that connection is possible. We shall diagnose why there could be a problem later on.
To solve problems with a particular application running on that remote machine, you must use the same account as that application. Otherwise it may not reveal the problem. Prefix the host description on the ssh command with the account name.
Use an IP address or a host name to establish the connection:
Now enter the password to gain access.
You may be challenged with a prompt to create and save a unique key. Answer yes and press [RETURN]. You only need to do this once. Now, you are logged in to your remote machine and the commands you type will be executed there.
To leave the remote session and go back to your own command line shell, type exit and press [RETURN].
If a network connection is not possible, visit the physical machine and access the command line shell directly on the operator console.
Conclusion
In this article, we have started to explore the Command Line Interface (CLI). Now familiarise yourself with the individual commands one at a time. Read the documentation and experiment on a test system before using them on a live system. This environment is unforgiving and you can cause serious disruption if you are careless.
The diagnostic process relies on collecting observations and recognising patterns. When this is understood, it can be encapsulated in monitoring tools that can watch for changes in the nominal performance of a system. These can generate alerts in the first place and call for help from a systems administrator. The most advanced systems will be able to deal with problems automatically and self-heal themselves. Let the computer do the work so you don't need to.
Learning how these features work equips you with a powerful text and data processing toolkit. The same tools you use for diagnosis and monitoring will improve your productivity in other areas as well if you learn how to use them skilfully.
In the next article, I will introduce you to some tools for examining the network and the connections between one machine and another.
You might also like...
Designing IP Broadcast Systems - The Book
Designing IP Broadcast Systems is another massive body of research driven work - with over 27,000 words in 18 articles, in a free 84 page eBook. It provides extensive insight into the technology and engineering methodology required to create practical IP based broadcast…
Demands On Production With HDR & WCG
The adoption of HDR requires adjustments in workflow that place different requirements on both people and technology, especially when multiple formats are required simultaneously.
If It Ain’t Broke Still Fix It: Part 2 - Security
The old broadcasting adage: ‘if it ain’t broke don’t fix it’ is no longer relevant and potentially highly dangerous, especially when we consider the security implications of not updating software and operating systems.
Standards: Part 21 - The MPEG, AES & Other Containers
Here we discuss how raw essence data needs to be serialized so it can be stored in media container files. We also describe the various media container file formats and their evolution.
NDI For Broadcast: Part 3 – Bridging The Gap
This third and for now, final part of our mini-series exploring NDI and its place in broadcast infrastructure moves on to a trio of tools released with NDI 5.0 which are all aimed at facilitating remote and collaborative workflows; NDI Audio,…