IP Monitoring & Diagnostics With Command Line Tools: Part 1 - The Command Line Interface
Being faced with an IP and file-based broadcast environment can be daunting if you are not already an experienced Information Technology systems administrator. This series of articles will help you acquire the skills one-by-one to monitor and diagnose problems from the UNIX command line. Then you can build more sophisticated tools of your own to manage and automate your systems.
All 12 articles in this series are now available in our free 80 page eBook ‘IP Monitoring & Diagnostics With Command Line Tools’ – download it HERE.
All articles are also available individually:
Introduction
When you build broadcast and media systems on a UNIX foundation, you also need Information Technology (IT) skills. Working under pressure with unfamiliar technology might be daunting at first. Don't panic! This series of articles will cover each topic in easy to digest pieces.
The Command Line Interface (CLI) in a UNIX system provides a lot of leverage to help you solve problems in an IP based infrastructure. That CLI is available in all of the operating systems you are likely to encounter. The Macintosh OS is based on a UNIX like foundation and Windows systems can be configured to look very similar. Most of the tools you need are based on open source projects which have been ported to all three platforms.
I will introduce some high-level topics to start with. Later articles will examine them in more detail:
In this article I introduce the Command Line Interface.
Next time, I cover connections between devices.
Then we look at implementing remote monitoring.
If you are new to this, some of the terminology may be opaque at first. Take it on faith for now. It will all become clear over the next few articles
Conceptually your entire system is simply a collection of distributed processes sending messages to each other.
The scenarios described in these articles have all happened in real-world situations. The solutions, advanced monitoring and diagnostic tools we will create later on have all been proved to work on live systems.
A problem-solving strategy
The diagnostic process is based on experience and the knowledge you acquire during the investigation. We will not have all the answers to start with. Just-in-time learning on the job is the key skill to develop. Look for familiar patterns that suggest probable causes. Apply your diagnostic methodology like this:
- Make plenty of notes during the entire process.
- Simplify the situation.
- Eliminate complexity.
- Reduce to the minimum required to reproduce the problem.
- Test for one thing at a time.
- Alter the configuration if necessary. Note how to change it back.
- When the test is complete, restore what you changed.
- Only change one thing at a time before testing again.
Documenting everything you do is vital. Your systems documentation should be complete and up to date and then accurately track any changes you apply to your systems.
When you discover a cure for the problem, publish a change note that describes the problem and how you solved it. Capture the details of any changes made to the system to fix it. Also describe how to undo the changes and remove the fix.
About the UNIX environment
UNIX is an architectural family in the same way that 'Windows' collectively describes the Microsoft operating systems.
Linux describes a sub-set, built on an open source foundation. There are many varieties of Linux. These systems are customised with configuration files or by rebuilding components from that open source code. Linux is a popular choice for deploying scalable infrastructures that are tailored to specific needs.
UNIX provides many powerful features for building your own tools:
- Command line tools are small, focussed and compact. Each is designed to accomplish a simple task efficiently.
- The input and output of any command line tool or process can be redirected to go to or come from a different place.
- Chain individual commands into pipelines to execute more complex tasks.
- Anything you type on the command line can be edited into a script to be called up again for repetitive tasks or to create new commands.
- Scripts can be scheduled for automatic execution on a regular basis.
- The file system manages the storage and makes the hardware devices and network available to your tools.
- Regular Expressions describe a syntax for manufacturing powerful text filters for searching, pattern matching, filtering and data validation.
- You have a great deal of control over how processes are executed.
- Helpful manual pages are available with the man command.
Terminal emulators
Before Graphical User Interfaces (GUI) were popular, the only way to interact with a computer was with a Command Line Interface using a terminal screen and keyboard. This was not hard, but required a large vocabulary of arcane commands. This is all still available but well-hidden on modern computers.
Start your journey into the world of Command Line Interfaces with one of these applications:
- Microsoft Windows Terminal app for keyboard and text-based interaction with the underlying operating system.
- Cygwin is also a useful addition to a Windows workstation because it creates a UNIX like environment to work in.
- Apple Macintosh systems have a Terminal app in the Utilities folder.
- Linux users have many alternative terminal emulators available. The default may be different depending on whether you have the Debian, Ubuntu, Suse or Red Hat (Fedora) version. The gnome-terminal app is often available. Install a different terminal app if you prefer it.
About command lines
When you start up your terminal emulator, it displays a small flashing cursor and waits for you to type something. You are now 'talking' to a shell on your own computer. The shell accepts your typed commands and echoes their output to the screen. Pressing the [RETURN] key calls the command to action.
On most systems, you are now in the UNIX environment. The exception is the Microsoft Windows Terminal app where you are still talking to a Windows OS. This is different to UNIX. Installing Cygwin to emulate UNIX instead might be helpful.
Some simple commands
Everything we build in this environment uses simple command line tools. Your skills will grow exponentially as you understand more of them. Start with these basic commands:
| Command | Description |
|---|---|
| man | Show the manual page for a command. |
| ls | List the contents of a directory. |
| cd | Change the current working directory. |
| pwd | Show the current working directory. |
| mkdir | Create a directory. |
| rmdir | Remove a directory. |
| cp | Duplicate a file to a new location. |
| mv | Move a file to a new location and/or rename it. |
| rm | Remove a file. Be careful, this may be dangerous. |
| uname | Describe the operating system. |
| hostname | Show the DNS name of this host. |
Use the online manual pages or find them on the web to read about each command before using it. In your command line type this:
man {command-name}
If you put "man ls" into a web search engine, it will list web pages about the ls command with the same help. The online versions may describe a different OS with options you cannot use. There is always a work-around with UNIX because it provides many different ways to accomplish the same thing.
The uname -a command tells you a lot about the version and build date of the operating system. Compare this against the other systems on your network to see if this machine is updated to the same revision as its siblings. Maintaining the same OS version across all your systems (if possible) will avoid version conflicts.
Connecting to another system
Use a Secure SHell (ssh) command to log in to a machine elsewhere in your broadcast infrastructure. Then you can run diagnostic tools directly on that computer. For now, we will assume that connection is possible. We shall diagnose why there could be a problem later on.
To solve problems with a particular application running on that remote machine, you must use the same account as that application. Otherwise it may not reveal the problem. Prefix the host description on the ssh command with the account name.
Use an IP address or a host name to establish the connection:
Now enter the password to gain access.
You may be challenged with a prompt to create and save a unique key. Answer yes and press [RETURN]. You only need to do this once. Now, you are logged in to your remote machine and the commands you type will be executed there.
To leave the remote session and go back to your own command line shell, type exit and press [RETURN].
If a network connection is not possible, visit the physical machine and access the command line shell directly on the operator console.
Conclusion
In this article, we have started to explore the Command Line Interface (CLI). Now familiarise yourself with the individual commands one at a time. Read the documentation and experiment on a test system before using them on a live system. This environment is unforgiving and you can cause serious disruption if you are careless.
The diagnostic process relies on collecting observations and recognising patterns. When this is understood, it can be encapsulated in monitoring tools that can watch for changes in the nominal performance of a system. These can generate alerts in the first place and call for help from a systems administrator. The most advanced systems will be able to deal with problems automatically and self-heal themselves. Let the computer do the work so you don't need to.
Learning how these features work equips you with a powerful text and data processing toolkit. The same tools you use for diagnosis and monitoring will improve your productivity in other areas as well if you learn how to use them skilfully.
In the next article, I will introduce you to some tools for examining the network and the connections between one machine and another.
You might also like...
HDR & WCG For Broadcast: Part 3 - Achieving Simultaneous HDR-SDR Workflows
Welcome to Part 3 of ‘HDR & WCG For Broadcast’ - a major 10 article exploration of the science and practical applications of all aspects of High Dynamic Range and Wide Color Gamut for broadcast production. Part 3 discusses the creative challenges of HDR…
IP Security For Broadcasters: Part 4 - MACsec Explained
IPsec and VPN provide much improved security over untrusted networks such as the internet. However, security may need to improve within a local area network, and to achieve this we have MACsec in our arsenal of security solutions.
Standards: Part 23 - Media Types Vs MIME Types
Media Types describe the container and content format when delivering media over a network. Historically they were described as MIME Types.
Building Software Defined Infrastructure: Part 1 - System Topologies
Welcome to Part 1 of Building Software Defined Infrastructure - a new multi-part content collection from Tony Orme. This series is for broadcast engineering & IT teams seeking to deepen their technical understanding of the microservices based IT technologies that are…
IP Security For Broadcasters: Part 3 - IPsec Explained
One of the great advantages of the internet is that it relies on open standards that promote routing of IP packets between multiple networks. But this provides many challenges when considering security. The good news is that we have solutions…