IP Monitoring & Diagnostics With Command Line Tools: Part 2 - Testing Remote Connections
In the previous article, we set the scene for working with the Command Line Interface (CLI) on a UNIX system. Now we will explore some techniques for performing basic tests on our network infrastructure to check for potential problems.
More articles in this series:
The examples will mainly use IP version 4 addresses because they are more concise. In general, IP version 6 addresses work in all the same places. Occasionally you will need to add a modifier to a command to select which type of addressing you want to use.
At each IP address, we connect to a numbered port. Many of these are well known and configured identically everywhere. Think of an IP address being like a street name and a port being a house number in that street.
Security concerns will become more important when you are connecting to machines outside of your local network.
Things that can break
There are many reasons for a communication breakdown between two machines. Here is a list of them arranged in a sensible order for checking:
- Faulty or unplugged cables and patch panels. You will need to visit the hardware and check this. Either end could be unplugged or the cable could be damaged.
- Hardware Mac addresses could be duplicated although it is very rare. This could happen if you buy a batch of cheap Ethernet interface cards from an unreliable supplier.
- Network switch configuration issues disallowing sockets to connect to each other. You will need to consult with your IT department so they can check the configuration.
- IP address could be misconfigured as illegal values. There are several strategies for discovering what devices are present on your local network. They won't see devices that are powered off though. Later, we will build a tool to aggregate the list from several sources and display the result.
- DNS issues that compromise the IP address. Work-around this by using IP addresses directly to avoid a DNS lookup.
- Sometimes a route stops working due to a breakdown or reconfiguration and the alternative route is less optimal (more hops and higher latency).
- Routers must be set up to allow port forwarding to reach the target port on the destination machine.
- TCP Ports could be blocked by a firewall in the destination machine.
- The target ports might not have a listener service configured to respond. Attempt a portscan on the target to see what is open and usable.
Checking the DNS
The Domain Name System (DNS) is a reference list of human readable names cross referenced against the physical IPv4 and IPv6 addresses. It is possible for DNS records to be legal but incorrect.
The first step is to use the nslookup command to resolve a fully qualified machine name to an IP address. Here is an example of one that resolves a simple DNS record to an IP address:
nslookup www.myorg.com
Server: 202.15.130.50
Address: 202.15.130.50#53>Non-authoritative answer:
Name: www.myorg.com
Address: 10.208.35.165
The first two lines in the result describe the DNS name server that resolved your query and which port was used to connect to it.
These are the most likely response codes you will see. There are a few others but they are very rare:
Result code | Meaning |
---|---|
NOERROR | A valid record was found. |
NXDOMAIN | The domain did not exist in any of the DNS name servers that were reachable from your computer. Check the spelling of the domain name you looked for. |
SERVFAIL | Might indicate that the DNS name server is broken or a firewall is blocking you. Escalate the problem to the people who look after your DNS server. |
REFUSED | The name server is prohibited from giving you an answer. This is also cause for escalation. |
If the nslookup succeeded, the IP address returned might still be incorrect even though it was valid. Compare the IP address against your documentation to ensure it is the one you expect.
If you know the correct IP address, then reverse the lookup with the whois command to see if it is configured with a different domain name.
whois 10.208.35.165
% IANA WHOIS server
% for more information on IANA, visit http://www.iana.org
% This query returned 1 object
...
{very detailed information about the owner
of that IP address here}
...
% This query was served by the RIPE Database Query Service
version 1.104 (BLAARKOP)
Checking the connection
The ping command is a useful tool for determining whether a target node is reachable. If it is, what is the latency of the connection?
If we don't tell it to stop, ping will run continuously until you manually halt it with a [Control] + [C] keystroke. Add the -c {count} option to the command to request a limited number of test cycles.
Here is a single ping test on a machine that is reachable:
ping -c 1 212.159.13.50
PING 212.159.13.50 (212.159.13.50): 56 data bytes
64 bytes from 212.159.13.50:
icmp_seq=0 ttl=61 time=12.600 ms
--- 212.159.13.50 ping statistics ---
1 packets transmitted, 1 packets received,
0.0% packet loss
round-trip
min/avg/max/stddev = 12.600/12.600/12.600/0.000 ms
If the device is not reachable, then the ping command will tell you there was a 100% packet loss. Indicating a count of 1 will fire the ping test just once but the result will not appear until that test completes the timeout after a few seconds. Prove this with a known to be illegal IP address:
ping -c 1 254.254.254.254
PING 254.254.254.254 (254.254.254.254): 56 data bytes
--- 254.254.254.254 ping statistics ---
1 packets transmitted, 0 packets received,
100.0% packet loss
Tracing the route
Probe the connection and see if we can reach 'there' from 'here'.Something may be blocking us along the way, or there may be a breakage in the route altogether.Rather like an open circuit in a traditional wired system.
The ping command established whether the other end is reachable. If it is, then we could use the traceroute command to examine the route that our packets are taking to get there. This may indicate a sub-optimal route which would cause performance issues.
The route is dynamic and might change unexpectedly. In a high availability environment, there may be more than one route to the destination. The connection will use the fastest route available but if that fails, the fallback alternative may be slower. You will need elevated privileges to run the traceroute command.
Here is a partial traceroute result to a working destination:
traceroute 212.159.13.50
br/> traceroute to 212.159.13.50 (212.159.13.50), 64 hops max,
52 byte packets
1 192.168.1.1 (192.168.1.1) 1.400 ms 1.125 ms 1.088 ms
2 195.166.130.249 (195.166.130.249) 7.685 ms
7.803 ms 7.750 ms
3 gi2-23-442.ptn-gw02.plus.net (84.93.253.75)
8.453 ms 8.334 ms
gi2-1-444.ptn-gw01.plus.net (84.93.253.79)
8.194 ms
4 * * gi2-23-442.ptn-gw02.plus.net (84.93.253.75)
8.178 ms !X
5 * * *
6 *
The asterisks indicate response timeouts. If you see multiple lines of three asterisks, the traceroute cannot get any further. It does not necessarily mean the node is unreachable. The mechanism that traceroute uses might be purposely blocked to prevent intruders deducing the inner workings of the network.
Checking for open ports
If we cannot ping the target system, but we think it is up and running, (perhaps with ICMP packets being blocked), then try an alternative. The nmap tool could run a portscan for example. You may need to install the nmap tool before using it.
Running unauthorised portscans on machines owned by other organisations can get you into serious trouble because it looks like an intrusion attempt. Always ask first and warn your own IT department too.
Once you have established that the target destination is reachable, check whether the ports that your applications needs are open. Consult your documentation to see what they are. Then use the nmap command to run a portscan on the target. This will report whether they are reachable and which listening services are configured on them.
nmap myserver.mynet.com
Starting Nmap 7.60 ( https://nmap.org ) at
2022-11-17 12:42 GMT
Nmap scan report for myserver.mynet.com (192.168.1.24)
Not shown: 995 closed ports
PORT STATE SERVICE
80/tcp open http
81/tcp open hosts2-ns
443/tcp open https
3306/tcp open mysql
8200/tcp open transit2
Nmap done: 1 IP address (1 host up) scanned in 0.05 seconds
Run the same check on your own system to see whether other nodes might be able to connect back to you. Substitute localhost for the target host name or use the IP address 127.0.0.1 instead.
Conclusion
After eliminating hardware problems due to faulty cabling and potential issues with the router or network switch, we used the command line to run some diagnostics. The ping tool tested basic connectivity. Then we traced the route between the two systems. Finally, we ran a portscan to see what remote services were available. Some tools require elevated privileges. Manage that carefully and grant them only on a temporary basis when needed. Operating with minimum privileges whenever possible will significantly reduce the risk of accidents. In the next article we will see how to send and receive information from one machine to another and how to ask remote machines to tell us about their status.
You might also like...
Designing IP Broadcast Systems
Designing IP Broadcast Systems is another massive body of research driven work - with over 27,000 words in 18 articles, in a free 84 page eBook. It provides extensive insight into the technology and engineering methodology required to create practical IP based broadcast…
If It Ain’t Broke Still Fix It: Part 2 - Security
The old broadcasting adage: ‘if it ain’t broke don’t fix it’ is no longer relevant and potentially highly dangerous, especially when we consider the security implications of not updating software and operating systems.
Standards: Part 21 - The MPEG, AES & Other Containers
Here we discuss how raw essence data needs to be serialized so it can be stored in media container files. We also describe the various media container file formats and their evolution.
NDI For Broadcast: Part 3 – Bridging The Gap
This third and for now, final part of our mini-series exploring NDI and its place in broadcast infrastructure moves on to a trio of tools released with NDI 5.0 which are all aimed at facilitating remote and collaborative workflows; NDI Audio,…
Microphones: Part 2 - Design Principles
Successful microphones have been built working on a number of different principles. Those ideas will be looked at here.