The first line of your script file

The first line of a UNIX shell script describes the interpreter that executes the script. It starts with a hash and exclamation symbol (#!) followed by the fully qualified path to the interpreter. Here are some examples:

Legacy scripts might need to run in an older version of the interpreter. Choose that here if you need it.

I/O redirection

Each process expects one stream of input data and delivers two streams of output text. By default, the command line passes the keyboard characters to the standard input stream and both output streams go to your screen. The streams are called:

• STDIN
• STDOUT
• STDERR

The normal output results are transmitted via STDOUT. If something goes wrong, the error messages are transmitted via STDERR. The two streams of text can be processed independently.

The shell can redirect the output streams to a different device, a log file or use them as input to another command. It can also redirect the input from another source instead of the keyboard.

Output redirection examples

Redirect the output of a command with the right facing caret (>) and save it in a file, overwriting any prior content:

ls -la > my_file_list.txt

Double caret symbols (>>) append to the file without overwriting it:

echo "Something went wrong" >> my_log_file.txt

In both cases, a new file will be created automatically if necessary.

This is how to discard the STDERR messages by redirecting them into a null device so they are ignored:

cat my_file.txt 2> /dev/null

Divert the STDOUT of a disk space check to a report file and save the STDERR messages in a log file:

df > output.txt 2>> error.log

Redirect both output streams to the same place. First, redirect STDOUT, then tell STDERR to follow it. This sends the contents of a data file through the sort command with a pipe and then stores the output with any errors that occur.

cat unsorted_data.txt | sort > output.txt 2>&1

Output redirection is incredibly powerful and will be used a great deal in a monitoring system to capture the results from the inspection commands.

Input redirection examples

There are subtle differences in the way the shell executes the commands when you use input redirection. You might only notice these if you monitor process listings while they run or inspect detailed file attributes and date stamps afterwards.

Print a file with one of these commands:

lpr file_to_print.txt
lpr < file_to_print.txt
cat file_to_print.txt | lpr

If you redirect the file using STDIN, or use cat to stream the file to the lpr command, the file name will not appear in the print queue listing. This enhances security and privacy.

Input redirection can read source text embedded within the script. The text is tagged to identify where it ends. We must append the redirected input with double carets (<<) to prevent the text from being executed as commands:

wc -c << EOF
How many characters are contained
in this example text before end of file tag?
EOF

Note:
Replace the -c on the wc command with -w to count words or -l to count lines.

Piped commands

The shell uses shorthand notation for chaining commands together by redirecting STDOUT and STDIN. The vertical bar character (|) creates a pipeline of instructions. Arranging the commands in the correct order is critical:

ls -la | cut -c35-37 | grep A | sort -r

The file listing from the ls command is sliced vertically with the cut command to only yield columns 35 to 37. The grep command discards any lines that do not contain a capital letter 'A'. Finally, the sort command arranges the remaining lines alphabetically in reverse order.

Shell meta-characters

Most punctuation symbols have a special meaning in the shell. We call them meta-characters. They may behave differently depending on the context:

• Command line parameters
• Describing file names
• Inside single quotes
• Inside double quotes
• Inside curly braces for variable substitution
• Inside back ticks for command substitution
• Inside dollar prefixed parenthesis for command substitution
• Conditional expressions between square brackets
• Regular expressions

Backslash symbols (\) placed in front of any meta-character deactivate any special treatment and render a literal character instead.

Using variables

Variables store values for later use in the script. By convention, system provided variables are upper case and user defined variable names are lower case.

Assign values to a variable using the equals symbol:

myvar="my_file_name.txt"

Prefix the name with a dollar sign to access the value. Optional curly braces help the interpreter distinguish variables from the surrounding code.

cat ${myvar}

Variable substitution

Using a variable before it has been defined will halt your script with an error. Defined but empty variables are problematic if you expect them to contain a non-null value.

Curly brace substitution can apply checks and alter the outcome before substituting the value. Adding a colon-dash (:-) or equals (:=) after the variable name will choose a specific behaviour.

If a variable contains a meaningful value, that will be returned by the substitution. Undefined and null variables will yield the replacement text instead. This handles the undefined and null variables more gracefully and script will not halt:

echo ${myvar:-replacement_string}
echo ${myvar:=replacement_string}

The colon-dash substitution leaves the source variable unchanged. The colon-equals substitution will assign the replacement_string to the source variable if it was called for.

Process hierarchy and inheritance

Calling a command to action creates a new child process for it. Each process has a unique ID value. List the processes and their PID values with the ps command:

ps -ef

UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 Jan24 ?        00:00:06 /sbin/init
root      1677     1  0 Jan24 ?        00:00:00 /usr/sbin/ssh -D
root     29400  1677  0 09:55 ?        00:00:00 ssh: user [priv]
user     29511 29400  0 09:55 ?        00:00:00 sshd: user@pts/0
user     29515 29511  0 09:55 pts/0    00:00:00 -bash
user     32536 29515  0 10:51 pts/0    00:00:00 ps -ef

Compare the (parent) PPID values with the PID values in the previous line to trace the provenance. The OS commences running with the init process. Observe the ps command in its own process at the end of the list.

Child processes only inherit user-defined variables from the parent if they have been prepared first with an export command. Special system variables are always available. Anything passed as a command line argument is accessible to the child process.

Command substitution

A sub-shell executes commands in a child process and substitutes the STDOUT result in its place. The older technique of using back-tick characters is superseded by using parenthesis with a leading dollar sign. Any valid command can be executed in the sub-shell, including running another script. Here is an example that assigns the contents of a data file to a variable:

myvar=$(cat my_data_file.txt)

This example should be used with great care because it could allow command injection if the source file contains valid commands:

$(cat my_data_file.txt)

Important:
Never take user input data at face value. Always clean and filter it to remove injected commands.

Loops

The loop constructs each determine how to continue looping in a different way. They are functionally similar to other languages but the syntax is slightly different.

Making it runnable

When your script is ready to test, use the chmod command to apply the execute flag to it:

chmod +x your_new_script.sh

Without this, your script will not be allowed to run. Your account may need elevated privileges to do this.