Advanced Bash Scripting Guide, Cheat Sheet and Use Cases

Advanced Bash Scripting

In addition to being a simple scripting tool, Bash also offers numerous advanced features and techniques that let you build powerful, robust automation scripts. From intricate parameter expansions to signal handling and co-processing, these capabilities make Bash highly adaptable to complex workflow scenarios. This guide will explore some of these advanced Bash features, demonstrating how to employ them in real-world use cases. By the end, you should have a strong grasp of sophisticated scripting methods that can streamline development, system administration, and DevOps tasks.

Advanced Bash scripting encompasses a wide range of features—from intricate parameter expansions and associative arrays to co-processes and robust error handling. Mastering these tools enables you to create sophisticated scripts capable of handling parallel tasks, responding to signals, and seamlessly integrating with other system processes.

By understanding and combining these advanced techniques, you can build powerful automation solutions, streamline workflows, and effectively manage complex operations directly from the command line.

If you’re looking for a more beginners guide and cheat sheet instead, follow this link: Bash scripting guide and cheat sheet.

Advanced Parameter Expansion

Bash provides various forms of parameter expansion to manipulate strings and variables in-place, without calling external utilities like sed or awk.

Substring Extraction

VARIABLE="HelloWorld"
echo "${VARIABLE:0:5}"    # Outputs 'Hello'
echo "${VARIABLE:5}"      # Outputs 'World'

• ${VAR:offset:length}: Extract substring from VAR at position offset of length length.
• ${VAR:offset}: Extract substring from VAR starting at position offset to the end.

Pattern Removal

FILEPATH="/home/user/docs/report.txt"
# Remove shortest match of pattern from front
echo "${FILEPATH#*/}"    # outputs 'home/user/docs/report.txt'
# Remove longest match of pattern from front
echo "${FILEPATH##*/}"   # outputs 'report.txt'
# Remove shortest match of pattern from end
echo "${FILEPATH%/*}"    # outputs '/home/user/docs'
# Remove longest match of pattern from end
echo "${FILEPATH%%/*}"   # outputs ''

Search and Replace

TEXT="The quick gray fox jumps over the lazy dog"
# Replace first occurrence
echo "${TEXT/fox/cat}"
# Replace all occurrences
echo "${TEXT//o/O}"

Indirect Expansion

Indirect expansion allows you to reference a variable whose name is stored in another variable:

VAR="HELLO"
REF="VAR"
echo "${!REF}"  # Outputs 'HELLO'

Process Substitution

Process substitution uses <( ) or >( ) to treat the output or input of a list of commands like a file name. This is handy when dealing with commands that expect file inputs:

# Compare two directory listings without creating temporary files
diff <(ls /path/dir1) <(ls /path/dir2)

In this example, both ls /path/dir1 and ls /path/dir2 outputs are made available to the diff command as if they were files.

Co-processes

A co-process runs asynchronously, allowing two-way communication with the invoking shell. You can create a co-process with |&.

# Create a co-process
coproc my_coproc { bc -l; }
# Send commands to the co-process
echo "10+5" >&"${my_coproc[1]}"
# Read result from the co-process
read RESULT <&"${my_coproc[0]}"
echo "Result: $RESULT"   # Outputs '15'

Co-processes are especially useful for interactive processes or large data streams that need to be handled in parallel.

Traps and Signal Handling

trap

The trap built-in intercepts signals (e.g., SIGINT, SIGTERM) or errors, letting you run cleanup or logging code before the script exits.

#!/usr/bin/env bash
cleanup() {
echo "Cleanup and exit."
# Additional cleanup tasks here
}
#Trap Ctrl-C (SIGINT) or script exit (EXIT)
trap cleanup INT EXIT

echo "Script running. Press Ctrl-C to trigger trap."
sleep 30

Common signals:

• INT (2): Interrupt signal (Ctrl-C).
• TERM (15): Termination request.
• EXIT: Triggered when the script exits, regardless of the reason.

set Built-in for Error Handling

set -euo pipefail
# -e: exit on any command returning non-zero
#-u: treat unset variables as errors
#-o pipefail: pipeline returns the exit code of the first failed command

Advanced Arrays and Indirection

Associative Arrays

As introduced in Bash 4.0, associative arrays map string keys to values:

declare -A my_map
my_map["key1"]="value1"
my_map["key2"]="value2"

for key in "${!my_map[@]}"; do
echo "$key => ${my_map[$key]}"
done

Namerefs (Indirect References)

Namerefs let you create a “reference” to a variable name:

#!/usr/bin/env bash
declare -A config
config["server"]="example.com"
config["port"]="8080"

config_ref() {
# $1 = name of the associative array
local -n arr=$1
echo "Server: ${arr["server"]}"
echo "Port:   ${arr["port"]}"
}

config_ref config

In this example, local -n arr=$1 references the associative array passed as $1, allowing direct manipulation inside the function without explicitly returning data.

Advanced Argument Parsing with getopts

The getopts built-in can parse complex arguments (including those with values) in a loop:

#!/usr/bin/env bash
usage() {
echo "Usage: $0 [-a] [-b arg] [-c]"
exit 1
}

FLAG_A=0
FLAG_B=""
FLAG_C=0

while getopts ":a b: c" opt; do
case "$opt" in
a) FLAG_A=1 ;;
b) FLAG_B="$OPTARG" ;;
c) FLAG_C=1 ;;
*) usage ;;
esac
done

if [ $FLAG_A -eq 1 ]; then
echo "Option -a is set."
fi

if [ -n "$FLAG_B" ]; then
echo "Option -b has value: $FLAG_B"
fi

if [ $FLAG_C -eq 1 ]; then
echo "Option -c is set."
fi

Advanced I/O Redirection and Named Pipes

Here-Strings and Here-Documents

• Here-string (<<<): Feeds a single string into a command’s stdin.
• Here-doc (<<EOF): Feeds multiple lines into a command.

cat <<EOF > /tmp/hello.txt
Hello
World
EOF

Named Pipes (FIFOs)

Named pipes allow separate processes to communicate by reading/writing to the same pipe file.

mkfifo /tmp/myfifo
#Process 1: Write to FIFO
echo "Hello from process 1" > /tmp/myfifo &
#Process 2: Read from FIFO
cat /tmp/myfifo

Parallelizing Tasks in Bash

Background Jobs

Use & to run a command in the background:

long_running_command &
fast_command
wait

• wait: Waits for background jobs to complete.

xargs -P

Combine xargs with -P to process multiple lines in parallel:

#Example: compress multiple files in parallel
ls *.log | xargs -P 4 -n 1 gzip

Command Substitution

Command substitution in Bash allows you to capture the output of a command and store it in a variable or use it directly within another command. For the full guide on Command Substitution in Bash, follow this link: Command Substitution in Bash.

Debugging Bash Scripts

Effective debugging and troubleshooting is a critical skill for every Bash script author. By combining built-in features like set -x, trap, and PS4 with external tools such as shellcheck, you can quickly isolate and resolve script issues. Implementing robust error handling, thorough logging, and best practices like incremental development will further ensure that your scripts run smoothly and remain maintainable in the long term (which you’ll always be your number one goal: stability and performance!).

You can access the full debugging and troubleshooting your bash scripts following this link: Debugging and Troubleshooting Your Bash Scripts.

Scripting Best Practices

1. Use Strict Mode: set -euo pipefail and handle potential errors.
2. Validate Inputs: Check for required arguments or files before processing.
3. Modularize Code: Organize reusable parts into functions.
4. Test on Multiple Shells: If portability is a concern, test with dash, sh, or older Bash versions.
5. Document Code: Provide usage messages and comment complex sections.