Part 3: Production Automation: Process Control & Robust Bash Scripting

Welcome to Part 3 of the Linux: Master Series Guide. In Part 2, we navigated the Filesystem Hierarchy Standard and learned how Linux manages multi-user security through permissions and ACLs.

Now, we move into operational automation: managing running workloads and writing robust shell scripts. A runbook step like "clear old logs when disk usage exceeds 90%" is only documentation until it is converted into a script. This part covers process lifecycles, signal handling, and building production-grade, fail-safe Bash scripts.

01: Process Lifecycles and Signal Management

A process is a running instance of a program, identified by a unique Process ID (PID) and tied to a parent process (PPID).

Process States and Monitoring

Inter-Process Signals

Linux uses signals to communicate with running processes. Signals can be sent using the kill or pkill commands:

Bash

# Gracefully request process termination by PID
kill 1234

# Forcefully terminate a process by name
pkill -9 nginx

Job Control: Foreground and Background Execution

• Append & to start a command in the background: python3 long_task.py &

• List background jobs running in the current terminal session: jobs

• Bring a background job to the foreground: fg %1

• Resume a suspended foreground job in the background: bg %1
  

02: Defensive Bash Scripting Standards

A script only earns trust in production when it is idempotent—safe to run once or ten times without duplicating work or erroring on existing state.

The Defensive Header: set -euo pipefail

Place set -euo pipefail at the very top of every production script:

Bash

#!/usr/bin/env bash
set -euo pipefail

• set -e: Exit immediately if any command exits with a non-zero (failure) status.

• set -u: Treat unset variables as errors and exit immediately.

• set -o pipefail: Ensures a pipeline fails if any command in the chain fails, rather than only checking the last command.

Parameter Expansions and Guard Clauses

Avoid bare variable expansions; double-quote all variables to prevent word splitting:

# Default parameter if variable is unset or empty
APP_ENV="${APP_ENV:-production}"

# Abort with an explicit error message if a variable is missing
LOG_DIR="${LOG_DIR:?LOG_DIR environment variable must be set}"

# Strip standard file extensions
filename="backup-2026.tar.gz"
echo "${filename%.tar.gz}"  # Outputs: "backup-2026"

• "${APP_ENV:-production}": Uses default value expansion. If APP_ENV is unset or empty, it evaluates to "production", but does not overwrite the variable globally.

• "${LOG_DIR:?LOG_DIR environment variable must be set}": Uses mandatory check expansion. If LOG_DIR is unset or empty, Bash prints the error message following :? to stderr and immediately aborts script execution.

• "${filename%.tar.gz}": Uses shortest suffix removal (%). It strips .tar.gz from the end of the string stored in $filename.

Resource Locking and Cleanup Traps

Use trap to release locks or delete temporary directories when a script exits or receives an error signal:

#!/usr/bin/env bash
set -euo pipefail

# Create a temporary working directory
TMP_DIR=$(mktemp -d)

# Guarantee cleanup executes regardless of how the script terminates
cleanup() {
    rm -rf "${TMP_DIR}"
}
trap cleanup EXIT ERR

# Prevent duplicate concurrent executions using flock
exec 200>/var/run/myapp-archive.lock
flock -n 200 || { echo "Another instance is already running" >&2; exit 1; }

• TMP_DIR=$(mktemp -d): Creates a uniquely named temporary directory inside /tmp and assigns its absolute path to TMP_DIR.

• trap cleanup EXIT ERR: Registers the cleanup function to run automatically whenever the script exits (EXIT) or encounters a command error (ERR).

• exec 200>/var/run/myapp-archive.lock: Opens file descriptor 200 pointing to the specified lock file path.

• flock -n 200: Attempts to acquire a non-blocking (-n) exclusive lock on file descriptor 200. If another instance holds the lock, it immediately fails, triggering the || fallback block to abort duplicate execution.

03: Text Processing Mastery

Ops work heavily involves filtering logs, API outputs, and configuration files. Three key utilities form the core text-processing pipeline:

1. grep (Pattern Matching)

• grep -E "^(ERROR|WARN)" app.log: Extended regular expressions.

• grep -v "DEBUG" app.log: Invert match (exclude lines matching DEBUG).

• grep -r "DATABASE_URL" /etc/: Search recursively through directories.

• zgrep "FATAL" /var/log/syslog*.gz: Search directly inside rotated, compressed .gz logs.

2. sed (Stream Editor)

• sed -i 's/max_connections=100/max_connections=200/' config.conf: In-place string substitution.

• sed -i '/^$/d' config.conf: Delete blank lines in place.

3. awk (Field Processing & Summaries)

• awk -F: '$3 >= 1000 {print $1}' /etc/passwd: Print usernames with UID >= 1000

• awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -n 10: Generate a top-10 list of IP addresses hitting a web server.

• sort | uniq -c | sort -rn | head -n 10:

  • sort: Sorts lines alphabetically so duplicate entries sit adjacent to one another (required for uniq).

  • uniq -c: Filters out adjacent duplicate lines and prefixes each line with its occurrence count (-c).

  • sort -rn: Sorts the counted list numerically (-n) in reverse descending order (-r) so the highest frequency appears at the top.

  • head -n 10: Truncates the output stream to show only the top 10 results.

04: Parsing Script Arguments (getopts)

getopts is the POSIX standard way to handle CLI flags safely inside scripts:

#!/usr/bin/env bash
set -euo pipefail

usage() { echo "Usage: $0 -d <directory> -n <days>" >&2; exit 2; }

target_dir=""
days=30

while getopts ":d:n:" opt; do
  case "${opt}" in
    d) target_dir="${OPTARG}" ;;
    n) days="${OPTARG}" ;;
    \?) echo "Invalid option: -${OPTARG}" >&2; usage ;;
    :) echo "Option -${OPTARG} requires an argument." >&2; usage ;;
  esac
done
shift $((OPTIND - 1))

if [[ -z "${target_dir}" ]]; then
    usage
fi

echo "Cleaning logs older than ${days} days in ${target_dir}..."

• ":d:n:": The option specification string:

  • Leading :: Enables silent error mode so getopts does not print default system error messages, letting you handle invalid flags in the \?) case.

  • d: & n:: The colon : following a letter indicates that the flag requires a value argument (e.g., -d /var/log).

• ${OPTARG}: A built-in getopts variable holding the value passed to a flag (e.g., /var/log for -d).

• shift $((OPTIND - 1)): OPTIND tracks the index of the next argument to be processed. Shifting by OPTIND - 1 removes processed option flags (-d, -n) from positional parameters ($1, $2), leaving only raw trailing arguments.

05: Task Scheduling (Cron & At)

Automation requires running tasks on a schedule or delaying execution.

Scheduling Recurring Tasks with Cron

Edit user cron jobs using crontab -e. The syntax consists of five time fields followed by the command:

Plaintext

# .---------------- minute (0 - 59)
# |  .------------- hour (0 - 23)
# |  |  .---------- day of month (1 - 31)
# |  |  |  .------- month (1 - 12)
# |  |  |  |  .---- day of week (0 - 6) (Sunday=0)
# *  *  *  *  *  command to be executed

Bash

# Run log archiving every night at 02:30 AM
30 2 * * * /usr/local/bin/archive-logs.sh -d /var/log/myapp >> /var/log/cron.log 2>&1

One-Off Deferred Tasks with at

For one-off scheduled actions where you do not want to create a permanent cron entry:

Bash

# Schedule a system maintenance check in 15 minutes
echo "/usr/local/bin/health-check.sh" | at now + 15 minutes

# View pending deferred jobs
atq

# Cancel a queued job by ID
atrm 3

Mastering process lifecycles and writing defensive, idempotent scripts lays the groundwork for system administration. In Part 4, we will take service supervision to the next level by exploring systemd, custom units, timers, and journald.