Bash Logging and Error Handling for 2 AM Ops

A script that fails silently is worse than one that crashes. The crash tells you something is wrong. The silent one lets you find out a week later, when the backup you assumed was running turns out to be empty.

This guide is the safety net I would put under any Bash script that runs unattended: strict mode, a log function, cleanup traps, meaningful exit codes, retries that do not hammer a struggling service, and a few habits that keep secrets out of logs. Each snippet is explained, including the places where it does not do what people expect.

Strict mode, and what it does not catch

set -Eeuo pipefail

• -e exits the script when a command fails.
• -u treats use of an unset variable as an error, which catches typos like $BACKUP_DRI before they turn into rm -rf /$BACKUP_DIR-style disasters.
• -o pipefail makes a pipeline fail if any stage fails, not only the last one. Without it, mysqldump db | gzip > out.gz reports success even when the dump failed.
• -E lets the ERR trap (covered below) fire inside functions and subshells too.

set -e is not a guarantee. It is ignored for commands in an if condition, on the left side of && or ||, and in any function called from those places. When a failure is expected, handle it explicitly so the intent is visible:

if ! grep -q pattern "$file"; then
  log "pattern missing, continuing"
fi

# only when the command is genuinely optional:
grep -q pattern "$file" || true

Use || true sparingly. Every one is a place where a real failure will be swallowed.

A log function that helps at 2 a.m.

LOG_FILE=/var/log/myapp/job.log

log()  { printf '%s INFO  %s\n' "$(date -Is)" "$*" | tee -a "$LOG_FILE" >&2; }
warn() { printf '%s WARN  %s\n' "$(date -Is)" "$*" | tee -a "$LOG_FILE" >&2; }
err()  { printf '%s ERROR %s\n' "$(date -Is)" "$*" | tee -a "$LOG_FILE" >&2; }

date -Is prints an ISO 8601 timestamp such as 2026-09-22T02:15:03+05:30. It includes the time zone and pastes cleanly into tickets. It sorts chronologically as long as every line uses the same offset, so on servers that span time zones or observe daylight saving, use date -u -Is to log in UTC. The level word (INFO, WARN, ERROR) lets you filter with grep ERROR later. tee -a writes to the log file and to the screen, so interactive runs and scheduled runs produce the same record, and >&2 sends the screen copy to stderr so it does not pollute stdout if the script’s output is piped somewhere.

Create the log file with sensible permissions once, and rotate it, or you will meet the disk-full problem again:

install -m 640 -o root -g adm /dev/null /var/log/myapp/job.log

Logs often contain hostnames, URLs and usernames, so 640 with a group such as adm is a better default than world-readable. A wrong owner shows up as a “permission denied” halfway through a run; see Linux users, groups and permissions.

Traps: clean up, and leave a breadcrumb

tmpdir=
cleanup() {
  local rc=$?
  trap - EXIT
  if [[ -n "${tmpdir:-}" && -d "$tmpdir" ]]; then rm -rf "$tmpdir" || true; fi
  if [[ $rc -ne 0 ]]; then err "aborted rc=$rc" || true; fi
  exit "$rc"
}
trap cleanup EXIT
tmpdir=$(mktemp -d)

An EXIT trap runs when the script ends: on success, on failure, and on most signals. It cannot run if the process is killed with SIGKILL, or if the script hands over to another program with exec. local rc=$? must be the first line, because any other command would overwrite $?. The cleanup removes the temporary directory, logs a message only when something went wrong, and re-exits with the original code so the caller still sees the failure. The || true guards matter: under set -e, a failing rm or log write inside the trap would otherwise abort the cleanup and replace the original exit status. Setting tmpdir= before defining the trap means cleanup is safe even if the script dies before mktemp runs.

To record where a failure happened, add an ERR trap:

on_err() { err "failed at line $1: $BASH_COMMAND"; }
trap 'on_err $LINENO' ERR

$LINENO is the line number and $BASH_COMMAND is the command that failed. Together they turn “the script died” into “line 47: rsync exited non-zero”, which is usually enough to start fixing. This relies on the -E flag from earlier.

Exit codes are your monitoring API

Exit status 0 means success and anything else means failure. If a wrapper or monitoring system needs to tell failures apart, agree on codes: for example 2 for bad usage, 3 for a missing dependency, 4 for a failed healthcheck. Document them at the top of the script.

Be careful about who actually reads the exit code. systemd does: a non-zero exit marks the unit as failed, which you can alert on. Cron does not act on it. By default cron only emails whatever the job printed. For cron, either make sure failures print something, or use the freshness-file approach in scheduling and wrapping cron jobs. If a systemd unit fails, this triage guide covers the next steps.

Retries that do not make an outage worse

retry() {
  local n=0 max=3
  until "$@"; do
    n=$((n + 1))
    [[ $n -ge $max ]] && return 1
    sleep $((n * 2))
  done
}

retry curl -fsS --max-time 10 "$URL"

"$@" runs whatever command you passed, keeping its arguments intact. The wait grows with each attempt (2 seconds, then 4), and after three tries the function gives up and returns failure instead of looping forever. The command in the until condition is exempt from set -e, which is exactly what you want here. Retries help with brief network blips. Unlimited retries only hide a real outage and add load to a service that is already struggling.

A reusable healthcheck function

check_health() {
  local code
  code=$(curl -sS -o /dev/null -w '%{http_code}' --connect-timeout 2 --max-time 5 "$1") || code=000
  [[ "$code" == "200" ]] || { err "health $1 returned $code"; return 4; }
}

The -w '%{http_code}' option prints only the status code and -o /dev/null discards the body. If curl cannot get a response at all it exits non-zero, and the || code=000 fallback makes sure you still have a value to log. (Assigning in the fallback, rather than using || echo 000 inside the substitution, avoids ending up with 000000, because curl already prints 000 through -w when no response arrives.) The function returns 4, matching the exit-code table above. For designing the endpoint itself, see REST API healthchecks for ops.

Long jobs and signals

abort=0
on_term() { warn "terminated, finishing current step"; abort=1; }
trap on_term TERM INT

for step in dump compress upload; do
  [[ $abort -eq 1 ]] && { err "aborted before $step"; exit 130; }
  "run_$step"
done
[[ $abort -eq 1 ]] && { err "aborted during final step"; exit 130; }

A backup that gets killed halfway and still reports success is a lie waiting to be discovered. This pattern catches SIGTERM and SIGINT, sets a flag, and stops at the next safe point with a non-zero exit. One thing to know: Bash defers its trap until the current foreground command returns. A SIGTERM sent only to the script therefore waits for that command to finish, while a Ctrl-C from the terminal also reaches the child process directly and usually stops it at once. If you need a guaranteed hard stop, wrap the long command in timeout -k 30s.

Keep secrets out of the logs

Log that an auth header was present, never the header itself. Do not dump the environment in debug mode. And do not leave set -x on in production: it prints every command with its expanded arguments, tokens included. Gate it behind an explicit flag such as DEBUG=1, and use it in the lab, not on a live system.

Correlating script logs with the journal

When a script restarts a service, log the unit name and a run id (for example run_id="$(date +%s)-$$") with every line. If you also send your script’s messages to the journal, for example with logger -t myapp-$run_id, you can grep that id in your log and in journalctl and read the whole incident in order. The id does not appear in the service’s own logs by itself, so only rely on it where you actually wrote it. This is especially valuable when several scheduled jobs touch the same service.

Practice drills

1. Write a script that fails on purpose and confirm the ERR trap logs the line and command.
2. Reference an unset variable and watch set -u stop the script.
3. Run it as a systemd oneshot service and read the result in the journal.
4. Fill a log until a disk-usage warning fires, then add rotation.

Common questions

Is set -e enough on its own?

No. Combine it with pipefail, traps, and explicit handling around commands that are allowed to fail.

Should I log to a file or to syslog?

Either works. logger -t myapp "message" sends to the local syslog interface, which on most systemd distributions ends up in the journal as well. Pick one primary destination so you always know where to look.

Where do I go next?

Read Bash scripting for sysadmins for the foundations, and parsing logs with grep, awk and sed for making use of what you now log.

Sources

• Bash manual: The Set Builtin, for -e, -u, pipefail and their exceptions
• Bash manual: trap
• crontab(5), for how cron handles output
• systemd.service, for exit status handling