Schedule and Wrap Cron Jobs Safely (Bash Ops)

Most cron problems are not cron problems. The job worked fine in your terminal, it fails at 02:15 with no output, and you spend the morning guessing. The cause is nearly always one of five things: a starved environment, the wrong working directory, two runs overlapping, a job that hangs forever, or a failure nobody was told about.

This guide shows one wrapper pattern that closes all five gaps, explains every line of it, and finishes with how to prove the job is still alive weeks later. If your job is already misbehaving, start with cron job not running, then come back here to make sure it cannot happen again.

Why a job that works in your shell fails under cron

Cron does not start your login shell. It runs the command with /bin/sh, a minimal PATH (often just /usr/bin:/bin), no .bashrc, no exported variables from your session, and your home directory as the working directory. Anything your script relies on implicitly, such as a binary in /usr/local/bin, a relative path, or an environment variable, quietly disappears.

Two more details catch people out. A bare % in a crontab line is treated as a newline, so a command like date +%F breaks unless you escape it as \%. And cron does not act on exit codes. By default it only emails whatever the job printed, so a job that fails silently produces silence. (Debian’s cron can log non-zero exit statuses if you start it with -L 4, but that is a log line, not an alert.)

The wrapper: one script that fixes the environment, overlap and hangs

Instead of putting the real command in the crontab, point cron at a small wrapper. The crontab stays boring and every safeguard lives in one reviewed file.

#!/usr/bin/env bash
set -euo pipefail

export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
cd /opt/myapp

exec /usr/bin/flock -n -E 99 /run/myapp-job.lock \
  /usr/bin/timeout -k 30s 30m /opt/myapp/bin/job.sh \
  >>/var/log/myapp/job.log 2>&1

Here is what each part does and why it is there:

• set -euo pipefail makes the wrapper itself strict: stop on errors (-e), treat unset variables as errors (-u), and fail a pipeline if any stage fails (pipefail). The full reasoning is in Bash logging and error handling.
• export PATH=... sets the search path explicitly, so the job behaves the same under cron, systemd and your terminal.
• cd /opt/myapp fixes the working directory. Because of set -e, a failed cd stops the script instead of running the job in the wrong place.
• flock -n takes a lock on /run/myapp-job.lock and, with -n (non-blocking), gives up immediately if another run holds it. -E 99 makes that “already running” case exit with code 99 instead of 1, so you can tell “skipped because of overlap” apart from a real failure, as long as your job never uses 99 itself (flock passes through the job’s own exit status otherwise).
• timeout -k 30s 30m sends SIGTERM if the job is still running after 30 minutes, and if it is still alive 30 seconds later, sends SIGKILL. A plain timeout 30m can wait forever on a job that ignores SIGTERM. The exit status is 124 when the timeout fired, or 137 if SIGKILL was needed. Pick a limit comfortably above the job’s normal runtime.
• >>/var/log/myapp/job.log 2>&1 appends both stdout and stderr to a log file. Without it, output goes to cron’s mail, which on most servers nobody reads.
• exec replaces the wrapper process with flock, so there is no idle shell hanging around holding resources.

A useful property of flock: the kernel releases the lock when every process holding that file descriptor has exited, even if the job crashed or was killed. You do not get stale lock files the way you do with hand-rolled touch /tmp/job.lock schemes. One caveat: a background child that inherited the descriptor can keep the lock alive after the parent dies.

The crontab entry

[email protected]
15 2 * * * root /usr/local/sbin/myapp-job-wrap.sh

This goes in a file under /etc/cron.d/, for example /etc/cron.d/myapp. Files in that directory have an extra sixth field, the user to run as (root here, but prefer a dedicated service account). On Debian and Ubuntu the filename may only contain letters, digits, underscores and hyphens, or cron silently ignores it. The file must end with a newline and must be owned by root and not writable by group or others, or Debian’s cron will refuse to load it.

Always use absolute paths for the script. MAILTO only helps if the server has a working mail transfer agent. Because the wrapper redirects everything, including the error messages from flock and timeout, into the log file, cron has almost nothing left to email. Treat the log and the freshness check below as your real alerting, and keep MAILTO as a backstop for problems before the redirect, such as a missing script.

Proving the job is still alive: freshness checks

A wrapper stops silent hangs and overlaps, but it cannot tell you the job stopped being scheduled at all, for example after someone edits the wrong cron file or the cron service is down. The fix is to monitor the outcome, not the schedule. Make the last line of job.sh record success:

touch /var/lib/myapp/job.ok

This only means something if job.sh itself starts with set -euo pipefail, so it can only reach the touch when everything before it succeeded. The wrapper’s strict mode does not carry into the job.

Then have your monitoring check that this file is recent. For a daily job, allow a little slack over 24 hours:

test -n "$(find /var/lib/myapp/job.ok -mmin -1500)" || echo "myapp job stale"

find -mmin -1500 matches files modified in the last 1,500 minutes (25 hours). If nothing matches, the command prints the warning, and your monitoring agent can turn that into an alert. This one check catches a stopped cron daemon, a deleted crontab, a permissions change and a hung job, all with the same alert.

Cron or a systemd timer?

Both are fine. Cron is everywhere and simple. A systemd timer gives you better logging through the journal, dependency ordering, and Persistent=true, which runs a missed job when the machine comes back from being off. Cron will simply skip it. If your fleet already uses systemd units heavily, a timer plus a oneshot service is worth it; the basics are in systemd for new admins. To check a timer expression before you trust it, run systemd-analyze calendar "*-*-* 02:15:00".

Whichever you pick, keep the wrapper. Its overlap lock, timeout and logging work identically under both.

Practice run: break it on purpose

Do this on a test box before you trust the pattern in production.

1. Schedule the wrapper every 5 minutes, wait for two runs, and confirm lines appear in the log.
2. Remove the PATH export and call a binary from /usr/local/bin in the job. Watch it fail, then restore the line.
3. Make the job sleep 400 so runs overlap. Confirm the second run exits with 99 and does not start a second copy.
4. Run timeout 10s sleep 60; echo $? by hand and confirm it prints 124. Then make a test job ignore SIGTERM (trap '' TERM) and see why the -k option matters.

Seeing each safeguard trigger once is what makes you trust it at 2 a.m.

If you want to practise this with guided labs, the automation and observability modules of the bootcamp (Module 6) and Module 8 build on exactly this pattern.

Common questions

Should cron jobs run as root?

Only if the job truly needs it. A dedicated user limits the damage if the script has a bug, and it makes file ownership in your logs and output directories predictable.

Where does the cron output go if I redirect it?

To the log file you chose. Cron only sends mail for output that reaches it, so a fully redirected job never emails. Use the freshness check or an alert on the log for failures.

What if the job legitimately takes longer than the interval?

Then the interval is too short or the job needs redesigning. The lock will skip overlapping runs safely, but if you see exit code 99 often, that is a signal to investigate, not to ignore.

Sources

• flock(1) manual page, for -n and -E.
• timeout(1) manual page, for the exit status 124 behaviour.
• crontab(5) manual page, for the environment and % handling.
• systemd.timer documentation, for Persistent=.

Related: A worked example with flock and a log line: replacing WP-Cron with a real 5-minute cron job for a WordPress container.