Bootcamp Module 4: Services & Disks

Production pain clusters here: services that won’t stay up, and disks that fill at the worst time. Learn the triage loops until they’re muscle memory.

Bootcamp hub · Prev: Module 3

Learning goals

  • Read systemctl status and journalctl -u
  • Explain a simple unit file’s ExecStart/User/Restart
  • Triage disk full with df, du, lsof, inodes

Required posts

Lab

  1. Create a tiny systemd user or system service that runs a script.
  2. Break ExecStart path on purpose; fix using the journal.
  3. Fill a lab filesystem safely (big file in a playground mount) and practice df/du.

Next: Module 5 — Networking →

systemd mental model

Units describe desired state. The manager starts/stops processes, collects logs, and restarts per policy. Your job is to make ExecStart true, environment complete, and dependencies honest.

Disk mental model

Capacity (bytes) and inodes (file counts) both matter. Deleted-open files confuse beginners. Separate mounts isolate blast radius — Docker on root is a common regret.

Combined lab scenario

  1. Create a service that writes a growing log file.
  2. Let it grow; watch df.
  3. Simulate rotate/delete without restart; observe deleted-open behavior with lsof.
  4. Restart the service; confirm space returns.
  5. Break the unit path; repair with journalctl.

Reading list

Exit criteria

You can explain to a junior why df and du disagree, and you can capture a failed unit’s first real error line without guessing.

journalctl fluency drills

journalctl -u ssh -b
journalctl --since "1 hour ago" -p err
journalctl -u myapp -o short-iso

Practice until flags are reflexes. During outages you won’t remember blog posts — you’ll remember drills.

Disk fill simulation ethics

Only fill disks on lab VMs. On shared cloud accounts, set quotas and budgets. Document how you recovered; that write-up is portfolio gold.

Link forward

High load with idle CPU often ties back to disks — peek at load average explained after this module.

Extended lab: the Friday night dual failure

Simulate a unit pointing at a missing binary while also filling a playground filesystem. Practice ordering: which do you triage first when both alerts fire? (Usually: prove disk isn’t lying, then fix the unit — but capture evidence for both.)

Posts: systemd, disk full.

Unit file reading practice

Open three vendor units on your lab (systemctl cat cron, ssh, nginx if present). Note Type=, User=, and Restart= lines. Guess what happens on crash before reading docs. Then confirm.

Disk alerts you should want

  • 80% and 90% bytes used
  • inode used percent
  • prediction / growth rate if available

If your lab monitoring is just df in cron mailing you, that’s already better than nothing — Module 6 will formalize it.

systemctl cheat sheet

systemctl status NAME
systemctl restart NAME
systemctl enable --now NAME
systemctl daemon-reload
systemctl list-units --failed

Know these cold. Then add journalctl filters. Together they cover most service incidents.

Logrotate awareness

Peek at /etc/logrotate.d/ on your lab. Understand that apps must reopen logs after rotation or you get deleted-open files — the star of the disk-full post. Config without signal handling is a time bomb.

When not to delete

Unknown large directories under /var/lib may be databases. If you didn’t put it there and it looks important, snapshot first. Deletion is irreversible; expansion is often reversible.

One more drill: failed units list

Run systemctl list-units --failed daily on your lab for a week. Intentionally fail a unit once and confirm it appears. Clearing with reset-failed after a fix is part of the hygiene — same as closing tickets.

Keep a personal snippet file of the systemctl and df commands you actually used — muscle memory beats bookmarks during pages.