Bootcamp Module 8: Observability Basics

Bootcamp hub · Prev: Module 7

Learning goals

• Separate logs, metrics, and healthchecks without buzzword fog, and know which one to reach for first
• Write one alert that carries an owner, a symptom, and the first command to run — not just a threshold
• Use journalctl and application logs to find a specific failure under mild time pressure

Required posts

• Load average high — the first metric everyone checks, and the one most often misread.
• OOM killer explained — what actually happens when memory runs out, and how to read the evidence afterwards.
• API healthchecks for ops — the contract your automation relies on.
• Parse logs like an admin — grep, awk and journalctl, explained flag by flag.
• systemd unit failed — reading a failure in the journal instead of guessing.

Mental model

Observability is how you ask questions of a system that cannot answer directly. Three signals cover most of what you need, and each answers a different question:

• Metrics show trends over time: CPU, memory, request rate, error rate. They tell you something changed, rarely why.
• Logs tell the story of one event: what request, what error, what line of code. They tell you why, once you know roughly when.
• Healthchecks are contracts with automation: a load balancer or an orchestrator uses them to decide whether to send traffic here. They tell you can this instance serve requests right now, nothing more.

Start an investigation with the signal you understand best, not the most sophisticated one. A metric graph that shows an error-rate spike at 2:14 tells you where to point journalctl; a failing healthcheck tells you this instance has stopped answering correctly, and, depending on how your load balancer or orchestrator is configured, that may or may not have already pulled it out of rotation — check the routing, don’t assume.

Lab

1. Schedule a curl healthcheck with explicit connect and total timeouts, the way REST API healthchecks for ops describes. To actually see the timeout fire rather than an instant “connection refused”, point it at an address that will not answer at all, for example a private IP nothing is listening on (10.255.255.1) or a nc -l listener you deliberately leave silent, and confirm curl gives up at the time you configured instead of hanging.
2. Inject a fake error line into a log file and find it with grep or awk without opening the file in an editor. Time yourself.
3. Write a one-line local alert that fires when disk usage passes 85%, even if it just writes to a file or prints to the terminal. The mechanism matters less than the habit of turning a threshold into an action.

Alert hygiene

Every alert that pages a human needs three things in the message itself: what broke, who is expected to act, and the first command to run. An alert that just says “disk usage high” sends the responder hunting for the hostname, the mount point and the runbook before they can do anything. Put all three in the alert text and the responder starts fixing instead of investigating.

Link to real runbooks instead of writing new ones from memory during an incident. This site already has several: disk, DNS, systemd, nginx. Reusing a tested runbook beats improvising folklore at 2 a.m.

Next → Module 9

FAQ

How long should this module take?

Plan one focused evening for reading plus one for the lab. The log-injection drill is worth doing slowly the first time, so the grep and awk patterns become automatic.

Do I need production access?

No. Use your lab. A fake error line and a local healthcheck teach the same skill without any risk to a real system.