SSL Certificate Expired: Emergency Fix + How to Never Miss Renewal

Users: “the site is broken.” Reality: TLS certificate expired at midnight UTC and Chrome decided to educate everyone. No malware. No “hack.” Just a calendar you ignored.

I’ve done the walk of shame. Here’s the emergency lane, then the boring system so you don’t repeat it. Pair with nginx 502 triage if the site is down for other reasons — expired cert and dead upstream are different animals wearing the same “site down” costume.

Confirm it’s actually expiry

echo | openssl s_client -servername example.com -connect example.com:443 2>/dev/null \
  | openssl x509 -noout -dates -subject -issuer

# or from another box:
curl -vI https://example.com 2>&1 | grep -i expire

Look at notAfter. If that date is in the past, you’re done diagnosing the “why browsers yell.” If the cert is fine but users still see warnings, you might have the wrong vhost cert, an incomplete chain, or HSTS memories — different ticket.

Emergency fix: Let’s Encrypt / certbot

Most of my boxes renew with certbot or an ACME client behind it.

sudo certbot certificates
sudo certbot renew --force-renewal
# or for one name:
sudo certbot certonly --nginx -d example.com -d www.example.com
sudo systemctl reload nginx   # or apache2 / caddy

If renew fails, read the error. Common ones:

• HTTP-01 challenge blocked — port 80 closed in cloud security group / firewall
• DNS-01 — API token expired for the DNS provider
• rate limits — you thrashed certbot in panic; wait or use staging carefully
• webroot path wrong after someone “cleaned” /var/www

Port 80 must reach the challenge for HTTP-01. People “harden” by closing 80 and then wonder why renew dies. Redirect HTTP→HTTPS is fine; blocking 80 entirely is how certificates rot.

Emergency fix: bought cert / PFX / corporate CA

You got a zip from the vendor. You need the leaf, the intermediates, and the key — and nginx wants a fullchain.

# nginx pattern
ssl_certificate     /etc/ssl/example/fullchain.pem;
ssl_certificate_key /etc/ssl/example/privkey.pem;

sudo nginx -t && sudo systemctl reload nginx

Missing intermediate = “works on some Android, fails on others.” Always test with SSL Labs or at least openssl s_client and check the chain length. Don’t paste private keys into Slack. Yes, someone always does.

Reload vs restart

Reload the proxy so new files are picked up without dropping every connection harder than needed. If the unit is failed, you’re in systemd failed start land first.

sudo nginx -t && sudo systemctl reload nginx
sudo apachectl configtest && sudo systemctl reload apache2

CDN / load balancer in front

Cert might be fine on the origin and expired on CloudFront / Cloudflare / ALB — or the opposite. Check what the browser hits. Terminate TLS where you think you terminate TLS. Teams can lose an hour renewing the origin certificate while Cloudflare still serves an old Universal SSL certificate, or a custom cert nobody owns.

How you never miss renewal again

Hope is not a strategy. Pick at least two of these:

1. ACME auto-renew — certbot timer / systemd timer actually enabled
2. Calendar + monitoring — alert when notAfter < 21 days
3. Synthetic check — external probe on HTTPS that fails on cert errors

systemctl list-timers | grep -i cert
sudo certbot renew --dry-run

Dry-run monthly. Put a note in the runbook. If renewal is DNS-01 via API, treat that API token like production creds — rotation day is also renewal-risk day.

Cron myths

Someone added a cron that “runs certbot” and never checked mail. Cron silent failure is a genre. See cron job not running. Prefer the distro’s certbot.timer. Log renewals somewhere you glance at.

Example: the staging cert on prod

A tempting “fix” for an expired certificate is to copy files from a staging box. If staging used a Let’s Encrypt staging-CA certificate, browsers still reject it because that CA is not trusted. Always check issuer, not only dates. openssl x509 -noout -issuer is your friend.

Preventing the next expiry

Process and tooling so this does not happen again.

While you’re in the TLS neighborhood

• Turn off ancient protocols if you haven’t (ops hygiene)
• Don’t expire and also open RDP to the world as a “backup plan” — RDP can’t connect is a separate headache
• SSH keys expire in people’s heads too — permission denied publickey

Checklist card

1. Confirm notAfter in the past (or wrong cert on the name)
2. Renew or replace cert + full chain
3. nginx -t / configtest → reload
4. Verify from outside: openssl/curl, not only localhost
5. Fix renew path (timer, port 80, DNS API)
6. Add expiry monitoring

If after reload you get 502s, upstream may have crashed during the scramble — back to the 502 checklist. Disk full can also break ACME writes: disk full triage.

Staging, wildcards, and multi-name certs

Wildcards (*.example.com) need DNS-01. HTTP-01 won’t cut it. If your renew job still tries webroot on a wildcard, it fails forever while apex renews fine — confusing dashboards.

SAN certs covering apex + www + api: renew the name set you actually serve. Dropping a name from the certbot line “to make it succeed” is how api.example.com starts showing the wrong cert next month.

echo | openssl s_client -servername api.example.com -connect api.example.com:443 2>/dev/null \
  | openssl x509 -noout -ext subjectAltName

HSTS makes expiry feel worse

With HSTS, browsers refuse easy “click through” on expiry. Good for security, painful during outages. That’s another reason to alert at day 21, not day 0. If you ever set a year-long HSTS max-age on a lab domain you reuse for demos — enjoy the lesson.

File permissions after renew

certbot usually handles this. Homegrown scripts sometimes write privkey.pem as 644 or put files in a directory nginx can’t traverse. After a manual copy:

sudo ls -l /etc/letsencrypt/live/example.com/
# keys should not be world-readable
sudo namei -l /etc/ssl/example/privkey.pem

SELinux contexts on custom paths can also block nginx from reading the new cert — SELinux permission denied.

Post-renew smoke test script

I keep a tiny check on the box (and an external check):

#!/usr/bin/env bash
set -euo pipefail
HOST="${1:?host}"
end=$(echo | openssl s_client -servername "$HOST" -connect "$HOST:443" 2>/dev/null \
  | openssl x509 -noout -enddate | cut -d= -f2)
end_epoch=$(date -d "$end" +%s)
now=$(date +%s)
days=$(( (end_epoch - now) / 86400 ))
echo "$HOST expires in $days days ($end)"
test "$days" -gt 14

Cron it. If cron is flaky, fix that habit: cron job not running. Alert when the script exits non-zero.

Team process (the unsexy part)

• One owner for ACME accounts and DNS API tokens
• Document every hostname → where TLS terminates (CDN vs origin)
• Renewal dry-run after any nginx/apache major change
• Blameless note in the incident channel when expiry happens — then add monitoring the same day, not “next sprint”

If you’re growing into automation, drop certbot checks into the same ops practice as Ansible baseline roles. Humans forget calendars; timers don’t — until someone disables them.

FAQ

Users still see expiry after renew?

CDN cache of the error page, wrong vhost, old worker, or they’re hitting a different hostname (www vs apex). Check the cert presented for the exact name they type.

Do I need to restart the app behind nginx?

Usually no — TLS terminates at the proxy. App restart only if the app itself serves TLS.

Can I extend a cert manually?

No usefully. Issue a new one. That’s how public CAs work.

What about internal CAs?

Same ops problem, different issuer. Monitor notAfter, automate issuance, document the trust store on clients. Internal doesn’t mean immortal.

Where does this sit on a learning path?

SysAdmin→DevOps roadmap and courses. TLS is bread-and-butter ops, not a specialty badge.

Sources

• Let’s Encrypt documentation, for certificate lifetimes and renewal
• openssl-x509(1) manual page, for checking notAfter