4.2Reliability: healthcheck and restart policies
The essentials in 30 seconds
- restart resurrects, healthcheck proves: the restart policy relaunches the container after a crash or reboot; the periodic health test says whether the service is truly ready (not just started).
- depends_on orders, condition: service_healthy waits for true readiness: without a condition, startup order is only a hope — with a healthcheck on the database, the API only starts on proven healthy.
- Idir's reflex: unless-stopped by default (respects voluntary stops), and Up is not healthy: an Up-but-unhealthy service is a patient to treat, not a running service.
Threaded mini-case — Idir ends Monday-morning crashes
Idir, junior DevOps in Algiers, gets woken up: after a power cut, the API is started but the database wasn't ready — cascading crashes. He adds restart: unless-stopped to each lasting service, a pg_isready healthcheck on the database, and makes the API depend on service_healthy. He then kills the database to verify: automatic relaunch and proven wait. Second milestone of the thread: the stack restarts on its own and only routes to proven healthy.
Sequel of case 4.1 — the image published by Idir now runs autonomously.
1. restart: the resurrection policy
restart: unless-stopped relaunches the container after a crash or a machine reboot, except voluntary stops — the right default for a lasting service. always relaunches even after a manual stop (reserved for critical services), on-failure only relaunches on error. Without a policy, like before Idir's fix, a night crash waits until morning.
Choose according to intent: lasting (unless-stopped), critical that must never stop (always), one-shot that should only restart on failure (on-failure). And avoid always everywhere: it restarts even your voluntary stops.
2. healthcheck: “ready” beats “started”
depends_on orders, healthcheck proves: a periodically tested command (pg_isready for Postgres, an HTTP request for an API) moves the service to healthy. You can then wait for the true ready (condition: service_healthy) instead of hoping the database has finished starting.
The test must target the real service (a request, not the mere presence of the process): that is what makes Idir's “database not ready” startup crashes disappear.
3. Three states, three reactions
starting (we wait), healthy (we route), unhealthy (we alert, we send no more traffic). docker compose ps shows the state: an Up-but-unhealthy service is a patient to treat (logs, diagnosis), not a running service.
Tune the frequency with measure: often enough to detect (10 to 30 seconds), with repeated retries to avoid false positives. A test that is too aggressive spends its time alerting; too rare, it misses outages.
Did you know?
Up is the pulse; healthy is the fitness.
🔎 Concrete example — 🔎 Concrete example — Postgres proving its health
Idir's stack: the API only starts on a proven-ready database.
services:
bdd:
image: postgres:16.4
restart: unless-stopped
healthcheck:
test: ["CMD-SHELL", "pg_isready -U app"]
interval: 10s
retries: 5
api:
build: ./api
depends_on:
bdd:
condition: service_healthyReading: the API only starts on a proven-ready database.
Lesson: the “database not ready” startup crashes disappear.
Section vocabulary
- restart
- Automatic container relaunch policy: no, on-failure, always, unless-stopped.
Simple example: Idir sets unless-stopped: relaunch after crash or reboot, but respects his manual stop.
What it is not: a manual restart at 3 a.m. after each crash.
Reflex: lasting service: declare a restart policy.
- healthcheck
- Periodic test PROVING the REAL HEALTH of the service (a request, not a mere process).
Simple example: pg_isready proves Idir's database really accepts connections.
What it is not: a mere startup order with no verification.
Reflex: critical dependency: test the real service.
- healthy / unhealthy
- Healthy or sick state according to the latest health test — independent of whether the process runs.
Simple example: Idir's database is Up but unhealthy: he diagnoses instead of routing.
What it is not: a mere on or off.
Reflex: Up + unhealthy: treat, don't route.
- condition: service_healthy
- In depends_on: wait until the dependency is proven healthy before starting.
Simple example: Idir's API waits for the healthy database instead of hoping.
What it is not: a launch order with no availability guarantee.
Reflex: ordered startup: require the condition, not just the order.
In practice — In practice — Making your stack autonomous
- Add restart: unless-stopped to each lasting service.
- Add a healthcheck to the database (pg_isready).
- Make the API depend on service_healthy.
- Kill the database (kill): observe relaunch + proven wait.
Ready-to-say script — explaining reliability to the team (30 s)
“This is Idir: each lasting service restarts on its own after a crash, the database proves it is ready with a real test, and the API waits for that proof before starting. No more wake-ups after a power cut: the stack gets back up on its own.”
Say in review: show the database kill test in front of the team.
Copy-paste template — proven database + waiting API (Compose)
services:
bdd:
image: postgres:16.4
restart: unless-stopped
healthcheck:
test: ["CMD-SHELL", "pg_isready -U app"]
interval: 10s
timeout: 5s
retries: 5
api:
build: ./api
restart: unless-stopped
depends_on:
bdd:
condition: service_healthyWeb API to test over HTTP
Like Idir but with a web-request test on the home page: the service is healthy when it truly responds.
24/7 critical service
restart: always assumed and documented: even a manual stop relaunches — to reserve for truly critical cases.
One-shot task
restart: on-failure: only relaunch on error, not after normal success.
False sense of security
depends_on without condition + test targeting the wrong container: fix toward service_healthy and a real test.
✓ Best practices
- unless-stopped by default for lasting services.
- Test the real service (request, not PID).
- Read the health state in compose ps.
✗ Common pitfalls
- always everywhere (restarts even your stops).
- depends_on without condition (false sense).
- Healthcheck testing the wrong container.
Key takeaways
- restart resurrects, healthcheck proves.
- depends_on + condition = true order.
- Up is not healthy.
- unless-stopped: the right default.
48h challenge — apply tomorrow
On your real stack: add unless-stopped + healthcheck + condition, kill the database and observe relaunch and proven wait.
- I can declare a suitable restart policy.
- I can write a healthcheck proving true readiness.
- I can make it wait for service_healthy and read the state.
Frequently asked questions
Healthcheck or simple retry in the app?
Both: retries make the app robust everywhere, the healthcheck makes the stack observable and schedulable. They are allies, not competitors.
What test frequency?
Often enough to detect (10 to 30 seconds), with repeated retries to avoid false positives. A test that is too aggressive alerts non-stop; too rare, it misses outages.
A service is “Up (unhealthy)”. What should you understand?