Skip to content

Health checks and supervision ​

Shipwick checks replicas while they are deployed and for as long as they run, over HTTP, over TCP or with a command inside the container, and restarts those that exit or stop answering. This page describes the three kinds of health check, the supervisor's restart and backoff rules, reconciliation of missing containers, the notifications and alerts an outage produces, and the application statuses that result.

The health check ​

A check is one of three kinds, decided by which health key is set. interval, timeout and retries mean the same for each.

yaml
health:
  path: /health          # HTTP GET, 2xx is healthy
  interval: 10s
  timeout: 3s
  retries: 3
yaml
health:
  tcp: 5432              # a TCP connection to this container port is accepted
yaml
health:
  command: ["pg_isready", "-U", "postgres"]   # exit 0 inside the replica
KindOne probe isFor
pathGET http://<container ip>:<port><path>Anything that speaks HTTP. Needs port.
tcpA TCP connection to <container ip>:<tcp>, closed once accepted; nothing is sentDatabases and queues that would log a protocol error on a stray HTTP request. It says the process is listening, which for a queue or a cache is usually all there is to know. port is not needed; the checked port and the proxied port need not be the same.
commandThe argv run inside the replica through the Docker Engine API's exec, no TTY, no shell; exit 0 is healthyA tool that knows the application can tell "listening" from "ready": a database can be listening and still refuse connections while it recovers. The tool must exist in the image; pg_isready, mysqladmin ping and redis-cli ping ship with theirs.

Exactly one of the three is set; a health block with none, or with two, is refused by validation. The engine dispatches on the kind in one place, for the deployment and the supervisor alike.

HTTP ​

The request is sent by the agent over the application network with the header User-Agent: shipwick-health-check.

  • A status from 200 to 299 is healthy. Anything else is a failed probe, reported as HTTP <status>.
  • No answer within timeout is a failed probe, reported as no response within 3s. So is a refused connection (connection refused) or one closed without an answer (connection closed before a response).
  • Redirects are not followed. A redirect is an answer, not a 2xx, and following it could walk the probe off the container network.
  • Every probe opens a fresh connection. A kept-alive connection can outlive a wedged listener and report a dead application as healthy.
  • The probe goes direct. An HTTP proxy configured in the agent's environment is never used.

TCP ​

The agent opens a connection to the container's address and closes it as soon as it is accepted, without sending a byte. The errors are the connection errors of the HTTP probe, prefixed with what was tried: TCP connect to port 5432: connection refused.

Command ​

The command is run inside the running replica as an argv, exactly as written, without a shell. Only the last 4 KB of what it prints are read, and only its last line is reported, so a chatty check cannot grow an event: command exited 2: pg_isready: no response. A command that does not finish within timeout is reported as command did not finish within 3s; the connection to the daemon is closed, which ends the read, but the process itself keeps running inside the container, since Docker has no way to kill an exec. Checks are meant to be tools that finish. A command that cannot be started at all, because the container is not running for instance, is command could not run: ….

Defaults and limits of interval, timeout and retries are in the deploy.yaml reference.

The same probe is used in two places with different patience.

During a deployment: the startup budget ​

Every new replica must answer its check once before it may take traffic. It has start_period + interval × retries to do so, 30 seconds with the defaults. This is the replica's startup budget.

During that time the replica is probed every second, not every interval. A connection refused by an application that is still booting is "not yet", not a strike, and a fast application is confirmed in about a second instead of waiting ten to be told it was ready after one. A slow starter gets its time: set health.start_period, up to 30 minutes, for a JVM or for an application that migrates its database on boot. Raising retries would also work, but would make a running replica's failures take longer to notice; start_period only stretches the budget.

yaml
health:
  path: /health
  start_period: 1m

A replica that never answers within the budget fails the deployment, and the error says why, naming the check that was tried:

text
✗ Deployment failed

  replica 1 did not become healthy within 30s: GET /health on port 8080: connection refused
text
  replica 1 did not become healthy within 30s: TCP connect to port 5432: connection refused
text
  replica 1 did not become healthy within 30s: command exited 2: pg_isready: no response

A replica that exits during the budget fails the deployment at once, without waiting for the budget to run out.

Without a health check ​

Without a health block, a deployment only verifies that replicas start and stay up: each new replica must still be running after a stabilization window of 3 seconds. A replica that exits within the window fails the deployment.

After deployment, such a replica is healthy for as long as it runs. The supervisor restarts it when it exits, but cannot notice a process that runs and no longer serves.

After deployment: continuous checking ​

The supervisor probes every running replica every interval. A single failed probe changes nothing. After retries consecutive failures the replica is marked unhealthy, leaves the proxy's rotation, and is restarted. This is the classic cure for a deadlocked process. The restart stops the replica gracefully: SIGTERM, then its grace period, 10 seconds or the application's deploy.stop_timeout, before it is killed. One passing probe resets the failure count.

A replica that the supervisor has just restarted, or that was started with shipwick start, is starting. It gets its startup budget again, start_period included, before failures count, is probed every second meanwhile, and receives no traffic until its first passing check; a passed check counts at once. If the budget runs out, it becomes unhealthy.

Each replica has one of these health values, shown by shipwick status and in the API's container list:

healthMeaning
"" (empty)The application defines no health check.
unknownNot probed yet, for example right after an agent restart. Counts as healthy. shipwick status shows it as checking.
startingStarted or restarted, still within its startup budget. Not routed to.
healthyThe last probe passed.
unhealthyFailed retries consecutive probes, never came up within its budget, or is not running. Not routed to.

unknown counts as fine on purpose: an application must not flap to DOWN because its supervisor was restarted.

The agent must be able to reach container addresses

HTTP and TCP probes go to container IP addresses. An agent running in a container, the recommended setup, finds itself through the Docker API and joins the application network on its own; since 0.8 it does not listen there, see Who can reach the API. An agent running as a host process reaches bridge networks directly on Linux. With Docker Desktop on macOS or Windows it cannot, and the agent warns at startup: applications with a path or tcp check will fail to deploy there. A command check goes through the Docker API and needs no address.

The supervisor ​

The supervisor is one loop in the agent, ticking every second. For each application that has an active deployment and is meant to be running, it compares the replicas of the active deployment with what Docker reports. A static application, served by the proxy from a folder, has no replicas and nothing for the supervisor to do. It acts on an application only while holding that application's lock, so it never interleaves with a deployment, a stop or a delete. If the lock is taken, the application is looked at again on the next tick. The same ticker starts scheduled jobs once a minute; see Run scheduled jobs and one-off commands.

text
exited  ──policy allows?──no──▶ leave stopped, say so once
   │yes
   ▼
wait backoff[restarts] ──▶ start ──▶ restarts++ ──restarts ≥ 5──▶ CRASH_LOOP (retry every 5m)

running + unhealthy ──policy ≠ never──▶ same backoff ──▶ restart
running + proven for 60s ──▶ restarts = 0, crash loop cleared
What happensWhat Shipwick does
A replica exitsRestarts it, as restart.policy allows.
A replica runs but fails retries checks in a rowMarks it unhealthy and restarts it, unless the policy is never.
A replica keeps dyingBacks off: 1s, 2s, 5s, 10s, 30s. After five restarts that did not hold, the application is CRASH_LOOP and the replica is retried every 5 minutes.
A replica stays up and proven for a minuteIts restart history is forgiven. The next crash starts again at 1s.
A replica's container is goneRecreates it from the deployment's stored configuration.

Health probes run outside the tick, at most one in flight per replica. A probe that takes its full timeout does not delay the restart of some other application's replica.

A pass asks Docker for its containers once, whatever the number of applications, since 0.9; until 0.8 it asked three times a second and once more for every application, which from about a hundred applications on two processors no longer fit in its second. Measured at 2 CPUs and 4 GB (scripts/measure-limits.sh apps), before and after that change, the server idled at 57% of one processor with 50 applications of two replicas and at 103% with 100; it idles at 14% and 29%. A replica the list predates is asked about by ID before anything is concluded, as below. Since 0.9 the pass also looks at the proxy's networks every half minute, and says which it is missing; see When things break.

The agent's own share of a pass, Docker aside, is small and was halved in 0.10. Until then the pass read each application's deployment and its replicas from the database one application at a time, seven statements for each, and parsing those statements was most of what it cost; since 0.10 it reads them all in one go. Measured on the same developer's machine as the figures above (an Intel Core i5-14600KF under Docker Desktop) with go test -bench in agent/internal/deploy, where Docker and the proxy are in-memory stand-ins that answer at once, so that what is left is the agent's — applications of two replicas, each with a hostname and a health check; medians of three runs — one pass took 5.5 ms of processor time with 50 applications, 10.9 ms with 100 and 22.4 ms with 200, and takes 2.6, 5.0 and 10.1 ms. GET /applications with 100 applications cost the agent 2.4 ms and costs 1.2; GET /server 2.2 ms and 1.0; the sampler's round every 30 seconds 6.3 ms and 4.7; on a database with a year of daily deployments the same read took 5.4 ms and takes 0.56. In the server, measured again with measure-limits.sh apps at 50 and 100 applications before and after, the agent's own processor went from 2% and 3% of one to 1% and 2% — the script's resolution is the percent — and the server's idle from 13% and 28% to 14% and 25%; shipwick ps, GET /applications and GET /server took the same 55 to 126 ms, nine tenths of which is Docker listing the containers.

Docker's own restart policy is set to no on every container. Two restart mechanisms would fight, and only the supervisor knows about backoff, health and crash loops.

Restart policy ​

restart.policy in deploy.yaml decides what the supervisor does with a replica that is down.

PolicyA replica that exitedA running replica that turned unhealthy
always (default)Restarted, whatever the exit code.Restarted.
on-failureRestarted only after a non-zero exit code or an out-of-memory kill. A clean exit (code 0) leaves it stopped.Restarted.
neverLeft stopped.Left running, marked unhealthy, out of rotation.

When the policy leaves a replica stopped, the supervisor records that once as an event and does not look at the exit again.

Backoff ​

The delay before a restart depends on how many restarts the replica has had without a stable run in between:

RestartDelay before it
1st1s
2nd2s
3rd5s
4th10s
5th30s
6th and later5m

The same schedule paces restarts of a replica that runs but never turns healthy.

CRASH_LOOP ​

After the fifth restart without a stable run, the replica is crash-looping. The application's status becomes CRASH_LOOP, an event says so, and the replica is retried every 5 minutes, indefinitely.

"Do not restart in a hot loop" and "give up" are different things. Most crash loops are caused by a dependency being down. The slow retry makes the application come back by itself when the dependency does.

text
my-api  ● CRASH_LOOP

REPLICA   CONTAINER            STATE     HEALTH      RESTARTS         STARTED
1         shipwick_my-api_3_1   running   healthy     0                2h ago
2         shipwick_my-api_3_2   running   unhealthy   5 (crash loop)   34s ago

WHEN      EVENT
28s ago   Replica 2 did not become healthy within 30s of starting: HTTP 503
34s ago   Replica 2 is crash-looping: 5 restarts without staying up. Retrying every 5m

Since 0.7 each attempt of a crash loop leaves its output behind: the agent copies what a replica printed when its run ends, and shipwick logs my-api --previous shows the last one. See Find out why it died.

There are three ways out of a crash loop:

  • The cause goes away and the replica has a stable run. The supervisor records "no longer crash-looping".
  • A new deployment. It replaces the containers, and new containers start with a clean slate.
  • shipwick stop followed by shipwick start. An explicit start resets the restart history of every replica.

The stable-run rule ​

A replica's restart count returns to zero, and its crash-loop flag is cleared, once it has been running for 60 seconds since its last restart and is proven.

Proven means more than running. A replica with a health check is proven only once it has actually passed it. Otherwise an application that runs but never answers would be forgiven every minute and never reach CRASH_LOOP. A replica without a health check is proven by running.

Reconciliation of missing containers ​

A replica whose container no longer exists (removed with docker rm, by docker system prune, by anything that is not Shipwick) is recreated from the active deployment's stored configuration within about a second: desired 3, present 2, create 1.

  • Before a replica is declared gone, Docker is asked once more, by container ID. The container list alone is not trusted.
  • A recreated replica with a health check is starting: it receives no traffic until it has passed its check.
  • If recreation fails, for example because the image is gone and cannot be pulled again, the attempts back off on the same schedule as restarts.
  • If the image was pruned from the server, it is pulled again before the container is created.
  • Containers of the application that belong to any other deployment are leftovers, of an interrupted cleanup for instance, and are removed in the same pass. A container that a deployment replaced and that is still using its grace period is not a leftover and is skipped. Job containers are not replicas and are left alone; they are removed when their run ends.

Reconciliation is also what completes an application after a failed rollback, or after a deployment that an agent restart interrupted and that could not be resumed: the active deployment in the database says how many replicas there should be, and the supervisor makes it so.

What the supervisor remembers ​

Health, backoff position, crash-loop flags and the active alerts live in the agent's memory. After an agent restart every replica starts with a clean slate and its health is unknown until probed, and an alert whose condition still holds is raised again. Only the restart counter is persisted, for display.

Applications keep running while the agent is down or being upgraded, but nothing restarts them during that time. After a server reboot, the agent brings every application back up according to its restart policy, and goes on with a deployment that the reboot interrupted; see Agent restarts during a deployment. "According to its policy" has a consequence worth knowing before the first reboot: an application with on-failure whose process exits with 0 when it is asked to stop, and every application with never, stays stopped. See When things break.

When Docker does not answer ​

Since 0.8 the supervisor asks the Docker daemon one question before each pass, and skips the pass when it goes unanswered for 15 seconds: it restarts nothing, changes no route and holds no application. It never concludes that a container is gone from a daemon that did not answer; a replica is declared gone only when Docker says that it does not exist. The agent's log says so once, the docker alert is raised when the silence has lasted 30 seconds, and supervision resumes with the daemon's first answer. See When things break.

Events ​

What the supervisor sees and does is recorded as application events, shown by shipwick status and served by GET /api/v1/applications/:name/events:

text
Replica 2 exited with code 137; restarting in 2s
Replica 2 restarted (attempt 2)
Replica 2 failed 3 health checks in a row: no response within 3s
Replica 2 is healthy
Replica 1 was killed for exceeding its memory limit; restarting in 1s
Replica 3's container shipwick_my-api_7_3 has disappeared
Recreated replica 3

Only the newest 500 events per application are kept. A stop or start made with a token other than root names it: Application stopped by ci. A scheduled job or one-off command that failed or timed out adds an event of type job; successful runs record nothing, since jobs run often. An alert about the application adds an event of type alert when it is raised and when it is cleared, and a replaced container that had to be killed at the end of its grace period is reported here too.

Notifications ​

With SHIPWICK_WEBHOOK_URL set on the agent, the supervisor reports two things about an application at once:

EventWhen
application.downNot one replica is ready: none is running, or those running are failing their health check. The message says which: my-api is down: 1 replica running but failing its health check. Shipwick restarts it as restart.policy allows; see why with: shipwick logs my-api.
application.recoveredEvery desired replica is ready and none has a restart still held against it. The message reads my-api is healthy again: 2/2 replicas running 1.4.2.

Recovered is stricter than not down on purpose. A crash-looping replica runs for a moment between crashes; if that counted as a recovery, every restart would produce one outage and one recovery. The restarts are forgiven after the stable run of 60 seconds described above, and that is when the recovery is reported: one outage and one recovery per incident. A single replica restarting is in the event feed, not in the webhook. Failed jobs are reported as job.failed. See Get notified.

Alerts ​

A notification says that something happened. An alert says that something is the case and, left alone, ends badly. Since 0.5 two alerts are raised from what the supervisor sees in its tick:

AlertRaised whenCleared when
restartsThe supervisor has restarted the same replica three times within ten minutes. Held back, neither raised nor cleared, while the application is down or the replica crash-looping: the outage has been reported, and three restarts are how every outage begins.Fewer than three restarts in the last ten minutes.
unhealthyAn application has had fewer healthy replicas than it should for five minutes: a warning. After an hour: critical.Every replica is healthy and has stayed up for a minute.

Healthy here is the definition recovered uses: every replica ready and none with a restart still held against it. A crash-looping replica, up for a moment between crashes, therefore neither ends the unhealthy period nor restarts its five minutes. application.down and application.recovered are sent at once, as before; unhealthy is what follows when the outage lasts, and it also covers an application that is only degraded, which nothing else reports.

Each alert is raised once and cleared once, however long it lasts; a warning that turns critical is told a second time. Alerts go to the webhook as alert.raised and alert.cleared, into the application's events, and into alerts of GET /server. Stopping or deleting an application drops its alerts without a message. Two more alerts, a replica close to its memory limit and the server's disk filling up, are read from the metric samples, and since 0.8 a fifth, docker, says that the Docker daemon has not answered for 30 seconds. All five, with their thresholds and where they show, are in Alerts and metrics.

Application status ​

An application's status is derived from its deployment history, its desired state and the live state of its replicas.

StatusMeaning
HEALTHYAll desired replicas of the active deployment are healthy.
DEGRADEDSome are.
DOWNNone are: not running, or running but failing their health check.
CRASH_LOOPA replica keeps dying or never turns healthy, and its restarts are now limited to one every 5 minutes. Outranks DEGRADED and DOWN: it says that restarting is not helping.
STOPPEDStopped on request. The supervisor leaves a stopped application alone.
DEPLOYINGThe first deployment is in flight and nothing is active yet.
FAILEDNo deployment has ever succeeded.

A replica counts as healthy when it runs and its health is healthy, unknown or empty. Without a health block, the healthy count equals the running count.

During a rollout, the replica counts describe the replicas that are serving at that moment, a mix of the old and the new version, and the desired count is the capacity the rollout maintains: the smaller of the two replica counts. An application being upgraded therefore reads HEALTHY, not DEGRADED. The separate deploying flag says that a deployment is in flight.