Skip to content

Deployments ​

A deployment is an immutable record of one attempt to run one configuration of an application. This page describes the states it moves through, how replicas are replaced and how long a replaced one gets to finish, when a deployment is finished, what a failure does to the version that was running, and what a restart of the agent does to a deployment that is running.

The deployment record ​

Every attempt is recorded, whether it succeeds or not. A record holds:

FieldMeaning
idIdentifier, unique on the server. This is what the API uses.
sequencePer-application counter: #1, #2, and so on. This is what shipwick status shows and what shipwick rollback --to takes. It is also part of the container names.
versionDerived from the image reference: its tag, or a shortened digest, or latest when the reference has neither. For an image built by shipwick deploy, the tag is the build's timestamp. For a static deployment, the first twelve characters of the folder's digest.
imageThe image reference. Empty for a static deployment, which has static: {digest, size_bytes, files} instead.
specThe complete configuration, as validated. Stored with the deployment and never changed afterwards.
statusThe current state of the state machine.
errorWhy the deployment failed, if it did.
kinddeploy, redeploy or rollback. Since 0.5 also import, for the configuration an export carried, deployed on another server, and standby, for the same deployed stopped on a standby server; see Move to a new server.
source_deployment_idFor a redeploy or rollback, the deployment whose configuration was re-used.
byThe name of the token that started it: root for the agent's own token, otherwise the token's name. Absent on deployments made before tokens had names.
started_at, completed_atWhen the record was created, and when the agent was done with it.

A deployment begins in one of three ways. deploy submits a deploy.yaml. redeploy takes the configuration of the active deployment, optionally with another image. rollback takes the configuration of an earlier successful deployment. All three resolve a configuration and hand it to the same entry point, so everything below applies to each of them. See Rollback. Where the image comes from is settled before any of this: a registry the agent pulls from, or the developer's machine, which builds it and sends it over first; see Images built where you are. A static application has no image at all; see A folder instead of a container.

Before a record is created, the agent checks the application's hostnames and published ports. If another application already answers to one of its domain, aliases or redirects, or the agent or the dashboard is served on it, the request is refused as a configuration error whose field names the offending line (aliases[1]). Since 0.5 a hostname is taken per path: two applications may share a hostname under different paths, but not the same path of it, and a redirected hostname is taken whole; see path. So is a publish entry whose server port another application publishes, or that the agent or the proxy listens on (publish[0].host). Nothing is recorded, pulled or started. A redeploy and a rollback are checked the same way, since a stored configuration's hostnames and ports may have been taken since.

The same checks can be asked for without deploying. POST /applications/:name/validate answers what a deployment of the file would answer — a hostname or a published port that something else holds, a ${NAME} in env that is not among the stored secrets — records nothing, and does not wait for a deployment that is running. shipwick deploy asks it before it builds an image or uploads a folder; see Images built where you are.

States ​

text
PENDING → BUILDING → STARTING → HEALTH_CHECKING → HEALTHY → ACTIVE → SUPERSEDED
    │         │          │              │             │
    └─────────┴──────────┴──────────────┴─────────────┴──→ FAILED
                                                              │
                                        ROLLED_BACK ← RESTORING ← ROLLBACK
StateWhat happens
PENDINGThe record exists and the application's lock is held. The API has answered 202 Accepted.
BUILDINGThe image is pulled. The agent does not build images; this state covers obtaining one. If the pull fails but the image exists locally, the local copy is used and a warning is recorded. An image built by shipwick deploy and sent to the server is not pulled at all: the step reads Using image shipwick.local/…, sent from a developer's machine, and if it is not on the server the deployment fails here. With pre_deploy, the command then runs from the new image, before any replica of it exists; see Before the replicas start. For a static deployment this state checks the uploaded folder.
STARTINGThe networks are ensured. The first new replica is created, recorded and started. On a first deployment, and under recreate, all replicas are.
HEALTH_CHECKINGEvery new replica must prove it is ready before it takes traffic. With a health block it must answer its check once within start_period + interval × retries, probed every second. Without one it must stay running through a 3 second stabilization window. A replica that exits fails the deployment at once. Replicas are replaced one at a time during this state.
HEALTHYEvery replica has been replaced and serves.
ACTIVEThe commit point. One database transaction promotes the deployment, marks the previous one SUPERSEDED and repoints the application. A final sweep then removes any container of the application that does not belong to the new deployment, except the replaced ones that are still using their grace period, and the images that only retired deployments refer to.
SUPERSEDEDA later deployment became ACTIVE. Superseded deployments are the possible targets of a rollback.
FAILEDThe deployment cannot succeed. If nothing of the old version had been retired, routing returns to it, the new containers are discarded, and this is the final state. Otherwise the rollback path follows.
ROLLBACK, RESTORING, ROLLED_BACKThe previous version is made whole again. See Rollback.

The final states are ACTIVE (later SUPERSEDED), FAILED and ROLLED_BACK. ACTIVE is the only success.

Illegal transitions are rejected by the engine, and every transition is a compare-and-swap in the database (UPDATE … WHERE status = <expected>), so a stale writer can never overwrite a newer state.

Before the replicas start: the pre-deploy command ​

yaml
pre_deploy:
  command: ["dotnet", "Migrate.dll"]
  timeout: 10m    # default 10m; 1s to 1h

pre_deploy runs a command from the new image, with the application's environment and limits, once the image is pulled and before any replica of the new version exists. It runs in a one-off container that is removed afterwards, like a scheduled job, and like a job it gets no volumes: a replica may be writing them. The deployment shows Running pre-deploy command and Pre-deploy command finished (12s).

  • A failure fails the deployment before anything was started. If the command exits non-zero or outlives its timeout, the deployment is FAILED with pre-deploy command exited 1 or pre-deploy command timed out after 10m, the last 20 lines of its output are saved with the deployment as a log event, and shipwick deploy prints them under the error. No replica of the new version was created, nothing of the serving version was touched, and there is nothing to roll back.
  • It runs next to the running version, under recreate too. The old replicas are stopped only afterwards. What the command does must therefore be safe next to the old code: add a column, do not drop one. That is the same backward compatibility a rolling update asks of migrations anyway.
  • Its timeout is added to the deployment's. A deployment normally has 15 minutes in total; with a hook, the hook's own timeout comes on top.
  • A rollback runs it too. A rollback deploys the stored configuration of an older deployment, hook included, from that older image. See Rollback.

Rolling replacement ​

Deployments are rolling by default (deploy.strategy: rolling). Replicas are replaced one at a time:

text
for every replica i:
    start new i → wait until it is ready → route to new i instead of old i → retire old i
then: retire old replicas that have no successor (scaling down) → commit
text
v1.4.1  ├── replica 1 ──▶ v1.4.2 replica 1 starts ─▶ healthy ─▶ takes traffic ─▶ old replica 1 retired
        ├── replica 2 ──▶ v1.4.2 replica 2 …
        └── replica 3 ──▶ v1.4.2 replica 3 …

The peak is N+1 containers. Starting the whole new version next to the old one would be simpler, because nothing of the old version would be touched before the commit. It would also need 2N containers for the duration, and Shipwick is for small servers: a 1 GB application with two replicas would need 4 GB to deploy. Rolling peaks at N+1 containers and never serves with fewer than N.

Replicas with nothing to replace start together. On a first deployment, or when scaling up, the additional replicas start as one batch, so that N replicas do not cost N waits. When scaling down, the surplus old replicas are retired last, after every new replica serves.

Both versions serve side by side for a moment. As with any rolling update, the two versions must be able to coexist. Database migrations, above all, must be backward compatible.

When traffic moves ​

A new replica joins the rotation only after it is ready. At each swap the order is fixed:

  1. The new replica is given the application's names on the services network. The proxy and the other applications find it at their next lookup.
  2. The rollout waits 1.5 seconds, so that the proxy's next lookup has found the newcomer.
  3. Only then is the predecessor stopped: SIGTERM, and SIGKILL after its grace period, 10 seconds unless deploy.stop_timeout says otherwise. It keeps its names until its process exits; exiting is what takes it out of Docker's DNS. The rollout does not sit through the grace period of the last replica it replaces; see below.

The proxy's configuration is not loaded at a swap. It names the application, not its replicas, and changes only when a domain or a port does. If Docker or the proxy cannot be updated, the deployment fails at that swap, before the predecessor is touched.

During a rollout no single deployment record describes who serves ("new 1, old 2, old 3"), and the database still names the old deployment as active. The rollout therefore dictates the application's routing until it commits or fails. At the commit the database says the same thing the rollout did; on failure, routing follows the database back to the previous deployment. See Routing and HTTPS.

Deploying to a stopped application starts it: the commit sets the application's desired state back to running.

Time to finish: the grace period ​

A replica that has been replaced is sent SIGTERM and then has its grace period to finish what it has in hand before it is killed: 10 seconds, or what deploy.stop_timeout sets, from 1s to 10m.

yaml
deploy:
  stop_timeout: 2m   # WebSockets, long uploads: let them finish

Since 0.5 the deployment does not wait for the last of them. By the time a replica is retired, its successor has been verified and has taken its place, so the old container gets its SIGTERM, its grace period and its removal in the background, outside the application's lock. Once every new replica serves, the deployment is ACTIVE and done, and how long it took no longer depends on how the old process takes the signal.

Three things still wait for a container that is leaving:

  • The next replica of the same rollout. A rollout waits for the replica it just replaced before it starts the next one, so an application with several replicas pays one grace period per replica but the last, and the peak stays at N+1 containers.
  • The next operation that needs it gone. A deployment started while a replaced container is still leaving waits for it before it starts a replica, and says how long: Waited 5s for a replaced container to stop. A rollback and shipwick delete wait the same way.
  • recreate, which does not hand anything to the background: the old version is stopped before the new one starts.

The grace period belongs to the version that is stopping, not to the one replacing it: it is read from the configuration of the deployment the container belongs to. It applies wherever Shipwick stops a replica gracefully — a rollout, shipwick stop, shipwick delete, the supervisor restarting an unhealthy replica — and those other three do wait for it. The containers of jobs and one-off commands keep the 10 second default.

What the time is used for is the application's business. A process that handles SIGTERM — stops accepting, finishes its requests, exits — leaves cleanly and usually at once. One that does not handle it keeps running, and keeps receiving a share of the requests, until the time is up and it is killed with whatever it was serving. A program started as the container's first process (CMD ["node", "server.js"]) ignores SIGTERM unless it installs a handler. The agent cannot tell the two apart beforehand, so it says so afterwards: a replaced container that had to be killed is reported in the application's events, in shipwick status and the dashboard, once it is gone, which is after the deployment has completed:

text
The replaced container shipwick_my-api_6_1 did not exit within 10s of SIGTERM and was killed. If its process has no handler for SIGTERM, set init: true and the signal ends it at once; to let it finish its requests, handle SIGTERM in the application; to give it longer, set deploy.stop_timeout

shipwick stop writes the same line about a replica it had to kill. Until the old container is gone, shipwick status lists it as stopping, apart from the replicas.

An init process ​

The kernel hands the first process of a container no signal it has not installed a handler for, which is why node server.js sits out its grace period. Since 0.6, init: true puts Docker's own init process in front of it:

yaml
init: true

The init process passes SIGTERM on to the application, which is then an ordinary process and ends on it at once, and it reaps the children the application leaves behind. The same ten-line Node server, stopped with shipwick stop, took 0.3 seconds with init: true and 10.4 seconds — its whole grace period, then the kill — without.

It applies to every container of the application: replicas, the pre-deploy hook, jobs and one-off commands. It is not the default, because an image that brings its own init does not want a second one: s6-overlay refuses to start unless it is the first process, and an image whose entrypoint is tini would run two. shipwick init writes init: true into the deploy.yaml of a Node project whose Dockerfile it writes.

An application that handles SIGTERM to finish its requests still does under an init process; ending at once is what happens to one that does not. See init and deploy in the reference.

Progress events ​

Each deployment carries a list of events. state events mark transitions; step events are meant to be shown to users, and shipwick deploy prints them as they appear. Since 0.9 one more event, probe, follows completed_at for a deployment with a routed domain: what the application answered when the agent asked it its own hostname through the proxy, a warning when the answer is a redirect loop (see A redirect loop is caught).

text
state  BUILDING
step   Pulled image ghcr.io/company/my-api:1.4.2     (or: "Using image shipwick.local/…, sent from a developer's machine")
step   Running pre-deploy command             (only with pre_deploy)
step   Pre-deploy command finished (12s)
state  STARTING
step   Started 1 container
state  HEALTH_CHECKING
step   Replica 1 passed health checks        (or: "running and stable" without a health block)
step   Replica 1/2 is serving 1.4.2; its 1.4.1 predecessor is retired
step   Replica 2 passed health checks
step   Replica 2/2 is serving 1.4.2; its 1.4.1 predecessor is retired
state  HEALTHY
step   Routed https://api.example.com to 2 replicas   (only with a domain)
state  ACTIVE
step   Removed 1 image of older versions      (only when there was one to remove)
step   Deployment successful

Recreate ​

deploy.strategy: recreate is for the application that cannot run twice: anything with a volume, or that holds a lock or a port it cannot share. A database, above all. Applications with volumes must use it; an application without volumes may.

yaml
deploy:
  strategy: recreate

The same rollout runs it with three differences:

  1. The old version is stopped first. After the image is pulled, and after the pre_deploy command has run if there is one, the running version is taken out of the proxy and stopped, one replica at a time, gracefully, each with its grace period. The containers are kept, not removed. The events say why: Stopped 1.4.1: 1.4.2 cannot run next to it.
  2. All new replicas start as one batch, since nothing runs that they could disturb. They are verified like any new replica, against the health check or the stabilization window, and then given the application's names. The stopped containers of the old version are removed at that point.
  3. A failure is undone by starting the old containers again. See Rollback.

The application is down from the stop to the moment the new version is ready: its domain answers 503, its name on the services network is carried by nobody, and shipwick status reads DOWN. The downtime is the new version's start time, plus what it takes to verify it. With a health block that is until the first passing probe; without one, the 3 second stabilization window.

text
Deploying postgres...

✓ Validated deploy.yaml (1 variable substituted)
✓ Pulled image postgres:17.1
✓ Stopped 17: 17.1 cannot run next to it
✓ Started 1 container
✓ Replica 1 running and stable
✓ Replica 1/1 is serving 17.1; its 17 predecessor is retired
✓ Deployment successful

Volumes without recreate, or with more than one replica, are refused by validation: two versions writing the same files at once is how data gets lost. See Run a database or other stateful application. Published ports have the same requirement, for the same reason: a server port has one holder, so the old version must be gone before the new one binds it. See Expose a service that is not HTTP.

When a deployment is done ​

A deployment's status settles (ACTIVE, FAILED) slightly before the agent is finished with it. Leftovers are swept, images pruned, and a failed rollout's containers removed. During that window the application is still locked.

completed_at is stamped only after cleanup and after the lock is released. It is the reliable signal:

Poll GET /api/v1/deployments/:id until completed_at is set. From that moment a new operation on the application is guaranteed not to be rejected as busy.

The replicas a rollout replaced may still be on their way out by then; nothing waits for them that does not have to, as described under Time to finish.

FAILED in particular is not necessarily the end: a deployment that fails after some replicas were replaced continues through ROLLBACK, RESTORING and ROLLED_BACK. shipwick deploy waits for completed_at, not for the first settled status.

Old images are removed ​

Part of the cleanup after a successful deployment is removing the images that only retired deployments of the application refer to, so a server that deploys daily does not fill its disk with versions nobody can return to. What stays, across all applications: the image of every active deployment, and the image of each application's most recent superseded one, which is the rollback target, so a rollback never waits for a pull. Only images that some deployment of the application named are candidates. An image any container still uses is left alone, so is anything pulled outside Shipwick, and a failure to remove one is logged, never returned. The same sweep runs after shipwick delete. The step reads Removed 1 image of older versions. Images sent from a developer's machine are pruned like any other; the two that matter stay. A static application's folders are kept the same way: the one serving and the one before it.

A deployment that ends FAILED or ROLLED_BACK removes the image it named as soon as it has ended, unless that image is the one running or the rollback target; an image any container still uses stays here too. The step reads Removed image ghcr.io/company/my-api:1.4.3, which nothing else uses. Before 0.5, an image sent with build: for a deployment that then failed stayed until the application's next successful deployment, and a first deployment that kept failing left one image behind per attempt. An image shipwick deploy built and sent for a deployment the agent then refused is removed by the next sweep once it is ten minutes old, and by shipwick delete at any age.

Images built where you are ​

With build: . in place of image, there is no registry in the picture. shipwick deploy runs docker build on your machine — the project and Docker are already there — for the server's architecture (--platform, asked of the agent, so a laptop of one kind builds for a server of another), tags the result shipwick.local/<name>:<UTC stamp>-<4 hex>, saves it and streams the archive to the agent, which loads it and deploys it like any other image.

Before it builds, the CLI has the agent validate the file, with the checks of a deployment and without the deployment, so that what only the server can know — a domain another application serves, a port already published, a secret that is not stored — is said before the build and the upload, not after them. The agent checks again when the deployment starts. An agent older than 0.5 is asked whether the domain is free, as before.

In a terminal the build is one line that moves, Building the image (12s) — #9 [build 4/6] RUN npm ci, and its whole output appears only when it fails. shipwick deploy --verbose shows every line as it is printed, and so does any run whose output is not a terminal, since a pipeline's log is where a build is read. Several applications of a shipwick.yaml building at once in a terminal are silent until each build ends.

The first deployment sends the whole image; since 0.5, only the layers the server does not have travel after that. The CLI asks the agent which of the image's layers its Docker lacks and leaves the others out of the archive, so a change to your code costs the size of the layer that holds it, not of the base image beneath it:

text
✓ Sent image to the server (22 KB; the server had the rest of 57.9 MB)

That line was measured with a Node application on node:22-alpine. This works with both of Docker's image stores, the classic and the containerd one, on your machine and on the server. It is an economy, never a condition: with an agent older than 0.5, an archive in a layout the CLI does not know (Docker before 25), or a server that lost a layer in the meantime, the whole image is sent as before. The agent trusts none of it: a reduced archive goes through the same checks as a whole one, and one that leaves out a layer the server lacks is refused with 409 IMAGE_INCOMPLETE, nothing stays loaded, and the CLI sends the whole archive.

shipwick.local is a host that does not exist, on purpose: nothing can pull from it, so such an image is either on the server or it is not, and a deployment whose image is gone (pruned, or a rollback to a version that was never sent to this server) says so and asks for another shipwick deploy from the project. The server never builds: a Dockerfile runs whatever it likes, with the network and CPU of the machine it runs on, and that machine should be yours. docker must be installed where shipwick deploy runs. shipwick validate describes the build and does not run it; --image does not apply to an application with build:, and a redeploy of one keeps the image it has. See build in the reference, and POST /applications/:name/images for the two requests the CLI makes.

A folder instead of a container ​

A built frontend — the dist/ of a Vite or Nuxt site, the out/ of a Next export, any folder of HTML, CSS and JavaScript — needs no container of its own; Caddy serves files. static names the folder, relative to deploy.yaml:

yaml
name: web
static: dist/
domain: example.com
redirects: [www.example.com]

shipwick deploy sends the folder as it is — run the build first — and the agent puts it in front of Caddy and routes the domain to it. The agent is asked to validate the file before the folder is uploaded, as before a build:

text
✓ Validated deploy.yaml
✓ Uploaded dist/: 42 files, 3.1 MB
✓ Received 42 files (3.1 MB)
✓ Copied 42 files into the proxy
✓ Found index.html
✓ Routed https://example.com to the uploaded files
✓ Deployment successful

The folder must hold an index.html; a request for a directory gets it, a request for a path that names no file gets a 404. A single-page application, whose router reads the path in the browser, names the page that answers those instead:

yaml
static:
  dir: dist/
  fallback: index.html

Every path that names no file is then answered with that page and status 200, a missing image or script included. The file must be in the folder: the CLI looks before it uploads, the agent before it routes.

A folder may be up to 512 MB. Nothing that describes a container applies, since there is none: no image, port, replicas, env, health, resources, volumes, jobs; domain, aliases and redirects work as for any application, and so do stop (the domain answers 503), start, rollback and delete. shipwick logs, metrics and run have nothing to show and say so. The version of a static deployment is the first twelve characters of the folder's digest: the same files make the same version, on any machine. The agent keeps the folder that serves and the one before it — the rollback target — and removes older ones; shipwick rollback needs no upload. An application that stops being a folder, because the next deployment names an image, keeps the last folder for as long as a plain shipwick rollback would return to it, which is until its second container version; after that nothing of it is left in the proxy. Before 0.5 the folders stayed until the application was deleted. Because the proxy's root changes with every new folder, deploying a static application reloads Caddy once, the one kind of change a rollout of containers never causes; see Routing and HTTPS. Static deployments go through the same record and state machine as every other; there are just no replicas in it. See static in the reference, and PUT /applications/:name/static.

Several applications at once ​

A shipwick.yaml holds several applications, each entry a complete deploy.yaml, with after naming the entries one must wait for. It is deployed as several ordinary deployments, one per application, each with its own record, lock, health checks and rollback; the agent never sees the file. The CLI holds the order: an application starts when every name in its after has completed an ACTIVE deployment in this run, at most four run at a time (--parallel), and one that waits for an application that failed, or was itself skipped, is skipped — a failure is reported, never propagated by timing. after is about readiness, not reachability: names on the services network resolve whatever the order, so after: [postgres] belongs on an application that would exit without its database, not on every consumer of another service. See Several applications.

Notifications ​

With a webhook configured on the agent, the outcome of every deployment is posted there when it is known: deployment.succeeded, deployment.failed or deployment.rolled_back, with a sentence that says what is running now. See Get notified.

One operation per application ​

Deploy, redeploy, rollback, stop, start, delete and the supervisor all contend for one in-memory lock per application. Different applications are fully independent.

There is no queue. A queue hides the conflict from the person who needs to know about it. What a client sees depends on who holds the lock:

Held byA user operation
Another user operationFails at once with 409 DEPLOYMENT_IN_PROGRESS. shipwick prints "Another operation is already in progress for this application."
The supervisorWaits, up to 30 seconds. The supervisor holds an application for moments, to restart a replica.

The supervisor itself never waits. A busy application is looked at again on its next tick, one second later.

Scheduled jobs and one-off commands hold no lock while they run: the lock is taken only to record the run and start its container, so a job that runs for an hour does not block a deployment, and a deployment that finishes leaves a running job of the old version alone until it ends. The pre_deploy command is the exception; it runs inside the deployment, which holds the lock anyway. A backup holds the lock while the archive streams, so a deployment asked for meanwhile is refused as busy.

While a deployment is in flight, the application's deploying field is true and in_flight_deployment_id names the deployment to follow. An application can be HEALTHY, with the old version serving, and deploying at once.

When a deployment fails ​

A new replica that crashes, is killed for exceeding its memory limit, or never becomes ready fails the deployment. So does an image that cannot be obtained — one sent from a developer's machine that is no longer on the server included — a pre_deploy command that exits non-zero or times out, a proxy that cannot be updated, a static folder without an index.html, and a deployment that takes longer than 15 minutes in total, image pull included (plus the pre_deploy timeout, when there is one).

If a replica crashed or never became healthy, its last 20 lines of output are saved with the deployment as a log event. The container is about to be deleted, and before 0.7 that was the only clue. A failed pre_deploy command leaves its last 20 lines the same way. Since 0.7 the agent also keeps more of it in its log archive, the last 2,000 lines of each replica of the failed deployment and the last 10,000 of the command: shipwick logs my-api --deployment 13. See Find out why it died.

What happens next depends on how far the rollout had come.

Before the first replica was replaced. This is the usual case: a bad image rarely survives its first health check, and a failed pre_deploy command fails before a replica exists. Nothing of the old version was touched. The new containers are removed and the deployment ends as FAILED.

text
✗ Deployment failed

  replica 1 exited with code 1 shortly after start

  Last output of replica 1:
  panic: DATABASE_URL is not set

my-api is still running 1.4.1; the failed deployment did not affect it.

After some replicas were replaced. Old replicas are retired before the commit, so a failure half-way leaves the previous version incomplete. The retired replicas are recreated from the previous deployment's stored configuration and verified, traffic returns to them, and the deployment ends as ROLLED_BACK. The new replicas that were already serving keep serving until the restored ones are ready, so capacity does not dip twice. See Rollback.

Measured under constant load: a rolling redeploy of 3 replicas answered 100 of 100 requests with 200, with 3 to 4 containers throughout. A rollout sabotaged at its second replica was rolled back, and 76 of 76 requests were answered with 200. A rollout does not reload the proxy, so this holds over a real network too: with a new connection per request, 12 clients and 50 ms of added latency, 18 consecutive rolling redeploys answered 15,774 of 15,774 requests. See What a deployment costs.

Under recreate. The old version has been stopped by the time a new replica can fail, so the failure is always undone: the new containers are removed, the stopped containers are started again, and the deployment ends as ROLLED_BACK. See Rollback.

Agent restarts during a deployment ​

Upgrading Shipwick restarts the agent, and so may a reboot or a crash. Since 0.5 a deployment that was running at that moment is not failed. An agent that shuts down leaves it exactly as it is — the record in flight, completed_at unset, its containers in place — and the agent that starts next resumes it before it serves a request; a crash is the same thing without the warning. Containers it had created are kept, replicas that were already serving keep serving, and only what was not done is done. The deployment's events say Resumed after the agent restarted.

Interrupted whileOn resume
The image was being pulled (BUILDING)The pull is repeated. An image that is gone for good, a shipwick.local one, fails the deployment as it would have.
The pre_deploy command ranThe deployment fails: the pre-deploy command was interrupted when the agent restarted, and is not run a second time: check what it left behind, then deploy again. A command that had finished is not repeated either, and the deployment goes on.
Containers were starting (STARTING)Created containers are adopted and started, missing ones created.
A replica was being health-checkedThe replica is adopted and checked again; its startup budget starts over.
Between two replicasReplicas already serving keep serving; the rollout continues with the next.
HEALTHY, before the commitCommitted.
Rolling back (ROLLBACK, RESTORING)The rollback is finished: missing replicas of the previous version are created, all of them verified, and what is left of the failed version removed.
After it settled (ACTIVE, FAILED)Nothing to resume. Leftovers are removed, and the supervisor's reconciliation completes whatever version is active.

The pre_deploy command is the one exception to resuming. Whether a migration that was cut off can run again is for its author to say, so the agent does not run it a second time: the deployment fails and says so, nothing of the running version has been touched, and you deploy again once you have looked at what the command left behind.

A resumed deployment gets a fresh deployment timeout; the restart is not counted against it. What cannot be resumed, a record the engine could not have left or a state that cannot be read back from Docker, fails as every interrupted deployment did before 0.5: FAILED, with an error that begins agent restarted during deployment, and its containers removed as leftovers. The previous deployment is then still the active one in the database, and the supervisor's reconciliation completes it again. See Health checks and supervision.

shipwick deploy waits through the restart: it shows Waiting for the agent to respond while the agent is away and goes on when it is back. It gives up after 20 polls in a row that did not reach the agent; the deployment is not affected by that, and shipwick status shows how it went.

A replaced container that was still using its grace period when the agent stopped is not waited for: Docker finishes the stop it was asked for, and the container is removed as a leftover at the next start. Running applications are not affected by agent restarts or upgrades.

Pressing Ctrl+C in shipwick deploy stops the waiting, not the deployment.