Deploy Scripts Should Refuse, Not Warn

Exit codes are the only signal that matters

A safe deploy script bash must verify its own output and exit with a non-zero status to signal failure. Warnings that allow the process to continue are ignored by the operator and the monitoring stack alike. The goal is not to inform the human standing in front of the terminal, but to stop the pipeline from proceeding with a broken state. When a script exits with code zero, every downstream tool assumes success. If the deployment is actually broken, that assumption propagates until the next scheduled check or user complaint occurs. The script itself is the best position to catch the error, because it knows exactly what it intended to do and what the system actually returned. A warning is a suggestion. An exit code is a fact. By treating any deviation from the expected state as a fatal error, you ensure that the system remains in a known, stable configuration.

Guarding the static assets

GamingCap and Invoiceful are static builds served directly from disk by Caddy. They have no server process and no database to query for health. This makes the deploy script’s job simpler but more critical. The script must verify that the new files are present on the disk before the old ones are removed. A common failure mode is a network interruption during the build or transfer, resulting in a partial file set. If the script simply copies files and exits zero, Caddy will serve 404 errors for missing assets. The guard here is a checksum comparison or a file count verification. If the number of files in the source directory does not match the destination, or if a critical index file is missing, the script must halt. This prevents the site from going offline or serving broken HTML. Since these sites run on a single VPS with 193GB of disk, space is rarely the issue, but integrity is. The script should atomically swap the directory symlink only after the new content is fully validated. This ensures that users never see a half-deployed state.

Verifying the dynamic services

The Next.js process serving four sites by host-based routing and Instalaz, which runs on port 3050, require more than file checks. These services depend on a shared PostgreSQL 16 database. A deploy script that restarts these processes without confirming they can bind to their ports and connect to the database is a ticking time bomb. The script should wait for the process to start, then issue a request to a health endpoint. If the response is not a 200 OK, or if the response time exceeds a defined threshold, the script should exit non-zero. This is particularly important because processes on one box compete for memory. With 12GB of RAM, a memory leak in the new version could starve the database or other services. The script should also verify that Caddy can still proxy the traffic. If Caddy cannot reach the upstream, the site is down even if the Node process is running. The script must treat a failed proxy check as a deployment failure, not a warning.

Systemd and the final check

Scheduled work runs as systemd oneshot services with timers, not cron. This means the deploy script is often triggered by a timer or a manual command. The script’s exit code is the only feedback loop for the systemd unit. If the script exits zero, systemd marks the unit as successful. If it exits non-zero, systemd logs the failure and can trigger alerts. The script should not rely on external monitoring to catch failures that it could have detected itself. For example, if the script updates the DNS records at Spaceship, it should wait for the propagation to complete before considering the deploy successful. This is difficult to verify programmatically, but the script can at least confirm that the API call to update the DNS succeeded. If the API returns an error, the script should exit non-zero. This prevents a situation where the code is deployed to the new server, but the DNS still points to the old one, or vice versa. The script should be the last line of defense before the change becomes live. By refusing to proceed when the state is uncertain, you maintain the integrity of the system. This approach is essential for running multiple products on a single infrastructure. For those interested