Skip to content

Deploy to an Existing VPS

Purpose

Use the repository-owned local orchestrator to deploy the exact reviewed origin/main commit into the existing immutable-release/systemd topology. The preferred artifact path builds outside the VPS, verifies the complete payload and stages a compatible fallback. The command does not provision a host, restore a database, delete a release, or roll back automatically.

Prerequisites

  • Run from the repository root after the intended commit is merged and pushed.
  • Local HEAD must match freshly fetched origin/main, with no tracked working-tree changes. Untracked paths are reported and excluded.

  • Complete the repository verification for the intended change before merge:

    python3 scripts/verify_docs.py
NO_MKDOCS_2_WARNING=true python3 -m pytest -q tests

Run the affected platform tests for code changes as described in AGENTS.md. The deployment helper does not rerun development test suites locally or on the production host. - Configure an SSH alias for the reviewed host using the supported topology and set every host, path and public-origin override in Deployment settings. The examples use the fictional alias mission-control-vps and service account mission-control; replace them with your installation's values. - Keep the documented Node 24, pnpm 11.22.0, Go 1.25.14, uv, systemd user units, restricted environment files, durable paths, and public TLS routes present. - Configure inference on 127.0.0.1:4010, private control on 127.0.0.1:4001, and point Agent runtime, Collector, and Console heartbeats to private control. Remove retired single-listener and telemetry-mode keys. - Keep PostgreSQL 16 or newer reachable from the host, with the database created by the Postgres superuser (libc C collation and character type, owned by a dedicated login role, as in Bootstrap and first sign-in) and migrated. The service account needs no sudo. - Define DATABASE_URL (a postgres:// or postgresql:// URL, password in the user part) in production's shared.env in DEPLOY_CONFIG_ROOT, mode 600. The script reads it only from that file, never from your shell, and never sends it over the SSH command line. shared.env must not define DATABASE_PATH or APPLICATION_DATABASE_PATH, even empty. An installation that still runs a SQLite release follows Migrate from SQLite first. - Install psql, pg_dump and pg_restore of the server's major version on the host. Keep at least 1 GiB plus the database size free on the backup filesystem. - Have the reviewed global/organization YAML and required static secrets ready before starting services. Sign in as organization.owner or organization.admin to inspect the loaded configuration afterward.

Build and deploy verified artifacts

Run from the repository root. Complete release verification and merge the candidate into main; the deployment command still requires local HEAD to match freshly fetched origin/main. Replace the fictional fallback SHA below with a reviewed ancestor that declares the same evaluation runtime contract as the compatible fallback. A pre-program release without the marker is not compatible with new case settings, comparison groups, sample-request authority or suite imports. The fallback's release directory must not exist on the host yet, because staging refuses an existing release. The fallback must also be a Postgres release: its build and its staging check that it loads the pg driver, so a pre-Postgres release is refused.

For the first Postgres deployment, the SQLite release that runs on the host cannot be the fallback. Set FALLBACK_COMMIT to the newest earlier commit of the Postgres release instead: the second parent of the merge commit, or the candidate's parent when main was fast-forwarded. Such a commit exists on main only when the Postgres release was merged with its history, by a merge commit or a fast-forward; a squash merge leaves none. Step 1 of Migrate from SQLite gives the commands that select it, check that its migrations match the candidate's and that its release directory is free, and build both artifacts. Because the migrations match, the fallback runs against the same imported database: if that deployment fails once the launch link points at the candidate, a code-only scripts/rollback_vps.sh to the staged fallback returns to earlier Postgres code, as described in Upgrade and rollback. The helper never switches to the fallback on its own; the runbook's Failure recovery covers each phase a failed first deployment can stop in. Returning to SQLite is a separate manual procedure, Roll back to the SQLite release, for when you decide to leave Postgres; scripts/rollback_vps.sh refuses a SQLite release before stopping anything.

CANDIDATE_COMMIT="$(git rev-parse HEAD)"
FALLBACK_COMMIT="bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
RELEASE_REF="$FALLBACK_COMMIT" RELEASE_OUTPUT_DIR=/private/tmp/mc-fallback scripts/build_release_artifact.sh
RELEASE_REF="$CANDIDATE_COMMIT" RELEASE_OUTPUT_DIR=/private/tmp/mc-candidate scripts/build_release_artifact.sh

The local Docker builder uses digest-pinned Node 24.13.1, pnpm 11.22.0, Go 1.25.14 and uv 0.12.3 images/tools. The default target is linux/amd64; linux/arm64 is also accepted. It archives the exact Git commit, installs locked dependencies (the pg driver is pure JavaScript; no native database module is compiled), builds the Console without any database setting, compiles the Collector, and prepares locked Linux Python wheels. Production configuration and secrets are never mounted into the builder. Scripts are snapshotted before execution; the Docker image context contains no repository source.

The resulting release-<commit>.tar.gz and release-<commit>.json are published only after safe extraction, a pg driver import probe, offline Python installation, relocation and actual runtime health succeed in Linux. The manifest declares mission-control.release/v1, commit, platform/architecture, Node major, pnpm/Python versions, evaluation contract, lockfile hashes and each file's SHA-256/size/mode or internal symlink target. Package-cache hardlinks are serialized as independent regular bodies. Extraction rejects unsafe paths, duplicate or unlisted members, hardlinks, special files, escaping/cyclic links and checksum/mode mismatches before creating the destination. The maximum payload is 8 GiB and 500,000 files. Checksums provide integrity for the trusted reviewed build; they are not an independent publisher signature.

Confirm the installed remote Python executable's exact version matches RELEASE_PYTHON_VERSION (default 3.14.7), then deploy:

DEPLOY_RELEASE_ARTIFACT="/private/tmp/mc-candidate/release-$CANDIDATE_COMMIT.tar.gz" \
DEPLOY_RELEASE_MANIFEST="/private/tmp/mc-candidate/release-$CANDIDATE_COMMIT.json" \
DEPLOY_FALLBACK_ARTIFACT="/private/tmp/mc-fallback/release-$FALLBACK_COMMIT.tar.gz" \
DEPLOY_FALLBACK_MANIFEST="/private/tmp/mc-fallback/release-$FALLBACK_COMMIT.json" \
DEPLOY_PYTHON_BIN=/home/mission-control/.local/share/uv/python/cpython-3.14.7-linux-x86_64-gnu/bin/python3.14 \
scripts/deploy_vps.sh

The artifact path requires Python 3.11 or newer for validation, a glibc Linux host with the matching architecture, Node 24, pnpm 11.22.0 and uv. It compiles nothing on the VPS. It creates a relocatable private virtual environment from verified wheels with --no-index, --no-build and --require-hashes; no dependency download is needed. Preflight checks at least 512 MiB available memory and disk headroom for both payload/upload copies, coordinated backups, Python installation and a reserve. It does not stop unrelated services to free resources.

Staging validates both releases while the current services remain live. A missing compatibility state on an established installation refuses deployment; only a first bootstrap without the launch link or guard drop-ins may initialize floor zero. A permanent service-release symlink binds each service's working directory, executable and startup guard; initially it points at the existing release. For a Postgres release, the startup guard also reads shared.env and refuses to start a service when it defines DATABASE_PATH or APPLICATION_DATABASE_PATH or lacks a postgres:// DATABASE_URL; it never prints the URL. A pre-Postgres release is exempt, so the manual rollback to the SQLite release starts through the same drop-ins. The deployment owns only two named systemd drop-ins per service and preserves other configuration. It verifies the effective launch commands and that the compatibility guard runs first.

After all four Mission Control services stop, the helper backs up the database with pg_dump --format=custom --no-owner, waiting at most 60 seconds for table locks, writes its catalog (pg_restore --list) beside it, reads the whole dump back, and copies the catalog, restricted environment files, unit configuration and runtime compatibility state. It writes the SHA-256 of every backup file, the dump included, to checksums.json. The monotonic compatibility floor advances before migration, so a crash or partial migration cannot silently restart an older worker. Migration is one db:migrate step against DATABASE_URL. The helper then verifies that every migration the candidate bundles is recorded (verify-migrations) and stops before promotion with STORAGE_MIGRATION_REQUIRED otherwise. After migration, the launch link switches atomically to the candidate; all four services and private readiness must pass before current is promoted. Service traffic can resume during this verification; this is a maintenance deployment, not a zero-downtime traffic switch.

current records the last verified healthy release; services launch through service-release. Never change one link manually to bypass the guard. Keep the compatible fallback, immutable payloads and current records. Python runtime/cache directories created during installation/startup are separate from the checksummed build payload.

Build setting Default / meaning
RELEASE_REF HEAD; exact commit to archive
RELEASE_OUTPUT_DIR Temporary-directory mission-control-artifacts; existing final outputs refuse
RELEASE_PLATFORM linux/amd64; alternatively linux/arm64
RELEASE_PYTHON_VERSION 3.14.7; must match the remote interpreter
RELEASE_BUILD_MEMORY 7g local Docker limit
RELEASE_NODE_HEAP_MB 4096 for the Console compiler

The named Docker volume mission-control-release-cache contains build dependency caches. It is not production storage. Build failures retain logs/partial output for inspection and do not upload or deploy anything.

Source-build compatibility path

The on-host source-build path remains available after the initial artifact cutover installs the compatibility fence. The first upgrade to the evaluation runtime contract requires the artifact path and its verified fallback. A later source build must declare the established contract; it cannot lower the floor.

  1. Run:

    scripts/deploy_vps.sh

  2. Let remote staging complete while the current release remains live. Staging installs the locked workspace dependencies and builds them first, then builds the Console with no database setting: DATABASE_URL and the retired SQLite path variables are unset for the build, which performs no query. The helper uses next build --webpack, whose separate build worker reduces compiler memory pressure on the running VPS; the compiler receives NODE_OPTIONS=--max-old-space-size=3072 so it can use up to 3 GiB of JavaScript heap. This applies only to the Console build, not running services. Allow additional host memory for native compiler allocations and the live services; use the external artifact build on memory-constrained hosts. A JavaScript heap allocation failure during staging leaves the current release running and does not create a database backup or promote the candidate. The production Playwright harness uses the same build mode. The build never opens the live database or loads the runtime environment files. Staging also discards the generated Webpack compiler cache after the Console build and on staging exit, including failure. Runnable bundles and runtime caches are retained; immutable releases do not need compiler caches for startup or rollback. Runtime startup still requires the configured database with every bundled migration applied.

  3. The immutable staging phase also builds the pinned Collector binary. The maintenance phase stops services, writes the pg_dump backup with its catalog and mission-control.dump.sha256 plus the catalog copy, runs db:migrate, verifies the migrations, promotes current, and starts services in dependency order.
  4. Startup loads the reviewed configuration automatically; no Console action or confirmation prompt is required. The script then requires private readiness after Agent runtime, Collector, and Console heartbeats, authenticated inference behavior, retired-route 404, Console sign-in, and the target current symlink before succeeding.

If the exact commit is already current, the script performs the complete service/process-path, private readiness, inference, and Console verification and exits as a no-op; it requires no configuration action.

Expected result: the command prints the full deployed commit, previous release path, and coordinated backup path. Gateway, Collector, agent runtime, and Console are active on the target release.

Deployment settings

Before any deploy or rollback command, set the installation-specific overrides in your workstation shell, from the repository root. These fictional examples override the helper's built-in installation settings. Replace every host, path, origin and Organization slug with the values for your reviewed deployment. Inspect the service account's installed executables and systemd EnvironmentFile declarations for the exact paths. Keep these exports in the same shell for the artifact and source-build procedures above and the rollback procedures.

export DEPLOY_HOST="mission-control-vps"
export DEPLOY_REMOTE_ROOT="/home/mission-control/mission-control"
export DEPLOY_CONFIG_ROOT="/home/mission-control/.config/mission-control"
export DEPLOY_STATE_ROOT="/home/mission-control/.local/share/mission-control"
export DEPLOY_NODE_BIN_DIR="/home/mission-control/.nvm/versions/node/v24.13.1/bin"
export DEPLOY_UV_BIN="/home/mission-control/.local/bin/uv"
export DEPLOY_GATEWAY_PUBLIC_URL="https://api.example.test"
export DEPLOY_CONSOLE_PUBLIC_URL="https://console.example.test"
export DEPLOY_ORGANIZATION_SLUG="acme"
Variable Meaning
DEPLOY_RELEASE_ARTIFACT / DEPLOY_RELEASE_MANIFEST Matching verified candidate files; together select artifact mode
DEPLOY_FALLBACK_ARTIFACT / DEPLOY_FALLBACK_MANIFEST Distinct compatible reviewed ancestor; required in artifact mode
DEPLOY_PYTHON_BIN Required exact remote interpreter path in artifact mode
DEPLOY_HOST Reviewed SSH host alias
DEPLOY_REMOTE_ROOT Root containing immutable releases and release symlinks
DEPLOY_CONFIG_ROOT Root containing protected service environment files
DEPLOY_STATE_ROOT Root containing durable data, backups and compatibility state
DEPLOY_NODE_BIN_DIR Directory containing the remote Node.js 24 executable
DEPLOY_UV_BIN Absolute path to the remote uv executable
DEPLOY_GATEWAY_PUBLIC_URL Deployment's public HTTPS inference origin
DEPLOY_CONSOLE_PUBLIC_URL Deployment's public HTTPS Console origin
DEPLOY_ORGANIZATION_SLUG Organization used by the Console verification probe

The public URL settings select verification targets; they do not configure DNS, tunnel routes, or the running services. Follow Change the public domain before using new targets. Until that cutover is verified, set both overrides to the deployment's existing HTTPS origins.

Remote preflight requires GATEWAY_PUBLIC_URL in both gateway.env and console.env (or their shared environment) to match DEPLOY_GATEWAY_PUBLIC_URL, and APP_URL in both services to match DEPLOY_CONSOLE_PUBLIC_URL. These are public HTTPS origins without a path, used by client snippets and OAuth discovery. The gateway still listens privately on 127.0.0.1:4010; runtime inference stays at http://127.0.0.1:4010, while management and heartbeats stay on private control at http://127.0.0.1:4001. The public origins do not replace these private routes.

The database location has no deployment setting: the helper reads DATABASE_URL from shared.env on the host during locked remote preflight. DEPLOY_DATABASE_PATH and DEPLOY_APPLICATION_DATABASE_PATH are retired; setting either locally stops the script before any Git or SSH step.

Remote paths must be absolute and cannot be /, contain dot segments, or contain shell syntax. Configured roots must be canonical non-symlink directories, distinct, non-overlapping, narrower than system roots such as /home or /var, and must not end in /.

Remote preflight checks that psql, pg_dump and pg_restore are installed, that the database answers select 1, that the server is PostgreSQL 16 or newer, that pg_dump and pg_restore have the server's major version, and that the backup filesystem has at least 1 GiB free beyond the database size. The helper passes the database password to these tools in the PGPASSWORD environment variable and the rest of the URL as their connection argument, so the password never appears on a command line. It therefore refuses a URL whose password contains a raw @ or /, or that carries a password= parameter. After staging and before the maintenance stop, the staged release also parses the URL with the platform's own driver (phase check-database-url).

Verification

The script fails unless archive checksums match, production builds pass, the pg_dump backup and its non-empty pg_restore --list catalog are written and the whole dump reads back, db:migrate succeeds and the migration check passes, all services start with valid file configuration, current resolves to the target commit, private control becomes ready, inference returns the expected authentication status, retired routes stay absent, and the Console probe succeeds.

Failure recovery

The script names the cause of each database problem and creates no database. These checks stop it before the maintenance stop and before it changes any service configuration, so the running release keeps serving:

  • gateway.env GATEWAY_PUBLIC_URL must match DEPLOY_GATEWAY_PUBLIC_URL (or console.env): align the service's canonical public gateway origin with the deployment probe target. An APP_URL must match DEPLOY_CONSOLE_PUBLIC_URL error requires the same correction for the Console origin in the named service. Follow Change the public domain and restart affected services after editing their environment. If an older helper instead reports GATEWAY_PUBLIC_URL must target inference on 4010, update the helper; do not replace a working public HTTPS origin with a loopback address.
  • DEPLOY_DATABASE_PATH is retired … or DEPLOY_APPLICATION_DATABASE_PATH is retired …: unset the local override.
  • shared.env still defines DATABASE_PATH; SQLite storage is retired … (or APPLICATION_DATABASE_PATH): complete Migrate from SQLite and remove the line.
  • shared.env must define DATABASE_URL, DATABASE_URL in shared.env must be a postgres:// or postgresql:// URL or DATABASE_URL in shared.env must carry its password percent-encoded in the user part (no raw @ or /) and no password= parameter: correct shared.env.
  • DATABASE_URL in shared.env is not a URL the platform can parse; percent-encode reserved characters in the user name and password (phase check-database-url): percent-encode the user name and password in shared.env. the staged release cannot load node-postgres (pg) to check DATABASE_URL in the same phase means that the staged release is incomplete: inspect it before you remove it and deploy again.
  • the Postgres database at DATABASE_URL is not reachable: check the server, the role's password and the host, port and database in the URL.
  • the Postgres server must be version 16 or newer: upgrade the server.
  • pg_dump is version … but the Postgres server is …; put the server's client tools first on PATH (or pg_restore): install the client tools of the server's major version on the host, first on the service account's PATH.
  • backup filesystem requires at least 1 GiB free beyond the database size: free space on the backup filesystem.

These failures come after the maintenance stop, with the services stopped:

  • storage_collation_refused: the database uses the collation … in the migrate phase: the database was not created with the libc C collation and character type, and nothing was migrated. Keep the services stopped. The database holds your data, so move it rather than recreating it empty: restore the backup directory the deployment printed as backup: with Restore into a new database, which recreates the database with the right collation. Then remove the staged release directories as described below and deploy again.
  • STORAGE_MIGRATION_REQUIRED in the verify-migrations phase: the migration did not complete. Keep services stopped, run db:migrate from the candidate release against the same DATABASE_URL, then remove the staged release directories as described below and deploy again.

For an artifact validation, pg import probe, Python version, disk or memory failure, correct the specific input/capacity issue before retrying. A partially staged release is preserved and must be inspected; the helper does not overwrite it, and staging refuses a release directory that already exists. Before you deploy the same commits again, inspect the failed attempt's release directories under DEPLOY_REMOTE_ROOT/releases/ and remove those that no link points at. If migration fails after the floor advances, old workers remain blocked. Keep services stopped and repair/resume the migration. Restoring the coordinated pre-writer backup is a separate explicit operation (scripts/rollback_vps.sh --restore-database, see Upgrade and rollback); never restore it over records written after candidate startup.

The script fails closed and performs no automatic rollback, restore, release deletion, or backup deletion. It prints the failed phase, target commit, previous release, promotion state, and backup path. Preserve that evidence and follow Upgrade and rollback before changing production state. A staging failure leaves the live release untouched; a later failure may leave services stopped or the migrated target promoted.