Skip to content

Upgrade and Rollback

Upgrade an existing current deployment

A Postgres deployment upgrades to a later release without resetting its current data. Stop writers, take and verify a backup, run the single migration step and restart the compatible gateway, Agent runtime, Collector and Console together. Keep configuration and purpose-separated secrets consistent. Verify private /readyz, an authenticated request and authorized/forbidden evidence reads before resuming clients.

The migration step runs from platform/ of the new release with DATABASE_URL set:

pnpm --filter @mission-control/database db:migrate

Expected result: Applied N migration(s); M recorded in total. A second run applies nothing. Only one process may migrate a database at a time. The existing-VPS deployment helper runs this step itself during maintenance, after its pg_dump backup, and then verifies the result before promoting the release.

Upgrade one minor version at a time and read that release's notes first; see Versions and compatibility. Services never migrate on their own. The gateway, the Console and connector:rotate-credentials check at startup that every migration their release bundles is recorded in the database, and refuse to start otherwise with STORAGE_MIGRATION_REQUIRED: run pnpm --filter @mission-control/database db:migrate. Run the migration against the same DATABASE_URL and start them again. Bootstrap applies missing migrations itself before it writes; config:check does not check migrations. Migrations are additive: the previous release keeps running against a migrated database, which is what makes a code-only rollback safe.

With several gateway instances, the upgrade rolls: run the migration step once, then restart the gateway instances one at a time, waiting for each instance's /readyz/traffic to answer 200 before the next (Run several gateway instances). Instances of the previous release keep running against the migrated database meanwhile. The database suite checks that every migration after the baseline is additive (no table or column drop, rename, type change, set not null or not-null column without a default); a migration that cannot be, marked -- breaking: in its first line, requires a full stop of every instance and says so in its release notes.

Startup then reconciles built-in preset role permissions against the current registry, so a release that adds a permission reaches existing Organizations without administrator action; each change is audited.

An installation that still runs a SQLite release follows the one-time Migrate from SQLite procedure instead of this section.

Evaluation release compatibility and fallback

Evaluation case settings, comparison groups, immutable sample requesters and bundle state require workers that understand their operating contract. Additive migrations alone do not make an older worker safe. The artifact deployment installs a durable startup guard for all four services and advances runtime-contract.json before applying migrations. Missing/corrupt state or an incompatible release fails closed before a worker starts; reinitialization never lowers an existing floor. For a Postgres release, the same start guard refuses a shared.env that defines DATABASE_PATH or APPLICATION_DATABASE_PATH, or lacks a postgres:// DATABASE_URL. A pre-Postgres (SQLite) release is exempt, so the manual rollback to the SQLite release starts through the same service units.

service-release selects the executable/working directory/guard target atomically. current names the last verified healthy release. A compatible fallback is staged and checked before maintenance. Both paths keep the current database and catalog; they do not replay a stale database backup unless you ask for one.

Roll back to an earlier release

A rollback is code-only by default: the earlier release runs against the current, migrated database, because migrations are additive. Run from the repository root of your checkout with the reviewed compatible release path printed by deployment:

ROLLBACK_RELEASE=/home/mission-control/mission-control/releases/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb scripts/rollback_vps.sh

Replace the fictional SHA and service-account paths with the actual retained fallback. The command reads the same DEPLOY_HOST, DEPLOY_REMOTE_ROOT, DEPLOY_STATE_ROOT, DEPLOY_CONFIG_ROOT and DEPLOY_NODE_BIN_DIR settings as the deployment; retain your explicit deployment settings in this shell. It takes the deployment lock and, before stopping anything, checks that no service-release.next or current.next link is left over from an interrupted operation, and that the target is a Postgres release that passes the runtime floor and the shared.env contract, matches its manifest checksums and imports its Python runtime. It refuses a pre-Postgres (SQLite) release as its target: returning to SQLite is the manual procedure in Roll back to the SQLite release. It then stops the four services, switches the launch link, and requires all four services, process paths and private readiness before updating current. Without --restore-database, it does not restore databases or lower the compatibility floor. Other services are not stopped. Preserve a failed activation for inspection; there is no automatic reverse switch.

Expected result: Activated compatible fallback <release>; the database was not restored (code-only rollback). A check that fails before the stop ends with Nothing was stopped or restored. and exit status 1. A held deployment lock ends the command with Deployment lock is held; inspect the active operation.

Restore the database with the rollback

To also return the database to the state of a deployment backup, name that backup directory. Use the Backup: directory printed by the deployment you are undoing: its name is pre-<first 7 characters of that deployment's commit>-<UTC timestamp>, and it holds the database as it was just before that deployment migrated. As ROLLBACK_RELEASE, name a release whose migrations the backup records, such as the release that was current before that deployment. From the repository root of your checkout:

ROLLBACK_RELEASE=/home/mission-control/mission-control/releases/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb \
  scripts/rollback_vps.sh --restore-database \
  /home/mission-control/.local/share/mission-control/backups/pre-ccccccc-20260924T120000Z

The directory must be a pre-… directory directly under DEPLOY_STATE_ROOT/backups: a deployment backup, or a safety dump described below. Any other value exits with status 2 before contacting the host. Before stopping anything, the command makes the code-only checks above and these, and refuses with Nothing was stopped or restored. when one fails:

  • The backup's mission-control.dump exists, matches the SHA-256 recorded beside it (checksums.json for artifact deployments, mission-control.dump.sha256 for source deployments and safety dumps) and reads completely, so a truncated or corrupt dump is refused.
  • The dump's migration history records every migration the target release bundles. Otherwise the message names the missing migrations: choose the backup taken for that release's migrations, or roll back to the release the backup belongs to.
  • shared.env defines a postgres:// DATABASE_URL with its password in the user part and no SQLite paths, the target release's node-postgres accepts the URL, the database is reachable, and psql, pg_dump and pg_restore on the host have the server's major version.
  • The backup filesystem has room for a safety dump of the current database: at least 1 GiB free beyond the database size. Nothing else is written to disk; the dump is streamed into the restore.

The command then stops the four services and:

  1. Writes a safety dump of the current database to a new pre-restore-<UTC timestamp> directory under DEPLOY_STATE_ROOT/backups: the dump, its catalog and its SHA-256, readable only by the service account. It prints Safety dump of the current database: <path>.
  2. Replaces the database with the backup in one transaction. It drops the drizzle and public schemas, recreates public as a new database has it, and loads the dump; the transaction commits only after the whole dump has been read. The result equals the backup exactly: tables, columns and rows that later migrations or writes added are gone.
  3. Prints the command that undoes the restore: To undo the restore: ROLLBACK_RELEASE=<previous release> scripts/rollback_vps.sh --restore-database <safety dump>.
  4. Runs the target release's migration check, then switches the links, starts the services and verifies them as a code-only rollback does.

Expected result: the safety dump line, the undo command, Restored database migrations are current for the target release, and Activated compatible fallback <release> with the database restored from <backup>; the replaced database is in <safety dump>.

The restore has three requirements:

  • The DATABASE_URL role must own the database, as Bootstrap and first sign-in creates it, because the restore drops and recreates the public schema.
  • The server's lock table must hold every lock of the one transaction: about 6,700 for the current schema, against a default capacity of about 7,800. Raise max_locks_per_transaction to 256, which needs a PostgreSQL restart, as described in Migrate from SQLite.
  • The disk of the server's data directory needs free space of about twice the database size: until the transaction commits, the server keeps a second copy of the data and its write-ahead log.

If the lock table or the disk runs out, the restore fails and its transaction is rolled back; a full disk can also make the PostgreSQL server restart. Restore into a new database is then the fallback.

A failure never leaves a half-restored database:

  • If the safety dump fails, nothing is restored and the database is unchanged. The services stay stopped and the release links unchanged; the message prints the command that starts the services again.

  • If the restore fails, the message shows the first error lines and, when the dump could not be read to its end, says that no COMMIT was sent. The command then asks the server what happened to the restore's transaction and reports the outcome:

    • …failed and its transaction was rolled back …: the database is unchanged., or …failed before its transaction changed anything …: the database is unchanged. When the error names the lock table (out of shared memory or max_locks_per_transaction), raise max_locks_per_transaction as the Postgres superuser, restart PostgreSQL and run the command again. Otherwise correct the named cause and run it again. Either way, you can instead restore into a new database as the Postgres superuser.
    • …committed, but psql reported a failure afterwards …: the database now holds the backup. The safety dump returns the database to its state before the restore.
    • Whether the restore of … committed is unknown (transaction <id>, …), followed by the query to ask the server once it settles: select pg_xact_status('<id>'::xid8);. It prints committed when the database holds the backup and aborted when it is unchanged.

    In each case the services stay stopped, the release links unchanged, and the message names the safety dump. - If the restored database fails the target release's migration check, the services stay stopped and the release links unchanged; the safety dump returns the database to its state before the restore.

A database restore discards every record written after the backup was taken, including Requests, usage and audit. Prefer a code-only rollback or a forward fix once the new release has served traffic.

If a migration fails before candidate writers start, keep services stopped and repair/resume it. After candidate writers have run, use a compatible fallback or forward fix; an old snapshot would lose new records and is not an ordinary rollback.

A rollback across the SQLite-to-Postgres cutover is a separate manual procedure: see Roll back to the SQLite release.

See Artifact deployment for required versions, resource preflight, exact files and failure recovery.

Earlier development cutovers

The historical Trace cutover describes an earlier irreversible boundary and is not the current activation procedure. There is no compatibility mode for old Agent keys/Project authority, and no automatic general old-data migration.