Control storage and recovery#

Luma Control uses a local SQLite database on one Manager. The database is the authoritative store for Control state and its indexed operational records. The database directory must be on a local filesystem that supports SQLite locking and WAL; an NFS share is not a supported database location. This setup does not provide multiple active Managers.

A new luma bootstrap initializes SQLite directly and automatically. There is no intermediate control.json state, separate database service, connection string or migration command to configure. Luma uses Python's standard-library sqlite3 module; the CLI installer creates a Python virtual environment, and the Control image runs on a Python base image.

The default database is /opt/luma/control/control.sqlite3; set LUMA_CONTROL_STATE_DIR in the Control process environment to use another state directory. The control.sqlite3-wal and control.sqlite3-shm files are SQLite runtime sidecars, not independent backup artifacts.

Writer contention and task receipts#

Control admits local writers in FIFO order with one 30-second acquisition budget shared with SQLite. Read snapshots do not enter this queue. Slow writes log their owner, wait duration and held duration without SQL values. Database contention returns HTTP 503 with database_busy and Retry-After: 1.

Build scheduling reads task metadata without loading event streams. Build output is appended in batches; heartbeat receipt maintenance runs in bounded background batches. A completed build/deployment keeps its executor ownership while retrying only its captured terminal receipt after transient database failures. It does not rerun the external operation or turn a temporary receipt failure into a failed build. Process interruption still requires runtime inspection after restart.

New node agents send a monotonically increasing batchSequence per task. Control commits the sequence and events atomically, acknowledges repeated/older batches without appending them again, and preserves the existing bounded progress window. Older agents remain accepted during rolling upgrades; upgrade Control first and then node agents to enable duplicate suppression end to end. Progress remains observational: the agent can discard output after its bounded send attempts.

History cleanup preview computes references and fingerprints in a read snapshot, then briefly acquires the writer to recheck policy and save the plan. Applying a plan still rechecks references and fingerprints before deleting anything.

Local maintenance commands#

Run these on the Manager using the same installed Luma version and state directory as Control. They operate on local files and do not require a remote management token:

python -m luma.control.maintenance status
python -m luma.control.maintenance check
python -m luma.control.maintenance backup /secure-backups/control-2026-09-05.tar.gz
python -m luma.control.maintenance restore /secure-backups/control-2026-09-05.tar.gz --destination /srv/luma-control-recovery

The backup destination must be a new path outside the Control state directory. Restore requires a destination that does not yet exist and does not overwrite the live Control. status reports database integrity, schema and row counts; check verifies the active database, and check --database PATH verifies a specific database copy. Review command exit status as well as the reported integrity result. A successful archive records a recent-backup receipt for the storage inventory; its timestamp does not prove an off-Manager copy or a completed recovery drill. Keep the archive and verification metadata private.

What grows, and what retention means#

A page containing 300 builds is a query result, not a storage policy. Build metadata, event text, source archives, images and application data have different lifetimes and must be managed separately.

Removing a build record does not itself delete its image, source archive or application volume. Conversely, deleting an image can remove the ability to redeploy an old build even while its history remains searchable.

History queries and reviewed retention#

The Dashboard history page and luma app history query indexed build and deployment attempts with cursor pagination. List pages contain summaries; record details and execution steps are fetched separately. luma app versions NAME continues to read Nomad job versions for rollback and is a different history. Build retries create a new attempt ID linked by retryOf/retryRootId; the failed attempt and its events are preserved. Former global 100-build, 200-deployment and 300-build-event trimming no longer governs these records.

Manager inventory snapshots refresh about hourly even when the Dashboard is closed; the page shows each measurement timestamp and its coverage.

In Dashboard storage governance, inspect the measured inventory, adjust the history policy, then create a preview. The default policy is 90 days for summaries, 14 days for detailed events and a 24-hour review grace period. Changing the policy does not start automatic deletion. A history cleanup plan stores identifiers and fingerprints, not copies of the deleted payloads; keep a separate verified backup if you need recovery after applying it.

A preview records up to 1000 candidates and estimated serialized payload bytes; when its result says truncated, repeat the preview after applying the reviewed batch to inspect further candidates. After its grace period, apply that specific plan explicitly. Control rechecks terminal status, retry/current-deployment references and each record's fingerprint before deleting. Active and referenced records remain protected. Plans expire seven days after they become eligible. A policy change or expired plan requires a new preview; individual records changed since preview are skipped and reported in the apply result. Depending on age, a candidate loses only old detailed events or the summary together with its events.

The same reviewed plan includes resolved/closed alert incidents and final delivery records older than summaryDays (90 by default). Active incidents and outstanding deliveries remain protected, and the apply result reports alert history cleanup separately. This is not an automatic background deletion.

The API entry points require the management token:

Reclaimed payload estimates are not promised filesystem savings. SQLite normally reuses freed pages; the database file may remain the same size after record removal. This flow does not automatically VACUUM the live database.

Builder and Registry capacity#

Builder inventory and cleanup run on the selected Builder through its agent. POST /v1/governance/builder accepts node, operation, and when required planId plus confirmed: true; poll the returned task through GET /v1/governance/builder/TASK_ID. Supported operations are inventory, preview, quarantine, restore and purge. The Dashboard presents the same sequence and its eligibility times.

The scope is its configured content-addressed snapshot store, including source archives, analysis artifacts and build/external-image evidence. Manager inventory does not guess remote disk usage; unknown Builder, Registry, BuildKit, Trivy and application-volume usage remains unknown until measured through the appropriate owner.

Builder cleanup requires a fresh complete Manager reference inventory and no active build tasks. All known snapshot references remain protected; only old, unreferenced content can become a candidate. An explicit preview records file identities. After at least 24 hours, explicit quarantine moves the unchanged candidates into a recoverable area; final purge requires another 24 hours and fresh reference checks. Restore never overwrites an existing file. The plan's seven-day expiry and the exact eligible time are returned with the plan.

Builder plan metadata (.governance.sqlite3) and quarantine (.trash) live inside the Builder snapshot root. They are remote Builder data and are not included in a Manager Control backup.

Quarantine moves bytes on the same filesystem and does not free disk space. Final purge does. Symlinks, unrecognized paths, changed content, stale reference coverage or active tasks block cleanup rather than broadening the deletion scope. This flow does not clean temporary workspaces, BuildKit/Trivy caches, Registry blobs, secrets or application volumes.

Registry has its own retention policy and protected deletion/GC flow. Its default mode is recommend, with 20 recent images, a 30-day age policy and a seven-day recovery window. Automatic deletion requires explicit enforce configuration; changing Control history retention does not enable it. Registry manifest deletion and physical blob garbage collection are separate steps, and shared layers can remain in use by retained images.

Existing installations: legacy state import#

This compatibility path applies only when upgrading an existing installation with legacy JSON state. It is not part of a new Manager setup and requires no manual migration command. Use luma update manager for the first upgrade and allow a short Control maintenance window. Its configuration reads and image prefetch do not initialize SQLite. Immediately before import, the installer stops only the luma-control Nomad job, confirms that no allocation can still write JSON, and saves a private control-pre-sqlite-* checkpoint beside the state directory. Failure to confirm termination aborts without importing. The first database initialization imports the available legacy control.json state, retains a private control.json.pre-sqlite.bak, and records the import checksum and counts in control-sqlite-migration.json. After that import, the database is the sole source of truth. Do not edit the legacy JSON file expecting to update the running Control, and do not run old JSON-writing and new database-writing Control versions against the same state directory.

The first migrated Control job disables automatic rollback to the old JSON image. A failed rollout keeps the pending cutover marker so a retry remains protected. Ordinary later SQLite updates retain their existing automatic rollback behavior. Do not directly run a new database/backup command against the live legacy directory before this upgrade sequence; those commands can initialize the database without fencing the legacy process.

An old JSON-writing image cannot consume the migrated database. Rolling back to that image requires stopping the new Control and restoring the final legacy checkpoint plus its matching configuration and old job spec. Keep the migrated SQLite directory separately. This rolls back Control state to the checkpoint time; it does not undo application-side operations that happened afterward. See the release cutover and rollback procedure.

Import preserves records still present in the legacy files. It cannot recover builds, deployment history or metrics already removed by earlier count limits. An existing database is not repeatedly overwritten from a legacy JSON snapshot. The committed control-sqlite-authority.json records the database identity. A missing, truncated or mismatched database after cutover fails closed, even if legacy JSON still exists. Restore a verified backup; deleting the database does not switch the Manager back to JSON.

Backup scope#

A consistent SQLite backup must include transactions still present in WAL. Use the supported database backup operation; copying only the active database file can omit recent commits. Do not concatenate, edit or restore WAL/SHM files from another database snapshot.

The database contains credentials and private configuration. Backup artifacts must remain private, and should be encrypted when transferred or stored off the Manager. Commands should report paths, counts and integrity results; there is no need to print token or key contents for verification.

Dashboard passkeys (public keys only) and browser sessions (hashes only) are SQLite tables, so a restore brings back the sessions the snapshot held. If a backup may have been exposed, run luma passkey sign-out after restoring.

The Control archive includes a consistent database snapshot and supported files inside the Control state directory, including private token files and fleet/prepare/Registry recovery artifacts. It excludes live WAL/SHM, lock and temporary files. Legacy import files are retained as recovery evidence. Keep configuration/token rotation quiescent while creating the archive: the SQLite snapshot is transactional, while external files are copied separately. Files configured outside the state directory are not automatically included.

A complete Manager recovery set also needs the files outside that archive:

• Control service environment and any secret files configured outside the state directory, such as a custom metrics token path. Notification channel App Secrets stored in SQLite are already part of the database snapshot; cached tenant access tokens are not persisted and are obtained again after restart.
• The Manager's Luma configuration, generated jobs/routes, Traefik configuration and ACME certificate state, plus the desired deployment manifests from Git.
• Nomad recovery material appropriate to the cluster, and independent backups of application volumes, databases and any Registry/Builder data that must be retained.

Keep a dated inventory of paths and permissions alongside each recovery set. A successful database backup is evidence for Control database consistency; it does not establish that the other recovery items are complete.

Restoring or moving the Manager#

1. Stop writes from the old Control before switching the authoritative Manager. Preserve its database and configuration as the rollback checkpoint.
2. Verify the backup's integrity, then restore it onto local storage on the target Manager with the restore command into a new, nonexistent directory. Point the target Control at that restored directory, and restore external configuration and private files with their original permissions.
3. Start a compatible Control version with the restored state directory. Check authentication, expected cluster/node identities, retained history and database health before redirecting clients and node agents.
4. Check Nomad leadership/connectivity, agent heartbeats, generated ingress routes and actual application endpoints. A database integrity check alone does not prove runtime availability.
5. Keep the original checkpoint until the target has passed these checks. Do not restart the old Manager as another writer for the same cluster.

Test this procedure with an isolated recovery copy before relying on it for a production incident. This guide describes the implementation and runbook; it is not a claim that a live backup, migration or recovery drill has been run.