Operations#

Resource history, resumable logs, disk samples, alerts and authenticated Prometheus scraping are described in Observability. For the single Manager SQLite store, history retention, backups and Manager recovery, see Control storage and recovery.

Day-2 tasks for running applications and nodes. Deploying is covered in Deploying applications.

Control Memory#

The Control job reserves 4 GiB (MemoryMB: 4096) with MemoryMaxMB: 0, so the reservation is also its memory limit. Bootstrap and Manager upgrades use the same renderer and apply this default each time they submit the Control job. A manual Nomad job edit is not a persistent override across Manager upgrades.

Size the Manager for this reservation plus the operating system, Nomad, Traefik and any other workloads; 8 GiB is the recommended starting point. Before upgrading a smaller Manager, check its available schedulable memory. After upgrading, run nomad job inspect -json luma-control on the Manager and confirm that the luma-control task has Resources.MemoryMB set to 4096.

Inspect Applications#

luma status                                   # cluster, nodes and application health
luma app list --region cn                     # services and replica health
luma app show api                             # one application or service
luma app events api                           # recent runtime events of the latest allocation
luma app logs api --tail 100
luma app logs api --allocation <alloc-id> --previous
luma app logs api --follow --format ndjson

--tail is a 1-500 line budget shared by all selected sources. Text output labels every line with [allocation/task/stream]; [partial] and [continued] mark fragments. Use JSON or NDJSON to keep the source and cursor metadata. --follow resumes from the last cursor after a dropped connection (backoff up to 15 seconds, at most eight reconnects without progress); press Ctrl-C to stop.

Control also keeps a searchable history of build and deployment attempts, separate from application logs:

luma app history api --kind deployment --status failed
luma app history --source dashboard --since 2026-09-01T00:00:00+08:00 --format json
luma app history --id <record-id> --kind deployment
luma build logs <build-id>

Pages hold 50 records by default (maximum 100); pass nextCursor as --cursor with the same filters. Retention is described in Control storage.

On the manager, Nomad's own tools work too: nomad job status <app>, nomad alloc logs -f <alloc-id>.

Update Image Tag#

Change the manifest:

image: ghcr.io/me/api:2026-05-29-2

Then deploy:

luma deploy api.yaml

Scale Replicas#

Change the manifest:

replicas: 3

Then deploy:

luma deploy api.yaml

Temporary scale from a manager node:

nomad job scale api 3

Temporary commands do not update Git. Commit the manifest change afterward if it should persist.

Pin To One Node#

Use this only when the service depends on local disk, hardware, or a specific home/worker machine:

region: home
node: home-mac-mini

node must be the Luma node name passed to luma node join --name. Luma renders it as a Nomad constraint on ${node.unique.name} (or meta.luma_node_name) and still keeps the region constraint on ${meta.region}, so the selected node must also be in that region. The Nomad node identity is stable across rejoins, so pinned placement does not need to be refreshed.

Roll Back#

Nomad keeps a version history per job, so Luma exposes runtime rollback through both the dashboard and CLI.

From the dashboard, open https://<control-domain>/dashboard/, choose Applications -> Versions, inspect the job versions, then choose a previous version and confirm rollback.

From the CLI:

luma app versions <app>
luma app rollback <app>
luma app rollback <app> --to-version <N>

luma app versions lists prior versions of the Nomad job (GET /v1/job/<id>/versions). luma app rollback reverts to the previous version, or the version chosen with --to-version, through Nomad job revert (POST /v1/job/<id>/revert). Jobspecs also render update { auto_revert = true }, so a new version that fails its health checks rolls back to the last healthy version automatically.

This is a running job rollback. It does not rewrite Git history, change the stored manifest/YAML in Luma Control, reverse database migrations, or restore volume contents. Compose rollback applies to the whole Compose job/stack. Use pinned image tags or digests for production; mutable tags such as latest can make an old Nomad job version pull newer bytes.

Git-first path, when you want the manifest and the running job to stay in sync:

git revert <deploy-commit>
luma deploy <app>.yaml

Restart A Service#

Restart a running deployment without pulling a new image or changing its stored manifest. Restart is a delivery reconcile, not merely a process signal: Control waits for replacement allocations, refreshes Nomad CNI host-port state, reconstructs HTTP/TCP route files from the stored deployment record and actual allocation node, synchronizes DNS, and verifies every public HTTP endpoint before returning success.

luma app restart <app>
luma app restart <app> --service <task>
luma app restart <app> --mode task

There are two restart modes:

• recreate — stops the allocation so Nomad reschedules a fresh one (picks up placement/rescheduling). This is the default for a whole stack.
• task — restarts the task in place inside the existing allocation. This is the default when --service targets one task.

Omitting --mode uses recreate for the whole stack and task when --service is set; pass --mode explicitly to override. For a Compose application, luma app restart <app> --service <svc> restarts one service's task in place, while luma app restart <app> recreates every allocation in the stack. Use --timeout <seconds> (default 120) to bound the control-plane response wait.

From Dashboard → Applications → application → Services, each running service has a Shell action. Nodes have their own Shell action under Infrastructure → Nodes. Both open dedicated terminal pages; leaving the page ends that browser session. That opens an interactive terminal in the service container through the node agent (docker exec), using the same terminal supervisor as node shells. It is for live diagnosis, not a substitute for logs or a restart. System stacks are blocked. Node agents need the container-terminal capability; update them if the action reports that the agent does not support container terminal.

The response includes replacementAllocations and a structured delivery result (routes, dns, and probes). A platform-managed deployment with no saved record reports delivery reconciliation as skipped; a managed public deployment does not report delivery.status=ready until its public probe succeeds. Reconciled file-provider HTTP routes use explicit priority so they safely override stale legacy Nomad-provider routes that advertise an unreachable provider-private node address.

Restart refuses the system stacks traefik, egress, and luma-control (Control runs inside the luma-control allocation, so cycling it from application management would kill Control itself).

Remove A Deployment#

Use the deployed service or Compose application name:

luma app remove <app>

The control plane uses the manifest recorded during the last successful deploy, deletes the Luma-managed Cloudflare DNS record for public services, deregisters and purges the Nomad job (DELETE /v1/job/<id>?purge=true), and deletes generated manager files. The same command removes single-service and Compose deployments. Because the control plane stores the manifest, this also works for deployments created through the web UI when the client no longer has a local YAML file. For tailscale-relay, it also deletes /opt/luma/routes/<service>.yml. For cloudflare-tunnel, Cloudflare Tunnel public hostname cleanup is skipped because that hostname is still managed in Cloudflare Zero Trust.

Storage data is preserved by default. To intentionally delete removable storage referenced by the recorded deployment, preview and then run:

luma app remove <app> --dry-run --delete-storage
luma app remove <app> --delete-storage

For single-service deployments, this deletes managed storage paths referenced by storage.<volume>.path and removes named Docker volume objects declared in the manifest; bind mounts are skipped. For Compose deployments, this deletes managed volume subdirectories referenced by the sidecar, not the storage class itself. It cannot be combined with --skip-orchestrator.

Preview the cleanup without changing the manager:

luma app remove <app> --dry-run

Keep DNS or the running Nomad job when you are doing a partial cleanup. --skip-orchestrator leaves the Nomad job in place:

luma app remove <app> --skip-dns
luma app remove <app> --skip-orchestrator

If the control plane is unavailable, remove the Nomad job directly on the manager, then remove generated files:

nomad job stop -purge <service>
sudo rm -rf /opt/luma/stacks/<region>/<service>
sudo rm -f /opt/luma/routes/<service>.yml

Remove A Node#

From any logged-in client:

luma node remove <node-name>

The request is handled by Luma Control on the manager. It deletes the Luma node registration and drains the matching Nomad client (PUT /v1/node/<id>/drain); a dead client is then garbage-collected by Nomad automatically. If Nomad returns 404 or node not found for that ID, Control skips drain (nomadDrainSkipped: nomad_node_not_found) and still deletes the Luma registration. Network, auth, and Nomad-unavailable errors still fail the remove. Use this for stale nodes that already left locally, failed joins, or decommissioned worker/home machines. Manager nodes (Nomad servers) are protected and must not be removed through this command.

Because the Nomad node identity is a stable UUID, a worker/home machine that leaves and rejoins with the same Luma node name keeps the same meta.luma_node_name, so services pinned by Luma node name do not need a NodeID refresh. Keep manifests pinned by Luma node name; do not replace them with Docker hostnames.

Drain A Node#

nomad node drain -enable <node-id>

Restore it:

nomad node drain -disable <node-id>

Refresh Egress#

export EGRESS_SUBSCRIPTION_URL='...'
luma manager egress

Verify image pulls on the target node:

sudo docker pull hello-world:latest

Update Luma#

luma update                                  # manager: CLI + Control; node: CLI + agent; client: CLI
luma update --install-ref v0.2.37           # a specific release, branch or full commit
luma update fleet                            # every non-manager node with a ready agent
luma update fleet --include-manager          # explicit repair only

Updates install the latest release unless --install-ref is given; use a full commit for a coordinated candidate rollout. The dashboard's Settings → Maintenance is the preferred manager path: it mirrors the Control image into the internal registry first, then rolls out with progress and route checks. A direct luma update on the manager pulls the image itself; where GHCR is unreachable, set LUMA_CONTROL_IMAGE to a pullable image.

On the manager, luma update refreshes the firewall, Traefik, the Tailscale watchdog, the journald size limit, Control configuration and the luma-control job (with Nomad auto-revert), then refreshes the local node agent. It does not restart Docker or the Nomad agent, run egress setup, or redeploy applications. Manager state under /opt/luma/control is root-only; without passwordless sudo run sudo ~/.local/bin/luma update.

Fleet updates run up to four nodes at a time. A node whose installer or reconnect check fails is retried up to five times; from Settings → Maintenance, a node that is offline at the start is also waited for and retried. Each install keeps the new and previous (rollback) runtimes and anything a process still uses, and removes older releases/candidate.* directories that contain only src and venv; set LUMA_KEEP_RELEASES=1 to keep them.

On a joined node, luma update refreshes the CLI and the node agent. A node whose agent is too old for fleet updates is reported as skipped; run luma update on it once. For an old node without saved agent metadata, pass --control-url https://luma.example.com --token <node-join-token>.

Managers and nodes run a small Tailscale watchdog that restarts Tailscale after consecutive peer failures, without touching Docker, Nomad or applications.

Builder Lifecycle#

A builder is an ordinary joined Luma node selected with luma build config --node <name> --default-node <name>. Its node agent leases build tasks; Docker Buildx creates the luma-builder and luma-builder-egress engines on demand. No additional background startup script is required.

luma node join installs the node service. Linux uses luma-node-agent.service, enabled at boot with Restart=always. macOS uses the io.luma.node-agent LaunchDaemon with RunAtLoad and KeepAlive. Nomad has its own system service, and Docker manages BuildKit containers. On macOS, Docker Desktop or OrbStack must still be configured to start at user login; a running agent does not mean Docker is available.

Node join, agent updates and the standalone installer share the service definitions and node Tailscale watchdog installation in luma/agent.py. The standalone installer only refreshes a node with an existing agent.json; it does not turn an ordinary CLI client into a builder. Machine-specific startup patches outside the source tree are not part of the installation procedure.

On Linux, luma-node-tailscale-watchdog.timer starts checking about two minutes after boot and then about once a minute. It reads Manager addresses from Nomad configuration, checks Tailscale connectivity and actual TCP ports, and restarts local Tailscale after three consecutive failures by default. Check this layer when services have started but Control still reports the node offline. An SSH connection and the agent's connection to its Manager are separate paths.

The Linux node watchdog reads /etc/default/luma-node-tailscale-watchdog, which installation and updates preserve. Set LUMA_NODE_TAILSCALE_WATCHDOG_THRESHOLD (default 3), LUMA_NODE_TAILSCALE_WATCHDOG_PORTS (space-separated, default 4647), or LUMA_NODE_TAILSCALE_WATCHDOG_MANAGERS (comma-separated tailnet addresses; unset means read Nomad). Changes take effect on the next timer run.

systemctl is-enabled luma-node-agent.service luma-node-tailscale-watchdog.timer
systemctl status luma-node-agent.service luma-node-tailscale-watchdog.timer --no-pager
journalctl -b -u luma-node-agent.service -u luma-node-tailscale-watchdog.service
luma node list

Verify recovery using service state, a fresh Control heartbeat and builder ready status together. Large clock jumps at boot also warrant checking host/VM RTC settings and the system time service. A clock jump alone does not establish the cause of a network outage, and Luma does not require every installation to adopt a particular NTP provider.

Node Disk Cleanup#

Every node agent cleans its own disk once a day, starting about ten minutes after the agent starts. A run:

• removes Docker images that no container uses and that are older than 7 days;
• prunes build cache that has not been used for 7 days, in the default builder and in a running luma-builder / luma-builder-egress;
• on Linux, keeps /etc/systemd/journald.conf.d/luma.conf capping the journal at SystemMaxUse=500M.

An image is always kept when a container uses it (running or stopped), when a Nomad allocation on the node references it, when it is the luma-control image of the agent's or Control's release, when it is a helper that Nomad or Luma starts on demand (the registry.k8s.io/pause bridge-network image, ubuntu:22.04 or LUMA_HOST_PREP_IMAGE for host repair, registry:2 and the macOS Compose network image), or when its age cannot be established. Age is the later of the build time and the time the image was tagged on the host; on Docker's containerd image store, which does not record tag times, an image that appears after the agent was upgraded counts from the time the agent first saw it. If the local Nomad API cannot be read, no image is removed. Removal never forces: an image that a new container started using in the meantime stays.

Open a node in the dashboard and use Disk cleanup: Preview cleanup shows the images that would be removed without changing anything, and the remove button runs the cleanup after a confirmation. The same request from a script:

curl -fsS -X POST https://luma.example.com/v1/nodes/disk-cleanup \
  -H "Authorization: Bearer $LUMA_DEPLOY_TOKEN" -H 'Content-Type: application/json' \
  -d '{"node": "manager", "dryRun": true}'

Requests are dry runs unless "dryRun": false is sent. The node's last result (time, removed images, reclaimed bytes) is shown on the same card.

Configure a node in /opt/luma/node-agent/agent.json, or with environment variables on the agent service, which take precedence:

{"diskHygiene": {"enabled": true, "minAgeDays": 14, "journaldMaxUse": "1G"}}

Restart the agent after editing agent.json. luma bootstrap, luma update on the manager and luma node join write the journald limit directly, honoring LUMA_JOURNALD_MAX_USE.

Repair The Control Plane#

luma bootstrap --domain luma.example.com
luma doctor

Bootstrap is idempotent but touches Docker, the firewall, Traefik, Nomad and egress; treat a full rerun as a maintenance-window operation.

Manager IP Change#

Run on the manager, preview first:

luma manager ip-change --old 203.0.113.10 --new 203.0.113.20 --domain luma.example.com --dry-run
luma manager ip-change --old 203.0.113.10 --new 203.0.113.20 --domain luma.example.com

It verifies the new address over HTTPS, then updates only the manager's publicIp, providers.dns.edgeTarget and the Cloudflare A records that exactly match the old address. It backs up luma.yaml, reuses the running Control image and does not redeploy applications. Rerunning finishes any records left by a partial failure.

Registry cleanup candidates#

Control protects images referenced by current Nomad jobs, active allocations (including an old version still running during a rollout), and queued/running/finalizing builds or active agent tasks. Successful build history, inactive Nomad versions, stale deployment records and per-repository/system version counts no longer grant indefinite protection.

An unreferenced image starts an inactivity clock on its first complete inventory scan. After seven consecutive observed unused days it becomes a cleanup candidate. Reuse resets the clock; incomplete scans reset it conservatively and report unknown. Image creation time is not used as evidence of inactivity, and direct external Registry pulls are not tracked as workload usage. The clock is keyed by repository and digest in registry-usage.json under the Control state directory, separate from the disposable inventory cache, and survives Control restarts.

On upgrade, the legacy policy migrates to seven days and recommend mode: candidates are marked, not automatically deleted. Existing image ages are not backdated into the new clock. --max-age-days now sets consecutive unused days; the deprecated --keep-last and --system-keep-last options no longer provide exemptions. An operator can explicitly change the mode later. This change takes effect after upgrading Control; it does not require redeploying Registry or upgrading node agents.

Private Registries On New Nodes#

luma node join and luma node nomad-join prepare managed HTTP registry access after Docker and Tailscale are set up and before Nomad starts. The host list comes from the control plane's recorded HTTP transports. A narrow legacy exception remains for the configured Builder Registry at a registered node's Tailscale IP on port 5000, when neither an explicit transport nor registry credentials are recorded. Other private hosts, public registries and recorded HTTPS endpoints are not inferred to use HTTP.

Before a single-service or Compose deployment submits its final Nomad job, Control checks the managed HTTP hosts in the resolved Docker images on every eligible target matching the job's placement and healthy task drivers. Every target must advertise managed-registry-access-v2 before any target configuration changes. Regional deployments also check newly joined eligible nodes; an old agent, unknown runtime or unsupported placement constraint stops submission. Manager targets are verification-only: missing configuration requires host-owned maintenance, and this preflight never changes the manager's Docker or proxy settings.

For rolling upgrades, the corrected agent advertises both v1 and v2 and routes the unchanged public action and payload to one implementation. Existing v1 Control clients can use the corrected agent; the new Control requires v2 and rejects v1-only agents. Upgrade all eligible target agents before upgrading Control.

Linux setup uses the local /var/run/docker.sock and reads Docker's actual JSON fields HttpProxy, HttpsProxy and NoProxy. It adds exact managed hosts only when transport or active proxy bypass needs configuration, preserving unrelated settings and backing up daemon.json before a change. Missing proxy facts are errors, not evidence of direct access.

On macOS, Luma identifies the actual OrbStack socket owner. It manages HTTP transport in that user's ~/.orbstack/config/docker.json, but reads proxy policy from OrbStack's native network_proxy and network.proxy.exclude settings. In auto mode it also checks the effective macOS system proxy settings. Docker's internal OrbStack broker address alone does not prove that an external proxy is active. Where a verified upstream proxy needs a bypass, Luma preserves existing exclusions and appends only the missing managed hosts through the native settings owner. Unknown modes, unsupported PAC/WPAD configurations, incomplete inspection results and other macOS Docker runtimes fail explicitly. Active state is read back and each registry's /v2/ is probed over HTTP; a saved file alone is not a successful verification.

When existing settings and the active engine state already satisfy the request, verification can run while containers are running. Any required configuration change, including native proxy exclusions that can apply without a Docker restart, requires an idle engine; otherwise drain the node in a maintenance window and retry. Luma restarts Docker only when transport settings require it and reports whether a restart occurred. Agent updates alone do not configure or restart Docker; the next normal join or deployment performs the required verification and, where permitted, setup.

luma registry serve records whether the managed registry uses HTTP or HTTPS and deploys the registry service. It no longer configures every node in the cluster. Its own resolved image uses the same target preflight; consumers prepare access to the new registry on their own join or deployment.

Required Network Ports#

For Linux nodes, Luma configures UFW during bootstrap/join. Mirror the same access in cloud security groups or Tailscale ACLs:

Nomad agents bind to 0.0.0.0 and advertise their Tailscale address. Luma-managed UFW rules only open 4646/4647/4648 on tailscale0; public isolation depends on those firewall rules, not on the advertise address. On hosts without UFW, configure equivalent host firewall rules and cloud security groups, and verify the ports are unreachable through the public interface.

Tailscale Relay#

tailscale-relay is explicit per service. It is suitable for home tools, previews, or low-frequency internal panels that need a public domain.

Custom regions can opt in without moving their nodes: luma region update render --exposure tailscale-relay. See custom-region ingress for cluster upgrade order, Mac networking, validation and rollback. It is not the default path for normal public traffic.

Node agent fails after an installer path change#

A Nomad node can remain ready while its Luma agent is offline. Before installing missing Python packages, inspect the service's actual ExecStart, its launcher, and the Python environment it selects. Hosts may have a working installation under /opt/luma-cli alongside an incomplete user-local installation.

The installer validates the target runtime before publishing its command shim or refreshing the node agent service: dependency imports, CLI/agent command loading, and pip check must succeed. A failed package installation can use source fallback only when these checks pass. The shim runs the validated Python with -m luma.cli, not an existing console script whose shebang may refer to a different environment. Validation failure returns nonzero and does not publish an installation-success message, replace the shim, or refresh/restart the service.

This publication gate is not a transactional rollback of source or package changes already made inside the target install directory. For an affected existing host, restore the service to a verified working environment, then check both systemd restart counts and fresh Control heartbeats; a running process alone does not prove that the agent has reconnected.