Observability#

Luma exposes node/container samples, current task queues, resource history, runtime events, persistent alerts and resumable application logs. These observations are not a measurement of end-user availability. Public-route verification during deploy/restart remains a separate capability.

Console navigation#

Observability has separate pages for alert incidents, application monitoring, resource metrics, logs, alert rules and notification channels. Each uses its own compact page title. The engine summary belongs to alert incidents; notification channels focus on channel configuration and delivery history. Built-in alert pages refresh every 15 seconds, and resource history refreshes every 30 seconds. See the console guide for UI workflows and the distinction between route relationships and connectivity probes.

Application alerts (luma-observe)#

Control dashboards show node/container samples. Live allocation tails remain an on-demand Control proxy. Application HTTP 5xx, latency (p90/p95/p99), Nomad failed allocations, restart storms, traces and searchable stdout/stderr are an optional observe/ Compose app. Deploy it with Luma; skip it and Control still runs. Operators look at Dashboard → Observability → App monitoring, which embeds Grafana at /grafana on the Control domain. HTTP, Nomad and trace series are labeled with the Luma app/stack name as soon as observe is deployed; applications do not opt in. Failed gauges are the allocations Nomad still wants to run, not lifetime failure counters. The stack uses host networking and loopback listeners (127.0.0.1:8082 Prometheus, 127.0.0.1:4318 OTLP, 127.0.0.1:3200 Tempo). Application tasks send traces to the manager Tailscale mesh listener on port 4319 with a Control-issued bearer token. Control uses host networking and queries http://127.0.0.1:8428 when observe is present. Grafana on 127.0.0.1:3100 is local debug only. The stack's bundled rules evaluate in vmalert/Alertmanager without Control SQLite. Separately, Control's dashboard-managed application alert presets read VictoriaMetrics and persist incidents and deliveries in Control SQLite. Traces retain 3 days; logs and metrics retain 15 days, with logs capped at 4 GiB. Logs and metrics stop ingesting when less than 2 GiB of the manager disk is free.

Sign in to /dashboard/ before opening /grafana. By default, Traefik checks the console session or management token with Control on every Grafana request; unauthenticated requests return 401. Grafana needs no second login, and the authenticated route depends on Control being available. See sign-in and passkeys.

External dashboard sharing#

Open Observability → Settings → Allow public dashboard links in the Control console. The switch is off by default. Enabling it opens a dialog that requires a sharing password of 8–128 characters. Canceling leaves sharing disabled. Control saves only a salted password hash and applies the routing change within a few seconds; no Observe or application restart is required. Password-protected settings survive Control restarts. Upgrades disable old passwordless sharing settings and remove their unauthenticated route until a password is set.

After enabling the switch, enable Share externally on each intended Grafana dashboard and share its /grafana/public-dashboards/<access-token> link. Turning on the Control switch does not share any dashboard. Give visitors the sharing password as well as the link. They enter it on a dedicated page before any dashboard data can load; no Luma account is needed. Grafana checks that the link is enabled and executes only the panel queries saved in that dashboard. Disabling or deleting the share blocks subsequent reads. Turning off the Control switch requires Luma sign-in for all public links again. Change password invalidates existing guest sessions, as does disabling sharing; re-enabling always requires a password. Guest sessions expire after eight hours and cannot access the console or its management APIs. Existing console sessions and management tokens continue to work. The route does not cache dashboard pages or responses.

The allowlist covers that page, its read-only public API (including the POST panel-query endpoint), and static assets. Dashboard search, normal /d/ pages, Explore, datasource proxies, /api/ds/query, and share management still require Luma sign-in. Control checks the guest or operator session on the page, API and static-asset routes before cookies and authorization headers are removed from the request sent to Grafana. A Control outage blocks these routes. Grafana plugins or later versions may require additional static paths; verify them without opening the general /api/ prefix.

Public page/API traffic shares a per-host rate limit of 5 requests/second with a burst of 32, a limit of 16 in-flight requests, 16 KiB query bodies, and a 15-second upstream response-header timeout. Static assets have a separate route. Overload returns an error instead of building an unbounded request queue. Metrics continue to use VictoriaMetrics' existing result cache and execution/memory limits; this adds no query backend, database, or application-side cache. Password verification accepts at most five attempts per client and 30 total per minute; excess attempts return 429. Verify password entry, a panel query, an invalid/revoked link, password rotation and a protected API after enabling the switch. Administrators can also read/update the setting through authenticated GET / POST /v1/dashboard/observe/settings, with the body {"grafanaPublicSharingEnabled": true, "grafanaPublicSharingPassword": "your sharing password"} to enable or change the password, or {"grafanaPublicSharingEnabled": false} to disable. Responses expose only the enabled flag, never the password or its hash. Control manages the separate luma-observe-grafana-public.yml file in its existing Traefik route directory; operators do not need to edit files. The bundled allowlist targets Grafana 11.5.x.

Deploy from observe/ through its recorded workflow. For a first Local Build on an AMD64 manager, use luma build local . --platform linux/amd64; resolve the actual manager architecture for other hosts. First deploy does not require notification secrets. For the stack's Alertmanager path, add scoped secrets and matching Compose environment references, then redeploy. For Dashboard rules, configure the built-in Feishu channel instead; see notification setup. Refresh Traefik after the current CLI includes the loopback metrics/OTLP flags so RED rules have a scrape target. Operators look at Dashboard → Observability → App monitoring. The charts are per Luma app (Traefik HTTP rate, 5xx and p90/p95/p99 latency + Nomad health), not each container's /metrics. Traefik histogram buckets are 0.05,0.1,0.25,0.5,1,2.5,5,10 seconds so those quantiles are distinguishable. Selecting observe also injects official OpenTelemetry environment variables into later Luma deploys (OTEL_SERVICE_NAME, luma.stack / luma.task / luma.region, OTLP HTTP to the mesh listener). Apps that already ship an OpenTelemetry distro emit spans without a Luma SDK. Redeploy existing apps after observe is first enabled. Apps export traces fail-open; the collector keeps HTTP 5xx, OTel errors, and traces slower than 1s, plus a 10% baseline. The collector uses a Go memory target, a memory limiter and bounded queues; sustained overload can refuse or drop telemetry. Do not ship raw access logs or traces off the manager public interface.

Trace storage sizing and upgrades#

The bundled Tempo 2.10.10 includes the upstream Parquet row-pool fix. Its default memory reservation is 1 GiB, container ceiling 3 GiB and Go memory target 2 GiB; Parquet row groups target 25 MB. Budget the ceiling alongside Control, Traefik and the other observe services before first deployment. A low memory ceiling can cause repeated file-page reclaim and disk reads without an OOM or full disk. Query buffer limits do not cap compaction memory. Tempo's 72-hour retention and ingest rate are not a disk quota, and Tempo has no free-space ingestion guard.

Rebuild and redeploy the complete observe/ stack from updated source to receive these defaults. Updating CLI/Control or restarting an old job does not update observe. A previously disabled Tempo also needs the collector's Tempo exporter restored. Preserve existing volumes; expired blocks are deleted asynchronously. See the first-deployment, upgrade and verification steps. CI exercises empty-storage startup, collector delivery and persisted trace retrieval after a Tempo restart using the packaged configs and memory limits.

Application integration#

For standard native metrics, independent machine credentials, external OTLP producers and application-owned dashboards/rules, see Standard application observability.

Deploying observe/ is the cluster opt-in. After that, new Luma deploys get OTel wiring automatically. A process restart of an old job is not enough; the Nomad job must be rendered again. Applications can use the optional luma-observe-sdk Python package or the official OTel SDK directly. Language agents are installed by the application's own image.

What you get without changing the app:

• Traefik HTTP RED (rate, 5xx, p90/p95/p99) labeled by Luma app name
• Nomad running / current failed / restart counts labeled by Luma app name
• An ingress span for public HTTP (cn-edge / external-edge) stored in Tempo

What you get after a new deploy, only if the image already speaks OpenTelemetry:

• Process spans exported to Tempo with luma.stack / luma.task / luma.region
• Look at Dashboard → Observability → App monitoring → Traces, or Grafana /grafana/d/luma-traces

What you still do not get: nginx / static Go / images with no OTel SDK, traffic that never hits Traefik, request bodies, SQL text, or every successful fast request (5xx and >1s traces are kept; the rest are sampled at 10%). Node agent connections to Control (/v1/node-agent/, /v1/terminal/) are held open by design, so they are left out of trace storage and only counted in Traefik metrics. Export is fail-open.

Manual spans#

Do not set OTEL_EXPORTER_OTLP_ENDPOINT or the bearer header. Control already injects them. Use the official OpenTelemetry API and keep exporters fail-open.

Install the official distro plus OTLP exporter in the image, for example Python:

opentelemetry-distro
opentelemetry-exporter-otlp

Then start with opentelemetry-instrument (or the equivalent for Node/Java) so the SDK reads the injected env. Add a span around a business step:

from opentelemetry import trace

tracer = trace.get_tracer("orders")

def charge(order_id: str) -> None:
    with tracer.start_as_current_span("charge_card") as span:
        span.set_attribute("order.id", order_id)
        provider.charge(order_id)

Node:

const { trace } = require("@opentelemetry/api");
const tracer = trace.getTracer("orders");

async function charge(orderId) {
  return tracer.startActiveSpan("charge_card", async (span) => {
    span.setAttribute("order.id", orderId);
    try {
      return await provider.charge(orderId);
    } finally {
      span.end();
    }
  });
}

Do not put secrets, tokens, or full request bodies on span attributes. Custom span names show up under the same luma.stack as the process. If the image has no OpenTelemetry SDK, these calls are no-ops unless you add the distro.

Dashboard and logs#

Dashboard → Observability opens incidents. Metrics have a dedicated page; the Logs view of App monitoring opens Grafana Explore against VictoriaLogs (15 days). Rules and notification channels have separate list and edit URLs. Storage governance is under Infrastructure → Storage → Capacity and cleanup.

The metrics page shows the actual sampled time span and retention, separates missing/stale/failed queries, and breaks resource lines across sampling gaps. Select a service for its resource history, or open observability logs filtered by its stack.

Observability logs are Nomad allocation stdout/stderr shipped by luma-observe into VictoriaLogs. Filter by app (the Nomad job / Luma stack). Traefik JSON access logs are the traefik job's stdout; unpack with LogsQL unpack_json. Line time is observation time because Nomad frames have no event timestamps. luma app logs and the application-detail live tail still follow Nomad files through Control and are bounded excerpts, not this 15-day store.

Resources are retained in 30-second buckets, independently of the number of nodes reporting a service. LUMA_METRICS_HISTORY_POINTS defaults to 720 (approximately six hours), bounded between 60 and 10000. Resource history remains a separate bounded metrics-history.json file in the Control state directory; collection outside Control is still useful for longer investigations and independent outage detection. See Control storage and recovery for backup scope. A service aggregate covers reporting nodes; missing contributions expire after 180 seconds. It is not proof that all replicas have reported.

Built-in alert rules and notifications#

The Dashboard alerting page manages rules, notification channels and incidents. Rules, incident transitions and the notification outbox persist in the Control SQLite database. The evaluator uses samples and task state already reported to Control; creating a resource rule does not start a public HTTP probe.

Available presets cover service container memory, node heartbeat age, sustained CPU, memory, disk and inode usage, queued-task age, the latest failed build per application, and — when luma-observe is deployed — application p95 latency, HTTP 5xx ratio and Nomad failed allocations. Observe samples are instant PromQL reads from VictoriaMetrics; if observe is down those rules keep their last state (noData: keep) instead of firing. Thresholds and the required continuous duration are configurable. A pending incident becomes firing only when its condition lasts for that duration; a healthy observation resolves it. One active incident is maintained per rule and target, with a configurable repeat interval for continued failures.

First task alert on a new cluster#

The public Control image and luma-infra distribution include these presets. After normal cluster bootstrap, an empty alert database initializes its tables automatically; no private configuration, old database or manual SQL is required. Other projects' rules, notification groups and credentials are never seeded.

1. Deploy the shared Observe stack. Updating a local CLI alone does not deploy Observe or upgrade Control.
2. Instrument application processes with the public Python SDK and the appropriate adapter. Explicitly set OTEL_METRICS_EXPORTER=otlp; Control still defaults it to none. Preserve injected OTLP endpoints, bearer headers and the luma.stack resource attribute. Traces alone are not task metrics.
3. Run luma observe tasks APP --view overview --since 1h --format json and check actual heartbeat and metric coverage. Use the reported OTel service_name, which can differ from the Compose name. Queue-wait metrics currently come from Taskiq; another framework must explicitly emit the same signals.
4. Configure your own Feishu application and group under Observability → Notifications. Create a rule with its metric, application, service, threshold and channel. Heartbeat rules should alert on missing samples; disable them when retiring the monitored service.
5. Check the last evaluation, incidents and delivery records. A configured channel does not establish delivery. Send a channel test explicitly when group-message verification is wanted.

An idle new worker can have no queue distribution yet. Missing telemetry is not zero. Resource alerts remain usable without SDK instrumentation.

Task SDK presets are app.task_heartbeat_age, app.task_queue_wait_p95, app.task_telemetry_dropped and app.task_tracking_overflow. Select one application and optionally scope it with taskQuery.service, using the OTel service_name from its heartbeat table. Heartbeat rules require the expected continuously running service. Up to 20 task metric rules are supported; bounded queries run outside SQLite and identical queries share a read.

Heartbeat age uses the newest process heartbeat within that service. The default is over 120 seconds for 180 seconds; use noData: alert to detect disappearance of all series. This can indicate a service or telemetry outage. Disable the rule when retiring a service. Queue p95 uses consumed-task waits over 15 minutes, requires at least 20 observations per service/queue, and takes the largest group. Observed low traffic clears the latency condition; missing telemetry does not become zero. This is not a backlog measurement. SDK drops count lifecycle log drops and instrumentation errors over 15 minutes. Tracking overflow sums untracked active tasks only in processes with fresh heartbeats; neither metric covers all OTLP or Collector losses.

These presets read existing VictoriaMetrics data and do not require redeploying instrumented applications or observe. Rules and notification destinations must be explicitly configured. A recovered metric condition does not establish that historically failed business tasks were successfully compensated.

The Service container memory near limit preset (app.memory) takes the highest Docker-reported memory percentage across an application's containers and replicas, including all tasks in a Compose stack. It does not average away a hot container. The default is above 80% for 2 minutes; select the application (for example, luma-observe) and a notification channel in Alert rules, then save the enabled rule. New installations include the preset; no rule or channel is created automatically. Upgrading Control adds it without redeploying observe or upgrading existing Agents that already report container statistics.

This preset uses Agent samples independently of the observe stack. Control resolves Nomad allocation IDs to application names before evaluating, outside the database transaction. Heartbeats and container samples must both be at most 120 seconds old, with a positive memory limit. Missing or invalid samples prevent recovery when a known replica lacks data; unresolved application identity also prevents claiming a healthy maximum. A fresh container already above the threshold still proves a breach even if other samples are missing. With noData: keep, missing data preserves firing incidents and breaks the pending duration. Docker's Linux usage excludes inactive file cache; this is an early warning against the Docker-reported memory limit, not a measurement of disk pressure or all cgroup cache. Keep host memory and disk alerts as well.

Build-failure cards and incident details include available service names, build IDs, full image references (per service for Compose), image repositories, builder nodes, source repositories and refs, failed stages, failure times and error summaries, with an entry to the build logs. The failed stage distinguishes image building, uploading and deployment. Missing image references are explicitly marked as unrecorded; a repository alone is never presented as a complete tag. Long errors retain their beginning and end; full output stays in the build logs.

Reminders refer to an existing failure, not a newly started build. Failure snapshots remain available after recovery. Following a Control upgrade, active failures with available build records are enriched on the next evaluation and subsequent notifications use the detailed format; messages already sent are unchanged. The Dashboard ships in the Control image, so this feature requires a Control/Manager upgrade without requiring Builder or other node Agent updates.

Missing data is explicit: the keep policy preserves an existing firing state without treating missing samples as recovery, while alert treats missing data as a condition to evaluate. CPU rules use fresh Linux host CPU observations; macOS load averages are not interpreted as CPU percentages.

Incident state and notification control are separate. pending waits for the duration, firing means the condition triggered, resolved requires observed recovery, and closed records an administrative end. These actions never start or retry a build.

Each observed new failed build gets its own incident. While a later build is queued or running, the old failure remains intact and waits for the outcome. A new failure closes the old incident as superseded, preserving its snapshot; the new incident does not inherit its acknowledgement. A later successful build resolves the current incident. A canceled retry leaves the old failure and its reminder policy active. Manual build dismissals survive Control restarts, rule re-enabling, and history cleanup. Metric dismissals end on observed recovery or changed rule conditions, allowing subsequent breaches to alert.

The list and detail page share the same actions. Management endpoints are POST /v1/alerting/incidents/{id}/ack and POST /v1/alerting/incidents/{id}/close. Both accept expectedBuildId to guard stale actions. Closing does not send a recovery notification. No action can retract an HTTPS request already in flight. Changing rule conditions closes old active incidents with a rule-change event; deleting a rule also removes its dismissals. Hiding a Dashboard overview card in one browser is only a display preference and does not silence alerts.

The built-in Feishu channel uses an application bot. Fill in App ID, App Secret and the destination group chat ID in the channel form. Enable the application's bot capability, grant and publish im:message:send_as_bot, and add the application bot to that group. Associate the saved channel with a rule, then explicitly send a test notification. Channel tests send a real message to the configured group and bypass silences.

Luma requests the tenant access token automatically and caches it only in process memory, refreshing before expiry. Operators do not configure a tenant access token. Channel reads return App ID, chat ID and appSecretConfigured; they never return App Secret. The App Secret is stored in the private Control SQLite database and therefore belongs to its protected backup scope. Omitting App Secret while editing preserves the stored value; replace it to rotate the credential and obtain a new cached token.

Evaluation runs independently about every 15 seconds. A separate delivery loop checks the persistent outbox about every second, so a slow provider request does not pause evaluation. Pending, sent, retrying and failed deliveries remain visible. Network/rate-limit failures retry with backoff up to eight attempts; invalid credentials, missing permission and an inaccessible group fail with an actionable category. Each HTTPS request has an eight-second timeout within a 16-second delivery budget.

A successful response confirms acceptance by Feishu, not that a human read the message. Outbox retries reuse a stable message UUID for provider deduplication; this remains at-least-once delivery, not an unlimited exactly-once guarantee. Check delivery history for failures before changing a rule to make it fire again. Resolved/closed incidents and final deliveries enter the reviewed storage retention plan after its summary retention period (90 days by default); active incidents and outstanding deliveries are protected. See Control storage.

Alerting management APIs under /v1/alerting/ require the management token; the metrics-only token cannot manage them. No notification channel is provisioned automatically. The Control must remain available to evaluate and deliver its own alerts; retain an independent Prometheus scrape or external probe for Manager/Control outages. Built-in resource alerts do not probe public URLs. Application p95, 5xx ratio and failed-allocation presets read luma-observe; they are not a substitute for an external Control-down probe.

Host disk samples#

Updated node agents sample the filesystem containing /opt/luma, or / if that directory does not exist. Set LUMA_METRICS_DISK_PATH in the node agent service environment to select another absolute path. The Dashboard identifies the sampled path; this does not measure every mount, remote NFS server capacity or Docker VM capacity on macOS. A missing filesystem or unsupported inode counter is unavailable, not zero usage.

Disk usage excludes reserved blocks from usable capacity, matching the usual df interpretation. The available-bytes value excludes reserved blocks too. Existing agents continue to work but will not supply these additional samples until updated.

Prometheus endpoint#

GET /v1/metrics emits Prometheus text exposition. It reads the persisted heartbeat snapshot without contacting Nomad, contacting agents or changing control state. It supports the management token. A remote collector should use a dedicated read-only metrics token.

The bundled luma-observe collector colocates on the manager host network and scrapes http://127.0.0.1:8080/v1/metrics without a token — the same loopback trust as Traefik :8082. A request that presents an Authorization header is still validated. Remote Prometheus scrapes continue to need the dedicated token.

On the manager, provision a random token of at least 32 ASCII characters in /opt/luma/control/metrics-token. The file must be a private regular file, readable by the Control process (typically mode 0600), not a symlink. The existing /opt/luma Control mount makes this default path visible to the container. It is read on each request, allowing atomic rotation without restarting Control. No token is created automatically.

An explicitly configured LUMA_METRICS_TOKEN_FILE overrides that path. For custom Control installations, expose the absolute path inside the process/container and set the environment variable there; an arbitrary manager shell environment is not automatically forwarded to the Nomad job.

Copy the token securely into the Prometheus collector's credentials file. This dedicated token is accepted only by /v1/metrics; it cannot query the Dashboard, read application logs, deploy or restart. Keep it distinct from every other Luma token. Endpoint output includes infrastructure names and region labels. Remote scrapes remain authenticated; the colocated luma-observe scrape is loopback-only.

The example Prometheus scrape config and alert rules are opt-in templates. Replace the hostname and credential path, check with your Prometheus version, and configure an Alertmanager destination before expecting notifications. These external templates are separate from Luma's built-in alert rules and Feishu channels described above. Applying Prometheus/Alertmanager configuration is a separate infrastructure change; it does not create a Luma notification channel.

Important metrics:

Host/container resource series expire after 120 seconds without fresh samples; they are omitted rather than kept at stale values or reported as zero. Separate collection timestamps are available after Control is updated; older stored records fall back to their heartbeat time until replaced. Historical counters, latency percentiles, desired replicas and HTTP error rates cannot be derived from this endpoint.

Service resource labels preserve the reported task identity. Registered Compose jobs map job + task to the Dashboard's job_task name; other jobs retain their job name plus a separate task label. Containers known only by allocation ID are counted as unresolved instead of guessing a service name or making a scrape query Nomad. Check that unresolved count when judging coverage.

For request rate, status codes and latency, enable Traefik metrics and join the router/service labels to Luma application identity. Private workers and routes that bypass Traefik need separate instrumentation. For Control outages, run the collector or an independent probe outside the Control process/failure domain.

Official references: Prometheus scrape configuration, Traefik metrics, Alertmanager grouping and silences.

Missing telemetry and periodic task stalls#

observe.metrics_age monitors the age of the shared Collector self-monitoring sample, with the fixed target metrics. Its defaults are 120 seconds of sample age for 180 seconds; select noData: alert. A stopped Collector, interrupted metric writes, unavailable metric reads, or no first sample can trigger it. It reports unavailable observability rather than failed business services. Control evaluates the rule and uses its existing notification channels; an unavailable Control still requires external monitoring. This covers the shared metrics path, not every application metric, log or trace.

app.task_completion_missing is for tasks expected to complete regularly. Select one application and set taskQuery.service, framework (taskiq or celery), and taskName. windowSeconds is an integer from 60 to 3600 (default 300), longer than the normal schedule interval plus runtime. Taskiq uses native execution counters; Celery uses Collector task-span counters and requires the existing luma.observe.span_metrics=service integration. The value is 0 with completions in the window and 1 without; use threshold 0 and noData: alert. All finished attempts count, including failures, which remain covered by failure rules. Increases are computed per series before aggregation to handle process replacement and counter resets.

Use service heartbeat rules for on-demand tasks that can legitimately be idle. Missing periodic completions can mean scheduling, blocked execution or telemetry failure; correlate with heartbeat and pipeline incidents. Disable or update expectations when retiring a service or changing its schedule. Recovery means qualifying observations returned, not that previous tasks were compensated.