Standard application observability#

luma-observe accepts official OpenTelemetry signals and exposes the standard VictoriaMetrics, VictoriaLogs and Tempo query APIs through an optional vmauth gateway. Applications own instrumentation, business state, delivery budgets, dashboards and alert expressions. Luma supplies a shared transport and storage service; the stack can be deployed independently of Control.

Data flow#

Official SDK ── OTLP ── Collector ── native Metrics ────── VictoriaMetrics
                            ├───── spanmetrics ───────── VictoriaMetrics
                            └───── tail sampling ─────── Tempo
Nomad stdout/stderr ── existing log shipper ────────────── VictoriaLogs

Backend / operator ── read credential ── vmauth ───────── standard query APIs
External producer ── ingest credential ── vmauth ──────── Collector OTLP HTTP
Application config ── read-only extension volume ──────── Grafana / vmalert

Native metrics bypass trace sampling. Span metrics are derived before the storage branch samples successful fast traces at 10%; OTel errors, HTTP 5xx and traces over one second remain eligible for storage. Metrics describe signals received by the collector, not proof of complete coverage. SDK sampling, bounded buffers, memory refusal and exporter failures can still leave gaps. Collector diagnostics are scraped locally at 127.0.0.1:8888/metrics; vmauth diagnostics use 127.0.0.1:8429/metrics.

Official auto-instrumentation alone cannot determine business completion. Applications can use official custom metrics and spans. Keep metric attributes bounded (service, task type, outcome), and place individual execution IDs on spans or structured logs. Keep secrets and request contents out of telemetry. Use asynchronous bounded exporters and preserve the application's return, exception and retry behavior when telemetry is unavailable.

Luma-managed producers#

Deploy the shared stack, then redeploy an OTel-capable application. Control injects the existing OTEL_* endpoint, bearer header and resource attributes. The manager Tailscale listener :4319 accepts both traces and native metrics. Application requests and task execution do not pass through the collector.

The injected auto-instrumentation default remains OTEL_METRICS_EXPORTER=none to avoid enabling automatic process metrics for every existing image. An application that opts into official metric readers can configure OTEL_METRICS_EXPORTER=otlp, or initialize its own official MeterProvider and bounded reader. There is no Luma SDK, business payload format, extra event database or per-application collector.

Optional machine access#

The default observe-access service uses official vmauth 1.114.0. With empty credentials it starts with zero authorized users. The startup script only writes vmauth's configuration; upstream vmauth implements authentication, routing, concurrency limits and timeouts.

The listener is 127.0.0.1:8427. Every backend store and internal diagnostics listener remains on loopback. A read credential grants cluster-wide telemetry read access to its fixed routes; it is not a tenant-level authorization layer. Use it only from trusted backend services or operator tools, never a browser bundle. Luma management credentials are separate.

Enable each needed role explicitly in the deployment's observe/docker-compose.yml, replacing that role's empty environment value:

environment:
  LUMA_OBSERVE_READ_TOKEN: ${LUMA_OBSERVE_READ_TOKEN}
  LUMA_OBSERVE_INGEST_TOKEN: ${LUMA_OBSERVE_INGEST_TOKEN}

Keep the service's other environment settings. A role that stays disabled keeps its literal "" value. Luma supports ${NAME} secret references; Compose shell default syntax is not resolved by Luma. Configure only the enabled roles using luma secret set LUMA_OBSERVE_READ_TOKEN --scope luma-observe and/or luma secret set LUMA_OBSERVE_INGEST_TOKEN --scope luma-observe. Generate distinct random bearer tokens of 32–512 characters. The CLI prompts privately. The safe variable list is in observe/.env.example.

Rebuild and deploy this independently managed stack from observe/ using the existing build workflow, for example luma build local . --platform linux/amd64. A CLI/Control update does not update the stack. The original storage paths, Tempo 2.10.10 memory settings and 72-hour retention remain in use.

Each role is limited to four concurrent requests, with eight total and a three-second upstream response-header timeout. Application clients still need their own overall timeout and response/row limits. Incoming Authorization is removed before the loopback upstream. Store management, reload, debug and write APIs are outside the read route set. vmauth diagnostics use a separate listener.

For a producer outside the mesh, explicitly install traefik-access.example.yml in the manager's existing Traefik file-provider directory after substituting a DNS/TLS host. Resolve that host through the existing ingress. The example publishes only the authenticated gateway; it creates no route automatically and changes no existing /grafana session authentication.

An external official SDK uses:

OTEL_EXPORTER_OTLP_ENDPOINT=https://observe-api.example.com/otlp
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer%20<ingest-token>

The header follows the official OTel SDK environment-variable format; URL-encode the token when required by that SDK. Existing Luma-managed applications keep the automatically injected mesh endpoint and token. A backend query client uses the same HTTPS host with /metrics, /logs, /traces base paths and its separate read token. Log ingestion remains the existing Nomad stdout/stderr shipper; this gateway adds no log write endpoint.

For rotation, replace the appropriate scoped secret and redeploy the stack, then update consumers. The gateway accepts one credential per role, so plan a short read outage or telemetry loss during rotation. Application execution must remain independent of telemetry delivery. To disable a role, restore its literal empty environment value and redeploy. Remove the explicit Traefik file when HTTPS machine access is no longer needed.

Application-owned dashboards and rules#

The observe-extensions volume is a shared read-only config mount in Grafana and vmalert. Its manager path is /srv/luma/data/luma-observe/extensions; initial creation accepts an empty directory. Existing store volumes keep their names and paths.

Install reviewed application configuration through the operator's normal file delivery process. Use unique dashboard UIDs, rule group names and recording metric names across applications:

extensions/
  dashboards/<application>.json    # standard Grafana dashboard JSON
  rules/<application>.yml          # standard vmalert/Prometheus rule groups

Grafana's Applications folder discovers dashboard JSON recursively. vmalert loads rules/*.yml and rules/*.yaml alongside the bundled infrastructure rules. Both re-scan every 30 seconds. Use an atomic rename from a temporary file with a different extension, with directories readable/traversable and files readable by Grafana/vmalert. Treat application rule configuration as trusted operator input; validate expressions and cardinality before installation.

Validate rule files with the pinned victoriametrics/vmalert:v1.114.0 binary using -rule=<file> -dryRun. Verify the actual dashboard and /api/v1/rules after installation; a parse failure keeps the prior valid rules active. Delete an application's file to remove that configuration on the next reload. This loads definitions, not a new notification receiver: use the existing Alertmanager routing, grouping, maintenance silences and authorized receivers.

Verification#

From the repository root with Luma dependencies and Docker installed:

python scripts/check-observe-protocols.py --platform linux/amd64
python scripts/check-observe-tempo.py --platform linux/amd64

CI runs both. The protocol check uses official packaged components and generic synthetic signals. It verifies zero-user startup, independent roles, protected mesh metrics, native counters, span metrics before sampling, retained error and slow traces, LogsQL/TraceQL reads, internal diagnostics, empty extension startup, and live dashboard/rule reload. It disables notification delivery and removes its isolated containers, volumes and temporary image tags. The separate Tempo check retains the fresh-storage and persisted-block restart verification.

After deployment, verify these paths using the actual configured roles, then check a representative instrumented application. Observe exporter failures, receiver refusal, source freshness, CPU/RSS, disk growth and storage retention. These protocol checks are not a throughput benchmark or a disk quota: Tempo still shares the manager disk and has no free-space guard. Existing storage and memory protections remain necessary; application-specific budgets and acceptance criteria belong in each application's deployment runbook.