Service-level span metrics#

中文

Applications can opt into service-level metrics derived from spans by setting the resource attribute luma.observe.span_metrics=service. The shared Collector then aggregates those derived metrics across process instances. Applications without that exact attribute value keep the existing spanmetrics behavior. Stored traces and native OTLP metrics retain their original resources, including service.instance.id.

When to use this mode#

This mode suits workers whose frequent process replacement creates many historical span-metric series. It uses standard OTel spans and Collector spanmetrics; applications still own task names, outcomes, dashboards and rules. It does not reuse process identities or convert native cumulative counters.

Choose the signals for each application:

The service flag and native metric exporter are independent. An opted-in application may retain native metrics; they still have per-process identities. Setting the metric exporter to none removes SDK queue-wait, active-attempt, heartbeat and oldest-active-age measurements, along with application custom native metrics. A completed-span counter cannot replace these measurements. Nomad allocation monitoring remains available; broker backlog and expected work need their own application or broker signals.

Enable and roll out#

Use Control 0.2.26 or later and deploy an Observe build containing the service-span pipelines. These are separate deployments: updating Control does not rebuild Observe. luma-observe-sdk==0.1.0 is compatible; this feature requires no SDK update. Other official OTel integrations can emit the same span contract.

1. Deploy the updated Control so application resource attributes merge by key.
2. Rebuild and redeploy the independent Observe stack through its existing deployment workflow.
3. Set the application resource flag and redeploy that application, starting in its test environment. Keep its existing endpoint, credentials and other environment values.

Example environment fragment for a service that chooses span-derived task metrics:

environment:
  OTEL_SERVICE_NAME: document-worker
  OTEL_RESOURCE_ATTRIBUTES: service.namespace=documents,deployment.environment.name=production,luma.observe.span_metrics=service
  OTEL_TRACES_EXPORTER: otlp
  OTEL_METRICS_EXPORTER: none

Use the application's own stable service and environment names. If it already sets OTEL_RESOURCE_ATTRIBUTES, add the flag to that existing value. The match is exact and case-sensitive: an absent flag or another value uses the default branch. Verify the flag on emitted span resources if application code also supplies resource attributes. SDK instrumentation and task lifecycle hooks are still required; environment injection alone does not create task spans.

Control preserves each application-defined resource key, including an explicitly empty value, and appends only missing injected keys such as luma.stack, luma.task and luma.region. It does not replace the application's other environment settings. Applications that previously supplied their own resource attribute string may gain these missing labels once on redeployment, changing their series identities. Applications that already received the complete injected attributes have no additional change from this merge.

Identities and metric contract#

The opt-in derived resources retain these six identity keys:

They also carry the fixed label luma_observe_span_metrics="service". Keep this mode label: it separates opt-in and default writers sharing the traces_span_metrics_* namespace, including during a rollout. Other resource attributes, including process identity and SDK version, are removed only from these opt-in derived metrics. The original trace and native-metric resources remain intact. Grouping all six identities prevents unrelated services, environments, Luma tasks or regions from being merged.

Opted-in spans still produce the generic traces_span_metrics_* RED family. Each accepted span reaches either the default or service RED connector once. The opt-in task family is an additional view of matching execution roots; do not add RED calls, task_span_calls_total and native task_executions_total together to calculate task volume. RED includes producer and child spans. The additional task family receives only spans satisfying all of these conditions:

• Span kind is CONSUMER.
• There is no parent span.
• Both task.name and task.framework attributes are present.

Producer spans and nested child spans do not enter the task family. Taskiq and Celery adapters in SDK 0.1.0 emit the matching execution roots. Each received completed root represents an execution attempt; a retry followed by success can contribute two attempts. A cancellation contributes only if a completed root is emitted and received.

The task dimensions are task_name, task_framework, task_queue and task_outcome, alongside the connector's standard span dimensions and the service identity labels above. Source outcomes such as succeeded, failed, retrying, skipped and cancelled remain separately selectable. Missing queue or outcome attributes are not inferred as business success. Arbitrary business fields do not become additional task-metric dimensions.

Query examples#

Use selectors for the intended service, environment and mode. Add luma_stack, luma_task or luma_region when the query should cover only one deployment. These examples use the environment fragment above.

Observed attempts over one hour, separated by outcome:

sum by (task_name, task_framework, task_queue, task_outcome) (
  increase(task_span_calls_total{service_name="document-worker", service_namespace="documents", deployment_environment_name="production", luma_observe_span_metrics="service"}[1h])
)

Failed attempts over the same window; retries and cancellations are separate outcomes:

sum by (task_name) (
  increase(task_span_calls_total{service_name="document-worker", service_namespace="documents", deployment_environment_name="production", luma_observe_span_metrics="service", task_outcome="failed"}[1h])
)

P95 duration of successful attempts, in seconds:

histogram_quantile(0.95,
  sum by (le, task_name) (
    rate(task_span_duration_seconds_bucket{service_name="document-worker", service_namespace="documents", deployment_environment_name="production", luma_observe_span_metrics="service", task_outcome="succeeded"}[5m])
  )
)

Choose windows long enough for the task's normal frequency and metric export interval. Sparse observations, absent series and counter resets affect rate/increase; these expressions are operational estimates, not an exact execution ledger. Application Grafana JSON uses the existing dashboard extension directory.

No-execution alert template#

The following is an example for a task expected to finish regularly. Choose the task, window and for duration from the application's workload, and apply its working hours or maintenance silences. An idle queue can be healthy; no completed spans can also mean long-running work or failed telemetry delivery. This document does not install an alert or enable notifications.

groups:
  - name: application-observation-example
    rules:
      - alert: ApplicationHasNoObservedExecutions
        expr: |
          (
            sum(increase(task_span_calls_total{
              service_name="document-worker",
              service_namespace="documents",
              deployment_environment_name="production",
              luma_observe_span_metrics="service",
              task_name="document.convert"
            }[30m])) or vector(0)
          ) == 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "No completed execution spans observed for document.convert"

This template treats missing series as no observed executions; it does not prove that the worker is down. Review it with deployment health, expected task demand and Collector freshness. Validate and deliver an approved rule through the existing application rule workflow.

Delivery limits and reconciliation#

The task counters describe completed roots received by the Collector before storage tail sampling. SDK head sampling, full export queues, timeout or delivery failures, Collector memory refusal, hard-killed workers and restarts can reduce coverage. Retransmitted duplicate roots can also change counts. Low average QPS does not establish burst safety: measure peak traffic, span volume per task, buffer refusal and exporter failures for the actual deployment.

The spanmetrics connectors keep cumulative state in Collector memory. The existing five-minute idle expiration for a resource group and Collector restarts can reset that state; individual metric series may retain state until their resource group is evicted. Historical storage samples remain, so a rate/increase window crossing these boundaries needs care. This mode does not add persistent counter state or change the existing expiration policy.

Compare logs and metrics using the same time window and timezone, task, framework, queue and outcome. Distinguish attempts from final business jobs, and account for the delivery gaps and reset boundaries above. A daily metric increase is not guaranteed to equal a daily log total. Service aggregation reduces process-driven series churn for stable span/task dimensions; it is not a storage quota and does not bound arbitrary span names or task cardinality.

Disable and verify#

Remove only luma.observe.span_metrics=service from the application's resource attributes and redeploy it. Newly received spans return to the default spanmetrics branch. Existing derived series remain queryable until lookback and storage retention exclude them; they are not deleted on redeploy. Keep the mode selector in dashboards during a transition. If native metrics were disabled, explicitly restore OTEL_METRICS_EXPORTER=otlp when they are wanted; removing the resource flag does not change that exporter setting.

After enabling or disabling, verify a representative application:

• The opt-in derived series have the service mode label and no process-instance label; an unflagged application still has its previous spanmetrics identity.
• Stored traces still identify the original process. Native metrics retain their original resources when their exporter is enabled.
• Task counts and duration include the expected completed roots and outcomes, while producer and child spans remain outside the task family.
• Query source freshness, exporter/receiver failures and actual series growth alongside the application's task and log observations.

Platform verification and standard access paths are documented in Standard application observability.