Observability investigation workspace#
The application overview shows business signals and telemetry coverage. Investigation context is retained across overview, dashboards, logs, metrics and alerts. Copying an investigation link fixes its start and end times; the deployment link opens records for that window. Environment, service and instance filters use reported OpenTelemetry identities. Custom Grafana charts must declare and use the corresponding variables to apply those filters; application owners maintain charts with fixed queries.
Open Observability → Platform health for collection and query status. Control bounds query concurrency, response size and timeouts, and briefly caches summaries. Empty samples, failed queries, stale samples and observed zero values remain distinct. Log coverage and Alertmanager alerts describe current state, even when a historical window is selected. HTTP summaries cover the entire application's ingress over five minutes; task statistics use the selected window. Broker backlog is not inferred from activity or publication counts.
luma observe status/app/alerts/tasks exposes the same capabilities. --format json preserves each source's availability. Command success does not establish application health. These commands and Web views require updated Control. Self-monitoring, durable export queues, structured log fields and persisted read positions require a separate Observe deployment; container identity completion requires updated node agents. Business SDKs are not upgraded automatically.
Logs retain collection time in _time for compatibility with existing alerts. They also record observed_at, a parseable event_time, normalized Trace/Span identifiers and task identifiers. Only allowlisted fields are extracted; arbitrary business JSON is not expanded into labels. Original messages remain searchable. Read positions are saved after the complete batch is accepted. Retries and crashes may replay records, providing at-least-once delivery; node log rotation can still create gaps.
The Collector's trace exporter uses a file_storage queue; Prometheus Remote Write uses its dedicated WAL. These protect only data that has reached the export stage, not SDK buffers or tail-sampling state. New volumes use Luma's initialize: empty preparation without clearing existing directories. Monitor queue usage and free disk space; reduce unnecessary telemetry or add capacity under pressure. Do not delete a running WAL. Queue request limits are not hard disk quotas.
Tempo 2.10.10's local storage enumeration reports directories that disappear during iteration as index polling errors. This implementation exposes the error count and related logs, without suppressing errors through configuration or upgrading Tempo without verification. Establishing the exact cause still requires comparing affected block IDs with deletion records.
Control and Alertmanager retain ownership of their notifications and silences. The shared Web entry reads external alerts without delivering them a second time. The restart rule uses each Allocation/Task counter's increase over the last ten minutes, avoiding a persistent restart-storm alert based solely on lifetime totals.