Skip to main content

Observability Architecture — Grafana LGTM Stack

internal engineering docs

flow: apps (OTel SDKs / exporters) → Alloy → { Mimir · Tempo · Loki } → Grafana · tenancy: X-Scope-OrgID on every write/read · interactive original ↗

Metrics · MimirPromQL · remote_writeTraces · TempoTraceQL · OTLPLogs · LokiLogQL · /loki/api/v1/pushAlertingrulers → AlertmanagerGCP · central control planedashed = traffic · animated AppsOTel SDKs / exportersscrape · OTLP · tailAlloycollector — all 3 signalsMimirmetrics store + PromQLTempotrace store + TraceQLLokilog store + LogQLGrafanaUI — dashboards · Explorecorrelationtenancy: X-Scope-OrgID on every write/read

0 · Stack at a glance — who does what​

Grafana Alloy — collector​

The frontline agent. One binary that scrapes, receives, processes and forwards metrics, logs and traces — replacing Prometheus agent + Promtail + OTel Collector.

  • Runs everywhere telemetry originates (DaemonSet / systemd)
  • Filtering + relabeling here controls backend cost
  • Buffers with WAL so backend outages don't lose data
  • Stamps tenant identity (X-Scope-OrgID) on everything
protocolin: OTLP, scrape, file/journald · out: remote_write, OTLP, Loki push

Grafana Mimir — metrics​

Horizontally scalable, multi-tenant metrics store AND PromQL query engine. Grafana sends PromQL; Mimir executes it — used for TPS, latency, CPU/memory, SLOs.

  • Prometheus-compatible: drop-in remote_write target
  • Long-term retention in object storage (years, not weeks)
  • Also runs recording/alerting rules (ruler) + Alertmanager
  • Scales to 1B+ active series across tenants
protocolquery: PromQL · ingest: remote_write

Grafana Tempo — traces​

Distributed tracing backend AND TraceQL query engine. Stores every span cheaply in object storage — powers request waterfalls and bottleneck analysis.

  • No sampling required at the backend: object storage is cheap enough to keep 100%
  • TraceQL: search spans by duration, attributes, structure
  • metrics-generator derives RED metrics + service graphs into Mimir
  • Trace ID is the correlation key across all three signals
protocolquery: TraceQL · ingest: OTLP

Grafana Loki — logs​

Log aggregation store AND LogQL query engine. Indexes only labels (not full text), so it's cheap; queries grep chunks in parallel — keyword and error-code search.

  • 'Like Prometheus, but for logs' — same label model
  • Tiny index → 10-100× cheaper than full-text log stores
  • LogQL can also compute metrics from logs (rate of errors)
  • Log lines carry trace IDs → one-click jump to Tempo
protocolquery: LogQL · ingest: /loki/api/v1/push

Grafana — UI​

The unified visualization layer. Stores no telemetry itself — it sends PromQL/TraceQL/LogQL to the three backends and renders dashboards, Explore views and alerts.

  • One UI over all three signals + correlation between them
  • Exemplars: latency spike on a graph → the exact trace behind it
  • Derived fields: trace ID in a log line → Tempo waterfall
  • Dashboards, on-call views, SLO reporting for the whole org
protocolspeaks PromQL · TraceQL · LogQL over HTTP