Grafana Tempo Tracing

Run distributed tracing at scale with Grafana Tempo. Generates Helm values, S3/GCS object-storage configs, TraceQL queries, service graphs, span metrics, and exemplar links from Prometheus.

This skill covers Grafana Tempo: deploying single-binary or microservices mode on K8s via Helm, configuring object storage backends (S3, GCS, Azure), tuning ingesters/queriers/compactors, writing TraceQL queries to filter by attributes/duration, generating service graph metrics and span metrics for RED dashboards, linking exemplars from Prometheus to traces in Grafana, OTLP ingestion from OpenTelemetry Collector, and multi-tenancy with per-tenant retention.

When to use

Use when deploying Tempo for traces, writing TraceQL queries, generating span metrics for SLOs, or wiring exemplars from Prom to Tempo in Grafana.

Examples

TraceQL slow-request query

Find p99 outliers on checkout

Write a TraceQL query that finds traces in the last 30 minutes where service.name=checkout-api and the root span duration exceeded 2s, grouped by http.route, and the Grafana panel JSON to chart it

Span metrics for SLOs

RED metrics generated from traces

Configure the Tempo metrics-generator to emit span metrics (rate, errors, duration histogram) per service and route, scrape them into Prometheus, and build a Grafana dashboard with availability SLO burn-rate panels