Vector Observability Pipeline

Build high-throughput log/metric/trace pipelines with Vector. Generates sources, transforms (VRL), sinks, end-to-end ACKs, disk buffers, and topologies that fan out to Datadog/Loki/S3/Kafka.

This skill covers Datadog's Vector: declaring sources (file, kubernetes_logs, journald, http_server, kafka, opentelemetry), writing VRL (Vector Remap Language) transforms for parsing/redaction/enrichment, sinks (Loki, Datadog, S3, Kafka, Elasticsearch, Clickhouse), reliable delivery with disk buffers and end-to-end ACKs, multi-tenant topologies, and running Vector as a DaemonSet agent + Aggregator pattern. Includes cost-cutting tactics: sampling, filtering, and reshaping at the pipeline.

When to use

Use when replacing Fluentd/Fluent Bit with Vector, cutting log volume with sampling, parsing/redacting fields before they hit a vendor, or building a multi-sink fan-out pipeline.

Examples

Log volume cut with VRL

Drop noisy logs and redact PII

Write a Vector pipeline that ingests Kubernetes logs, drops anything matching health-check paths, redacts email and credit-card patterns in the message field, and routes the rest to Datadog with disk buffering

Multi-sink fan-out

Cheap storage + searchable tier

Configure Vector to fan out app logs: full-fidelity to S3 in JSON for compliance, sampled at 10% to Loki for live debugging, and ERROR-only to PagerDuty webhook