Skills / Engineering / Prometheus Alertmanager

Prometheus Alertmanager

Route, group, silence, and deliver alerts with Prometheus Alertmanager. Build routing trees, wire receivers for Slack, PagerDuty, and email, manage silences and inhibition rules, and debug flaky alert-delivery pipelines.

This skill helps you tame alerting with Prometheus Alertmanager. It designs routing trees that match on labels, groups related alerts to cut noise, configures receivers (Slack, PagerDuty, email, webhook), sets repeat and group intervals, writes inhibition rules so downstream alerts stay quiet during outages, and manages silences via the API and UI. It also covers templating alert messages, HA Alertmanager clustering, and testing routes with amtool.

alertmanager prometheus alerting observability incident-response

When to use

Use when configuring Prometheus Alertmanager — routing trees, receivers (Slack/PagerDuty/email), grouping, silences, inhibition rules, alert templates, or debugging alert delivery.

Examples

Routing tree with receivers

Page critical, Slack the rest

Write an Alertmanager config that routes severity=critical alerts to PagerDuty and everything else to a Slack channel, grouping by alertname and cluster with a 5m group_wait

Inhibition and silences

Suppress noisy alerts

Add an Alertmanager inhibition rule so that when a cluster-down alert fires, all node-level alerts for that cluster are suppressed, and show how to create a 2-hour silence with amtool
Added to wishlist