Alert routing by team

A routing rule sends the alerts it matches to a list of channels. The first matching rule wins, so give specific rules a lower priority number. The seeded catch-all (priority 10000) sends everything else to every notifier, unless you set a default channel.

Say who owns a service

Rules match a service's labels. Two keys matter: team and owner. A service gets them in one of three ways. Later ones win.

1. From its telemetry. The gateway reads these OpenTelemetry resource attributes: team, service.team, service.owner, owner and k8s.pod.label.team. The last one appears when the collector's k8sattributes processor copies the pod's team label.
2. From a Backstage catalog. POST /api/v1/atlas/import/backstage sets each Component's team from its spec.owner (group:default/payments becomes payments).
3. By hand. PUT /api/v1/service-labels with {"service_id": "payment-service", "key": "team", "value": "payments"}. Telemetry and imports never overwrite a label set this way. GET lists a tenant's labels, and DELETE ?service_id=&key= removes one.

Labels reach the routing engine within a minute.

Send a team's alerts to the team

POST /api/v1/routing/rules
{
  "name": "Payments team",
  "priority": 100,
  "label_matchers": {"team": "payments"},
  "channels": ["slack:payments", "pagerduty:payments"]
}

A channel is <notifier>:<target>:

Channel	Goes to	Configure
slack:payments or slack:#payments	the webhook named payments	SLACK_WEBHOOKS="payments=https://hooks.slack.com/...,platform=https://..."
pagerduty:payments	the Events API key named payments	PAGERDUTY_ROUTING_KEYS="payments=<key>,platform=<key>"
email:[email protected]	that address	SMTP settings
opsgenie:Payments	Opsgenie, with that team as responder	OPSGENIE_API_KEY
slack, pagerduty, ...	the notifier's default destination	SLACK_WEBHOOK_URL, PAGERDUTY_ROUTING_KEY, ... (Slack works with named webhooks only; PagerDuty needs the default key)

Webhook URLs and routing keys are secrets. Put SLACK_WEBHOOKS and PAGERDUTY_ROUTING_KEYS in the infrasage-secrets Secret, not in the ConfigMap.

If a target names no configured destination, delivery falls back to that notifier's default, because a typo shouldn't lose a page. GET /api/v1/routing/rules lists such targets under unresolved_targets, and saving a rule with one returns a warning.

A resolution goes to the channels its alert went to.

A default channel

When team rules send owned services to their teams, the rest should go to one triage channel, not to every notifier. Set:

ROUTING_DEFAULT_CHANNELS=slack:triage

This replaces the seeded catch-all's *. If you have edited the catch-all to name its own channels, those are kept.

Other labels

Rules can also match environment (from the alert). Labels in label_matchers are AND-joined with service_pattern, severity and source.