Alert Rules

The detectors learn what normal looks like and alert on departures from it. Some things they cannot infer: that this queue must never exceed 40 because the downstream batch job breaks at 41, that a licence server's connection count is capped by a contract, that finance needs paging when settlement lag passes a regulatory number. That knowledge lives in your team, and it deserves a home.

Alert rules are that home. AI-first does not mean rules-never. It means rules become the minority, with the detectors as the floor beneath them.

Writing a rule

Console → Alert Rules, or the API:

curl -X POST https://api.infrasage.dev/api/v1/alert-rules \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{
    "name": "Checkout queue backing up",
    "type": "threshold",
    "service_id": "checkout",
    "metric": "queue_depth",
    "comparison": "above",
    "threshold": 40,
    "for_duration": "10m",
    "severity": "high"
  }'

Set ALERT_RULES_ENABLED=true on the engine to evaluate them.

Field	Meaning
metric	The metric name as your telemetry reports it
comparison	above or below
threshold	The number
for_duration	How long the condition must hold: 5m, 1h (max 2h)
service_id	Optional; omit to watch every service emitting the metric
severity	low / medium / high / critical. The rule's own tier wins

Sustained means sustained

A rule with for_duration: 10m fires only when the condition held for every window in those ten minutes, and only when enough windows actually exist (a 60% coverage floor). above compares the window minimum, below the window maximum. One lucky sample cannot satisfy "for 10 minutes", and one dip below the line resets it.

The alert shows the boundary value: the closest the metric came to not breaching. When a rule fires at "boundary value 62" against a threshold of 40, you know the entire window sat comfortably above the line.

What happens when one fires

Rule fires enter the same pipeline as every detector signal: deduplication, cooldowns, severity-gated paging, Slack and PagerDuty routing, incident grouping. The alert carries your rule's name (not a generated one) and your severity, and the evidence line states the condition in full. Rules are exempt from post-restart warmup suppression: explicit operator intent outranks statistical hedging.

GET /api/v1/alert-rules/fires?hours=24 lists recent fires.

Migrating from Prometheus

Port the rules that encode decisions: contractual limits, downstream breaking points, regulatory thresholds. Leave behind the ones that exist because static thresholds were the only tool available: "CPU > 80%", "memory > 90%", "error rate > 5%". Those are what the detectors and the saturation rules do better, because they know what your service's normal actually is.

A good migration ends with fewer rules than it started with. That is the point.