Alert Intelligence
Detection is easy; trustworthy detection is the product. This page covers the layer between "something looks wrong" and "a human got paged": the part that decides severity honestly, suppresses noise without hiding incidents, and keeps repeat incidents visible.
Independent signals
Several independent signals evaluate every service each minute:
| Signal | What it notices |
|---|---|
| Metric deviation | A latency, error or business metric moving away from its normal, materially |
| Behavior shift | The service's overall behavior changing when no single metric names it |
| SLO burn | Error budget being spent fast: user-facing impact, whatever the metrics look like |
| Cascade | Related anomalies across services that call each other: blast radius, even when each hop looks marginal |
| Silence | A service that stopped reporting entirely: an outage until proven otherwise |
| Manual override | An operator saying "treat this as anomalous now", for the things only humans know yet |
When distinct signals agree about the same service at about the same time, the alert is promoted one severity tier. Corroboration earns priority; one weak signal on its own does not.
Not every wiggle deserves a page
Not all metrics deserve the same bar. Per-operation metrics (POST /process, db.insert) can swing
wildly in relative terms while changing almost nothing in absolute terms, especially on a young
service. InfraSage holds them to a higher bar and requires the change to be large in practice, not
just unusual in statistics. Curated business metrics and service-level aggregates keep the normal
sensitivity. The bar was set against live fault runs: real faults clear it by a wide margin, ambient
noise doesn't.
Severity calibration
Severity encodes evidence strength, not detector enthusiasm:
- Silence → critical. SLO burn and cascades → high.
- A single metric deviation → medium, promoted by corroboration.
- Strength scales with magnitude: strong signals rate high, moderate medium, and weak, uncorroborated ones low, which never page.
Noise controls
- Restart warmup. For a few minutes after an engine restart, the parts of detection that need to re-learn stay quiet while history-based checks keep running. A platform deploy does not storm your tenants.
- Re-alert throttle. After an alert fires, the same service cools down briefly. A genuinely different signal bypasses the cooldown.
- Deduplication with an identity window. Repeat signals within an hour fold into the existing alert as occurrence counts. A firing alert older than the window stops absorbing them, so a fresh incident gets a fresh identity and a fresh page instead of disappearing into a stale alert's counters.
- Silence windows suppress notification fan-out (not detection) for planned work.
- Noise candidates (Insights → Noise): rules that fired often, were never acknowledged, and self-resolved fast (the definition of weather) queue for review with their numbers attached. You decide; the platform only nominates.
Routing & lifecycle
Alerts route through configurable rules to your channels: Slack, email, and webhooks (see Integrations). The winning rule is stamped into the alert's labels, so why did this page me is always answerable. Acknowledge, resolve, snooze, and assignment are first-class, recorded with the actor, and feed the noise analytics.
Measured behavior
From InfraSage's own control-scenario suite, run on the pilot:
- Hard 5× latency step: paged in 46 seconds.
- Gradual ramp (5-minute onset): paged in 138 seconds, mid-ramp.
- Innocent deploy with no fault: 15 minutes correctly quiet.
These scenarios re-run as regression gates whenever detection changes.