Alert Intelligence

Detection is easy; trustworthy detection is the product. This page covers the layer between "a detector fired" and "a human got paged": the part that decides severity honestly, suppresses noise without hiding incidents, and keeps repeat incidents visible.

The detector ensemble

Six independent signals evaluate every service each minute:

Detector	Signal	Notes
Named-metric z	Robust modified z-score (median/MAD) per metric against a rolling baseline	The workhorse for latency/error/business metrics. Class-aware thresholds (below).
ML novelty	A learned model over the service's whole telemetry shape	Catches "something is different" when no single metric names it.
SLO burn	Error-budget burn rate	Fires on user-facing impact regardless of metric shapes.
Cascade	Correlated anomalies across calling services	Catches blast radius even when each hop looks marginal.
Silence	A service stopped reporting entirely	Critical by default: silence is an outage until proven otherwise.
Manual override	An operator says "treat this as anomalous now"	For the things only humans know yet.

When two or more distinct detectors corroborate within a five-minute window, the ensemble booster promotes the alert one severity tier. Corroboration earns priority; one weak signal on its own does not.

Class-aware thresholds

Not all metrics deserve the same bar. Per-operation span metrics (POST /process, db.insert) have tiny variance on young baselines, so trivial absolute shifts can produce large z-scores. InfraSage classifies metric names and applies:

• Curated & aggregate metrics (business metrics, service-level duration/error aggregates): the configured threshold (WATCHDOG_NAMED_METRIC_Z_THRESHOLD).
• Operation-class metrics: at least max(2× configured, 6.0) and a practical-significance gate: the current value must deviate ≥ 50% from the baseline median. A statistically loud but practically meaningless wiggle does not page.

This split was tuned against live control runs: injected faults measure z ≈ 5-100 with several hundred percent deviation; ambient noise measures z ≈ 2-5 with single-digit deviation.

Severity calibration

Severity encodes evidence strength, not detector enthusiasm:

• Silence → critical. SLO burn and cascades → high.
• Named-metric fires → medium (single-signal), promoted by corroboration.
• The statistical path scales with magnitude: strong signals rate high; moderate → medium; weak, uncorroborated fires → low and never page. Every tier is env-overridable per detector.

Noise controls

• Warmup suppression. For a few minutes after an engine restart, the state-dependent ML path is suppressed while its models re-warm (baseline-driven detectors stay live from the first tick). A platform deploy does not storm your tenants.
• Re-alert throttle. After an alert fires, the same service's alert path cools down briefly (default 3 minutes). A genuinely distinct signal (a large z change) bypasses the cooldown.
• Deduplication with an identity window. Repeat fires within an hour fold into the existing alert as occurrence counts. A firing alert older than the window stops absorbing new fires, so a fresh incident gets a fresh identity and a fresh page instead of disappearing into a stale alert's counters.
• Silence windows suppress notification fan-out (not detection) for planned work.
• Noise candidates (Insights → Noise): rules that fired often, were never acknowledged, and self-resolved fast (the definition of weather) queue for review with their numbers attached. You decide; the platform only nominates.

Routing & lifecycle

Alerts route through configurable rules to your channels: Slack, email, and webhooks (see Integrations). The winning rule is stamped into the alert's labels, so why did this page me is always answerable. Acknowledge, resolve, snooze, and assignment are first-class, recorded with the actor, and feed the noise analytics.

Measured behavior

From the platform's own control-scenario suite, run against production:

• Hard 5× latency step: paged in 46 seconds.
• Gradual ramp (5-minute onset): paged in 138 seconds, mid-ramp.
• Innocent deploy with no fault: 15 minutes correctly quiet.

These scenarios re-run as regression gates whenever the detection stack changes.