Anomaly Detection

InfraSage uses a multi-layer anomaly detection pipeline that combines statistical methods with unsupervised machine learning. Detection latency is under 100 milliseconds.

How it works

The Watchdog runs on a configurable interval (default: every 60 seconds) and makes three passes:

ClickHouse telemetry
        │
        ▼
  1. Statistical Watchdog    (Z-score per metric)
        │
        ▼
  2. Isolation Forest        (multivariate ML scoring)
        │
        ▼
  3. Adaptive Thresholds     (seasonal + infrastructure-aware)
        │
        ▼
  Anomaly declared → trigger RCA

Layer 1: statistical Watchdog (Z-score)

For each (service_id, metric_name) pair, the Watchdog keeps a ring buffer of recent values. On each poll cycle, it computes the Z-score of the latest value:

Z = (current_value - rolling_mean) / rolling_stddev

If |Z| > WATCHDOG_Z_SCORE_THRESHOLD (default 3.0), an anomaly is declared.

The buffer holds the last N samples per metric, over a configurable window. It resets when a service starts reporting metrics for the first time, but a gap in the data does not reset it.

Sensitivity tuning

Threshold	Behavior
2.0	Very sensitive, many false positives
3.0 (default)	Balanced; flags 3-sigma deviations
4.0	Conservative, only flags extreme outliers
5.0	Very conservative, production-critical services only

# Set via environment variable
WATCHDOG_Z_SCORE_THRESHOLD=3.5

Layer 2: Isolation Forest

Isolation Forest is an unsupervised ML algorithm that finds anomalies by randomly partitioning the feature space. Points that are isolated quickly (short average path length) are anomalies.

InfraSage runs Isolation Forest across several metrics at once for a single service, so it catches multivariate anomalies that per-metric Z-scores miss. CPU slightly elevated, error rate slightly elevated, latency slightly elevated: each is normal on its own, but the combination is not.

Features used:

• Raw metric value
• First derivative (rate of change)
• Second derivative (acceleration)
• Hour of day (encoded)
• Day of week (encoded)
• Correlated metric values

Layer 3: adaptive thresholds

Adaptive thresholds add context to the Z-score. Seasonality is one input: higher CPU on weekday mornings and lower CPU on weekends is expected. Infrastructure events are another, so a spike after a known deployment is expected and the baseline adapts to it. Thresholds also tighten on their own as a rolling baseline's variance decreases over time.

Adaptive thresholds are built from ClickHouse historical data and recalculated on each Watchdog cycle.

RCA cooldown

After an anomaly triggers RCA for a (service_id, metric_name) pair, InfraSage enforces a cooldown period before triggering RCA for the same pair again (default: 15 minutes). This prevents RCA spam during sustained incidents.

WATCHDOG_RCA_COOLDOWN_MINUTES=15

During the cooldown window, anomalies are still detected and stored in ClickHouse; they just don't trigger another RCA cycle.

Detection quality controls

Four mechanisms keep the alert stream honest. Each one exists because a specific false-positive mode was measured in production and closed.

Severity scales with signal strength

An alert's tier is calibrated per detector. On the statistical path, high requires a strong signal (|z| ≥ 3 or a strong ML score); moderate signals land at medium and weak, uncorroborated ones at low. Corroboration still promotes: when two independent detectors fire for the same service within a short window, the ensemble raises the tier by one. The practical consequence is that high means high, which is what makes a pager integration humane.

Class-aware named-metric thresholds

Per-operation span metrics (POST /checkout, db.insert) have tiny variance bands on young baselines, so trivial absolute shifts can produce large modified-z values. These operation-class series must clear both a much higher bar (at least 2× the configured threshold, floor 6.0) and a practical-significance gate (≥ 50% deviation from the baseline median) before they can alert. Curated business metrics and aggregate duration series keep the configured sensitivity. Real faults measure z = 5 to 100 at hundreds of percent deviation, so detection is unaffected; the noise class simply can't page anymore.

Post-restart warmup

The state-driven ML path re-learns after an engine restart, and re-learning looks like novelty. For a warmup window after boot (default 8 minutes, WATCHDOG_WARMUP_MINUTES), only independent signals may alert: named-metric z, SLO burn, stale service, cascade, and manual override. The ML/state path stays quiet until its models have settled, so deploys stop being alert storms.

Incident identity

Alert deduplication groups repeat fires of the same incident. But a firing alert older than the dedup window (default 60 minutes, ALERT_DEDUP_MAX_AGE_MINUTES) stops absorbing new fires: a fresh fire gets a fresh alert identity, a fresh timestamp, and a fresh page. A stale alert can never silently swallow a new incident.

Saturation: bounded metrics get limit-aware rules

Deviation-based detection structurally weakens near a ceiling. A series whose normal is 95% has five points of headroom, so the fatal move is a tiny relative change, and a disk filling at its usual steady rate is never statistically anomalous at any instant. The saturation detector (SATURATION_DETECTOR_ENABLED) reasons about the physical limit instead, for bounded series (percent or ratio utilization: disk, memory, CPU, pools, quotas):

• Time-to-exhaustion. Fits short (90 min) and long (12 h) slopes and projects when the series hits its limit: high within ~1 h, medium within ~6 h, low for slow steady fills landing within ~24 h. The alert says so concretely: "disk_used_percent at 87.4/100, exhausts in ~3.1h at the current rate."
• Headroom breach. The series crossed the absolute floor (92% of its limit) while its own baseline lives lower. This is an application that climbed to 95%: it fires even with no other symptom, and promotes to high when the anomaly ensemble corroborates.
• Resident-high. The baseline itself lives above 85% by design (a database buffer pool at a permanent 95%). This stays silent unless the series has additionally consumed half of its remaining headroom and an independent weirdness signal corroborates. The database at its usual 95% never pages; the same database at 97.5% while misbehaving does.

The same change also exempts utilization and percent-style names from the operation-class practical-significance gate above. Near a ceiling, a ≥50% relative move is mathematically impossible, so that gate would have permanently muted exactly the emergencies this detector exists to catch.

Declared limits. Percent and ratio series self-describe their ceiling. Raw-unit series cannot, whether that is memory bytes against a container limit or connections against a pool max. Declare them and the same rules apply, with readable alert text ("jvm_heap_bytes at 3.60G/4.00G, exhausts in ~4.0h at the current rate"):

curl -X PUT https://api.infrasage.dev/api/v1/saturation/limits \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"service_id": "checkout", "metric": "jvm_heap_bytes",
       "limit_value": 4294967296, "unit": "bytes"}'

Omit service_id for a tenant-wide default covering every service that emits the metric. GET /api/v1/saturation/limits lists declarations, DELETE /api/v1/saturation/limits?service=&metric= removes one, and GET /api/v1/saturation/fires shows recent saturation findings. The detector picks up changes on its next tick (~2 minutes).

Dead-man's switch: the monitor must fail loudly

Every alerting channel InfraSage owns (PagerDuty, Opsgenie, Slack) is invoked by the engine. If the engine crashes, its database fills, or the cluster loses networking, those channels go silent, and silence is indistinguishable from "everything is fine". A monitoring system must be the last thing to fail, and it must fail loudly.

Set DEADMAN_HEARTBEAT_URL to an external heartbeat endpoint: healthchecks.io, a PagerDuty heartbeat integration, or any service that alerts when pings stop. InfraSage pings it on every evaluation tick, so the page for "InfraSage is down" comes from outside InfraSage. The ping is tied to the evaluation loop rather than the process, so a live process with a wedged watchdog still trips it.

Resources that drain toward zero

The saturation rules above watch series climbing toward a ceiling. Their mirror image drains toward zero: disk_free_bytes, quota remaining, seats left, credits available. Half the world's exporters report free space rather than used space, so those series get a zero-floor time-to-exhaustion rule of their own: "disk_free_bytes down to 200G remaining, exhausts in ~4h at the current burn rate." Names matching both shapes (disk_free_percent) are read as remaining, so a mostly-empty disk at 95% free is correctly read as healthy.

Per-instance visibility

Service-level averaging hides single-node failures: a disk at 98% on one node of ten averages to 27.8%, and a per-pod memory leak divides by the replica count. Resource metrics are also rolled up per instance, and the saturation rules evaluate each instance independently, so the alert names the pod ("… (instance gateway-8489cc9b5f-qmx4d)") instead of drowning it in the mean.

Baseline maturity

Detection quality tracks baseline age, and the console says so. Each service carries a baseline maturity tier, shown on its dashboard: young (< 24 h), settling (< 7 days), or mature. Expect a somewhat more sensitive first week on new services; the chip is there so week-one behavior reads as calibration, not crying wolf.

Metric fidelity at ingest

Two conversions happen before any detector sees your data, because getting them wrong silently corrupts every signal downstream:

Cumulative counters become per-interval deltas. OpenTelemetry SDKs default to cumulative temporality, so orders_total is an ever-growing number. Stored raw, its average is a lifetime artifact, its z-score sits permanently elevated (the latest value always exceeds the median), and its chart is a staircase to infinity. InfraSage converts at the boundary, with counter-reset detection so a pod restart reads as a small delta rather than a negative spike. Delta-temporality senders and gauges pass through untouched.

Histogram buckets become percentiles. A histogram carries its bucket distribution; discarding it leaves only count and average, and averages hide tail latency, the exact thing teams alert on. Every histogram also emits .p50, .p95, and .p99 estimated from its buckets, so p99 > 500ms is expressible as an alert rule and visible in dashboards. Summaries emit their pre-computed quantiles directly.

OTEL_TEMPORALITY_NORMALIZE=false reverts to raw storage if you need the old behavior.

Anomaly score

Each detected anomaly has a numeric score between 0 and 1:

Score	Meaning
0.0-0.4	Mild deviation; logged, not alerted
0.4-0.7	Moderate; alert triggered
0.7-1.0	Severe; alert, RCA, and runbook evaluation

Viewing anomalies

Via Grafana

Open $GRAFANA_URL → the Anomaly Detection dashboard. It shows:

• Anomaly timeline by service
• Score heatmap
• Top anomalous metrics

Via ClickHouse SQL

SELECT
  service_id,
  metric_name,
  anomaly_score,
  z_score,
  timestamp
FROM infrasage.infrasage_anomalies
WHERE timestamp > now() - INTERVAL 1 HOUR
ORDER BY anomaly_score DESC
LIMIT 50

Via API

curl $INFRASAGE_URL/api/v1/anomalies \
  -H "Authorization: Bearer $YOUR_JWT" \
  -G --data-urlencode "service_id=payment-api" \
     --data-urlencode "since=2026-04-10T00:00:00Z"

Dead-letter queue (DLQ)

Records that fail validation are not silently dropped. They go to the DLQ with their full payload, so you can audit validation failures and replay corrected records.

# Check DLQ stats
curl $INFRASAGE_URL/api/v1/debug/dlq-stats

# Response
{
  "total_failed": 142,
  "by_reason": {
    "timestamp_too_old": 98,
    "invalid_value": 31,
    "missing_service_id": 13
  },
  "oldest_entry": "2026-04-09T08:00:00Z"
}