Skip to main content

Anomaly Detection

InfraSage learns what normal looks like for each of your services and tells you when one stops behaving that way. You don't set thresholds per metric. You send telemetry; within a day each service has a baseline, and within a week that baseline knows its daily and weekly rhythm.

What it watches​

Every service is judged on the signals that matter for its kind of work, not on a fixed list of metric names:

  • Request-serving services: how much work they do, how much of it fails, and how slow it is, including the slow tail.
  • Message consumers: whether they keep up, how many messages fail, how far behind they fall.
  • Databases and caches: as their callers experience them, even when the store sends no telemetry of its own.
  • Infrastructure: containers near their memory or CPU limits, nodes under pressure, Kubernetes workloads that are down, crash-looping or killed for running out of memory.
  • Logs: a service that starts logging errors while every request still returns success.

The judgment is made per minute, against the service's own history at that time of day and day of week. A Monday-morning traffic peak is normal for a service that always has one; the same peak on a Sunday night is not.

What makes an anomaly worth an alert​

A deviation has to be both real and large enough to matter. A busy service failing 0.2% instead of 0.1% is statistically clear and practically irrelevant; a quiet service with three failed requests is neither. InfraSage requires both evidence and a material effect before it calls something degraded, so a high-volume service can't page on a trivial shift and a low-volume one can't page on noise.

Each alert carries a severity that tracks the strength of the evidence. high means high, which is what makes a pager integration humane.

Detection quality controls​

Each of these exists because a specific false-positive pattern was measured and closed.

Restarts don't cause alert storms​

When the engine restarts, it restores what it had learned and what was already degraded, so a deploy of InfraSage itself doesn't look like an outage of your services.

A new incident is a new alert​

Repeat signals of the same problem are grouped into one alert. A stale alert never silently absorbs a new incident: once a problem has been quiet for a while, a fresh one gets a fresh alert, a fresh timestamp and a fresh page.

Saturation: bounded metrics get limit-aware rules​

Deviation-based detection weakens near a ceiling. A disk that is normally 95% full has five points of headroom, and one filling at its usual steady rate is never "unusual" at any single moment. For bounded resources (disk, memory, CPU, connection pools, quotas), InfraSage reasons about the limit itself:

  • Time to exhaustion. It projects when the resource runs out and says so: "disk_used_percent at 87.4/100, exhausts in ~3.1h at the current rate." Sooner exhaustion means higher severity.
  • Headroom breach. A resource that climbed into its last few percent while its own normal sits lower is flagged even with no other symptom.
  • Resident-high. A resource that lives near its limit by design (a database buffer pool at a permanent 95%) stays quiet until it uses up what headroom remains and something else is also wrong. The database at its usual 95% never pages; the same database at 97.5% while misbehaving does.

Declared limits. Percent and ratio series describe their own ceiling. Raw-unit series can't, whether that's memory bytes against a container limit or connections against a pool maximum. Declare them and the same rules apply, with readable alert text ("jvm_heap_bytes at 3.60G/4.00G, exhausts in ~4.0h at the current rate"):

curl -X PUT https://api.infrasage.dev/api/v1/saturation/limits \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"service_id": "checkout", "metric": "jvm_heap_bytes",
       "limit_value": 4294967296, "unit": "bytes"}'

Omit service_id for a tenant-wide default covering every service that emits the metric. GET /api/v1/saturation/limits lists declarations, DELETE /api/v1/saturation/limits?service=&metric= removes one, and GET /api/v1/saturation/fires shows recent saturation findings. Changes take effect within a few minutes.

Resources that drain toward zero​

The mirror image of a filling disk is free space running out: disk_free_bytes, quota remaining, seats left, credits available. Those get the same time-to-exhaustion treatment: "disk_free_bytes down to 200G remaining, exhausts in ~4h at the current burn rate." A mostly-empty disk reporting 95% free is read as healthy.

Per-instance visibility​

Service-level averages hide single-node failures: a disk at 98% on one node of ten averages to 27.8%, and a per-pod memory leak divides by the replica count. Resource rules evaluate each instance, so the alert names the pod ("… (instance gateway-8489cc9b5f-qmx4d)") instead of drowning it in the mean.

Dead-man's switch: the monitor must fail loudly​

Every alerting channel InfraSage owns (PagerDuty, Opsgenie, Slack) is invoked by InfraSage. If it crashes, its storage fills, or the cluster loses networking, those channels go silent, and silence looks exactly like "everything is fine".

Set DEADMAN_HEARTBEAT_URL to an external heartbeat endpoint: healthchecks.io, a PagerDuty heartbeat integration, or any service that alerts when pings stop. InfraSage pings it every time it finishes evaluating, so the page for "InfraSage is down" comes from outside InfraSage, and a process that is alive but no longer evaluating still trips it.

Baseline maturity​

Detection quality tracks how long InfraSage has watched a service, and the console says so. Each service shows a baseline maturity tier on its dashboard: young (< 24 h), settling (< 7 days), or mature. Expect a somewhat more sensitive first week on a new service; the chip is there so week-one behavior reads as calibration, not crying wolf.

Metric fidelity at ingest​

Two conversions happen before any detection sees your data, because getting them wrong silently corrupts everything downstream.

Cumulative counters become per-interval deltas. OpenTelemetry SDKs default to cumulative temporality, so orders_total is an ever-growing number. Stored raw, it would always look higher than normal and chart as a staircase. InfraSage converts at the boundary, with counter-reset detection so a pod restart reads as a small delta rather than a negative spike. Delta-temporality senders and gauges pass through untouched.

Histogram buckets become percentiles. Discarding a histogram's buckets leaves only count and average, and averages hide tail latency. Every histogram also emits .p50, .p95 and .p99 estimated from its buckets, so p99 > 500ms is expressible as an alert rule and visible in dashboards. Summaries emit their pre-computed quantiles directly.

OTEL_TEMPORALITY_NORMALIZE=false reverts to raw storage if you need the old behavior.

Viewing anomalies​

In the console. Each service's dashboard shows what is degraded now and what was, with the evidence. The Incidents page groups related anomalies into one incident with its likely root cause.

Via the API:

curl $INFRASAGE_URL/api/v1/anomalies \
  -H "Authorization: Bearer $YOUR_JWT" \
  -G --data-urlencode "service_id=payment-api" \
     --data-urlencode "since=2026-04-10T00:00:00Z"

Dead-letter queue (DLQ)​

Records that fail validation are not silently dropped. They go to the DLQ with their full payload, so you can audit validation failures and replay corrected records.

# Check DLQ stats
curl $INFRASAGE_URL/api/v1/debug/dlq-stats

# Response
{
  "total_failed": 142,
  "by_reason": {
    "timestamp_too_old": 98,
    "invalid_value": 31,
    "missing_service_id": 13
  },
  "oldest_entry": "2026-04-09T08:00:00Z"
}