Skip to main content

Log Anomaly Detection

Some failures leave no trace in metrics: every request returns success, latency is fine, and the only sign is a service writing errors to its logs. InfraSage reads your log streams and surfaces three kinds of log-based anomaly, independently of any metric or trace signal:

TypeWhat it catchesWhen
Novel patternAn error or fatal log pattern this service has never produced beforeAs it happens
BurstA known pattern suddenly appearing far more often than it normally doesAs it happens
Critical eventLines that announce trouble in so many words: a circuit breaker opening, a pool exhausted, a failover, a state change into a degraded modeAs it happens

A service can be clean on metrics and still be flagged on logs, or the reverse. No application changes are needed: InfraSage works on the log lines you already send.

How patterns are recognized​

InfraSage groups log lines into patterns by separating the fixed text from the parts that vary, such as IDs, timestamps and values:

"User 42 logged in from 10.0.0.1"   → "User <*> logged in from <*>"
"DB query took 234ms for user 99"   → "DB query took <*> for user <*>"

That is what lets it tell "a new kind of error" from "the same error, more often". Each pattern keeps its own history per service, so a connection-refused message that normally appears twice an hour is a burst at two thousand an hour, and the same message from a service that has never produced it is novel.

Grace period for new services​

A brand-new service has no log history, so every pattern it writes would look novel. Novel-pattern detection stays off for a service until it has some history:

LOG_GRACE_PERIOD_MINUTES=60   # default

AI summaries of new patterns​

When a novel pattern appears, InfraSage can ask an AI model to classify how serious it is and summarize it in one sentence for the alert. Patterns it judges benign, and near-duplicates of patterns your team has already seen, are suppressed. Detection never waits on this step: if the model is slow or unavailable, the alert fires with its structural evidence alone.

LOG_SEMANTIC_ENRICH=true   # default; set false to disable

Only the pattern and a few sample lines are sent, and only for novel patterns, not for every log line. See sub-processors for where that data goes, and data redaction to mask sensitive values before anything leaves your environment.

What an alert shows​

Alert evidence for a novel or bursting pattern carries the pattern, up to three matching raw lines, fields extracted from them, and other services that started logging trouble at the same time:

Novel log pattern: a1b2c3 [fatal]: <*> failed to connect to <*>:5432 after <*> retries
Sample log 1: payment-worker failed to connect to db-primary:5432 after 5 retries
Sample log 2: payment-worker failed to connect to db-primary:5432 after 5 retries
Sample log 3: payment-api failed to connect to db-replica:5432 after 5 retries
Extracted fields: http_status=500, request_id=req-abc123, user_id=user-789
Simultaneous log burst in: auth-service, order-processor

Extracted fields​

InfraSage pulls these fields out of raw log lines when they are present, in the common spellings (request_id=, requestId: and so on):

Field
request_id
user_id
tenant_id
error_code
http_statusany 3-digit HTTP status
trace_id
span_id
duration_msduration=, latency=, elapsed= with a number

They help size the blast radius: which users, tenants or requests were affected.

Cascades across services​

When one service starts logging trouble, InfraSage checks whether others started at the same moment and lists them. That is how a cascade shows up in the evidence: a database connection pool exhausted behind db-proxy produces simultaneous bursts in payment-api, order-service and auth-service.

Volume and retention​

A chatty service logging at 100k events/s could fill storage before old logs expire. InfraSage caps how many raw lines per second it stores for each service at the ingestion gateway. Pattern recognition still sees every line, so detection is unaffected; only the stored samples are thinned.

LOG_MAX_RAW_PER_SECOND=1000   # per service; 0 disables the cap

When the cap drops lines, the log_writes_dropped_total Prometheus counter increments.

Raw log lines are kept for one day. Pattern history is kept for 30 days, and patterns from services that have sent nothing for a week are pruned, so decommissioned services don't skew anyone's normal.

Configuration reference​

VariableDefaultServiceDescription
LOG_GRACE_PERIOD_MINUTES60AIOps EngineMinutes of pattern history a service needs before novel-pattern alerts fire
LOG_SEMANTIC_AUTO_DETECTtrueAIOps EngineFlag critical-event lines (breaker open, pool exhausted, failover, degraded state)
LOG_SEMANTIC_ENRICHtrueAIOps EngineAI classification and summaries of novel patterns; needs an AI provider key
LOG_MAX_RAW_PER_SECOND1000Ingestion GatewayPer-service cap on stored raw log lines; 0 disables it