Enterprise Onboarding FAQ

Everything an enterprise team asks in the first month, edge cases included, with honest answers where the honest answer is "not yet, here's the workaround." Questions are ordered the way you'll hit them: setup → detection → data edge cases → logs → trust → access & safety → advanced.

Setup

How do I configure InfraSage?

The happy path, in order:

1. Install the Helm chart into your Kubernetes cluster. It's self-hosted and takes about five minutes. Pick a scale profile sized to your event volume.
2. Get telemetry in. Choose per source, and you can mix:
   • OpenTelemetry (recommended): point your OTel Collector's OTLP exporter at the Ingestion Gateway (/v1/metrics, /v1/logs, /v1/traces).
   • Prometheus: add InfraSage as a remote_write target, or use federated mode, where InfraSage pulls per-minute aggregates from your Prometheus on a schedule and stores no raw samples (Connect → Bring your own stack).
   • Logs without OTel: Splunk HEC endpoint or Elasticsearch bulk endpoint (Logstash/Filebeat-compatible).
   • Cloud metrics: the CloudWatch integration poller (EC2, RDS, Lambda, ALB, DynamoDB, S3, SNS).
3. Set up identity and access: SSO (SAML/OIDC), SCIM provisioning, MFA, and role assignments (viewer → operator → admin). Do this before inviting the team.
4. Map services to teams in the Connect hub. Routing and escalation hang off ownership, and the Home page keeps showing your mapping percentage until it's done.
5. Configure notification channels (Slack, Teams, email, webhook), then routing rules (service pattern + severity → channel) and escalation policies. Use the routing dry-run endpoint to test a rule before you rely on it; the API warns you if a rule targets a channel with no configured notifier.
6. Wait 24 to 48 hours while anomaly baselines build from your real traffic. Explicit threshold rules detect from the first minute; learned-baseline detection reaches full strength after two or three days, or a week if your traffic has strong weekly seasonality.

Feature flags are default-off: runbook execution, runbook authoring, triggers, the DAG engine, causal grouping, federated telemetry, and pillars are all opt-in env vars. Nothing acts on your systems until you turn the relevant surface on, and even then nothing executes without a human approval (see the access section).

Detection

How will InfraSage detect anomalies?

Several independent detectors run in parallel, all on cheap pre-aggregated data (the LLM is not in the detection loop):

• Service behavior vectors: each service gets a per-minute embedding of its metric, log, and trace behavior. A learned baseline turns that into a "weirdness" score, with adaptive per-service thresholds, seasonality awareness, and a holiday calendar.
• Named-metric scoring: per-metric z-scores against rolling baselines ("p99 latency is 3.2σ above its Tuesday-afternoon normal").
• Log-shape detection: log templates are mined continuously, with no LLM involved. Novel templates and template bursts are anomaly signals, as are error/warn/fatal rate shifts. Semantic rules flag specific patterns (OOM kills, deadlocks).
• Trend and change-point detectors: slow drifts and sharp level shifts (CUSUM) that z-scores miss.
• Causal pre-fault signals (CIAD): learned invariants between services whose violation often precedes a customer-visible fault.
• Deploy-aware baselines: a deploy event suppresses the "everything changed" false-positive window and becomes evidence instead.
• Probes and business KPIs (via Pillars, below): synthetic checks and revenue-shaped metrics with their own baselines.

Alerts fire through the watchdog with dedup (repeats collapse into one alert with a count), grouping into incidents (correlated alerts across services cluster via the dependency graph), routing, silences, and escalation on top. You can always add explicit threshold rules alongside the learned detection. Many teams keep their five "contractual" thresholds and let InfraSage learn the rest.

InfraSage's default vector is not considering the metrics that I have

Three tools, in escalating order of effort:

1. Check what was discovered first. The telemetry catalog (per-service metric inventory) lists every metric name InfraSage has seen from your service. If yours isn't there, it's an ingestion or labeling problem, not a vector problem.
2. Map your names onto the standard slots. The embedding has canonical dimensions (latency, error rate, throughput, saturation…). If you emit app_req_duration_p99 instead of a recognized latency name, add a dimension alias (/api/v1/embedding/dim-aliases) that points your metric at the canonical slot. No re-instrumentation needed.
3. Add custom slots for metrics with no canonical equivalent (queue depth, cache hit ratio, GC pause), either globally (/api/v1/embedding/global-slots) or per service (embedding config API). The auto-select machinery also proposes slot assignments from observed data, so review its suggestions rather than starting blind.

Two things to know. The named-metric scorer watches metrics independently of the vector, so a metric that matters to you gets per-metric anomaly scoring the moment it's ingested, before you have tuned any slots. And after you change slot config, use the re-embed and re-ingest tooling to rebuild recent baselines instead of waiting for the window to roll forward on its own.

My service name label is different from what InfraSage supports

Service identity is the one label you must get right. Where it comes from, per source:

• OTLP: the service.name resource attribute. Fix it in your OTel Collector with a resource processor if your SDKs emit something else.
• Prometheus remote_write: the job label becomes the service id, and every other label rides along as a tag. If job is wrong for you (say it's your scrape-config name, not your service), add a write_relabel_configs block in Prometheus to rewrite job before sending.
• Federated (pull) mode: fully configurable. Each pull query declares service_label ("use the service label", "use app", and so on), with job as the fallback.
• Splunk/Elastic log ingestion: service is derived from the event's source/fields; set it explicitly in your shipper config.

Edge cases: multi-tenant deployments prefix ids (tenant/service). The UI displays the bare name and the APIs accept both. If two sources emit different names for the same service (payments from OTel, payments-svc from Prometheus), unify them at the source. InfraSage treats them as two services, and there is deliberately no server-side "merge two service ids" magic today, because it makes evidence provenance ambiguous.

I have multiple stamps. How does InfraSage detect anomalies across stamps?

InfraSage models a six-part identity: tenant, environment, service, cell, instance, and operation. A "stamp" maps onto the cell, with environment above it.

• Label your telemetry with environment and cell, using OTel resource attributes or Prometheus external labels per stamp. That's the only requirement.
• Per-stamp baselines: identity-level detection learns baselines per (service, environment, cell), so payments in stamp-eu-2 is judged against its own normal, not a blended global average. A stamp with 10× the traffic doesn't drown out a small stamp's anomaly.
• Per-stamp alerts: alert dedup fingerprints include environment and cell, so the same fault in three stamps produces three attributable alerts, rather than one ambiguous alert or thirty duplicates.
• Routing and runbooks can match on environment (and labels), so stamp-specific on-call routing works.

The honest edge case is cross-stamp correlation. If one shared dependency breaks all stamps at once, incident grouping clusters the alerts using the service dependency graph within a time window: stamps connected through a shared upstream land in one incident, while stamps that look topologically independent surface as parallel per-stamp incidents that you can merge yourself in the incident list. A first-class "same fingerprint across N cells → one meta-incident" collapse is on the roadmap, not shipped.

Logs

How do I configure alerts on logs?

Three layers, cheapest first:

1. Automatic (zero config): error, warn, and fatal counts per service-minute feed the behavior vector and the watchdog, so an error-rate spike alerts without any rule. Log level detection reads structured level fields and falls back to [ERROR]-style prefixes. If your logs use a nonstandard scheme, normalize levels in your shipper or the counts will under-detect.
2. Shape-based (zero config): novel log templates and template bursts ("a message we've never seen appeared 400×/min") are first-class anomaly signals.
3. Pattern-specific (explicit): semantic rules match regexes to named events ("OutOfMemory", "connection pool exhausted") that fire as detections. Absence rules ("no logs from X for 5 min") cover silent-death cases. Threshold alert rules can also target log-derived metrics.

For "alert me if this exact string appears," use a semantic rule rather than a threshold rule. You get a named, deduplicated event instead of a raw counter.

How do I know log anomaly detection is working properly?

Verify at four levels, in order:

1. Ingestion: the service's telemetry-quality score shows log coverage. If coverage is red, detection has nothing to work with. Also check the per-service signal counts; log totals per minute should match what you expect.
2. Template mining: the log-templates inventory for the service should populate within minutes of log flow. No templates = your log bodies aren't reaching the miner (check field mapping in your shipper).
3. Fire a controlled test: inject a burst of distinctive ERROR lines into one service (or use Demo Control on a demo tenant) and watch: signal counts rise → novel template registered → anomaly score moves → alert fires. That end-to-end trace is your proof, and it doubles as an on-call drill.
4. Close the loop on misses: when something real slips through, file it with the missed-detection feedback on the service page. It lands in the telemetry-quality system with remediation hints (usually a silence that was too broad, or levels that weren't parsed), and it's what we use to tune detection for your tenant.

Trust

How do I know an RCA is correct?

You're not asked to trust it. You're shown its work, at five levels:

1. Evidence citations: claims in the analysis pin to evidence items with a coverage score (≥80% = well-evidenced; below 40% the UI bands it as "hypothesis only"). Numbers drill to the query and window behind them.
2. Deterministic origin: separately from the LLM, a graph-based origin resolver names the suspect service with a confidence, and abstains with a reason when the signal is ambiguous rather than guessing.
3. The judge and the gate: an automated acceptability judge scores each analysis, and low-coverage or judge-rejected RCAs are gated: visible in the UI as unpublished, and never pushed to Slack as if they were confident findings.
4. Degraded-mode honesty: if every LLM provider was down and the rule-based fallback produced the analysis, the output is labeled fallback_rules with a banner. It's a heuristic checklist, never dressed up as a grounded analysis.
5. Your verdicts feed back: rate an RCA wrong and it becomes a counter-exemplar in future prompts; the Accuracy page tracks precision on evaluation scenarios and flags weak classes. Agent-mode analyses also keep a full tool-call trace you can replay step by step.

How do I trust a runbook generated by InfraSage?

Trust is earned per-runbook through a ladder, never granted by default:

• Born untrusted: every generated or imported runbook starts at trust score 0 and tier T2, so the whole runbook needs explicit human approval before any step runs. LLM output cannot execute anything by itself, ever.
• Reviewed with receipts: drafts show each step beside its source (the incident evidence or the document sentence it came from). You edit, fill in parameters, and then promote or reject.
• Bounded blast radius: a static analyzer scores each runbook's blast radius (which services/namespaces it touches, how many mutating steps) against your tenant's policy cap.
• Supervised execution: T2 = one-shot approval; T1 = pause at every step for approval; human-checkpoint steps ("verify the dashboard recovered") stop the run until an operator confirms. Every run has a per-step audit trail and replay.
• Earned autonomy, deliberately: a runbook becomes eligible for a lower tier only after ≥10 supervised runs across ≥3 days with ≥90% trust and no recent failure. Even then a human makes the downgrade, one runbook at a time. There is no path where the system loosens its own leash.

I have a lot of runbooks in Confluence, but many are outdated and could harm production. Will InfraSage detect that on import? Will it detect and fix dangerous commands?

What import does protect you from:

• Nothing imported can run. Imports land as trust-0, tier-T2 drafts. An outdated runbook sitting in the drafts queue is inert. The danger window is a human promoting it without reading it, and the review screen is built to make the review real: every generated step carries the verbatim source sentence, so "this says scale the old deployment name" is visible rather than buried.
• No invented commands. The converter is forbidden from fabricating shell commands. Only commands quoted verbatim in your document survive, and anything that needs judgment becomes a blocking human-checkpoint step rather than an action.
• Strict parameters. Ambiguous blanks ({{cluster}}, {{replica_count}}) fail compilation loudly instead of silently becoming empty kubectl arguments.
• Guarded execution even after promotion: blast-radius caps, the closed kubernetes-operation allowlist, the shell executor being globally disableable, and T2 approval all still apply. A stale target (a deployment that no longer exists) fails at execution with a clear error rather than doing something else.
• Future drift is caught: imported runbooks remember their source revision. The drift-sync worker re-fetches nightly and queues a new draft for review when the Confluence page changes. It never hot-swaps the approved version.

What it honestly does not do (yet):

• It does not semantically validate against production at import time. "This references a service that was decommissioned in 2024" is not auto-detected today. (A planned improvement: cross-check step targets against live topology at review time, so stale targets get flagged before promotion instead of at execution.)
• It does not have a "dangerous command classifier," and it will not rewrite your procedure to "fix" it. Silently editing an operational document is its own hazard. The model flags what it can, the tier system contains what it can't, and the human reviewer stays the authority.

Practical guidance for a big legacy corpus: import in bulk (everything becomes searchable, cited drafts), promote only the runbooks you'd actually hand a new on-call engineer, and let the rest age out of the drafts queue (90-day TTL).

Access & safety

How will I give production access to InfraSage? Will it use that access during RCA? How do I stop it running dangerous commands?

Three separate grants, each independently scoped:

1. Telemetry (no grant needed): ingest is push-based, or pull with read-only credentials you supply (Prometheus/Loki/Jaeger endpoints, CloudWatch read keys).
2. Read access for investigation: a Kubernetes ServiceAccount you create, scoped to get/list/watch. It powers live topology and the "inspect this pod" capability. Yes, RCA uses it, under a discipline: live-infrastructure reads are allowed only after the agent has established that ingested telemetry can't answer the question, and every live read is written to an audit table with its justification. You can review exactly what was looked at, when, and why.
3. Write access for remediation (optional, off by default): only the runbook executors mutate anything, and only through:
   • a closed operation allowlist for Kubernetes (restart pod, scale deployment, restart deployment, drain node; nothing else compiles),
   • the shell executor being disabled in multi-tenant mode and gate-able everywhere (leave it off and no arbitrary command can ever run),
   • tiered human approval on every execution (Slack card or console; deploy-class actions require a two-person-style second confirmation),
   • blast-radius policy caps, daily mutation counters, and full audit rows.

And one directive that's enforced in code, not policy: RCA never triggers itself and never triggers remediation. Analyses run on human request or your webhook; actions run on human approval. If you grant only reads, InfraSage is a pure observer with an audit trail.

Advanced

Does InfraSage understand my network topology if I have no traces, only NAT/VPC flow logs?

Honest answer: flow-log topology inference is not implemented today. What you can use instead, in practice:

• Kubernetes live topology: the k8s informer discovers runtime structure (workloads, services, and their relationships) without any traces. For in-cluster estates this recovers most of the graph.
• Manual or seeded dependencies: the dependency API accepts explicit edges, and teams usually export their known service graph once (from architecture docs or their flow-log tooling) and seed it. The causal grouper, blast-radius scoring, and RCA traversal all consume this graph identically to a discovered one.
• Partial traces beat no traces: instrumenting only your gateways and edges with OTel gives you service-pair discovery for the paths that matter most.

If flow logs are your system of record for topology, tell us. Parsing VPC flow logs into dependency edges is a well-shaped feature request: the ingestion and graph sides both exist, and only the parser is missing. It's the kind of thing a design partnership prioritizes.

I have a custom tool. How do I ingest its events, and how will InfraSage detect anomalies on them?

Pick the shape that matches your data:

• It's telemetry-shaped (things happening over time): send OTLP logs/events, or generic events through the gateway. Event volume feeds the behavior vector automatically (an event-rate anomaly is detected like any other), and log-shape detection applies if the events carry text bodies.
• It's alert-shaped (your tool already decides something is wrong): post to the alerts webhook, and the event enters the normal alert lifecycle (dedup, grouping, routing, RCA-on-demand) with your tool as the source.
• It's business-metric-shaped (orders, signups, declines): push KPIs via the business-KPI API, with CSV/API backfill for history. KPI baseline detection gives them learned normal ranges and anomaly alerts, and they appear in RCA evidence when a technical incident moves a business number.
• It needs custom logic ("alert when refund-rate ÷ order-rate exceeds X for segment Y"): define a custom pillar, a sandboxed SQL detector over your ingested data, with a dry run before you enable it. This is the full-control path.

What are Pillars, and when should I use them?

Pillars are the business layer above services: named product journeys (checkout, payments, search) mapped to the services that implement them, with their own probes, KPI baselines, severity policy, and anomaly scoring at the product level. Use them when:

• executives ask "is checkout healthy?" and a wall of service graphs isn't an answer;
• you need synthetic coverage (HTTP/DNS/TLS probes) for paths with thin telemetry;
• you want business KPIs (revenue, decline rate) guarded with baselines, not just infra metrics;
• an infra anomaly needs automatic translation into product impact ("payment-service latency → checkout pillar degraded").

Skip them until your service-level detection is trusted. Pillars build on a good foundation, they don't replace one. When you're ready, the guided wizard (with dry-run) sets a pillar up in minutes, and everything stays default-off until you enable it.

If I connect InfraSage to my repos, will it detect which line of code / PR / commit caused an incident? Will it raise a fix PR?

Split into three honest tiers:

• Deploy/PR-level correlation: yes, shipped. Connect the GitHub/GitLab deploy webhooks and every RCA automatically carries the changes that landed in the hour before onset: version, commit ref, actor, and PR link ("v2.14.1 deployed 6 minutes before onset"). For most incidents, "which deploy did this" is the answer, and it arrives with a link to the PR.
• Line-level blame: no. InfraSage does not clone or statically analyze your source. It will hand your engineer the suspect PR and the failing behavior (specific endpoint, error template, trace evidence); the diff reading stays human.
• Auto-raising a fix PR: no, and deliberately not today. Our remediation posture is approved runbooks with bounded blast radius, not machine-authored code changes. If you want a "draft a PR" motion, the supported pattern is a runbook step that calls your own workflow (through the n8n integration, say) so PR creation happens in your tooling under the same human-approval gate. InfraSage proposes, your engineer reviews, and nothing merges itself.