Skip to main content

Common issues

Fixes for the problems people hit most often when running InfraSage.

Services won't start​

ClickHouse fails to start​

Symptom: infrasage-clickhouse container exits immediately.

Causes and fixes:

  1. Port 9000 already in use

    lsof -i :9000
    # Kill the conflicting process or change the port in docker-compose.yml

  2. Insufficient disk space

    df -h /var/lib/docker
    # Free up disk space; ClickHouse needs at least 10 GB

  3. SQL init scripts failed

    docker logs infrasage-clickhouse | grep -i error
    # Check /scripts/sql/ for syntax errors

Ingestion Gateway can't connect to ClickHouse​

Symptom: Gateway logs show clickhouse: connection refused or no such host.

Fix:

# Verify ClickHouse is healthy
docker exec infrasage-clickhouse clickhouse-client \
  --user infrasage --password infrasage-dev \
  --query "SELECT 1"

# Check network connectivity from gateway container
docker exec infrasage-ingestion-gateway \
  wget -qO- http://clickhouse:8123/ping

The gateway waits for ClickHouse to be healthy before starting (via depends_on condition). If it still fails, increase the healthcheck timeout.

Redpanda topic not created​

Symptom: Gateway logs show topic 'raw-telemetry' does not exist.

Fix:

# Manually create the topic
docker exec infrasage-redpanda \
  rpk topic create raw-telemetry --brokers localhost:29092 --partitions 3

No data in ClickHouse​

Symptom: You're sending telemetry but ClickHouse shows 0 rows.

Debugging steps:

  1. Verify the gateway is receiving requests

    curl -s '$PROMETHEUS_URL/api/v1/query?query=infrasage_ingestion_events_total' | jq
    # Should show non-zero counter

  2. Check if events are failing validation

    curl $INFRASAGE_URL/api/v1/debug/dlq-stats
    # If DLQ is growing, your records are failing validation

  3. Check Kafka consumer lag

    docker exec infrasage-redpanda rpk group describe infrasage-ingestion-group
    # If LAG is growing, the Telemetry Operator is not consuming fast enough

  4. Check Operator logs

    docker logs infrasage-telemetry-operator | grep -i error

Anomalies not being detected​

Symptom: You're ingesting data with extreme values but no anomalies appear.

Causes:

  1. Not enough history. A service needs some history before anything it does can be called unusual. Send data continuously for a while first; the service's dashboard shows its baseline maturity.

  2. One extreme point isn't an incident. Detection looks for a sustained, material change, not a single outlier sample. Keep the abnormal values coming for a few minutes.

  3. Detection runs once a minute. Allow a couple of minutes after the change. For testing you can shorten it with WATCHDOG_INTERVAL_SECONDS=10.

# Send an abnormal value (repeat for a few minutes)
curl -X POST $INFRASAGE_URL/api/v1/telemetry \
  -H "Content-Type: application/json" \
  -d '{
    "service_id": "test-service",
    "metric_name": "cpu_usage_percent",
    "value": 9999.0,
    "timestamp": "'$(date -u +%Y-%m-%dT%H:%M:%SZ)'"
  }'

Root cause analysis returns a fallback or nothing​

Symptom: An analysis comes back labeled analysis_mode: fallback_rules, or fails.

RCA runs when you ask for it (from an incident, chat or the API), not automatically on every anomaly.

Causes:

  1. No AI provider configured. Check that LLM_PROVIDER and its key are set for the AIOps Engine (see AI provider). Without one, RCA falls back to a rule-based analysis by design.

  2. Provider errors. Check the AIOps Engine logs:

    docker logs infrasage-aiops-engine | grep -i "llm\|rca"

  3. Daily budget reached. Analyses stop when the tenant's daily AI budget is used up; the Insights page shows the 30-day spend.

  4. Too soon. Evidence lands a few minutes behind live data. If the analysis abstains with a "re-run shortly" hint, do that.

Rate limit errors (HTTP 429)​

Symptom: Ingestion returns 429 Too Many Requests.

Causes:

  1. Per-key rate limit exceeded. Increase rate_limit_rps on the API key, or create additional keys with shared rate limits.

  2. Tenant plan quota exceeded. Check monthly event usage:

    curl $INFRASAGE_URL/api/v1/billing/usage -H "Authorization: Bearer $JWT"

    Upgrade your plan or wait for the next billing period.

  3. Sending too fast. Implement exponential backoff in your ingestion client:

    import time
    for attempt in range(5):
        resp = requests.post(url, json=payload)
        if resp.status_code != 429:
            break
        time.sleep(2 ** attempt)

High Kafka consumer lag​

Symptom: rpk group describe shows growing LAG for infrasage-ingestion-group.

Fixes:

  1. Increase OPERATOR_WORKER_COUNT (e.g., from 2 to 4)
  2. Increase Kafka partition count and scale Telemetry Operator replicas
  3. Check ClickHouse write performance: 
    docker exec infrasage-clickhouse clickhouse-client \
      --query "SELECT * FROM system.metrics WHERE metric LIKE '%Write%'"

Grafana dashboards show no data​

Symptom: Grafana dashboards are empty or show "No data".

Fixes:

  1. Prometheus data source not configured. In Grafana: Settings → Data Sources → Add Prometheus → URL: http://prometheus:9090

  2. Prometheus not scraping InfraSage. Check $PROMETHEUS_URL/targets; all targets should be UP.

  3. Dashboards not provisioned. Restart Grafana: docker-compose restart grafana