Cost Optimization

This page covers strategies to reduce InfraSage event volume, infrastructure costs, and LLM API spend, without degrading detection quality.

Understand your event volume

Before optimizing, measure what you're ingesting. Query your top event producers:

-- Top services by event count (last 24h)
SELECT
    service_id,
    type,
    count() AS event_count,
    round(event_count / sum(event_count) OVER () * 100, 1) AS pct
FROM telemetry
WHERE timestamp >= now() - INTERVAL 24 HOUR
GROUP BY service_id, type
ORDER BY event_count DESC
LIMIT 20;

-- Top metric names by cardinality
SELECT
    metric_name,
    uniqExact(service_id) AS services,
    count() AS total_events
FROM telemetry
WHERE type = 'metric'
  AND timestamp >= now() - INTERVAL 24 HOUR
GROUP BY metric_name
ORDER BY total_events DESC
LIMIT 20;

Reduce event volume

1. Sample high-frequency metrics

Not all metrics need to be reported every second. Metrics that change slowly (CPU, memory) don't benefit from 1-second resolution.

// Report slowly-changing metrics less frequently
type AdaptiveSampler struct {
    lastValues map[string]float64
    minChange  float64  // only report if value changed by this much
}

func (s *AdaptiveSampler) ShouldReport(metric string, value float64) bool {
    last, ok := s.lastValues[metric]
    if !ok || math.Abs(value-last)/last > s.minChange {
        s.lastValues[metric] = value
        return true
    }
    return false
}

Or configure a reporting interval per metric type:

Metric type	Recommended interval
Request latency (P99)	15 seconds
Error rate	15 seconds
CPU / memory	60 seconds
Disk / network I/O	60 seconds
Queue depth	30 seconds
Business events (payments, signups)	Real-time

2. Drop redundant logs

Logs are typically the largest event category. Filter at the source:

# OTEL Collector: drop debug logs before sending to InfraSage
processors:
  filter/drop_debug:
    logs:
      exclude:
        match_type: strict
        severity_texts: ["DEBUG", "TRACE"]

service:
  pipelines:
    logs:
      processors: [filter/drop_debug]

Or via InfraSage's ingestion config:

# Drop log levels below this severity
LOG_MIN_SEVERITY=info

# Truncate log bodies over this length
LOG_MAX_BODY_BYTES=2048

3. Exclude high-volume, low-signal metrics

Health check endpoints typically generate thousands of log lines with no anomaly signal. Exclude them:

LOG_EXCLUDE_PATHS=/health,/ready,/metrics

4. Aggregate before ingestion

Instead of sending every HTTP request as an event, aggregate into per-second summaries:

# Instead of: 1 event per request
# Send: 1 event per 15 seconds with count + latency stats
import statistics

class MetricAggregator:
    def __init__(self, flush_interval_s=15):
        self.buckets = {}
        self.interval = flush_interval_s

    def record(self, service_id, metric, value):
        key = (service_id, metric)
        self.buckets.setdefault(key, []).append(value)

    def flush(self):
        events = []
        for (service_id, metric), values in self.buckets.items():
            events.extend([
                {"type": "metric", "service_id": service_id, "metric_name": f"{metric}.p50",   "value": statistics.median(values),       "timestamp": int(time.time() * 1000)},
                {"type": "metric", "service_id": service_id, "metric_name": f"{metric}.p99",   "value": statistics.quantiles(values, n=100)[98], "timestamp": int(time.time() * 1000)},
                {"type": "metric", "service_id": service_id, "metric_name": f"{metric}.count", "value": len(values),                     "timestamp": int(time.time() * 1000)},
            ])
        self.buckets.clear()
        return events

Choose the right scale profile

The scale profile determines resource allocation. Over-provisioning is the most common cost driver.

Profile	Recommended for	Monthly infra cost (est.)
small	< 1M events/day, < 10 services	~$50-150/mo
medium	1M-50M events/day, 10-50 services	~$300-800/mo
large	50M+ events/day, 50+ services	~$1,500+/mo

Check your current throughput:

curl http://infrasage:8080/metrics | grep ingestion_events_total

If you're on large but processing < 10M events/day, downgrade:

helm upgrade infrasage ./infrasage-chart \
  --set global.scaleProfile=medium

See Scale Profiles for full resource specs.

Tune retention policy

Storing 1 year of raw logs is expensive and rarely useful. Set per-type retention:

# Aggressive retention for high-volume types
RETENTION_DAYS_LOG=14       # logs are searchable for 2 weeks
RETENTION_DAYS_METRIC=90    # metrics kept for 3 months
RETENTION_DAYS_TRACE=7      # traces for 1 week
RETENTION_DAYS_EVENT=180    # events for 6 months
RETENTION_DAYS_SLO=365      # SLO data for compliance (1 year)

To check current storage usage by type:

SELECT
    type,
    formatReadableSize(sum(bytes_on_disk)) AS disk_usage,
    count() AS row_count
FROM system.parts
WHERE database = 'infrasage'
GROUP BY type
ORDER BY disk_usage DESC;

Reduce ClickHouse storage

Compression

ClickHouse uses LZ4 compression by default. For log bodies (high compressibility), ZSTD gives better ratios:

ALTER TABLE telemetry
MODIFY COLUMN log_body String CODEC(ZSTD(3));

Tiered storage

Move old data to cheaper object storage (S3, GCS) automatically:

<!-- ClickHouse storage policy -->
<storage_policy>
  <volumes>
    <hot>
      <disk>default</disk>
      <max_data_part_size_bytes>1073741824</max_data_part_size_bytes>
    </hot>
    <cold>
      <disk>s3_cold</disk>
    </cold>
  </volumes>
  <move_factor>0.2</move_factor>
</storage_policy>

Data older than 30 days moves to S3 automatically. Cold queries are slower but data is still queryable.

Reduce AI provider costs

AI analysis only runs when someone asks for it (an incident's "Analyze", the copilot, a webhook), never automatically on every alert, so cost tracks how often your team asks. To bound it further:

1. Cap the spend

RCA_PER_INCIDENT_BUDGET_USD=2          # stop an analysis that would cost more than this
LLM_TENANT_MONTHLY_BUDGET_USD=200      # per-tenant monthly ceiling

2. Reuse recent answers

RCA_CACHE_ENABLED=true
RCA_CACHE_TTL_MINUTES=30

A repeat request for the same trouble within the window returns the previous analysis instead of calling the model again.

3. Use a cheaper model for simple copilot turns

BEDROCK_HAIKU_MODEL sends short, simple copilot turns to a faster, cheaper model; see AI provider.

4. Measure it

Cost per analysis and 30-day spend are on the Insights page of your console.

Cost monitoring

Track InfraSage's own resource consumption in Grafana:

# ClickHouse disk growth rate (bytes/hour)
rate(clickhouse_disk_bytes_total[1h])

# LLM API calls per hour
rate(rca_llm_calls_total[1h])

# Ingestion rate (events/second)
rate(ingestion_events_total[1m])

Set an alert if ingestion rate grows unexpectedly (could indicate a misconfigured service flooding logs):

# Alertmanager rule
- alert: InfraSageIngestionSpike
  expr: rate(ingestion_events_total[5m]) > 50000
  for: 5m
  annotations:
    summary: "Ingestion rate spike — possible log flood"