Scale Profiles

InfraSage handles anywhere from a handful of services to thousands. Treat the profiles below as starting points, then tune against the throughput and latency you actually see.

Choosing a profile

Profile	Services	Events/sec	RAM needed	Typical use
Small	10-50	Up to 10K/sec	8 GB	Staging, small teams, dev
Medium	50-500	Up to 100K/sec	32 GB	Mid-size production
Large	500-5,000	Up to 1M/sec	128 GB+	Enterprise, multi-region

Small (10-50 services)

Single Kubernetes node. Suitable for development, staging, and small production environments.

INGESTION_WORKER_COUNT=4
BATCH_FIREHOSE_SIZE=10000
BATCH_FIREHOSE_TIMEOUT_MS=5000
BATCH_EXEMPLAR_SIZE=2000
BATCH_EXEMPLAR_TIMEOUT_MS=10000
WATCHDOG_INTERVAL_SECONDS=60
WATCHDOG_Z_SCORE_THRESHOLD=3.0
WATCHDOG_RCA_COOLDOWN_MINUTES=15
VECTOR_HNSW_M=16
VECTOR_HNSW_EF_CONSTRUCTION=200
OPERATOR_WORKER_COUNT=2

Recommended hardware:

• t3.xlarge (4 vCPU, 16 GB RAM), which runs every component on one host
• 50 GB gp3 EBS for ClickHouse data

Medium (50-500 services)

Kubernetes deployment. Separate ClickHouse and Redpanda instances.

INGESTION_WORKER_COUNT=16
BATCH_FIREHOSE_SIZE=50000
BATCH_FIREHOSE_TIMEOUT_MS=5000
BATCH_EXEMPLAR_SIZE=10000
BATCH_EXEMPLAR_TIMEOUT_MS=5000
WATCHDOG_INTERVAL_SECONDS=60
WATCHDOG_Z_SCORE_THRESHOLD=3.0
WATCHDOG_RCA_COOLDOWN_MINUTES=15
VECTOR_HNSW_M=16
VECTOR_HNSW_EF_CONSTRUCTION=200
OPERATOR_WORKER_COUNT=4

Recommended hardware:

• Ingestion Gateway: 3× m5.large (2 vCPU, 8 GB) with HPA
• Telemetry Operator: 2× m5.large
• AIops Engine: 1× m5.xlarge (4 vCPU, 16 GB)
• ClickHouse: r5.2xlarge (8 vCPU, 64 GB RAM) with 500 GB gp3
• Redpanda: 3-node cluster on m5.large

Large (500-5,000 services)

Kubernetes with dedicated node pools. ClickHouse cluster.

INGESTION_WORKER_COUNT=32
BATCH_FIREHOSE_SIZE=100000
BATCH_FIREHOSE_TIMEOUT_MS=2000
BATCH_EXEMPLAR_SIZE=20000
BATCH_EXEMPLAR_TIMEOUT_MS=2000
WATCHDOG_INTERVAL_SECONDS=60
WATCHDOG_Z_SCORE_THRESHOLD=3.0
WATCHDOG_RCA_COOLDOWN_MINUTES=15
VECTOR_HNSW_M=24
VECTOR_HNSW_EF_CONSTRUCTION=400
VECTOR_HNSW_EF_SEARCH=100
OPERATOR_WORKER_COUNT=8

Recommended hardware:

• Ingestion Gateway: 5-10× m5.2xlarge (8 vCPU, 32 GB) with HPA
• Telemetry Operator: 4× m5.xlarge
• AIops Engine: 2× m5.2xlarge
• ClickHouse: 3-node r5.4xlarge cluster (16 vCPU, 128 GB each)
• Redpanda: 3-5 node cluster on m5.2xlarge
• Dedicated Kafka partition count: 9-12

Tuning tips

High ingestion throughput

• Increase INGESTION_WORKER_COUNT to match the CPU cores you have
• Increase BATCH_FIREHOSE_SIZE, which cuts ClickHouse write overhead
• Decrease BATCH_FIREHOSE_TIMEOUT_MS if latency matters more than throughput

Reduce false-positive anomalies

• Increase WATCHDOG_Z_SCORE_THRESHOLD (e.g., 4.0 or 5.0)
• Increase WATCHDOG_RCA_COOLDOWN_MINUTES to avoid RCA spam

Faster anomaly detection

• Decrease WATCHDOG_INTERVAL_SECONDS (min practical: 10)
• Keep WATCHDOG_Z_SCORE_THRESHOLD at 3.0

Large incident memory / better RCA similarity

• Increase VECTOR_HNSW_M to 24 or 32
• Increase VECTOR_HNSW_EF_CONSTRUCTION to 400
• Note: higher values increase memory usage and index build time

Estimated monthly costs (AWS)

Profile	ClickHouse	Kafka	Compute	LLM API	Total
Small	$200	$100	$300	$50	~$650
Medium	$1,000	$500	$2,000	$500	~$4,000
Large	$5,000	$2,000	$5,000	$2,000	~$14,000