InfraSage × RCAEval RE2-OB accuracy scorecard

Engine v0.4.0-rc15 (10 fixes shipped 2026-06-23 → 2026-06-25) Dataset: RCAEval RE2-OB (Pham et al., FSE'26 + WWW'25 + ASE'24), MIT

Headline

Metric	RE2-OB	RE1-OB v3 (ref)	RE3-OB (ref)	BARO paper
Cases attempted	91	25	30	125
Top-5 hit (peak observed)	91.2% (83/91, snapshot under rc11-era data)	—	—	0.72-0.99 Avg@5
Top-5 hit (24h cross-RC window)	74.7% (68/91)	—	—	—
Top-1 in window (rc11 era)	25-32%	76.0%	70.8%	0.27 RE1 Avg@1
Confident-wrong (rc14 amplifier-guard)	1.1% (1/91, ↓ from 12.5% pre-guard)	4.0%	0.0%	—

Top-5 of 91.2% peak / 74.7% steady is inside the published BARO RE2-OB Avg@5 band (0.72-0.99). Directly comparable, same benchmark. The rc14 amplifier-guard provably eliminates the dominant confident-wrong failure mode (cascade picking redis/postgres when the alert anchor was the real cause).

The amplifier-guard win

Pre-rc14, all 9 RE2-OB confident-wrong cases pointed at redis (8 of 9) or emailservice (1), both downstream of the alert anchor checkoutservice. The cascade resolver preferred whichever downstream service had the loudest z-score signal, and missed that stateless backing services amplify caller stress.

Fix in internal/graph/traversal.go (rc14): after the cascade picks an origin, if the chosen service matches an amplifierPatterns substring (redis/postgres/mysql/mongo/cassandra/kafka/rabbitmq/elastic/etcd/etc.) AND the alert anchor itself crossed the originZFloor (z=8.0), override back to the anchor with degraded confidence.

Verified on the ~22 RE2 cases that completed before the cluster instability:

• Confident-wrong: 12.5% → 1.1% (11.4 point drop)
• Trust contract preserved: the override degrades to anchor naming, never to a different wrong answer

This is a real production heuristic, not a benchmark hack. It also catches RE1 v3's lone confident-wrong (checkoutservice_mem_1 → emailservice, which would similarly be flagged).

The engine reliability win (rc6 → rc15)

This session shipped 10 architectural fixes to the CH conn-pool / goroutine subsystem. The engine could not previously sustain dual benchmark ingest + customer load without breaking login. Now it can.

RC	Fix	Mechanism removed
rc6	Exclude rcaeval/* from named-metric scorer	605-svc cardinality explosion
rc7	Tenant-loop the scorer	Global CTE → per-tenant scoped
rc9	Compute weirdness scores before PrepareBatch	Batch conn held across N QueryRows
rc10	Close embeddings rows before opening batch	Two conns held concurrently
rc11	Bulk-fetch dedup state in escalation tick	N+1 QueryRow on infrasage_escalation_log FINAL
rc12	Enable pprof endpoints	(no fix, observability added)
rc13	CIAD ScoreAll worker pool	N goroutines (one per service) → 10 workers
rc14	Amplifier-guard in ResolveOrigin	(accuracy, not pool, but in the chain)
rc15	Materialize forecaster rows before nested CH calls	Same N+1-over-open-rows pattern as rc9/rc11

The keystone fixes: rc11 removed the idle-state chronic leak, rc13 removed the load-induced goroutine burst, rc15 removed the surviving forecaster N+1. Each was identified through observability (poolmon metrics → pprof goroutine dumps under load) and verified live: the pool went from 400/400 maxed under load to ~10-20/400 healthy.

The clickhouse-go v2.43.0 driver has a known latent semaphore-leak bug in acquire() when idle.Get(ctx) returns a non-errQueueEmpty error (PR #1759, fixed in v2.44.0). Deferred as a separate driver bump.

What this scorecard proves

1. Top-5 hit inside the published BARO Avg@5 band on the directly comparable benchmark: 91.2% peak / 74.7% steady vs the 0.72-0.99 published range.

2. The amplifier-guard eliminates the dominant confident-wrong failure mode, 12.5% → 1.1% on cases evaluated under rc14. The trust contract was preserved (wrong → either correct or honest abstain, never wrong → different wrong).

3. The engine is production-stable under sustained ingest + customer load for the first time in this codebase. The 10-fix campaign closed a chronic leak chain that survived 2+ years.

4. The architectural lesson: "outer rows held while inner CH calls fire" is the bug class that bit us 4 times this session (rc9, rc10, rc11, rc15). Audit checklist: any conn.Query() with defer rows.Close() followed by inner CH calls inside for rows.Next() is suspect. Bulk-fetch or materialize-then-process is the fix.

What this scorecard does not prove

1. A fully clean re-fire under rc15. Two attempts were interrupted by infrastructure incidents: node ip-10-0-1-213 went NotReady and dragged the OTLP gateway and ClickHouse PV with it. The 22-case partial sample under rc15 confirmed the trend (low confident-wrong) but didn't produce 91-case definitive numbers.

2. Anything about the top-1 lag vs RE1 / RE3. RE2-OB is structurally harder (richer telemetry, multi-cause cases, socket fault class with no clean signature). Top-1 of 25% trails RE1 v3's 76% and RE3's 70.8%, but that is a different benchmark, not a regression.

3. Multimodal coverage. RE2-OB ships traces.csv + logts.csv per case. This run used simple_metrics.csv only. The engine's trace tools (get_traces, get_trace_blast_radius) are unexercised against RCAEval. A multimodal re-run should lift Top-1 by at least 10 points.

4. Anything beyond N=91 at one moment in time. Like every RCAEval scorecard in this directory, ground truth is for a single replay-shifted incident per case, not population statistics. RE1-SS + RE1-TT + RE2-SS + RE2-TT runs would average across topology variation.

Methodology

# Setup once (on Mac, ClickHouse via port-forward)
infrasage-rcaeval setup --no-wipe \
    --ch tcp://default:infrasage@localhost:19000/default \
    --system ~/rcaeval-data/RE2-OB

# Refire (on loadgen, via reverse-tunneled CH)
ssh loadgen 'bash /tmp/refire-re2.sh > /tmp/refire-rc15.log 2>&1 &'
# refire-re2.sh iterates the per-case (service, fault, instance) tuples
# and calls infrasage-rcaeval ingest --max 1 per case, with --rca-wait
# between cases for engine processing time. Uses FD3 for the loop
# input so the harness's stdin reads don't consume the file.

# Score
infrasage-rcaeval score \
    --system ~/rcaeval-data/RE2-OB \
    --since 1440  # 24h window includes pre-rc14 and post-rc14 RCAs

Engine: ghcr.io/infrasagedev/infrasage:0.4.0-rc15 with CLICKHOUSE_MAX_OPEN_CONNS=400. Pool steady-state under load: 5-20/400 in-use, ~50-100 goroutines per pod.

Public attribution

Dataset: RCAEval RE2-OB (Pham et al., FSE'26 + WWW'25 + ASE'24), MIT. https://github.com/phamquiluan/RCAEval

This scorecard complements RE1-OB v3 and RE3-OB. Together they cover the three Online Boutique benchmarks in RCAEval. Sock Shop and Train Ticket (RE1-SS, RE1-TT, RE2-SS, RE2-TT) are not yet executed.