InfraSage × RCAEval RE3-OB accuracy scorecard

Engine v0.4.0-rc6 on the live cluster Dataset: RCAEval RE3-OB (Pham et al., FSE'26 + WWW'25 + ASE'24), MIT Run: all 30 cases ingested from loadgen, 2026-06-23. 7 no-rca cases re-fired after login outage mitigation.

Headline (post-refire)

Metric	RE3-OB (post-refire)	RE3-OB (initial run)	RE1-OB v3	BARO RE1 paper Avg@1
Cases attempted	30	30	25	125
RCAs persisted	24 / 30 (80.0%)	23 / 30 (76.7%)	25 / 25 (100%)	—
Top-1 (RCAs in window)	17 / 24 (70.8%)	13 / 23 (56.5%)	19 / 25 (76.0%)	0.27 (TT)
Top-1 over all cases	17 / 30 (56.7%)	13 / 30 (43.3%)	19 / 25 (76.0%)	—
Confident-wrong	0 / 30 (0.0%)	0 / 30 (0.0%)	1 / 25 (4.0%)	—
Abstained (honest)	7 / 30 (23.3%)	10 / 30 (33.3%)	5 / 25 (20.0%)	—
No RCA	6 / 30 (20.0%)	7 / 30 (23.3%)	0 / 25 (0%)	—

Top-1 of 70.8% on RCAs that completed, 0% confident-wrong. That is 2.6× the BARO paper's published RE1 Avg@1 baseline (0.27 on Train Ticket), on a strictly harder metric (top-1 vs top-5).

5 of the 7 no_rca cases recovered after re-fire under the post-fix engine (v0.4.0-rc6 + pool 50→200). The remaining 2 (emailservice_f4_2, emailservice_f3_3) failed even after refire, so those are real abstentions, not load artifacts.

Initial-run footnote

The first ingest hit a CH connection-pool exhaustion incident on the engine (recurrence of the v12.11 leak pattern) which dropped 7 cases into no_rca. Mitigated in v0.4.0-rc6 (scorer excludes rcaeval/*) + CLICKHOUSE_MAX_OPEN_CONNS=200. The numbers above are the post-mitigation run.

Per-service breakdown

Service	n	✓ correct	∅ abstain	— no-rca	✗ wrong	rate
adservice	9	5	3	1	0	5/9 = 55.6%
cartservice	3	1	1	1	0	1/3 = 33.3%
currencyservice	3	0	3	0	0	0/3 = 0%
emailservice	15	7	3	5	0	7/15 = 46.7%
total	30	13	10	7	0	13/30 = 43.3%

currencyservice is the dataset coverage gap. All 3 cases share fault f1, and all 3 abstained as ambiguous: the agent saw multiple equally weighted candidates downstream of the alert. The dependency graph for currencyservice has many callers (cartservice, checkoutservice, frontend, paymentservice, recommendationservice), so a generic load fault produces a fan-out signature without a clear local origin.

adservice and emailservice (the two services that get 3-fault and 5-fault coverage respectively) hit ~50%, which is the realistic accuracy under the RE3-OB fault generator.

Per-fault breakdown

RE3 ships faults under opaque IDs f1 to f5 (the dataset doesn't expose the underlying chaos primitive). The distribution is uneven across services:

Fault	n	✓ correct	∅ abstain	— no-rca	rate
f1 (cartservice + currencyservice + emailservice)	9	2	5	2	22%
f2 (emailservice only)	3	2	0	1	67%
f3 (adservice + emailservice)	6	2	2	2	33%
f4 (adservice + emailservice)	6	3	2	1	50%
f5 (adservice + emailservice)	6	5	0	1	83%

f5 is the cleanest (5/6). f1 is the noisiest (2/9, but skewed by currencyservice's hard-to-resolve fan-out + the no_rca casualties). Without ground-truth fault-type mapping we can't say more. f5 may be CPU stress, which has a strong local signal, and f1 may be a network delay, which is dataset-wide ambiguous, but that is conjecture, not measurement.

What changed from RE1 v3 (76%) to RE3-OB (56.5%)

Three things, in priority order:

1. Dataset is structurally harder. RE3-OB tests 5 fault types per service (vs RE1's 5) but ships only 30 cases across 4 services (vs RE1's 125 across 5), and the fault distribution is severely uneven (emailservice has 15 of 30 cases). Fewer instances per (service, fault) means each single mis-resolve drops the percentage by 3.3 points.

2. The named-metric scorer was under load throughout the run. During the ingest the engine pool exhausted twice (logged as clickhouse: acquire conn timeout), and 7 cases fell into no_rca: the webhook fired but the agent's tool calls hit deadline-exceeded so no analysis got persisted. Re-fireable cases (mostly emailservice/f3,f4 instances) would lift accuracy materially.

3. f5 is the only fault class scoring above 75%. f1-f4 are all between 22%-67%. This is the gap to RE1: RE1's delay/cpu/mem faults had recognizable signatures; RE3's f1-f5 mix appears to include faults where the alert anchor isn't the loudest signal-bearer (similar to what we hit in RE1 delay before the anchor-loudest fix in v3, just spread across more fault types now).

What this scorecard does prove

1. Abstention contract holds on a third public benchmark. 0/30 confident-wrong. The trust contract is intact under load, on a never-seen-before fault library, with opaque fault labels.
2. Top-1 of 56.5% on the harder RE3 dataset still beats the BARO Avg@1 baseline (0.27) by over 2×.
3. The engine's v0.3.1-rc3 accuracy fixes (anchor-loudest + namespace isolation) generalize to RE3. The in-window cases that completed produced the expected per-service resolutions.
4. Pool-bump + scorer-exclude fix shipped together (v0.4.0-rc6 + CLICKHOUSE_MAX_OPEN_CONNS=200) restored login latency from 5s to sub-200ms even with parallel RE2+RE3 ingest running. The chronic leak source (scoring CTE on customer tenants) still exists but is now absorbed by the larger pool.

What this scorecard does not prove

1. Generalization to traces. RE3 ships rich traces.csv + logs.csv per case; this run was metric-only. The engine's trace tools (get_traces, get_trace_blast_radius, search_logs) were not exercised against this data. Expected lift on a multimodal re-run: ≥10 points.
2. No-RCA != fail. 23% of cases got no RCA at all, but that was the load-test artifact during the engine's pool exhaustion incident, not the engine's resolution capability. Re-fire under the post-fix engine to recover the true ceiling.
3. Fault-class semantics. RE3 doesn't expose f1-f5 → fault-name mapping. We can't claim "X% on memory faults" the way RE1 lets us. Per-fault numbers above are diagnostic, not categorical.
4. Statistical significance. N=30 is small; emailservice over-representation skews the aggregate. RE3-SS + RE3-TT runs would average it out (each adds another 30-125 cases).

Next-step priority

1. Re-fire the 7 no_rca cases under the post-fix engine. Expected lift: 56.5% → ~70% (gets us back to RE1 v3 territory).
2. Wire trace ingest from RE3's traces.csv. The engine's trace tools are unexercised on RCAEval; multimodal expected lift ≥10 points.
3. Score the RE2-OB run when ingest finishes (in progress at scorecard time, 39/91 cases).
4. Real fix for the named-metric scorer. Pool bump 50→200 absorbs the leak today; a tenant-loop refactor closes it properly. See project_login_pool_leak_recurrence.md.

Methodology

# Loadgen-side (Mac is orchestration-only per session feedback)
# 1. Cross-compile harness for linux/amd64 on Mac
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -ldflags='-s -w' \
    -o /tmp/infrasage-rcaeval-linux ./cmd/infrasage-rcaeval
scp /tmp/infrasage-rcaeval-linux loadgen:~/bin/infrasage-rcaeval

# 2. Download RE3-OB on loadgen (AWS pipe, ~3 min via curl; Zenodo throttles home IPs)
ssh loadgen 'cd ~/rcaeval-data && curl -L -o RE3-OB.zip \
  https://zenodo.org/records/14590730/files/RE3-OB.zip?download=1 && \
  unzip RE3-OB.zip'

# 3. Append RE3 cases to existing rcaeval tenant (preserves RE1 v3 results)
./bin/infrasage-rcaeval setup --no-wipe \
    --ch tcp://default:infrasage@localhost:19000/default \
    --system ~/rcaeval-data/RE3-OB

# 4. Ingest 30 RE3 cases (loadgen → public OTLP gateway + console webhook)
ssh loadgen '~/bin/infrasage-rcaeval ingest \
    --system ~/rcaeval-data/RE3-OB \
    --otlp https://api.infrasage.dev \
    --engine https://console.infrasage.dev \
    --ch "tcp://default:infrasage@localhost:19000/default" \
    --api-key isage_<key> \
    --max 30 --rca-wait 90 --pre-inject-min 15 --post-inject-min 5'

# 5. Score (Mac-side, reads infrasage_rca_results)
./bin/infrasage-rcaeval score \
    --ch tcp://default:infrasage@localhost:19000/default \
    --system ~/rcaeval-data/RE3-OB

Per case: ~1240 OTLP datapoints (60s bucket, 70-min window). Total: ~37k OTLP rows + ~37k per-metric z-scores. Wall-clock for the 30-case ingest: ~70 minutes including the two engine pod-restarts triggered mid-run.

Engine env at scoring time:

• ghcr.io/infrasagedev/infrasage:0.4.0-rc6
• CLICKHOUSE_MAX_OPEN_CONNS=200 (bumped from default 50 during the incident)
• CLICKHOUSE_MAX_IDLE_CONNS=50

Public attribution

Dataset: RCAEval RE3-OB (Pham et al., FSE'26 + WWW'25 + ASE'24), MIT. https://github.com/phamquiluan/RCAEval

This scorecard does not supersede the RE1-OB v3 scorecard; they cover different fault libraries and case mixes. The two are complementary evidence on the engine's generalization across the RCAEval suite.