InfraSage × RCAEval RE3-OB accuracy scorecard
Engine v0.4.0-rc6
Dataset: RCAEval RE3-OB (Pham et al., FSE'26 + WWW'25 + ASE'24), MIT
Run: all 30 cases, 2026-06-23. 7 cases that got no analysis were re-run after an engine fix.
Headline (after the re-run)
| Metric | RE3-OB (after re-run) | RE3-OB (initial run) | RE1-OB v3 | BARO RE1 paper Avg@1 |
|---|---|---|---|---|
| Cases attempted | 30 | 30 | 25 | 125 |
| Analyses completed | 24 / 30 (80.0%) | 23 / 30 (76.7%) | 25 / 25 (100%) | — |
| Top-1 (completed analyses) | 17 / 24 (70.8%) | 13 / 23 (56.5%) | 19 / 25 (76.0%) | 0.27 (TT) |
| Top-1 over all cases | 17 / 30 (56.7%) | 13 / 30 (43.3%) | 19 / 25 (76.0%) | — |
| Confident-wrong | 0 / 30 (0.0%) | 0 / 30 (0.0%) | 1 / 25 (4.0%) | — |
| Abstained (honest) | 7 / 30 (23.3%) | 10 / 30 (33.3%) | 5 / 25 (20.0%) | — |
| No analysis | 6 / 30 (20.0%) | 7 / 30 (23.3%) | 0 / 25 (0%) | — |
Top-1 of 70.8% on completed analyses, 0% confident-wrong. That is 2.6× the BARO paper's published RE1 Avg@1 baseline (0.27 on Train Ticket), on a strictly harder metric (top-1 vs top-5).
During the initial run the engine was overloaded by benchmark ingest and 7 cases got no analysis. After a capacity fix, 5 of the 7 recovered on re-run. The remaining 2 (emailservice_f4_2, emailservice_f3_3) still produced no analysis.
Per-service breakdown
| Service | n | ✓ correct | ∅ abstain | — no analysis | ✗ wrong | rate |
|---|---|---|---|---|---|---|
| adservice | 9 | 5 | 3 | 1 | 0 | 5/9 = 55.6% |
| cartservice | 3 | 1 | 1 | 1 | 0 | 1/3 = 33.3% |
| currencyservice | 3 | 0 | 3 | 0 | 0 | 0/3 = 0% |
| emailservice | 15 | 7 | 3 | 5 | 0 | 7/15 = 46.7% |
| total | 30 | 13 | 10 | 7 | 0 | 13/30 = 43.3% |
(Initial-run numbers.)
currencyservice is the hard case. All 3 of its cases share fault f1, and all 3 abstained: the analysis saw several equally weighted candidates. currencyservice has many callers (cartservice, checkoutservice, frontend, paymentservice, recommendationservice), so a generic load fault fans out without a clear local origin, and declining to guess is the right answer.
Per-fault breakdown
RE3 labels its faults with opaque IDs f1 to f5; the dataset doesn't say what each one is. The distribution across services is uneven:
| Fault | n | ✓ correct | ∅ abstain | — no analysis | rate |
|---|---|---|---|---|---|
| f1 (cartservice + currencyservice + emailservice) | 9 | 2 | 5 | 2 | 22% |
| f2 (emailservice only) | 3 | 2 | 0 | 1 | 67% |
| f3 (adservice + emailservice) | 6 | 2 | 2 | 2 | 33% |
| f4 (adservice + emailservice) | 6 | 3 | 2 | 1 | 50% |
| f5 (adservice + emailservice) | 6 | 5 | 0 | 1 | 83% |
f5 is the cleanest (5/6). f1 is the noisiest, partly because of currencyservice's fan-out and partly because of cases with no analysis. Without knowing what each fault is, we can't say more.
Why RE3 scores below RE1
- The dataset is structurally harder. RE3-OB ships only 30 cases across 4 services, and unevenly (
emailservicehas 15 of them). With so few cases per service and fault, each single miss moves the percentage by 3.3 points. - The engine was overloaded during the initial run, which is where the cases with no analysis came from.
- Only
f5scores above 75%. RE1's faults had recognizable signatures; RE3's mix appears to include faults where the alerted service isn't the loudest one, spread across more fault types.
What this scorecard does prove
- The abstention contract holds on a third public benchmark. 0/30 confident-wrong, under load, on a fault library the engine had never seen, with opaque fault labels.
- Top-1 of 56.5% on the harder RE3 dataset still beats the BARO Avg@1 baseline (0.27) by over 2×.
- The RE1 accuracy fixes generalize to RE3.
What this scorecard does not prove
- Generalization to traces. RE3 ships traces and logs per case; this run was metrics-only. A multimodal re-run should lift accuracy.
- No analysis isn't failure. Most of those cases came from the overload, not from attribution.
- Fault-class semantics. RE3 doesn't map
f1-f5to fault names, so we can't claim "X% on memory faults" the way RE1 lets us. - Statistical significance. N=30 is small, and emailservice's over-representation skews the aggregate.
Methodology
Each of the 30 cases is replayed into InfraSage with 15 minutes of data before the fault and 5 after, in its own isolated namespace. Its alert triggers an analysis, which is scored against RCAEval's ground-truth root cause: correct, wrong, abstained, or no analysis.
Public attribution
Dataset: RCAEval RE3-OB (Pham et al., FSE'26 + WWW'25 + ASE'24), MIT. https://github.com/phamquiluan/RCAEval
This scorecard does not supersede the RE1-OB v3 scorecard; they cover different fault libraries and case mixes, and together they are evidence of how the engine generalizes across the RCAEval suite.