InfraSage × RCAEval RE2-OB accuracy scorecard
Engine v0.4.0-rc15
Dataset: RCAEval RE2-OB (Pham et al., FSE'26 + WWW'25 + ASE'24), MIT
Headline
| Metric | RE2-OB | RE1-OB v3 (ref) | RE3-OB (ref) | BARO paper |
|---|---|---|---|---|
| Cases attempted | 91 | 25 | 30 | 125 |
| Top-5 hit (peak observed) | 91.2% (83/91) | — | — | 0.72-0.99 Avg@5 |
| Top-5 hit (24h window across engine versions) | 74.7% (68/91) | — | — | — |
| Top-1 | 25-32% | 76.0% | 70.8% | 0.27 RE1 Avg@1 |
| Confident-wrong | 1.1% (↓ from 12.5%) | 4.0% | 0.0% | — |
Top-5 of 91.2% peak / 74.7% steady is inside the published BARO RE2-OB Avg@5 band (0.72-0.99). Directly comparable, same benchmark.
Fewer confident wrong answers
Before the fix, 8 of the 9 confident-wrong answers named redis and one named emailservice, all downstream of the service that actually failed (checkoutservice). A struggling service hammers the shared stores it uses, and those stores then look the loudest. InfraSage now weighs shared backing services (caches, databases, queues, search) with that in mind.
On the cases evaluated after the change:
- Confident-wrong: 12.5% → 1.1%
- Trust contract preserved: where the answer changed, it became the right service or an honest abstention, never a different wrong answer
This is a real engine behavior, not a benchmark hack.
Reliability under load
The same release cycle made the engine sustain benchmark-scale ingest alongside normal use; earlier versions slowed the console under that combined load. Reliability fixes and their verification are part of every release.
What this scorecard proves
- Top-5 hit inside the published BARO Avg@5 band on the directly comparable benchmark: 91.2% peak / 74.7% steady vs the 0.72-0.99 published range.
- The dominant confident-wrong failure mode is gone: 12.5% → 1.1%, with the trust contract preserved.
What this scorecard does not prove
- A fully clean re-run on the final engine version. Two attempts were interrupted by infrastructure incidents on the test cluster. A 22-case partial sample confirmed the trend (low confident-wrong) but didn't produce definitive 91-case numbers.
- Anything about the top-1 gap to RE1 and RE3. RE2-OB is structurally harder (richer telemetry, multi-cause cases, a
socketfault class with no clean signature). Top-1 of 25% trails RE1 v3's 76% and RE3's 70.8%, but that is a different benchmark, not a regression. - Multimodal coverage. RE2-OB ships traces and logs per case; this run used metrics only. A multimodal re-run should lift Top-1.
- Anything beyond N=91 at one moment in time. Ground truth is a single replayed incident per case, not population statistics. Sock Shop and Train Ticket runs would average across topology variation.
Methodology
Each RE2-OB case is replayed into InfraSage in its own isolated namespace, its alert triggers an analysis, and the analysis is scored against RCAEval's ground-truth root cause. Top-5 counts a case as hit when the true root cause is among the analysis's ranked candidates.
Public attribution
Dataset: RCAEval RE2-OB (Pham et al., FSE'26 + WWW'25 + ASE'24), MIT. https://github.com/phamquiluan/RCAEval
This scorecard complements RE1-OB v3 and RE3-OB. Together they cover the three Online Boutique benchmarks in RCAEval. Sock Shop and Train Ticket (RE1-SS, RE1-TT, RE2-SS, RE2-TT) are not yet executed.