Skip to main content

RCA accuracy leaderboard

InfraSage is the first commercial RCA vendor to publish per-benchmark top-1 accuracy and confident-wrong rate under the industry-standard RCAEval methodology. Confident-wrong is the metric no correlation-based competitor reports. It measures how often the RCA points at a specific wrong service instead of abstaining.

Our discipline: we say we don't know when we don't.

Headline results​

BenchmarkCasesTop-1Top-5Confident-wrongDetails
RCAEval RE2-OB9125% (steady)91.2% peak / 74.7% steady1.1% (↓ from 12.5% pre-amplifier-guard)scorecard
RCAEval RE3-OB3070.8%—0%scorecard
RCAEval RE1-OB2576.0%—4.0%scorecard
PetShop2588%91.2%—scorecard

Where we sit vs published baselines​

  • RE2-OB Top-5: 91.2% peak / 74.7% steady lands inside BARO's published Avg@5 band of 0.72-0.99. Directly comparable, same benchmark.
  • RE1-OB Top-1: 76% vs BARO's RE1 Avg@1 of 0.27. 2.8× on the same benchmark.
  • Confident-wrong: we report it. Neither HolmesGPT, Aurora, Traversal, OpenDerisk, Datadog Bits AI SRE, nor Dynatrace Davis CoPilot publishes this number on any public RCA benchmark.

What this leaderboard proves​

  1. Directly comparable to academic baselines (BARO, RCD, CIRCA, MicroCause, PyRCA) using the same RCAEval harness. No custom benchmark, no unreproducible metrics.
  2. The abstention contract works. After shipping the amplifier-guard (a stateless-backing-service override for redis / postgres / kafka / mongo / etc.), confident-wrong dropped from 12.5% to 1.1% on RE2-OB. The trust contract held: wrong answers turned into honest abstentions, never into different-wrong answers.
  3. Multi-benchmark coverage. RE1-OB + RE2-OB + RE3-OB + PetShop spans metrics-only, metrics+traces, and cascade-heavy scenarios.

What this leaderboard does not prove​

  1. Sock Shop and Train Ticket are unexercised. RE1-SS, RE1-TT, RE2-SS, RE2-TT, RE3-SS, RE3-TT scorecards are next. Different service topologies may shift the numbers.
  2. Multimodal lift is not yet captured on RE2/RE3. These runs used simple_metrics.csv only. RE2/RE3 ship trace and log data, and a multimodal re-run should lift Top-1 by at least 10 points.
  3. Evaluation is a single timestamp per case. Ground truth is a single replay-shifted incident. Population statistics would require running each case across multiple time windows.

Methodology​

All scorecards use github.com/phamquiluan/RCAEval datasets replayed against the InfraSage engine via cmd/infrasage-rcaeval. Case-level isolation (per-case service namespaces) prevents cross-case baseline contamination, a documented pitfall other benchmark harnesses fall into.

Each scorecard page includes:

  • The exact engine version + config
  • The reproduction command
  • Per-case ground-truth vs prediction table
  • The specific failure modes we abstain on
  • Fault-type breakdown (CPU vs memory vs disk vs delay vs loss vs socket)

Attribution​

Dataset: RCAEval (Pham et al., FSE'26 + WWW'25 + ASE'24), MIT license. https://github.com/phamquiluan/RCAEval

Please cite the RCAEval paper when comparing against these scorecards.

Contribute​

Running a benchmark of your own? Open a PR against github.com/sushant-115/infrasage with your bench/*.md scorecard. We accept RCAEval reruns, adversarial suites, and per-domain custom scenarios. The goal is to make published RCA accuracy the industry norm, and the OSS peers who currently publish nothing are the place to start.