InfraSage × PetShop accuracy scorecard
Engine v0.2.0 → v0.3.0-rc1 Dataset: PetShop RCA Benchmark (Amazon Science), Apache-2.0 code + CC-BY-4.0 data Run: 18 issues, high_traffic scenario, 2026-06-19/20
Headline
| Metric | Value |
|---|---|
| Issues attempted | 18 |
| Analyses scored | 2 |
| Top-1 correct | 0 / 2 (0.0%) |
| Confident-wrong | 0 / 2 (0.0%) |
| Abstained (honest) | 2 / 2 (100.0%) |
| No analysis | 16 / 18 (88.9%) |
The confident-wrong rate is 0%: both analyses that ran abstained, and an independent review judged that the right call, rather than inventing a root cause.
The no-analysis rate is the caveat that matters. Of 18 alerts, only 2 produced an analysis; why is explained below. This run does not measure PetShop accuracy, and we don't claim a number for it.
What ran
| Scenario / issue | Verdict | Review | Expected root cause | Answer |
|---|---|---|---|---|
| high_traffic/issue_0 | ∅ abstain | acceptable | lambdastatusupdater_AWS::Lambda::Function | (abstained) |
| high_traffic/issue_9 | ∅ abstain | acceptable | payforadoption_AWS::ECS::Container | (abstained) |
| high_traffic/issue_1-8, 10-17 | — no analysis | — | various | (per-service cooldown) |
Cost per analysis: $0.149 on average ($0.1771 for issue_0, $0.1214 for issue_9).
Why 16 of 18 didn't run
Every PetShop issue alerts on the same customer-facing service, PetSite. InfraSage spaces out repeated analyses of the same service with a cooldown, and the benchmark fired a new alert every ~5 minutes, inside it. The two analyses that ran, issue_0 at the start and issue_9 about 30 minutes in, are exactly the ones that fell outside the cooldown. That is selection bias, not a representative sample.
What this scorecard does prove
- The pipeline works end to end through public endpoints: ~10,000 metric rows ingested over the public API, alerts delivered by webhook, analyses completed and stored, with no in-network access.
- Detection found the right service. For issue_0, InfraSage alerted on the ground-truth root cause and its neighbors in the cascade. Detection worked; attribution is the open question.
- The abstention contract held on real-world data. Both analyses abstained instead of guessing, and the independent review approved both abstentions.
What this scorecard does not prove
- Top-1 accuracy. N=2 is far too small. Published baselines on PetShop (DoWhy, CausalRCA) use N=68 across all scenarios.
- Generalization across scenarios. Only
high_trafficran;low_traffic,temporal_traffic1andtemporal_traffic2(30 more issues) haven't. - That abstaining was the right rate. 100% might be too cautious for PetShop's signal-to-noise ratio; a larger sample will tell.
Methodology
Each issue's PetShop metrics are replayed into InfraSage, shifted to the present, through the public ingestion API. The issue's alert is then fired by webhook, and the resulting analysis is scored against PetShop's ground-truth root cause.
Next run
A publishable scorecard (N ≥ 30) needs each issue analyzed independently of the cooldown. The next run gives each issue its own alert identity.
Public attribution
This scorecard uses the PetShop RCA dataset from Amazon Science (Hardt et al., NeurIPS 2023), distributed under CC-BY-4.0.
Dataset: PetShop RCA (Amazon Science), CC-BY-4.0. Hardt et al., NeurIPS 2023.