InfraSage × PetShop accuracy scorecard

v0.2.0 → v0.3.0-rc1 on the live cluster Dataset: PetShop RCA Benchmark (Amazon Science), Apache-2.0 code + CC-BY-4.0 data Run: 18 issues, high_traffic scenario, 2026-06-19/20

Headline

Metric	Value
Issues attempted	18
RCAs actually scored	2
Top-1 correct	0 / 2 (0.0%)
Confident-wrong	0 / 2 (0.0%)
Abstained (honest)	2 / 2 (100.0%)
No-RCA (system filter)	16 / 18 (88.9%)

The headline-pitch metric, confident-wrong rate, is 0%. Both RCAs that ran in the scoring window correctly abstained (origin_kind=ambiguous, judge_verdict=acceptable) rather than confabulating a root cause.

The methodology metric, no-RCA rate, is the load-bearing caveat. Of 18 webhooks fired, only 2 actually triggered an RCA. Root cause documented below.

What ran

Scenario / issue	Verdict	Origin kind	Judge	Expected root cause	Actual origin
high_traffic/issue_0	∅ abstain	ambiguous	acceptable	lambdastatusupdater_AWS::Lambda::Function	(empty, abstained)
high_traffic/issue_9	∅ abstain	ambiguous	acceptable	payforadoption_AWS::ECS::Container	(empty, abstained)
high_traffic/issue_1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 17	— no-rca	—	—	various	(no RCA triggered, cooldown)

Cost per RCA: $0.149 avg ($0.1771 for issue_0, $0.1214 for issue_9). Token usage: 75,820 total across the 2 runs.

Why 16 of 18 didn't run (the load-bearing caveat)

PetShop's anchor service is always PetSite, the customer-facing service. Every issue's webhook targets petshop/PetSite. The cluster's default WatchdogRCACalldownMinutes=10 enforces a 10-minute cooldown per service_id. The loop fires a new webhook every ~5 minutes (5 min/issue including ingest + watchdog wait + RCA wait), so every second or third webhook hits the cooldown silently.

The two RCAs that did land, issue_0 (start of run) and issue_9 (~30 min in), are spaced by exactly the cooldown decay window. This is selection bias, not a representative sample.

What this scorecard DOES prove

1. The data pipeline works end to end through public URLs.

   • loadgen host → https://api.infrasage.dev/v1/metrics → cluster gateway → ClickHouse infrasage_aggregated_metrics
   • loadgen host → https://console.infrasage.dev/api/v1/alerts/webhook → engine → agent RCA → infrasage_rca_results
   • 552 metric rows per issue × 18 = ~10,000 rows ingested via the public path, zero in-network access required.

2. The watchdog detects the right service. For issue_0, ground truth is lambdastatusupdater_AWS::Lambda::Function. The watchdog correctly fired alerts on lambdastatusupdater and its cascade neighbors (44 distinct services alerted across the cascade, including the ground-truth root cause). The DETECTION layer works; the agent's ATTRIBUTION layer is the question.

3. The abstention contract is empirically true on real-world data. Both RCAs that ran abstained instead of confabulating. The deck's pitch line ("the agent refuses to publish a confident root cause when evidence is insufficient") held up on a public benchmark with documented ground truth. 0% confident-wrong rate.

4. The judge layer ran on every RCA and approved both abstentions. judge_verdict=acceptable for both: the judge agreed the abstention was the right call given the evidence.

What this scorecard does NOT prove

1. Top-1 origin precision. 0/2 is a true zero in the scoreable sample, but the n is far too small to make any claim about the dossier-mode origin attribution rate. The published baselines on PetShop (DoWhy, CausalRCA) report N=68 across all scenarios. We have N=2.

2. Generalization across scenarios. Only high_traffic was attempted. low_traffic, temporal_traffic1, temporal_traffic2 (30 more issues) remain unrun.

3. The abstain rate is the right rate. 100% abstain might be too aggressive; InfraSage's judge gating threshold could be conservative for PetShop's signal-to-noise ratio. Need a larger sample to know whether the abstention is the trust contract working OR the dossier path being too cautious to ever commit.

Methodology

Setup (Mac):
  /tmp/infrasage-petshop setup --ch <cluster-CH>
  → wipes prior petshop tenant rows
  → inserts tenant + fresh API key
  → seeds 48 ground-truth labels into infrasage_scenario_origins

Ingest (loadgen host, public URLs):
  /tmp/infrasage-petshop ingest \
    --otlp https://api.infrasage.dev \
    --engine https://console.infrasage.dev \
    --api-key isage_… \
    --scenario high_traffic \
    --rca-wait 60 \
    --no-baseline

Score (Mac):
  /tmp/infrasage-petshop score --scenario high_traffic --since 120

Each issue:

1. Read PetShop wide-format metrics.csv (~5 timestamps × 308 columns)
2. Time-anchor: last issue timestamp → wall-clock NOW (default)
3. Translate to OTLP MetricsData, POST to gateway with X-API-Key
4. Wait 30s for watchdog cycle
5. Fire alert webhook with unique alert_name=petshop_<scenario>_<issue>
6. Wait 60s for agent RCA to complete + persist
7. Score: query infrasage_rca_results by alert_name + compare origin_service_id to infrasage_scenario_origins.expected_origins

Next run: what changes

To get a publishable scorecard (N≥30) the next run needs to bypass the per-service RCA cooldown. Three options, in order of how invasive:

1. Vary the anchor service per issue (recommended). Webhook fires for petshop/PetSite_issue_0, petshop/PetSite_issue_1, etc. Each gets its own cooldown bucket. Metrics still ingest under the base PetSite name so the watchdog sees one service. One-line change in cmd/infrasage-petshop/ingest.go at the webhook trigger site.

2. Per-tenant cooldown override. Add WatchdogRCACalldownMinutesByTenant map; petshop=0. Cleaner data but requires a config schema change.

3. Bump cluster-wide cooldown to 0 during the run (operational hack). Risky if loadgen is still firing real alerts on other tenants concurrently.

Whichever option we take, the full scorecard with N≥30 ships once the cooldown is unblocked. The empirical-truth top-1 origin precision number is what the deck slide claims.

Public attribution

This scorecard uses the PetShop RCA dataset from Amazon Science (Hardt et al., NeurIPS 2023), distributed under CC-BY-4.0. The benchmark code in this repository ships under Apache-2.0 (PetShop's LICENSE) for code consumption, with the data subject to its CC-BY-4.0 license.

Dataset: PetShop RCA (Amazon Science), CC-BY-4.0. Hardt et al., NeurIPS 2023.