Skip to main content

Runbooks & Remediation

A runbook is a declarative YAML spec that InfraSage compiles and executes as a dependency graph, behind human approval gates sized to how much the runbook has earned your trust. This page covers the execution side; see Authoring & Import for how runbooks get written, reviewed, and promoted.

The spec​

runbook_id: reset-payment-pool
name: Reset payment DB connection pool
description: >
  Recycles the payment service's DB pool. Use when RCA points at pool
  exhaustion / DB latency on payment-service.
service_id: payment-service
tier: T2
enabled: true
steps:
  - step_id: confirm-impact
    name: Confirm pool saturation before recycling
    skill_id: legacy:manual_confirm
    params:
      instruction: "Confirm payment p99 is still elevated before recycling."
  - step_id: reset-pool
    name: Recycle the connection pool
    skill_id: legacy:http
    params: { url: "https://ops.internal/payment/reset-pool", method: "POST" }
    depends_on: [confirm-impact]
    on_failure: stop
    max_retries: 2
  - step_id: announce
    name: Announce remediation
    skill_id: legacy:slack_notify
    params: { message: "payment DB pool recycled via runbook" }
    depends_on: [reset-pool]
    on_failure: continue

Step types: HTTP calls, Kubernetes operations (scale / restart / drain), gated shell, Slack notifications, timed waits, n8n workflow triggers, manual_confirm (a checkpoint that pauses the run until a human answers in the console or Slack), and runner_http, which executes an HTTP call inside your cluster via the in-cluster runner. That last one buys you cluster-local URLs, no public exposure of ops endpoints, and deterministic cluster targeting.

HTTP steps retry transport-level dial failures automatically, meaning a connection refused or a timeout before the request was sent. That is safe for any method, since the request never reached the server. HTTP status errors are never retried implicitly; use max_retries for that, deliberately.

Semantics worth knowing:

  • depends_on builds a DAG; independent branches run concurrently.
  • on_failure: stop | continue per step; downstream steps of a stopped branch are skipped.
  • max_retries re-attempts a failed step. HTTP steps additionally retry transport-level failures (connection refused/reset, where the request never reached the server) automatically, since those are safe for any method.
  • Strict variables: {{vars}} resolve from the trigger context; unknown variables fail compilation loudly rather than becoming empty strings.
  • Trigger context flows in. {{alert.severity}}, {{service_id}} and friends resolve from the alert or manual trigger that started the run.

How runs start​

  1. Manually, from an incident. The console's applicable runbooks panel curates promoted, enabled runbooks matching the alert's source; one click queues the run for approval.
  2. From the RCA. Analyses surface remediation candidates, and executing one goes through the same approval gate.
  3. From triggers (optional, flag-gated). Alert-matching rules on service glob, severity, detector, alert-name glob, environment, and labels fire runbooks automatically into the approval queue, not into execution.

The approval loop​

Every T2/T1 run pauses as a pending action in the Decision Queue (console and Slack). The card shows the runbook, target service, trigger context, and tier. Approve dispatches to the DAG engine and returns the execution id; reject records why. manual_confirm steps pause mid-run the same way. Who decided, when, and from which surface is all recorded.

Observing a run​

Console → Knowledge → Runbooks shows the catalog (tier, trust, last runs); each runbook's pane shows execution history with per-step status, inputs, outputs, and errors. For finished runs:

  • Replay reconstructs the step-by-step timeline for review.
  • Post-mortem export produces a downloadable markdown dossier of the run.
GET  /api/v1/runbooks                                  # catalog
GET  /api/v1/runbooks/{runbook_id}                     # spec + recent executions
POST /api/v1/runbooks/{runbook_id}/manual-trigger      # queue for approval
POST /api/v1/agent/actions/{action_id}/approve         # dispatch (or /reject)
GET  /api/v1/runbook-executions/{execution_id}         # status + step results
GET  /api/v1/runbooks/executions/{execution_id}/replay
GET  /api/v1/runbooks/executions/{execution_id}/postmortem

Verified end to end​

The full loop runs as a live control scenario against production, and it must pass before detection or remediation changes ship: detector fires → RCA names the cause and cites the deploy → operator approves reset-payment-pool → the HTTP step clears a sticky injected fault → recovery observed from the target's side.

:::note Network prerequisites Runbook steps execute from InfraSage's control plane, so HTTP targets must be reachable from its egress; private targets typically expose an allowlisted endpoint. An in-cluster runner that executes steps inside your network (no inbound exposure) is on the roadmap. :::