Custom Pillars

A pillar is a data domain.

Metrics, logs, traces and events are the four InfraSage ships with, because they are what an observability platform ingests by default. They are not the definition. Anything a company wants watched can be a pillar: SMS delivery, call detail records, payment transactions, queue backlogs, conversion funnels. That data has nothing to do with infrastructure and everything to do with whether the business is working.

A custom pillar gives that data the same treatment your services get: a statistical baseline, anomaly detection with calibrated severity, alerts in the shared list, and a place in cross-domain correlation.

The shape of a pillar

Field	What it is
pillar_id	Stable identifier, unique per tenant
subject_type	What the keys identify: sms_route, merchant, queue, service
sql_query	A SELECT over your data. The last column is the value
dimension_columns	Which columns form the subject key
z_threshold	Sigma from baseline before firing (default 3.0)
direction	up, down, or both
tick_interval_seconds	How often to evaluate (minimum 60)

subject_type is the field that makes a domain a first-class citizen. Without it a key like vodafone-in is opaque text: it can be detected on, but nothing can look it up, relate it to anything, or investigate it. With it, the anomaly says SMS route vodafone-in and the platform knows what kind of thing that is.

Getting data in

Two routes, depending on where your data lives.

Push it

curl -X POST https://api.infrasage.dev/api/v1/pillars/sms_delivery/observations \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '[
    {"ts": "2026-09-01T10:00:00Z", "value": 0.94, "dimension_key": "route:vodafone-in"},
    {"ts": "2026-09-01T10:00:00Z", "value": 0.99, "dimension_key": "route:airtel-in"}
  ]'

Any data domain, any shape, as long as it reduces to a number per key per moment. POST /api/v1/business/kpis remains as an alias for this endpoint.

Query it in place

If the data already reaches ClickHouse (through a connector, the OTLP pipeline, or a KPI push), a pillar can read it directly with SQL, without keeping a second copy.

Writing the SQL

Operator SQL runs in a sandbox: a dedicated read-only ClickHouse user, a table allowlist, a five-second cap, and an automatic tenant predicate. Three rules follow from that, and all three are easy to trip.

Project tenant_id, but never filter on it

The sandbox injects its own tenant predicate and refuses any query that references tenant_id in a filter position. Otherwise WHERE tenant_id='someone_else' OR 1=1 would slip past it.

-- Good: tenant_id projected before the WHERE
SELECT any(tenant_id) AS tenant_id, dimension_key AS subject, avg(value) AS obs
FROM infrasage_business_kpis
WHERE kpi_name = 'sms_delivery_rate'
GROUP BY dimension_key

:::caution The guard is a text scan The check looks for tenant_id within 400 characters after a WHERE, HAVING or ON. A GROUP BY tenant_id that happens to sit inside that window is rejected too, even though it is harmless. Project tenant_id with any() before the WHERE and group by your subject column instead. :::

Use a closed, lagged window

WHERE ts >= toStartOfMinute(now() - INTERVAL 3 MINUTE)
  AND ts <  toStartOfMinute(now() - INTERVAL 2 MINUTE)

An open window (ts > now() - INTERVAL 5 MINUTE) races your own ingest. Rows arrive a beat after their timestamp, so a query run at the boundary systematically undercounts, by around 19% in our own measurements. A closed window lagged two minutes is stable and independently recomputable, which also means you can verify what the pillar saw.

The last column is the value

Everything named in dimension_columns forms the subject key, joined by |. The final projected column is the measurement.

Templates

Three domain templates ship for the common shapes, alongside the infrastructure probe and KPI templates:

Template	Watches	Fires on
custom.delivery_rate	Success or delivery rate per route, carrier or channel	A drop
custom.decline_rate	Decline or failure rate per gateway, processor or provider	A rise
custom.queue_depth	Backlog per queue, topic or partition	A rise

Each is a starting point, not a constraint. The SQL is yours to edit.

Authoring in the UI

Pillars → Author a pillar walks through it:

1. Describe what you want watched, in plain language
2. Suggest: the wizard proposes templates and fills them in
3. Dry-run: the SQL executes in the sandbox against your real data, and you see the rows it would produce
4. Promote: the pillar is created

Promoted pillars are created disabled. Enabling is a second, deliberate step, so authoring a pillar can never start billing or paging on its own. Where the per-tenant policy gate is enabled, a tenant must also opt the custom-pillar kind in.

What happens when it fires

An anomaly becomes an alert in the shared list, labelled with its subject:

[high] sms-delivery down — sms_route vodafone-in
z=-5.20 obs=10.00 mean=100.00 stddev=10.00

Severity comes from the same calibrated z-score buckets your infrastructure alerts use, so a high from a custom pillar means what a high from a service means.

Cross-domain correlation

When several pillars have something to say about the same subject in the same minute, they are correlated into one verdict rather than N separate alerts:

Subject	Pillar	Score	Active pillars	Combined
sms_route/vodafone-in	sms-delivery	5.2	2	5.2
sms_route/vodafone-in	sms-latency	3.8	2	5.2
sms_route/airtel-in	sms-delivery	4.1	1	4.1

combined_score is the maximum across pillars, not a sum: a subject watched by ten pillars should not outrank one watched by two purely on breadth. active_pillars carries the corroboration separately, because two independent domains agreeing is a stronger signal than one shouting.

Investigation

Every subject a pillar observes is registered in the topology graph. A service-typed subject resolves to the same identity the topology discoverer uses, so it lands on the existing dependency graph and inherits its edges immediately.

RCA can anchor on a subject directly:

{
  "subject_type": "sms_route",
  "subject_key": "vodafone-in",
  "tenant_id": "acme",
  "max_hops": 2
}

:::note A new domain starts unconnected A non-service subject is registered as a node but has no edges until something relates it to the rest of your estate. RCA will say so rather than implying the graph can help. Investigate through the pillar's own observations and any co-occurring alerts in the same window. :::

Limits

• Minimum tick interval is 60 seconds
• Operator SQL is capped at 5 seconds and may read only infrasage_* tables
• A baseline needs history before it fires; expect a quiet first few hours
• system.*, table functions, and every DDL/DML verb are rejected outright