Service level objectives¶

Sandloom exports the metrics an SLO needs; it does not choose your objectives. This page defines SLIs on the shipped metrics, gives example SLOs with starting targets measured on a small stand, and a template for the error budget and its policy. The SLIs are Prometheus recording rules in deploy/observability/prometheus/rules/sandloom-slo.yml; the Sandloom / SLOs Grafana dashboard plots their budgets and burn rates.

SLIs¶

An SLI is the share of good events among valid events. The recording rules store the bad share (1 - good/valid) per job, over 5m, 30m, 1h and 6h windows, as sandloom:sli_<name>:ratio_rate<window>.

Definitions and their limits:

• Probes are not traffic. /health*, /ready, /metrics and unmatched paths (<unmatched>, i.e. 404 scans) are excluded.
• 4xx are good events. 401, 403, 404, 409, 422 and 429 are the API working as designed. A deployment that wants "rate limited" to count against it adds status="429" to the bad set.
• Reads vs writes is the HTTP method. Routes carry no class label (route policy is derived from the method, the authenticator and the declared risk; none of it is a metric label). For an SLO on one route (or a group of routes), add a path=~"…" matcher to the rules.
• The latency histogram has no status label: a fast 500 counts as fast. Read latency and availability together.
• Thresholds must be bucket boundaries. The HTTP histogram uses prometheus_client's default buckets (… 0.1, 0.25, 0.5, 0.75, 1, 2.5, 5 … seconds); the event histogram … 1, 2.5, 5, 10, 30, 60 …. Pick an SLO threshold from those, or the SLI is interpolated guesswork.
• Server-side only. The app measures from its own socket: TLS, proxies, the gateway and the network are outside the SLI. An edge SLO needs the balancer's or a synthetic probe's metrics.
• Event latency includes clock skew between the producer's database (emitted_at) and the consumer host. Keep NTP on both.
• Consumer lag has no ratio SLI. Track it as a threshold instead: sandloom_streams_group_oldest_unacked_age_seconds (alert SandloomConsumerGroupBehind).

Example SLOs¶

For a 2 vCPU monolith (2 Gunicorn workers, PostgreSQL on 2 vCPU), measured over a rolling 30 days. "Load envelope" is the load under which the target was met on the stand — above it, expect to miss the latency SLO first.

Why these numbers and not tighter ones:

• Reads: p99 at 500/s sits at 190–230 ms, so 250 ms leaves little headroom; it is the tightest bucket boundary the stand met. Planned single-route reads and /auth/me saturate near 1 200/s, ordinary reads near 690/s — the envelope is ~70 % of the ordinary-read saturation point.
• Writes: at 200/s the p99 is ≤ 211 ms; 500 ms absorbs bursts towards 290/s (where p99 already reaches 409–634 ms). Financial (F) writes saturate around 310–360/s, ordinary (W) around 470/s in closed loop.
• Login is dominated by the password hash on purpose. Its capacity (≈ 9/s on 2 vCPU) is a planning number: see PASSWORD_HASH_PROFILE in docs/operate/operator-reference.md → Password hashing. Do not "fix" a login SLO by weakening the hash.
• Events: the durable consumer's whole callback costs 13–15 ms under HTTP load; a pure drain without HTTP traffic reaches ≈ 510–520 events/s, but that is burst-drain capacity, not a sustained mixed rate. Between 200/s and 300/s the consumer group starts to fall behind — set the envelope below the rate where SandloomConsumerGroupBehind would fire, and scale consumers (more processes) before raising it.

Where the numbers come from¶

Oracle qualification of 2026-10-22 (f898857a vs b609b5ee), private evidence benchmarks/qualification/.state/oracle-main-20261022-f898857a/: four ARM Neoverse N1 vCPUs host, app limited to 2 vCPU / 1 GiB with two Gunicorn workers, PostgreSQL 16 on 2 vCPU / 1 GiB, separate cache, security and broker Redis (AOF appendfsync=always), pool 5 + 5 per worker, strict production RLS. The load generator ran on the same host without TLS, proxy or network in the path, so production latencies are higher by your edge's cost. Windows were 30–60 s and small in number; they establish orders of magnitude, not confidence intervals.

Error budget¶

The budget is what the SLO allows to go wrong: 1 − target of the valid events in the window.

Burn rate = observed bad ratio ÷ budget. A burn rate of 1 spends exactly the budget over the window; 14.4 spends 2 % of a 30-day budget per hour (gone in ~2 days).

Burn-rate alerts¶

sandloom-slo.yml ships multi-window, multi-burn-rate alerts (Google SRE workbook) for read and write availability at the example 99.9 % target. A long window proves the burn is real; the short one makes the alert resolve quickly once it stops.

The latency, login and event SLIs are recorded but not alerted on by burn rate: their thresholds depend more on your hardware than availability does. Copy the availability pair, change the record name and budget (0.01 for a 99 % SLO), and add a minimum-traffic guard (and sum by (job) (rate(sandloom_http_requests_total[1h])) > 1) for low-traffic services, where a handful of errors is a large ratio.

Changing a target touches three places together: the alert expressions in sandloom-slo.yml, slo() in deploy/observability/build_dashboards.py (regenerate the dashboards) and the table above. CI checks the SLO board and the burn alerts agree on the budget.

Error-budget policy (template)¶

Agree on it before the budget runs out; adapt the thresholds.

Record for each SLO: the owner, the target and window, the dashboard link, the alerts that protect it, and the date it was last reviewed (quarterly, or after any capacity change).

Related reading¶

• Runbook
• Security model