# Runbook

Find the situation, then follow the procedure it links to in the [operator reference](<https://sandloom.dev/docs/operate/operator-reference/index.md>).

Operator commands (`sandloom migrate`, `outbox`, `streams`, `projection`, `doctor`) boot your application, so set `SANDLOOM_COMPOSITION_ROOT=<app>.asgi` for them, as the generated profile does. Run them with `--apply` only after reading the dry run.

## Something is wrong

| Situation | Start with | Procedure |
| --- | --- | --- |
| an alert fired | its row in the alert table | [Alerts](<https://sandloom.dev/docs/operate/operator-reference/#alerts>) |
| a process won't start or isn't ready | `sandloom doctor` | [Preflight](<https://sandloom.dev/docs/operate/operator-reference/#preflight-and-effective-configuration>) |
| events stopped arriving, or some were dead-lettered | `sandloom outbox list-dead`, `sandloom streams consumers` | [Events](<https://sandloom.dev/docs/operate/operator-reference/#events-outbox-relay-and-dead-letters>) |
| a saga is stuck | `sandloom sagas list` | [A parked saga](<https://sandloom.dev/docs/operate/operator-reference/#runbook-a-parked-saga>) |
| a task may or may not have been enqueued | `TaskEnqueueUncertain`: don't retry with another key | [Uncertain enqueues](<https://sandloom.dev/docs/operate/operator-reference/#uncertain-task-enqueues>) |
| a database or a key leaked | freeze deploys, `sandloom security audit-export` | [Security incident](<https://sandloom.dev/docs/operate/operator-reference/#security-incident-runbook-database-leak-credential-leak-key-compromise>) |
| data was lost | stop writers, restore in order, `sandloom backup reconcile-security` | [Restoring](<https://sandloom.dev/docs/operate/operator-reference/#restoring-the-order-and-the-traffic-gate>) |
| Redis is slow or unavailable | the `CACHE_*` limits and each caller's failure policy | [Redis cache](<https://sandloom.dev/docs/operate/operator-reference/#redis-cache-timeouts-pool-revocation-cache>) |
| login is slow or CPU spikes at login | `PASSWORD_HASH_PROFILE` | [Password hashing](<https://sandloom.dev/docs/operate/operator-reference/#password-hashing-work-factor-capacity-planning>) |
| latency spikes with no slow queries | the `GC_*` settings and metrics | [Garbage collector](<https://sandloom.dev/docs/operate/operator-reference/#garbage-collector-tuning>) |

## Routine work

| Task | Procedure |
| --- | --- |
| store or rotate secrets and keys | [Secrets management](<https://sandloom.dev/docs/operate/operator-reference/#secrets-management>) |
| apply migrations | [Database migrations](<https://sandloom.dev/docs/operate/operator-reference/#database-migrations>) |
| plan and verify backups | [Backups](<https://sandloom.dev/docs/operate/operator-reference/#backups-restore>) |
| set retention and handle erasure requests | [Data retention and PII](<https://sandloom.dev/docs/operate/operator-reference/#data-retention-pii>) |
| answer "who did this?" | [Audit log](<https://sandloom.dev/docs/operate/operator-reference/#audit-log>) |
| connect logs, metrics, traces and dashboards | [Observability](<https://sandloom.dev/docs/operate/operator-reference/#observability>) |
| upgrade asyncpg, SQLAlchemy or aiohttp | [Database drivers](<https://sandloom.dev/docs/operate/operator-reference/#upgrading-asyncpg-sqlalchemy>), [aiohttp](<https://sandloom.dev/docs/operate/operator-reference/#upgrading-aiohttp>) |
| upgrade Sandloom | [Upgrade an application](<https://sandloom.dev/docs/operate/upgrade/index.md>) |