Runbook¶
Find the situation, then follow the procedure it links to in the operator reference.
Operator commands (sandloom migrate, outbox, streams, projection, doctor)
boot your application, so set SANDLOOM_COMPOSITION_ROOT=<app>.asgi for them, as the
generated profile does. Run them with --apply only after reading the dry run.
Something is wrong¶
| Situation | Start with | Procedure |
|---|---|---|
| an alert fired | its row in the alert table | Alerts |
| a process won't start or isn't ready | sandloom doctor |
Preflight |
| events stopped arriving, or some were dead-lettered | sandloom outbox list-dead, sandloom streams consumers |
Events |
| a saga is stuck | sandloom sagas list |
A parked saga |
| a task may or may not have been enqueued | TaskEnqueueUncertain: don't retry with another key |
Uncertain enqueues |
| a database or a key leaked | freeze deploys, sandloom security audit-export |
Security incident |
| data was lost | stop writers, restore in order, sandloom backup reconcile-security |
Restoring |
| Redis is slow or unavailable | the CACHE_* limits and each caller's failure policy |
Redis cache |
| login is slow or CPU spikes at login | PASSWORD_HASH_PROFILE |
Password hashing |
| latency spikes with no slow queries | the GC_* settings and metrics |
Garbage collector |
Routine work¶
| Task | Procedure |
|---|---|
| store or rotate secrets and keys | Secrets management |
| apply migrations | Database migrations |
| plan and verify backups | Backups |
| set retention and handle erasure requests | Data retention and PII |
| answer "who did this?" | Audit log |
| connect logs, metrics, traces and dashboards | Observability |
| upgrade asyncpg, SQLAlchemy or aiohttp | Database drivers, aiohttp |
| upgrade Sandloom | Upgrade an application |