Testing modules¶

Each module owns a tests/ directory. Tests run against a real PostgreSQL (from .env.test) through the real application. They are not mocked. The harness (env loading, schema, sessions, HTTP client, tokens) belongs to the framework, not to each module: it ships as the sandloom.sdk.testing.plugin pytest plugin (ADR-0002).

This page explains how to run the tests and how the harness behaves. Every fixture and helper, with copy-paste examples, is in the testing reference; exact signatures are in sandloom.sdk.testing or sandloom docs <name>.

For the general test workflow, use this guide; for copy-paste fixtures, use Testing reference; for exact public helper signatures, use sandloom.sdk.testing.

First run on a fresh checkout¶

You do not need a .env to run the tests. Start the test services and run the gate:

docker compose -f deploy/compose/test.yml up -d --wait   # PostgreSQL :5433, Redis :6380
uv sync
uv run sandloom test

Framework suite. Its root conftest (conftest.py) fills every setting the shell leaves unset from .env.test before the settings are first read.
.env.test beats a developer's .env, so tests never depend on dev values. ENVIRONMENT stays the dev default: the suite gates on is_prod, never on is_test.
A shell that exports DATABASE_URL itself (CI) is left alone.
Apps. The plugin below loads .env.test instead.

.env.example is the template for running the app (uvicorn, celery, sandloom migrate). Copying it to .env does not change the tests.

Your own database and Redis (shared machines)¶

SANDLOOM_TEST_DATABASE_URL, SANDLOOM_TEST_CACHE_URL and SANDLOOM_TEST_BROKER_URL win over .env.test on every load path. A plain shell DATABASE_URL / CACHE_URL does not: .env.test overrides those for the test sessions.

The database is created on first use.
Pick two Redis indexes no one else uses (tools/checks/test_resources.py list).
Set SANDLOOM_TEST_OWNER so the ownership guard knows the resources are yours.

export SANDLOOM_TEST_OWNER=me SANDLOOM_TEST_DATABASE_URL=postgresql+asyncpg://test_user:test_password@127.0.0.1:5433/sandloom_test_me
export SANDLOOM_TEST_CACHE_URL=redis://127.0.0.1:6380/120 SANDLOOM_TEST_BROKER_URL=redis://127.0.0.1:6380/121

The harness refuses to reset any database other than the one SANDLOOM_TEST_DATABASE_URL names (or its pytest-xdist _gw<N> copies).

CACHE_URL and SANDLOOM_TEST_CACHE_URL: do they have to differ?

The old requirement. A comment in .github/workflows/ci.yml says the settings-level CACHE_URL must name a different Redis database than the test cache. The reason given: the billing conftest pointed its idempotency cache at the test CACHE_URL, and the redis driver shares its process-wide client only when the two URLs are equal, so billing failed closed with 503.
It no longer reproduces. Measured on 2026-10-25, with the snapshot CACHE_URL equal to SANDLOOM_TEST_CACHE_URL (what the fresh-checkout seeding produces):
the billing suite (307 passed);
the whole fast suite (3,783 passed).
CI still sets different values. Keep them different if you run the slow suites with a hand-written .env and see 503s from billing.

Activate the harness (once per application)¶

# pyproject.toml — `sandloom new` generates this
[tool.pytest.ini_options]
pythonpath = ["."]
asyncio_mode = "auto"
addopts = ["-p", "sandloom.sdk.testing.plugin"]
# optional, instead of SANDLOOM_COMPOSITION_ROOT in .env.test:
# sandloom_composition_root = "myapp.asgi"

That is all: modules need no conftest.py and the app needs no root conftest. The plugin is deliberately not a pytest11 entry point: installing the framework never changes another test suite (nor the framework's own).

A module in the framework tree (src/sandloom/<name>, whose suite does not load the plugin) gets a one-line conftest instead:

from sandloom.sdk.testing.fixtures import *  # noqa: F403

What the plugin does:

Loads .env.test first. The nearest .env.test walking up from the pytest rootdir (then from the test paths) is loaded with override=True in pytest_load_initial_conftests, before any conftest or test module can read the settings. No file found is a clear usage error.
Boots the app through the composition root (SANDLOOM_COMPOSITION_ROOT, else the sandloom_composition_root ini option, else sandloom.main), which also registers every loaded module's models.
Guards the database. db_engine refuses any DATABASE_URL whose database name lacks test, even with ENVIRONMENT=test. The schema (every model plus the core outbox / audit / consumer-inbox tables) is built once per process and emptied before each test: only the tables holding a row are emptied (DELETE children-first; a table with more than 1,000 rows is TRUNCATEd) and every used sequence is rewound. That leaves exactly what a TRUNCATE … RESTART IDENTITY CASCADE of every table leaves, at a fraction of the cost. The probe runs with row_security = off, so a row-level security policy can never hide a row from it; any error falls back to truncating everything.
Records events and resolves contracts. published(E) lists the events the code under test published; Contract(REF) resolves through the test application, and fake_contract overrides one contract for a block.
Releases pools per test, so loop-bound asyncpg connections never leak into the next test's event loop.

What a test looks like¶

A test takes the client fixture, mints a user with as_user, and asserts on the response and on published(...):

async def test_create_and_read_an_item(client: Any) -> None:
    _, headers = as_user(tenant_id=uuid4())
    with fake_contract(STATS_QUOTA_V1, Quota(allow=True)):
        created = await client.post(
            "/api/v1/tasks/items", json={"title": "first"}, headers=headers
        )
    assert created.status_code == 201, created.text
    assert len(published(ItemCreated)) == 1
    item = created.json()
    fetched = await client.get(f"/api/v1/tasks/items/{item['id']}", headers=headers)
    assert fetched.status_code == 200
    assert fetched.json()["title"] == "first"


async def test_the_quota_refuses_a_new_item(client: Any) -> None:
    _, headers = as_user(tenant_id=uuid4())
    with fake_contract(STATS_QUOTA_V1, Quota(allow=False)):
        response = await client.post(
            "/api/v1/tasks/items", json={"title": "x"}, headers=headers
        )
    assert response.status_code == 402
    assert published(ItemCreated) == []


async def test_items_need_an_authenticated_user(client: Any) -> None:
    response = await client.post("/api/v1/tasks/items", json={"title": "x"})
    assert response.status_code == 401

A consumer is driven with deliver. The envelope id is stable across times, so times=2 is a redelivery; session=db runs the deliveries in the test transaction:

async def test_a_redelivered_event_is_counted_once(client: Any, db: Any) -> None:
    tenant = uuid4()
    _, headers = as_user(tenant_id=tenant)
    event = ItemCreated(item_id=uuid4(), tenant_id=tenant)
    await deliver(event, times=2, session=db)  # the same envelope id twice
    response = await client.get("/api/v1/stats/count", headers=headers)
    assert response.status_code == 200, response.text
    assert response.json() == {"items": 1}


async def test_counts_are_per_tenant(client: Any, db: Any) -> None:
    await deliver(ItemCreated(item_id=uuid4(), tenant_id=uuid4()), session=db)
    _, headers = as_user(tenant_id=uuid4())  # another tenant
    response = await client.get("/api/v1/stats/count", headers=headers)
    assert response.json() == {"items": 0}

Fixtures and helpers in brief¶

The beginner kit is client, as_user, published, deliver, fake_contract and fake_access_fact. The rest of the harness (db, real_uow_client, db_uow, ambient_uow, sandloom_cache, scratch_database, the migration, revocation and RPC helpers) is listed with examples in the testing reference. Two behaviours matter for every test:

client never commits. Every request runs on the test's one db session, bound as the Unit of Work. Events and audit rows land in the test transaction. A request that fails with a 4xx does not roll its writes back. Assert rollback behaviour with real_uow_client, which runs the real request Unit of Work (commit on success, rollback on any error).
Direct service calls need a Unit of Work. publish_event and record_audit raise outside one. Use db_uow (the test session as the Unit of Work) or ambient_uow (an in-memory recording one).

Unhandled errors are re-raised¶

Production answers an unhandled endpoint exception with an opaque 500 {"code": "internal_error"}. In a test that hides the cause, so client and real_uow_client raise UnhandledServerError from the original exception instead: route, exception type and message, and the catalogue fix:

error[SANDLOOM-R001]: POST /api/v1/notes answered 500 internal_error because the endpoint raised:
  MissingTenantError: an ORM SELECT on TenantScoped model NotesNote ran with no tenant bound. ...
  fix: add a TenantScope() parameter to the endpoint ... (sandloom explain SANDLOOM-R001)

A test that asserts the 500 itself marks it @pytest.mark.sandloom_allow_server_errors.

Keep only module-specific fixtures (seed data, stubs) in a module conftest.py, built on the harness fixtures. See examples/helpdesk/.../tickets/tests/conftest.py, and the bundled sandloom_tenancy/tests/conftest.py / sandloom_billing/tests/conftest.py (a domain fixture overriding async_client on top of client, for example to install a fake payment provider, is the pattern).

Deprecations fail the suite¶

Generated apps (and the example apps) set filterwarnings = ["error::DeprecationWarning"]: a deprecated call fails the tests now instead of breaking on the next release. Add targeted ignore::DeprecationWarning:<module> entries only for third-party noise you cannot fix. The framework's own suite errors on deprecations attributed to sandloom.* code (error::DeprecationWarning:sandloom).

Run the suites¶

sandloom test is the gate. It runs the tests in parallel, then the static gates:

uv run sandloom test <module>   # iterate: the module's tests, the fast suite, the gates
uv run sandloom test --full     # once, at the end: the full parallel suite (slow tests included) + gates
uv run sandloom test --no-gates # tests only;  sandloom test --plan prints the steps

Never re-run a serial pytest -q after each fix. Plain pytest still works for a single test:

uv run pytest src/sandloom/<module>/tests/test_x.py::test_y --no-cov

Fast suite. -m "not slow" skips the multi-process and subprocess-heavy suites: the split conformance matrix, the e2e split deployment, the generated-project and migration-CLI tests, module data extraction, migration chain walks, the real pg_dump round trip. The full suite (CI, the pre-push hook) still runs everything; mark a new test @pytest.mark.slow when it spawns servers / CLIs or walks migration chains.

Parallel. pytest -n auto --dist loadfile gives each xdist worker its own database (<db>_gw<N>) and Redis indexes; -n auto is capped to the connections the PostgreSQL server has free. On a Redis shared with other runs set SANDLOOM_TEST_REDIS_DB_STRIDE=auto (sandloom test does): the controller then claims free indexes for the workers before they start (this owner's indexes from earlier runs first, then unmarked empty ones, each with an atomic check-and-set) instead of the fixed index + 16 × (N + 1) layout, which collides with other runs' markers. Too few free indexes abort the session with exit code 4 (the pre-push hook then falls back to a serial run).

In the framework tree the benchmark security suite (benchmarks/tests) is a separate CI job (benchmarks-security: it needs a superuser connection for its RLS roles and runs about 6 minutes): tools/ci/ci_local.sh <ref> --jobs benchmarks-security.

Isolation guards¶

The harness protects concurrent runs on one machine from each other:

Resource ownership. Every test database carries an owner marker (its comment, sandloom-test-owner:<owner>) and every test Redis index the key sandloom:test:owner. The owner is SANDLOOM_TEST_OWNER, else the checkout root (one per git worktree). An unmarked resource is claimed by the first run (a database the harness creates is stamped on creation); one marked by another owner aborts the session before any test runs (exit code 4, nothing truncated or flushed), under xdist for every worker's _gw<N> copy too. SANDLOOM_TEST_TAKE_OWNERSHIP=1 takes a resource over deliberately. Empty a test Redis database with flush_owned_redis(client) (keeps the marker).
Self-connection. A URL with localhost and a port from 32768 up is refused (exit code 4). A service published on 127.0.0.1 leaves ::1 with no listener, and a connection to that dead port can be given the same port as its source, so the socket connects to itself and reads its own request back. It takes thousands of connections, so it looks like a random failure. Use 127.0.0.1 or a lower port.
Stale markers. Every claim also stamps when the resource was last used (Redis key sandloom:test:last_used, database setting sandloom_test.last_used; the owner marker keeps its format for older checkouts). Markers of finished runs otherwise pile up until no Redis index is free:

uv run python tools/checks/test_resources.py list            # owner, age, connections
uv run python tools/checks/test_resources.py prune --days 7  # dry run: what would go
uv run python tools/checks/test_resources.py prune --days 7 --apply

A resource is pruned only when nothing is connected to it and its last use is known and older than --days (at least 1; a run re-stamps its resources when it starts). Legacy markers without a stamp fall back to the youngest key idle time (Redis, read under CLIENT NO-TOUCH) or the database directory's modification time (superuser role); unknown is never pruned. A Redis index is emptied by an atomic check-and-FLUSHDB (skipped when its marker or stamp changed since the scan); a database is dropped with a plain DROP DATABASE, which PostgreSQL refuses while anything is connected. Unmarked databases (scratch copies leaked by killed runs) are only touched with --include-unmarked; this checkout's own owner is always protected. - Connection budget. A test process gets small pools by default (DB_POOL_SIZE=2, DB_MAX_OVERFLOW=3, CONTRACT_DB_POOL_SIZE=1, CONTRACT_DB_MAX_OVERFLOW=1, DB_POOL_OVERFLOW_LINGER_SECONDS=0; an explicit value wins): at most 13 connections with the harness engine. The session starts only if the server's free connections (max_connections − reserved − in use) cover workers × 13, else it fails fast with the numbers instead of too many clients mid-run; -n auto is capped to what fits while keeping 40 free for the multi-process suites. The split conformance matrix boots in waves of at most 40 connections and checks each wave's room. SANDLOOM_TEST_SKIP_CONNECTION_CHECK=1 skips the check. - Process-global state. create_app binds its app as the process's active core / infra registry / event bus and fills the settings' module lists. The harness builds the composition-root app before the first test module and makes its binding the baseline: every module starts from it and gets it back afterwards, every test gets its starting bindings back, so an app a test builds never leaks into the next test, and a test that resolves the active runtime without asking for sandloom_app passes alone too. What a test registers on the bound core (an event schema, a service, a startup task, a declared operation, ...) is rewound after it (CoreFacade.restore_registrations). - Pre-push hook. backend-full-tests refuses to run the full suite unless SANDLOOM_TEST_DATABASE_URL names this checkout's own database (tools/checks/prepush_full_tests.sh; SKIP=backend-full-tests skips it once).

Test isolation¶

Every test must pass alone, in any order, serially or in parallel. A parallel run with a fixed distribution hides a test that passes only because an earlier one left a registry, a binding, a cache key or a row behind, so the gates run the tests in a seeded random order (sandloom.core.testing.shuffle: the files, then the tests inside each file; a file's module fixtures are still set up once):

sandloom test shuffles every test step with a fresh seed. It prints the seed before the steps and again after a failure: sandloom test --seed <seed> replays the order, --no-shuffle runs in file order.
pytest directly: --sandloom-shuffle=<seed> (or random). The framework's pyproject.toml loads the plugin; an application adds -p sandloom.core.testing.shuffle (inert without the option). The seed is in the report header and the summary.
CI runs the parallel suites (backend-tests, examples) with --sandloom-shuffle=random, and the job backend-tests-isolation runs the in-process framework tests serially in one process in a fresh order: the check a fixed xdist distribution cannot make. Locally: tools/ci/ci_local.sh <ref> --jobs backend-tests-isolation.

The serial job excludes subprocess_probe: package build/install checks and CLI/plugin probes whose tested runtime lives entirely in a child process. They cannot leak that runtime into the parent's next test and remain in both unfiltered full suites (Python 3.11 and 3.12). Tests of the parent's globals, registrations, caches and database state stay in the shuffled serial pass. --durations=25 reports the slowest setup/call/teardown phases in CI.

Replay a failure of the serial job in one process:

uv run pytest -p no:xdist --sandloom-shuffle=<seed>      # the whole run, same order
uv run pytest -p no:xdist src/sandloom/x/tests/test_y.py  # the failing file alone

What the harness resets for you after every test (sandloom.core.testing.isolation): the process's runtime bindings and what a test registered on the bound core (above), SQLAlchemy's class-level event flags (event.listen(Engine, ...) sets Engine._has_events and event.remove never clears it), and the RPC client's described-contract caches (PROCESS_CACHES). The test database is emptied before every test that uses db_engine, the cache by sandloom_cache. What a test must not do: rely on another test's setup (register what you need in your own fixture), keep state in module globals between tests, or read a module-level definition's derived state before the harness registered it. A new process-wide cache or class flag that tests warm belongs in PROCESS_CACHES / CLASS_FLAGS, not in a per-test patch.

Conventions¶

Tests import the framework only through sandloom.sdk.testing (plus the public sandloom.sdk modules, contract stubs and DTOs). The test-imports rule of sandloom check (SANDLOOM-C013) is a whitelist: a module test may import those and its own package, not kernel internals and not another business module.
Mint users via JWT only. Use as_user (or the make_headers fixture). Do not create identity rows from another module's tests. Tokens carry token_type="access", iat, tenant_id and the signed tenant_ids allowlist, email / email_verified; extra keywords become claims, tenant=False mints a tenant-less token.
Unit tests without a database. Run services against a mocked session under recording_uow() (its .audits / .events record what was written); bind_test_uow(session) binds a real session as the Unit of Work by hand; run_on_commit(session) treats the session as committed: it runs the queued before_commit checks first (a failing one propagates and no post-commit callback runs), then the post-commit callbacks; InMemoryCache is a full cache provider for idempotency / cache tests.
Contracts read the test's data. On the client path the test session is bound for contract calls automatically. Elsewhere wrap the call in bind_contract_session(db) (from sandloom.sdk.testing).
In-memory bus delivery is asynchronous. publish only enqueues on the dev/test memory bus. Drive a consumer directly with deliver instead of waiting for the bus.
Every versioned channel needs a schema in every profile (SANDLOOM-R003). A typed Event class carries its own; a probe channel in a test is registered with register_event_schemas.
Module wiring is the module's own test. The generated tests/test_module_wiring.py pins the approved route snapshot, the boots-alone outcome and the registration surface. A module's tests are collected without editing testpaths (src/sandloom in the framework, the app package in an app).
No driver resets. The Redis cache driver's clients follow the running event loop, so tests never reset driver singletons.
Don't gate on is_test. In the framework's own suite the settings snapshot .env (ENVIRONMENT=dev). Gate behaviour on is_prod, never on is_test.
Snapshot IDs before a savepoint rollback. After a service method that may roll back a begin_nested() savepoint, snapshot entity.id before the call: rollback expires ORM instances on async sessions.
Sign webhooks. Billing webhook tests sign payloads with sandloom_billing.psp.sign_stripe_payload.

Cross-module test imports¶

Module tests are held to a whitelist (sandloom check rule test-imports): the public sandloom.sdk modules, the contract stubs and DTOs, the testing kit sandloom.sdk.testing and the module's own package. Another business module's code (its services, models, schemas) and kernel internals are out.

Build another module's state the way a client would: through its HTTP API on the shared client (rows land in the test db), its contracts (fake_contract, or the real provider through the test app) or its events (deliver); carry its facts in the token (as_user(tenant_id=..., roles=[...])) or answer them with fake_access_fact. See sandloom_billing/tests/tenancy_api.py (organizations and memberships seeded through tenancy's routes) and examples/helpdesk/.../tickets/tests/conftest.py. Take the app and its core from the sandloom_app / sandloom_core fixtures, never from the composition root module. A test that is really about kernel behaviour belongs in the kernel's own suite; a missing seam belongs in sandloom.sdk.testing.

The quality gates¶

sandloom test runs the static checks after the tests. Checks explains what each one verifies, the error codes they print, and how to run sandloom check on its own.

Inspect before changing a module¶

Use module context to gather scoped rules, SDK and wire interfaces, and the declared architecture graph to inspect contract/event connections. Both commands can be invoked by an agent and read their index without booting the app.