CI performance research¶

Related: CI profiles · Testing guide.

Research date: 2026-10-29. This is an observed sample, not a ranking of framework quality or a universal target for test duration.

Public CI from mature Python projects does run for several minutes. In the sample below, five test-executing PR runs per project had median elapsed times between 5:43 and 9:55. Broad compatibility/build workflows could take much longer. This does not establish that Sandloom's current duration is optimal; it establishes that a nine-minute full validation is not, by itself, evidence of an unusual testing practice. The useful comparison is the work performed, the time to actionable feedback, and the resources spent.

Measurement method¶

The source is the first-party GitHub Actions REST API: list recent successful workflow runs, then enumerate their jobs, including all pages. Select the five newest successful first-attempt PR runs that actually execute the project's test jobs. Do not select the fastest five. Samples were collected on October 29; several FastAPI/SQLAlchemy samples are from October 26–28.

• Elapsed time: latest non-skipped job completion minus run_started_at. It includes setup, queues between jobs, tests, downstream jobs and cleanup.
• Sum of job minutes: sum of non-skipped jobs' completed_at - started_at. Parallel jobs overlap in elapsed time. This is an execution-time sum, not GitHub's billed amount; it does not apply OS or runner pricing multipliers.
• Workflow timeout-minutes is a failure limit, not a measured duration.
• Canceled/failed runs and second attempts were excluded. Successful runs alone do not describe a project's failure rate or typical time to find a defect.
• These are whole workflow samples, not pytest-only benchmarks. They are not normalized for test count, native code, runner hardware, platform, database, coverage, benchmarks or the number of compatibility combinations.

The API fields and endpoints are documented in GitHub's workflow-run API and workflow-job API. Each sample below links to the originating run so its scope and job breakdown can be inspected.

Observed durations¶

The SQLAlchemy push workflow is a broad cross-platform/build matrix, not a single serial test process. Its long elapsed time includes scheduling: in the 76:34 sample, the longest individual job was 9:00. A workflow with many queued jobs can take much longer than its slowest test command. These are observations from the sample's job timestamps, not a claim that SQLAlchemy's tests themselves need 76 minutes in one process.

FastAPI had two newer successful Test runs lasting only 17 and 22 seconds; the test matrix was skipped. They are deliberately excluded from the table: run 40883353243, run 40686442620. Comparing those runs with Sandloom's full release validation would be misleading.

Pydantic's three latest sampled successful pushes to main took 21:53, 27:23 and 26:22, with 116 active jobs each. Their workflow enables additional native-wheel builds on main/tags or an explicit full-build label; this is additional work beyond its PR profile: 40933290063, 40929898819, 40929802413, pinned workflow.

Sample ledger¶

Elapsed seconds are unrounded; the duration table formats them as minutes and seconds. Run configuration was read at the first listed run's head SHA. A PR checkout may execute GitHub's synthesized merge commit; the head SHA identifies the inspected source snapshot rather than asserting a universal checkout policy.

What these projects actually do¶

FastAPI: select changes, parallelize, then combine coverage¶

The inspected workflow selects source-related changes, runs a compatibility matrix across Python/OS/dependency combinations and combines coverage after its test jobs. Its test script uses pytest -n auto --dist loadgroup, so related groups stay on one worker. Regression checks for changed tests and benchmarks are separate jobs. The ten-minute job timeout is not its observed runtime. Sources: pinned workflow, pinned test script.

Django: local test clients and different database isolation modes¶

The sampled GitHub workflow runs SQLite on Windows and free-threaded Ubuntu, plus JavaScript and script tests. It excludes docs-only changes and caches package downloads. It is not a measurement of all supported production database configurations. Django's own runner defaults to parallel processes and has shuffle/bisect tools for order-dependent failures. Sources: pinned workflow, contributor testing documentation.

Django's in-process client avoids real HTTP. TestCase uses nested transactions and rollback; class-level setUpTestData avoids recreating seed data per method. TransactionTestCase provides the more expensive mode needed to observe real commit/rollback behavior. This is a test-semantics choice: replacing a real commit test with rollback-only isolation would change what it proves. Source: testing tools.

SQLAlchemy: a deliberately smaller PR matrix¶

The PR workflow tests two Python versions on Linux with three build types, plus static/type checks; its TOX_WORKERS=-n4 is passed into pytest through nox. The inspected GitHub nox sessions use SQLite, with an asyncio driver in the greenlet variant. The push workflow expands the OS/architecture/Python matrix. The difference between the two profiles is explicit in source: PR workflow, nox sessions, push workflow.

Pydantic: parallel jobs, targeted coverage and separate build scope¶

Pydantic's PR workflow includes Python/platform tests, Rust/core tests, typing, memory and documentation jobs. Broad native wheel builds are conditional on main/tags or an explicit full-build label. It caches dependencies in test jobs; its PyPy test command omits coverage because coverage is expensive there. The ordinary test matrix, free-threaded cases and native builds have different purposes; its total workflow time is not a benchmark of one Python unit-test command. Source: pinned workflow.

What is applicable to Sandloom¶

These are recommendations inferred from the source and observations above, not claims that copying another project's exact worker count will improve Sandloom.

1. Keep fast feedback distinct from release confidence. Sandloom already has changed-area PR checks and an opt-in full profile. Compare its PR profile with PR profiles, and full validation with full validation. Track both.
2. Optimize repeated preparation before adding workers. Measure aggregate fixture setup/teardown, database reset, schema compilation, imports and child process startup. Preserve disposable-resource ownership, RLS checks, sequence reset and fallback behavior while removing redundant work.
3. Use the cheapest faithful test mode. Pure policy/serialization tests need no live servers. In-process ASGI tests can cover request behavior. Keep real PostgreSQL, Redis, committed UoW and multi-service tests wherever those are the guarantees under test. Do not substitute SQLite for PostgreSQL/RLS proof.
4. Reuse expensive immutable preparation and retain local state isolation. Transactions/savepoints are useful for tests that can operate within them; tests of independent commits, reconnects, migrations or multiple services need their existing real boundaries. Broadening fixture scope is not a safe automatic optimization.
5. Balance parallel work without duplicating module-scoped setup. Sandloom's loadfile choice keeps a test file together. Other xdist scheduling modes change grouping guarantees; measure the longest-worker tail before changing one. Group scheduling is documented by pytest-xdist.
6. Preserve the serial isolation gate's purpose. Sharding that gate changes which tests share a process and can hide cross-file leaks. Parallel matrix coverage does not replace that evidence. Improve fixture/engine/SQL/import overhead inside the serial pass; any reduced gate would need an explicit change to the assurance contract.
7. Measure elapsed time and execution cost separately. Extra jobs can reduce wait time while increasing total job minutes and queues. Check the critical path and retain before/after runs with the same selection and seed.

A reasonable engineering objective is faster actionable PR feedback and a measured reduction in full validation, while keeping the proof each suite provides. This research does not establish a universal two- or three-minute standard for mature frameworks.