Benchmarking Notes¶

This document captures the benchmark harnesses used in the repo and a few reference results from named development and comparison environments.

Reference Environments¶

Developer laptop reference: Apple M5 MacBook Air (16 GB), PostgreSQL 17 in Docker (OrbStack). Several awa-only regression examples below come from this environment.
Dedicated enqueue reference: PostgreSQL 17 on a separate Linux host. These runs are useful for producer-path shape comparisons and are not published throughput claims.
Cross-system comparison reference: postgresql-job-queue-benchmarking. That repo owns fair-comparison hardware, adapter configuration, raw output, and result summaries. Its default Postgres image is 17.2; --pg-image postgres:18-alpine selects PG18.
Example commands assume a developer database URL: postgres://postgres:test@localhost:15432/awa_test
Benchmarks live in: awa/tests/benchmark_test.rs awa/tests/scheduling_benchmark_test.rs awa/tests/failure_benchmark_test.rs
Python worker benchmarks live in: awa-python/scripts/benchmark_runtime.py
Shared output schema: awa/tests/bench_output.rs (Rust) awa-python/scripts/bench_output.py (Python)

These are engineering benchmarks, not published vendor-style numbers. The main goal is to compare shapes, validate architecture changes, and catch regressions. Treat each result as tied to the environment named beside it; do not compare numbers from different sections as though they came from one machine.

North-Star Metrics¶

Awa's performance target is sustainable end-to-end queue work, not the largest enqueue-only number. A benchmark table should lead with:

completed jobs/sec — durable terminal transitions, not just handler returns
p99 end-to-end latency — producer enqueue to durable completion
queue depth / backlog growth — whether offered load is being absorbed or stored as future latency
WAL bytes per completed job — the main Postgres durability budget for the hot path
transaction commits per completed job — connection and commit pressure
dead tuples and prune lag — whether churn stays bounded under overlap readers, retries, and terminal bursts

Producer throughput is still measured, especially for INSERT vs COPY and enqueue_shards sweeps, but it is a secondary producer-path diagnostic. A configuration that can enqueue 50k/s and complete 5k/s is a backlog and latency problem unless the workload is intentionally bursty and has enough headroom to drain before the SLA window closes.

Positioning Proof Checklist¶

Queue storage changes Awa's comparison set.

The interesting public question is no longer "can Postgres run jobs?" It is "can a Postgres job queue keep dispatch latency low while also keeping hot-path dead tuples bounded?"

That means the benchmark burden for positioning is different from the benchmark burden for internal regressions. Before making the strongest public claim, the comparison needs to be rerun cleanly on the same hardware against the right reference points:

Awa queue storage
PgQue
River
optionally Oban Pro as a paid partitioned reference

The benchmark set should include:

idle pickup latency
sustained runtime throughput and durable completion throughput
overlap readers / MVCC horizon pressure
mixed workload soak
terminal-failure burst
WAL bytes/job, commit pressure, queue depth, and dead-tuple pressure for every headline table

Operational differences should be called out, not hidden:

Awa does not require pg_cron; the worker runtime owns dispatch, rescue, rotation, and prune.
PgQue is designed around a periodic ticker and recommends pg_cron.
River and Oban are job frameworks rather than shared-log event queues, so they remain the more comparable references for worker semantics.

Commands and raw output for any public-facing table should live in-repo under docs/adr/bench/ or another stable location so the resulting claim is auditable.

Methodology Notes¶

The most important lesson from this round of work is that benchmark isolation matters.

The maintenance service is global to the worker instance. If a benchmark leaves behind millions of deferred rows in awa.scheduled_jobs, later "hot path" benchmarks may accidentally measure background promotion work as well. The sustained hot/deferred benchmarks reset runtime state first to avoid that pollution.

Two benchmark shapes are used:

Burst/frontier benchmarks: seed a backlog, then drain it
Steady/sustained benchmarks: warm up, then measure a fixed time window

Steady numbers are the better indicator of sustained runtime behavior.

MVCC Horizon Benchmark¶

awa/tests/scheduling_benchmark_test.rs also includes an ignored test_mvcc_horizon_overlap_benchmark scenario for the failure mode described in PlanetScale's "Keeping a Postgres queue healthy" post: overlapping long-lived transactions pin the MVCC horizon while queue churn continues.

The benchmark:

runs a steady producer feeding awa.jobs_hot
enables aggressive completed-job cleanup to generate update/delete churn
starts overlapping REPEATABLE READ reader transactions on separate connections to hold old snapshots open
samples per-second throughput, queue depth, and pg_stat_user_tables (n_dead_tup, vacuum counters) for awa.jobs_hot

Reader modes:

idle_snapshot: pins a snapshot and waits inside the transaction; useful for the pure MVCC-horizon case
active_scan: runs repeated queue scans inside the open transaction so the reader behaves more like overlapping analytics work on the primary

Example:

DATABASE_URL=postgres://postgres:test@localhost:15432/awa_test \
  cargo test --release --package awa --test scheduling_benchmark_test \
  test_mvcc_horizon_overlap_benchmark -- --ignored --exact --nocapture

This scenario is also wired into .github/workflows/nightly-chaos.yml and runs on PostgreSQL 18 in the Rust nightly benchmark lane.

The nightly lane uses a shorter CI profile so the benchmark stays cheap on shared runners while still exercising overlap readers and cleanup pressure:

AWA_MVCC_JOB_RATE=400
AWA_MVCC_BASELINE_SECS=5
AWA_MVCC_OVERLAP_SECS=12
AWA_MVCC_COOLDOWN_SECS=5
AWA_MVCC_OVERLAP_READERS=2
AWA_MVCC_OVERLAP_HOLD_SECS=12
AWA_MVCC_OVERLAP_STAGGER_SECS=4
AWA_MVCC_READER_MODE=active_scan
AWA_MVCC_ANALYTICS_TICK_MS=500

Nightly regression checks use per-run ratios (overlap_handler_per_s and cooldown_handler_per_s relative to the same run's baseline window) plus guardrails on dead_tup_delta and max_available.

That nightly profile is intentionally short. It checks that Awa still behaves reasonably under overlapping analytical readers, but it is not the same thing as the 15-minute mixed-workload soak discussed in the PlanetScale post. For a closer reproduction, increase duration and reader hold times and keep the readers in active_scan mode so queries overlap continuously rather than simulating only idle in transaction.

A second ignored benchmark target now exists for that longer profile: test_mvcc_horizon_planetscale_soak. Its default shape is closer to the blog's mixed-workload setup:

800 jobs/sec producer rate
3 overlapping analytics readers
120s reader hold time
20s stagger between readers
15m overlap window with active_scan

That soak benchmark is wired into CI as a weekly/manual run, while the shorter test_mvcc_horizon_overlap_benchmark remains the daily nightly smoke.

MVCC bench knobs¶

AWA_MVCC_JOB_RATE — steady producer rate in jobs/sec
AWA_MVCC_BASELINE_SECS / AWA_MVCC_OVERLAP_SECS / AWA_MVCC_COOLDOWN_SECS
AWA_MVCC_OVERLAP_READERS / AWA_MVCC_OVERLAP_HOLD_SECS / AWA_MVCC_OVERLAP_STAGGER_SECS
AWA_MVCC_READER_MODE — idle_snapshot or active_scan
AWA_MVCC_ANALYTICS_TICK_MS — cadence for repeated analytics scans in active_scan mode
AWA_MVCC_CLEANUP_INTERVAL_MS / AWA_MVCC_CLEANUP_BATCH_SIZE

Cross-system comparison¶

The awa benches in this file are the awa-only regression detection track — fast, precise, hard thresholds, awa code only. They are not designed for fair comparison with other systems.

For cross-system comparison, the long-horizon harness lives in a separate repository so each system-under-test can be benchmarked through its public API on equal footing:

hardbyte/postgresql-job-queue-benchmarking

The companion repo currently benchmarks awa against pgque, procrastinate, pg-boss, river, oban, and pgmq using a shared subprocess contract: each adapter is a self-contained binary that emits one JSON sample per line. Headline numbers, per-system architectural notes, and reproducible run instructions are in that repo's SYSTEM_COMPARISONS.md and results/2026-04-28/SUMMARY.md.

The two tracks are deliberately separate. If they ever diverge on workload shape or thresholds, the awa-only benches in this file are canonical for awa's own numbers and the cross-system runner defers.

Python Runtime Benchmarks¶

The Python benchmark script exercises the real awa-python worker path while reusing the same database-facing benchmark shapes as the Rust runtime:

Baseline scenarios (--scenario baseline):

copy: Python client enqueue_many_copy direct queue-storage COPY throughput
hot: sustained worker throughput over pre-seeded awa.jobs_hot
scheduled: sustained deferred promotion over pre-seeded awa.scheduled_jobs

Failure scenarios (--scenario failures):

terminal_1pct / 10pct / 50pct: terminal failures
retryable_1pct / 10pct / 50pct: retry-once failures
callback_timeout_10pct: callback registration with short timeout
mixed_50pct: rotating through terminal, retryable, and success modes

The worker-focused scenarios seed with SQL directly so the reported number is about Python handler dispatch and runtime behavior, not enqueue serialization.

Reference Results¶

Immediate Enqueue Throughput¶

The enqueue path that is being measured here has a few important architectural properties:

homogeneous inserts now route directly to awa.jobs_hot or awa.scheduled_jobs instead of going through the compatibility view
admin metadata maintenance moved from row-level triggers to statement-level trigger batches
COPY staging now reuses a session-local temp table and stages typed values instead of reparsing text on the final INSERT ... SELECT

Example reference result from the developer laptop reference:

developer laptop reference (Apple M5, release build, v0.5.0-alpha.0):
insert_only_single: about 30k inserts/s
copy_single: about 43k inserts/s
insert_contention_distinct (4 producers x 3k): about 46k inserts/s
copy_contention_distinct (4 producers x 3k, chunk 1000): about 100k inserts/s
insert_contention_same_queue (4 producers x 3k): about 95k inserts/s
copy_contention_same_queue (4 producers x 3k, chunk 1000): about 67k inserts/s

These are engineering comparisons, not product guarantees. Their main value is showing where the architecture bottlenecks move as the implementation changes.

Sustained Hot Path¶

Measured with test_runtime_sustained_hot_path after resetting runtime state:

warmup: 2s
measurement window: 10s
queue size seeded: 200,000 immediately-available jobs

Example reference result from the developer laptop reference (release mode, v0.5.0-alpha.0):

handler returns: about 5.6k jobs/s
DB completed transitions: about 5.6k jobs/s

This benchmark enables the in-memory OpenTelemetry exporter and the production alerting metrics path (queue depth, lag, wait-duration histogram). Back-to-back A/B testing in the same reference environment shows v0.5.0 is ~44% faster than v0.4.1 (5.9k vs 4.1k/s) — the promotion query optimizations more than offset the metrics instrumentation cost.

Python Runtime Baseline¶

Measured with awa-python/scripts/benchmark_runtime.py on the developer laptop reference database:

enqueue_many_copy: about 16.2k jobs/s (50,000 jobs in 3.09s)
sustained hot path:
handler returns: about 3.2k jobs/s
DB completed transitions: about 3.1k jobs/s

The copy scenario measures producer serialization and direct queue-storage COPY. The worker-focused scenarios seed with SQL so the runtime number is not dominated by Python-side enqueue serialization.

Deep Backlog Drain¶

test_queue_storage_deep_backlog_drain_benchmark is an ignored regression benchmark for the spike shape where a large ready backlog drains more slowly than the same fleet processes steady offered load. It seeds with direct queue-storage COPY, optionally analyzes ready_entries, then measures claim/complete throughput and claim latency while draining a fixed backlog.

DATABASE_URL_QUEUE_STORAGE=postgres://postgres:test@localhost:15432/awa_test_queue_storage \
AWA_QS_DEEP_BACKLOG_JOBS=1000000 \
AWA_QS_DEEP_BACKLOG_SECONDS=120 \
AWA_QS_DEEP_BACKLOG_CONSUMERS=16 \
AWA_QS_DEEP_BACKLOG_SHARDS=16 \
cargo test --package awa --test queue_storage_benchmark_test \
  test_queue_storage_deep_backlog_drain_benchmark -- --exact --ignored --nocapture

Useful knobs:

AWA_QS_DEEP_BACKLOG_JOBS — ready rows to seed (default 100000)
AWA_QS_DEEP_BACKLOG_COPY_BATCH — direct COPY chunk size (default 1000)
AWA_QS_DEEP_BACKLOG_CONSUMERS — concurrent claim/complete loops (default 8)
AWA_QS_DEEP_BACKLOG_CLAIM_BATCH_SIZE — claim batch size (default 512)
AWA_QS_DEEP_BACKLOG_SHARDS — awa.queue_meta.enqueue_shards upserted before seeding (default 16)
AWA_QS_DEEP_BACKLOG_ANALYZE — run ANALYZE after seeding (default 1)

Large Deferred Frontier¶

Measured with test_scheduled_steady_10m_due_1k_per_sec:

total deferred backlog: 10,000,000 rows
due rate target: 1,000 jobs/s
measurement window: 10s

Isolated 4-thread Tokio runtime result (release mode, v0.5.0-alpha.0):

9,000 of 10,000 due jobs completed within the window
per-second completions: 0, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000 — perfectly steady after the first-tick startup delay
pickup lateness:
p50: 229 ms
p95: 332 ms
p99: 343 ms
promotion: 354 batches, mean 4.3 ms, max 59 ms
claim latency: mean 5.0 ms

This demonstrates that the hot/deferred split with literal-state promotion queries handles a 10M-row deferred frontier with steady, predictable throughput.

Key optimization (v0.5.0): Promotion queries use literal state values (e.g., WHERE state = 'scheduled') instead of parameterized (WHERE state = $1). This allows the Postgres planner to match the partial index idx_awa_scheduled_jobs_run_at_scheduled at plan time. With a parameterized query, the planner falls back to a full bitmap scan on multi-million-row tables, degrading promotion from ~4ms to ~400ms per batch (100x slower).

Moderate Deferred Frontier — Higher Due Rate¶

Measured with test_scheduled_steady_2m_due_4k_per_sec:

total deferred backlog: 2,000,000 rows
due rate target: 4,000 jobs/s
measurement window: 10s

Result (v0.5.0-alpha.0):

all 40,000 due jobs were picked and completed
pickup lateness: p50: 0 ms, p95: 0 ms, p99: 57 ms
promotion: 214 batches, mean 10.0 ms, max 176 ms
claim latency: mean 12.4 ms

This validates the architecture at a realistic production scale: 2M deferred rows with 4k/s throughput and reliable promotion.

High-Rate Deferred Frontier: 10M at 6k/s¶

Measured with test_scheduled_steady_10m_due_6k_per_sec:

total deferred backlog: 10,000,000 rows
due rate target: 6,000 jobs/s
measurement window: 10s

Result:

58,686 of 60,000 target jobs completed within the window (98%)
per-second completions: 2942, 6834, 6907, 4758, 5915, 5662, 7246, 6446, 5305, 6671
pickup lateness: p50: 0 ms, p95: 310 ms, p99: 476 ms
promotion: 242 batches, mean 6.4 ms, max 99 ms

This rate was previously a documented scaling limit (only 20-57k of 60k promoted, promotion at 1.7-3.6s per batch). The literal-state promotion fix (v0.5.0) eliminated the bottleneck entirely.

Promotion and completion throughput knobs:

promote_interval (default 250 ms): how often promotion runs
AWA_COMPLETION_FLUSH_MS (default 1 ms): completion batcher flush interval
AWA_COMPLETION_BATCH_SIZE (default 512): max rows per completion-batcher flush. Queue-storage short-job completion uses a fused receipt-claim lock, compact terminal insert, and compact claim-closure insert, so the default keeps finalization latency low while still amortising durable completion work under load.
AWA_COMPLETION_SHARDS: number of parallel completion flushers. The runtime default is storage-dependent: 8 for canonical storage. Queue storage starts at 1 for ordinary runtimes and uses 4 once the configured runtime worker capacity is at least 512. Queue storage starts conservative because the terminal / receipt path already batches heavily and the effective fleet-wide flusher count is processes × AWA_COMPLETION_SHARDS. Override only after measuring end-to-end throughput, p99 latency, WAL bytes/job, and dead tuples for the target worker-process topology.

Internal promotion constants are fixed in the runtime: PROMOTE_BATCH_SIZE = 4,096 rows per promotion batch and PROMOTE_MAX_BATCHES_PER_TICK = 32 batches per maintenance tick. They are not environment variables or builder options.

Concurrent Multi-Queue Lifecycle¶

Measured with concurrent_lifecycle_test.rs. Four queues (email:32, payments:16, analytics:64, webhooks:16 workers), 128 total workers, full insert → claim → execute → complete lifecycle.

Queue-count sweep with 128 total workers, pool=50, 20k jobs (release mode):

Config	Workers/queue	Throughput	Per-worker
1 queue × 128	128	`~1.9k/s`	`~14/s`
2 queues × 64	64	`~2.0k/s`	`~15/s`
4 queues × 32	32	`~350/s`	`~2.7/s`

Key finding: 1-queue and 2-queue throughput is essentially identical, but 4 queues drops 5-6x. The cliff between 2 and 4 queues indicates that the per-queue dispatcher overhead (separate claim query, PgListener, semaphore) compounds non-linearly when many small queues share a pool.

For comparison, the hot-path benchmark (200k pre-seeded into jobs_hot via SQL) reaches ~10k/s because it bypasses insert triggers and has a fully warmed dispatch pipeline. The lifecycle benchmarks exercise the complete path including job-state triggers and notification.

Tuning guidelines:

Size the connection pool to at least num_queues * 4 + 20. With 4 queues, that's 36+ connections.
Prefer fewer queues with larger worker pools over many small queues. 2 queues × 64 workers performs the same as 1 × 128.
Use multi-queue for isolation, priority, or rate limiting — not for throughput.

A unified cross-queue claim query (one SQL round-trip claiming across all queues via LATERAL JOIN) could eliminate the per-queue dispatch overhead. Prototyping shows 16ms for 48 jobs across 2 queues — comparable to single-queue performance. This is tracked as a future optimization.

Progress Feature Overhead¶

ADR-014 introduced the user-facing structured progress API and a two-tier heartbeat flush. In the current queue-storage engine, mutable progress lives in {schema}.attempt_state only after an attempt first needs per-attempt mutable state; short jobs that never report progress do not allocate an attempt_state row. Older benchmark notes refer to progress columns on jobs_hot / scheduled_jobs, which was the canonical-storage implementation.

Performance impact was validated:

Zero overhead when no progress is set. The heartbeat service partitions jobs by pending progress; jobs without mutations use the heartbeat-only path, and snapshot_pending_progress returns empty when no generation has been bumped.
Completion remains narrow. Successful completion clears the progress snapshot while closing the attempt. On queue storage this is handled through the attempt-state / terminal-snapshot path rather than rewriting the ready row.
Sustained hot-path throughput was unchanged when the progress feature was added. Current hot-path throughput (~5.6k/s) reflects the additional v0.5.0 OTel metrics instrumentation, not progress overhead.

Failure-Mode Benchmarks¶

The failure-mode benchmark suite measures throughput, drain time, and recovery behaviour when a configurable percentage of jobs fail, retry, hang, or trigger rescue paths. This answers a question the happy-path benchmarks cannot: how does failure impact healthy-job throughput?

Benchmark matrix¶

Scenario	Description
`terminal_1pct` / `10pct` / `50pct`	N% of jobs fail terminally
`retryable_1pct` / `10pct` / `50pct`	N% of jobs fail once then succeed on retry
`callback_timeout_10pct`	10% register a callback that times out, then succeed on retry
`deadline_hang_10pct`	10% hang until deadline rescue fires, then succeed on retry
`snooze_once_10pct`	10% snooze once, then succeed
`mixed_all_modes`	50% success, 10% each of terminal/retryable/callback/deadline/snooze
`stale_heartbeat_rescue`	All jobs seeded as "running" with stale heartbeat — measures rescue-to-completion time

Rust harness¶

Tests live in awa/tests/failure_benchmark_test.rs. Each scenario seeds jobs deterministically by mode, starts a Client with aggressive rescue intervals, drains to terminal states, and emits both human-readable output and a JSONL record. The full matrix command covers the 10 failure scenarios; the stale-heartbeat rescue benchmark is a separate test.

Python harness¶

The Python benchmark (awa-python/scripts/benchmark_runtime.py) supports a failure-mode subset via --scenario failures:

terminal_1pct / 10pct / 50pct
retryable_1pct / 10pct / 50pct
callback_timeout_10pct
mixed_50pct

Python also includes heartbeat rescue via --scenario rescue. It still does not include the Rust-only deadline_hang and snooze_once scenarios. The worker returns RetryAfter, WaitForCallback, Cancel, or raises exceptions based on the job's mode field.

Structured output¶

Both Rust and Python benchmarks emit one JSONL record per scenario, prefixed with @@BENCH_JSON@@ for extraction. Schema version 2:

{
  "schema_version": 2,
  "scenario": "terminal_10pct",
  "language": "rust",
  "seeded": 5000,
  "metrics": {
    "throughput": {
      "handler_per_s": 4200.0,
      "db_finalized_per_s": 4100.0
    },
    "drain_time_s": 1.22,
    "rescue": {
      "deadline_rescued": 0,
      "callback_timeouts": 0
    }
  },
  "outcomes": {
    "completed": 4500,
    "failed": 500
  }
}

Extract JSONL from mixed stdout: grep '@@BENCH_JSON@@' output.txt | sed 's/^@@BENCH_JSON@@//'

For enqueue-only benchmarks (insert_only_single, copy_single, and the contention matrix), metrics.enqueue_per_s is emitted instead of metrics.throughput. Those records still include "measurement": "enqueue" in metadata, plus optional Postgres-side deltas such as wal_bytes, temp_bytes_delta, and xact_commit_delta.

Interpreting The Results¶

Some practical guidelines:

Compare like with like. Burst/frontier benchmarks and steady-state benchmarks answer different questions.
Reset runtime state before sustained measurements if you want to isolate one path. Global background work can distort results.
Prefer the db_completed_delta view when you care about end-to-end queue completion, not just handler return rate.
Treat the numbers here as environment-specific reference points, not portable guarantees. For cross-system claims, use the companion comparison repo and its recorded environment.

How To Run¶

Happy-path benchmarks¶

DATABASE_URL=postgres://postgres:test@localhost:15432/awa_test \
  cargo test --package awa --test scheduling_benchmark_test \
  test_runtime_sustained_hot_path -- --exact --ignored --nocapture

DATABASE_URL=postgres://postgres:test@localhost:15432/awa_test \
  cargo test --package awa --test scheduling_benchmark_test \
  test_scheduled_steady_2m_due_4k_per_sec -- --exact --ignored --nocapture

Enqueue contention benchmarks¶

These are the most useful benchmarks when you want to compare single-producer enqueue against multi-producer contention, or compare chunked INSERT with the COPY staging path under concurrent writers.

DATABASE_URL=postgres://postgres:test@localhost:15432/awa_test \
  AWA_BENCH_CONTENTION_PRODUCERS=4 \
  AWA_BENCH_CONTENTION_JOBS_PER_PRODUCER=3000 \
  AWA_BENCH_INSERT_BATCH_SIZE=1000 \
  AWA_BENCH_COPY_CHUNK_SIZE=1000 \
  cargo test --package awa --test benchmark_test \
  test_enqueue_contention_matrix -- --exact --ignored --nocapture

This emits six JSONL records:

insert_single
copy_single
insert_contention_distinct
copy_contention_distinct
insert_contention_same_queue
copy_contention_same_queue

The matrix hard-resets Awa runtime tables before each scenario so later cases do not inherit a larger or dirtier jobs table from earlier ones.

By default, AWA_BENCH_COPY_CHUNK_SIZE should match AWA_BENCH_INSERT_BATCH_SIZE if you want the closest apples-to-apples comparison between chunked INSERT and chunked COPY staging. If you want to test the current "one bulk COPY per producer" shape instead, set AWA_BENCH_COPY_CHUNK_SIZE to AWA_BENCH_CONTENTION_JOBS_PER_PRODUCER.

The optional Postgres profile block in metadata.db_profile is meant to make server runs easier to interpret. In particular:

wal_bytes shows how much WAL the scenario generated
temp_bytes_delta and temp_files_delta show temp-file pressure
xact_commit_delta helps explain why many small commits degrade throughput
tup_inserted_delta shows how much table churn the database observed

Failure-mode benchmarks (Rust)¶

# Full matrix (10 failure scenarios)
DATABASE_URL=postgres://postgres:test@localhost:15432/awa_test \
  cargo test --package awa --test failure_benchmark_test \
  test_failure_bench_full_matrix -- --exact --ignored --nocapture

# Single scenario
DATABASE_URL=postgres://postgres:test@localhost:15432/awa_test \
  cargo test --package awa --test failure_benchmark_test \
  test_failure_bench_terminal_10pct -- --exact --ignored --nocapture

# Stale heartbeat rescue
DATABASE_URL=postgres://postgres:test@localhost:15432/awa_test \
  cargo test --package awa --test failure_benchmark_test \
  test_failure_bench_stale_heartbeat_rescue -- --exact --ignored --nocapture

Python benchmarks¶

cd awa-python
uv run maturin develop

# Baseline scenarios (copy, hot, scheduled)
PYTHONPATH=scripts DATABASE_URL=postgres://postgres:test@localhost:15432/awa_test \
  uv run python scripts/benchmark_runtime.py --scenario baseline

# Failure scenarios
PYTHONPATH=scripts DATABASE_URL=postgres://postgres:test@localhost:15432/awa_test \
  uv run python scripts/benchmark_runtime.py --scenario failures

# Everything
PYTHONPATH=scripts DATABASE_URL=postgres://postgres:test@localhost:15432/awa_test \
  uv run python scripts/benchmark_runtime.py --scenario all

Caveats¶

Each number is tied to the environment stated near the result. The developer commands use a localhost database URL for convenience, but that URL is not a claim about where every result was produced.
Ignored benchmark tests are not part of the normal unit/integration test pass.
The current focus is relative behavior and architectural validation, not cross-machine leaderboard comparisons.

Cross-system numbers¶

Cross-system comparison (awa vs pgque / procrastinate / pg-boss / river / oban / pgmq) lives in hardbyte/postgresql-job-queue-benchmarking. That repo is the source of truth for fair-comparison numbers, chaos results across systems, and the long-horizon harness itself.

The reasoning for the split: cross-system benches need to evolve at the pace of the systems being compared (each gets new releases, new public APIs, new adapter caveats), and the awa-only regression track in this file needs to evolve at awa's pace. Keeping them in one repo coupled their schedules and conflated "did awa regress?" with "is awa still faster than X?". They're separate questions; they're now in separate places.