Configuration

AWA has three configuration surfaces: the Rust runtime (ClientBuilder + QueueConfig), the Python runtime (client.start()), and the CLI (awa serve, awa job, etc). This guide explains how they work rather than listing every option — use --help, IDE autocomplete, or the source for exhaustive reference.

Terms used throughout this guide:

• A queue is the worker subscription and capacity boundary.
• A claim is the act of reserving a ready job for one attempt.
• A claim lease (run_lease) is the monotonically increasing attempt guard on a job; stale completions with an old value are rejected.
• A scheduled job has a future run_at and waits outside the ready claim path. Retry backoff and snooze use the same deferred backlog until their run_at is due.
• A lane is one ordered (queue, priority, enqueue_shard) stream.
• A segment is a rotating partition that Awa eventually truncates.
• A receipt is the lightweight short-attempt claim record; a lease row is materialized when an attempt needs mutable execution state.

How configuration flows

┌────────────────────────────────────┐
│  Worker process (Rust or Python)   │
│  ─ QueueConfig per queue           │
│  ─ ClientBuilder for runtime knobs │
│  ─ Connects directly to Postgres   │
└──────────────┬─────────────────────┘
               │
          PostgreSQL
               │
┌──────────────┴─────────────────────┐
│  awa serve  (admin UI + API)       │
│  ─ CLI flags / AWA_* env vars      │
│  ─ Read-only safe (auto-detected)  │
└────────────────────────────────────┘

Workers and the UI server are separate processes. Workers own all queue machinery — the UI is a read-mostly dashboard with optional admin actions.

Worker scope: which queues and which kinds

A single worker process handles work for the set of queues it was configured with and the set of job kinds it registered handlers for. There is no implicit fan-out across queues, and no implicit filter on kinds within a queue — these are two separate concerns.

Targeting specific queues

A worker only claims jobs from queues passed to its builder; everything else on the database is invisible to it. To run a worker dedicated to one queue, only declare that queue:

// Rust: this worker only ever processes the "email" queue.
let client = Client::builder(pool)
    .queue("email", QueueConfig::default())
    .register::<SendEmail, _, _>(handle_email)
    .build()?;
client.start().await?;

# Python: same idea — only "email" is in the start list.
@client.task(SendEmail, queue="email")
async def handle(job): ...

await client.start([("email", 8)])

Run separate worker processes (or separate fleets) per queue when you want isolation: a stuck etl queue can't starve email, deployment of a slow handler doesn't pause unrelated queues, and per-queue scaling is just a deployment knob. Run one worker process across multiple queues with global_max_workers and weighted mode when you want elastic capacity sharing — see Weighted mode.

Targeting specific job kinds within a queue

register::<SomeJob, _, _> (Rust) or @client.task(SomeJob, ...) (Python) tells the worker how to execute one specific job kind. At execute time the worker looks up the kind on the claimed job and runs the matching handler.

Sharp edge: the claim path is per-queue, not per-kind. If a queue holds jobs of kinds the worker didn't register, the worker still claims those jobs (it can't tell ahead of time) and then fails them terminally with unknown job kind: <name>. There is no soft re- enqueue. So the supported patterns for "this worker only handles a subset of kinds" are:

1. Recommended: put each kind set on its own queue. Queues are cheap; this is the operator-shaped boundary. Workers with different kind responsibilities subscribe to different queues, and the queue boundary becomes the routing decision.
2. Acceptable: every worker on the queue registers handlers for every kind that lands there. Heterogeneous workloads on a single queue work fine as long as no worker is missing a registration.
3. Not supported: have some workers on a queue claim jobs and silently leave kinds they don't know for someone else. This will terminal-fail or DLQ jobs.

If kinds drift out of sync (e.g. a deploy lags), the descriptor catalog in the admin UI flags missing handlers; see Queue and job-kind descriptors below.

Queue configuration

Every queue needs a QueueConfig. The two fundamental choices are:

1. Hard-reserved mode (default) — each queue gets a fixed max_workers slot count
2. Weighted mode — call global_max_workers(N) to share a pool, with min_workers as a floor and weight for overflow

Rust

let client = Client::builder(pool.clone())
    .queue("email", QueueConfig {
        max_workers: 20,
        rate_limit: Some(RateLimit { max_rate: 50.0, burst: 50 }),
        ..Default::default()
    })
    .queue("reports", QueueConfig {
        max_workers: 5,
        deadline_duration: Duration::from_secs(600),
        ..Default::default()
    })
    .register::<SendEmail, _, _>(handle_email)
    .register::<GenerateReport, _, _>(handle_report)
    .build()?;

The key QueueConfig fields:

Field	Default	When you'd change it
max_workers	50	Always — this is your concurrency cap per queue
rate_limit	None	External API rate limits, backpressure
deadline_duration	5m	Hard upper bound on a single attempt. Set to Duration::ZERO to skip the deadline rescue path; receipts mode (the 0.6 default storage) supports both shapes — the deadline lands on lease_claims.deadline_at and the maintenance rescue path force-closes expired claims.
poll_interval	200ms	Tune if NOTIFY latency matters (rare)
min_workers / weight	0 / 1	Only in weighted mode
claimers	1	Hot queue-storage queues that need more than one dispatcher/claimer loop inside a single runtime. Claimers share the queue's worker permits.
claim_batch_size	512	Maximum jobs each dispatcher tries to claim in one DB round-trip. Lower this for latency-sensitive small queues; benchmark before combining large batches with multiple claimers.

Defaults intentionally favor the smallest blast radius: enqueue_shards = 1, claimers = 1, and claim_batch_size = 512. Raise enqueue_shards only when the queue can accept partitioned FIFO semantics. For a single hot queue, a larger claim batch usually helps before extra claimers: it reduces claim round-trips without adding more concurrent head coordinators. Benchmark claimers = 2 or 4 only when a single claimer cannot keep worker permits full.

Partitioned Queues

A logical queue is the application concept: for example email or customer-updates. A physical queue is the queue name Awa stores in Postgres and workers claim from. Most applications use one physical queue per logical queue.

Very hot workloads can partition one logical queue over multiple physical queues when the ordering contract allows it. This creates independent durable claim and completion streams. It is the most direct way to reduce hot-head coordination without weakening durability: each job is still written, claimed, leased, completed, retried, and rescued through the normal Awa tables.

Rust producers and workers can share PartitionedQueue:

use awa::{Client, InsertOpts, QueueConfig, PartitionedQueue};

let customer_updates = PartitionedQueue::new("customer-updates", 4)?;

let client = Client::builder(pool.clone())
    .partitioned_queue(&customer_updates, QueueConfig {
        // Applied per physical queue: 4 × 32 hard-reserved workers.
        max_workers: 32,
        ..Default::default()
    })
    .register::<UpdateCustomer, _, _>(handle_update)
    .build()?;

let opts = customer_updates.route_opts_by_key(
    InsertOpts::default(),
    format!("customer-{customer_id}").into_bytes(),
);

With one partition, PartitionedQueue::new("email", 1) uses the plain email queue. With more than one partition, partition 0 remains the logical queue name and later partitions use stable suffixes: email, email__p1, email__p2, and so on. Key-based routing sets both the selected physical queue and InsertOpts::ordering_key, so related jobs keep per-key FIFO inside the selected partition even if each physical queue later raises enqueue_shards.

partitioned_queue applies the QueueConfig to each physical queue. In hard-reserved mode, total logical capacity is roughly partitioned_queue.partitions() * max_workers; rate_limit is also per physical queue. Divide those values yourself, or use global_max_workers, when you need a logical total cap across the partitioned queue.

Python exposes the same deterministic router through the awa.PartitionedQueue class:

queue = awa.PartitionedQueue("customer-updates", 4)

@client.task(UpdateCustomer, queue=queue.physical_queues[0])
async def update_customer(job):
    ...

await client.start(
    queue.queue_configs(
        max_workers_per_partition=16,
        claim_batch_size=512,
    )
)

await client.insert(
    UpdateCustomer(customer_id=customer_id, payload=payload),
    **queue.route_by_key(f"customer-{customer_id}"),
)

Register the handler once and pass explicit partition queue configs to start(). Python handlers are dispatched by job kind; the queue name on @client.task gives start() a declared queue to validate.

route_by_key() returns {"queue": ..., "ordering_key": ...} and can be passed directly to insert(). For insert_many_copy() and enqueue_many_copy(), pass opts=[queue.route_by_key(key) for ...] when a batch contains jobs for multiple partitions. route_by_index() returns only a queue for round-robin partitioning when per-key FIFO is not needed.

Choosing a throughput lever

These knobs solve different bottlenecks:

Knob	What changes	Use it when
PartitionedQueue	Routes one logical workload across multiple physical queues. Each physical queue has its own queue-level capacity, rate limit, claim cursor, completion stream, and metrics.	The workload can be partitioned by key or round-robin, and you want more end-to-end throughput from independent queue coordination paths.
enqueue_shards	Keeps one physical queue name, but splits that queue into multiple ordered lanes. FIFO becomes per (queue, priority, enqueue_shard) instead of global per (queue, priority).	Operators should still see one queue, but the workload can accept partitioned FIFO inside that queue.
claimers	Adds more dispatcher/claimer loops for the same physical queue inside one worker runtime. They share the queue's worker permits and rate limiter.	One claimer is leaving worker permits idle. This is not the first lever for a hot queue because extra claimers can add transaction pressure without creating independent capacity domains.

For a hot workload that can be partitioned, start with PartitionedQueue. If the public queue should stay singular, consider enqueue_shards. Raise claimers only when measurements show the runtime is claim-starved rather than Postgres-bound.

Python

Tuple form for simple cases, dict form for full control:

# Hard-reserved — just (name, max_workers)
await client.start([("email", 10), ("reports", 5)])

# Dict form — rate limiting, deadlines, priority aging, retention
await client.start([
    {"name": "email", "max_workers": 10, "rate_limit": (50.0, 50)},
    {
        "name": "reports",
        "max_workers": 5,
        "deadline_duration_ms": 600_000,
        "priority_aging_interval_ms": 30_000,
        "retention": {"completed_hours": 12, "failed_hours": 168},
    },
])

Weighted mode requires dict form and global_max_workers:

await client.start(
    [{"name": "email", "min_workers": 5, "weight": 2},
     {"name": "reports", "min_workers": 2, "weight": 1}],
    global_max_workers=20,
)

Weighted mode

Enabled by global_max_workers(N) (Rust) or global_max_workers=N (Python). Each queue's min_workers is guaranteed; remaining capacity is distributed by weight. This is useful when queue load is unpredictable and you want elastic sharing rather than static partitioning.

What can be tuned per queue vs per job

Queue configuration is the normal way to express operational policy. Job kinds are primarily handler registrations and descriptor/catalog entries; if two kinds need different runtime policy, put them on different queues.

Scope	Knobs	Notes
Per queue	max_workers, rate_limit, deadline_duration, priority_aging_interval, poll_interval, min_workers, weight	Runtime dispatch policy. deadline_duration is the hard per-attempt wall-clock timeout for every job claimed from that queue.
Per job enqueue	queue, priority, max_attempts, run_at, tags, metadata, unique	Stored with the job. Use this for routing, priority, retry budget, scheduling, identity, and operator context.
Per job kind	handler registration plus JobKindDescriptor	Descriptors cover display name, owner, docs link, tags, and extra metadata. They do not set dispatch limits or timeouts.
Per callback wait	callback timeout	Applies only to jobs that enter wait_for_callback; see Callback timeout.
Per queue / global retention and DLQ policy	completed_retention, failed_retention, queue_retention(...), dlq_enabled_by_default, queue_dlq_enabled, dlq_retention	Retention has global defaults with per-queue overrides. DLQ enablement is queue-scoped. Split job kinds across queues if only some kinds should DLQ.
Per runtime / fleet	heartbeat timings, rescue scan intervals, cleanup batch size, descriptor retention, queue-storage slots/stripes/rotation	Process or storage-engine policy, not job-kind policy. queue_storage_queue_stripe_count currently applies to the queue-storage engine rather than to one named queue.

There is no active per-job-kind hard timeout today. The hard timeout is the queue's deadline_duration; moving long-running kinds onto their own queue is the intended way to give them a different timeout without making every job on a busy queue slower to rescue.

Scheduled jobs and deferred promotion

Set run_at when enqueueing a job that should not be claimable yet. Awa stores that row in the deferred backlog and the maintenance leader promotes it to the ready ring once run_at <= now().

Retry backoff and Snooze use the same mechanism: the current attempt closes, a deferred row is written with its next run_at (retryable for backoff and RetryAfter, scheduled for a snooze), and promotion makes it ready again later. Cron schedules are producers on top of the same enqueue path; a cron fire atomically records the schedule fire and inserts the resulting job.

Operational knobs:

• promote_interval controls how often maintenance checks for due deferred rows.
• Per-job run_at controls the due time.
• Periodic schedules are declared in worker code with periodic() and managed through the CLI/UI cron surface.

Job priority and aging

Every job carries a priority (i16, lower number = higher priority). The default is 2. Conventional usage:

Priority	Typical meaning
1	Urgent / customer-facing / SLA-critical
2	Default
3	Background work
4	Batch / catch-up / bulk reprocess

Values outside 1..=4 are rejected by the insert path.

Priority enters the queue at insert time:

// Rust
awa::insert_with(
    &pool,
    &SendEmail { to: "alice@example.com".into(), subject: "Welcome".into() },
    awa::InsertOpts {
        queue: "email".into(),
        priority: 1, // urgent
        ..Default::default()
    },
).await?;

# Python — kwargs on insert
await client.insert(
    SendEmail(to="alice@example.com", subject="Welcome"),
    queue="email",
    priority=1,
)

Priority aging (escalation for fairness)

Without aging, a steady stream of priority-1 work can starve every priority-2 job behind it. AWA escalates priority over time — the longer a job has been waiting, the higher (numerically lower) its effective priority becomes at claim time. A priority-4 job that has waited 3 × aging_interval ages all the way down to priority 1 and is no longer starvable.

The cadence is per-queue via QueueConfig.priority_aging_interval (default 60s):

// Rust
.queue("etl", QueueConfig {
    max_workers: 8,
    // Drop one priority level every 30 seconds of waiting.
    priority_aging_interval: Duration::from_secs(30),
    ..Default::default()
})

# Python — same semantics, dict form
await client.start([
    {
        "name": "etl",
        "max_workers": 8,
        "priority_aging_interval_ms": 30_000,
    }
])

Aging is computed at claim time on queue-storage runtimes: ready rows keep their stored lane priority, and the claim SQL compares candidate lanes by effective priority. When a job is claimed, the live attempt records the effective priority used for that claim. In canonical storage, the maintenance leader physically rewrites waiting rows from priority N to N-1 and preserves the enqueue priority in metadata._awa_original_priority.

The admin UI shows the priority on the current job snapshot and, when _awa_original_priority is present, the original enqueue priority as (enqueued as N). Set the value to Duration::ZERO (Rust) or priority_aging_interval_ms: 0 (Python) to disable escalation entirely (strict static priority).

There is a separate top-level ClientBuilder::priority_aging_interval that controls the legacy canonical-storage maintenance pass that physically rewrites stored priorities. With queue storage (the 0.6 default) it is a no-op; the per-queue setting above is the one to tune.

Changing priority after enqueue

Queued jobs can be reprioritized through durable batch operations. In queue storage, priority is part of the physical lane key (queue, priority, enqueue_shard, lane_seq), so changing an already-queued job's priority means tombstoning the old lane and appending a replacement ready row with a fresh lane sequence. That is a semantic operation, not a metadata update.

Use one of these patterns:

• Choose the priority when inserting the job, periodic schedule, or direct-COPY batch.
• Tune priority_aging_interval when the goal is fairness under sustained high-priority load.
• For DLQ recovery, use retry_from_dlq(..., priority=...) (Python) or RetryFromDlqOpts { priority: Some(...) } (Rust/model API) to revive a DLQ row at a different priority.
• For pending available or scheduled jobs, submit op_kind = "set_priority" through /api/batch-ops or awa batch-ops submit. The operation snapshots matching job ids, previews/counts the eligible population, and then applies changes asynchronously from the maintenance leader.

Plain retry of a failed/cancelled/waiting job keeps the job's existing priority. Running, waiting, terminal, and DLQ rows are not reprioritized by batch operations; use cancel/retry semantics if an in-flight attempt must be interrupted.

awa batch-ops preview \
  --op-kind set_priority \
  --filter '{"queue":"default"}' \
  --spec '{"priority":1}'

awa batch-ops submit \
  --op-kind set_priority \
  --filter '{"queue":"default"}' \
  --spec '{"priority":1}'

Python clients expose the same durable control-plane operation. Async and sync clients use the same method names; omit await when using awa.Client:

preview = await client.preview_set_priority(
    1,
    filter={"queue": "default", "state": "available"},
)
print(preview["total_matched"])

operation = await client.set_priority(
    1,
    filter={"queue": "default"},
    submitted_by="ops@example.com",
)

operation = await client.cancel_batch_operation(operation["id"])

The generic Python envelope is also available when you want exact parity with the HTTP API:

operation = await client.submit_batch_operation(
    "move_queue",
    spec={"queue": "escalations", "priority": 1},
    filter={"tag": "incident-123"},
)

Batch-operation history is retained for AWA_BATCH_OP_RETENTION_DAYS days (default 90). Use awa batch-ops purge --before <timestamp> for explicit cleanup of finalized operations.

Queue and job-kind descriptors

Queues and job kinds can carry operator-facing metadata: display names, descriptions, owners, docs links, tags, and arbitrary JSON extra. This is separate from runtime scheduling config and drives the labels the admin UI / API surface.

The runtime catalogs and propagates these — see Architecture → Descriptors And Runtime Liveness for how sync, staleness, and drift detection work.

Rust

use awa::{Client, JobArgs, JobKindDescriptor, QueueConfig, QueueDescriptor};

let client = Client::builder(pool)
    .queue("email", QueueConfig::default())
    .queue_descriptor(
        "email",
        QueueDescriptor::new()
            .display_name("Email")
            .description("Transactional outbound email")
            .owner("messaging")
            .tag("customer-facing"),
    )
    .job_kind_descriptor::<SendEmail>(
        JobKindDescriptor::new()
            .display_name("Send email")
            .description("Deliver a single transactional email"),
    )
    .register::<SendEmail, _, _>(handle_email)
    .build()?;

Python

client = awa.AsyncClient(database_url)

@client.task(SendEmail, queue="email")
async def handle(job):
    ...

client.queue_descriptor(
    "email",
    display_name="Email",
    description="Transactional outbound email",
    owner="messaging",
    tags=["customer-facing"],
)
client.job_kind_descriptor(
    "send_email",
    display_name="Send email",
    description="Deliver a single transactional email",
)

await client.start([("email", 8)])

Both surfaces must be called before start() / build(). Declaring a descriptor for a queue the client doesn't run is an error, so dead references show up at startup instead of silently producing stale rows.

Reliability timings: heartbeat, deadline, rescue

These knobs control how fast a stuck or crashed handler is noticed and rescued. They live on ClientBuilder (Rust) and client.start() kwargs (Python). All of them have _ms-suffixed kwargs on the Python side (e.g. heartbeat_interval_ms=15000).

Heartbeat — detecting crashed workers

A running job updates a heartbeat row periodically while its handler is alive. If the heartbeat goes stale, the maintenance leader rescues the job (re-enqueues it for another attempt). Three knobs participate:

Knob	Default	What it does
heartbeat_interval	30s	How often each running handler refreshes its heartbeat row.
heartbeat_staleness	90s	How long the row may go un-refreshed before the maintenance leader treats the job as crashed.
heartbeat_rescue_interval	30s	How often the maintenance leader scans for stale heartbeats.

Pick them in this order:

1. Decide your detection target. "Crashes should be noticed within X seconds." That target is roughly heartbeat_staleness + heartbeat_rescue_interval in the worst case.
2. Set heartbeat_staleness to at least 3× heartbeat_interval. The 3× rule absorbs scheduler hiccups, GC pauses, and the rescue scan's own jitter; tighter ratios produce false rescues. The builder logs a warning if you violate it.
3. heartbeat_rescue_interval can match heartbeat_interval for low-latency rescue, or be higher to reduce maintenance load on big fleets.

For a 5-second crash detection target: heartbeat_interval=1s, heartbeat_staleness=4s, heartbeat_rescue_interval=1s. For a 1-minute target with the cheapest possible maintenance: keep all the defaults.

Deadline — bounding a single attempt

Each queue has a deadline_duration (default 5m on QueueConfig). At claim time the runtime stamps now() + deadline_duration onto the claim, and a maintenance scan force-closes attempts that pass it without completing. This bounds a single attempt's wall-clock time independently of heartbeats — if a handler is hanging, looping forever, or wedged in a sync wait, the deadline rescues it even if its heartbeat is fresh.

Knob	Default	What it does
QueueConfig.deadline_duration (Rust) / deadline_duration_ms (Python dict form)	5m	Per-queue hard upper bound on one attempt. Duration::ZERO / 0 skips deadline rescue for that queue.
ClientBuilder::deadline_rescue_interval (Rust) / deadline_rescue_interval_ms (Python kwarg)	30s	How often the maintenance leader scans for expired deadlines.

Receipts mode (the 0.6 default storage) supports both shapes: the deadline lands on lease_claims.deadline_at and is rescued there for short claims, or carried onto leases.deadline_at if the claim materializes for a long-running attempt. See Queue storage tuning and ADR-023.

Callback timeout — bounding wait_for_callback

If you suspend a handler with wait_for_callback() and the external system never resumes, a callback-timeout rescue brings the job back to ready (or DLQ if attempts are exhausted).

Knob	Default	What it does
ClientBuilder::callback_rescue_interval	30s	How often the maintenance leader scans for callback_timeout_at < now(). The per-callback timeout itself is set when registering the callback in the handler.

Retention and cleanup

Retention depends on the storage path.

Queue storage is the worker engine in 0.6: ordinary terminal history (completed, non-DLQ failed, and cancelled) is reclaimed by queue-ring prune after the matching ready segment is no longer live. Successful receipt completions can live in compact receipt_completion_batches_* rows; failed, cancelled, non-receipt, and wide terminal snapshots live in done_entries_*. Custom metadata, tags, errors, and unique-key state make a successful completion wide; Awa-owned provenance metadata from priority aging or queue/priority moves stays eligible for compact receipt completion. Both physical histories hydrate through {schema}.terminal_jobs while immutable job-body fields remain in the retained ready_entries_* row until queue prune reclaims the segment. Direct SQL against done_entries should therefore expect nullable body columns and should not assume all completed jobs are present there. Use {schema}.terminal_jobs for read-only SQL inspection.

Ready rows are not deleted for unclaimed cancellation, priority aging, or SQL compatibility deletes through awa.jobs. Those paths append to {schema}.ready_tombstones_*; claim treats tombstoned lanes as spent evidence, exact-count queries skip the tombstone ledger, and queue prune truncates it with the matching ready/done segment.

failed_retention is a retention floor for non-DLQ failed terminal rows in both storage engines: a failed terminal row stays retryable for at least failed_retention past the moment it failed. On the canonical compatibility path the floor is enforced by row cleanup leaving recent failed rows in place. In queue storage the floor is enforced by carry-forward at prune time: when a queue slot is reclaimed, failed rows still inside the floor are widened into self-contained terminal rows and re-inserted into the live segment before the old slot is truncated, so they survive rotation regardless of how fast the ring turns over. Failed rows that have aged past the floor are pruned; their count is recorded in queue_terminal_rollups.pruned_failed_count and surfaced as QueueCounts.pruned_failed, a cumulative, monotonic count of failed rows no longer retryable. See ADR-032. cancelled rows are not held by the floor — cancellation is an explicit decision and cancelled rows are reclaimed with their segment.

completed_retention is canonical-only: it bounds how long completed jobs stay queryable on the compatibility path. Queue storage keeps ordinary completed terminal history in the rotating ring and reclaims it by queue-ring prune; use the queue-ring sizing and rotation knobs below to control how much remains queryable.

DLQ rows are different: dlq_entries is a separate hold table and the maintenance leader deletes rows older than dlq_retention in bounded cleanup passes.

Knob	Default	What it does
completed_retention	24h	Canonical compatibility path only: how long completed jobs stay queryable before row cleanup deletes them. Queue storage reclaims completed history by queue-ring prune instead.
failed_retention	72h	Both engines: a non-DLQ failed terminal row stays retryable for at least this long past failure. Canonical enforces it by leaving recent failed rows in place; queue storage enforces it by carry-forward at prune time. Does not apply to cancelled rows or to the DLQ.
dlq_retention	30d	How long DLQ rows stay in dlq_entries before bounded cleanup deletes them.
descriptor_retention	30d	How long stale queue/kind descriptor catalog rows survive.
cleanup_interval	60s	How often the cleanup pass runs.
cleanup_batch_size	1000	Max rows deleted per pass. Raise for very high throughput; lower if you want gentler IO.
dlq_cleanup_batch_size	1000	DLQ-specific batch size.
ClientBuilder::terminal_count_rollup_interval (Rust) / terminal_count_rollup_interval_ms (Python kwarg)	30s	Queue storage only: how often maintenance folds pending done_entries terminal-count deltas for sealed queue slots into compact live counters. Exact reads include pending deltas and retained compact receipt batches, so this affects compaction/WAL pressure rather than correctness.

Queue storage tuning

Queue storage is the runtime engine in 0.6, and most deployments can keep the defaults. Queue-storage tables live in the canonical awa schema; the main knobs are there for large fleets, very bursty queues, or operators who want to trade off the partition-reclaim window against rotation churn.

Producer path choice

For bulk producers on queue storage, prefer direct queue-storage COPY: Python enqueue_many_copy() or Rust QueueStorage::enqueue_params_copy(). Those paths stream rows straight into ready_entries / deferred_jobs and use the queue-storage enqueue heads directly.

Direct-copy producers must use the same queue-storage routing config as the worker fleet. This matters most for queue_stripe_count / queue_storage_queue_stripe_count: a producer using the default unstriped config can write to queue while striped workers claim from queue#0, queue#1, etc. Rust producers should construct QueueStorage with the same QueueStorageConfig used by workers. Python producers should pass the same queue_storage_queue_stripe_count to enqueue_many_copy() when the fleet uses non-default striping.

insert_many_copy() is the compatibility insert API. It is still useful when a caller needs the canonical insert surface, but in queue-storage mode it routes through the compatibility function rather than being the primary producer fast path.

Terminal count rollup

Queue storage keeps exact terminal counts without updating mutable counter rows on the completion hot path. done_entries terminal inserts append positive rows to queue_terminal_count_deltas_*; retry, discard, DLQ move, and compatibility delete of done_entries rows append negative rows. Compact receipt completions do not write count deltas: exact count reads add retained receipt_completion_batches_* counts and subtract retained receipt_completion_tombstones_*.

A queue slot is one partition in queue storage's rotating ready/done ring. A sealed slot is no longer the current write slot, so maintenance can fold its terminal-count deltas without touching the active completion segment.

The maintenance leader folds pending done_entries deltas from sealed queue slots every terminal_count_rollup_interval (default 30s) and processes at most four slots per tick. Rollup is MVCC-horizon aware: if another backend is holding a visible snapshot open, or is idle in a transaction with an assigned transaction id, maintenance leaves pending delta rows append-only and tries again on a later tick. Exact counts remain correct because they read folded counters, pending deltas, retained compact batches, compact tombstones, and permanent rollups.

Raising the interval reduces folded-counter churn but leaves more pending delta rows for exact reads to sum. Lowering it folds counters sooner, which can help if exact-count reads are frequent and the done_entries delta ledger is growing faster than maintenance drains it. Under long reader transactions, lowering the interval does not force rollup; the useful fix is to release the pinned snapshot or let queue prune reclaim the whole sealed segment. Compact receipt successes are already retained as compact batch rows, so this interval does not change their hot-path write count.

Rust

let client = Client::builder(pool.clone())
    .queue("email", QueueConfig::default())
    .queue_storage(
        QueueStorageConfig {
            queue_slot_count: 16,
            lease_slot_count: 8,
            claim_slot_count: 8,
            queue_stripe_count: 1,
            ..Default::default()
        },
        Duration::from_millis(1_000),
        Duration::from_millis(1_000),
    )
    .claim_rotate_interval(Duration::from_millis(1_000))
    .build()?;

Python

await client.start(
    [("email", 8)],
    queue_storage_schema="awa",
    queue_storage_queue_slot_count=16,
    queue_storage_lease_slot_count=8,
    queue_storage_claim_slot_count=8,
    queue_storage_queue_stripe_count=1,
    queue_storage_queue_rotate_interval_ms=1000,
    queue_storage_lease_rotate_interval_ms=1000,
    queue_storage_claim_rotate_interval_ms=1000,
)

What the knobs mean

Knob	Default	What it controls
queue_slot_count	16	Number of rotating ready, claim-attempt, tombstone, and terminal queue partitions. Together with queue rotation cadence and how quickly segments become prunable, this bounds ordinary terminal-history visibility in queue storage. Ready-backed terminal rows and claim-attempt evidence rely on the matching ready partition until queue prune reclaims the segment.
lease_slot_count	8	Number of rotating lease partitions
claim_slot_count	8	Number of rotating ADR-023 claim-ring partitions (lease_claims, lease_claim_closures, and lease_claim_closure_batches children). All three tables share the same claim_slot so each partition's claims and closure evidence are reclaimed together by TRUNCATE; compact closure batches also carry ready segment metadata so prune can prove sealed segments by counts and skip conservatively when counts do not prove closure. Each slot has tiny stale-rescue and deadline-rescue cursors in claim_ring_slots.
queue_stripe_count / queue_storage_queue_stripe_count	1	Number of physical stripes behind each logical queue. 1 is the normal unstriped path. For a single very hot queue on many small replicas, 2 is the current tuned recommendation; higher values should be benchmarked before use.
lease_claim_receipts	true	Use the receipt-plane short path (claim writes compact attempt-emission range evidence into ready_claim_attempt_batches and receipt evidence into row-local lease_claims or compact lease_claim_batches; successful compact completion validates the returned receipt identity, writes receipt_completion_batches terminal history, and writes lease_claim_closure_batches closure evidence; non-success exits write explicit closures into lease_claim_closures; receipt tables are reclaimed by claim-ring prune, and attempt-emission evidence is reclaimed by queue-ring prune). Receipts mode supports per-claim deadlines: when QueueConfig.deadline_duration > 0, the claim writes clock_timestamp() + interval onto lease_claims.deadline_at and the maintenance rescue path force-closes expired claims with a 'deadline_expired' closure. Set to false to force every claim through the legacy leases materialization path. See ADR-023.
queue_rotate_interval	1000ms	How often ready/terminal segments rotate
lease_rotate_interval	1000ms	How often lease segments rotate
claim_rotate_interval	matches queue_rotate_interval	How often the ADR-023 claim-ring rotates. Set with ClientBuilder::claim_rotate_interval (Rust) or queue_storage_claim_rotate_interval_ms (Python). Test harnesses sometimes set this to a long interval to pin claim-ring layout for deterministic count assertions.

The benchmark harness in postgresql-job-queue-benchmarking reads QUEUE_SLOT_COUNT, LEASE_SLOT_COUNT, CLAIM_SLOT_COUNT, QUEUE_STRIPE_COUNT, and LEASE_CLAIM_RECEIPTS from the environment. Those env vars are benchmark configuration, not general worker-runtime configuration.

Use the defaults unless you have a reason not to:

• Increase queue_slot_count if queue partitions stay unprunable for too long because readers, active leases, or pending ready rows keep old segments live.
• Increase lease_slot_count if lease churn is high enough that dead tuples in the lease ring stop collapsing promptly.
• Increase claim_slot_count if the rotation cadence (claim_rotate_interval) plus the slot count combine to a partition retention window shorter than your longest in-flight zero-deadline short job; running out of empty slots forces rotate_claims to return SkippedBusy and the receipt-plane churn falls back onto a smaller working set of partitions.
• Increase queue_stripe_count only for measured workloads where many small replicas contend on the same logical queue. This is queue-storage configuration, not a per-queue QueueConfig field: it gives each logical queue the same number of physical stripes. Striping spreads a hot logical queue over queue#N physical coordination paths, but it weakens perfect global ordering and can regress calmer shapes if overused. For single hot-queue workloads, benchmark 2 stripes first; higher values should be treated as workload-specific tuning rather than a general default.
• Increase rotation intervals to reduce partition churn and metadata activity.
• Decrease rotation intervals to tighten dead-tuple bounds at the cost of more frequent rotate/prune work.

Sharding the enqueue head per queue

Each queue's awa.queue_meta.enqueue_shards (default 1, range 1..=64) governs how many independent enqueue head rows the queue has. With the default value, the queue uses a single head row and the ordering contract is strict FIFO per (queue, priority) — identical to v016.

Raising enqueue_shards is a semantic mode switch, not a hidden performance knob. The queue's contract becomes partitioned FIFO — strict order within (queue, priority, enqueue_shard), no order promised across shards. It is the same kind of decision as choosing SQS Standard over SQS FIFO, raising Kafka partition count, or using Pub/Sub ordering keys.

-- Opt a contended queue into 4 shards.
INSERT INTO awa.queue_meta (queue, enqueue_shards)
VALUES ('my_hot_queue', 4)
ON CONFLICT (queue)
DO UPDATE SET enqueue_shards = EXCLUDED.enqueue_shards;

Use an upsert rather than a plain UPDATE: the first enqueue can create queue-storage lane rows before an operator-owned queue_meta row exists, so a plain UPDATE may quietly affect zero rows.

A 16-producer same-queue reference sweep measured 1.0× / 1.60× / 2.75× / 3.69× at S=1/2/4/8. Application authors who need per-key FIFO at S>1 (per-customer, per-order, per-account) pass InsertOpts::ordering_key (Rust) or ordering_key=... (Python) on insert — jobs sharing the key always land on the same shard regardless of which producer batch emitted them.

Observability: the awa.job.claimed OTel counter carries an awa.enqueue.shard attribute on the queue-storage claim path. Dashboards can sum by that attribute to confirm the claim ordering is rotating fairly across shards.

Lowering the value is safe at any time. See ADR-025 for the full design and contract.

Hot-Queue Claim Control

Queue storage also uses a bounded-claimer control plane (queue_claimer_state / queue_claimer_leases) so not every replica hammers a hot queue's claim path at once. For a single hot queue, raising QueueConfig.claimers lets one runtime run multiple dispatcher/claimer loops while still sharing the queue's max_workers / min_workers permits.

Keep claimers modest. In the hot-queue reference shape, enqueue_shards = 4 is the change that turns overload into bounded end-to-end throughput; raising claimers to 4 increases transaction pressure and worsens tail latency in that shape. For extreme single-queue workloads, raise enqueue_shards first when the ordering contract allows partitioned FIFO. Benchmark claimers = 2 or 4 only for a workload-specific reason, and judge the result by durable completion rate, p99 end-to-end latency, queue depth, WAL bytes per completed job, transaction commits per completed job, and dead tuples. Lower claim_batch_size from 512 for small or latency-sensitive queues; raise claimers only with workload-specific evidence.

Queue storage defaults AWA_COMPLETION_SHARDS to 1 for ordinary runtimes and 4 when the configured runtime worker capacity is at least 512 (8 on canonical storage). Extra completion flushers can improve large single-process shapes, but they also multiply terminal-write contention across a worker fleet. Override this only after measuring the same north-star metrics for the target topology.

AWA_COMPLETION_BATCH_SIZE defaults to 512 and AWA_COMPLETION_FLUSH_MS defaults to 1. The queue-storage short-job path fuses receipt closure and terminal insertion into one statement, so the lower flush interval reduces the worker capacity tied up waiting for durable completion while the batch size still amortises the claim/complete path under load. That pairing is what moves the lower WAL budget into durable throughput.

For very high-throughput no-op queues, size max_workers for durable completion latency, not handler CPU alone. The 10k/s offered-rate reference shape needs 1,024 in-flight worker permits to absorb the load consistently; real jobs with non-trivial handler time need workload-specific sizing.

The terminal write path is already narrow for running/waiting jobs: successful receipt completions use compact receipt_completion_batches_* rows, and other retained terminal facts use done_entries_* without duplicating immutable ready body fields when the ready row is still retained. Both surfaces hydrate through terminal_jobs, so the default completion settings stay conservative without giving up durable terminal history.

Live tuning: per-queue runtime overrides

The dispatch knobs can be changed on a running fleet (ADR-038) — no restart:

$ awa queue overrides set email --poll-interval-ms 1000 --claim-batch-size 64
$ awa queue overrides show email
$ awa queue overrides clear email

Overridable: poll_interval, claim_batch_size, rate_limit (retunes a queue built with a rate limit; enabling limiting live is not supported), and deadline_duration (must stay on the same side of zero — the claim mode is fixed at startup). Semantics: the builder value is the default, a set override wins, clearing reverts. set replaces the whole override state, so show first when adjusting a single knob. Workers apply changes within their refresh cadence (10s by default; AWA_OVERRIDE_REFRESH_MS tunes it).

Worker health listener

Set AWA_HEALTH_ADDR (or ClientBuilder::health_addr) to serve GET /healthz and GET /readyz from the worker runtime for infrastructure probes:

AWA_HEALTH_ADDR=0.0.0.0:8321

Port 0 binds an ephemeral port (read it back with Client::health_listener_addr()). Unset means no listener. Endpoint semantics, response fields, and Kubernetes probe examples live in Deployment; awa health covers probe-less environments from the CLI.

Distributed tracing

When a producer enqueues inside an OpenTelemetry span (any tracing span under a tracing-opentelemetry layer), awa captures the W3C traceparent into the job's metadata under the reserved awa:traceparent key and reconnects it at execution (ADR-039):

• First attempt — the job.execute span joins the producer's trace as a remote child: enqueue → execute reads as one trace.
• Retries — each retry starts a fresh root trace carrying a span link back to the enqueue site, so a job retried hours later doesn't stretch the original trace across its backoff schedule or inherit its sampling decision.

Both sides carry OTel messaging semantic conventions (messaging.system = "awa", producer/consumer span kinds, send {queue} / job.execute {kind} names). To propagate onward from a Rust handler (outgoing HTTP headers), use the ambient context — awa_model::trace::current_traceparent() — so downstream spans nest under the execution span; ctx.traceparent() (Rust) and job.traceparent (Python) return the stored enqueue-site context for inspection or for continuing the trace where no ambient context exists.

Worker-side traces

The dispatcher's own poll and the heartbeat tick are debug-level root spans — receive {queue} (consumer kind, with the messaging attributes and messaging.batch.message_count) and heartbeat.tick. Neither loop is instrumented as a whole: a span covering a loop that lives as long as the worker never closes, and every child accumulates under it until the trace exceeds what a backend will accept (#449).

They are debug rather than info because they tick whether or not there is work: at the default 200 ms poll_interval, an info-level poll span would open ~5 traces/s per queue-claimer, nearly all of them recording "claimed nothing". A pipeline filtering at info — the usual production setting — creates neither span, and nothing is allocated or exported. Raise your filter to debug for the dispatcher target when you want them.

What you keep either way: empty and non-empty polls alike are counted by awa.dispatch.empty_claim and timed by the claim-duration histogram, which are metrics and unaffected by span filtering. With no poll span at info, a job enqueued without trace context executes under its own root span instead of nesting into the poll that claimed it; a job carrying awa:traceparent still joins its producer's trace as before.

Capture costs well under a microsecond per enqueue and is on by default; AWA_TRACE_CAPTURE=off disables ambient capture process-wide (Rust and Python alike). Python producers are captured at the binding layer whenever opentelemetry-api is importable, and Python handlers get the job's trace context attached as their ambient OpenTelemetry context — see the Python guide. An explicit metadata["awa:traceparent"] supplied at enqueue always wins — that is also the participation path for SQL-native producers.

Dead Letter Queue

The DLQ is the separate, durable hold-table for jobs that exhausted retries or hit a non-retryable terminal failure. Without it, terminal failures live in ordinary queue-storage done_entries_* partitions and are reclaimed when queue-ring prune can safely truncate their segment. With DLQ enabled, those rows land in dlq_entries, are visible to the admin UI / API as a discrete backlog, and can be retried or purged by an operator.

When to enable

• Enable DLQ for queues whose terminal failures need an operator decision. Payment, notification, billing — anything where you'd rather a human triages a failure than have the job silently age out.
• Leave DLQ disabled for high-throughput queues whose failures are fire-and-forget. Logging, telemetry, ETL retries that get re-driven from upstream — accumulating dead rows here is just storage cost.
• Default to disabled unless you've decided either way; the builder's dlq_enabled_by_default is the global switch and queue_dlq_enabled is the per-queue override.

Configuring

use std::time::Duration;

let client = Client::builder(pool.clone())
    .dlq_enabled_by_default(true)            // default for every queue
    .queue_dlq_enabled("metrics_flush", false) // exception: this one stays off
    .dlq_retention(Duration::from_secs(60 * 60 * 24 * 30))  // 30 days
    .dlq_cleanup_batch_size(1000)
    .build()
    .await?;

DLQ policy is per-queue, not per-job-kind. If you need a single queue to handle some kinds with DLQ and some without, split them onto two queues (the same shape recommended for worker scope by kind).

Per-queue retention overrides go through RetentionPolicy.dlq on queue_retention(queue, policy) — handy if one queue's failures need to live longer than the global dlq_retention.

Operator workflow

Once DLQ is on, operators interact with it through:

• Web UI — DLQ tab with retry, purge, and per-row failure detail.
• CLI — awa dlq list, awa dlq retry <id>, awa dlq purge.
• REST — /api/dlq/* endpoints (same actions as the UI).
• Python / Rust — client.dlq_* admin methods for programmatic retry / purge.

Queue-policy declaration is still a runtime-side concern; the UI / API report state but don't toggle DLQ on/off (that's a code-level decision so it doesn't drift between deployments). See ADR-020.

CLI and awa serve

The CLI resolves its database target in this order: --database-url > --context/AWA_CONTEXT > the DATABASE_URL environment variable > default_context from the config file (see Named contexts below). The resolved target is echoed, password-stripped, to stderr before any command acts. All subcommands except serve use a single database connection.

awa serve starts the admin UI and API. It has its own connection pool and response cache, configurable via CLI flags or environment variables:

awa serve --pool-max 10 --cache-ttl 5

Every flag has a corresponding AWA_* environment variable (shown in --help):

Flag	Env var	Default	Purpose
--pool-max	AWA_POOL_MAX	10	Max database connections
--pool-min	AWA_POOL_MIN	2	Min idle connections
--pool-idle-timeout	AWA_POOL_IDLE_TIMEOUT	300	Idle connection timeout (seconds)
--pool-max-lifetime	AWA_POOL_MAX_LIFETIME	1800	Max connection lifetime (seconds)
--pool-acquire-timeout	AWA_POOL_ACQUIRE_TIMEOUT	10	Connection acquire timeout (seconds)
--cache-ttl	AWA_CACHE_TTL	5	Dashboard query cache TTL (seconds)

Dashboard cache

The cache deduplicates repeated poll requests — multiple browser tabs or rapid refresh cycles within the TTL window hit memory rather than the database. The frontend polling interval is derived from the cache TTL (minimum 5s) and served via /api/capabilities, so clients automatically back off to match the server's refresh rate.

If you're connecting to a read replica, increase --cache-ttl to reduce load. The dashboard will feel slightly less real-time but won't overwhelm the replica.

Read-only mode

awa serve disables mutation endpoints (retry, cancel, pause, drain) whenever the server is running in read-only mode. /api/capabilities reports read_only: true and the frontend hides the corresponding buttons. Mutation requests against a read-only server return 503 Service Unavailable with a clear error body.

There are two ways to opt in:

Mode	Trigger	When to use
Auto-detect (default)	Server probes current_setting('transaction_read_only') on startup	Pointed at a read replica or a Postgres role without write grants
Forced	--read-only flag or AWA_READ_ONLY=1 env var	Writable DB but you want mutations off — incident read-outs, shared debugging instances, less-trusted public UI sessions

# Auto-detect (current behaviour)
awa --database-url "$DATABASE_URL" serve

# Explicit — force read-only regardless of DB privileges
awa --database-url "$DATABASE_URL" serve --read-only

# Same via env var
AWA_READ_ONLY=1 awa --database-url "$DATABASE_URL" serve

Once forced, there is no way for a frontend user to flip back to writable without restarting the server — that's the whole point.

Instance identity

The UI is single-backend by design — run one awa serve per database (#437). When you operate several databases (say Cloud SQL and AlloyDB production instances), give each UI a name so open tabs — and mutation targets — are distinguishable at a glance:

awa --database-url "$AWA_CLOUDSQL_URL" serve \
  --instance-name cloudsql-prod \
  --instance-color '#0ea5e9' \
  --peer alloydb=https://awa-alloydb.internal

Flag	Env var	Purpose
--instance-name	AWA_INSTANCE_NAME	Name rendered in the header badge and browser tab title
--instance-color	AWA_INSTANCE_COLOR	Accent color (CSS hex, e.g. #0ea5e9) tinting the header badge and favicon
--peer NAME=URL (repeatable)	—	Plain links to peer Awa UIs in the header — a zero-data-plane "switcher"

The identity is exposed via /api/capabilities (instance_name, instance_color, peers). There is deliberately no multi-database mode: every identifier in the UI (job ids, queue names, kinds) is database-scoped, so a merged view would need a backend qualifier on every route and mutation — and would double the consequence of a mis-click. Two labeled tabs fail into confusion; one merged UI fails into mutating the wrong production database.

If peers are reachable from less-trusted networks, pair this with read-only mode (above) so instance links don't encourage exposing writable UIs.

Named contexts

Operating several Awa databases from one workstation with a bare DATABASE_URL is a foot-gun: nothing tells you which database awa migrate or awa dlq purge is about to hit. Named contexts (#437) fix that with a kubectl-style config file at ~/.config/awa/config.toml (override with $AWA_CONFIG; $XDG_CONFIG_HOME is honored) — but deliberately without a sticky "use-context" for mutations, because a sticky current context is exactly how people mutate the wrong production cluster:

default_context = "cloudsql"   # honored by read-only commands only

[contexts.cloudsql]
url_env = "AWA_CLOUDSQL_URL"   # indirection: no secret in the file
production = true

[contexts.alloydb]
url_env = "AWA_ALLOYDB_URL"
production = true

[contexts.local]
url = "postgres://postgres:test@localhost:15432/awa_test"

Each context sets exactly one of url (inline) or url_env (the named environment variable is read at resolution time — fits Cloud SQL / AlloyDB IAM connector auth, and keeps secrets out of the file). If a plaintext url embeds a password and the file is group/other-readable, the CLI warns and suggests chmod 600 or url_env.

Resolution order: --database-url > --context/AWA_CONTEXT > DATABASE_URL env > default_context. Three safety rules carry the weight:

1. Read/write asymmetry. Read-only commands (job list, queue stats, dlq depth, health, storage status, …) may fall back to DATABASE_URL or default_context. Mutating commands (migrate, storage transitions and flips, dlq retry/purge/move, queue drain/pause, batch-ops submit, cron remove, and every other write) require an explicit --context or --database-url whenever more than one context is defined.
2. Target echo. The password-stripped target is printed to stderr before any command acts: target: context 'alloydb' [production] — postgres://app@10.0.0.9:5432/awa.
3. Production gate. Mutating commands against a context marked production = true prompt for interactive y/N confirmation; pass --yes for automation (a non-interactive stdin without --yes refuses rather than hanging).

Inspect the config with awa context list (marks the default and production contexts, shows password-stripped targets) and awa context show [name]. With no config file, behaviour is unchanged: --database-url and DATABASE_URL work exactly as before.

Next

• Deployment guide
• Migration guide
• Troubleshooting