Scaling a hot queue¶
One busy queue can outgrow the coordination around a single lane. A lane admits one producer at a time, because a producer holds it until it commits, and one claimer at a time, because a claim locks the lane's cursor. When many producers and workers share one (queue, priority), they queue behind those two locks while the database still has room.
This page is the tuning procedure: what to check first, which lever to try in which order, and how to tell whether it helped. Queues, lanes and splitting work explains what each lever changes about a queue and what it gives up.
Check it is short of lanes¶
Before spreading a queue, check that it is short of lane throughput rather than of workers. A queue whose utilization stays above about 0.8 needs more workers or a split by service time. In hard-reserved mode, raising max_workers adds workers, and so does a PartitionedQueue, whose partitions each get their own; in weighted mode capacity comes from global_max_workers, min_workers and weight instead. Enqueue shards and claimers add none.
Which to try first¶
- Batch. A larger claim or insert batch moves more jobs through each lock. Workers sitting idle while the queue has a backlog points at claiming, not at the lane: raise
claim_batch_sizefirst. - Partition by key. If the workload has a natural key (customer, account, order) and can live with per-key ordering, use a partitioned queue. It also gives each partition its own worker capacity and rate limit.
- Shard. If the queue must stay one queue to operators, raise
enqueue_shards. - Add claimers last, and only with evidence that one claim loop leaves workers idle; see claimers.
Measured results¶
- Enqueue shards. In a 16-producer benchmark on one queue, 2, 4 and 8 shards gave 1.6, 2.75 and 3.7 times the throughput of one shard. In the hot-queue reference shape,
enqueue_shards = 4is the change that turned overload into bounded end-to-end throughput. - Claimers. In the same reference shape, raising
claimersto 4 increased transaction pressure and made tail latency worse.
Measuring the result¶
Judge a change by durable completions per second, end-to-end p99 latency, queue depth, and WAL bytes and transactions per completed job. Compare against a run of the same workload before the change. The awa.job.claimed counter carries an awa.enqueue.shard attribute, so a dashboard can confirm that claims are spread evenly across shards.
Related¶
- Queues, lanes and splitting work: the problem each tool solves, what it gives up, and which combine.
- Lanes and claiming: what a lane is and how a claim takes from it.
- Batching: moving more jobs through each statement.
- Sharding the enqueue head, Hot-queue claim control and ADR-025: settings and design records.