Troubleshooting¶
This guide covers the operational issues that come up most often in practice: stuck running jobs, leader-election delays, and heartbeat timeouts.
Quick Checks¶
Start with these:
awa --database-url "$DATABASE_URL" queue stats
awa --database-url "$DATABASE_URL" job list --state running
awa --database-url "$DATABASE_URL" serve
The web UI is usually the fastest way to see queue depth, runtime health, and individual job details.
Jobs Stuck In running¶
What It Usually Means¶
A job staying in running is not automatically a bug. It usually means one of these:
- the handler is still genuinely working
- the process is draining during shutdown
- the worker stopped heartbeating and rescue has not fired yet
- the job hit a long deadline and is waiting for deadline rescue
Inspect Running Jobs¶
In 0.6 with the default queue-storage backend, running attempts live in {schema}.leases (lease-materialized attempts) and unmaterialized short-job claims live in {schema}.lease_claims. The {schema} is awa unless the operator chose a different name; replace below as needed.
-- Materialized leases. heartbeat_at and deadline_at both populated.
SELECT
job_id,
queue,
attempt,
heartbeat_at,
deadline_at,
EXTRACT(EPOCH FROM (now() - heartbeat_at)) AS heartbeat_age_s,
EXTRACT(EPOCH FROM (deadline_at - now())) AS deadline_remaining_s
FROM awa.leases
WHERE state = 'running'
ORDER BY heartbeat_at ASC;
-- Receipt-mode short-job claims that haven't materialized yet. No
-- heartbeat row; deadline_at lives on the claim if deadline_duration > 0
-- and the receipt-side deadline rescue (`rescue_expired_receipt_deadlines_tx`)
-- is what kicks in here. Anti-join with closure evidence to filter
-- "still open".
SELECT
c.job_id,
c.queue,
c.attempt,
c.claimed_at,
c.deadline_at,
EXTRACT(EPOCH FROM (now() - c.claimed_at)) AS claimed_age_s,
EXTRACT(EPOCH FROM (c.deadline_at - now())) AS deadline_remaining_s
FROM awa.lease_claims c
WHERE NOT EXISTS (
SELECT 1 FROM awa.lease_claim_closures cx
WHERE cx.claim_slot = c.claim_slot
AND cx.job_id = c.job_id
AND cx.run_lease = c.run_lease
)
AND NOT EXISTS (
SELECT 1 FROM awa.lease_claim_closure_batches cb
WHERE cb.receipt_ranges @> c.receipt_id
)
ORDER BY c.claimed_at ASC;
Pre-0.6 / canonical-only deployments still query awa.jobs directly:
SELECT id, kind, queue, attempt, heartbeat_at, deadline_at
FROM awa.jobs
WHERE state = 'running'
ORDER BY heartbeat_at ASC;
Interpretation:
- low
heartbeat_age_s: the worker is still alive. If its handler has already returned and its log showsAnother transaction holds this attempt's completion lock; still retrying, see a completion waits on an application's transaction heartbeat_age_swell aboveheartbeat_staleness(default90s): stale-heartbeat rescue should pick it up on the next scan- very negative
deadline_remaining_s: deadline rescue should pick it up soon (lease side or receipt side, depending on which table the row lives in)
Inspect Runtime Health¶
SELECT
instance_id,
hostname,
pid,
last_seen_at,
healthy,
postgres_connected,
poll_loop_alive,
heartbeat_alive,
maintenance_alive,
leader,
shutting_down
FROM awa.runtime_instances
ORDER BY leader DESC, last_seen_at DESC;
If heartbeat_alive or maintenance_alive is false for the active runtime, fix that before manually changing job state.
Recovery Options¶
- if the job is still making progress, leave it alone
- if the worker is gone, wait for rescue to move it back to
retryable - if you need operator intervention now, cancel it:
Then investigate why the worker stopped.
Admin Cancel Doesn't Wake A Running Worker¶
awa job cancel <id> writes the cancellation to the database and emits pg_notify('awa:cancel', {job_id, run_lease}). Each worker runtime runs a CancelListener (PgListener-based) that picks up the notification and fires the matching Arc<AtomicBool> cancel flag the handler's JobContext holds — so the handler sees ctx.is_cancelled() == true on its next check and can stop cleanly. If the notification arrives after the claim but before the attempt registers to run, the runtime keeps it for that attempt, for up to five minutes and at most 256 per registry shard, and does not run the handler; the worker logs Skipping a handler whose attempt was cancelled before it started. This depends on the notification arriving before the handler starts, not on the cancel's commit: a notification that arrives later, or never, leaves the handler running until it checks is_cancelled() or finishes, and the run_lease check then rejects its completion.
If the listener fails to start, it logs Failed to start the admin cancel listener; retrying in the background and retries with backoff from 250 ms up to 30 s until it connects, then logs Admin cancel listener started after retrying; if its connection drops, it logs PG cancel listener connection lost; reconnecting and listens again, and cancels committed while it was down do not wake their handlers. In both cases admin cancels fall back to heartbeat / deadline rescue for detection (i.e. the cancel does take effect on the next completion-time run_lease check, but the handler doesn't get the early wake-up). Symptoms: awa job cancel returns success and the database shows the job as cancelled, but a worker reports completing the cancelled run_lease long after the cancel was issued; the completion is rejected by StaleCompleteRejected so no harm done, just wasted work.
To diagnose:
- Search the worker logs for
Failed to start the admin cancel listener. The runtime is in fallback mode untilAdmin cancel listener started after retryingfollows it. - Search for
PG cancel listener error; will retry— the listener saw a transient error and is sleeping 1s before reconnecting; not a steady-state failure unless it spams the log. - Confirm
pg_listenerworks from outside the runtime:psql -c "LISTEN \"awa:cancel\";"and watch a separate session fireSELECT pg_notify('awa:cancel', '{}');.
To recover:
- The listener retries on its own; a restart is not needed once the database accepts its connection.
- If the failure is steady (e.g. a Postgres pool sized so all connections are saturated by claim/complete traffic and
LISTENcan't acquire one), increase the pool max or dedicate a connection to listening.
In all cases the cancel itself is durable — only the early wake-up is lost. Long-running handlers that don't poll ctx.is_cancelled() between heartbeats won't notice the cancel until they finish or heartbeat-rescue fires.
Claiming Slows While Handlers Are Idle¶
What It Usually Means¶
Durable completion, or lifecycle hooks, are behind the handlers. A runtime lets only a bounded number of finished attempts wait for their completion to commit (see completion settings). Once that allowance is used up, finished handlers keep their worker slots, awa_job_in_flight sits at max_workers while handler CPU is idle, and the runtime claims only as fast as completions commit. Worker memory stays flat. 0.6 releases before 0.6.15 kept claiming instead, and their memory grew with every finished attempt until the process was killed.
What To Check¶
awa_completion_flush_duration_seconds: long flushes point at the database, usually lock waits on the completion or terminal tables. Look for long transactions and lock waits on the queue-storage schema, as in Inspect Long Transactions.awa_completion_deadlock_retry_total: a steady rate means completions are re-running after deadlocks.- Lifecycle hooks (
on_event,on_event_kind): a hook that waits on a slow service or on the same database holds its attempt's slot until it returns. Move slow work into a job or a follow-up.
Fixes¶
Clear the lock contention or long transaction; claiming recovers as soon as completions commit again. Raising max_workers does not help while completion is the bottleneck. For jobs with large payloads, lowering AWA_COMPLETION_BATCH_SIZE reduces how much memory waiting attempts can hold.
A Completion Waits On An Application's Transaction¶
A transaction that changes a running attempt, such as admin::cancel_tx run inside an application's own transaction, holds that attempt's completion lock until it commits or rolls back. If the attempt's handler returns meanwhile, its completion cannot proceed until then. Once the attempt has waited a second, the worker logs Another transaction holds this attempt's completion lock; still retrying with the attempt's job_id and waited_ms.
In queue storage with receipts on (the default), a successful attempt in that state is left out of its completion batch and retried with backoff, without holding a pool connection, so completions of other jobs carry on. It keeps its finalizer slot and its heartbeat while it retries; its worker slot was released when the attempt was handed to the finalizer. When the transaction commits, the retry sees the committed change: a cancelled attempt's completion is ignored as stale. If the lock is still held after two minutes of finalization budget (time spent waiting for an accepted completion to be acknowledged does not count), the worker gives up, logs Failed to complete job, stops heartbeating the attempt, and leaves it to stale-heartbeat rescue.
Releases before 0.6.17, and 0.7.0-alpha.1, waited on the lock inside the batch instead. Every completion the runtime sent through its completion batcher then waited behind that one transaction, and once the in-flight completion allowance filled, the runtime stopped claiming in every queue.
Not every completion is covered:
- Attempts that end in a retry, snooze, cancel, failure or callback wait, and successful attempts of a kind with follow-up enqueues or whose handler staged follow-ups, finalize one job at a time. They still wait on the lock while holding a pool connection, a finalizer slot and their heartbeat, but no other attempt waits with them.
- With receipts off (
lease_claim_receipts = false, only possible on a non-default queue-storage schema), a successful attempt's batch still waits on the lock, and the rest of the batch waits with it.
Keep transactions that touch running jobs short, and commit or roll back before doing slow work in them.
Rescue Fails With idx_awa_jobs_unique¶
Symptom: maintenance repeats, every tick (canonical engine — the queue-storage
engine wedges identically but logs error: unique conflict instead):
Failed to rescue stale heartbeat jobs, error: ... duplicate key value violates unique constraint "idx_awa_jobs_unique"
Failed to rescue deadline-expired jobs, error: ... duplicate key value violates unique constraint "idx_awa_jobs_unique"
Cause: a unique job whose unique_states mask excludes running was running without a claim, a newer duplicate took the key, and the old attempt then tried to enter a claiming state. Rescue reports this as rescued as duplicate. During a storage transition, snooze/retry migration reports the same race as rescheduled as duplicate. In both fixed paths the old attempt becomes cancelled terminal evidence, never executable work, and the newer claim holder wins. Since 0.6.1 rescue degrades row-at-a-time; the transition-time reschedule behavior is fixed by #456. On affected older builds the whole rescue batch can abort every tick, or a migrated successor can lose its claim — upgrade rather than deleting a live claim by hand:
SELECT j.id AS stuck_job, j.kind, j.queue, j.unique_key,
c.job_id AS claim_holder, h.state AS holder_state
FROM awa.jobs_hot j
JOIN awa.job_unique_claims c ON c.unique_key = j.unique_key AND c.job_id <> j.id
LEFT JOIN awa.jobs h ON h.id = c.job_id
WHERE j.state = 'running';
Cancel the stuck_job (cancelled is outside the default mask, so the transition succeeds). Delete a claim row only after proving its recorded holder no longer exists; deleting a live holder's claim admits a duplicate.
Prevention: choose unique_states masks that are closed under runtime transitions — retry/rescue move running -> retryable, snooze moves running -> scheduled, and promotion moves retryable -> available or scheduled -> available. A mask a transition can enter from outside (for example {scheduled, available, retryable} without running) intentionally permits a newer enqueue during execution, so the superseded old attempt can end in a fallback cancellation. {} (no dedup) and the full non-terminal set {scheduled, available, running, retryable} avoid that race; pair the full mask with a short by_period bucket if you need "a change during a run still gets a fresh run" semantics.
Producer Enqueue Is Slower Than Expected¶
What It Usually Means¶
Producers are enqueuing well below the rate the application offers, batches are taking longer than expected, or the consumer fleet is permanently undersaturated despite plenty of producer concurrency.
Reading The Producer Histograms¶
The direct queue-storage COPY path (QueueStorage::enqueue_params_copy in Rust, Client.enqueue_many_copy in Python) records two histograms per batch:
| Metric | What it measures |
|---|---|
awa.enqueue.batch_size |
Rows per COPY call |
awa.enqueue.duration |
Wall-clock time per COPY call |
Reading them together separates the two usual failure modes:
- Batches are tiny.
batch_sizep50 sits at a handful of rows when the upstream flow is bursty or chunked too aggressively. Throughput is bounded by the per-batch round-trip cost, not the COPY itself. Increase chunk size or batch the upstream side. - Batches are slow.
batch_sizep50 is reasonable (say ≥100) butdurationp99 is high. The COPY itself is contended on the DB side; see the diagnoses below.
Diagnoses¶
1. Producer is going through the compatibility insert path.
Before migration v055, insert_many_copy_from_pool / client.insert_many_copy routes every row through awa.insert_job_compat() once per row, and a batch with unique jobs adds a savepoint round trip per row. On any database with non-trivial per-statement latency (auth-proxy hop, managed Postgres, network) the per-row cost dominates — measured at ~100–150 ms per row through an auth proxy, which caps a single producer at ~7 rows/sec regardless of batch or chunk size. Run awa migrate to install awa.insert_jobs_compat_batch; COPY then inserts the batch in one function call. Generic insert_many keeps its one-statement per-row path either way.
Switch the producer to the direct queue-storage COPY entry point:
- Rust:
QueueStorage::enqueue_params_copy(pool, &jobs) - Python:
client.enqueue_many_copy(jobs)
See Producer path choice.
2. enqueue_shards is 1 for the contended queue.
The default enqueue_shards = 1 means every producer contends on a single enqueue-head row per (queue, priority). Multi-producer enqueue serialises through that row. Check the live value:
If no row is returned the queue is running with the default 1. Raise it explicitly (a 16-producer same-queue reference sweep measured 1.0× → 1.60× → 2.75× → 3.69× at S = 1/2/4/8):
INSERT INTO awa.queue_meta (queue, enqueue_shards)
VALUES ('<queue>', 4)
ON CONFLICT (queue)
DO UPDATE SET enqueue_shards = EXCLUDED.enqueue_shards;
Use an upsert: first-enqueue may create lane rows before any operator inserts a queue_meta row, so a plain UPDATE quietly affects zero rows. Raising enqueue_shards is a semantic switch from strict to partitioned FIFO; see ADR-025.
3. Producers wait on one queue's enqueue lane.
Many connections wait with wait_event = 'advisory' while the database is otherwise idle, batch inserts take seconds although the work per batch is milliseconds, and promotion into the same queue runs late while producers saturate it.
On queue storage every enqueue holds a transaction-scoped advisory lock on its (queue, priority, shard) lane until it commits, so concurrent producers for one lane run one at a time; a lane's ceiling is a batch's rows divided by the time the lock is held. Generic insert_many and per-row COPY take the lock at the batch's first ready row and hold it while the rest of the batch is processed. From migration v055, insert_many_copy and insert_many_skip_unique_conflicts prepare the whole batch (job ids, unique claims, payloads) before they take any lane lock, and take each lane's lock when they write that lane; it is still held until the transaction ends.
- Run
awa migrateand move hot producers toinsert_many_copyorinsert_many_skip_unique_conflicts. - When
insert_manyfails on a unique conflict, retry withinsert_many_skip_unique_conflictsrather than inserting the batch again row by row: each single-row insert queues for the lane separately. - Keep producer transactions short after the enqueue: the lane stays locked until commit.
- Raise
enqueue_shards(above) for a queue whose producers still saturate one lane.
4. WAL or commit pressure on the database.
If batch size and shard count both look healthy, sample pg_stat_activity for LWLock:WALWrite / LWLock:WALSync waits and confirm the database is sized for the offered rate. The per-vCPU sustained-completion and burst-enqueue numbers in Managed Postgres sizing are useful reference points.
Leader Election Delays¶
Expected Behavior¶
Only one instance runs maintenance tasks at a time. Non-leaders retry every 10s by default.
That means failover is not instant by default. A short delay after a pod restart or network blip is normal.
What To Check¶
From the runtime snapshot table or your app health endpoint, confirm:
- at least one live instance exists
- in steady state, one live instance reports
leader = true - the leader also reports
maintenance_alive = true
During failover there may be a brief window with no leader before the next election retry succeeds.
If no leader appears:
- verify Postgres connectivity from worker pods
- confirm the worker process actually called
start() - make sure the pool is not exhausted
Tuning¶
If you want faster leader failover:
- Rust: lower
ClientBuilder::leader_election_interval(...) - Python: lower
leader_election_interval_ms=...inclient.start(...)
Use lower values carefully. Faster checks mean more advisory-lock traffic.
A Maintenance Statement Waits On A Lock¶
The leader runs promotion and the three rescues (stale heartbeat, deadline, callback timeout) in one loop, and its other maintenance in another: cleanup, cron sync, metadata, batch operations, terminal-count rollup and ring rotation and prune. A statement in the second loop that waits on a lock, such as the cron sync upserting a schedule row another session holds, delays only that loop. Promotion and rescue keep running, and schedules already stored keep firing, because the cron evaluator runs on its own task.
Everything else in the second loop does wait. While that statement waits, cleanup does not run and the queue, lease and claim rings do not rotate or prune, so queue-storage partitions are not reclaimed. Shutdown also waits for it to finish.
A leader whose scheduled jobs are not promoted is therefore not blocked by its own cleanup or cron sync. Look instead at promotion's own lane locks (awa.maintenance.promote.lane_contention), at pool exhaustion, and at a long rescue pass. To find a stuck maintenance statement, list the sessions waiting on locks, and pick out the leader's by application_name or by its pid in pg_locks:
SELECT pid, application_name, wait_event_type, wait_event,
now() - query_start AS waiting, left(query, 120)
FROM pg_stat_activity
WHERE wait_event_type = 'Lock' AND datname = current_database()
ORDER BY query_start;
0.6.16 and earlier, and 0.7 alpha builds, ran every branch in one loop, so there a single waiting statement stopped promotion, rescue and cron until its lock was released. See ADR-051.
Heartbeat Timeouts¶
Defaults¶
Current runtime defaults:
- heartbeat interval:
30s - stale-heartbeat rescue tick:
30s - stale-heartbeat cutoff: about
90s - deadline duration:
5m
Common Causes¶
- the worker process crashed or was OOM-killed
- Postgres became unreachable
- the pod was terminated before graceful shutdown finished
- the pool was undersized and the runtime could not get connections reliably
For Python workers specifically, the heartbeat loop runs on the Rust runtime rather than the Python event loop. If heartbeats still stop, suspect process health or database connectivity first.
Fixes¶
- increase container
terminationGracePeriodSeconds - make sure your app calls
shutdown(...)on SIGTERM - increase
deadline_durationfor legitimately long jobs - increase pool size if the runtime is starved for DB connections
If long-running jobs are expected, remember:
- heartbeats keep the job alive
- they do not override
deadline_duration
If a job really needs 20m, set a longer deadline for that queue.
Claim Ring Stops Rotating After A Runtime Is Killed¶
Symptoms on queue storage, usually after a high-throughput runtime was OOM-killed or lost without a graceful shutdown:
awa.maintenance.rotate.attemptsfor the claim ring reportsskipped_busywith blockerclaim.rows, andawa.ring.generationfor the claim ring stops advancing- queue prune skips the slot those claims reference with reason
queue.unclosed_claim_refs, and the queue ring stops wrapping too - the current claim slot keeps growing
- jobs the dead runtime held wait well past
heartbeat_stalenessbefore they run again
The dead runtime's claims stay open until rescue closes them, and the ring cannot rotate back into a slot that still holds open claims. Rescue drains such a backlog in passes of 500-attempt batches (Draining a backlog of abandoned claims). To see where it is:
awa.maintenance.rescue.oldest_attempt_agebyawa.rescue.kindshows how old the attempts rescue is closing are. It should fall towardsheartbeat_staleness(or the queue's deadline) as the backlog drains.awa.maintenance.rescue.pass_sizebyawa.rescue.pass_endshows why passes stop.time_budgetorrow_limitmeans a drain is running at its configured rate.owners_unconfirmedmeans stale-heartbeat rescue is waiting for evidence that the claims' runtime is gone and is taking one batch per ordinary rescue interval.
If stale-heartbeat passes stay at owners_unconfirmed, look at the runtime snapshots:
SELECT instance_id, hostname, last_seen_at, snapshot_interval_ms, heartbeat_alive
FROM awa.runtime_instances
ORDER BY last_seen_at DESC;
A snapshot that is no longer fresh but is younger than heartbeat_staleness holds acceleration back, as does a fresh snapshot with heartbeat_alive = false. The unconfirmed phase requires 3 × max(runtime_snapshot_interval, 1s) < heartbeat_staleness for every runtime; otherwise a stalled live runtime can pass directly from fresh to gone (owner-evidence limits). The drain also needs heartbeat_rescue_interval below half of heartbeat_staleness. With the defaults, acceleration usually starts about 3–3.5 minutes after the last runtime snapshot, and a leadership change restarts the evidence window. Deadline rescue is not held back this way, so queues with a deadline_duration drain through it regardless.
To drain faster, raise rescue_pass_budget (at most half the rescue interval, so 15 s by default) and rescue_pass_max_rows, keeping in mind that a pass holds the maintenance leader for its budget and can exceed rescue_pass_max_rows by one final batch of closures and retry rows. The measured rates show what to expect. Retry due times can cluster; this cap does not bound promotion throughput.
Queue Ring Stops Wrapping Behind An Unclaimed Lane¶
Symptoms on queue storage, while claims, completions and the other queues stay healthy:
awa.maintenance.prune.attemptsfor the queue ring reportsskipped_activewith reasonqueue.pending_ready, and once rotation reaches the same slotawa.maintenance.rotate.attemptsreportsskipped_busywith blockerqueue.ready_rowsawa.ring.generationfor the queue ring stops advancing- the current slot's
ready_entries_*,done_entries_*,ready_segments_*andready_claim_attempt_batches_*partitions grow at the cluster's enqueue rate, for every queue - since 0.6.18,
awa.maintenance.prune.pinned_ready_agereports the lane responsible, and the maintenance leader logsQueue ring prune is pinned by unclaimed ready rowswith itsqueue,priority,enqueue_shardandoldest_ready_age_secs
Queue prune truncates the oldest sealed slot only once every lane's ready rows in it are behind that lane's claim cursor, and rotation cannot reuse a slot until prune has truncated it. A lane that nobody claims therefore holds the oldest slot, and with it storage reclamation for the whole ring, until it is claimed. The usual causes are:
- a paused queue with ready rows
- a queue no running worker serves, such as a deployment scaled to zero, or one rate-limited far below its arrival rate
- a lower priority under strict priority (
priority_aging_intervalof zero) while a higher priority of the same queue stays backlogged
A claim on the lane releases the slot:
- on a queue with strict priority, set a non-zero
priority_aging_interval; the waiting lower-priority rows then age ahead of newer higher-priority work and are claimed, which is the fastest fix for a starved lane - resume a paused queue
- start workers for the queue, or raise its rate limit
Moving the stuck jobs to another queue or changing their priority does not release the slot: the old ready rows stay in place, tombstoned, and count until a claim on their lane passes them. Cancelling them releases it only when the cancels reach the lane's claim cursor in lane order, as drain_queue, one bulk_cancel of all the lane's waiting jobs, or cancel one job at a time in enqueue order do. Jobs cancelled in another order, or with cancel_tx, keep pinning the slot until the lane is claimed.
awa.maintenance.prune.pinned_ready_age is a gauge of the age of each pinning lane's first pending row, by awa_job_queue, awa_job_priority and awa_queue_enqueue_shard. The leader updates it when queue prune runs and, while a lane pins the slot, at least every 15 seconds or queue rotation tick, whichever is longer, even while prune backs off; it sets a lane to 0 once it stops pinning the slot or the runtime loses leadership. A short-lived value is normal while a backlog drains through a full ring. Alert when a lane stays old:
- alert: AwaQueueRingPinnedByLane
expr: max by (awa_job_queue, awa_job_priority) (awa_maintenance_prune_pinned_ready_age_seconds) > 3600
for: 15m
annotations:
summary: "Queue {{ $labels.awa_job_queue }} priority {{ $labels.awa_job_priority }} keeps the queue ring from being reclaimed"
Pick the threshold above the longest time a healthy lane's oldest row waits to be claimed. To list the lanes from Rust, QueueStorage::queue_ring_pin returns the oldest sealed slot and each pinning lane with its first pending row's lane_seq, run_at and age.
The ring is shared by every queue in the database. To keep queues that are expected to sit paused, unserved or starved from holding back reclamation for the rest, serve them from a separate database; see storage isolation. A separate queue-storage schema in the same database is not a boundary, since a database has one active queue-storage schema.
Ring Prune Stays Blocked Under Claim Load¶
Symptoms on queue storage under sustained claim traffic from several runtimes (#721):
awa.maintenance.prune.attemptsfor the queue ring, the claim ring or both keeps rising with outcomeblocked(reasonnone), andprunedstops advancing; prune retries every tick, since ablockedoutcome does not back it offawa.maintenance.rotate.attemptsfor the same ring reportsskipped_busy, usually with several blockers at once (queue.ready_rows,queue.ready_segments,queue.done_rowsand the other slot tables, orclaim.rows,claim.closure_rowsandclaim.closure_batch_rows)awa.ring.generationfor that ring stops advancing- the current slot's
ready_entries_*andready_segments_*partitions keep growing, andawa.dispatch.claim_durationrises as they grow
No jobs are lost: a blocked prune rolls back and changes nothing. The cost is storage that is not reclaimed and claim latency that grows with the current slot.
Claims hold AccessShareLock on the ring partitions they read, which can include the slot prune wants to truncate. When claims from several runtimes overlap continuously, prune's lock request never finds the slot free within its wait, the ring cannot rotate into the unpruned slot, the current slot grows, and slower claims overlap more. The loop sustains itself without any other blocker.
A deployment with 14 runtimes serving about 13 queues on 0.6.16 kept its queue ring on one slot for about two days this way, with that slot's ready partition at about 55M rows (64 GB). Sampling pg_locks on the next slot's partitions every 25 ms never found them free: each sample showed four to six AccessShareLock holders, all short transactions running the claim function, including claims for a queue with no backlog.
0.7 does not fix it yet. Queue prune already runs its skip checks before it requests ACCESS EXCLUSIVE (and checks them again once it holds it), but it makes one request per attempt: it does not have 0.6.17's retry of the ACCESS EXCLUSIVE request with doubling waits up to 320 ms.
Confirm It¶
Tell it apart from the other ring stalls by the prune outcome:
| Fault | awa.maintenance.prune.attempts |
|---|---|
| Claims starve prune (this section) | blocked, reason none |
| Unclaimed lane pins the slot | skipped_active, reason queue.pending_ready |
| Abandoned claims after a killed runtime | skipped_active, reason queue.unclosed_claim_refs or claim.open |
Do not use awa.maintenance.prune.pinned_ready_age to rule this fault in or out. The leader updates the gauge only after a skipped_active prune and leaves it unchanged when prune is blocked, so during this fault it can keep showing a lane's age from an earlier pin, or 0.
To see what holds the slots prune wants, list the locks on the oldest sealed slot of the queue ring and of the claim ring; the query picks each slot the way prune does, from the ring cursor that ring_cursor resolves for either cursor authority. Replace <schema> with the active schema_name from Find the active queue-storage schema, and run it as a role that may execute <schema>.ring_cursor, such as the migration role, and that can see other roles' sessions (a superuser, or a member of pg_read_all_stats): without that, pg_stat_activity hides the query, xact_start and age of the runtime role's sessions, and the second query below leaves them out. The query reads catalog and lock views, the ring state rows and the rotation ledger only, and takes no lock on the partitions; run it several times, since one sample can miss a short claim. Look at the rows for the ring whose prune reports blocked.
WITH cursors AS (
SELECT 'queue' AS ring, c.slot, c.generation, s.slot_count
FROM <schema>.ring_cursor('queue') AS c, <schema>.queue_ring_state AS s
WHERE s.singleton
UNION ALL
SELECT 'claim', c.slot, c.generation, s.slot_count
FROM <schema>.ring_cursor('claim') AS c, <schema>.claim_ring_state AS s
WHERE s.singleton
), oldest AS (
SELECT ring,
((slot - (LEAST(generation + 1, slot_count) - 1)) % slot_count + slot_count) % slot_count AS slot
FROM cursors
WHERE LEAST(generation + 1, slot_count) > 1
), targets AS (
SELECT o.ring, p.prefix || o.slot AS relname
FROM oldest o
CROSS JOIN (VALUES
('queue', 'ready_entries_'), ('queue', 'ready_segments_'), ('queue', 'done_entries_'),
('queue', 'ready_claim_attempt_batches_'), ('queue', 'ready_tombstones_'),
('queue', 'receipt_completion_batches_'), ('queue', 'receipt_completion_tombstones_'),
('queue', 'queue_terminal_count_deltas_'),
('claim', 'lease_claims_'), ('claim', 'lease_claim_batches_'),
('claim', 'lease_claim_closures_'), ('claim', 'lease_claim_closure_batches_')
) AS p(ring, prefix)
WHERE p.ring = o.ring
)
SELECT t.ring, c.relname, l.mode, l.granted, a.pid, a.application_name,
now() - a.xact_start AS xact_age, left(a.query, 80) AS query
FROM targets t
JOIN pg_class c ON c.relname = t.relname
JOIN pg_namespace n ON n.oid = c.relnamespace AND n.nspname = '<schema>'
JOIN pg_locks l ON l.relation = c.oid
AND l.database = (SELECT oid FROM pg_database WHERE datname = current_database())
JOIN pg_stat_activity a ON a.pid = l.pid
ORDER BY t.ring, c.relname, l.granted, a.xact_start;
This fault shows as a few holders in every sample, each a different short transaction: on the queue ring, AccessShareLock from claim_ready_runtime or a claim_ready_slot_<N> function; on the claim ring, claims and completions that read or write the claim tables. A single long xact_age, or a reader that is not a claim, points instead to the slow readers described under prune_lock_timeout.
Long readers of the queue-storage tables stall prune the same way, without any claim load. A query that scans leases, lease_claims, lease_claim_closures, done_entries, ready_entries or another ring table through its parent holds ACCESS SHARE on every partition of that table, sealed slots included, until its transaction ends, and prune cannot truncate the oldest slot until it finishes. Monitoring and dashboard queries are the usual source: a count over these tables can run for minutes with millions of jobs in the ring, and polls issued back to back keep prune blocked and the ring from rotating for as long as they run. The query above covers only the queue and claim rings' prune targets; this one lists every transaction older than five seconds that holds a lock on any table in the queue-storage schema, for every ring:
SELECT a.pid, a.application_name, now() - a.xact_start AS xact_age,
count(DISTINCT l.relation) AS relations, left(a.query, 80) AS query
FROM pg_locks l
JOIN pg_class c ON c.oid = l.relation
AND l.database = (SELECT oid FROM pg_database WHERE datname = current_database())
JOIN pg_namespace n ON n.oid = c.relnamespace AND n.nspname = '<schema>'
JOIN pg_stat_activity a ON a.pid = l.pid
WHERE a.xact_start < now() - interval '5 seconds'
GROUP BY a.pid, a.application_name, a.xact_start, a.query
ORDER BY xact_age DESC;
Run such readers against a replica or with a short statement_timeout, poll them less often, or use Awa's metrics instead. A reader that finished its query but left its transaction open (state = 'idle in transaction') keeps its locks until the transaction ends, which statement_timeout does not do: end it, or set idle_in_transaction_session_timeout for the reporting role.
Mitigations¶
None of these has been measured against this fault. Change one at a time and watch pruned, awa.ring.generation and awa.dispatch.claim_duration.
- Reduce overlapping claims. Serve each queue from fewer runtimes, especially queues that are usually idle, since every runtime that serves a queue claims on it. A dispatcher whose fallback poll finds no ready jobs does not claim, so idle queues add fewer claims than busy ones.
- Raise
prune_lock_timeoutmodestly, for example to50msor100ms. Only theready_segmentspartition waits for this timeout; the other queue partitions are lockedNOWAIT. While prune'sACCESS EXCLUSIVErequest waits, every new claim on that partition queues behind it, so a longer wait trades claim latency for reclamation. Keep it well belowdeadlock_timeoutandstatement_timeout; startup warns when it reaches either. - Stop long readers of the ring tables. If either lock query shows a monitoring, dashboard or reporting query rather than claims, bound or move that query instead; see above.
- Do not lengthen the rotation interval for this fault. It does not help prune get its lock, and it makes each slot larger.
- Split workloads across databases. Claims lock only their own database's ring partitions, so moving some queues and their runtimes to a separate database leaves fewer overlapping claims on each ring, at the cost of a second installation to migrate and operate.
Once prune gets its lock, for example when claim traffic drops, the ring rotates again and the oversized slot is reclaimed when the ring reaches it.
Dead Tuples Growing In Queue Storage¶
What It Usually Means¶
If queue throughput starts to sag while lease tables keep accumulating dead tuples, the usual cause is not duplicate dispatch. The more common pattern is:
- workers are still claiming and completing jobs
- the maintenance leader is still rotating and pruning
- one or more long-lived transactions on the primary are holding an old snapshot open or touching older segments
- prune cannot truncate old lease or terminal segments aggressively enough because the MVCC horizon is pinned or a reader still holds a lockable view of the segment
This is the same general failure mode described in PlanetScale's "Keeping a Postgres queue healthy" post.
Find The Active Queue-Storage Schema¶
Inspect Table Churn¶
Replace <schema> with the active schema_name from the previous query.
SELECT
relname,
n_live_tup,
n_dead_tup,
vacuum_count,
autovacuum_count,
last_vacuum,
last_autovacuum
FROM pg_stat_user_tables
WHERE schemaname = '<schema>'
AND (
relname = 'attempt_state'
OR relname = 'deferred_jobs'
OR relname = 'dlq_entries'
OR relname LIKE 'ready_segments_%'
OR relname LIKE 'ready_entries%'
OR relname LIKE 'ready_tombstones%'
OR relname LIKE 'done_entries%'
OR relname LIKE 'queue_terminal_count_deltas%'
OR relname LIKE 'leases%'
)
ORDER BY n_dead_tup DESC, relname;
Interpretation:
ready_entries%,ready_tombstones%, andqueue_terminal_count_deltas%should usually stay at or near zero dead tuplesready_segments_*is compact control-plane metadata; live rows should track retained ready lane ranges, and queue prune truncates reclaimed slot children with the rest of the ready familyleases%can rise within the current rotation window, but should fall again after pruneattempt_stateshould roughly match live long-running attempts, not total queue depth, and should return close to zero after drainautovacuum_countstaying flat for a long time can indicate vacuum is not keeping up on churn-heavy lease partitions- persistent dead tuples across many
leases%tables usually means prune is blocked or the maintenance leader is unhealthy
Inspect Long Transactions¶
SELECT
pid,
usename,
application_name,
client_addr,
state,
now() - xact_start AS xact_age,
wait_event_type,
query
FROM pg_stat_activity
WHERE datname = current_database()
AND xact_start IS NOT NULL
ORDER BY xact_start ASC;
Pay particular attention to:
- sessions with large
xact_age - sessions in
idle in transaction - read-only / reporting tools that keep a snapshot open for a long time
Recovery Options¶
- terminate or fix the long-lived transaction that is pinning the horizon
- move analytical reads to a replica
- shorten transaction scope in admin or reporting code
- confirm one maintenance leader is healthy; rotation and prune are leader-owned
- reduce terminal-row retention if the terminal history is much larger than needed; note that
failed_retentionis a floor — non-DLQfailedrows stay retryable for at least that long in both engines (queue storage carries in-floor failed rows forward at prune time), and the cumulative count of failed rows aged past the floor is visible asQueueCounts.pruned_failed - review autovacuum settings if lease churn is expected continuously
If you want to reproduce the behavior locally before changing settings, run the MVCC benchmark documented in Benchmarking. Preventative guidance — reader placement, session timeouts, alerting, and autovacuum capacity flags — lives in Managed Postgres: MVCC discipline.
Something's In The DLQ¶
The Dead Letter Queue is where terminal failures land when a queue has DLQ enabled. These rows are never claimed. Operators retry them, purge them, or let retention remove them.
Quick checks:
awa --database-url "$DATABASE_URL" dlq depth
awa --database-url "$DATABASE_URL" dlq list --limit 20
awa --database-url "$DATABASE_URL" dlq list --queue email
dlq_reason tells you how the row got there:
terminal_errormax_attempts_exhaustedheartbeat_staledeadline_exceededcallback_timeout- an operator-supplied reason from
awa dlq move
A burst of heartbeat_stale or deadline_exceeded from one kind usually means a poison job: one that crashes or hangs its worker on every attempt. See Poison jobs.
Useful actions:
awa --database-url "$DATABASE_URL" dlq retry <job-id>
awa --database-url "$DATABASE_URL" dlq retry-bulk --queue email
awa --database-url "$DATABASE_URL" dlq purge --queue email
awa --database-url "$DATABASE_URL" dlq move --queue email --reason backfill
If DLQ depth is growing quickly:
- inspect a few rows and compare
dlq_reason - sample
awa job dump <id>for the full error/progress chain - pause the upstream queue or producer if one failure mode is dominating
- tune
dlq_retentionorRetentionPolicy.dlqif forensic retention is too long for the incident volume
Common Error Cases¶
SchemaNotMigrated¶
Cause:
- application code is newer than the database schema
Fix:
deadlock detected while enqueueing¶
Cause:
- on queue storage every enqueue holds a transaction-scoped advisory lock on its
(queue, priority, shard)lane until the transaction commits; theDETAILnames two 64-bit advisory keys, and two transactions that took lanes in opposite orders hold one each and wait on the other's insert_many,insert_many_copy,insert_many_skip_unique_conflictsand theQueueStoragebatch enqueues register and lock their lanes one at a time in one canonical order,(queue, priority, shard)ascending with queue names compared bytewise, so two batch calls cannot hold each other's lanes in opposite orders- a
QueueStorageprocess (batch enqueue, or a worker promoting scheduled or retried jobs) from 0.6.15 or earlier, or from a 0.7 build before this fix, creates a new(queue, priority)lane by taking its provisioning lock before writing itsqueue_lanesrow; every other producer, and currentQueueStorage, writes the row first, so one of each that are both the first to use the lane can deadlock (#692). Upgrade those processes, or create a queue's lanes before traffic reaches it withawa storage prepare-queue - from migration v055,
insert_many_copyandinsert_many_skip_unique_conflictstake all of a batch's unique claims before any lane lock, while per-row inserts (genericinsert_many, single-job inserts, and binaries that predate the batch function) take each job's claim and lane together. From migration v057, for 2 s the function waits at most 500 ms (or half of its session'sdeadlock_timeout, if lower) while it holds claims or lanes, then releases them and retries, so it gives way to a per-row transaction that holds a lane the batch needs and waits for one of its keys, provided that transaction'sdeadlock_timeoutis above 500 ms (the default is 1 s) and the batch inserts its unique claims within 500 ms (10,000 to 20,000 keys on a fast server, fewer under load; split larger batches). Its last attempt waits as any statement does, so a per-row transaction that holds the lane for longer than the retries (about 2.5 s) before it inserts the key still deadlocks with it, and a cycle that retrying cannot break, for example through a lock the batch's own transaction took in an earlier statement, still ends in40P01, after 3 to 4 s and usually by aborting the batch's transaction. The v055 body of the function waited instead, and the two deadlocked;awa migratereplaces it - unique keys and the application's own locks can form other cycles
- a transaction that enqueues with several separate statements (
insertper job, handler-staged follow-ups,Transaction.insertin a loop) takes locks in statement order and can deadlock against a genericinsert_many, aQueueStoragebatch enqueue or another such transaction that touches the same lanes in a different order; a batch throughawa.insert_jobs_compat_batchnormally gives way to it instead
Fix:
- enqueue every lane an application transaction touches with one
insert_manycall, or issue the single-job inserts in(queue, priority)order - keep transactions that hold enqueue locks short, since the locks last until commit
- retry on
40P01: Postgres aborts one of the two transactions and the other proceeds
Find the lane behind a key from the DETAIL line ([database, high, low, 1], the key is high << 32 | low):
SELECT queue, priority, enqueue_shard
FROM <queue_storage_schema>.queue_enqueue_heads
WHERE hashtextextended(format('%I.%I', '<queue_storage_schema>', seq_name), 0)
= (<high>::bigint << 32) | <low>::bigint;
relation "awa.job_id_seq" does not exist¶
Cause:
- A producer started before the queue-storage substrate had been materialised on a fresh database, or the schema was dropped under a running fleet.
awa.job_id_seq is part of the queue-storage substrate created by the runtime's prepare_schema step the first time a worker boots against a schema — not by awa migrate alone, which builds the canonical surface. Migrations + worker startup together produce a usable schema.
Fix:
- Wait for the first consumer pod to log
Awa worker runtime startedbefore scaling up producers. - If the schema was deliberately reset, re-run
awa migrateand let one worker pod boot to completion before resuming producer traffic. - If the error persists after both, the runtime may be pointing at a different schema than the producer. Confirm with:
cannot delete from table ... because it does not have a replica identity¶
cannot delete from table "queue_terminal_rollup_deltas" because it does not
have a replica identity and publishes deletes
The awa schema is in a logical-replication publication (CREATE PUBLICATION
... FOR ALL TABLES, which Debezium creates by default, or FOR TABLES IN
SCHEMA awa) and the database is on a schema older than v047, where some
tables had no primary key or replica identity. PostgreSQL then rejects every
UPDATE and DELETE on those tables, so maintenance fails on each tick and
dirty-mark drains, rollup folds and unique-claim releases stop.
Fix: run awa migrate to v047 or later, which gives every awa table a replica
identity. Independently, prefer keeping the awa schema out of the publication;
see Logical replication and
CDC for the
recommended publication shape and what to expect if the queue is published.
register at least one worker before starting the runtime¶
Cause:
client.start()was called before any worker registration
Fix:
- Rust: call
.register(...)or.register_worker(...)before.build() - Python: add
@client.worker(...)handlers beforeclient.start(...)
weighted mode requires explicit queue configs¶
Cause:
- Python
client.start(global_max_workers=...)was used without explicit queue config dicts
Fix:
Queue or job-kind shows "stale descriptor" in the UI¶
The descriptor catalog is refreshed by live workers on every runtime snapshot tick. A descriptor is flagged stale when no live runtime has touched last_seen_at within the snapshot window — typically because the code that declared it has been retired or no workers are currently running.
- If the queue/kind is genuinely retired, delete the row:
- If workers should be running, check
/runtimefor missing instances. - If the declaration moved to a different worker role that isn't deployed yet, the stale status will clear once that rollout completes.
Queue or job-kind shows "descriptor drift" in the UI¶
Two or more live runtimes are reporting different BLAKE3 hashes for the same descriptor — i.e. they disagree on its fields.
- During a rolling deploy this is normal and clears when the old revision finishes draining.
- If it persists, two branches of your worker code are running simultaneously (e.g. a partial rollback or a split between the Rust and Python runtimes with different descriptor declarations). Reconcile the declaring code and re-deploy.
When To Escalate¶
Escalate beyond normal operator actions when:
runningjobs are accumulating and no instance reports a healthy leaderlast_seen_atis stale for every runtime instance- jobs repeatedly rescue and retry without making progress
- you need to change the schema state manually
At that point, inspect worker logs, Postgres health, and recent deployment events together.