ADR-032: Failed terminal retention floor¶

Status¶

Accepted.

Context¶

Queue-storage terminal rows live in the rotating done_entries_* ring and are reclaimed by queue-ring prune once the matching ready segment is no longer live. Prune is segment-coarse: when the oldest sealed queue slot is reclaimable, the prune transaction TRUNCATEs the ready, tombstone, terminal, and delta children for that slot in one shot. ADR-026 made this cheap and correct for ordinary terminal history by treating ready and terminal rows as one retention unit.

That coarseness creates a silent-loss cliff for failed terminal rows. A job that exhausts its handler with retries left, or a queue that does not opt into the DLQ, lands a failed terminal fact in done_entries. An operator who wants to re-drive those failures — retry_failed_by_queue, retry_failed_by_kind, or a manual scan — has only until the slot rotates out and prune truncates it. There is no per-row retention applied to a failed row before the segment is reclaimed: the row is dropped on the floor of the prune TRUNCATE with no warning and no record that it ever existed past the terminal counters. The window is whatever queue_slot_count × rotation cadence happens to be, which is a churn-control knob, not a retention policy. Operators who reasonably expect "I can retry recent failures" get a window that silently shrinks as throughput rises.

The mechanism that drops them is the prune TRUNCATE itself. From the prune path:

FOR UPDATE on queue_ring_state and queue_ring_slots[slot], then LOCK TABLE ... ACCESS EXCLUSIVE under a short transaction-local lock_timeout, recheck active rows, then TRUNCATE; ready-backed terminal rows, ready tombstones, and pending terminal-count deltas are reclaimed with their retained ready bodies.

ADR-019 later tightened automatic prune paths to bound child-table lock acquisition, so a contended prune tick returns Blocked when it cannot own the segment promptly. That lock-mode change does not alter the failed-retention floor: carry-forward still happens only after prune owns the child locks and before the TRUNCATE in the same transaction.

A failed row in the truncated slot is reclaimed alongside the rest of the segment. Nothing carries it forward, and nothing tells the operator it is gone.

Decision¶

Queue-storage prune gains a failed-retention floor: a failed terminal row is retryable for at least failed_retention past its finalized_at, regardless of how fast its queue slot rotates. The floor is enforced by carry-forward inside the prune transaction, not by blocking the slot.

Carry-forward¶

QueueStorage::prune_oldest takes a failed_retention: Duration parameter (Duration::ZERO disables the floor and reproduces the old behavior; granularity is whole seconds). Inside the single prune transaction, before the TRUNCATE:

Identify carried rows. In the slot being pruned, select done_entries rows with state = 'failed' AND finalized_at >= now() - failed_retention. These are inside the floor and must survive.
Widen them. ADR-026 made ready-backed failed rows narrow — args, max_attempts, run_at, created_at, unique_key, and unique_states are NULL and hydrate from the retained ready_entries row in the same segment. The carry hydrates those fields from the ready child and produces self-contained wide rows, the same shape as a synthetic (non-ready-backed) terminal row, so the carried fact no longer depends on any ready row.
Re-insert into the live segment. The widened rows are inserted through the {schema}.done_entries parent with ready_slot = current_slot and ready_generation = the current slot's generation, both read in-transaction from the queue_ring_state row already locked FOR UPDATE (which also serializes against rotate — see Guarantees). LIST partitioning routes them to the live done child, which is not in the TRUNCATE list. The PK cannot collide because lane_seq comes from never-resetting global sequences; there is deliberately no ON CONFLICT clause so a violation aborts the transaction loudly rather than silently dropping a terminal fact.
Re-append the terminal-count deltas. The carried rows' positive terminal-count deltas are re-appended under the current slot, mirroring the completion path's delta write exactly (state dimension and job_id counter bucketing included), so exact counts stay consistent across the move.

The carried rows are then excluded from the rollup fold that the prune transaction writes for the truncated slot. The fold splits the truncated terminal rows into two count dimensions: non-failed terminal rows go to pruned_completed_count, and failed rows whose finalized_at < cutoff (older than the floor, genuinely being dropped) go to pruned_failed_count. Carried failed rows are in neither — they remain live in the new slot.

Count split and operator visibility¶

Migration v032 adds pruned_failed_count to {schema}.queue_terminal_rollups. pruned_completed_count keeps its name (additive-only migration policy) and now means "non-failed pruned terminal rows". The sum of the two columns is always the correct total of terminal rows that have left the live ring — see Consequences for the mixed-version honesty argument.

QueueCounts gains a pruned_failed: i64 field, sourced from SUM(queue_terminal_rollups.pruned_failed_count) over the queue. It is cumulative and monotonic: rollups never decrease, so this is a standing, queryable record of how many failed rows have aged past the floor and are no longer retryable. Operators get visibility of the loss that the old design hid.

retry_failed_by_kind and retry_failed_by_queue return a RetryFailedOutcome { retried, matched, pruned_failed_count }. matched is the unique-job scan count before retry, so matched - retried.len() reports jobs that raced or were pruned out from under the retry rather than silently dropping them. pruned_failed_count is Some(cumulative rollup sum for the queue) on the by-queue path and None on the by-kind path, because rollups have no kind dimension and faking a global number there would be dishonest. The CLI surfaces both: it reports the matched/raced split when they differ and, when pruned_failed_count is Some(n > 0), appends that N failed rows have been pruned past retention and are no longer retryable.

Reuse of `failed_retention`¶

The floor reuses the existing failed_retention knob (default 72h) rather than introducing a new one. On the canonical compatibility path failed_retention already bounds how long failed rows stay queryable; in queue storage it now bounds how long they stay retryable before carry-forward stops carrying them. One knob, one intent — failed terminal rows survive for at least failed_retention — implemented differently per engine. MaintenanceService::rotate_queue_storage_queue passes self.failed_retention to prune_oldest; direct callers pass an explicit Duration.

`cancelled` is excluded¶

The floor applies to state = 'failed' only. cancelled rows are always prunable when their segment is. Cancellation is an explicit operator (or application) action that already had its decision moment; re-driving a cancelled job is not the recovery path the floor protects. Holding cancelled rows past their segment would carry retention cost for a class nobody asked to keep retryable.

Guarantees¶

Crash safety¶

Carry-forward, the rollup fold, the TRUNCATE, and the delta re-append all run inside the single existing prune transaction. There is no intermediate committed state where a carried failed row has been truncated from the old slot but not yet re-inserted into the live slot, or where its count delta has been re-appended but the source row not yet truncated. The transaction either commits as a whole — old slot truncated, carried rows wide in the live slot, rollup reflecting only the genuinely-dropped rows, deltas consistent — or rolls back to the pre-prune state. A crash mid-prune leaves the carried rows in the old slot, retryable, and a later prune tick retries the whole operation. No terminal fact is observable in zero or two places.

Rotate-vs-carry serialization¶

The destination of the carry is the current slot. If a rotate advanced current_slot between the carry's read of the destination and the carry's insert, the rows could land in a slot that is itself about to be sealed and pruned — re-creating the cliff one slot over. This cannot happen because both prune and rotate take queue_ring_state FOR UPDATE, and prune holds that lock for the whole transaction:

Prune acquires queue_ring_state FOR UPDATE, reads current_slot / generation, carries the rows into that slot, folds, truncates the old slot, re-appends deltas, and commits — all under the held lock.
A concurrent rotate_ready that wants to advance current_slot must take the same queue_ring_state FOR UPDATE and therefore blocks until the prune transaction commits or aborts.

So the destination slot the carry reads is stable for the life of the prune transaction: a rotate can only advance current_slot after prune commits, by which point the carried rows are already durably in what was the live slot, and that slot is now simply one slot back in the ring with a full retention window ahead of it. The interleaving "carry reads slot N, rotate flips to N+1, prune truncates N+1's predecessor" is unreachable because the rotate cannot proceed while prune holds the lock.

Relationship to ADR-019, ADR-020, ADR-023, and ADR-026¶

Amends ADR-026. ADR-026's "Queue-prune logic must continue treating ready and terminal rows as one retention unit" consequence no longer holds without qualification for failed rows inside the floor. A carried failed row is deliberately decoupled from its retained ready body: the carry widens it into a self-contained terminal row (the synthetic / non-ready-backed shape ADR-026 already defines for unclaimed-cancellation and scheduled-cancellation rows) and re-inserts it into the live segment, leaving the original ready body to be reclaimed with the truncated slot. The "one retention unit" property still holds for ordinary terminal history; the floor is the documented exception, and it is safe precisely because ADR-026 already has a wide synthetic terminal shape that does not depend on a ready row.
Refines ADR-019 the same way ADR-026 does — it touches the queue-storage terminal-family prune path, not the lifecycle or lease/claim planes. Prune's lock discipline (ADR-019 / the lock-order model) is unchanged; carry-forward adds work inside the same locked transaction without taking new locks.
ADR-020 (DLQ) is unaffected. The DLQ is a separate opt-in hold table with its own dlq_retention. The floor is for non-DLQ failed terminal rows — the failures a queue without DLQ leaves in done_entries. A queue that routes terminal failures to the DLQ never produces the failed done_entries rows the floor protects.
ADR-023 (receipt plane) is unaffected. carried_failed_rows is always 0 for the lease and claim rings; the floor lives only on the queue-storage terminal ring.

Consequences¶

Positive¶

Failed terminal rows are retryable for at least failed_retention regardless of queue rotation speed. The retention window is a policy, not a side effect of churn-control sizing.
The loss that does happen — failures aged past the floor — is recorded in pruned_failed_count and surfaced through QueueCounts.pruned_failed, so it is observable instead of silent.
Retry outcomes report matched vs retried and a pruned-past-retention signal, so an operator re-driving failures knows when ids raced or were dropped rather than assuming a clean retry.
No new locks, no new background pass, no second terminal family: the floor is extra work inside the prune transaction that already exists.

Negative¶

pruned_completed_count is misnamed for what it now holds. The additive-only migration policy forbids renaming or retyping a column, so it keeps its name and now means "non-failed pruned terminal rows". Code and SQL reading rollups must read both columns to get the terminal total. (Documented as-is; the rename cost is not worth a breaking migration.)
Pre-v032 mixed-version honesty. A binary that predates v032 has no pruned_failed_count column to write, so it folds all pruned terminal rows — failed and non-failed — into pruned_completed_count. During a rolling upgrade the failed/non-failed split is therefore best-effort: rows pruned by an old binary are counted as completed. The total is never wrong — pruned_completed_count + pruned_failed_count is always the correct count of terminal rows that have left the ring — but pruned_failed_count undercounts for failures an old binary pruned. Once every binary is on v032 the split is exact going forward.
Carrying rows forward keeps failed history in the live ring longer, costing some ring space proportional to failed-row volume × failed_retention. For queues with high failed volume and a long floor this is a real cost; lower failed_retention or route failures to the DLQ if it matters.
Prune does slightly more work per tick when carrying rows: a select, a wide insert, and a delta append on top of the fold and truncate. It stays one transaction and one slot.

Alternatives Considered¶

Block the slot until the youngest failed row ages out¶

Refuse to prune a sealed slot while it holds a failed row inside the floor. Rejected: prune scans the single coldest slot per tick by design, and a slot can carry an in-floor failed row for the full failed_retention window. Blocking it stalls reclamation of everything in that slot — ready bodies, tombstones, completed terminal rows, deltas — behind one young failure, for up to failed_retention. That re-introduces exactly the dead-tuple / unprunable-segment pressure ADR-026 and the #169 work removed. Carry-forward keeps the coldest-slot scan free to reclaim on schedule.

Row-level `DELETE` of the prunable rows¶

Delete just the genuinely-prunable rows from the sealed slot and leave the in-floor failed rows in place. Rejected: it violates the AwaDeadTupleContract PartitionTruncate reclaim pin and ADR-019's contract that these partitions are reclaimed by TRUNCATE, not row vacuum. Row-level deletes turn an append-only / truncate-reclaimed partition into a row-vacuum table, which is the dead-tuple shape the storage redesign exists to avoid.

A separate failed-hold table¶

Carry in-floor failed rows to a dedicated retention table outside the queue ring. Rejected: it adds a second terminal family with its own read path, prune, counts, and retry surface — exactly the cost ADR-026 rejected for the success-receipt-table alternative ("another terminal source for counts, admin reads, retry/replay tools, and prune"). The carry-forward design keeps one durable terminal family by reusing the wide synthetic terminal shape that already exists.

Validation¶

Runtime tests assert that a failed row inside failed_retention survives a prune of its original slot as a wide, self-contained terminal row in the live slot, hydrates correctly, and remains retryable; a failed row past the floor is pruned and folded into pruned_failed_count.
The terminal-count consistency oracle (test_queue_terminal_live_counts_rebuild_restores_invariant) and the terminal-counter trust-marker test cover that carry-forward keeps exact counts consistent: carried rows' deltas re-appended under the new slot, dropped rows' counts folded into rollups.
correctness/storage/AwaShardedPrune.tla models ready-backed (narrow) terminal rows whose done rows come from a ready row in the same segment. Carried failed rows enter the synthetic / wide terminal class, which that model does not represent; correctness/storage/MAPPING.md documents the boundary.

Relationship to Other ADRs¶

Amends ADR-026's one-retention-unit consequence for in-floor failed rows (see above).
Refines ADR-019's queue-storage terminal-family prune path.
Independent of ADR-020 (DLQ) and ADR-023 (receipt plane).