ADR-023 Receipt-Ring Validation Artifact (2026-04-26)

This file records the exact commands, configuration, and raw results backing the validation numbers in ADR-023. The run exercises the post-Phase-6 system: lease_claim_receipts defaulted on, the legacy open_receipt_claims table deleted, and lease_claims / lease_claim_closures partitioned by claim_slot with the claim ring rotating in lockstep with the lease ring.

Note on paths. This validation artefact pre-dates the extraction of the long-horizon harness into its own repository. References to benchmarks/portable/... below describe the in-tree layout at the time of the run; the harness now lives at hardbyte/postgresql-job-queue-benchmarking. The data files referenced by run-id (custom-20260426...) are retained on the host that produced the run; they are not in the new public repo.

Run identity

• Run id: custom-20260426T032000Z-839ddc
• Branch: feature/vacuum-aware-storage-redesign at 7b688e4
• Started: 2026-04-26 05:15:45Z (15:15 NZST), finished ~17:14 NZST
• Duration: 115 min (5m warmup + 25m clean + 30m idle-in-tx + 30m recovery + 25m clean)

Host

• 24-core x86_64, 91 GiB RAM
• Linux 6.18.21, glibc 2.42
• Postgres 17.2-alpine in compose
• max_connections = 400 (raised from default 200 — see benchmarks/portable/postgres.conf for the reasoning; smaller ceilings caused pool exhaustion on the 4×8 profile)
• shared_buffers = 256MB, autovacuum = on, autovacuum_naptime = 1min, autovacuum_vacuum_scale_factor = 0.2

CLI

uv run python benchmarks/portable/long_horizon.py \
  --systems awa \
  --replicas 4 --worker-count 8 --producer-rate 200 \
  --phase warmup=warmup:5m \
  --phase clean_1=clean:25m \
  --phase idle_1=idle-in-tx:30m \
  --phase recovery_1=recovery:30m \
  --phase clean_2=clean:25m \
  --skip-build

Per-replica producer rate is 200/s, so aggregate offered load is 800/s.

Headline result

Receipt-plane peak dead tuples stayed at ≤49 across all phases — including a 30-min idle-in-tx phase that holds open transactions to block autovacuum. The lease_claim_closures_<0..7> partitions stayed at exactly 0 dead tuples in every phase, confirming the append-only-then-truncate property the partitioning was designed to preserve.

open_receipt_claims is absent from the schema and from manifest.adapters.awa.event_tables. The receipt control table no longer exists.

Throughput by phase

Aggregate completion / enqueue rates summed across the 4 replicas (median over the 5s sample window of each phase):

phase	type	aggregate completion/s	aggregate enqueue/s	p99 e2e median (ms)	p99 e2e peak (ms)
clean_1	clean	769.8	791.9	92.0	244.1
idle_1	idle-in-tx	359.2	640.5	n/a (no clean drain)	67 076
recovery_1	recovery	744.3	752.0	n/a	n/a
clean_2	clean	614.0	736.0	n/a	n/a

The e2e p99 spike of 67 s during idle_1 is the expected blocked-vacuum latency: held-open transactions prevent claim-ring partitions from being truncated and the steady-state pipeline backs up. The recovery and following clean phases drained without losing any claims (see receipt plane numbers below).

Receipt plane — peak n_dead_tup per partition per phase

table	clean_1	idle_1	recovery_1	clean_2
awa_exp.lease_claims_0	7	8	1	6
awa_exp.lease_claims_1	0	0	3	2
awa_exp.lease_claims_2	3	0	0	8
awa_exp.lease_claims_3	0	0	8	1
awa_exp.lease_claims_4	4	0	0	8
awa_exp.lease_claims_5	2	0	8	8
awa_exp.lease_claims_6	7	2	8	11
awa_exp.lease_claims_7	8	0	0	5
awa_exp.lease_claim_closures_0..7	0	0	0	0
receipt-plane total	31	10	28	49

Closure partitions are append-only between rotations; their truncate runs under ACCESS EXCLUSIVE on a partition that the rotate path has already moved off. Zero-dead-tuple steady state is the designed shape.

Receipt plane — peak total_relation_size_mb

Largest single partition during clean_2 was lease_claims_1 at 1.26 MB. Aggregate across all 16 receipt-plane partitions stayed < 12 MB through the entire run.

Warm tables — peak n_dead_tup per phase

table	clean_1	idle_1	recovery_1	clean_2
awa_exp.attempt_state	0	0	0	0
awa_exp.queue_lanes	0	0	0	0
awa_exp.queue_claim_heads	0	0	0	0
awa_exp.queue_enqueue_heads	0	0	0	0
awa_exp.queue_ring_state	76	665	40	40
awa_exp.queue_ring_slots	95	665	158	0
awa_exp.lease_ring_state	103	28 503	232	104
awa_exp.lease_ring_slots	0	0	0	0
awa_exp.claim_ring_state	78	1 631	84	76
awa_exp.claim_ring_slots	0	0	0	0

The *_ring_state tables are singletons (one row each); their dead tuples are HOT-update churn that recovers within one autovacuum pass once held-open transactions release. The 28 503 spike on lease_ring_state during idle_1 collapses back to 232 in recovery_1 and 104 by clean_2. This matches the architectural contract in correctness/storage/AwaDeadTupleContract.tla: ring-state rows are Warm (bounded by row count, churn-prone but reclaim-fast).

ADR-019 baseline comparison

ADR-019 (docs/adr/bench/019-queue-storage-validation-2026-04-19.md) is dominated by short throughput tests against a single-replica DB. The closest comparable number is the mixed-workload soak's exact_dead_total = 276 after a 10s, 1000 jobs/s drain. The Phase 6 multi-replica long-horizon equivalent is the 49 receipt-plane peak in clean_2 — a different shape (longer, multi-replica, includes an idle-in-tx phase), but the order-of-magnitude reduction confirms the prediction in ADR-023: by removing the open_receipt_claims row from the receipt path, receipt-plane MVCC churn drops from "low hundreds" to "low tens" and is fully partition-truncate-reclaimed instead of autovacuum-reclaimed.

Validation status

• All four TLA+ specs in correctness/storage/ pass under TLC (AwaSegmentedStorage, AwaSegmentedStorageRaces, AwaStorageLockOrder, AwaSegmentedStorageTrace, AwaDeadTupleContract). TLC is wired into .github/workflows/nightly-chaos.yml.
• cargo test --workspace green at 7b688e4.
• 4 receipt-plane chaos scenarios in awa/tests/receipt_plane_chaos_test.rs pass (#[ignore] group, exercised in nightly-chaos.yml).
• open_receipt_claims is absent from prepare_schema() and from the live schema after install (test test_open_receipt_claims_is_absent_after_install).

Known limitations / follow-ups

Residual ~2.4 GB/h/replica RSS growth (resolved 2026-04-27)

The 115-min run saw per-replica RSS climb from ~zero at start to 3.9– 4.6 GB at T+95m. Initial investigation traced this to glibc malloc arena fragmentation and applied MALLOC_ARENA_MAX=2 in bench_harness/adapters.py:_base_env. That capped arena count (1-2 distinct arenas) but a 12h validation attempt at T+1h still showed 2.7-3.6 GB/replica (~3.2 GB/h/replica), with the 1-2 capped arenas growing via repeated 64 MB chunk allocations.

A jemalloc heap profile (and a parallel code audit) traced 99.1% of allocations to tokio::runtime::task::core::Cell::new from awa-worker/src/dispatcher.rs:662: the Dispatcher::drain_ready path spawns every job-runner future into a shared Arc<Mutex<JoinSet<()>>>, but JoinSet::join_next() was only called during shutdown drain. Completed JoinHandles — and the task Cell each one keeps alive (which captures the entire execute_task closure: cloned Arcs of pool/storage/metrics, the job payload, the dispatch permit) — accumulated indefinitely under steady-state load.

Fixed in commit c0df1df by reaping completed handles via set.try_join_next() before each spawn batch. Validated end-to-end with the 12h overnight run below; per-replica RSS held at 17-18 MB for the entire 11.6h (vs 3.6 GB at T+1h pre-fix).

Pool sizing

The pre-fix max_connections = 200 ceiling caused pool-timeout errors on the 4×8 profile because each replica's bench pool defaults to 80 connections (worker_count × 4 + 48), and 4 × 80 = 320 exceeds the ceiling. The new defaults — max_connections = 400 in benchmarks/portable/postgres.conf and per-replica pool size in benchmarks/portable/awa-bench/src/long_horizon.rs — are pinned and captured in the manifest for cross-run comparability.

12-hour overnight run (2026-04-27, after JoinSet leak fix)

Run id: custom-20260426T101003Z-ba9cb4. Started 2026-04-26 22:10 NZST, completed 2026-04-27 09:46 NZST (11.6 h wall time). Same branch + commit c0df1df (JoinSet drain fix). Profile:

--phase warmup=warmup:5m
--phase clean_1=clean:4h
--phase idle_1=idle-in-tx:30m
--phase recovery_1=recovery:30m
--phase clean_2=clean:4h
--phase idle_2=idle-in-tx:30m
--phase recovery_2=recovery:30m
--phase clean_3=clean:90m

Receipt plane — 7-phase summary

phase	type	claims peak dead	closures peak dead	claims peak MB	closures peak MB
clean_1	clean	85	0	3.58	3.37
idle_1	idle-in-tx	38	0	4.35	3.19
recovery_1	recovery	42	0	6.50	4.55
clean_2	clean	68	0	57.57	36.45
idle_2	idle-in-tx	33	0	59.67	37.56
recovery_2	recovery	37	0	101.63	63.73
clean_3	clean	42	0	98.18	61.55

Closure partitions stayed at 0 dead tuples through every phase across two full idle-in-tx stress cycles — the architectural property ADR-023 set out to deliver. Receipt-plane peak dead-tuple count over the entire 11.6 h was ≤85.

Warm singletons — peak n_dead_tup per phase

table	clean_1	idle_1	recovery_1	clean_2	idle_2	recovery_2	clean_3
awa_exp.queue_ring_state	78	668	668	44	44	44	44
awa_exp.lease_ring_state	108	28 541	28 584	148	23 563	237	2 505
awa_exp.claim_ring_state	78	1 635	1 635	79	186	77	76
awa_exp.attempt_state	0	0	0	0	0	0	0
awa_exp.queue_lanes	0	0	0	0	0	0	0
awa_exp.queue_claim_heads	0	0	0	0	0	0	0

Ring-state row counts spike under blocked vacuum and recover within an autovacuum pass once held-open transactions release — exactly the Warm reclaim shape in the dead-tuple contract spec.

Per-replica RSS

Hourly samples taken from /proc/<pid>/status:

sample	replica 0	replica 1	replica 2	replica 3
T+1h	18 080 kB	17 240 kB	17 176 kB	17 308 kB
T+5h	18 284 kB	17 376 kB	17 232 kB	17 396 kB
T+9h (idle_2)	18 404 kB	17 428 kB	17 260 kB	17 396 kB
T+11h	18 436 kB	17 448 kB	17 260 kB	17 396 kB

Net per-replica drift over 11 h: < 400 kB. Total allocator-arena mappings stayed at 16-18 anonymous regions, ~16 MB total per replica. Compare pre-fix: 2.7-3.6 GB at T+1h, projecting to ~150 GB at T+12h.

Throughput regression — out of scope for ADR-023

phase	aggregate completion/s	aggregate enqueue/s	total backlog peak
clean_1	759	776	95
idle_1	365	645	354 878
recovery_1	747	746	360 925
clean_2	344	704	4 956 477
idle_2	167	596	5 732 775
recovery_2	221	707	6 591 795
clean_3	200	700	9 229 258

After idle_1 leaves a ~360k-job backlog, recovery_1's drain rate just matches the steady enqueue rate — it does not actually drain the backlog. From clean_2 onward completion rate falls to 200-350/s while enqueue holds at ~700/s, so the backlog grows monotonically to 9.2 M jobs / 12 GB ready_entries by the end. This is not an ADR-023 issue — the receipt plane stayed clean throughout (closures stayed at 0 dead tuples; claim partitions reclaimed each rotation). The slowdown is a separate scaling characteristic exposed by holding a ~Million-row ready_entries accumulation, tracked separately.

Three 30-minute regression-confirmation runs (2026-04-27)

After the 12 h validation surfaced the post-idle throughput regression fixed in c0df1df / ab99a31 / d21e5db, three short bench runs were taken at the same 4×8×200/s topology to confirm the regression stays gone under different pressure shapes. All three runs stream metrics through the OTel side-stack at docker/observability/ (commit 7905c74), so the dashboard is the canonical operator view and these tables are the textual record.

Run 1 — clean baseline

custom-20260427T043422Z-ab7901. 4×8×200/s, warmup 2m + clean 28m. Headline: completion 783/s vs enqueue 792/s, claim peak 25, closures 0. Steady-state shape; no surprises.

Run 2 — single-cycle idle-in-tx stress (the original regression scenario)

custom-20260427T051026Z-5bc479. warmup 2m + clean_pre 8m + idle-in-tx 5m + clean_post 15m.

phase	comp/s	enq/s	claim peak	closure peak
clean_pre	787	800	16	0
idle_1	779	781	3	0
clean_post	784	792	36	0

The original 12 h regression had clean_pre 759 → idle_1 365 → clean_post 344. This run is the post-fix shape: clean_post within 3/s of clean_pre, and within 8/s of enqueue. The post-idle drop is gone.

Run 3 — two-cycle stress (most informative)

custom-20260427T054607Z-6a8f52. warmup 2m + clean_1 5m + idle_1 5m + recovery_1 5m + clean_2 5m + idle_2 3m + recovery_2 3m + clean_3 2m. Two full stress cycles in 30 minutes.

phase	comp/s	enq/s	gap	claim peak	closure peak
clean_1	783	800	17	15	0
idle_1	755	757	3	15	0
recovery_1	798	800	2	1	0
clean_2	790	792	2	21	0
idle_2	773	785	12	3	0
recovery_2	773	787	14	8	0
clean_3	774	788	14	13	0

Receipt-plane closure peak: 0 across every one of the seven phases, including two full idle-in-tx stress cycles. Claim peak ≤21 across the entire run. Throughput keeps within 2% of the producer rate in all phases — the post-idle regression does not re-emerge across repeated stress cycles.

Warm singleton (lease_ring_state) dead-tuple peaks per phase, for the dead-tuple contract spec's Warm reclaim shape:

phase	lease_ring_state	queue_ring_state	claim_ring_state
clean_1	101	68	69
idle_1	5 909	296	296
recovery_1	6 010	75	301
clean_2	93	72	79
idle_2	3 510	176	175
recovery_2	3 608	181	181
clean_3	93	65	66

Spikes under held-tx, full recovery within one clean phase. Exactly the Warm/bounded-by-row-count shape declared in correctness/storage/AwaDeadTupleContract.tla.

Files

• 115-min run: benchmarks/portable/results/custom-20260426T032000Z-839ddc/
• 12-hour run: benchmarks/portable/results/custom-20260426T101003Z-ba9cb4/
• Run 1 (clean baseline): benchmarks/portable/results/custom-20260427T043422Z-ab7901/
• Run 2 (single-cycle stress): benchmarks/portable/results/custom-20260427T051026Z-5bc479/
• Run 3 (two-cycle stress): benchmarks/portable/results/custom-20260427T054607Z-6a8f52/
• All include summary.json, manifest.json, raw.csv, plots, html

Second 12-hour overnight run (2026-04-27 → 2026-04-28)

Run id: custom-20260427T081444Z-2471c6. Started 2026-04-27 20:14 NZST, completed 2026-04-28 07:56 NZST (11h 35m wall). Branch tip a45dca6 (post-doc-consolidation). Same 7-phase profile as the prior 12h run (4 replicas × 8 workers × 200/s producer = 800/s aggregate; warmup:5m → clean_1:4h → idle_1:30m → recovery_1:30m → clean_2:4h → idle_2:30m → recovery_2:30m → clean_3:90m).

Headline

Receipt-plane invariant held across the entire 11.6 h including two full idle-in-tx stress cycles:

• lease_claim_closures_0..7 peak n_dead_tup = 0 in every phase.
• lease_claims_0..7 peak n_dead_tup = 24 across the run (well under the 200 threshold).

Per-replica RSS: held at 17184–17552 kB across 5 sampled checkpoints spanning 8 hours of the run. Net drift ~150 kB. The JoinSet leak fix from c0df1df continues to hold.

Run log: 0 ERROR/FATAL/panic. 474 sqlx WARN slow-statement events, all on the queue_counts query during high-backlog phases — see "queue_counts perf characterisation" below.

Receipt plane — 8-phase summary

Peak n_dead_tup per partition family per phase. Dashes mean the metric was not sampled in that phase (no events fell within the sample window).

phase	claims peak	closures peak
warmup	0	0
clean_1	16	0
idle_1	14	0
recovery_1	8	0
clean_2	9	0
idle_2	4	0
recovery_2	8	0
clean_3	0	0

Warm singletons — peak n_dead_tup per phase

table	clean_1	idle_1	recovery_1	clean_2	idle_2	recovery_2	clean_3
lease_ring_state	105	28 563	(≤ 132)	132	26 686	(≤ 88)	88
claim_ring_state	78	1 616	(≤ 79)	79	323	(≤ 76)	76
queue_ring_state	78	633	(≤ 9)	9	9	(≤ 9)	9

Same Warm/bounded-by-row-count reclaim shape as the first 12h run. Spike under held-tx, full reclaim within one clean phase.

Throughput by phase

Per-replica completion / enqueue rates (median over the 5s sample window). Multiply by 4 for aggregate.

phase	completion/s	enqueue/s
warmup	198.5	198.5
clean_1	190.2	190.2
idle_1	118.1	164.9
recovery_1	184.5	183.4
clean_2	95.7	174.0
idle_2	42.8	158.3
recovery_2	52.0	175.4
clean_3	31.5	175.8

Same post-idle backlog-growth shape as the first 12h run. Receipt plane stayed clean throughout, so the slowdown is not an ADR-023 property — it is the queue_counts hot-path described next.

queue_counts perf characterisation (targeted bench, off-pool)

Driven by a fresh-postgres microbench (artifacts/queue_counts_bench/run.sh) that seeds N rows into a single ready_entries partition for one queue and runs EXPLAIN (ANALYZE, BUFFERS) on the exact-count CTE used by QueueStorage::queue_counts_exact. Three repeats per size; the run isolates the count from concurrent enqueue/claim traffic.

backlog rows	execution_ms (median)	total_ms (median)
100 000	14.1	29.6
1 000 000	116.1	134.7
5 000 000	460.2	475.1
10 000 000	845.9	860.9

Linear in the backlog. The cost is dominated by a single sub-CTE, lane_counts, which scans every row in ready_entries and joins against queue_claim_heads to filter by lane_seq >= claim_seq. At 10 M rows the EXPLAIN plan reports 877 ms in Finalize Aggregate over a Parallel Seq Scan + Hash Join across ready_entries_0 partition, with the parallel workers each reading ~55 k buffer pages (3 × 8 GB residency across the partition heap).

Every other sub-CTE finishes in under a millisecond:

sub-CTE	actual time (10 M backlog)
lane_counts	877 ms (full ready_entries scan)
live_running (leases)	0.04 ms (8 partitions, all empty)
live_running (lease_claims anti-join)	0.04 ms (claim ring TRUNCATE'd hot)
live_terminal (done_entries)	0.08 ms
pruned_terminal (queue_lanes ⨝ rollups)	0.13 ms

The dispatcher caches queue_counts for CLAIMER_QUEUE_COUNTS_MAX_AGE = 250 ms (awa-worker/src/dispatcher.rs:20). Once the backlog crosses ~3 M rows the exact query takes longer than the cache window, so every replica's claim batch ends up waiting on a fresh exact count. Across 4 replicas this saturates one connection per replica with the count query alone, which is why throughput collapses from ~750/s aggregate in clean_1 to ~125/s in clean_3.

Fix direction (not landed in this branch)

The receipt-plane numbers tell us this is not an ADR-023 issue; the partitioned receipt tables are doing exactly what they were designed to do. The fix space is:

1. Replace the ready_entries scan with a counter read. queue_lanes.available_count already tracks per-(queue, priority) available entries and is updated transactionally by enqueue/claim/rotate. Summing that column over the matching logical-queue physical stripes is O(few rows) regardless of backlog size and gives the same semantic answer as the current row-count scan.
2. Lift queue_counts out of the dispatcher claim hot path. The dispatcher only uses queue_counts to size target_claimers. That is a slow-moving control-loop input — a 1–5 second cache window would be safe and would absorb the worst-case query cost without visible impact. Resolution: both directions landed before the 0.6 cut. The queue_counts_exact lane_counts CTE now reads sum(queue_lanes.available_count); the dispatcher's claimer-target loop uses an AvailableSignal from the same lane-head counter; the legacy queue_counts_cached method, the queue_count_snapshots cache table, and the dispatcher's CLAIMER_QUEUE_COUNTS_MAX_AGE constant are gone. The receipt-plane validation in this artifact holds independently of those throughput-side changes.

Files

• Long-horizon results bundle: benchmarks/portable/results/custom-20260427T081444Z-2471c6/ (summary.json, manifest.json, raw.csv, index.html, plots/). raw.csv is 529 MB and is gitignored — the bundle is local-only archive evidence; this artifact captures the derived numbers.
• Overnight check log: artifacts/overnight/checks.log (5 sampled health checkpoints from the autonomous cron).
• queue*counts microbench: artifacts/queue_counts_bench/ (run.sh, timings.csv, four explain*<N>\_<r>.txt per repeat).