Skip to content

Scoped runtime controls: source audit

Evidence supporting proposed ADR-052. No controls or parameter choices described by the ADR have been implemented or tested by this audit.

The audit is pinned to the commits below. Since then, 0.6.16 added a schema patch registry to the 0.6 migration runner and shipped the maintenance.prune control as its prune_controls patch; 0.6.17 moved promotion and rescue into their own maintenance loop (ADR-051); and main gained migrations v053 to v059. Line references describe the pinned commits.

A check of a plain control table inside claim_ready_runtime is a plausible enforcement point for queue-storage runtime claims, but is not a complete dispatch halt. Canonical dispatch bypasses that function on both audited branches; main also has callable per-slot claim functions. A database read can avoid a client polling interval, but does not establish a barrier against claims already in progress or handler launches after the halt commits.

This source audit pins main to d16ef5c4572cb96b25a8f26ad85bd5915ab76f64 and release/0.6.x to 8cec9e45a45735c300a89c98dc96573589100386. It inspects source, not a running halt implementation or a mixed-version rehearsal. The prospective patch examples below are separate commits, not evidence that the pinned release contains their behavior.

Claim routes

  • Canonical storage bypasses the proposed check. The dispatcher embeds a WITH claimed … UPDATE awa.jobs_hot SET state = 'running' query. Its predicate checks queue pause, with no call to claim_ready_runtime. This exists on main, awa-worker/src/dispatcher.rs:745–795 and 0.6, lines 563–612. Main explicitly selects canonical storage for canonical-drain runtimes and can fall back to it when auto-finalization does not succeed: client.rs:1529–1605. Guarding only the queue-storage function leaves these workers outside the halt boundary, including old binaries executing the embedded query.
  • The audited queue-storage runtime APIs converge on the SQL wrapper. claim_batch, claim_runtime_batch_with_aging, its instance-gated wrapper, and claim_job_batch reach claim_ready_rows_tx, which issues FROM {schema}.claim_ready_runtime($1, $2, $3, $4). See main, queue_storage.rs:5261–5309, 7161–7306, 7731–7828, and 0.6, 3575–3623, 5172–5310. No separate Rust-generated new compact-claim INSERT was found in these pinned runtime paths. The Rust INSERT into lease_claims during schema preparation copies existing legacy claims; it is not new dispatch: main, 4339–4357.
  • Receipt mode changes the write target, not the runtime entry point. On 0.6, receipts use compact lease_claim_batches for zero deadlines and row-based lease_claims for positive deadlines; legacy mode writes leases: v023:2684–2689. Main uses compact batches for both receipt deadline modes and retains legacy leases: v023:3282–3294. A guard must precede every claim-producing arm, not only a leases write.
  • Main has a lower SQL entry point. The wrapper dynamically invokes claim_ready_slot_N: v023:4155–4175. Those functions independently select ready rows and execute the claim CTE: 4241–4254, 4291–4357. A permitted direct slot caller bypasses a wrapper-only check. Source inspection found no such Rust runtime caller at the pinned commits; permissions and external SQL callers remain part of the enforcement proof.

What the database check can guarantee

A fresh per-claim read can reject claims without waiting for a runtime control polling loop. Describe this as no additional client polling interval on covered claim paths, not instantaneous fleet cessation. PostgreSQL Read Committed reads committed data at query snapshot time; Repeatable Read retains its transaction snapshot. A claim that read permission before a concurrent halt committed can still write and commit afterward. A plain read of the control row creates no serialization barrier with the halt writer. Serializable ordering alone also does not require overlapping claims to be ordered after the halt. These are consequences of PostgreSQL transaction isolation, not behavior verified for an implemented halt.

The current SQL functions default to VOLATILE; PostgreSQL documents fresh snapshots for queries inside volatile functions. An entry check followed by later SQL therefore still leaves an inter-statement race; moving the check into a claim statement still leaves claims whose statement snapshot predates the halt. Prepared statements do not themselves freeze table values, but a wrongly declared IMMUTABLE control helper can cache stale results: function volatility. On main, ADR-052 requires awa-owned claim transactions to begin explicitly at Read Committed, including when database/role defaults differ. Only externally owned transactions at other isolation levels are rejected. Overlapping evaluations remain outstanding work until settled.

Main currently begins queue-storage claims with BEGIN; SET LOCAL synchronous_commit = off, which does not override isolation; canonical claims run directly through a pool query. There are concrete transaction-creation seams to change before any data/control query, while leaving caller-owned enqueue transactions alone. See the begin constant, physical claim transaction, and canonical query. PostgreSQL permits setting transaction isolation before its first query; it must not be changed late or left to a persistent session setting.

A control-table read adds an ACCESS SHARE relation lock. Fast-path eligibility is not a guarantee that every such lock uses that path; inspect pg_locks.fastpath in the claim-overhead rehearsal. The proposed table is ordinary, unpartitioned and never truncated, so this relation lock is outside reclamation's partition-lock set. See table-lock modes and pg_locks. This does not eliminate ordinary migration/catalog locking or its performance cost.

Claims commit before the runtime returns jobs for execution, and dispatch later spawns handlers: main, queue_storage.rs:7327–7356, dispatcher.rs:929–946. A database claim guard alone cannot prevent a previously claimed job from launching after halt, cancel an executing handler, or stop maintenance and cron. Replacing a compatible SQL function can constrain older binaries that still call it; it does not constrain their other SQL paths or add runtime control observation.

Custom schemas and 0.6 delivery

Every installed queue-storage schema owns a wrapper and its claim representation. Main's v051 refresh discovers trusted substrates, preserves their slot counts and receipt mode, then reinstalls them: v051:25–91. The discovery bootstrap rejects unsafe configured schemas and warns about skipped untrusted schemas: queue_storage_schema_discovery.sql:202–243. Updating only awa.claim_ready_runtime cannot establish coverage for custom schemas. New installs, later prepare_schema calls, migrations, and restores must preserve the guard; preparation invokes the installer again: queue_storage.rs:4243–4257.

The pinned 0.6 release has versioned migrations through v040 and skips already-applied versions; it has no schema-patch registry in that migration runner: migrations.rs:172–187, 281–300. Editing an old migration alone would not update an existing v040 database.

Prospective commit b5016b7d, migrations.rs:233–249 provides versionless, probed schema patches because 0.7 owns v041 onward. Its definer_lane_provisioning patch composes updated installer definitions with a custom-schema sweep: 410–423. It preserves old helper arguments and cursor semantics, exports repeatable SQL, and needs ownership-capable migration credentials: patch:21–27. Its function-only lock profile does not transfer to a halt patch that creates a table or refreshes the whole substrate. Main explicitly warns that substrate reinstallation can hold table locks through commit: v051:17–22.

Separate prospective commit 54f0bc6a, migrations.rs:233–253 adds an awa.schema_patches ledger, with probe backfill and no application on newer schemas. It is a delivery example, not a merged prerequisite or halt implementation. ADR-052 limits 0.6 to Phase 1: create the standalone control table and operator functions additively, then gate compatible maintenance runtimes. Do not refresh claim functions, reinstall substrates or change claim isolation there. Full dispatch fencing and its substrate/return-shape/rollout proofs are main-only work. The Phase 1 patch still requires privilege, replay and later-main preservation checks.

Maintenance observation and reclamation

The leader holds a dedicated checked-out pool connection for its session advisory lock, while maintenance operations acquire other pool connections. The existing liveness SELECT 1 runs every 30 seconds in the same tokio::select! loop as maintenance. Each selected branch awaits its work inline, so a blocked branch delays that check and cancellation. Initial cron synchronization also precedes the select loop and launches a separate cron evaluator. See main maintenance.rs:1002–1133, 1254–1299, and 0.6:893–1065.

Replacing that inline query with a controls read alone cannot bound observation latency. ADR-052 instead proposes independently scheduled ownership of the same leader connection, serializing reads and advisory-lock operations. It adds no observer connection, but requires restructuring task ownership. Before admitting initial cron or maintenance work, the observer must have read the relevant controls. No elapsed-time observation bound proves that an already admitted database operation has stopped.

Queue, lease and claim timers each combine rotation with pruning. A maintenance.prune gate must cover both stages, not just calls named prune_oldest*. Terminal-count rollup independently acquires ACCESS EXCLUSIVE and truncates delta children, so it belongs behind the same gate. Main's rollup wrapper also folds prune deltas, trims rotation ledgers and considers automatic storage-authority changes. See maintenance.rs:1206–1252, 1411–1476, queue_storage.rs:18607–18635, and 0.6:13924–13960. PostgreSQL documents that TRUNCATE acquires an exclusive table lock.

Both branches default prune lock timeout to 10 ms. This limits a particular lock wait, not the whole maintenance operation: preliminary singleton/slot row locks and prechecks can occur before the timeout is installed. The 0.6 setter is hidden and has no worker-builder, CLI or Python wiring at the audited baseline. Existing blocked/active backoff does not pace successful reclamation after resume. See main default, 0.6 setter, prechecks and later lock timeout, and 0.6 backoff.

There are 17 leader select arms: 15 work timers, cancellation and leader-connection liveness, plus initial cron sync and the independent evaluator. Cleanup includes completed jobs, DLQ rows, batch operations, runtime snapshots and descriptors; other branches refresh queue health, dirty keys, metadata and priority aging or execute administrative batches. ADR-052's single branch map explicitly assigns each operation and requires a closed enum with an exhaustive mapping. There is no aggregate-rule fallback. See main maintenance.rs:1136–1267.

Dirty-key recomputation updates dashboard counters/catalogs, not ready cursors or execution eligibility. Its assignment to non-recovery housekeeping is a design choice supported by the canonical dirty-mark recompute/refresh SQL. Priority aging, by contrast, changes waiting-job priority and belongs with promotion. Queue-health sampling is explicitly housekeeping, not exempt control reporting. Administrative batch execution belongs to dispatch; cleanup of its expired records belongs to prune.

The proposed reclamation budget applies permanently, not only after resume, and has no dependency on the control schema. It caps transactions that can request relation modes SHARE, SHARE ROW EXCLUSIVE, EXCLUSIVE or ACCESS EXCLUSIVE, not ordinary ROW EXCLUSIVE writes. One unfinished capped operation is allowed per leader across rings/substrates, followed by at least one monotonic second before the next admission. Each nested transaction needs a permit, not just its enclosing timer callback. An individual transaction can request multiple locks; the cap does not count lock objects. Uncertain server-side SQL cannot free the slot just because its local future was dropped. These properties require rehearsal; the audit is not evidence that the cap exists today.

Rotation and pruning share a scope but differ in cap eligibility. At the audited baselines, rotation uses advisory/singleton row serialization and cursor/ledger DML, while queue/lease/claim pruning takes ACCESS EXCLUSIVE. Terminal-delta rollup also does so per slot transaction. Prune-delta and rotation-ledger folds use row DML; authority flip uses row locks and DML. See rotation serializers/cursor operations, queue pruning, lease pruning, claim pruning, folds, and authority flip.

The audited 0.6 canonical refresh_admin_metadata is another capped path: it truncates dirty tables at v006:433 and 523. Main's v046 replaces those with DELETE; dirty-key recomputation itself uses DML on both baselines. The standalone terminal-counter rebuild API also takes a strong lock but has no maintenance-loop caller in this audit. The leader cap does not claim to pace explicit operator/migrator DDL; adding such an API to automatic work requires a classified, capped entry.

Observation and authority

Runtime snapshots default to 10 seconds and run on an independent reporter with delayed missed ticks. That reporter also publishes descriptors and cron reconciliation evidence, so continued control reporting during a halt must not silently continue halted application work. Snapshot age is not control-observation age; publish actual observed revisions and monotonic ages, and distinguish draining, cancellation requested, quiescent and unknown. See default interval, cron reconciliation, descriptor publication, and reporter loop.

The proposed cancellation contract adds a durable participant registry, not a new claim-path write per job. Registration must commit before a runtime admits work and serialize with control changes; the cancel transaction must record pending owner evidence before signaling cancellation. A compatible implementation can use the rare startup/control-change path for this interlock rather than lock a shared control row on every claim. Old unregistered runtimes require an enforced boundary before fleet coverage can be claimed. Missing snapshots, failed observation and expired heartbeats do not delete pending evidence. No such interlock or resolution implementation has been validated here.

Drain-mode resume has no reconciliation dependency and does not wait for missing participants; observed/drained reporting is separate. Cancellation creates the durable obligation. An unresolved cancel cannot be downgraded to drain to evade it, and a superseding cancel retains its pending evidence. The snapshot UPSERT itself also has cron-protocol safety triggers: preserve their evidence invalidation while separating liveness from new cron declarations and descriptor publication.

Control-specific privileges are a proposed addition. Current documented roles include a broad combined awa_runtime principal; an operator-only command must revoke direct control-table writes from that role as well as producers, remove default PUBLIC EXECUTE, constrain the definer owner's privileges and fix its search path. ADR-043 proposes the capability direction; PostgreSQL's definer-function guidance explains why merely marking a function SECURITY DEFINER is insufficient.

Phase 1

The independently shippable subset and its acceptance criteria are defined in ADR-052.

Proof still needed

  • Enumerate supported binary versions and actual claim SQL, including canonical drains, fallback selection, direct slot calls, receipt modes, custom schemas, and external callers. Exercise released old binaries against the patched database.
  • On main, test concurrent halt/claim/launch under Read Committed, including blocked statements, delayed commits, rollbacks, prepared statements and multiple physical queue stripes. With Repeatable Read or Serializable database/role defaults, awa-owned claim transactions must explicitly choose Read Committed; incompatible external transactions must be refused without altering caller isolation. Include in-flight claims and SQL before reporting drained.
  • Verify least-privilege access to the control table, operator-only changes, producer access, and resistance to guard loss through schema preparation, upgrade, restore or a skipped custom substrate.
  • Rehearse Phase 1's additive patch and compatible maintenance runtime on 0.6 and main, including patch-first/binary-first ordering, replay, partial failures and later migration. On 0.6, verify claims and substrate installers remain unchanged. Full claim fencing and its live-load DDL proof belong only to main.
  • Hold a maintenance SQL operation or exhaust the worker pool while observing controls on the already-held leader session. Exercise startup, disconnect, leadership handoff, missing/malformed rows and monotonic freshness expiry. Confirm no replacement observer connection is required per poll.
  • Check every entry point and reporter side effect against the exhaustive enum/table. Fail stale prune/cron permission closed while promotion and rescue keep their last observed state on a held leader session. Test actual session loss separately from query timeout. Measure claim read overhead and maintenance availability.
  • Exercise the permanent cap in normal operation, budget-only mode, startup, leadership changes, backlog and resume across all ring families/substrates. Drain-mode resume must succeed without participant/reconciliation evidence. Cancel-mode uncertainty must survive snapshot expiry and prevent cancellation downgrade or resume until positively reconciled or fenced.