Skip to content

ADR-038: Hot-reloadable per-queue runtime overrides

Status

Accepted — Tier 1 implemented for 0.7 (#396, from #385); Tier 2 tracked in #397.

Context

Worker runtime parameters are read from the builder at startup, so tuning a live fleet (poll cadence, batch size, rate limits, attempt deadlines) requires a rolling restart. Graceful drain makes restarts safe but slow, and online tuning during an incident — the moment it matters most — is exactly when restarts are least welcome (#385).

awa already has one hot-reloadable control: queue_meta.paused, polled by dispatchers and applied live. The database is already the fleet's control plane.

Decision

Extend awa.queue_meta with nullable override columns (migration v041): override_poll_interval_ms, override_claim_batch_size, override_rate_limit (jobs/second), override_deadline_ms, plus overrides_updated_at.

Semantics:

  • The builder value is the default; a non-NULL override wins; NULL (or clearing) reverts to the builder value. Whole-state writes: set replaces the full override row.
  • Dispatchers refresh overrides on a slow cadence (10s) — one single-row read per claimer per cadence, not a per-poll control query — and apply them in the dispatch loop. A change is fleet-consistent within roughly one cadence.
  • Mutation surfaces: awa_model::admin::{set,clear,queue}_runtime_overrides and awa queue overrides set|clear|show.

Two guard rails, both logged when they bite:

  1. Rate-limit overrides only retune an existing limiter. The shared token bucket is created at build time only when the builder configured a rate limit; enabling limiting on an unlimited queue live (or disabling it) changes the hot claim path and is Tier 2.
  2. Deadline overrides never cross the zero boundary. The receipts-vs-legacy claim mode is selected by zero/non-zero deadline_duration at startup; an override that would flip the mode is ignored with a warning. Retuning a non-zero deadline is fully supported and applies to the next claim.

Consequences

Positive

  • Incident-time tuning (slow a hot queue's polling, shrink batches, tighten deadlines) without restarts, consistently across the fleet, auditable via overrides_updated_at.
  • No new infrastructure and no new hot mutable rows: queue_meta is one low-churn row per queue, written at operator frequency — the #169 dead-tuple discipline is unaffected.

Negative / limits

  • Changes take up to one refresh cadence to apply. The cadence has a process-wide escape hatch (AWA_OVERRIDE_REFRESH_MS, floored at 50ms) but is deliberately not configurable per queue in Tier 1.
  • Per-claimer application means a brief window where claimers of one queue disagree.
  • Live concurrency (max_workers), correctness-coupled cadences (heartbeat/rescue), and Python/UI parity are explicitly Tier 2 (#397).

Alternatives considered

  • SIGHUP / config-file reload — awa config lives in builder code inside embedding applications; library workers have no file to reload. Rejected.
  • In-process update API only — no fleet consistency; every embedder rebuilds the distribution story. The DB override path subsumes it.
  • Per-poll override reads — simplest freshness, but adds a control query to every poll on every claimer; the 10s cadence costs almost nothing and is fast enough for tuning.

Relationship to other ADRs

Extends the queue_meta control-plane pattern (pause/resume); respects ADR-026's no-new-hot-rows discipline; interacts with ADR-023's claim-mode selection (hence the zero-boundary guard); effective-value reporting pairs with the runtime-snapshot work in #391.