> For the complete AWA documentation index, see [`llms.txt`](../llms.txt).

# Awa 0.7 — Design & Roadmap

> **Status:** Proposed. This is the design document for the 0.7 release cycle: strategic
> decisions, workstreams, fully-scoped issues (existing and new), experiments with decision
> rules, milestones, and release gates. Companion context: [0.7 planning brief](https://github.com/hardbyte/awa/blob/main/docs/0.7-planning-brief.md)
> (0.6 recap, open-issue inventory, the #169 hot-row chronology, and the #197 process template).
> No timeline, effort, or cost assumptions are made; sequencing is expressed as dependencies
> and gates, not dates.

---

## 1. Theme and thesis

**0.6 was the storage release** — it made the substrate honest: bounded degradation under
pinned MVCC horizons, append-only lifecycle, one-way-door migration tooling, and a passed
benchmark gate.

**0.7 is the operations, observability, and multi-tenancy release** — it makes Awa the
Postgres job queue a serious team can adopt *without a spelunking expedition*: deploy it with
one Helm command, secure it by default, watch a job's trace from enqueue to completion, protect
tenants from each other, chain A→B without a workflow engine, and enqueue from any language.

The storage-engine rewrite (#295) does **not** headline 0.7. It runs as an **evidence track**
with an explicit decision gate, for three reasons grounded in the 0.6 record:

1. **The gate passed.** The corrected long-horizon shape holds 798/s through a 60-minute pinned
   reader with full dead-tuple reclaim. The burning platform that motivated the RFC in May was
   substantially extinguished by rc.1.
2. **The RFC was partly pre-paid.** Its highest-value components — cursor allocation (#321) and
   append-only ready segments + tombstones (#323) — already shipped *inside* the current engine
   via staged migrations. The 0.6 endgame demonstrated the engine can absorb structural change
   in place (v016 → v039) without a new engine identity.
3. **The remaining gap is unquantified.** The visible deltas vs. the rotation-model reference are
   WAL/job (~2.2–2.9 KiB vs ~452 B) and the single-shard completion ceiling. Nobody has yet
   measured how much of the WAL gap is *intrinsic to Awa's contract* (per-attempt identity,
   receipts, terminal evidence, retries) versus *architectural waste*. Rewriting storage twice
   in consecutive releases — with restore-only rollback — to chase an unquantified number would
   repeat the exact mistake the 0.6 process avoided.

If the experiments (§5) show a ≥2× win reachable through staged in-place migrations, storage
work enters 0.7 scope through Gate A. Otherwise the RFC graduates to 0.8 with real data behind
it, and 0.7 ships targeted mitigations.

---

## 2. Strategic decisions

Each decision is stated with its rationale and its reversal condition. These are the calls that
shape everything below.

### D1 — Storage evolution over storage replacement
**Decision:** The default path for #295's ideas is *evolution of the existing queue-storage
engine via staged migrations (v041+)*, not a third engine identity with its own
`prepare/enter/finalize` transition. A new engine identity is justified only if experiments
prove the lifecycle shape cannot be fixed in place.

**Rationale:** The transition machinery, the one-way door, and operator retraining are the
expensive parts of an engine swap — 0.5→0.6 proved it. The 0.6 endgame also proved the converse:
v016→v039 restructured lane cursors, ready segments, terminal counts, and claim routing *inside*
the engine, each individually testable, each individually benchmarked. "Fix one hot row, the
next surfaces" is a viable delivery strategy when each fix is a migration; it is not viable when
each fix requires a fleet-wide cutover.

**Reversal condition:** Gate A (§5, E1/E3) demonstrates that the mutable receipt/lease plane
itself — not any individual table — is the bottleneck, and that no in-place restructuring
closes it.

### D2 — Canonical engine: gate in 0.7, remove in 0.8
**Decision:** `awa migrate` for 0.7 **refuses to run unless the cluster's storage state is
`active`** (queue-storage finalized) or the install is fresh. The canonical engine is formally
deprecated in 0.7 (docs, release notes, startup warning) and its claim/execution/trigger paths
are deleted in 0.8. Upgrades therefore step 0.5 → 0.6 (finalize) → 0.7.

**Rationale:** Canonical exists today only as the transition source. Every feature in this
roadmap would otherwise need dual-engine implementation, dual-engine tests (#360 documented the
cost of *not* doing that), and dual-engine docs. Requiring finalization at the 0.7 boundary
converts #360's dual-engine matrix from a permanent tax into a bounded compatibility suite, and
gives operators one unambiguous instruction. Stepping-stone upgrades are the established norm
(Oban, River, Postgres itself).

**Reversal condition:** Beta feedback showing a meaningful population stuck mid-transition for
reasons Awa can fix.

### D3 — Deployment surfaces become products, not patterns
**Decision:** Promote ADR-027 (callback ingress) and ADR-028 (maintenance-only role) from
Proposed to Accepted and ship all four deployment shapes as first-class, documented,
Helm-packaged artifacts: **worker**, **maintenance-only runtime** (#282), **callback ingress**
(`awa callbacks serve`), and **admin UI** — plus the **serverless `tick()`** shape (#118) for
zero-infrastructure deployments.

**Rationale:** The surface taxonomy in ADR-027 (private admin / public signed callbacks /
internal workers / internal maintenance) is correct and already partially implemented (#291/#293
shipped the callback-only router). What's missing is the last mile: binaries/entrypoints, health
endpoints, auth, and an installable chart. This is the highest-leverage adoption work in the
tracker.

### D4 — Multi-tenancy ships tiered, honestly
**Decision:** #340 (per-key execution control) is a headline 0.7 feature, delivered in tiers:
**Tier 1 (committed):** worker-local per-key rate limiting and concurrency caps (approximate
across a fleet, exact per worker, with no fleet-wide grant/counter state — documented as such).
**Tier 2 (experiment-gated):** storage-accurate fleet-wide per-key concurrency via serialized,
append-only keyed claim/grant evidence at claim time, shipped only if E5 finds a design with **zero
new hot mutable rows**, bounded claim-path cost, and no ready-queue false-idle condition.

**Rationale:** The #169 lesson is absolute: no per-key counter family may become a new hot
mutable table. A worker-local limiter captures most real-world value (noisy-neighbor damping,
per-tenant API-budget respect) immediately; the exact fleet-wide semantics are a hard storage
design problem that deserves the same evidence-first treatment as the engine. Shipping Tier 1
as "approximate, per-worker" with clear docs beats shipping nothing or shipping a lie.

### D5 — Safe by default on the admin surface
**Decision:** Implement #343 (built-in token auth + documented OIDC proxy pattern), and change
the default posture: **when no auth is configured and the bind address is non-loopback, `awa
serve` starts in read-only mode with a visible banner** explaining how to enable mutation
(configure `AWA_ADMIN_TOKEN` or an auth proxy, or pass `--i-understand-open-admin`).

**Rationale:** The UI exposes retry, cancel, DLQ purge, batch ops, and storage-transition
controls. "Open mutating admin on 0.0.0.0" should be a choice, never an accident. Loopback
stays frictionless for development.

### D6 — The public surface gets a written contract
**Decision:** Publish a **stability policy** (`docs/stability.md`) mapping every public surface
to a semver promise: Rust API, Python API, the SQL producer contract (#342 — once shipped,
`insert_job` covered as a public contract with cross-language BLAKE3 `unique_key`, domain-separated
SHA-256 `concurrency_key`,
and shard-routing test vectors), the ADR-042 SQL worker finalization contract
once shipped (`complete_job` receipt, error, and version semantics), the
HTTP admin API (unblocked by the #143 `awa-api` crate split), metric names/attributes, and the
CLI. Everything not listed is internal and may change in any release.

**Rationale:** 0.6's `QueueFanout→PartitionedQueue` beta churn was fine *within* a beta series
but showed the absence of a written boundary. Polyglot producers (#342) and UI/API consumers
(#143) cannot exist without one. This document is cheap and permanently valuable.

### D7 — Job dependencies, not workflows
**Decision:** Ship #14 as minimal A→B chaining built on ADR-029's transactional follow-up
mechanism: `InsertOpts::after(JobRef)` parks B in the deferred backlog in a `waiting_on` state;
A's guarded finalization transactionally promotes B (success) or applies B's declared
`on_parent_failure` policy (`cancel` default | `enqueue_with_context` | `discard`). Exactly one
parent. No fan-in, no fan-out, no DAG type, no workflow state — restated as a non-goal in the
ADR.

**Rationale:** The promotion mechanism already exists (ADR-029 inserts follow-ups atomically
with the finalize UPDATE); #14 is mostly a parking state and a policy enum. Single-parent A→B
covers the dominant request ("after the PDF, send the email") while the #81 research issue
guards the workflow boundary.

### D8 — Observability is end-to-end or it isn't done
**Decision:** #110 ships fully in 0.7: W3C `traceparent` captured at enqueue (write-once on the
job row — append-only, no MVCC concern), restored as span parent at claim/dispatch, propagated
across the PyO3 boundary, span status recorded at guarded finalization, linked spans for
retries/callbacks/cron fires. Paired with a new **admin-UI attempt timeline** (claims, receipts,
heartbeats, callbacks, finalizations per attempt) that link-outs to the operator's tracing
backend by trace id, and a **Grafana alert pack** codifying the watch-list from the 0.6 upgrade
guide.

---

## 3. Workstreams and issues

Priorities: **P0** = release-defining (0.7 does not tag without it), **P1** = strongly
committed, **P2** = opportunistic, **P3** = research/candidate. "NEW-n" issues are drafted by
this roadmap and not yet filed.

### WS-1 · Foundations (everything else builds on these)

| Item | Pri | Scope & acceptance |
| --- | --- | --- |
| **#335** CI sharding | P0 | Matrix-sharded Rust tests (own Postgres per shard), cargo-nextest, audit of every >60s test to pin behavior with sub-second configured windows. **Accept:** worst shard <8 min; no real-time production defaults load-bearing in tests. Do first — it multiplies every later item. |
| **#360** Engine-aware test harness | P0 | Core landed on main via #362: `AWA_TEST_ENGINE` parameterization, `skip_unless_canonical` guard, engine-aware `work_one_in_queue`, and the `rust-test-queue-storage` CI leg. Residual scope: triage any remaining canonical-only artifacts, and the forward-compat piece now split out as #367. Under D2 this becomes a *bounded* compat suite: full matrix until 0.8 removes canonical, then retires to the forward-compat matrix. **Accept:** broad suite green under `queue_storage`; `cancel_by_unique_key`-class defects structurally impossible to ship again. |
| **#367** Forward/backward-compat matrix | P0 | Automated: pinned old binaries (0.5.7, 0.6.0) enqueue→claim→complete against a 0.7-migrated schema; 0.7 binary against 0.6 schema (pre-migrate) fails *loudly and legibly*. Closes the "no test runs a genuinely old binary against the newest schema" gap from #360. **Accept:** matrix in nightly; documented support statement. |
| **#143** `awa-api` crate split | Re-scoped | The April spike on #143 already built and reverted the split (no second consumer; the imposition is ~1 MB of embedded bundle). 0.7 ships the hygiene half only — response-type discipline inside `awa-ui` (#403), which is all D6's HTTP promise needs (and #368 shipped without the split). The crate split moves to 0.8 (#143), where the MCP server (#389) is the first genuine external consumer of the types. |
| **#368** Worker health & readiness endpoints | P0 | Workers currently run **no HTTP server at all**. Add an opt-in, tiny health listener (`AWA_HEALTH_ADDR`): `/healthz` (process live), `/readyz` (DB reachable, schema compatible, claim loop ticking, maintenance heartbeat fresh). Also `awa health --database-url` CLI probe for probe-less environments. Required by #344. **Accept:** k8s liveness/readiness probes work out of the box; documented. |
| **#369** Stability policy (`docs/stability.md`) | P0 | Per D6. Surface-by-surface semver map; deprecation policy (announce in N, warn in N+1, remove in N+2 unless security); what beta/rc mean for each surface. |
| **#370** Canonical deprecation & 0.7 upgrade gate | P0 | Per D2. Migration refuses non-`active` clusters with a message that names the exact 0.6 finalize steps; startup warning on any canonical code path; 0.8 removal plan documented; `docs/upgrade-0.6-to-0.7.md` (short — the storage step is "be finalized"; everything else is additive). |
| Housekeeping | P1 | Verify-and-close #256 (SeaORM shipped), #347 (partitioned queues shipped; fold unresolved open questions into ADR-031 or new issues), triage #346 (nightly failure). File **#383: 0.7 release-readiness tracker** — the #197 analogue, opened on day one with the six gates from §7 as live checklists. |

### WS-2 · Storage & performance (evidence track — see §5 for experiments)

| Item | Pri | Scope & acceptance |
| --- | --- | --- |
| **#246** Deadline-rescue claim regression | P0 | First step per the 0.6 record: **reproduce on 0.6.0 final** (E2) — the release decision explicitly said don't chase it unless it reproduces on current main. If it reproduces: name the mechanism (working-set inflation on `lease_claims` is the standing hypothesis), then fix via candidates: deadline-bucketed partial index (needs partitioned-table migration plan), deadline coarsening (round `deadline_at` to reduce index churn), or rescue-scan via segment cursor. Safety rail from the issue stands: per-claim deadline rescue stays default-on. **Accept:** ≤5% throughput overhead with rescue ON at the 1×256 shape, or a named mechanism + documented mitigation if the fix lands in the Gate-A storage work instead. |
| **#295** Storage RFC → Gate A | P0 (decision), scope conditional | Run E1 (allocator bake-off) + E3 (WAL decomposition). Author the draft ADR from results. **Gate A decision rule (§5)** determines whether restructuring migrations enter 0.7 or the RFC graduates to 0.8 with data. Either way #295's RFC questions 1–5 get *answered*, not deferred. |
| **#371** Ring-state metadata striping | P1 | The residual pinned-horizon accumulators after #355 are `lease_ring_state` (~14k dead tuples/hr) and `claim_ring_state` (~3.5k). Apply the ADR-026 discipline (striping or delta-append) to ring bookkeeping. Small, measurable, keeps the "no dominant hot row" invariant tight. **Accept:** long-horizon idle-phase dead tuples for ring-state ≤ noise. |
| **#341** Backpressure | P1 | Per the issue's own design lean: **soft-signal default, opt-in hard rejection.** Concretely: (a) enqueue paths can return a depth signal (`EnqueueOutcome::pressure`) sourced from lane-head cursors (index-only, no scans); (b) `InsertOpts::backpressure: Off \| Signal \| Reject{limit}` — `Reject` returns a typed error and is documented as *changing transactional-enqueue semantics* (ADR-006 tension made explicit); (c) `PacedProducer` helpers in Rust + Python (the pattern the bench harness already proved externally); (d) metrics `awa.enqueue.backpressure.{signaled,rejected}`. E6 validates defaults. **Accept:** 2×-capacity offered load with paced producer holds bounded depth; hard-reject never enabled implicitly. |
| **#380** Adaptive claimers (experiment-gated) | P2 | E4: bounded controller adjusting per-queue claimers within `[1, max]` on lane-lag signal, opt-in. Addresses the single-shard default-shape ceiling (10k offered / 4k completed) without making users hand-tune `claimers`. Ships only if E4 shows stability under bursty load; otherwise document tuning presets instead. |
| **#303** Maintenance-task split | P2 | Unchanged: telemetry-gated. Check `awa_maintenance_branch_overrun_total` across beta fleets; implement the rescue+promote split only if non-trivial overruns appear; else close wontfix as the issue itself proposes. |

### WS-3 · Deployment & operations (D3, D5)

| Item | Pri | Scope & acceptance |
| --- | --- | --- |
| **#282** + ADR-028 Maintenance-only role | P0 | `.maintenance_only()` builder + `awa maintenance run`. Runs promote/rescue/rotate/prune/cleanup/metadata; never claims, never invokes handlers, never dispatches HttpWorker; preserves leader election + shutdown semantics. Promote ADR-028 to Accepted. **Accept:** the issue's acceptance list; chaos nightly includes a maintenance-only + callback-only + HttpWorker topology with zero always-on general workers. |
| **#372** + ADR-027 Callback ingress deployable | P0 | `awa callbacks serve` — the callback receiver as its own process (router from #291/#293), BLAKE3-signed, no admin surface, no UI assets, minimal deps. Promote ADR-027 to Accepted with the surface-taxonomy table as normative docs. **Accept:** public-ingress deployment documented end-to-end (Helm values included); admin UI reachable only privately in the reference topology. |
| **#118** Serverless `tick()` | P0 | `Client::tick(budget) -> TickReport` (every sub-step bounded, e.g. LIMIT 100; returns promoted/rescued/rotated/pruned/dispatched counts) + authenticated `POST /api/tick`. No persistent leader election — short advisory try-locks + SKIP LOCKED, per the issue's lean. Documented drivers: Cloud Scheduler, Vercel cron, GitHub Actions cron, `pg_cron` SQL wrapper. Explicit non-goals restated: no sub-second latency, no LISTEN/NOTIFY, dispatcher doesn't scale to zero *invisibly*. **Accept:** a demo app runs on scheduler-driven ticks alone with correct promotion/rescue; maintenance-gap behavior documented. |
| **#344** Helm chart | P0 | `charts/awa` in-repo, OCI-published, chart-lint + kind smoke in CI. Templates: worker Deployment (optional auth-proxy sidecar), migration Job (helm hook; `awa migrate` as single DDL owner), UI Deployment (readOnly **defaults true** per D5), optional callback-ingress Deployment, optional maintenance-only Deployment, ServiceMonitor. Chart docs state maintenance/cron are leader-elected in-process — **no CronJob**, except the documented tick() variant. Depends on #368 probes. |
| **#343** Admin auth | P0 | Per D5: `AWA_ADMIN_TOKEN` (constant-time compare; header for API, cookie session for UI), documented oauth2-proxy/OIDC pattern with a worked example, non-loopback-unauthenticated ⇒ read-only default + banner, structured audit log line for every mutating action. Authz tiers explicitly deferred. |
| **#373** `awa doctor` | P1 | One command, human + `--json` output: schema/binary version match, storage state + transition blockers, oldest `xact_start` / xmin horizon age, dead-tuple hotspots vs known table families, autovacuum recency, ring rotation health (`skipped_busy`/`blocked` rates), orphaned runtime instances, NOTIFY round-trip check, DLQ depth, failed-retention pruning stats. Subsumes the `awa diagnose mvcc` promised in #169's operational playbook (never shipped). **Accept:** every troubleshooting.md scenario has a corresponding doctor check. |
| **#374** Connection-pooler compatibility | P1 | pgbouncer/pgcat/RDS-Proxy are undocumented today and transaction-pooling silently breaks LISTEN/NOTIFY. Ship: startup NOTIFY self-test → explicit warning + automatic documented polling fallback cadence; docs matrix (session vs transaction mode, pickup-latency deltas from E7); CI leg running the integration suite through pgbouncer. |

### WS-4 · Observability (D8)

| Item | Pri | Scope & acceptance |
| --- | --- | --- |
| **#110** End-to-end tracing | P0 | Per D8 and the issue's own plan: `trace_context` captured at enqueue (W3C tracecontext, <200 B, write-once — composes with append-only storage), parent restored at claim, PyO3 propagation (ADR-004 boundary), span status at guarded finalization (run_lease-checked), linked spans for retry/callback/cron, sampler + propagator config. Rust→execution first, Python boundary second, per the issue. Gated by E8: <2% enqueue overhead. |
| **#375** Attempt timeline in UI + trace link-out | P1 | Per-job attempt view assembled from receipts/leases/closures/terminal evidence: claims, heartbeats, snoozes, callback park/resume, finalization — with configured tracing-backend link-out by trace id. Turns the storage engine's evidence trail into operator UX. Depends on #143. |
| **#376** Grafana alert pack | P1 | Codify the 0.6 upgrade-guide watch-list as importable alerts: `xact_start` age, `rotate…skipped_busy` sustained, `prune…blocked` non-zero, queue-lag SLO, DLQ depth delta, `maintenance_branch_overrun_total`, backpressure signals (#341). Ship beside the existing dashboard JSON. |

### WS-5 · Workload control & flow (D4, D7)

| Item | Pri | Scope & acceptance |
| --- | --- | --- |
| **#340** Per-key control — Tier 1 | P0 | **ADR-033.** `InsertOpts::concurrency_key` (defaulting to `ordering_key` when unset), per-queue `KeyPolicy { max_in_flight_per_key, rate_per_key }` enforced in the dispatcher: worker-local exact, fleet-approximate (documented formula: fleet cap ≈ per-worker cap × workers). The key digest travels durably with the job so retries retain identity, but Tier 1 adds no fleet-wide grant/counter family. Gated jobs are *not* claimed-then-requeued; the dispatcher holds them un-dispatched within its claimed batch or snoozes with jitter (bounded, measured in E5). Metrics: `awa.dispatch.key_gated`, per-key fairness gauge. |
| **#340** Per-key control — Tier 2 | P2 (E5-gated) | **ADR-033.** Fleet-exact concurrency via a database-authoritative policy plus an append-only execution grant encoded in keyed row-local claim evidence (preferred) or a separate E5 ledger candidate. Claim decisions use transaction-scoped key try-locks; keyed-only jobs route shard-locally; the allocator admits only a contiguous FIFO prefix, never advances over a gated head, probes other bounded lanes, and distinguishes gated from idle. Grant closure emits a transactional queue wakeup; gated dispatchers wait on notify plus a safety poll. Limit reductions fence admission in a short epoch transition, verify outside the exclusive row-lock window under a bounded timeout, and refuse while any key exceeds the target; an incomplete or over-target verification leaves the transition pending for epoch-guarded resume or cancel. Complete-coverage enablement rejects legacy rows without authoritative key identity until drained or backfilled; an explicit partial-legacy mode may admit them ungoverned with non-green diagnostics and metrics. Ships only if E5 shows acceptable claim/pickup cost, no hot mutable counter family, no cursor-monotonicity violation, and no #246 regression. Cross-key fairness is separate. |
| **#401** Caller-owned finalization | P1 | **ADR-042.** A distinct caller-owned handler return type plus Rust `complete_in_tx` / a versioned SQL completion function let application rows and completion of the exact run lease commit together. Both bindings use one hardened `SECURITY DEFINER` function, so the application role needs no direct Awa-table grants. The runtime reconciles a durable `FinalizationToken` on every handler exit; after a bounded database outage it may release the heavyweight local permit while leaving the durable attempt/grant open for rescue. Completion-only v1, queue-storage only, opt-in single-job slow path; explicit in-transaction follow-ups replace process-local Completed specs for participating kinds. This 0.7 item owns the narrow ADR-043 substrate it requires: provision the bounded `NOLOGIN` execution owner, install and transfer the exact `complete_job` function, revoke `PUBLIC`, apply only its manifest-listed application grant, and validate ownership/ACL plus role-graph closure on the oldest and newest supported PostgreSQL majors. It does not implement the ordinary-runtime capability split. **Accept:** `awa doctor` diagnoses missing, excessive, transitively reachable, and incorrectly owned finalizer grants for the configured application role. |
| **#14** Job dependencies | P1 | **ADR-034**, per D7: `InsertOpts::after(JobRef)`, `waiting_on` deferred state, transactional promotion on parent finalize via ADR-029 machinery, `on_parent_failure: Cancel \| EnqueueWithContext \| Discard`. Single parent only; DAG/fan-in/fan-out named non-goals. TLA+ witness for the promote-on-finalize race (parent finalizes concurrently with child cancel). Rust + Python parity, UI shows the dependency edge on both jobs. |
| **#342** SQL producer contract | P1 | Introduce `awa.insert_job` as the single-signature public capability (D6), backed by internal compatibility/storage helpers. Its one extension point is a final `opts jsonb DEFAULT '{}'::jsonb` argument; optional keys evolve additively without overloads. Document request/result/error semantics versioned against `schema_version`; compute the domain-separated SHA-256 `concurrency_key` digest (PostgreSQL's built-in `sha256()` — `unique_key` stays client-side BLAKE3 per ADR-002) and ADR-033 shard precedence (`ordering_key`, else `concurrency_key`, else rotor) authoritatively in PostgreSQL rather than trusting a client shard; publish cross-language BLAKE3 `unique_key`, SHA-256 concurrency-digest, and routing vectors; provide a conformance script any producer implementation can run against a live schema; and ship `docs/sql-producer-contract.md`. Reference snippets (Node/Go) are *examples*, explicitly unsupported — no official third-language clients (issue's own out-of-scope). |
| **#81 / #83** Research | P3 | Deliver the written recommendations the issues ask for, during 0.7, so 0.8 planning inherits decisions instead of open questions. #81's recommendation should explicitly evaluate whether ADR-034 (A→B) + ADR-029 (follow-ups) already constitute the "very constrained first-class API" endpoint — my expectation is yes, and the workflow-engine boundary holds there. |

### WS-6 · Developer experience & documentation

| Item | Pri | Scope & acceptance |
| --- | --- | --- |
| **#377** Docs site | P1 | Versioned docs site (mdBook or equivalent) built from `docs/` in CI, published per release tag. The 25 KB README slims to overview + quickstart + links. Information architecture: Learn (concepts/quickstarts) / Operate (deploy, upgrade, troubleshoot, doctor) / Reference (config, API, SQL contract, metrics, stability policy) / Internals (architecture, ADRs, TLA+). |
| **#378** Python DX pass | P1 | Complete `.pyi` stubs audited against the PyO3 surface with a CI check (mypy/pyright gate on stubs + examples); error-message audit (every raised error names the fix or links docs); `awa-pg` wheel matrix confirmed for current CPython + platform set. |
| **#379** Operations handbook | P1 | Consolidate deployment.md / deploying-on-managed-postgres.md / troubleshooting.md / upgrade guides into one coherent "run Awa in production" narrative on the docs site, including the pooler matrix (#374), MVCC discipline, autovacuum guidance, and a production-readiness checklist. |
| **#381** Example gallery | P2 | Full runnable reference apps (CI-tested like the existing quickstarts): FastAPI + Awa (bridge + callbacks), Django sync bridge, axum + SeaORM, serverless tick() on a cron driver, and a mixed Rust-producer/Python-worker pipeline. |
| **#382** `awa dev` sandbox | P3 | Candidate: one command that runs a disposable Postgres (docker), migrates, starts a demo worker + UI with seeded jobs. Pure onboarding sugar; do only if cheap. |

---

## 4. New ADRs to write

| ADR | Title | Status target | Source |
| --- | --- | --- | --- |
| 027 | Callback ingress as a deployable surface | Proposed → **Accepted** | #372 |
| 028 | Maintenance-only runtime role | Proposed → **Accepted** | #282 |
| 033 | Per-key execution control (tiered) | Accepted | #340, E5 |
| 034 | Job dependencies (A→B on ADR-029) | Accepted | #14 |
| 035 | Backpressure and producer flow control | Accepted | #341, E6 |
| 036 | Public surface stability policy | Accepted | #369 / D6 |
| 037 | Canonical engine deprecation & removal | Accepted | #370 / D2 |
| 042 | Caller-owned finalization transactions | Accepted | #401 / #342 |
| 03x | Segment/cursor storage evolution | Draft from E1/E3; status decided at Gate A | #295 |

---

## 5. Experiments

Each experiment has a hypothesis, method, metrics, and a **decision rule** — the 0.6 lesson is
that gates must be numeric and written down *before* the run.

**E1 — Claim-allocator bake-off** *(feeds Gate A; answers #295 Q1/Q2)*
Prototypes on the existing harness: **P-a** current engine, tuned (baseline); **P-b** batch-tick
allocator (pgq-style tick ranges + Awa's deferred backlog as the retry sidecar); **P-c**
claim-ledger v2 (per-segment cursor with the indexed short-circuit lesson from #355 — the
spike's naive anti-join is known-wrong).
Shapes: pinned-horizon at 800/s **and 1,600/s**; single-shard 10k/s; skewed-key.
Metrics: complete/s, depth, WAL/job, dead tuples, claim p99, fairness.
*Decision input to Gate A.*

**E2 — #246 reproduction on 0.6.0** *(feeds the #246 fix)*
Re-run the issue's reproducer (1×256, rescue ON/OFF) on the 0.6.0 tag. If reproduced: attribute
via `pg_stat_io`/waits/`pgstattuple` on `lease_claims`; A/B the candidate fixes.
**Decision rule:** overhead ≤5% ⇒ close with docs; >5% with named mechanism ⇒ fix in 0.7;
mechanism implicates receipt-plane shape ⇒ fold into Gate A scope.

**E3 — WAL/job decomposition** *(the keystone experiment; answers "is the pgque gap intrinsic?")*
`pg_waldump` attribution of one soak's WAL across record types and table families: lifecycle
inserts vs receipt/claim evidence vs terminal batches vs index maintenance vs ring bookkeeping.
**Decision rule:** if ≥40% of WAL/job is attributable to *removable architecture* (not contract
evidence), storage restructuring has proven headroom → strong Gate A input. If the gap is mostly
contract-intrinsic, the honest answer is "the ~2 KiB/job is the price of per-attempt identity +
receipts" — document it, close that part of #295's motivation, and stop chasing pgque's number.

**Gate A — storage scope decision** *(consumes E1 + E2 + E3)*
Restructuring migrations enter 0.7 **only if all three hold**: (i) a prototype ≥ parity clean and
strictly better at 1,600/s pinned; (ii) ≥40% WAL/job reduction attributable per E3; (iii) the
change is deliverable as staged in-place migrations (D1) with TLA+ deltas identified. Otherwise:
0.7 ships #246 fix + #371 + tuning presets; the draft ADR + all evidence graduates to 0.8.

**E4 — Adaptive claimers** *(feeds #380)*
Lag-signal controller vs static settings across steady/bursty/skewed load.
**Decision rule:** no oscillation, no ordering-contract violation, ≥1.5× default-shape completion
⇒ ship opt-in; else ship documented presets only.

**E5 — Per-key enforcement prototypes** *(feeds #340 Tier 2 / ADR-033)*
(a) worker-local token bucket (Tier 1 baseline), (b1) keyed attempts routed through row-local
`lease_claims` with existing closure evidence, (b2) a separate append-only grant/closure ledger,
(c) shard-pinning + per-shard caps, and (d) committed key parking plus closure-time promotion. The
exact claim-gating prototypes use transaction-scoped key try-locks, bounded cross-lane probes,
concurrency-key shard locality, and per-call same-key memoization. Include keyed jobs without
ordering keys and differing ordering/concurrency keys. Skewed tenant distributions (Zipf); metrics: fairness
(Jain's index), hot-tenant starvation bound, claim p99 delta, storage write amplification
(**must be zero new hot mutable rows**), retained row-local claims, false-idle outcomes,
grant-close notification amplification, and notify/poll pickup latency.
**Decision rule:** (b1), (b2), or a fully proved (d) within 10% claim p99 of baseline, zero hot-row
footprint, no ready-but-reported-idle outcome, and notify-bounded session-mode pickup ⇒ Tier 2
ships. (c) is a performance/control comparison, not a fleet-exact result; otherwise Tier 1 only,
data published in the ADR.

**E6 — Backpressure defaults** *(feeds #341 / ADR-035)*
Paced producer vs naive at 1×/2×/4× capacity; find the depth-target and pacing defaults that
hold bounded depth without throughput sacrifice under 1×.

**E7 — Pooler matrix** *(feeds #374)*
Integration suite + latency probes through pgbouncer session and transaction modes; quantify
NOTIFY loss and fallback-polling pickup latency; prove ADR-042 finalization remains atomic through
transaction pooling and ADR-033 grant wakeup uses the gated safety cadence there; produce the
compatibility table.

**E8 — Tracing overhead** *(gate for #110)*
Enqueue/claim/complete throughput with trace capture on/off, sampled and unsampled.
**Decision rule:** <2% enqueue overhead sampled ⇒ default-on capture with sampling; 2–5% ⇒
default-off, opt-in; >5% ⇒ redesign storage of the context before shipping.

---

## 6. Milestones

Gate-sequenced, not date-sequenced. Later milestones may start early where dependencies allow;
a milestone is *done* when its exit criteria hold in CI/nightly, not when its PRs merge.

**M0 — Foundations.**
Issues #335, #360, #367, #143, #368, #369, #370 groundwork, housekeeping closes, and the
#383 tracker opened with §7 gates.
*Exit:* worst CI shard <8 min; dual-engine suite green; compat matrix in nightly; `awa-api`
published as a crate; health endpoints merged; stability policy merged.

**M1 — Evidence & decisions.**
E1–E3 run; Gate A decided and recorded on #295; E5 run; ADR-033/034/035/042 drafted; #246 mechanism
named.
*Exit:* Gate A recorded with data; every 0.7 ADR in review or accepted; #246 has a fix plan or a
closure with evidence.

**M2 — Deployment shapes.**
Issues #282, #372 (ADR-027/028 → Accepted), #118, #343, #373, #374, and #344 (last — consumes
all of the above).
*Exit:* the four-surface reference topology (private admin, public signed callbacks, workers,
maintenance-only) installable from the Helm chart with kind smoke in CI; tick()-only demo green.

**M3 — Observability.**
Issue #110 (Rust path → Python boundary), the E8 gate, #375, and #376.
*Exit:* a job enqueued in a traced FastAPI request shows one connected trace through a Python
worker retry to completion; UI attempt timeline links to it.

**M4 — Flow & tenancy.**
Issue #340 Tier 1 (+Tier 2 if E5 passed), #401, #14, #341, #342, and the #81/#83 recommendations
written.
*Exit:* ADR-033/034/035/042 accepted and implemented per their acceptance sections; SQL contract
conformance script green cross-language.

**M5 — Storage (scope per Gate A).**
Either: targeted track (#246 fix, #371, #380 if E4 passed, tuning presets) — or additionally
the Gate-A restructuring migrations with TLA+ deltas and a fresh long-horizon acceptance run at
the raised bar.
*Exit:* §7 performance gate numbers hold on clean main.

**M6 — Release train.**
`0.7.0-beta.1` when M0+M2+M3 are done and M4/M5 are merge-complete (judgment-call gates resolve
under beta exposure, per the 0.6 precedent). RC when every §7 gate is green. Stable when the
tracker checklist is fully checked, release notes carry caveated evidence (not universal
claims), and publish mechanics (crates in dependency order, wheels, chart OCI push, Docker,
GitHub release) are green.

---

## 7. Release gates for 0.7.0 stable

The #383 tracker opens with these as live checklists (the #197 template, six gates):

1. **Correctness.** TLA+ green including new models: dependency promotion race (ADR-034),
   claim-time key gating if Tier 2 ships, allocator/segment deltas if Gate A passed. Trace
   witnesses for each new lifecycle path. Model-to-code mapping docs updated.
2. **Performance.** (a) The 0.6 pinned-MVCC long-horizon shape **still passes** — no regression,
   ever; (b) #246 shape: ≤5% rescue-ON overhead or named mechanism + shipped mitigation;
   (c) partitioned-queue preset sustains ≥9k jobs/s e2e on the reference 24-CPU harness;
   (d) tracing overhead within its E8 gate.
3. **Stability.** 14 consecutive green nightlies (chaos + TLA + failover + pooler leg);
   dual-engine suite green; compat matrix green.
4. **Operations.** Helm kind-smoke in CI; health endpoints + doctor shipped; upgrade-0.6-to-0.7
   guide with the D2 gate front-and-center; every new surface has troubleshooting entries.
5. **Documentation.** Stability policy published; SQL contract v1 + conformance script; docs
   site live and versioned; release notes framed as caveated point-in-time evidence.
6. **Security.** #343 shipped with the D5 default posture; security.md updated for the
   four-surface topology; secret scan clean; callback signing docs current.

---

## 8. Risks and mitigations

| Risk | Mitigation |
| --- | --- |
| Gate A passes and storage work destabilizes the cycle | D1 keeps it staged in-place migrations, each individually benchmarked and revertible pre-release; the raised #169-style bar applies to *every* storage migration, not just the last one; M5 is the only milestone allowed to slip out to 0.8 wholesale. |
| D2 strands un-finalized 0.6 clusters | The gate message names exact finalize steps; `awa doctor` diagnoses transition blockers; beta feedback is the explicit reversal trigger. |
| Tier-1 per-key semantics get mistaken for fleet-exact | Docs state the approximation formula everywhere the knob appears; metrics expose per-worker vs fleet counts; ADR-033 records the tiering rationale. |
| Scope breadth (five P0 workstreams) | M0 is genuinely first (CI speed + harness pay for themselves); everything else is dependency-ordered with per-milestone exit criteria; the beta-first pattern lets judgment calls resolve under exposure instead of blocking the tag. |
| #110 trace storage interacts with future storage changes | trace_context is write-once at enqueue on append-only rows — deliberately shape-agnostic; confirmed in ADR review against the Gate-A design if it proceeds. |
| Nightly flake debt erodes the 14-green gate | #335's real-time-window audit is P0 precisely for this; any new flake gets an issue within 24h per the tracker rules. |

---

## 9. Explicitly out of scope for 0.7

- **Workflow engine features**: DAGs, fan-in/fan-out, sagas, workflow state (D7; #81 guards).
- **Official non-Rust/Python client libraries** (#342 ships the contract, not clients).
- **Replica-aware topologies / read-replica routing** (unchanged from the 0.6 decision).
- **Authz granularity tiers** on the admin surface (auth ships; roles deferred, per #343).
- **Strict PostgreSQL capability roles for the ordinary runtime** (#452 / ADR-043). #401 ships the
  narrow bounded-owner and exact-manifest slice required by ADR-042's finalizer; replacing the
  ordinary runtime's blanket table/function grants, including the bounded maintenance dispatcher,
  is a later expand/use/tighten program with a direct-COPY performance decision.
- **A third storage-engine identity** absent Gate A (D1).
- **Multi-database / sharded-cluster federation** — one Postgres per Awa remains the model.

---

## 10. Disposition of tracked open issues

| Issue | 0.7 disposition |
| --- | --- |
| #295 storage RFC | Evidence track: E1/E3 → Gate A; ADR drafted either way; implementation conditional (WS-2). |
| #360 dual-engine harness | **In**, P0, M0 (bounded by D2). |
| #347 partitioned queues | Verify shipped in 0.6; fold residual open questions into ADR-031; close. |
| #346 nightly failure | Triage in M0 housekeeping; close or spawn a scoped bug. |
| #344 Helm | **In**, P0, M2. |
| #343 UI auth | **In**, P0, M2, with D5 default-posture change. |
| #342 SQL contract | **In**, P1, M4 (under D6). |
| #341 backpressure | **In**, P1, M4 (ADR-035; E6). |
| #340 per-key control | **In**, tiered per D4 (ADR-033; E5), M4. |
| #401 caller-owned finalization | **In**, P1, M4 (ADR-042; composes with #342 and #340). |
| #452 PostgreSQL capability boundary | ADR-043 design only beyond #401's narrow 0.7 bounded-owner/finalizer-manifest slice; ordinary-runtime implementation deferred beyond 0.7. |
| #335 CI sharding | **In**, P0, M0, first. |
| #303 maintenance split | Telemetry-gated during beta; implement or close-wontfix per its own rule. |
| #295-adjacent #246 regression | **In**, P0 (E2 first; fix or fold into Gate A). |
| #282 maintenance-only role | **In**, P0, M2 (ADR-028 → Accepted). |
| #256 SeaORM | Verify shipped (crate exists, publish chain updated); close. |
| #246 deadline-rescue | (see above) |
| #143 awa-api split | Re-scoped: hygiene half in 0.7 (#403); crate split deferred to 0.8 with #389 as the consumer. |
| #118 tick() | **In**, P0, M2. |
| #110 tracing | **In**, P0, M3 (full scope — 0.6 shipped only the opt-in propagation subset). |
| #83 ingress adapters | Research deliverable written during 0.7; likely outcome: reference adapters only. |
| #81 orchestration | Research deliverable written during 0.7; expected outcome: ADR-034 + ADR-029 are the boundary. |
| #14 dependencies | **In**, P1, M4 (ADR-034, per D7). |

New issues filed for the original roadmap: **#367–#382** (§3) plus **#383** (release tracker).
Later issue **#401** adds caller-owned finalization. ADR numbers 033–037 and 042 are claimed by
files in `docs/adr/` per the numbering convention.

---

## 11. What "best possible" means here, in one paragraph

Awa's differentiation is *earned honesty*: TLA+ models, raised benchmark bars, caveated release
notes, one-way doors documented before they're crossed. 0.7 extends that ethos from the storage
engine to the whole product surface: deployments that are safe by default, claims about
per-key fairness that state their approximation, a storage rewrite that must beat a written
number before it may exist, a public contract document instead of implied stability, and
operator tooling (`doctor`, health, alerts, timelines) that turns the engine's evidence trail
into something a human on call at 3 a.m. can actually use. That — not a features-per-release
count — is what makes an open-source infrastructure project the best in its category.