Deployment Guide¶
This guide covers how to deploy Awa workers and the web UI with Docker or Kubernetes.
Deployment Model¶
Awa workers are stateless. Queue state lives in Postgres. A worker process typically contains:
- one dispatcher per configured queue
- one heartbeat loop
- the maintenance service: on the leader, a recovery loop (promotion and rescue) and a hygiene loop (cleanup, cron sync, rotation and the rest)
- one elected leader per database for rescue, promotion, cleanup, and cron evaluation
That means you scale by running more worker processes, not by adding local state.
Process Roles¶
In production, treat these as separate concerns:
- application worker: your Rust or Python service embedding Awa and calling
Client::start()orclient.start() - migration/admin/UI process:
awaCLI formigrate,storage,job,queue,cron, andserve - PostgreSQL: the only required external dependency
awa serve is an operator UI and admin API. It is not the worker runtime.
Queue Storage Cutover¶
Fresh 0.6 installs use queue storage as the worker engine. Existing 0.5.x clusters must follow the staged storage-transition procedure; Migrations covers the general migration contract and tooling.
Operationally:
- queue storage is the 0.6 worker engine
- canonical tables remain in the schema for migration, rollback boundaries, and compatibility SQL surfaces
- while state is
canonicalorprepared, 0.5.x and 0.6 pods may coexist and all writes/execution remain canonical - after
enter-mixed-transition, new writes route to queue storage while 0.6 drain workers finish the canonical backlog - once queue-storage rows exist, rollback to a 0.5.x-only fleet is not supported; finish the transition or restore from backup
For a 0.5.x cluster, do not stop canonical workers and start queue-storage workers as a separate manual cutover. Use the storage commands so producer routing, canonical drain, queue-storage executor readiness, and rollback interlocks move together.
Current Rust and Python worker starts default to the correct effective engine for the storage-transition state. Only set queue_storage_schema / ClientBuilder::queue_storage(...) when you need to override the default schema name or segment sizing.
Connection Pool Sizing¶
Practical starting point, based on the current runtime internals:
- each runtime (client) keeps one
LISTENconnection that carries enqueue wakeups for all of its queues - each runtime (client) keeps a second
LISTENconnection for admin-cancel notifications - the elected leader holds one advisory-lock connection while it remains leader
- claim, heartbeat, completion, cleanup, and handler SQL all share the same pool
Conservative starting heuristic, not a hard requirement:
With the default claimers = 1, this is the old shortcut: max_connections >= queue_count + 5.
Then add headroom for:
- handler code that also talks to Postgres
- producer traffic sharing the same pool
- health checks and admin queries
Examples:
- one queue, light worker-only process:
10is a reasonable start - three queues plus handler SQL: start around
12-20 - combined API + worker process: either use separate pools or size for both workloads explicitly
The Python client defaults to max_connections=10. awa serve defaults to a pool of 10 connections (configurable via --pool-max / AWA_POOL_MAX). Other CLI subcommands use a single connection.
Idle connections and Postgres memory¶
Each pooled connection is a Postgres backend with its own relation, plan and function caches, and those caches only grow while the backend lives. Queue storage has several hundred partitions and indexes, and the claim path runs per-slot functions, so a backend that has run claims, completions and maintenance holds tens of MB of backend-local memory. Postgres memory for an Awa process is roughly the number of open backends times that size.
The pool keeps every connection it opened at the busiest moment. sqlx hands out the connection that has been idle longest (its idle connections form a FIFO queue), so even an idle runtime, which acquires a connection several times a second for polling and maintenance, touches every pooled connection within seconds. While a runtime keeps using the pool this often, idle_timeout rarely closes any of them, and each one ends up having run every statement Awa issues. A process that holds a pool but seldom uses it, such as a client with no workers started, does have idle connections closed.
To return memory after busy periods, set max_lifetime on the pool so connections are recycled. A connection past its lifetime is closed when it is next released or when the pool's reaper finds it idle, and the pool opens replacement connections as needed to maintain min_connections or to meet concurrent demand. The pool can therefore shrink toward what concurrent demand needs, but not below min_connections. A replacement's caches build up again as it is used.
let pool = PgPoolOptions::new()
.max_connections(20)
.max_lifetime(Duration::from_secs(300))
.connect(&database_url)
.await?;
sqlx's default lifetime is 30 minutes. Shorter lifetimes reconnect more often; at a few minutes the reconnect cost is negligible next to the memory returned.
awa serve sets these from --pool-max-lifetime / AWA_POOL_MAX_LIFETIME (default 1800 seconds) and keeps at least --pool-min / AWA_POOL_MIN connections (default 2). The Python client takes the lifetime as the keyword argument max_lifetime_seconds (default 1800.0, sqlx's default). None disables it, as PgPoolOptions::max_lifetime(None) does in Rust, so connections stay open until closed for another reason. Zero, negative and non-finite values raise ValidationError.
Do not disable the statement cache with PgConnectOptions::statement_cache_capacity(0) to save memory. In sqlx 0.9 every query then leaves a named prepared statement on the server that is never closed, and backend memory grows without bound.
Primary Database Hygiene¶
Queue storage keeps the main ready path append-only, but Awa is still a high-churn Postgres workload. The main operational pitfall is no longer one giant mutable queue heap; it is long-lived readers or stale transactions that block lease or segment prune while the maintenance leader keeps rotating forward.
Recommended practice:
- keep analytical reads and admin transactions short on the primary
- run long-lived reporting queries against a replica when possible
- avoid leaving sessions
idle in transaction - monitor
pg_stat_activityfor long transactions - monitor
pg_stat_user_tableson the active queue-storage schema; ready, tombstone, done, and terminal-delta segments should stay near zero dead tuples, lease segments may spike within the rotation window but should collapse after prune, andattempt_stateshould roughly track live long-running attempts - tune autovacuum for the database if the lease tables churn heavily
- monitor
increase(awa_maintenance_prune_attempts_total{awa_ring_outcome=~"blocked|error"}[15m])for contention or failures, alongside successfulprunedoutcomes, ring generations and accumulated rows. A positive increase can persist while recycling succeeds. If recycling has stalled, inspect long transactions and conflicting locks; for short readers, considerprune_lock_timeoutand its effect on waiting queries - alert when
awa_maintenance_prune_pinned_ready_age_secondsstays above an hour: a paused, unserved or strictly starved lane is keeping the queue ring from being reclaimed. See queue ring stops wrapping behind an unclaimed lane
Queue-storage depth metrics are designed for high-cadence monitoring. The worker records available from lane-head cursor differences and probes queue lag from the next claimable row per lane, instead of running the exact queue_counts ready-row scan on every metrics tick. Use the admin API or CLI when you need exact queue counts for an operator action.
The nightly MVCC benchmark exists to catch changes that make this failure mode worse, but it is not a substitute for keeping the primary free of stale snapshots.
Docker¶
CLI / UI Container¶
The repository already includes Dockerfiles for the awa CLI:
Build the CLI image locally:
Run migrations:
docker run --rm \
-e DATABASE_URL="$DATABASE_URL" \
awa-cli \
--database-url "$DATABASE_URL" migrate
Run the web UI:
docker run --rm -p 3000:3000 \
-e DATABASE_URL="$DATABASE_URL" \
awa-cli \
--database-url "$DATABASE_URL" serve --host 0.0.0.0 --port 3000
Worker Containers¶
Your worker image is your application image. Build and run it the same way you deploy the rest of your Rust or Python service, with Awa embedded in-process.
Recommended pattern:
- one image for your worker application
- one CLI/UI image for migrations and
awa serve - one Postgres service
Kubernetes¶
Deploy workers as a Deployment. Awa does not require sticky sessions, local disks, or a sidecar.
Guidelines:
- scale replicas horizontally;
SKIP LOCKEDhandles concurrent claiming - split queues across deployments when you want isolation
- remember only one replica will be the maintenance leader at a time
- set
terminationGracePeriodSecondsabove your shutdown drain timeout
Example worker deployment settings:
If your code calls:
or:
then 40 seconds is a reasonable Kubernetes grace period.
Health Checks¶
Workers ship an opt-in HTTP health listener (#368).
Enable it with the AWA_HEALTH_ADDR environment variable or the builder:
let client = Client::builder(pool)
.queue("email", QueueConfig::default())
.health_addr("0.0.0.0:8321".parse()?) // or AWA_HEALTH_ADDR=0.0.0.0:8321
.build()?;
Two endpoints:
GET /healthz— process liveness.200for as long as the runtime is up, including during graceful drain — a draining worker must not be killed by its liveness probe.GET /readyz— readiness.200only when the worker can do useful work: Postgres reachable, schema at least the binary's expected version, every queue's claim loop ticking, heartbeat + maintenance services alive, and not shutting down. Otherwise503with a JSON body naming each failing check (postgres_connected,schema_version,expected_schema_version,schema_compatible,poll_loop_alive,heartbeat_alive,maintenance_alive,shutting_down).leaderis also reported but is informational — it never gates readiness.
Kubernetes probes against the listener:
spec:
template:
spec:
containers:
- name: worker
env:
- name: AWA_HEALTH_ADDR
value: "0.0.0.0:8321"
ports:
- name: health
containerPort: 8321
livenessProbe:
httpGet: { path: /healthz, port: health }
periodSeconds: 10
readinessProbe:
httpGet: { path: /readyz, port: health }
periodSeconds: 5
The listener is deliberately minimal (no TLS, no auth): bind it to the pod IP or loopback and keep it off public ingress.
The in-process API remains available without the listener:
- Rust:
client.health_check().await - Python:
await client.health_check()
awa health — probe-less environments¶
For cron drivers, systemd ExecStartPre, or CI smoke checks, awa health
answers the cluster-level question from the database alone — reachable,
schema migrated for this binary, and fleet heartbeat counts:
$ awa health --database-url "$DATABASE_URL"
ready: yes
postgres: connected
schema version: 39 (binary expects >= 39)
storage: active (queue_storage)
live runtimes: 3 (0 unhealthy)
--json emits the same report as a machine-readable object. Exit code 0
means ready; 1 means unreachable or unmigrated. Fleet numbers are
informational — an idle cluster with zero workers is still "ready" for
producers and migrations.
awa serve is separate; its HTTP server is for the web UI/admin API, not for worker liveness.
Rolling Deployments¶
Let old pods drain with shutdown(...), and keep terminationGracePeriodSeconds slightly above that drain timeout. Release-specific procedures are listed in the migration guide; for the current release boundary, follow Upgrading from 0.6 to 0.7. Representation changes use the expand, flip, and contract policy in ADR-041.
Queue Isolation Patterns¶
Common patterns:
- critical vs bulk queues in separate deployments
- Python workers for I/O-heavy integration queues, Rust workers for high-throughput internal queues
- a dedicated deployment for cron-heavy workloads
Use:
- Rust:
ClientBuilder::queue("name", QueueConfig { ... }) - Python:
client.start([("name", 10)])or dict configs in weighted mode
Separate deployments isolate worker capacity and handler failures, but not storage reclamation: every queue in the database still shares one queue ring, one claim ring and one maintenance leader. To isolate those, see Storage isolation.
Storage Isolation¶
On queue storage, all queues in a database share one set of rings. A queue that stops being claimed while it holds ready jobs, such as a paused queue, a queue with no running workers, or a lower priority starved under strict priority, stops queue-ring reclamation for every queue in that database until it is claimed again. Claims that starve ring prune are also per ring: every runtime that claims from the database adds to the overlap that keeps prune from its lock. To keep workloads with different risk profiles from stalling each other's reclamation, for example long-paused, batch or strictly prioritised queues apart from latency-sensitive ones, give each its own database.
The boundary is the database, not the queue-storage schema. A database holds one Awa installation: the awa control schema records a single active queue-storage schema, every producer path (Rust and Python inserts, awa.insert_job and the other SQL compatibility functions, cron, follow-up enqueues, retries, the DLQ and admin operations) writes to that schema, and a runtime configured with a queue_storage_schema other than the active one refuses to start. A custom queue-storage schema replaces the default as the one active substrate; it does not run beside it.
Each database has its own:
- queue, lease and claim rings, so a lane pinned in one database does not hold back reclamation in another, and claims in one database do not lock another's partitions
- maintenance leader, elected by an advisory lock that Postgres scopes to the database, so a long rescue pass or a stalled prune in one database does not delay rotation, promotion or rescue in another
- cron schedules, unique-key claims, DLQ, queue controls and admin view
The databases can share one Postgres server. Isolation then stops at the server: CPU, I/O, WAL, max_connections and autovacuum workers are shared, and a replication slot whose xmin is set (in pg_replication_slots), or a standby with hot_standby_feedback, holds back vacuum of table rows in every database on it. A slot that only retains WAL, or a logical slot with only catalog_xmin set, does not hold back vacuum of Awa's table rows. Within one database, a stalled lane still pins that database's ring; the fixes in the troubleshooting guide still apply there.
To set it up:
- Create a database per isolation domain and run
awa migrateagainst each. Every release's migrations, and any staged storage transition, must then be applied to each database. - Assign each queue to exactly one database. Run its workers with a pool connected to that database:
Client::builder(pool)in Rust,awa.AsyncClient(database_url)in Python. Every runtime contends for maintenance leadership, so each database's leader is one of the runtimes connected to it. - Route producers by queue: insert with a pool or client connected to the database that serves the queue. A job inserted into a database whose runtimes do not serve its queue waits there; nothing moves it to another database. Follow-up jobs that a handler enqueues, including lifecycle-hook follow-ups, go to the handler's own database, so keep a workflow's queues in one database.
- Declare each unowned periodic job on every client connected to the database that serves its queue. Any of them may lead, and a leader writes to
awa.cron_jobsonly the unowned schedules its own client declares, so a schedule that only some clients declare is not registered until one of them leads. Use owner-scoped schedules to make registration independent of which client leads. - When a queue moves from one database to another, its jobs and schedules in the old database stay there:
move_queuecannot move jobs between databases. Keep a worker for the queue connected to the old database until its jobs there finish, or drain them. Retire each unowned schedule the queue's jobs come from in the old database (awa cron retire <name> --applywith--database-urlset to that database) as the queue moves, or the old database's leader keeps enqueuing them there; owned schedules follow their owner's reconciliation in each database. - Run
awa serveand the otherawacommands once per database, with--database-urlorDATABASE_URLset to it; one UI shows one database.
Costs:
- more partitions and maintenance work in total, since each database keeps its own rings and leader
- a separate pool, and more server connections, in every process that serves or enqueues into more than one database
- no transaction spans databases: a handler or a producer transaction can enqueue atomically only into its own database, unique keys deduplicate only within one database, and a job cannot be moved between databases with
move_queue - Awa's metrics carry no database attribute, so the domains can be told apart only by the resource attributes of the processes that serve them: run each database's workers in processes of their own if you need per-database metrics
When not to bother: if every queue is claimed continuously, queues that mix priorities under a sustained backlog have a non-zero priority_aging_interval, queues are not left paused or unserved with jobs waiting, and you alert on awa.maintenance.prune.pinned_ready_age, on prune attempts with outcome blocked, and on awa.ring.generation that stops advancing, one database is simpler and a stall is caught before it costs much storage. pinned_ready_age catches a lane pinning the ring; it does not report overlapping claims that keep prune blocked, which only the blocked count and a flat ring generation show.
Web UI Deployment¶
awa serve binds to 127.0.0.1:3000 by default. In containers, set:
The UI is read/write admin surface. Put it behind your normal authentication, network policy, and ingress controls.
For a UI pinned to a less-trusted network or shared with stakeholders who shouldn't trigger retries, cancels, or pauses, run with --read-only (or set AWA_READ_ONLY=1):
This forces read-only even when the Postgres connection is fully writable — the UI hides mutation buttons and every mutation endpoint returns 503. See Read-only mode for the tradeoff versus pointing at a read replica.
If you expose the callback receiver endpoints for HttpWorker, also configure AWA_CALLBACK_HMAC_SECRET (or --callback-hmac-secret) so awa-ui verifies X-Awa-Signature on callback requests. See HTTP workers and callback signatures for the worker, function, and receiver contract.
Next¶
- Deploying on managed Postgres — Cloud SQL / AlloyDB specifics, sizing data, IAM grants, auth-proxy sidecar
- PostgreSQL roles and privileges
- Migration guide
- Configuration
- Troubleshooting