Skip to content

Deployment Guide

This guide covers how to deploy Awa workers and the web UI with Docker or Kubernetes.

Deployment Model

Awa workers are stateless. Queue state lives in Postgres. A worker process typically contains:

  • one dispatcher per configured queue
  • one heartbeat loop
  • the maintenance service: on the leader, a recovery loop (promotion and rescue) and a hygiene loop (cleanup, cron sync, rotation and the rest)
  • one elected leader per database for rescue, promotion, cleanup, and cron evaluation

That means you scale by running more worker processes, not by adding local state.

Process Roles

In production, treat these as separate concerns:

  • application worker: your Rust or Python service embedding Awa and calling Client::start() or client.start()
  • migration/admin/UI process: awa CLI for migrate, storage, job, queue, cron, and serve
  • PostgreSQL: the only required external dependency

awa serve is an operator UI and admin API. It is not the worker runtime.

Queue Storage Cutover

Fresh 0.6 installs use queue storage as the worker engine. Existing 0.5.x clusters must follow the staged storage-transition procedure; Migrations covers the general migration contract and tooling.

Operationally:

  • queue storage is the 0.6 worker engine
  • canonical tables remain in the schema for migration, rollback boundaries, and compatibility SQL surfaces
  • while state is canonical or prepared, 0.5.x and 0.6 pods may coexist and all writes/execution remain canonical
  • after enter-mixed-transition, new writes route to queue storage while 0.6 drain workers finish the canonical backlog
  • once queue-storage rows exist, rollback to a 0.5.x-only fleet is not supported; finish the transition or restore from backup

For a 0.5.x cluster, do not stop canonical workers and start queue-storage workers as a separate manual cutover. Use the storage commands so producer routing, canonical drain, queue-storage executor readiness, and rollback interlocks move together.

Current Rust and Python worker starts default to the correct effective engine for the storage-transition state. Only set queue_storage_schema / ClientBuilder::queue_storage(...) when you need to override the default schema name or segment sizing.

Connection Pool Sizing

Practical starting point, based on the current runtime internals:

  • each runtime (client) keeps one LISTEN connection that carries enqueue wakeups for all of its queues
  • each runtime (client) keeps a second LISTEN connection for admin-cancel notifications
  • the elected leader holds one advisory-lock connection while it remains leader
  • claim, heartbeat, completion, cleanup, and handler SQL all share the same pool

Conservative starting heuristic, not a hard requirement:

max_connections >= sum(queue.claimers) + 5

With the default claimers = 1, this is the old shortcut: max_connections >= queue_count + 5.

Then add headroom for:

  • handler code that also talks to Postgres
  • producer traffic sharing the same pool
  • health checks and admin queries

Examples:

  • one queue, light worker-only process: 10 is a reasonable start
  • three queues plus handler SQL: start around 12-20
  • combined API + worker process: either use separate pools or size for both workloads explicitly

The Python client defaults to max_connections=10. awa serve defaults to a pool of 10 connections (configurable via --pool-max / AWA_POOL_MAX). Other CLI subcommands use a single connection.

Idle connections and Postgres memory

Each pooled connection is a Postgres backend with its own relation, plan and function caches, and those caches only grow while the backend lives. Queue storage has several hundred partitions and indexes, and the claim path runs per-slot functions, so a backend that has run claims, completions and maintenance holds tens of MB of backend-local memory. Postgres memory for an Awa process is roughly the number of open backends times that size.

The pool keeps every connection it opened at the busiest moment. sqlx hands out the connection that has been idle longest (its idle connections form a FIFO queue), so even an idle runtime, which acquires a connection several times a second for polling and maintenance, touches every pooled connection within seconds. While a runtime keeps using the pool this often, idle_timeout rarely closes any of them, and each one ends up having run every statement Awa issues. A process that holds a pool but seldom uses it, such as a client with no workers started, does have idle connections closed.

To return memory after busy periods, set max_lifetime on the pool so connections are recycled. A connection past its lifetime is closed when it is next released or when the pool's reaper finds it idle, and the pool opens replacement connections as needed to maintain min_connections or to meet concurrent demand. The pool can therefore shrink toward what concurrent demand needs, but not below min_connections. A replacement's caches build up again as it is used.

let pool = PgPoolOptions::new()
    .max_connections(20)
    .max_lifetime(Duration::from_secs(300))
    .connect(&database_url)
    .await?;

sqlx's default lifetime is 30 minutes. Shorter lifetimes reconnect more often; at a few minutes the reconnect cost is negligible next to the memory returned.

awa serve sets these from --pool-max-lifetime / AWA_POOL_MAX_LIFETIME (default 1800 seconds) and keeps at least --pool-min / AWA_POOL_MIN connections (default 2). The Python client takes the lifetime as the keyword argument max_lifetime_seconds (default 1800.0, sqlx's default). None disables it, as PgPoolOptions::max_lifetime(None) does in Rust, so connections stay open until closed for another reason. Zero, negative and non-finite values raise ValidationError.

client = awa.AsyncClient(database_url, max_connections=20, max_lifetime_seconds=300)

Do not disable the statement cache with PgConnectOptions::statement_cache_capacity(0) to save memory. In sqlx 0.9 every query then leaves a named prepared statement on the server that is never closed, and backend memory grows without bound.

Primary Database Hygiene

Queue storage keeps the main ready path append-only, but Awa is still a high-churn Postgres workload. The main operational pitfall is no longer one giant mutable queue heap; it is long-lived readers or stale transactions that block lease or segment prune while the maintenance leader keeps rotating forward.

Recommended practice:

  • keep analytical reads and admin transactions short on the primary
  • run long-lived reporting queries against a replica when possible
  • avoid leaving sessions idle in transaction
  • monitor pg_stat_activity for long transactions
  • monitor pg_stat_user_tables on the active queue-storage schema; ready, tombstone, done, and terminal-delta segments should stay near zero dead tuples, lease segments may spike within the rotation window but should collapse after prune, and attempt_state should roughly track live long-running attempts
  • tune autovacuum for the database if the lease tables churn heavily
  • monitor increase(awa_maintenance_prune_attempts_total{awa_ring_outcome=~"blocked|error"}[15m]) for contention or failures, alongside successful pruned outcomes, ring generations and accumulated rows. A positive increase can persist while recycling succeeds. If recycling has stalled, inspect long transactions and conflicting locks; for short readers, consider prune_lock_timeout and its effect on waiting queries
  • alert when awa_maintenance_prune_pinned_ready_age_seconds stays above an hour: a paused, unserved or strictly starved lane is keeping the queue ring from being reclaimed. See queue ring stops wrapping behind an unclaimed lane

Queue-storage depth metrics are designed for high-cadence monitoring. The worker records available from lane-head cursor differences and probes queue lag from the next claimable row per lane, instead of running the exact queue_counts ready-row scan on every metrics tick. Use the admin API or CLI when you need exact queue counts for an operator action.

The nightly MVCC benchmark exists to catch changes that make this failure mode worse, but it is not a substitute for keeping the primary free of stale snapshots.

Docker

CLI / UI Container

The repository already includes Dockerfiles for the awa CLI:

Build the CLI image locally:

docker build -f docker/Dockerfile -t awa-cli .

Run migrations:

docker run --rm \
  -e DATABASE_URL="$DATABASE_URL" \
  awa-cli \
  --database-url "$DATABASE_URL" migrate

Run the web UI:

docker run --rm -p 3000:3000 \
  -e DATABASE_URL="$DATABASE_URL" \
  awa-cli \
  --database-url "$DATABASE_URL" serve --host 0.0.0.0 --port 3000

Worker Containers

Your worker image is your application image. Build and run it the same way you deploy the rest of your Rust or Python service, with Awa embedded in-process.

Recommended pattern:

  • one image for your worker application
  • one CLI/UI image for migrations and awa serve
  • one Postgres service

Kubernetes

Deploy workers as a Deployment. Awa does not require sticky sessions, local disks, or a sidecar.

Guidelines:

  • scale replicas horizontally; SKIP LOCKED handles concurrent claiming
  • split queues across deployments when you want isolation
  • remember only one replica will be the maintenance leader at a time
  • set terminationGracePeriodSeconds above your shutdown drain timeout

Example worker deployment settings:

spec:
  replicas: 3
  template:
    spec:
      terminationGracePeriodSeconds: 40

If your code calls:

client.shutdown(Duration::from_secs(30)).await;

or:

await client.shutdown(timeout_ms=30000)

then 40 seconds is a reasonable Kubernetes grace period.

Health Checks

Workers ship an opt-in HTTP health listener (#368). Enable it with the AWA_HEALTH_ADDR environment variable or the builder:

let client = Client::builder(pool)
    .queue("email", QueueConfig::default())
    .health_addr("0.0.0.0:8321".parse()?)   // or AWA_HEALTH_ADDR=0.0.0.0:8321
    .build()?;

Two endpoints:

  • GET /healthz — process liveness. 200 for as long as the runtime is up, including during graceful drain — a draining worker must not be killed by its liveness probe.
  • GET /readyz — readiness. 200 only when the worker can do useful work: Postgres reachable, schema at least the binary's expected version, every queue's claim loop ticking, heartbeat + maintenance services alive, and not shutting down. Otherwise 503 with a JSON body naming each failing check (postgres_connected, schema_version, expected_schema_version, schema_compatible, poll_loop_alive, heartbeat_alive, maintenance_alive, shutting_down). leader is also reported but is informational — it never gates readiness.

Kubernetes probes against the listener:

spec:
  template:
    spec:
      containers:
        - name: worker
          env:
            - name: AWA_HEALTH_ADDR
              value: "0.0.0.0:8321"
          ports:
            - name: health
              containerPort: 8321
          livenessProbe:
            httpGet: { path: /healthz, port: health }
            periodSeconds: 10
          readinessProbe:
            httpGet: { path: /readyz, port: health }
            periodSeconds: 5

The listener is deliberately minimal (no TLS, no auth): bind it to the pod IP or loopback and keep it off public ingress.

The in-process API remains available without the listener:

  • Rust: client.health_check().await
  • Python: await client.health_check()

awa health — probe-less environments

For cron drivers, systemd ExecStartPre, or CI smoke checks, awa health answers the cluster-level question from the database alone — reachable, schema migrated for this binary, and fleet heartbeat counts:

$ awa health --database-url "$DATABASE_URL"
ready:              yes
postgres:           connected
schema version:     39 (binary expects >= 39)
storage:            active (queue_storage)
live runtimes:      3 (0 unhealthy)

--json emits the same report as a machine-readable object. Exit code 0 means ready; 1 means unreachable or unmigrated. Fleet numbers are informational — an idle cluster with zero workers is still "ready" for producers and migrations.

awa serve is separate; its HTTP server is for the web UI/admin API, not for worker liveness.

Rolling Deployments

Let old pods drain with shutdown(...), and keep terminationGracePeriodSeconds slightly above that drain timeout. Release-specific procedures are listed in the migration guide; for the current release boundary, follow Upgrading from 0.6 to 0.7. Representation changes use the expand, flip, and contract policy in ADR-041.

Queue Isolation Patterns

Common patterns:

  • critical vs bulk queues in separate deployments
  • Python workers for I/O-heavy integration queues, Rust workers for high-throughput internal queues
  • a dedicated deployment for cron-heavy workloads

Use:

  • Rust: ClientBuilder::queue("name", QueueConfig { ... })
  • Python: client.start([("name", 10)]) or dict configs in weighted mode

Separate deployments isolate worker capacity and handler failures, but not storage reclamation: every queue in the database still shares one queue ring, one claim ring and one maintenance leader. To isolate those, see Storage isolation.

Storage Isolation

On queue storage, all queues in a database share one set of rings. A queue that stops being claimed while it holds ready jobs, such as a paused queue, a queue with no running workers, or a lower priority starved under strict priority, stops queue-ring reclamation for every queue in that database until it is claimed again. Claims that starve ring prune are also per ring: every runtime that claims from the database adds to the overlap that keeps prune from its lock. To keep workloads with different risk profiles from stalling each other's reclamation, for example long-paused, batch or strictly prioritised queues apart from latency-sensitive ones, give each its own database.

The boundary is the database, not the queue-storage schema. A database holds one Awa installation: the awa control schema records a single active queue-storage schema, every producer path (Rust and Python inserts, awa.insert_job and the other SQL compatibility functions, cron, follow-up enqueues, retries, the DLQ and admin operations) writes to that schema, and a runtime configured with a queue_storage_schema other than the active one refuses to start. A custom queue-storage schema replaces the default as the one active substrate; it does not run beside it.

Each database has its own:

  • queue, lease and claim rings, so a lane pinned in one database does not hold back reclamation in another, and claims in one database do not lock another's partitions
  • maintenance leader, elected by an advisory lock that Postgres scopes to the database, so a long rescue pass or a stalled prune in one database does not delay rotation, promotion or rescue in another
  • cron schedules, unique-key claims, DLQ, queue controls and admin view

The databases can share one Postgres server. Isolation then stops at the server: CPU, I/O, WAL, max_connections and autovacuum workers are shared, and a replication slot whose xmin is set (in pg_replication_slots), or a standby with hot_standby_feedback, holds back vacuum of table rows in every database on it. A slot that only retains WAL, or a logical slot with only catalog_xmin set, does not hold back vacuum of Awa's table rows. Within one database, a stalled lane still pins that database's ring; the fixes in the troubleshooting guide still apply there.

To set it up:

  1. Create a database per isolation domain and run awa migrate against each. Every release's migrations, and any staged storage transition, must then be applied to each database.
  2. Assign each queue to exactly one database. Run its workers with a pool connected to that database: Client::builder(pool) in Rust, awa.AsyncClient(database_url) in Python. Every runtime contends for maintenance leadership, so each database's leader is one of the runtimes connected to it.
  3. Route producers by queue: insert with a pool or client connected to the database that serves the queue. A job inserted into a database whose runtimes do not serve its queue waits there; nothing moves it to another database. Follow-up jobs that a handler enqueues, including lifecycle-hook follow-ups, go to the handler's own database, so keep a workflow's queues in one database.
  4. Declare each unowned periodic job on every client connected to the database that serves its queue. Any of them may lead, and a leader writes to awa.cron_jobs only the unowned schedules its own client declares, so a schedule that only some clients declare is not registered until one of them leads. Use owner-scoped schedules to make registration independent of which client leads.
  5. When a queue moves from one database to another, its jobs and schedules in the old database stay there: move_queue cannot move jobs between databases. Keep a worker for the queue connected to the old database until its jobs there finish, or drain them. Retire each unowned schedule the queue's jobs come from in the old database (awa cron retire <name> --apply with --database-url set to that database) as the queue moves, or the old database's leader keeps enqueuing them there; owned schedules follow their owner's reconciliation in each database.
  6. Run awa serve and the other awa commands once per database, with --database-url or DATABASE_URL set to it; one UI shows one database.

Costs:

  • more partitions and maintenance work in total, since each database keeps its own rings and leader
  • a separate pool, and more server connections, in every process that serves or enqueues into more than one database
  • no transaction spans databases: a handler or a producer transaction can enqueue atomically only into its own database, unique keys deduplicate only within one database, and a job cannot be moved between databases with move_queue
  • Awa's metrics carry no database attribute, so the domains can be told apart only by the resource attributes of the processes that serve them: run each database's workers in processes of their own if you need per-database metrics

When not to bother: if every queue is claimed continuously, queues that mix priorities under a sustained backlog have a non-zero priority_aging_interval, queues are not left paused or unserved with jobs waiting, and you alert on awa.maintenance.prune.pinned_ready_age, on prune attempts with outcome blocked, and on awa.ring.generation that stops advancing, one database is simpler and a stall is caught before it costs much storage. pinned_ready_age catches a lane pinning the ring; it does not report overlapping claims that keep prune blocked, which only the blocked count and a flat ring generation show.

Web UI Deployment

awa serve binds to 127.0.0.1:3000 by default. In containers, set:

awa --database-url "$DATABASE_URL" serve --host 0.0.0.0 --port 3000

The UI is read/write admin surface. Put it behind your normal authentication, network policy, and ingress controls.

For a UI pinned to a less-trusted network or shared with stakeholders who shouldn't trigger retries, cancels, or pauses, run with --read-only (or set AWA_READ_ONLY=1):

awa --database-url "$DATABASE_URL" serve --read-only

This forces read-only even when the Postgres connection is fully writable — the UI hides mutation buttons and every mutation endpoint returns 503. See Read-only mode for the tradeoff versus pointing at a read replica.

If you expose the callback receiver endpoints for HttpWorker, also configure AWA_CALLBACK_HMAC_SECRET (or --callback-hmac-secret) so awa-ui verifies X-Awa-Signature on callback requests. See HTTP workers and callback signatures for the worker, function, and receiver contract.

Next