Skip to content

ADR-014: Structured Progress and Metadata Persistence

Status

Accepted. This ADR records the original canonical-storage implementation. Queue-storage deployments keep the user-facing progress API but store mutable progress snapshots in {schema}.attempt_state; see Relationship to ADR-019.

Context

Long-running handlers had no way to report progress or update checkpoint metadata during execution. The only mid-execution signal was the binary heartbeat (alive/dead). This made it impossible to:

  • Show users what percentage of a batch job is complete
  • Checkpoint work so a retried job can resume rather than restart
  • Attach handler-specific metadata (e.g., last processed ID) that survives across attempts

Decision

Add a nullable progress JSONB column to both jobs_hot and scheduled_jobs. The column stores a structured object:

{
  "percent": 50,
  "message": "Processing batch 5 of 10",
  "metadata": { "last_processed_id": 1234 }
}

Why JSONB instead of separate columns

A single JSONB column avoids namespace collision with the user-owned metadata column (which is free-form, set at insert time via multiple paths including COPY ingestion and cron metadata). It also allows the progress schema to evolve without migrations.

Write path: generation-buffered, three flush modes

Handlers write to an in-memory ProgressState buffer shared with the heartbeat service via Arc<std::sync::Mutex<...>>. The buffer uses a generation counter for change tracking:

  1. Heartbeat piggyback — on every heartbeat cycle, jobs with pending progress updates get a combined SET heartbeat_at = now(), progress = v.progress query. Jobs without changes get the original heartbeat-only query. At most two queries per cycle regardless of job count.

  2. State-transition atomic — when complete_job() runs, the latest progress snapshot is included in the same UPDATE that transitions state. If the UPDATE fails (stale run_lease), the progress is lost with the transition — which is correct.

  3. Explicit flush — flush_progress() performs a direct UPDATE jobs_hot SET progress = $2 WHERE id = $1 AND run_lease = $3. This is the reliable path for critical checkpoints and does not return until the write is confirmed or the job is no longer running.

Why std::sync::Mutex, not tokio::Mutex

The critical section is pure in-memory work: JSON cloning, integer increment, Option swap. No .await points are ever held under the lock. std::sync::Mutex is cheaper and simpler for this pattern.

Lifecycle semantics

Completed jobs clear progress to NULL (ephemeral — the job succeeded and progress is no longer relevant). All other transitions preserve progress: retries and snoozes preserve it for checkpoint resumption, failures and cancellations preserve it for operator inspection. Rescue operations (stale heartbeat, expired deadline, callback timeout) preserve progress because the rescue queries in maintenance.rs target awa.jobs_hot directly and don't include progress in the SET clause — PostgreSQL leaves unmentioned columns unchanged.

Shallow merge for metadata

update_metadata performs a shallow merge on the metadata sub-object: top-level keys overwrite, nested objects are replaced (not deep-merged). This is an in-memory operation — no JSONB || at the database level. The entire progress JSON is written as a unit on flush.

Consequences

  • Zero overhead for jobs that don't use progress. The heartbeat partitions jobs into two tiers; jobs without pending progress use the original query unchanged.
  • No additional round-trips. Progress writes piggyback on existing heartbeats rather than requiring separate queries.
  • Checkpoint resumption is opt-in. Handlers read ctx.job.progress.metadata to pick up where they left off. Nothing forces handlers to use progress.
  • Unbounded metadata growth risk. There is no enforced size limit on the progress JSONB. A pathological handler could accumulate large values via repeated update_metadata calls. A size guard should be added if this becomes a production concern.

Alternatives Considered

  • Separate progress_percent / progress_message columns. Simpler schema but inflexible — no room for handler-specific checkpoint data without another column.
  • External progress store (Redis). Violates the Postgres-only principle (ADR-001).
  • Write progress on every set_progress call. Too many round-trips for high-frequency updates. The buffered approach coalesces writes naturally.

Relationship to ADR-019

ADR-019 moved progress storage off the jobs_hot / scheduled_jobs progress columns described here and onto {schema}.attempt_state (keyed by (job_id, run_lease)) plus the deferred / terminal payload for persisted snapshots. The user-facing progress model — set_progress, update_metadata, flush_progress, generation-based buffering, lifecycle semantics — is unchanged. Short jobs now complete without ever allocating an attempt_state row, so the buffered approach matters even more: only long-running or callback-heavy jobs pay the upsert cost. See ADR-019.