Operator troubleshooting index

Start with the policy, then follow its reason to the failing boundary.

First five commands

Set the policy name and namespace once:

NAMESPACE=default
POLICY=my-policy

Then collect the controller's view before changing anything:

kubectl get pgr "$POLICY" --namespace "$NAMESPACE" -o wide
kubectl describe pgr "$POLICY" --namespace "$NAMESPACE"
kubectl get pgr "$POLICY" --namespace "$NAMESPACE" \
  -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{" reason="}{.reason}{" -- "}{.message}{"\n"}{end}'
kubectl get pgplan --namespace "$NAMESPACE"
kubectl get events --namespace "$NAMESPACE" \
  --field-selector involvedObject.name="$POLICY" \
  --sort-by='.lastTimestamp'

status.conditions[].reason is the primary index below. status.last_error usually preserves the underlying PostgreSQL, Kubernetes, or connection error:

kubectl get pgr "$POLICY" --namespace "$NAMESPACE" \
  -o jsonpath='{.status.last_error}{"\n"}'

For controller-side detail, read the operator log:

kubectl logs --namespace pgroles-system \
  -l app.kubernetes.io/instance=pgroles-operator \
  --all-containers --tail=200

Adjust pgroles-system and the instance label if you used different Helm release or namespace names.

Reason index

No status or no reconcile

If kubectl does not recognize PostgresPolicy, install or update the CRDs. Helm installs CRDs on the first install but does not upgrade them on helm upgrade:

kubectl get crd postgrespolicies.pgroles.io
kubectl get crd postgrespolicyplans.pgroles.io
helm status pgroles-operator --namespace pgroles-system
kubectl get pods --namespace pgroles-system \
  -l app.kubernetes.io/instance=pgroles-operator

If the object exists but has no status, inspect the operator pod and logs. Check that its ServiceAccount can get, list, watch, and patch status for the CRDs, and can read Secrets in the policy namespace. An admission error from kubectl apply happens before the controller sees the object; read that error directly rather than waiting for a status condition.

Secret errors

SecretMissing means the named Secret or key is absent. Connection URL Secrets default to the key DATABASE_URL; role password keys have their own defaults. Confirm the reference, namespace, and available keys without printing secret values:

kubectl get pgr "$POLICY" --namespace "$NAMESPACE" \
  -o jsonpath='{.spec.connection}{"\n"}'
kubectl get secret --namespace "$NAMESPACE"
SECRET=quick-start-database
kubectl get secret "$SECRET" --namespace "$NAMESPACE" \
  -o go-template='{{range $key, $_ := .data}}{{$key}}{{"\n"}}{{end}}'

SecretFetchFailed means the Kubernetes read itself failed, or generated-role password storage failed. Check the operator ServiceAccount permissions and the specific API error in last_error. Updating a referenced Secret's resourceVersion triggers reconnection automatically.

Connection and authentication

For InvalidConnectionParams, inspect last_error for the exact field. Common causes are an empty Secret value, invalid port, invalid sslMode, or malformed URL. For DatabaseConnectionFailed, test four boundaries from inside the cluster: DNS resolution, network reachability, TLS requirements, and the PostgreSQL username/password/database combination. A host that works on your laptop may not be reachable from the operator pod.

For GcpAuthFailed, verify the Kubernetes ServiceAccount annotation, Workload Identity binding, Cloud SQL login permission, IAM database username, and token endpoint response. For SetRoleFailed, the login succeeded but connection.params.setRole did not: the authenticated identity must be a member of the target PostgreSQL role.

See database connections for every connection shape and the Cloud SQL IAM flow.

Policy validation

InvalidSpec covers policy rules that are more contextual than the CRD schema: manifest expansion, interval parsing, password rules, and other cross-field constraints. The condition message names the invalid field or relationship. Fix the source manifest and re-apply it; do not edit status.

If kubectl apply rejects the YAML, that is CRD admission rather than InvalidSpec. Useful distinctions:

• unknown or misspelled fields are rejected by the structural schema
• unquoted role config values are rejected because those values are strings
• invalid combinations such as a password on a non-login role may be caught by CRD validation or by the reconcile-time validator

Use the PostgresPolicy resource and the manifest reference to check field spelling and semantics.

Executor privileges

InsufficientPrivileges means the database credential is valid but cannot perform the requested inspection or mutation. The PostgreSQL error in last_error identifies the failed operation, such as permission denied to create role.

Do not respond by granting broad privileges blindly. Map the failed statement to the executor requirements: role management normally needs CREATEROLE and sometimes ADMIN OPTION; object grants need ownership or WITH GRANT OPTION; default privileges need membership in their owner role. The complete matrix and bootstrap SQL are in executor privileges.

Missing database objects

Before issuing DDL, the operator checks external schema references. Schemas declared in spec.schemas are excluded because the operator can create them; schemas referenced only by grants or default privileges must already exist.

MissingDatabaseObject therefore usually means one of three things:

• the policy references a misspelled or not-yet-migrated schema
• the operator should own the schema, but it is not declared in spec.schemas
• the connection points at the wrong PostgreSQL database

The same reason can classify PostgreSQL undefined-object errors that pass the preflight. Read last_error, then create or declare the object, remove the reference, or correct the database connection.

Wildcard grants

UnsatisfiableWildcardGrant is a safety stop. A wildcard matched at least one table, sequence, or function where the executor lacks the ability to grant the requested privilege. The operator does not create a partial plan for that reconcile.

The condition message includes example objects and missing privileges. Give the executor ownership or the required grant option, narrow the wildcard, or manage those objects under a different ownership boundary. See grants and privileges for wildcard semantics.

Policy conflicts

ConflictingPolicy means another PostgresPolicy targeting the same database has overlapping ownership claims. The condition message names the other namespace/policy and summarizes the overlap.

Keep one policy per database and credential boundary where possible. Otherwise make their roles, schemas, grants, memberships, and other ownership claims disjoint. Changing reconcile timing does not solve an ownership conflict.

Unsafe role drops

UnsafeRoleDrops blocks a role removal when PostgreSQL dependencies make a plain DROP ROLE unsafe. Decide explicitly where owned objects should go, then use a retirement with reassign_owned_to and/or drop_owned as appropriate. If active sessions are the issue, a retirement can terminate them when the executor has the required privilege.

Review the generated SQL in observe mode before approving destructive retirement steps. Do not remove the safety blocker merely to make the policy green.

A plan does not execute

First inspect all three execution gates:

kubectl get pgr "$POLICY" --namespace "$NAMESPACE" \
  -o jsonpath='suspend={.spec.suspend}{" mode="}{.spec.mode}{" approval="}{.spec.approval}{"\n"}'

• suspend: true stops reconciliation entirely
• mode: observe computes plans but never executes mutating SQL
• mode: apply with approval: manual waits for a terminal Approved decision on the current plan's status
• mode: apply with approval: auto applies without a human gate

Follow the current plan and inspect its phase and conditions:

PLAN="$(kubectl get pgr "$POLICY" --namespace "$NAMESPACE" \
  -o jsonpath='{.status.current_plan_ref.name}')"
kubectl get pgplan "$PLAN" --namespace "$NAMESPACE" -o yaml

A decision recorded on an observe-mode policy's plan produces ApprovalIgnored; switch the policy to mode: apply if you intend SQL to execute. ApprovalUnset means the operator inferred an approval mode from spec.mode; set approval explicitly because that inference is deprecated.

If policy or database state changed after review, the approved plan may become Superseded. Review and approve the newly referenced plan. If a plan was rejected, its replacement is created on the next reconcile rather than in the same cycle. Reject a plan by writing a terminal Denied condition and decidedBy to its status subresource; see plan and approval for the exact command and the full lifecycle.

A superseded plan names its cause in the Superseded=True condition message — effects changed, effects cleared, replaced by a newer plan, the policy stopped referencing it, another candidate's content was promoted and applied (reason SupersededByPromotion), or the target moved (reason TargetChanged, TargetIdentityUnavailable, or TargetIdentityAppeared).

Transient and infrastructure failures

ApplyFailed and DatabaseInspectionFailed preserve the underlying SQL error in last_error and use transient backoff. Check database availability, failovers, statement cancellation, and network stability before changing the policy.

KubernetesApiError points to API reachability, authorization, or an update conflict. PlanSqlStorageFailed means the operator could not persist a large SQL preview in its ConfigMap-backed storage. Check API errors, namespace quota, ConfigMap permissions, and object-size limits. The operator never executes SQL stored in the ConfigMap; it re-renders from current state before apply.

Use transient_failure_count to distinguish a repeating infrastructure problem from a single retry:

kubectl get pgr "$POLICY" --namespace "$NAMESPACE" \
  -o jsonpath='{.status.transient_failure_count}{"\n"}'

Lock contention

LockContention means another reconcile currently holds the in-process or PostgreSQL advisory lock for the same database target. The operator retries after a short jittered delay; an occasional occurrence needs no intervention.

If it persists, look for multiple policies or replicas targeting the same database and for slow inspection/apply cycles. Do not disable the lock: it is what prevents overlapping inspect/diff/apply transactions.

Deletion is stuck

Deleting a policy means “stop managing,” not “undo database changes.” The finalizer first waits for attached ephemeral-access policies to be removed. EphemeralAccessCleanupPending reports that guarded wait.

Inspect attached EphemeralAccessPolicy and EphemeralAccessRequest resources, allow their revocation/finalizers to complete, and then retry deletion. Do not strip finalizers until you have verified that temporary PostgreSQL memberships were revoked; forced Kubernetes deletion can otherwise leave access behind.

Force a fresh reconcile

After fixing the cause, you can wait for spec.interval, edit the policy, or request an immediate reconcile with a unique timestamp:

kubectl annotate pgr "$POLICY" --namespace "$NAMESPACE" \
  reconcile.pgroles.io/requestedAt="$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  --overwrite

Then re-read conditions rather than relying only on Events. Several reasons, including MissingDatabaseObject, InvalidConnectionParams, and UnsatisfiableWildcardGrant, do not emit dedicated Events.