PostgreSQL Backups and Disaster Recovery

New in 2.3.0. The built-in PostgreSQL is backed up by pgBackRest, which the Crunchy Postgres Operator (PGO) runs on NetBox Enterprise's behalf. pgBackRest takes consistent base backups. It archives the write-ahead log (WAL) continuously into a dedicated repository volume, separate from the PostgreSQL data volume.

Velero then captures that repository along with the rest of the install. So the backup archive holds a self-consistent database backup set, not a copy of a running data directory.

This applies to both install paths. It applies to the built-in database only. An external database (spec.postgresql.external: true) gets no repository, and its backups are yours to run.

Why the data volume is not backed up

Velero's filesystem backup walks the PostgreSQL data directory file by file while PostgreSQL writes to it. The result is a torn copy. The backup reports success, and the restore is unrecoverable.

The usual remedy does not apply here. PostgreSQL 15 and later scope a backup session to the connection that opened it, so a Velero hook that starts a backup and then exits aborts it. Embedded Cluster's default storage has no volume-snapshot support either, so there is no atomic snapshot to fall back on.

pgBackRest closes that gap. The operator therefore excludes the PostgreSQL data volume from Velero on purpose, and the repository is the restorable artifact.

A completed Velero restore therefore always leaves the PostgreSQL data volume empty. Something must rebuild the database from the repository. The operator does that for you, and the restore procedure covers what to do when the automatic path does not run.

Storage requirements

Every storage class behind a NetBox Enterprise volume must provision PersistentVolumes whose source is local, csi, or a network volume. It must not provision hostPath volumes.

Check what your cluster provisions:

kubectl get pv -o custom-columns=\
'NAME:.metadata.name,SC:.spec.storageClassName,CSI:.spec.csi.driver,LOCAL:.spec.local.path,HOSTPATH:.spec.hostPath.path'

Any row with a value in the HOSTPATH column is not backed up. A row with a value in CSI or LOCAL is fine.

Why Velero skips a hostPath volume

Velero's backup agent reads a volume through the kubelet pod-volumes directory. Kubelet puts a local, csi, or network volume there, and bind-mounts a hostPath volume straight into the container instead, so the agent has nothing to read.

Velero then skips it. No warning, no PartiallyFailed, and no per-volume backup record. This is upstream Velero behavior, and NetBox Enterprise cannot work around it.

A hostPath class therefore costs you the whole application state: the PostgreSQL data volume and the pgBackRest repository, NetBox media and scripts, the plugin wheelhouse, Redis persistence, and in-cluster object storage.

Known-good and known-bad storage

Embedded Cluster ships OpenEBS LocalPV, which provisions local volumes. An Embedded Cluster install meets the requirement with no action.

A pure-Helm install depends on the cluster you bring:

The local-path-provisioner rows are the ones to watch. It is a common default on a single-node or edge cluster. If you evaluate NetBox Enterprise on one of those and you intend to test disaster recovery, change the storage class first. Otherwise the test misleads you.

local-path-provisioner cannot be configured to emit local volumes. Install a CSI driver instead, or install OpenEBS LocalPV and make it the default:

helm repo add openebs https://openebs.github.io/openebs
helm install openebs openebs/openebs -n openebs --create-namespace \
  --set engines.replicated.mayastor.enabled=false
kubectl patch storageclass local-path \
  -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
kubectl patch storageclass openebs-hostpath \
  -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

Despite the name, openebs-hostpath provisions local volumes, not hostPath ones. It is safe. An existing claim keeps its original class, so re-provision NetBox Enterprise storage after you change the default, or set the class per volume.

Configuration

Every setting lives under spec.postgresql.backups. On a supported storage class, the defaults give a default install working disaster recovery with no configuration. On a hostPath class they give none, and say nothing — see Storage requirements.

spec:
  postgresql:
    backups:
      enabled: true
      repoStorageSize: 16Gi
      repoStorageClassName: ""
      retentionFull: 4
      fullSchedule: "0 1 * * 0"
      incrementalSchedule: "0 1 * * 1-6"

See spec.postgresql.backups for the full field reference.

Three points are worth stating on their own:

1. Retention is by count, not by age. The repository volume has a fixed size, so a bound on the number of backup sets is what bounds the disk. Expiring a full backup also expires the incrementals and the WAL that depend on it.
2. Point-in-time recovery reaches only as far back as the retained WAL. The window is therefore retentionFull backup cycles, not an independent setting.
3. Set either schedule to an empty string to disable that run. The two can be disabled independently. enabled: false removes pgBackRest altogether.

Disabling backups leaves no PostgreSQL backup at all

enabled: false removes the repository, and the PostgreSQL data volume stays excluded from Velero. Nothing then backs the database up.

That is deliberate. A torn copy is not a backup, and one that looks restorable is worse than none. So disabling pgBackRest is an opt-out from PostgreSQL disaster recovery, not a choice of a different method.

A Velero restore of such a cluster brings back everything except the database. PostgreSQL then bootstraps a fresh, empty, usable instance, because there is no repository to rebuild from.

Disabling it on a running cluster is not a clean revert either. PGO leaves the repository StatefulSet and its volume in place, and it pauses its own reconciliation of the cluster. The pause is total, which is what makes it easy to miss: no spec change reaches the database, no upgrade applies, and no other status updates, while the database keeps serving.

The operator reports the pause on the NetBoxEnterprise resource as PostgresReconciliationPaused=True. The condition is advisory and does not gate Ready, because the way out is a configuration change and Ready gates both the Admin Console install gate and helm upgrade --wait. To resume, set the PostgreSQL backups option back to enabled.

Upgrading: a pre-2.3.0 backup is not a restore point

Take a full backup immediately after you upgrade to 2.3.0. A backup taken before the upgrade cannot restore your database.

Every release before 2.3.0 disabled pgBackRest, and it did not exclude the PostgreSQL data directory from Velero. An older archive therefore holds a torn file-by-file copy of a live data directory, and no pgBackRest repository, because none existed.

There is nothing in such an archive to recover from. The torn copy is not a valid PostgreSQL data directory, and there is no backup set or WAL to replay in its place.

The first valid PostgreSQL restore point is the first pgBackRest backup taken after the upgrade.

Turning pgBackRest on for a running cluster is safe and non-destructive. Existing data is untouched. PGO adds a repository host and the pgBackRest sidecars, and stanza creation runs against the live database.

Expect short-lived UnableToCreateStanzas warnings during the upgrade, while PGO rolls the instance to add the sidecar. They clear on their own, so do not react to them.

Wait for a successful backup instead — and take that backup yourself rather than wait for the schedule. fullSchedule defaults to weekly, so an upgrade on a Monday otherwise leaves six days with no valid restore point. See Taking a backup on demand, then confirm the new set with Verifying backups are working.

Verifying backups are working

Set the namespace and two selectors first. Every command on this page reads them, so set them in the shell you work in. PGCLUSTER is the NetBoxEnterprise resource name plus -postgres.

NS=netbox          # your release namespace
PGCLUSTER=netbox-postgres

CLUSTER="postgres-operator.crunchydata.com/cluster=$PGCLUSTER"

# The dedicated pgBackRest repository host.
REPO_HOST="$CLUSTER,postgres-operator.crunchydata.com/pgbackrest-dedicated="

# The PostgreSQL instance pods.
INSTANCES="$CLUSTER,postgres-operator.crunchydata.com/data=postgres"

An unset NS is the failure to watch for. kubectl -n "" silently falls back to your current context, so several checks below return No resources found and read as a real defect. Confirm the namespace resolves before you trust any of them:

kubectl -n "$NS" get postgrescluster "$PGCLUSTER"

Select the repository host on pgbackrest-dedicated, not on the bare postgres-operator.crunchydata.com/pgbackrest label. PGO puts the bare label on the instance pods too, because they run pgBackRest sidecars, so a selector on it picks an arbitrary pod.

# The repository host pod should be Running.
kubectl -n "$NS" get pods -l "$REPO_HOST"

# List the backup sets pgBackRest currently holds. This is the authority.
kubectl -n "$NS" exec -it \
  "$(kubectl -n "$NS" get pod -l "$REPO_HOST" -o name | head -1)" \
  -c pgbackrest -- pgbackrest info --stanza=db

Read the newest backup's timestamp, not just the count. "At least one full backup" is satisfied by a six-week-old set on a cluster whose repository volume filled up, or whose schedules the operator is holding down after a restore. Compare the newest set against fullSchedule and incrementalSchedule. A newest set older than one schedule interval means backups have stopped.

An empty repository prints No stanzas exist in the repository. and still exits 0. That means no backup has ever run: either the schedule has not come due, or stanza creation failed.

Events are a weaker signal, so read them second and only to explain what pgbackrest info already showed:

kubectl -n "$NS" describe postgrescluster "$PGCLUSTER" | sed -n '/Events/,$p'

An UnableToCreateStanzas warning here means backups are not initialized -- unless you are within an upgrade window, where it is expected and clears on its own. Absence of the warning proves nothing either way: Kubernetes expires events after about an hour, so a failure from yesterday leaves no trace. Judge on pgbackrest info.

Do not trust a Completed Velero backup on its own. Confirm that volume data was captured:

kubectl -n velero get podvolumebackups \
  -l velero.io/backup-name="$BACKUP" \
  -o custom-columns='VOLUME:.spec.volume,PHASE:.status.phase,BYTES:.status.progress.bytesDone'

Expect one Completed row per persistent volume, each with a non-zero byte count. The pgBackRest repository volume must be among them, because it is the only copy of your database in the archive.

An empty result has two causes. Check the cheap one first: confirm the backup name against velero backup get, because a name that matches nothing returns the same empty list. Once the name is known good, an empty result means no volume data was captured at all, and the likely cause is a hostPath storage class.

Taking a backup on demand

A manual full backup before an upgrade is worth taking. Run pgBackRest on the repository host:

REPO_HOST_POD="$(kubectl -n "$NS" get pod -l "$REPO_HOST" -o name | head -1)"

kubectl -n "$NS" exec "$REPO_HOST_POD" -c pgbackrest -- \
  pgbackrest backup --stanza=db --type=full

--type=incr takes an incremental backup against the newest full set instead. Confirm the new set landed:

kubectl -n "$NS" exec "$REPO_HOST_POD" -c pgbackrest -- pgbackrest info --stanza=db

The pod-volume-restore plugin config

Velero restores a filesystem-backed volume through a restore-wait init container, and it reads that container's security context from a ConfigMap in the velero namespace. It finds the ConfigMap by label selector, and it needs exactly one match.

With no matching ConfigMap, Velero copies the security context from the restored pod's first container. PGO's containers ask to run as a non-root user without naming a numeric user ID, so kubelet refuses the init container. No volume restore runs, and the pgBackRest repository comes back empty while every other object restores normally.

Embedded Cluster needs no action. NetBox Enterprise merges its settings into the ConfigMap the platform already ships.

A pure-Helm install supplies its own Velero, and the chart renders the ConfigMap for it. Set disasterRecovery.createVeleroSpecs: true in your values. See Backups (Helm).

The operator checks the count on every reconcile and reports the result as a VeleroRestoreConfigInvalid condition. See Status & Conditions for the five reasons it can carry. The condition is advisory. Nothing is wrong with the running cluster; what it reports is that a future restore cannot return your data.

Its absence proves nothing. The condition is absent when Velero is not installed, and absent under rbac.scope: namespace, because the reads are cross-namespace and the operator emits nothing rather than a false alarm. On a namespace-scoped install, count the ConfigMaps yourself:

kubectl get configmap -n velero \
  -l 'velero.io/plugin-config,velero.io/pod-volume-restore=RestoreItemAction'

One is correct.

On upgrade, remove your own copy

Do this on any upgrade to 2.3.0 or later, whether or not you change your values. An earlier revision of this guidance told you to apply the ConfigMap by hand, in the same step that told you to set disasterRecovery.createVeleroSpecs: true. If you followed it, the gate is already on, and helm upgrade alone lands a second claimant.

Two ConfigMaps then carry both discovery labels, Velero's lookup fails, and every volume restore aborts before the init container is injected.

Nothing warns you at the time. Backups keep succeeding, because the plugin config is read only on restore. The first symptom is a recovery that returns empty volumes. The chart cannot see your copy, so this step is yours.

Find it by label, not by name. Velero counts matches on the label pair, so any ConfigMap that carries both is a claimant whatever it is called:

kubectl get configmap -n velero \
  -l 'velero.io/plugin-config,velero.io/pod-volume-restore=RestoreItemAction' \
  -o custom-columns='NAME:.metadata.name,MANAGED-BY:.metadata.labels.app\.kubernetes\.io/managed-by,KEYS:.data'

Anything in that list other than netbox-enterprise-pod-volume-restore-config has to go. The common name is velero-fs-restore-action-config. That is both the name the earlier guidance used and what the upstream Velero chart produces, so your copy may belong to your Velero release rather than to you.

There are three outcomes:

1. Nothing found. Nothing to do.

2. A secCtx key only, and no app.kubernetes.io/managed-by: Helm label. This is the copy the earlier guidance told you to apply. Delete it in the same change that sets createVeleroSpecs: true:

   kubectl delete configmap -n velero velero-fs-restore-action-config

3. Anything else. This covers a managed-by: Helm label, or any of the other keys Velero reads from this ConfigMap, such as image, command, cpuRequest, cpuLimit, memRequest, and memLimit. Do not delete it. A Helm-owned copy returns on the next helm upgrade of your Velero release, so remove it from that release's values instead. The chart's copy carries only secCtx, so it cannot express the others: an image that pins a relocated restore-helper, which an air-gapped install needs, is lost if you delete the ConfigMap that holds it. In that case, keep your ConfigMap as the single claimant, add the security context below to it, and leave disasterRecovery.createVeleroSpecs off. You then supply the Velero Backup, the volume policy, and the restore-modifier ConfigMap yourself.

Confirm the count afterwards. One is correct. Two is the collision. Zero means the chart did not render its copy, so check that createVeleroSpecs is set in the values the release was installed with.

kubectl get configmap -n velero \
  -l 'velero.io/plugin-config,velero.io/pod-volume-restore=RestoreItemAction'

The setting is global to your Velero

Velero resolves one pod-volume-restore plugin config per Velero installation. There is no per-namespace or per-application scope.

On Embedded Cluster that Velero belongs to the appliance. A pure-Helm install supplies its own, and it may serve other workloads on the same cluster. With createVeleroSpecs: true, every filesystem-backed volume restore anywhere in that cluster runs its init container as user and group 26. Files land owned by 26:26, and a volume whose fsGroup excludes 26 may not be writable by the init container. That shows as a PodVolumeRestore for the other workload in phase Failed with a permission-denied message, while its Restore still reports Completed. If you share a Velero, check kubectl -n velero get podvolumerestores after the next restore of any workload, not just this one.

The alternative is worse, because with no config at all no volume restore works, including yours. But if that Velero is shared, know the scope before you turn it on.

Three rules govern the content. Read them before you change what the chart renders, or before you supply your own:

1. Supply the whole security context, not the shorthand keys. Velero builds the context from an empty one and fills in only the keys you give, and any key at all makes it skip its own hardened defaults. A partial security context therefore drops the rest without a message.
2. Keep the user ID and the group ID at 26. That is the ID Crunchy's images run PostgreSQL as, so restored files land with the ownership pgBackRest expects.
3. Keep the count at one.

This is what the chart renders:

apiVersion: v1
kind: ConfigMap
metadata:
  name: netbox-enterprise-pod-volume-restore-config
  namespace: velero
  labels:
    velero.io/plugin-config: ""
    velero.io/pod-volume-restore: RestoreItemAction
data:
  secCtx: |
    runAsUser: 26
    runAsGroup: 26
    runAsNonRoot: true
    allowPrivilegeEscalation: false
    capabilities:
      drop:
        - ALL
    seccompProfile:
      type: RuntimeDefault

Restore procedure

A running PostgreSQL is not evidence that the data came back. From an empty data directory, PostgreSQL does one of two things, and both happen on real restores:

• It fails to bootstrap, and the instance pod parks forever. The operator reports this on the NetBoxEnterprise resource as PostgresRestorePending. See Restore reports success but PostgreSQL never starts.
• It bootstraps cleanly in a couple of minutes, and you get a healthy cluster over a database with none of your data in it. The operator detects this and reports PostgresRecoveryRequired, which holds Ready down rather than let the cluster present as healthy.

So always confirm that your data is present, at step 3, before you treat a restore as finished. It does not matter how healthy the pods look.

While a recovery is outstanding, scheduled backups are held down. The operator empties this cluster's pgBackRest schedules for as long as that is the case. The restored cluster shares the source's repository, and its own backups would compete for the same retention slots and expire the source's, which are the only copy of your data. There is no flag to clear and nothing to switch back on. The operator reads the state again on every pass, so the schedules return by themselves once the database is back.

0. Confirm the installer matches your newest backup

This step applies to Embedded Cluster only.

sudo ./netbox-enterprise restore does not offer every backup in the store. It compares each backup against the application version and the Embedded Cluster version compiled into the installer binary on disk, and it drops every backup that differs.

An upgrade through the Admin Console replaces neither the binary nor those compiled-in versions. So on a cluster that was upgraded, the restore hides every backup taken since the original install.

Embedded Cluster lists its reasons only when no backup survives the filter. When one old backup does survive, the newer ones disappear with no message at all. The prompt then reads Found 1 restorable backup!, which reads as "there is one backup". To accept it restores the cluster to its original-install state, and every row written since then is gone.

The filter cuts the list down. Nothing scopes it to this cluster. A backup store holds backups from every cluster that writes to it. The restore lists all of them, offers the newest, and shows you a name and a timestamp:

○  Found 2 restorable backups!
Restore from backup "netbox-enterprise-019814bb" (2026-09-01 13:55:40 UTC)?  [Y/n]:

Neither field tells you which cluster produced it. To accept another cluster's backup completes, reports success, and brings the cluster up healthy over someone else's data. There is no error to investigate afterwards.

So give every cluster its own bucket, or its own prefix within one bucket. Do that when you configure the backup storage location, not at restore time. At restore time the store already holds the mixture.

If you have inherited a shared store, read the cluster ID and the versions off the backup before you accept it:

set -o pipefail   # without this the pipeline reports jq's status, not the copy's

aws s3 cp s3://"$BUCKET"/"$PREFIX"/backups/<name>/velero-backup.json - \
  | jq -er '.metadata.annotations["kots.io/embedded-cluster-id"],
            .metadata.annotations["kots.io/apps-versions"],
            .metadata.annotations["kots.io/embedded-cluster-version"]'

Three non-empty lines, or stop. This command is the only thing standing between you and a restore over another cluster's data, and without the two flags above it cannot fail loudly. A failed aws s3 cp writes to stderr and the pipeline still exits 0. A missing annotation makes plain jq -r print null and exit 0.

set -o pipefail surfaces the copy's failure, and jq -e exits non-zero on a null. If you see blank lines, null, or a non-zero exit, the lookup failed. That is not evidence the backup is unlabelled. Fix the credentials, the bucket, or the object name and run it again.

Once you have three values: compare the cluster ID against the cluster you are restoring, and the two versions against your installer binary. If they differ, download the installer that matches the newest backup before you start.

That object path is Velero's own layout. NetBox Enterprise neither documents it nor depends on it, so treat the lookup as reliable but unsupported.

1. Restore the cluster

On Embedded Cluster, run sudo ./netbox-enterprise restore. On Helm, apply a Velero Restore -- see Restoring a pure-Helm install. Either path brings back the PostgresCluster resource, the PGO Secrets, and the pgBackRest repository volume.

Run it on a host with no cluster on it.

# 1. The backup completed.
kubectl -n velero get backup <name> -o jsonpath='{.status.phase}{"\n"}'

# 2. It actually holds volume data — one Completed row per volume, non-zero bytes,
#    and the pgBackRest repo volume among them.
kubectl -n velero get podvolumebackups -l velero.io/backup-name=<name> \
  -o custom-columns='VOLUME:.spec.volume,PHASE:.status.phase,BYTES:.status.progress.bytesDone'

If you restore onto the same machine, run sudo ./netbox-enterprise reset --yes first. The reset drops the SSH connection partway through, and the command reports a non-zero status because of it. Reconnect and confirm that the host is bare, rather than treat that status as a failure.

Check the state the installer leaves behind, not the binary. A $PATH lookup passes on a reset that removed the binary and left the data directories:

for p in /var/lib/embedded-cluster /var/lib/k0s /etc/k0s; do
  [ -e "$p" ] && echo "still present: $p"
done
systemctl list-units --all 'k0s*' --no-legend | grep . && echo "k0s units remain"
command -v k0s >/dev/null && echo "k0s binary still on PATH"
echo "check complete"

Any line before check complete means the host is not bare. Re-run the reset, or clean the named path by hand.

An HA restore stops partway and waits for you to add the other nodes. After the Admin Console comes back, node 1 prints its URL and then holds:

Type 'continue' when you are done adding nodes: You are restoring a
high-availability cluster, which requires at least 3 controller nodes. You
currently have 1. Please add more controller nodes.

It re-asks rather than fails, so there is no race. Join the others while it waits. Mint the join command on node 1, then run it on each remaining node:

# On node 1:
cd && sudo ./netbox-enterprise join print-command

# On each remaining node, using the command it printed:
cd && sudo ./netbox-enterprise join <node-1-ip>:30000 <token> --yes

--yes matters. Every join runs host preflight checks, and a warning otherwise stops at an interactive prompt. It also confirms the HA prompt on the node that forms quorum.

It also accepts every preflight warning without showing you one. Read each node's preflight output before you answer continue on node 1. A node that joined over a warning about disk space, a kernel setting, or a missing storage prerequisite joins silently and fails later.

Once three controllers are present, answer continue, and the restore proceeds to the application.

An external database is a different shape. There is no pgBackRest repository and no PostgresCluster, so the database is never part of what the restore returns. It lives outside the cluster and comes through untouched, and steps 2 and 3 do not apply.

2. Rebuild the database from the repository

Do this only when the operator's automatic path did not run or did not work.

kubectl -n "$NS" get netboxenterprise -o json \
  | jq -r '.items[].status.conditions[]
           | select(.type|test("PostgresRecoveryRequired|PostgresRestorePending|PostgresRestoreFailed"))
           | "\(.type)=\(.status) \(.reason)"'

kubectl -n "$NS" get postgrescluster "$PGCLUSTER" -o jsonpath='{.spec.dataSource}{"\n"}'

PGO restores from the repository when the resource carries a data source and the restore is enabled:

kubectl -n "$NS" patch postgrescluster "$PGCLUSTER" --type=merge -p '{
  "spec": {
    "backups": {
      "pgbackrest": {
        "restore": {
          "enabled": true,
          "repoName": "repo1",
          "options": []
        }
      }
    }
  }
}'

# Trigger the restore.
kubectl -n "$NS" annotate postgrescluster "$PGCLUSTER" \
  postgres-operator.crunchydata.com/pgbackrest-restore="$(date +%s)" --overwrite

Read the patch back before you trigger it. The operator owns parts of this resource, so a reconcile between the two commands can revert the field, and the annotation then fires against a cluster with no restore enabled:

kubectl -n "$NS" get postgrescluster "$PGCLUSTER" \
  -o jsonpath='{.spec.backups.pgbackrest.restore}{"\n"}'
# {"enabled":true,"options":[],"repoName":"repo1"}

Recovery target

An empty options list recovers the most that is available. pgBackRest then restores the newest backup and replays the WAL archive to its end. Use this unless you have a reason not to.

There are two alternatives, and one trap.

--type=immediate stops at the backup. It recovers only until the database is consistent, and it discards every transaction committed after the backup ran. Full backups are weekly by default, so a run that stops here can lose days of writes, and it reports success. The WAL that closes that gap is in the same repository. Choose this only when you want the backup and not the tail.

Point-in-time recovery needs a target with no space in it, and a time target also needs --set. Two constraints meet here, and a form that satisfies only one of them fails.

The first constraint is PGO's. It renders options onto the restore Job's command line unquoted, so a target written as "2026-08-28 23:20:00+00" splits on the space:

ERROR: [096]: command does not allow parameters

The second is pgBackRest's. Its automatic backup-set selection cannot parse a timestamp written with T, so a target with the space removed trips a different error:

ERROR: [029]: automatic backup set selection cannot be performed with
provided time '2026-08-28T23:20:00+00'

That message reads as though the target is outside the archive. Usually it is not. Only the set selection fails to parse the value. pgBackRest still passes the target through to PostgreSQL, whose own parser accepts ISO 8601 with T. To name the backup set yourself removes the step that cannot parse it.

Either of these forms works, and neither contains a space. Recover to a time, and name the set:

"options": [
  "--type=time",
  "--target=2026-08-28T23:20:00+00",
  "--set=20260828-230000F"
]

Recover to a log sequence number, which needs no --set:

"options": ["--type=lsn", "--target=0/A000000"]

Take the --set value from pgbackrest info. Choose a backup that finished before your target time. The target must also fall inside the WAL archive range that pgbackrest info reports.

A rejected form does not fail once. The Job retries and fails repeatedly, with PostgreSQL shut down throughout. That is an outage on top of the disaster you are recovering from.

Verify that the restore wrote the target you meant, while the recovery is still in progress.

Resolve the pod and the data directory first. Both vary: PGO names the instance StatefulSet after your cluster and its instance set, and the directory is named for your PostgreSQL major version. A wrong value for either makes the check below fail in a way that looks exactly like the benign case described afterwards.

PGPOD="$(kubectl -n "$NS" get pod -l "$INSTANCES" -o name | head -1)"
PGDIR="$(kubectl -n "$NS" exec "$PGPOD" -c database -- \
  sh -c 'ls -d /pgdata/pg[0-9]* 2>/dev/null | grep -E "/pg[0-9]+$" | head -1')"
echo "pod=$PGPOD dir=$PGDIR"

Both must print a value. An empty PGDIR means the data directory does not exist yet, which is its own answer: the recovery has not written it.

kubectl -n "$NS" exec "$PGPOD" -c database -- \
  grep recovery_target "$PGDIR/postgresql.auto.conf"

Only now is a non-zero exit meaningful. During recovery, an absent recovery_target_* line means the target never reached PostgreSQL, and the restore ran to the end of the WAL instead.

PostgreSQL removes those settings when recovery ends, so after a fast restore this command exits 1 against a cluster that recovered correctly. Its absence then means nothing either way.

The restore pod's log carries its own record, and is worth reading while the pod still exists:

kubectl -n "$NS" logs -l postgres-operator.crunchydata.com/pgbackrest-restore \
  --tail=-1 | grep -iE "recovery_target|starting point-in-time recovery"

PGO collects the restore Job once it finishes, and kubectl logs against a selector that matches no pod prints nothing and exits 0. So empty output here means the pod is gone, not that the target was absent. Capture this log during the recovery if you need it.

Watching the restore

Watch the PostgresCluster status, not a Job named *restore*. PGO collects the Job, so a poll for it races and looks like a failure even on a restore that succeeded.

kubectl -n "$NS" get postgrescluster "$PGCLUSTER" \
  -o jsonpath='{.status.pgbackrest.restore}{"\n"}'
# {"id":"1785424164","startTime":"...","completionTime":"...","finished":true}

finished: true with an id that matches the annotation you set means pgBackRest finished the write of the restored data.

That is necessary and not sufficient. It reports on the pgBackRest restore only. PostgreSQL still has to bootstrap on top of the restored files, and that step can fail in a loop while the status above keeps saying the restore succeeded. Do not call the restore done until the database container is Ready. See step 3.

How long to let it run before you treat it as stuck. Two measured points, same build and same backup store, both successful:

The slow one spent that time in a retry cycle and reported nothing for the duration. It then completed on its own and returned every row, including one that existed only in the WAL archive.

So a single-node restore that has said nothing for forty minutes is not evidence of a hang. Judge it on movement, not on elapsed time. Watch for pgbackrest info to gain backups, for the restore pod's log to advance through WAL segments, and for PostgresDataInitialized on the PostgresCluster. If none of those has moved in half an hour, start to investigate.

The restore status goes stale after a success. PGO stops refreshing status.pgbackrest.restore once it collects the Job, so a field read hours later can still show a start time and finished: false on a cluster that serves all its data. Check the startTime against the clock, and confirm against a live fact: whether a restore Job exists, and whether the database container is Ready.

A non-zero failed count in that same status is a failing restore. The operator reports it as PostgresRestoreFailed on the NetBoxEnterprise resource, holds Ready down, and emits a Warning event. See Restore fails in a loop.

Disable the restore setting again once it completes. A later reconcile can otherwise re-trigger it, and a re-trigger is another destructive restore over a database that is by then live and serving.

kubectl -n "$NS" patch postgrescluster "$PGCLUSTER" --type=merge \
  -p '{"spec":{"backups":{"pgbackrest":{"restore":{"enabled":false}}}}}'

# Read it back. Expect false.
kubectl -n "$NS" get postgrescluster "$PGCLUSTER" \
  -o jsonpath='{.spec.backups.pgbackrest.restore.enabled}{"\n"}'

3. Verify

Four checks. Do all four -- the first two say PostgreSQL is running, and only the last two say your data came back and stays protected.

# 1. Every instance pod is fully Ready. This, not the restore status, is what
#    says PostgreSQL came back. A pod short of its full container count has not
#    bootstrapped.
kubectl -n "$NS" get pods -l "$INSTANCES"

# 2. All instances report ready.
kubectl -n "$NS" get postgrescluster "$PGCLUSTER" \
  -o jsonpath='{.status.instances[*].readyReplicas}{"\n"}'

# 3. NetBox can reach the database, and its data is present. Compare both
#    numbers against the baseline you recorded before the loss.
NBPOD="$(kubectl -n "$NS" get pod -l 'app.kubernetes.io/component=netbox' \
  -o name | head -1)"

kubectl -n "$NS" exec "$NBPOD" -c netbox -- \
  python manage.py shell -c \
  'from core.models import ObjectChange
from dcim.models import Site
print("sites:", Site.objects.count())
print("newest change:", ObjectChange.objects.order_by("-time").values_list("time", flat=True).first())'

# 4. Scheduled backups have returned. The operator empties this cluster's
#    pgBackRest schedules while a recovery is outstanding, so an unrecovered
#    cluster silently has no backups.
kubectl -n "$NS" get postgrescluster "$PGCLUSTER" \
  -o jsonpath='{.spec.backups.pgbackrest.repos[0].schedules}{"\n"}'

# No restore condition should remain on the resource.
kubectl -n "$NS" get netboxenterprise -o json \
  | jq -r '.items[].status.conditions[]
           | select(.type|startswith("PostgresR"))
           | "\(.type)=\(.status) \(.reason)"'

Expect the first command to print your fullSchedule and incrementalSchedule values, and the second to print nothing.

Empty output from the first command is the held state, not an error. The operator omits the field entirely while a recovery is outstanding. Nothing needs switching back on: the operator re-reads the state every pass, so the schedules return by themselves once the database is back. If they have not returned and the database is serving, the second command tells you which condition is still standing.

A cluster that comes up healthy but empty means the restore recovered an empty backup set, not your data. Treat that as a failed restore.

Restoring a pure-Helm install

An Embedded Cluster install gets its Velero Backup from the platform, and the installer builds a Restore for you at recovery time. A Helm install supplies its own Velero, so it supplies both itself.

The chart supplies the Backup half. Enable it in your values:

disasterRecovery:
  createVeleroSpecs: true

That renders four objects into the velero namespace:

1. The Backup.
2. The volume-policy ConfigMap the Backup references.
3. The pod-volume-restore plugin config, which Velero reads the restore init container's security context from.
4. The resource modifier that makes a restored PostgresCluster arrive paused.

If you hand-applied the plugin config from an earlier revision of this guidance, deal with your copy in the same change. See On upgrade, remove your own copy. Otherwise Velero sees two claimants and restores no volumes at all.

Leave this setting off on Embedded Cluster. The platform already supplies the Backup there, and a second copy in the velero namespace breaks the restore rather than improves it.

Install order matters on a rebuilt cluster

Two of those four objects have to exist in the velero namespace before you create the Restore. Velero resolves the resource modifier at creation time, and it reads the plugin config when it injects the init container. The chart is what renders them, so on a rebuilt cluster you install the chart first.

Install it with netboxEnterprise.enabled left at its default false:

netboxEnterprise:
  enabled: false
disasterRecovery:
  createVeleroSpecs: true

That renders the operator and all four velero-namespace objects, and no NetBoxEnterprise resource. So there is no PostgresCluster and no volumes.

To install with your production values instead brings up a live, empty cluster. The Restore below uses existingResourcePolicy: none, so Velero skips an object that already exists by name and namespace, and a skipped object is never patched. The archived PostgresCluster and both volumes would be skipped, the resource modifier would never land, and the Restore would report Completed while it recovered nothing that holds data.

Confirm there is nothing for Velero to skip before you go on:

# Positive control first: prove the namespace exists and you are looking at it.
# A wrong or unset NS makes the next command pass unconditionally.
kubectl get namespace "$NS"

kubectl -n "$NS" get postgrescluster

Expect the namespace to exist, and No resources found from the second command. Those two together are the check; the second one alone is not, because it prints the same line for a namespace that does not exist.

The NetBoxEnterprise resource comes back from the archive, because it carries the labels the Backup selects. The operator then clears the pause and supplies the data source. Set netboxEnterprise.enabled: true only if the resource did not come back.

The chart ships no Restore. Apply one by hand

A Restore is not part of the chart, and it must not be added to one. A Velero Backup is a declaration, and Velero acts on it when the schedule says so. A Restore is an action that runs the moment the object exists. A chart manifest is applied on every helm install and every helm upgrade, so a Restore shipped there would restore over your live cluster as soon as a matching backup exists.

Apply the manifest below only when you are recovering. Three checks come first, and none of them can be skipped.

1. The backup completed, and it holds volume data. Completed on its own is not enough. A backup taken on a hostPath storage class reports Completed with none of your data in it:

kubectl -n velero get backup netbox-enterprise-backup \
  -o jsonpath='{.status.phase}{"\n"}'

kubectl -n velero get podvolumebackups \
  -l velero.io/backup-name=netbox-enterprise-backup \
  -o custom-columns='VOLUME:.spec.volume,PHASE:.status.phase,BYTES:.status.progress.bytesDone'

Expect Completed, then one Completed row per volume with a non-zero byte count, the pgBackRest repository volume among them. An empty second result means there is nothing to restore.

2. Exactly one pod-volume-restore plugin config exists. Count it by label, not by name. Velero needs exactly one match on the label pair, and a second claimant under any name breaks every volume restore:

kubectl get configmap -n velero \
  -l 'velero.io/plugin-config,velero.io/pod-volume-restore=RestoreItemAction'

One row is correct. Two is the collision -- see On upgrade, remove your own copy. Zero means the chart did not render its copy.

3. The resource modifier exists, by name, because the Restore names it:

kubectl -n velero get configmap netbox-enterprise-restore-modifiers

Both ConfigMaps come from the chart under disasterRecovery.createVeleroSpecs, and their absences do not look alike:

• No plugin config, or two of them. Velero injects the init container and copies the security context from the restored pod's first container, which kubelet then refuses. The init container sits at CreateContainerConfigError, no volume restore runs, and the Restore does not reach a terminal phase. It stays InProgress with every API object restored.
• No modifier. Nothing is visible. The PostgresCluster arrives unpaused, PGO bootstraps an empty database over the recovery, the pods come up healthy, and the Restore reports Completed.

Then apply the Restore. Set NS to your release namespace in the same shell that runs the kubectl apply, because the heredoc substitutes it:

NS=netboxlabs   # your release namespace
kubectl apply -f - <<EOF
apiVersion: velero.io/v1
kind: Restore
metadata:
  # DESTRUCTIVE. Apply this only during a recovery. Velero validates a Restore
  # once and never re-validates it, so a failed attempt cannot be retried under
  # the same name. Use a fresh name each time.
  name: netbox-enterprise-restore-1
  namespace: velero
spec:
  backupName: netbox-enterprise-backup

  includedNamespaces:
    - $NS
    - velero

  includeClusterResources: true
  restorePVs: true

  # Velero's default. Every object that already exists is SKIPPED with a
  # warning, and the Restore still reports Completed. So on a partly-live
  # cluster, Completed means "restored what was absent", not "restored
  # everything". Read the warning count before you trust it.
  existingResourcePolicy: none

  # REQUIRED. This is what makes the restored PostgresCluster arrive paused, so
  # PGO cannot create an instance and initialise an empty data directory before
  # the pgBackRest restore happens. Without it every pod comes up healthy over
  # an empty database and this Restore still reports Completed.
  #
  # Velero resolves this reference when the Restore is created, and an
  # unresolvable one skips the modifier step with NO error. Confirm the
  # ConfigMap exists before you apply this.
  resourceModifier:
    kind: ConfigMap
    name: netbox-enterprise-restore-modifiers

  orLabelSelectors:
    - matchLabels:
        app.kubernetes.io/managed-by: netbox-operator
    - matchLabels:
        netboxlabs.com/managed-by: netbox-operator
    - matchLabels:
        app.kubernetes.io/part-of: netbox-enterprise
    - matchLabels:
        postgres-operator.crunchydata.com/control-plane: pgo
EOF

Watch it to completion:

kubectl -n velero get restore netbox-enterprise-restore-1 \
  -o jsonpath='{.status.phase}  {.status.warnings}  {.status.validationErrors}{"\n"}' -w

FailedValidation with BackupStorageLocation.velero.io "" not found means the backup was not resolvable when Velero validated the object. Confirm that the Backup reports Completed, then apply the manifest again under a new name.

A non-zero warning count on a Completed restore is usually existingResourcePolicy: none as it skips objects that were already present. Read them before you treat the restore as complete:

velero restore describe netbox-enterprise-restore-1 --details

Completed is not evidence that your data came back. Return to Restore procedure. A completed Velero restore always leaves the PostgreSQL data directory empty, and something must rebuild the database from the repository.

Do not widen the backup selectors

The selectors decide what enters the archive. Do not widen them, and do not replace them with a namespace-wide backup.

A namespace-wide backup captures the PostgreSQL cluster-membership Endpoints, which hold the database's system identifier. When a restore brings that back, PGO reads the identifier, concludes the cluster is already bootstrapped, and skips the pgBackRest restore. The database then comes up empty, and no error names the cause.

Troubleshooting

These commands use the $REPO_HOST and $INSTANCES selectors from Verifying backups are working. Set them first if you jumped straight here.

Stanza creation fails

Backups are not initialized. Get the real error rather than the exit code:

# The bare pgbackrest label on purpose, not "$REPO_HOST": stanza creation can
# fail on either side, so this collects the repository host and the sidecars.
kubectl -n "$NS" logs -l postgres-operator.crunchydata.com/pgbackrest= \
  -c pgbackrest --tail=100

REPO_HOST_POD="$(kubectl -n "$NS" get pod -l "$REPO_HOST" -o name | head -1)"
kubectl -n "$NS" exec -it "$REPO_HOST_POD" -c pgbackrest -- \
  pgbackrest stanza-create --stanza=db --log-level-console=detail

These are the pgBackRest exit codes seen on Kubernetes:

PGO re-runs stanza creation only when it sees new repository configuration. To force a retry after you fix the cause, clear the marker annotation:

kubectl -n "$NS" annotate postgrescluster "$PGCLUSTER" \
  pgbackrest.crunchydata.com/stanzas-created- --overwrite

Restore fails in a loop

status.pgbackrest.restore carries a non-zero failed count and finished: false, and the count climbs as the Job retries. The operator reports this as PostgresRestoreFailed and holds Ready down:

kubectl -n "$NS" describe netboxenterprise | grep -A3 PostgresRestoreFailed
kubectl -n "$NS" get events --field-selector reason=PostgresRestoreFailed

Ready shows one of two states here, and both are correct. On a cluster that already serves data, Ready is False, because the restore failure gates it. On a cluster with no database yet, Ready is Unknown, because the reconcile waits for a PostgreSQL that cannot start. Read the condition, not Ready, to tell a failed restore from a slow one.

Read the restore Job's log for the pgBackRest error code, which is the part that says why:

kubectl -n "$NS" logs -l postgres-operator.crunchydata.com/pgbackrest-restore \
  --tail=50

Two causes account for most of these, and both are in the options you set in step 2:

• --set names a backup set that is not in the repository. Check it against pgbackrest info. A label that does not exist fails every attempt, identically, forever.
• The recovery target falls outside the WAL archive range. The same command reports the range.

Fix the options, then re-trigger with a fresh annotation value. The condition clears once PGO reports a database.

The count itself is sticky. PGO copies failed from the restore Job and stops refreshing it once it collects the Job, so a restore that failed twice and then succeeded keeps failed: 2 in its status. A non-zero count on a cluster that serves your data is not a problem.

The condition is a useful signal when it is present, and its absence proves nothing. On a cluster whose failing restore is what keeps PostgreSQL down, the condition can be absent while Ready reads Unknown. Read status.pgbackrest.restore on the PostgresCluster directly in that case. PostgresDataInitialized=False with reason PGBackRestRestoreFailed on the same object says the same thing in one line.

After a Velero restore there are no PostgreSQL pods at all

This is not a stalled restore. A restored PostgresCluster arrives paused on purpose, so that PGO cannot create an instance and initialise an empty data directory before the pgBackRest restore. While the pause holds, PGO creates nothing:

kubectl -n "$NS" get postgrescluster "$PGCLUSTER" \
  -o jsonpath='{.spec.paused}{"  "}{range .status.conditions[?(@.type=="Progressing")]}{.status}/{.reason}{end}{"\n"}'
# true  False/Paused

The operator releases the pause on the same reconcile that supplies the data source, and it records an event when it does. So this state should last seconds. It persists only when the operator is not reconciling:

# Is the operator running at all?
kubectl -n "$NS" get deploy -l app.kubernetes.io/component=operator

# What did it say about the pause?
kubectl -n "$NS" get events --field-selector reason=PostgresRestorePauseReleased
kubectl -n "$NS" get events --field-selector reason=PostgresRestorePauseHeld

A PostgresRestorePauseHeld warning means the operator saw the pause and kept it on purpose. The cluster has a pgBackRest repository, but the reconcile built no data source, and to release the pause would let PGO initialise an empty database over the recovery. Fix why the data source is missing. A paused cluster is recoverable; an overwritten one is not. The next reconcile then releases the pause on its own.

Do not clear the pause by hand to unstick it unless the resource already carries a data source. Without that field PGO has no restore to run, so it creates an instance and initialises an empty database. Check first:

kubectl -n "$NS" get postgrescluster "$PGCLUSTER" -o jsonpath='{.spec.dataSource}{"\n"}'

The opposite state: PostgreSQL pods appear immediately

This one is easy to read as success. If the restored cluster is not paused, the resource modifier did not apply, and the restore raced PGO.

Check the pause, not the restore phase. A restore that skipped the modifier reports Completed with zero errors, because from Velero's side nothing went wrong. That is the same phase a correct restore reports.

kubectl -n "$NS" get postgrescluster "$PGCLUSTER" \
  -o jsonpath='paused={.spec.paused} dataSource={.spec.dataSource}{"\n"}'

An absent paused on a freshly restored cluster is the symptom. If PostgreSQL has already started over an empty data directory, this is a failed restore: see Rebuild the database from the repository.

The restore phase is worth reading second, because PartiallyFailed catches a different miss:

kubectl -n velero get restore -o custom-columns=\
'NAME:.metadata.name,PHASE:.status.phase,ERRORS:.status.errors,WARNINGS:.status.warnings'

The ConfigMap the Restore names has to exist in the velero namespace before the restore runs. The two install paths supply different objects, so check the one that applies to you:

# Embedded Cluster
kubectl -n velero get configmap velero-netbox-enterprise-restore-modifiers
# Helm
kubectl -n velero get configmap netbox-enterprise-restore-modifiers

Also confirm that your Restore names the object your install actually has. Velero skips the modifier step with no error when the reference does not resolve.

Restore reports success but PostgreSQL never starts

status.pgbackrest.restore shows finished: true, but the instance pod never reaches Ready and NetBox cannot connect. Check the instance pod's log:

kubectl -n "$NS" logs -l "$INSTANCES" -c database --tail=50

A repeating cycle of about 10 seconds like this is the known failure:

P00 ERROR: [040]: unable to restore to path '/pgdata/pg18_wal' because it contains files
ERROR: Error creating replica using method pgbackrest: ... '--type=standby' exited with code=40
ERROR: failed to bootstrap (without leader)
INFO: Removing data directory: /pgdata/pg18

The restore left /pgdata holding a bootstrap directory and a populated WAL directory, but no data directory. PostgreSQL deletes the data directory on every failed attempt and never touches the WAL directory, so the loop cannot clear itself. To empty that WAL directory by hand breaks the loop.

INSTANCE_POD="$(kubectl -n "$NS" get pod -l "$INSTANCES" -o name | head -1)"

# Check first. Expect pg18_bootstrap and pg18_wal, and NO pg18.
kubectl -n "$NS" exec "$INSTANCE_POD" -c database -- ls /pgdata

Only if that matches, and with your own major version substituted for pg18:

kubectl -n "$NS" exec "$INSTANCE_POD" -c database -- \
  bash -c 'rm -rf /pgdata/pg18_wal/*'

PostgreSQL promotes itself on the next cycle, roughly 30 seconds later, and the pod becomes fully Ready. This is a bootstrap problem, not a data problem, so the recovered data is intact.

Confirm that rather than assume it. Run all four checks in Verify. A Ready pod is the signal this whole page warns against reading as proof, and you have just deleted files to reach it.

If the pod is still not Ready a few minutes later, the loop has a different cause. Re-read the instance log, and treat the restore as unfinished.

The repository volume is full

pgbackrest info shows backup sets beyond retentionFull, or backups start to fail on write. Watch for this before it happens:

kubectl -n "$NS" exec "$REPO_HOST_POD" -c pgbackrest -- df -h /pgbackrest

Either raise repoStorageSize, which is a volume expansion and needs a storage class that allows it, or lower retentionFull.

To lower retention expires older sets, and with them the ability to recover to a point in time inside those sets.

An expansion is not immediate, and it can stall silently. Read the claim back and wait for the pending condition to clear:

kubectl -n "$NS" get pvc -l netboxlabs.com/managed-by=netbox-operator \
  -o custom-columns='NAME:.metadata.name,SIZE:.status.capacity.storage,CONDITIONS:.status.conditions[*].type' \
  | grep repo

A claim stuck at FileSystemResizePending has not grown, and backups are still failing.

Backups run but the archive does not contain them

There are two causes, and the second is the common one. Check both.

1. Velero did not capture the volume. This is what a hostPath storage class does, and it is also what happens when repoStorageClassName points the repository at a different class from everything else. Ask Velero directly:

kubectl -n velero get podvolumebackups -l velero.io/backup-name="$BACKUP" \
  -o custom-columns='VOLUME:.spec.volume,PHASE:.status.phase,BYTES:.status.progress.bytesDone'

The repository volume must appear, Completed, with a non-zero byte count. Its absence from an otherwise populated list is the answer. See Storage requirements.

2. The repository volume does not carry the backup-selection label, so Velero never considers it:

kubectl -n "$NS" get pvc -l netboxlabs.com/managed-by=netbox-operator \
  -o custom-columns='NAME:.metadata.name'

Expect a claim whose name contains repo. Confirm NS resolves first -- a wrong namespace prints an empty list here too, and that reads as a missing label.

Limitations

• On a single-node install, the repository shares a failure domain with the database. On Embedded Cluster both volumes are node-local. pgBackRest protects against database-level corruption and operator error. Survival of node loss depends on the Velero archive that lives off-cluster.
• There is no cross-cluster repository. pgBackRest supports an object-store repository, which NetBox Enterprise does not expose. Database backups and WAL are kept out of the NetBox media bucket on purpose: that bucket has different retention, access, and capacity expectations. A separate backup-only object store target is future work.
• Point-in-time recovery is bounded by the WAL retained beside the newest full backup. Expiring a full backup expires its WAL, so the window is retentionFull backup cycles.
• Air-gapped disaster recovery is not covered on either install path. An air-gapped install needs a relocated restore-helper image, and the chart's plugin config cannot express one.

See Also

• Backups (Embedded Cluster) - Admin Console backup and restore, and manual dumps
• Backups (Helm) - Velero installation and the chart's disaster-recovery objects
• spec.postgresql.backups - Full field reference
• Status & Conditions - The restore conditions the operator reports