PostgreSQL Backups and Disaster Recovery

From 2.3.0, pgBackRest backs up the built-in PostgreSQL. The Crunchy Postgres Operator (PGO) runs pgBackRest for NetBox Enterprise. pgBackRest takes consistent full and incremental backups on a schedule. It also archives the write-ahead log (WAL) all the time. The WAL is the record of each change to the database. pgBackRest writes the backups and the WAL to a repository volume, which is separate from the PostgreSQL data volume.

This page applies to Embedded Cluster and Helm installs. It applies to the built-in database only. An external database (spec.postgresql.external: true) has no repository, and you must back it up yourself.

Terms

This page uses these terms, and uses each one for one thing only:

A disaster recovery restore always includes a PostgreSQL restore, because the database comes back from the repository. You can also do a PostgreSQL restore by itself, on a running install, to return the database to an earlier point.

This page has three parts:

1. How disaster recovery and PostgreSQL backups work together: what a disaster recovery backup holds, and what a disaster recovery restore returns.
2. Disaster recovery: disaster recovery backups and disaster recovery restores.
3. PostgreSQL backups: PostgreSQL backups, and a PostgreSQL restore to an earlier point.

How disaster recovery and PostgreSQL backups work together

A disaster recovery backup copies the whole repository volume with the rest of the install. It therefore holds all PostgreSQL backups and all WAL that the repository held at that time.

A disaster recovery restore has two steps:

1. Velero restores the volumes, which include the repository.
2. The operator does a PostgreSQL restore from the repository. By default, pgBackRest restores the newest PostgreSQL backup and replays the WAL to its end.

The database therefore comes back at about the time of the disaster recovery backup. To go back further, use an earlier disaster recovery backup. You can also do a PostgreSQL restore to an earlier point after the disaster recovery restore. See PostgreSQL restore to an earlier point.

The two kinds of backup have separate schedules:

PostgreSQL backups alone do not protect against the loss of the cluster, because the repository is in the cluster. Set a schedule for disaster recovery backups. The disaster recovery schedule sets how much data a disaster recovery restore can lose.

The pod-volume-restore plugin config

A disaster recovery restore needs exactly one pod-volume-restore plugin config in the velero namespace. On Embedded Cluster, the platform supplies it, and you need no action. On Helm, the chart supplies it. See The pod-volume-restore plugin config.

Disaster recovery

A disaster recovery restore returns the whole install from a disaster recovery backup. The disaster recovery backup holds the repository volume. After Velero restores that volume, the operator does a PostgreSQL restore from the repository. The database comes back at about the time of the disaster recovery backup.

The tools to make and restore disaster recovery backups are different on each install path:

• Embedded Cluster: the admin console and the installer. See Backup and Restore in Depth.
• Helm: your own Velero and the chart's Velero objects. See Backups (Helm).

This section covers what is the same on both install paths.

Why the data volume is not backed up

The Velero filesystem backup copies the PostgreSQL data directory file by file while PostgreSQL writes to it. The result is a torn copy: a set of files that does not agree with itself. The disaster recovery backup reports success, but you cannot recover from the copy.

The usual fix does not work here. In PostgreSQL 15 and later, a backup session belongs to the connection that opened it. A Velero hook that starts a backup session and then exits therefore stops that session. The default storage on Embedded Cluster also has no volume snapshots, so no atomic snapshot is available.

pgBackRest solves this problem. The operator excludes the PostgreSQL data volume from disaster recovery backups, and the database comes back from the repository instead.

A completed Velero restore therefore always leaves the PostgreSQL data volume empty. A PostgreSQL restore from the repository must then rebuild the database. The operator starts it automatically. The disaster recovery restore procedure tells you what to do when the automatic PostgreSQL restore does not run.

Storage requirements

Each storage class for a NetBox Enterprise volume must provision PersistentVolumes with a local, csi, or network volume source. It must not provision hostPath volumes.

Check what your cluster provisions:

kubectl get pv -o custom-columns=\
'NAME:.metadata.name,SC:.spec.storageClassName,CSI:.spec.csi.driver,LOCAL:.spec.local.path,HOSTPATH:.spec.hostPath.path'

A row with a value in the HOSTPATH column is not in a disaster recovery backup. A row with a value in CSI or LOCAL is in it.

Why Velero skips a hostPath volume

The Velero backup agent reads each volume through the kubelet pod-volumes directory. Kubelet puts a local, csi, or network volume in that directory. Kubelet mounts a hostPath volume directly into the container instead, so the agent cannot find it.

Velero then skips the volume. It shows no warning, sets no PartiallyFailed phase, and writes no per-volume backup record. This is upstream Velero behavior, and NetBox Enterprise cannot change it.

A hostPath storage class therefore loses all application state: the PostgreSQL data volume, the repository, NetBox media and scripts, the plugin wheelhouse, Redis persistence, and in-cluster object storage.

Known-good and known-bad storage

Embedded Cluster ships OpenEBS LocalPV, which provisions local volumes. An Embedded Cluster install meets the requirement with no action.

A pure-Helm install depends on the storage in your cluster:

local-path-provisioner is a common default on single-node and edge clusters. If you use one of these clusters to test disaster recovery, change the storage class first. If you do not, the test gives a false result.

You cannot configure local-path-provisioner to provision local volumes. Install a CSI driver, or install OpenEBS LocalPV and make it the default:

helm repo add openebs https://openebs.github.io/openebs
helm install openebs openebs/openebs -n openebs --create-namespace \
  --set engines.replicated.mayastor.enabled=false
kubectl patch storageclass local-path \
  -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
kubectl patch storageclass openebs-hostpath \
  -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

The name openebs-hostpath is misleading: it provisions local volumes, not hostPath volumes. An existing claim keeps its original storage class. After you change the default, provision the NetBox Enterprise volumes again, or set the storage class for each volume.

Disaster recovery restore procedure

A running PostgreSQL does not prove that your data came back. From an empty data directory, PostgreSQL does one of two things. Both occur on real disaster recovery restores:

• It cannot bootstrap, and the instance pod stops there. The operator reports PostgresRestorePending on the NetBoxEnterprise resource. See Restore reports success but PostgreSQL never starts.
• It bootstraps in a few minutes, and you get a healthy cluster with an empty database. The operator detects this and reports PostgresRecoveryRequired. This condition keeps Ready from True, so the cluster does not look healthy.

Before you call the disaster recovery restore finished, confirm that your data is present at step 3. The pod status does not prove it.

During a disaster recovery restore, the operator stops the PostgreSQL backup schedules. It removes them until the database is recovered. The restored cluster uses the same repository as the source cluster. Its own PostgreSQL backups would take retention slots and expire the PostgreSQL backups of the source, which are the only copy of your data. You do not need to turn the schedules on again. The operator checks the state on each reconcile, and the schedules come back when the database is back.

1. Restore the cluster

Restore the disaster recovery backup with the procedure for your install path:

• Embedded Cluster: Restore the NetBox Enterprise installation. It also tells you how to choose the correct disaster recovery backup and how to prepare the host.
• Helm: Disaster recovery restore. Read these sections of it: The chart's Backup is not a restore point, Install the chart before the Restore, and The chart has no Restore: apply one manually. Also see Do not widen the backup selectors.

Both paths restore the PostgresCluster resource, the PGO Secrets, and the repository volume. Then do step 2 only if the automatic PostgreSQL restore does not run. Always do step 3.

With an external database, the disaster recovery restore is different. No repository and no PostgresCluster exist, so the restore does not include the database. The database is outside the cluster and does not change. Steps 2 and 3 do not apply.

2. Rebuild the database from the repository

This step is a manual PostgreSQL restore. Do it only when the automatic PostgreSQL restore did not run or did not work.

The commands in steps 2 and 3 use $NS, $PGCLUSTER, and $INSTANCES. Set them for the restored cluster before you start. See Verify that both kinds of backup work.

kubectl -n "$NS" get netboxenterprise -o json \
  | jq -r '.items[].status.conditions[]
           | select(.type|test("PostgresRecoveryRequired|PostgresRestorePending|PostgresRestoreFailed"))
           | "\(.type)=\(.status) \(.reason)"'

kubectl -n "$NS" get jobs -l postgres-operator.crunchydata.com/pgbackrest-restore
kubectl -n "$NS" get postgrescluster "$PGCLUSTER" \
  -o jsonpath='{.status.pgbackrest.restore}{"\n"}'

PGO does a PostgreSQL restore when the resource has a data source and the restore is enabled:

kubectl -n "$NS" patch postgrescluster "$PGCLUSTER" --type=merge -p '{
  "spec": {
    "backups": {
      "pgbackrest": {
        "restore": {
          "enabled": true,
          "repoName": "repo1",
          "options": []
        }
      }
    }
  }
}'

# Trigger the restore.
kubectl -n "$NS" annotate postgrescluster "$PGCLUSTER" \
  postgres-operator.crunchydata.com/pgbackrest-restore="$(date +%s)" --overwrite

Read the patch back before you trigger the PostgreSQL restore. The operator owns parts of this resource, so a reconcile between the two commands can revert the field. The annotation then triggers a cluster that has no restore enabled:

kubectl -n "$NS" get postgrescluster "$PGCLUSTER" \
  -o jsonpath='{.spec.backups.pgbackrest.restore}{"\n"}'
# {"enabled":true,"options":[],"repoName":"repo1"}

Recovery target

An empty options list recovers as much as possible. pgBackRest restores the newest PostgreSQL backup and replays the WAL archive to its end. Use this value unless you have a reason to use a different one.

Two other targets are available, and one form fails.

--type=immediate stops at the PostgreSQL backup. It recovers only until the database is consistent, and it discards each transaction after the backup. Full PostgreSQL backups are weekly by default, so this target can lose days of changes, and it still reports success. The WAL for that period is in the same repository. Use this target only when you want the PostgreSQL backup without the later changes.

For point-in-time recovery, the target must not contain a space. A time target also needs --set. Two separate rules apply, and a form that obeys only one rule fails.

The first rule comes from PGO. PGO adds options to the command line of the restore Job without quotes. A target such as "2026-08-28 23:20:00+00" therefore splits at the space:

ERROR: [096]: command does not allow parameters

The second rule comes from pgBackRest. Its automatic backup-set selection cannot read a timestamp with a T. A target with the space removed therefore causes a different error:

ERROR: [029]: automatic backup set selection cannot be performed with
provided time '2026-08-28T23:20:00+00'

This message suggests that the target is outside the archive. Usually it is not. Only the set selection cannot read the value. pgBackRest still sends the target to PostgreSQL, which accepts ISO 8601 with a T. When you name the backup set, pgBackRest does not do the selection. A backup set is one PostgreSQL backup, as pgbackrest info lists it.

These two forms work, and neither contains a space. Recover to a time, and name the set:

"options": [
  "--type=time",
  "--target=2026-08-28T23:20:00+00",
  "--set=20260828-230000F"
]

Recover to a log sequence number (LSN). This form does not need --set:

"options": ["--type=lsn", "--target=0/A000000"]

Get the --set value from pgbackrest info. Choose a PostgreSQL backup that finished before your target time. The target must also be inside the WAL archive range that pgbackrest info shows.

A rejected form does not fail only once. The Job tries again and fails again, and PostgreSQL stays stopped for all of that time. That adds an outage to the disaster that you recover from.

Make sure that the PostgreSQL restore wrote the correct target while the recovery is still in progress.

First, find the pod and the data directory. Both names vary. PGO names the instance StatefulSet after your cluster and its instance set. The directory name includes your PostgreSQL major version. A wrong value for either one makes the check below fail in the same way as the harmless case that follows it.

PGPOD="$(kubectl -n "$NS" get pod -l "$INSTANCES" -o name | head -1)"
PGDIR="$(kubectl -n "$NS" exec "$PGPOD" -c database -- \
  sh -c 'ls -d /pgdata/pg[0-9]* 2>/dev/null | grep -E "/pg[0-9]+$" | head -1')"
echo "pod=$PGPOD dir=$PGDIR"

Both variables must have a value. An empty PGDIR means that the data directory does not exist yet, so the recovery did not write it.

kubectl -n "$NS" exec "$PGPOD" -c database -- \
  grep recovery_target "$PGDIR/postgresql.auto.conf"

Only now is a non-zero exit status important. During recovery, if no recovery_target_* line exists, the target did not reach PostgreSQL. The PostgreSQL restore then ran to the end of the WAL.

PostgreSQL removes these settings when recovery ends. After a fast PostgreSQL restore, this command therefore exits 1 on a cluster that recovered correctly. In that case, the missing line tells you nothing.

The log of the restore pod also records the target. Read it while the pod exists:

kubectl -n "$NS" logs -l postgres-operator.crunchydata.com/pgbackrest-restore \
  --tail=-1 | grep -iE "recovery_target|starting point-in-time recovery"

PGO removes the restore Job when the Job finishes. kubectl logs with a selector that matches no pod prints nothing and exits 0. Empty output here therefore means that the pod is gone, not that the target was missing. If you need this log, save it during the recovery.

Monitor the PostgreSQL restore

Monitor the PostgresCluster status, not a Job named *restore*. PGO removes the Job, so a check for the Job can show a failure even after a successful PostgreSQL restore.

kubectl -n "$NS" get postgrescluster "$PGCLUSTER" \
  -o jsonpath='{.status.pgbackrest.restore}{"\n"}'
# {"id":"1785424164","startTime":"...","completionTime":"...","finished":true}

finished: true with an id that matches your annotation means that pgBackRest finished the write of the restored data.

This is necessary, but it is not sufficient. The status reports on the pgBackRest part only. PostgreSQL must still bootstrap on the restored files. That step can fail again and again while the status above still shows success. The PostgreSQL restore is not complete until the database container is Ready. See step 3.

How long to wait before you treat the restore as stuck. These two disaster recovery restores used the same build and the same backup store, and both succeeded:

The slow restore was in a retry cycle and reported nothing for that time. It then completed and returned every row, which included one row that existed only in the WAL archive.

A single-node restore that shows no progress for forty minutes is therefore not necessarily stuck. Look for progress, not for elapsed time. Monitor these three signs:

• New PostgreSQL backups in pgbackrest info.
• New WAL segments in the log of the restore pod.
• PostgresDataInitialized on the PostgresCluster.

If none of them changes for 30 minutes, start to investigate.

The restore status becomes stale after a success. PGO stops the updates to status.pgbackrest.restore when it removes the Job. Hours later, the field can still show a start time and finished: false on a cluster that serves all its data. Compare the startTime with the current time. Then check two live facts: whether a restore Job exists, and whether the database container is Ready.

A failed count in the same status shows failed attempts. The first one or two attempts of a disaster recovery restore can fail, and a later attempt can still succeed. A count that continues to increase means that the PostgreSQL restore fails in a loop. After several failed attempts, the operator reports PostgresRestoreFailed on the NetBoxEnterprise resource, keeps Ready from True, and sends a Warning event. See Restore fails in a loop.

Disable the restore setting when the PostgreSQL restore completes. If you do not, a later reconcile can trigger it again. A second trigger is another destructive PostgreSQL restore, over a database that is live.

kubectl -n "$NS" patch postgrescluster "$PGCLUSTER" --type=merge \
  -p '{"spec":{"backups":{"pgbackrest":{"restore":{"enabled":false}}}}}'

# Read it back. Expect false.
kubectl -n "$NS" get postgrescluster "$PGCLUSTER" \
  -o jsonpath='{.spec.backups.pgbackrest.restore.enabled}{"\n"}'

3. Verify

Set $NS, $PGCLUSTER, and $INSTANCES for the restored cluster first. See Verify that both kinds of backup work. If one of them is empty, the checks below look at the wrong cluster or at nothing.

Do all four checks. The first two checks show that PostgreSQL runs. Only the last two show that your data came back and that PostgreSQL backups run again.

# 1. Every instance pod is fully Ready. This, not the restore status, is what
#    says PostgreSQL came back. A pod short of its full container count has not
#    bootstrapped.
kubectl -n "$NS" get pods -l "$INSTANCES"

# 2. All instances report ready.
kubectl -n "$NS" get postgrescluster "$PGCLUSTER" \
  -o jsonpath='{.status.instances[*].readyReplicas}{"\n"}'

# 3. NetBox can reach the database, and its data is present. Compare both
#    numbers against the baseline you recorded before the loss.
NBPOD="$(kubectl -n "$NS" get pod -l 'app.kubernetes.io/component=netbox' \
  -o name | head -1)"

kubectl -n "$NS" exec "$NBPOD" -c netbox -- \
  python manage.py shell -c \
  'from core.models import ObjectChange
from dcim.models import Site
print("sites:", Site.objects.count())
print("newest change:", ObjectChange.objects.order_by("-time").values_list("time", flat=True).first())'

# 4. Scheduled backups have returned. The operator empties this cluster's
#    pgBackRest schedules while a recovery is outstanding, so an unrecovered
#    cluster silently has no backups.
kubectl -n "$NS" get postgrescluster "$PGCLUSTER" \
  -o jsonpath='{.spec.backups.pgbackrest.repos[0].schedules}{"\n"}'

# No restore condition should remain on the resource.
kubectl -n "$NS" get netboxenterprise -o json \
  | jq -r '.items[].status.conditions[]
           | select(.type|startswith("PostgresR"))
           | "\(.type)=\(.status) \(.reason)"'

The first command must print your fullSchedule and incrementalSchedule values. The second command must print nothing.

Empty output from the first command is the held state, not an error. The operator removes the PostgreSQL backup schedules while a disaster recovery restore is in progress. You do not need to turn anything on again. The operator checks the state on each reconcile, and the schedules come back when the database is back. If the database serves data and the schedules are still missing, the second command shows the condition that remains.

If the cluster is healthy but empty, the PostgreSQL restore recovered an empty backup set, not your data. Treat this as a failed disaster recovery restore.

PostgreSQL backups

pgBackRest makes full and incremental PostgreSQL backups in the repository. They run on a schedule by default. The repository has two uses:

• Disaster recovery. A disaster recovery backup copies the repository, so a disaster recovery restore can rebuild the database. See Disaster recovery.
• A PostgreSQL restore to an earlier point. On a running install, you can restore the database to an earlier PostgreSQL backup, or to a time between them, without a disaster recovery restore. See PostgreSQL restore to an earlier point.

The repository is not a high-availability feature. In an HA install, Patroni can also build a replica from the repository, but that is a side effect.

Configuration

All settings are under spec.postgresql.backups. On a supported storage class, the default settings give working PostgreSQL backups with no configuration. They do not give disaster recovery backups. Disaster recovery backups need backup storage and a schedule, and on Helm they also need Velero. See How disaster recovery and PostgreSQL backups work together. On a hostPath storage class, disaster recovery backups do not hold the repository, and no message tells you. See Storage requirements.

spec:
  postgresql:
    backups:
      enabled: true
      repoStorageSize: 16Gi
      repoStorageClassName: ""
      retentionFull: 4
      fullSchedule: "0 1 * * 0"
      incrementalSchedule: "0 1 * * 1-6"

See spec.postgresql.backups for the full field reference.

Three rules apply:

1. Retention is by count, not by age. The repository volume has a fixed size, so the number of PostgreSQL backups controls the disk usage. When pgBackRest expires a full backup, it also expires the incremental backups and the WAL that depend on it.
2. Point-in-time recovery reaches back only to the oldest retained full backup. The recovery window is therefore retentionFull backup cycles. It is not a separate setting.
3. In 2.4.0, you cannot turn off one schedule. Each schedule must be a five-field cron expression. The operator rejects an empty string: the reconcile fails, and Ready becomes False with ReconcileFailed. enabled: false removes pgBackRest completely.

PostgreSQL backup schedule

PostgreSQL backups run on a schedule by default. This schedule is separate from the schedule for disaster recovery backups, which has no default. See How disaster recovery and PostgreSQL backups work together.

The admin console sets Full Backup Schedule only. See Change the retention or the full-backup schedule. To change incrementalSchedule, change the NetBoxEnterprise resource. On a Helm install, set the value under netboxEnterprise.spec.postgresql.backups in your values.

Annotations do not control the schedule. Neither the operator nor PGO reads an annotation to set, pause, or start a scheduled PostgreSQL backup. To take a PostgreSQL backup now, see Taking a backup on demand.

PGO makes one CronJob for each schedule. To list them, set the variables in Verifying backups are working, then run:

kubectl -n "$NS" get cronjob -l "postgres-operator.crunchydata.com/cluster=$PGCLUSTER"

The PostgresCluster status records the recent scheduled PostgreSQL backups:

kubectl -n "$NS" get postgrescluster "$PGCLUSTER" \
  -o jsonpath='{.status.pgbackrest.scheduledBackups}{"\n"}'

During a disaster recovery restore, the operator removes the schedules, and PGO deletes the CronJobs. They come back when the database is recovered. See Disaster recovery restore procedure.

Why 4 full backups

The default keeps 4 full PostgreSQL backups for these reasons:

• Two is too few. After a disaster recovery restore, the restored cluster makes its own PostgreSQL backups in the same repository. It makes an incremental backup immediately, and then a full backup each Sunday. With 2 kept, the second Sunday full backup expires the last full PostgreSQL backup of the source cluster, so the source is gone within two weeks.
• One is too few. If a full backup fails, you have nothing to restore.
• Four gives about three weeks after a disaster recovery restore before the PostgreSQL backups of the source expire.
• Four goes with the default repoStorageSize of 16Gi. A larger default would make existing volumes grow on upgrade, and Embedded Cluster storage cannot make a volume larger. 16Gi is not enough for every database. The space that you need depends on the database size, the incremental backups, and the WAL. See Repository size and retention.

With PostgreSQL backups off, nothing backs up the database

enabled: false removes the repository, and disaster recovery backups still exclude the PostgreSQL data volume. Nothing then backs up the database.

This is intentional. A torn copy is not a backup, and a torn copy that looks restorable is worse than no copy. enabled: false therefore turns off PostgreSQL disaster recovery. It does not select a different backup method.

A disaster recovery restore of such a cluster returns everything except the database. PostgreSQL then starts a new, empty instance, because no repository exists to rebuild from.

On a running cluster, enabled: false is also not a clean revert. PGO keeps the repository StatefulSet and its volume, and it pauses its reconciliation of the cluster. The pause is total, and it is easy to miss. No spec change reaches the database, no upgrade applies, and no status changes, but the database continues to serve.

The operator reports the pause on the NetBoxEnterprise resource as PostgresReconciliationPaused=True. This condition is advisory and does not change Ready. The fix is a configuration change, and Ready controls both the Admin Console install gate and helm upgrade --wait. To resume, enable PostgreSQL backups again.

Volume sizes and disk space

NetBox Enterprise makes five persistent volumes. These are the default sizes:

The two PostgreSQL volumes are the ones that grow without your action. The database grows with your data. The repository grows with each PostgreSQL backup and with the WAL archive. See Repository size and retention to calculate the repository size.

Embedded Cluster: all volumes share one disk

Embedded Cluster provisions each volume with openebs-hostpath, under /var/lib/embedded-cluster/openebs-local on the node. This storage class does not enforce a size. Each volume can grow until the disk of the node is full, and a full disk stops all of the install, not only the volume that filled it. df in a pod shows the whole disk of the node, not the size of the volume.

The admin console has no setting for the volume sizes, and this storage class cannot make a volume larger. Plan the disk instead. The requirements give 150 GB under /var/lib. On a test install, the install itself used about 20 GB of the disk before it held any data. The database, the repository, and the media share the rest.

To keep the repository smaller, lower the retention, or take full backups more frequently. See Change the retention or the full-backup schedule.

Helm: the storage class decides

Set the sizes and the storage classes in your values when you install:

netboxEnterprise:
  spec:
    postgresql:
      storageSize: "20Gi"
      backups:
        repoStorageSize: "64Gi"
    netbox:
      mediaStorageSize: "20Gi"

To use a storage class other than the default, set storageClassName and repoStorageClassName. See spec.postgresql. A storage class that enforces sizes, such as most CSI drivers, makes each size a hard limit. openebs-hostpath does not enforce sizes, as on Embedded Cluster.

You can make a volume larger after the install only if its storage class has allowVolumeExpansion: true:

kubectl get storageclass

You cannot make a volume smaller. Kubernetes rejects a size that is less than the current capacity.

A size change that Kubernetes rejects has a different result for each volume:

After a size change, compare the size that you set with the claims:

kubectl -n "$NS" get postgrescluster "$PGCLUSTER" \
  -o jsonpath='{range .status.conditions[?(@.type=="PersistentVolumeResizing")]}{.status} {.reason}: {.message}{"\n"}{end}'

kubectl -n "$NS" get pvc \
  -o custom-columns='NAME:.metadata.name,REQUEST:.spec.resources.requests.storage,CAPACITY:.status.capacity.storage'

If the first command prints False, set the size back to the current capacity.

What fills the PostgreSQL data volume

The data volume holds the database and its WAL. PostgreSQL normally keeps about 1 GB of WAL, the value of max_wal_size. With the 4Gi default, a database larger than about 2 GB therefore leaves little free space.

PostgreSQL keeps each WAL segment until pgBackRest archives it to the repository. If archiving fails, the WAL on the data volume grows with each change, and nothing sets a limit. When the data volume is full, PostgreSQL stops.

Archiving fails, for example, when the repository is full or when the repository host does not run. The NetBoxEnterprise resource does not show this failure. In a test with blocked archiving, Ready stayed True, and the WAL on the data volume grew from 1.1 GB to 2.0 GB in a few minutes of writes. PostgresBackupRepoPressure measures the repository only, so it does not show the WAL that waits on the data volume.

When you correct the cause, pgBackRest archives the WAL that waits. PostgreSQL then frees the space at the next checkpoint.

Measure the usage

Set the variables in Verify that both kinds of backup work. Then run:

PGPOD="$(kubectl -n "$NS" get pod -l "$INSTANCES" -o name | head -1)"
REPO_HOST_POD="$(kubectl -n "$NS" get pod -l "$REPO_HOST" -o name | head -1)"

# 1. The data volume: used and free space, and the size of each database.
kubectl -n "$NS" exec "$PGPOD" -c database -- df -h /pgdata
kubectl -n "$NS" exec "$PGPOD" -c database -- psql -qAt -c \
  "SELECT datname, pg_size_pretty(pg_database_size(datname))
     FROM pg_database WHERE NOT datistemplate
    ORDER BY pg_database_size(datname) DESC"

# 2. The WAL on the data volume, and the state of WAL archiving.
kubectl -n "$NS" exec "$PGPOD" -c database -- sh -c 'du -sh /pgdata/pg*_wal'
kubectl -n "$NS" exec "$PGPOD" -c database -- psql -qAt -c \
  "SELECT last_archived_time, last_failed_time, failed_count
     FROM pg_stat_archiver"

# 3. The repository: used and free space, the PostgreSQL backups, and the WAL archive.
kubectl -n "$NS" exec "$REPO_HOST_POD" -c pgbackrest -- df -h /pgbackrest/repo1
kubectl -n "$NS" exec "$REPO_HOST_POD" -c pgbackrest -- \
  du -sh /pgbackrest/repo1/backup /pgbackrest/repo1/archive

Read the results:

• The data volume. Compare the database sizes and the WAL size with storageSize. On openebs-hostpath, df shows the disk of the node.
• WAL archiving. If last_failed_time is later than last_archived_time, archiving fails now. failed_count is a total since PostgreSQL started, so a count that is not zero alone is not a problem. A new cluster usually has one failure from before pgBackRest made the stanza.
• The repository. backup holds the PostgreSQL backups, and archive holds the WAL archive.

To project the growth, run the commands again after 24 hours, at a time when no full backup ran between the two runs. The growth of archive is the WAL that your database writes each day, after compression. Use it in the formula in Repository size and retention. The growth of the database sizes gives the growth of the data volume.

Repository size and retention

The repository holds retentionFull full PostgreSQL backups, their incremental backups, and all WAL back to the oldest full backup. PostgreSQL archives WAL all the time, independent of the PostgreSQL backup schedule. The repository size is therefore about:

retentionFull × (one compressed full backup)
  + (the incremental backups that depend on those full backups)
  + (days between the oldest full backup and now) × (WAL written per day)

On a busy database, the WAL is the larger part. With weekly full backups and the default of 4, the repository keeps up to about four weeks of WAL. For example, a database that writes 19 GB of WAL each day, with compressed full backups of 29 GB, needs about:

The peak is one full backup larger, because pgBackRest expires the oldest set only after the next full backup finishes. The table does not include incremental backups, which add to each row. More frequent full backups make the repository smaller, but each full backup copies the whole database.

To size repoStorageSize, follow these steps:

1. Calculate the formula at your current WAL rate and incremental backup size.
2. Add one full backup for the peak.
3. Add free space for growth. The PostgresBackupRepoPressure condition becomes True when the filesystem is 70% full. Choose a size so that the total from steps 1 and 2 is less than 70% of it.

Do not delete or move files in the repository manually. If backup/ is missing, every PostgreSQL backup fails, and the WAL archive continues to grow.

The size is not a limit on Embedded Cluster

Embedded Cluster provisions the repository volume with openebs-hostpath, which does not enforce repoStorageSize. The repository can grow until the disk of the node is full. This storage class also cannot expand a volume. A larger repoStorageSize on an existing Embedded Cluster install does not make the volume larger, and the NetBoxEnterprise resource shows no message. See Helm: the storage class decides for how to see the rejected change. Lower the retention, or make full backups more frequent, instead.

On a pure Helm install, choose repoStorageSize when you install. A storage class that cannot expand a volume cannot make it larger later.

Monitor the PostgresBackupRepoPressure condition

From 2.4.0, the operator measures the repository and its filesystem every hour. The PostgresBackupRepoPressure condition becomes True when the filesystem is 70% full or more, or when the repository is larger than repoStorageSize:

kubectl -n "$NS" get netboxenterprise -o jsonpath='{range .items[*].status.conditions[?(@.type=="PostgresBackupRepoPressure")]}{.status} {.reason}: {.message}{"\n"}{end}'

The condition does not change Ready. See Status Conditions for all reasons.

Change the retention or the full-backup schedule

On an admin console install, set Full Backups to Keep and Full Backup Schedule in the PostgreSQL group of the Config page. Then deploy. These options control PostgreSQL backups, not disaster recovery backups. They show only when the built-in database is enabled. If you leave them empty, the defaults apply: 4 full backups, and a full backup on Sundays at 01:00. On a Helm install, set netboxEnterprise.spec.postgresql.backups.retentionFull and netboxEnterprise.spec.postgresql.backups.fullSchedule in your values.

To lower the retention with a patch, run the command below. For example, use it on Embedded Cluster, where you cannot make the repository larger. Replace 2 with your value. Then expire the old PostgreSQL backups as described in Free space now.

NBE="${PGCLUSTER%-postgres}"
kubectl -n "$NS" patch netboxenterprise "$NBE" --type merge \
  -p '{"spec":{"postgresql":{"backups":{"retentionFull":2}}}}'

• Keep at least 2 full backups. The admin console does not accept fewer. After a disaster recovery restore, the next full PostgreSQL backup of the restored cluster takes a slot. With one slot, that backup expires the PostgreSQL backup that you restored from.
• Do not schedule a full backup at 01:00 from Monday to Saturday. The incremental backup runs at that time, and PGO does not run two PostgreSQL backups at the same time. For a nightly full backup, use 0 3 * * *.
• A value that you set with kubectl patch stays while Full Backups to Keep is empty. If you then set a different value in the admin console, the deploy fails with a field-manager conflict. The SpecManuallyEdited condition names the field and the field manager. Remove the entry for that field manager from metadata.managedFields on the NetBoxEnterprise resource, then deploy again.

Free space now

A lower retention frees space only when pgBackRest expires the old PostgreSQL backups. pgBackRest does this after the next PostgreSQL backup. To free the space now, set NS and REPO_HOST as in Verifying backups are working. Then run:

REPO_HOST_POD="$(kubectl -n "$NS" get pod -l "$REPO_HOST" -o name | head -1)"

# 1. Wait until the repository host has the new value. Replace 2 with your value.
until kubectl -n "$NS" exec "$REPO_HOST_POD" -c pgbackrest -- \
    sh -c 'grep -h "retention-full =" /etc/pgbackrest/conf.d/*.conf' | grep -q '= 2$'; do
  sleep 5
done

# 2. Expire the sets that the new retention does not keep.
kubectl -n "$NS" exec "$REPO_HOST_POD" -c pgbackrest -- pgbackrest expire --stanza=db

# 3. Confirm "status: ok" and the expected number of full backups.
kubectl -n "$NS" exec "$REPO_HOST_POD" -c pgbackrest -- pgbackrest info

On a test install, the new value took about 75 seconds to reach the repository host.

Upgrades: a disaster recovery backup from before 2.3.0 does not hold the database

After you upgrade from a release earlier than 2.3.0, take a full PostgreSQL backup immediately, and then a disaster recovery backup. A disaster recovery backup from before the upgrade cannot restore your database.

Each release before 2.3.0 disabled pgBackRest and did not exclude the PostgreSQL data directory from Velero. An earlier disaster recovery backup therefore holds a torn copy of a live data directory and no repository.

You cannot recover the database from such a backup. The torn copy is not a valid PostgreSQL data directory, and no PostgreSQL backup or WAL exists to use instead.

The first valid restore point for the database is the first PostgreSQL backup after the upgrade. A disaster recovery restore can use it only from a disaster recovery backup that you take after that PostgreSQL backup.

You can safely turn on pgBackRest for a running cluster. Existing data does not change. PGO adds a repository host and the pgBackRest sidecars, and it creates the stanza against the live database.

During the upgrade, short UnableToCreateStanzas warnings can occur while PGO restarts the instance to add the sidecar. They clear without action.

Wait for a successful PostgreSQL backup, and take that backup yourself. Do not wait for the schedule. fullSchedule is weekly by default, so after an upgrade on a Monday you can have no valid restore point for six days. See Taking a backup on demand, then confirm the new PostgreSQL backup with Verifying backups are working.

Verify that both kinds of backup work

Set the namespace and two selectors first. All commands on this page use them, so set them in your shell. NS is the namespace of the NetBoxEnterprise resource. On Embedded Cluster, it is kotsadm. On Helm, it is your release namespace. PGCLUSTER is the name of the NetBoxEnterprise resource plus -postgres.

NS=netbox          # kotsadm on Embedded Cluster; your release namespace on Helm
PGCLUSTER=netbox-postgres

CLUSTER="postgres-operator.crunchydata.com/cluster=$PGCLUSTER"

# The dedicated pgBackRest repository host.
REPO_HOST="$CLUSTER,postgres-operator.crunchydata.com/pgbackrest-dedicated="

# The PostgreSQL instance pods.
INSTANCES="$CLUSTER,postgres-operator.crunchydata.com/data=postgres"

An unset NS is the problem to look for. kubectl -n "" uses your current context and shows no message. Some checks below then return No resources found, which looks like a real defect. Before you trust a check, make sure that the namespace is correct:

kubectl -n "$NS" get postgrescluster "$PGCLUSTER"

PostgreSQL backups. Select the repository host with pgbackrest-dedicated, not with the plain postgres-operator.crunchydata.com/pgbackrest label. PGO also puts the plain label on the pods of its backup Jobs, and those pods stay after the Job ends. A selector on the plain label can therefore select a finished Job pod, and kubectl exec into that pod fails.

# The repository host pod should be Running.
kubectl -n "$NS" get pods -l "$REPO_HOST"

# List the backup sets pgBackRest currently holds. This is the authority.
kubectl -n "$NS" exec -it \
  "$(kubectl -n "$NS" get pod -l "$REPO_HOST" -o name | head -1)" \
  -c pgbackrest -- pgbackrest info --stanza=db

Read the timestamp of the newest PostgreSQL backup, not only the count. A six-week-old backup satisfies "at least one full backup" on a cluster whose repository volume is full, or whose schedules the operator holds after a disaster recovery restore. Compare the newest PostgreSQL backup with fullSchedule and incrementalSchedule. If it is older than one schedule interval, PostgreSQL backups stopped.

An empty repository prints No stanzas exist in the repository. and still exits 0. This means that no PostgreSQL backup ran. Either the schedule time did not come yet, or the stanza creation failed.

Events are a weaker signal. Read them second, and only to explain what pgbackrest info showed:

kubectl -n "$NS" describe postgrescluster "$PGCLUSTER" | sed -n '/Events/,$p'

An UnableToCreateStanzas warning here means that PostgreSQL backups are not initialized. During an upgrade, this warning is expected and clears without action. If you do not see the warning, that proves nothing. Kubernetes deletes events after about an hour, so a failure from yesterday leaves no record. Use pgbackrest info to decide.

Disaster recovery backups. A Completed disaster recovery backup alone is not proof. Make sure that it holds volume data:

kubectl -n velero get podvolumebackups \
  -l velero.io/backup-name="$BACKUP" \
  -o custom-columns='VOLUME:.spec.volume,PHASE:.status.phase,BYTES:.status.progress.bytesDone'

You must see one Completed row for each persistent volume. The repository volume must be in the list with a non-zero byte count, because it is the only copy of your database in the disaster recovery backup. An empty volume, such as media on a new install, shows <none> in the BYTES column, and that is correct.

An empty result has two causes. Check the easy one first: compare the backup name with velero backup get, because a name that matches nothing gives the same empty list. If the name is correct, an empty result means that Velero captured no volume data. The probable cause is a hostPath storage class.

Take a PostgreSQL backup on demand

Take a full PostgreSQL backup before an upgrade. Run pgBackRest on the repository host:

REPO_HOST_POD="$(kubectl -n "$NS" get pod -l "$REPO_HOST" -o name | head -1)"

kubectl -n "$NS" exec "$REPO_HOST_POD" -c pgbackrest -- \
  pgbackrest backup --stanza=db --type=full

--type=incr makes an incremental PostgreSQL backup against the newest full backup instead. Make sure that the new PostgreSQL backup exists:

kubectl -n "$NS" exec "$REPO_HOST_POD" -c pgbackrest -- pgbackrest info --stanza=db

A PostgreSQL backup stays in the cluster. To also have it off the cluster, take a disaster recovery backup after it.

PostgreSQL restore to an earlier point

On a running install, you can restore the database to an earlier PostgreSQL backup, or to a time between them. This is a PostgreSQL restore only. It does not need a disaster recovery restore. The repository keeps older full backups until pgBackRest expires them.

1. Find the target. Run pgbackrest info on the repository host, as in Verify that both kinds of backup work. Record the label of the backup set and the WAL archive range.
2. Choose the options for the target. Recovery target gives the rules and examples.
3. Enable maintenance mode. On Embedded Cluster, see Enable and disable maintenance mode. On Helm, see Maintenance Mode. Maintenance mode stops NetBox, its workers, and Diode, and PostgreSQL continues to run. If you do not enable it, NetBox writes to the database again as soon as PostgreSQL restarts.
4. Apply the patch and the annotation from step 2, with your options in place of the empty list. Read the patch back before you add the annotation.
5. Monitor the PostgreSQL restore. See Monitor the PostgreSQL restore. Continue only when every instance pod is Ready.
6. Disable the restore setting.
7. Disable maintenance mode.
8. Do the checks in Verify. Compare the results with your recovery target, not with the baseline from before the restore. The changes after the target are intentionally absent.
9. Take a full PostgreSQL backup. See Take a PostgreSQL backup on demand. The restore starts a new timeline, and the PostgreSQL backups before it are on the old one. Without a full backup on the new timeline, a later disaster recovery restore can fail with ERROR: [058]. See PostgreSQL restore fails in a loop.

What not to do

• Do not delete or move files in the repository manually. See Repository size and retention.
• Do not increase repoStorageSize on Embedded Cluster. Its storage class cannot expand a volume, so the change has no effect, and no message shows it. Lower retentionFull instead.
• Do not turn off PostgreSQL backups. Nothing then backs up the database. See With PostgreSQL backups off, nothing backs up the database.
• Do not keep fewer than 2 full PostgreSQL backups. See Change the retention or the full-backup schedule.
• Do not use an annotation to change the PostgreSQL backup schedule. See PostgreSQL backup schedule.

Troubleshooting

See Troubleshoot PostgreSQL Backups and Disaster Recovery. It covers these problems:

• Stanza creation fails
• PostgreSQL restore fails in a loop
• No PostgreSQL pods after a Velero restore
• PostgreSQL pods appear immediately after a Velero restore
• PostgreSQL restore reports success but PostgreSQL never starts
• The repository volume is full
• PostgreSQL backups run but the disaster recovery backup does not contain them

Limitations

• On a single-node install, the repository and the database can fail together. On Embedded Cluster, both volumes are on the node. PostgreSQL backups protect against database corruption and operator error. To survive the loss of the node, you need a disaster recovery backup, which is outside the cluster.
• No cross-cluster repository is available. pgBackRest supports an object-store repository, but NetBox Enterprise does not make it available. PostgreSQL backups and WAL are not in the NetBox media bucket on purpose, because that bucket has different retention, access, and capacity requirements. A separate object-store target for PostgreSQL backups is not available yet.
• Point-in-time recovery reaches back only to the oldest retained full backup. When pgBackRest expires a full backup, it also expires the WAL of that backup. The window is therefore retentionFull backup cycles.
• Neither install path supports air-gapped disaster recovery. An air-gapped install needs a relocated restore-helper image, and the plugin config of the chart cannot specify one.

See Also

• Backup Essentials - The short version
• Backup and Restore in Depth - Admin Console backup and restore, and manual dumps
• Backups (Helm) - Velero installation and the chart's disaster-recovery objects
• spec.postgresql.backups - Full field reference
• Status & Conditions - The restore conditions the operator reports