Backups

This page covers disaster recovery backups on a Helm install. A disaster recovery backup is a Velero backup of the whole install. On Helm, you install Velero and you schedule the disaster recovery backups. NetBox Enterprise does not schedule them for you.

PostgreSQL backups are separate. pgBackRest makes them on a schedule by default, on both install paths. They stay in the cluster, so they do not protect against the loss of the cluster. PostgreSQL Backups and Disaster Recovery defines the terms, and covers PostgreSQL backups and the parts of a restore that are the same on both install paths.

Prerequisites

• A running NetBox Enterprise Helm install.
• A storage class that provisions local, csi, or network PersistentVolumes. On a hostPath storage class, a disaster recovery backup holds no volume data and shows no error. See Storage requirements.
• Velero 1.13 or later. An earlier Velero fails each disaster recovery backup at validation.
• S3-compatible object storage, such as AWS S3 or MinIO, for the disaster recovery backups.
• The velero CLI on your workstation (installation guide).

Install Velero

Install Velero with the node agent enabled. The node agent makes the filesystem backups of the volumes. The example below uses the AWS S3 provider. See Velero supported providers for the other providers.

1. Create a credentials file for your S3 bucket:

[default]
aws_access_key_id=<YOUR_ACCESS_KEY>
aws_secret_access_key=<YOUR_SECRET_KEY>

2. Install Velero in the cluster:

velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.11.1 \
  --bucket <BUCKET_NAME> \
  --secret-file ./credentials-velero \
  --backup-location-config \
    region=<REGION>,s3ForcePathStyle="true",s3Url=<S3_ENDPOINT> \
  --use-node-agent

For MinIO or another S3-compatible store, set s3Url to the endpoint, such as https://minio.example.com. For AWS S3, remove s3Url and s3ForcePathStyle.

If more than one cluster uses the same bucket, give each cluster its own prefix with --prefix <CLUSTER_NAME>. With a shared bucket and prefix, velero backup get lists the disaster recovery backups of all the clusters, and nothing identifies the cluster of each one. A disaster recovery restore from the backup of another cluster completes, reports success, and brings your cluster up with the data of the other cluster.

3. Make sure that Velero is ready:

velero version
kubectl -n velero get pods

Enable the chart's disaster recovery objects

Set one value:

disasterRecovery:
  createVeleroSpecs: true

The chart then creates four objects in the velero namespace:

The chart does not include a Schedule or a Restore. See Schedule disaster recovery backups and Disaster recovery restore.

The pod-volume-restore plugin config

Velero restores a filesystem-backed volume with a restore-wait init container. It reads the security context of that container from a ConfigMap in the velero namespace. It finds the ConfigMap by label selector, and it needs exactly one match.

If no ConfigMap matches, Velero copies the security context from the first container of the restored pod. The PGO containers request a non-root user but do not give a numeric user ID, so kubelet refuses the init container. No volume restore runs. The repository comes back empty, and all other objects restore normally.

Embedded Cluster needs no action. NetBox Enterprise adds its settings to the ConfigMap that the platform supplies.

A pure-Helm install uses its own Velero, and the chart creates the ConfigMap for it. Set disasterRecovery.createVeleroSpecs: true in your values. See Enable the chart's disaster recovery objects.

The operator counts the ConfigMaps on each reconcile. When the count is not one, it sets the VeleroRestoreConfigInvalid condition. On a correct cluster, the condition is absent. See Status & Conditions for its five reasons. The condition is advisory. The running cluster has no problem. The condition tells you that a future disaster recovery restore cannot return your data.

If the condition is absent, that alone proves nothing. The condition is also absent when Velero is not installed. It is also absent under rbac.scope: namespace, because the checks read other namespaces, and the operator sends nothing instead of a false alarm. On a namespace-scoped install, count the ConfigMaps yourself:

kubectl get configmap -n velero \
  -l 'velero.io/plugin-config,velero.io/pod-volume-restore=RestoreItemAction'

One is correct.

On upgrade, remove your own copy

Do this on each upgrade to 2.3.0 or later, even if you do not change your values. Earlier guidance told you to apply this ConfigMap manually, in the same step that set disasterRecovery.createVeleroSpecs: true. If you did that, the setting is already on, and helm upgrade alone adds a second matching ConfigMap.

Then two ConfigMaps have both labels, the Velero lookup fails, and every volume restore stops before Velero adds the init container.

You get no warning at that time. Disaster recovery backups continue to succeed, because Velero reads the plugin config only during a restore. The first symptom is a disaster recovery restore that returns empty volumes. The chart cannot see your copy, so you must do this step.

Find the copy by label, not by name. Velero counts matches on the label pair, so each ConfigMap with both labels is a match, whatever its name:

kubectl get configmap -n velero \
  -l 'velero.io/plugin-config,velero.io/pod-volume-restore=RestoreItemAction' \
  -o custom-columns='NAME:.metadata.name,MANAGED-BY:.metadata.labels.app\.kubernetes\.io/managed-by,KEYS:.data'

Remove each item in the list except netbox-enterprise-pod-volume-restore-config. The usual name is velero-fs-restore-action-config. The earlier guidance used that name, and the upstream Velero chart also makes a ConfigMap with that name. Your copy can therefore belong to your Velero release, not to you.

There are three possible results:

1. Nothing found. No action is necessary.

2. Only a secCtx key, and no app.kubernetes.io/managed-by: Helm label. This is the copy from the earlier guidance. Delete it in the same change that sets createVeleroSpecs: true:

   kubectl delete configmap -n velero velero-fs-restore-action-config

3. Anything else. This includes a managed-by: Helm label, or another key that Velero reads from this ConfigMap, such as image, command, cpuRequest, cpuLimit, memRequest, and memLimit. Do not delete this ConfigMap. A copy that Helm owns comes back on the next helm upgrade of your Velero release, so remove it from the values of that release instead. The copy from the chart has only secCtx and cannot hold the other keys. For example, an air-gapped install needs an image key for a relocated restore helper, and you lose it if you delete its ConfigMap. In that case, keep your ConfigMap as the only match, add the security context below to it, and leave disasterRecovery.createVeleroSpecs off. You then supply the Velero Backup, the volume policy, and the restore-modifier ConfigMap yourself.

After the change, count the ConfigMaps again. One is correct. Two is the collision. Zero means that the chart did not create its copy. In that case, make sure that createVeleroSpecs is set in the values that you installed the release with.

kubectl get configmap -n velero \
  -l 'velero.io/plugin-config,velero.io/pod-volume-restore=RestoreItemAction'

The setting applies to all of your Velero

Velero uses one pod-volume-restore plugin config for each Velero installation. It has no scope for each namespace or each application.

On Embedded Cluster, the appliance owns Velero. A pure-Helm install uses its own Velero, which can also serve other workloads on the same cluster. With createVeleroSpecs: true, every filesystem-backed volume restore in that cluster runs its init container as user 26 and group 26. Restored files then belong to 26:26. If the fsGroup of a volume excludes 26, the init container can fail to write to that volume. For the other workload, this shows as a PodVolumeRestore in phase Failed with a permission-denied message, while its Restore still reports Completed. If you share a Velero, run kubectl -n velero get podvolumerestores after the next restore of any workload, not only this one.

Without a config, no volume restore works, which includes yours, so that choice is worse. But if your Velero is shared, know this scope before you turn the setting on.

Three rules apply to the content. Read them before you change what the chart creates, or before you supply your own config:

1. Supply the full security context, not the shorthand keys. Velero builds the context from an empty one and adds only the keys that you give. Any key makes Velero skip its own hardened defaults. A partial security context therefore loses the other keys with no message.
2. Keep the user ID and the group ID at 26. The Crunchy images run PostgreSQL with this ID, so restored files get the ownership that pgBackRest expects.
3. Keep the count at one.

This is what the chart creates:

apiVersion: v1
kind: ConfigMap
metadata:
  name: netbox-enterprise-pod-volume-restore-config
  namespace: velero
  labels:
    velero.io/plugin-config: ""
    velero.io/pod-volume-restore: RestoreItemAction
data:
  secCtx: |
    runAsUser: 26
    runAsGroup: 26
    runAsNonRoot: true
    allowPrivilegeEscalation: false
    capabilities:
      drop:
        - ALL
    seccompProfile:
      type: RuntimeDefault

Label selection

The operator labels each resource that it manages. The Backup selects resources with any one of these labels:

The selection excludes the PostgreSQL data volume on purpose. It includes the repository volume, which is the source of the database in a disaster recovery restore. See Why the data volume is not backed up.

Do not widen the backup selectors

The selectors decide what goes into a disaster recovery backup. Do not widen them, and do not replace them with a backup of the whole namespace.

A backup of the whole namespace includes the PostgreSQL cluster-membership Endpoints, which hold the system identifier of the database. When a disaster recovery restore returns these Endpoints, PGO reads the identifier and decides that the cluster is already bootstrapped. PGO then skips the PostgreSQL restore. The database comes up empty, and no error shows the cause.

Make disaster recovery backups

The commands below use NS for the release namespace. Set it in your shell:

NS=netbox

Velero does not treat an --include-namespaces value that matches nothing as an error. The disaster recovery backup then reports Completed with zero items.

Schedule disaster recovery backups

The chart creates a one-time Backup, not a Schedule. Create a Schedule for recurring disaster recovery backups. This example runs each day at 02:00 UTC and keeps each backup for 30 days. It uses the same selectors and volume policy as the chart:

apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: netbox-enterprise-daily
  namespace: velero
spec:
  schedule: "0 2 * * *"
  template:
    includeClusterResources: true
    includedNamespaces:
      - netbox # your release namespace
      - velero
    defaultVolumesToFsBackup: true
    ttl: 720h
    resourcePolicy:
      kind: ConfigMap
      name: netbox-enterprise-backup-resource-filters
    orLabelSelectors:
      - matchLabels:
          app.kubernetes.io/managed-by: netbox-operator
      - matchLabels:
          netboxlabs.com/managed-by: netbox-operator
      - matchLabels:
          app.kubernetes.io/part-of: netbox-enterprise
      - matchLabels:
          postgres-operator.crunchydata.com/control-plane: pgo

A disaster recovery restore returns the install at about the time of the disaster recovery backup. The schedule therefore sets how much data a disaster recovery restore can lose.

Make a disaster recovery backup on demand

Make a dated disaster recovery backup, for example before an upgrade:

BACKUP="netbox-backup-$(date +%Y%m%d%H%M)"

velero backup create "$BACKUP" \
  --or-selector "app.kubernetes.io/managed-by=netbox-operator or netboxlabs.com/managed-by=netbox-operator or app.kubernetes.io/part-of=netbox-enterprise or postgres-operator.crunchydata.com/control-plane=pgo" \
  --include-namespaces "$NS",velero \
  --include-cluster-resources=true \
  --default-volumes-to-fs-backup \
  --resource-policies-configmap netbox-enterprise-backup-resource-filters \
  --wait

If you have a schedule, you can also start a disaster recovery backup from it now:

velero backup create --from-schedule netbox-enterprise-daily --wait

Check a disaster recovery backup

Use the name of the disaster recovery backup that you made. $BACKUP from the step above holds it. A cluster that followed the guidance from before 2.3.0 can still have an old backup with the name netbox-backup. velero backup describe on that backup reports Completed, but the backup can be months old.

velero backup get
velero backup describe "$BACKUP" --details
velero backup logs "$BACKUP"

Completed alone is not proof. Make sure that the disaster recovery backup holds volume data, and that pgBackRest holds a recent PostgreSQL backup. See Verify that both kinds of backup work.

Disaster recovery restore

A disaster recovery restore on Helm has two parts. First, you apply a Velero Restore, as this section describes. Then the operator does a PostgreSQL restore from the repository. The second part is the same on both install paths. See Disaster recovery restore procedure.

The chart's Backup is not a restore point

The chart creates one Velero Backup, netbox-enterprise-backup, and no Schedule. Velero runs a Backup one time, when the object is created. Velero deletes this Backup after 30 days. The Backup is a plain release resource, so the next helm install or helm upgrade after that creates it again, and Velero runs it again. This disaster recovery backup therefore holds the cluster at the last time Helm created it, which depends on your upgrade history. Between the deletion and the next upgrade, it does not exist. A disaster recovery restore from it loses all changes after that time.

Make your disaster recovery backups with a Schedule. Scheduled backup gives a Schedule that uses the same selectors and volume policy as the chart. On-demand backup shows how to make a dated disaster recovery backup before an upgrade.

Install the chart before the Restore

Two of the four objects must exist in the velero namespace before you create the Restore. Velero reads the resource modifier when you create the Restore. It reads the plugin config when it adds the init container. The chart creates both objects, so on a rebuilt cluster, install the chart first.

Install the chart with netboxEnterprise.enabled at its default value, false:

netboxEnterprise:
  enabled: false
disasterRecovery:
  createVeleroSpecs: true

This creates the operator and the four objects in the velero namespace, but no NetBoxEnterprise resource. There is therefore no PostgresCluster and no volume.

If you install with your production values, you get a live, empty cluster. The Restore below uses existingResourcePolicy: none. With this policy, Velero skips each object that already exists with the same name and namespace, and it does not patch a skipped object. Velero would skip the archived PostgresCluster and both volumes, and the resource modifier would not apply. The Restore would report Completed, but it would recover none of your data.

Before you continue, confirm that Velero has nothing to skip:

# Positive control first: prove the namespace exists and you are looking at it.
# A wrong or unset NS makes the next command pass unconditionally.
kubectl get namespace "$NS"

kubectl -n "$NS" get postgrescluster

The namespace must exist, and the second command must show No resources found. You need both results. The second command alone is not enough, because it shows the same line for a namespace that does not exist.

The disaster recovery backup holds the NetBoxEnterprise resource only if the chart created it, that is, if netboxEnterprise.enabled was true when the backup ran. The Backup selects objects by the labels of the chart. A resource that you applied manually has none of these labels unless you added one, so the disaster recovery backup does not hold it. In that case, apply your copy again when the Restore completes. In both cases, the operator then removes the pause and adds the data source. Set netboxEnterprise.enabled: true only if the chart created the resource and the resource did not come back.

The chart has no Restore: apply one manually

The chart does not include a Restore, and you must not add one to it. The Backup in the chart is safe. Each time Helm creates it, Velero makes a disaster recovery backup, which overwrites nothing. A Restore is different: it runs as soon as the object exists. Helm applies the chart on each helm install and each helm upgrade. A Restore in the chart would therefore restore over your live cluster as soon as a matching disaster recovery backup exists.

Apply the manifest below only during a disaster recovery restore. First, do all three checks below.

1. Choose a recent disaster recovery backup that completed and holds volume data. List the disaster recovery backups, newest last:

kubectl -n velero get backup --sort-by=.status.completionTimestamp \
  -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,COMPLETED:.status.completionTimestamp

A backup that did not finish has no completion time, so it sorts first. If no backup finished, kubectl cannot sort on the field. Remove --sort-by.

Do not choose netbox-enterprise-backup unless you want the state of the cluster at the last time Helm created it. See The chart's Backup is not a restore point. On a rebuilt cluster, the chart install in the step above created a new netbox-enterprise-backup. If the cluster uses the same bucket and prefix as before, that Backup fails with backup already exists in object storage, and the old backup of that name does not show in the list. Set BACKUP to the name that you chose, in the shell that you use for the rest of this procedure.

Completed alone is not enough. A disaster recovery backup on a hostPath storage class reports Completed and holds none of your data:

BACKUP=REPLACE-WITH-BACKUP-NAME

kubectl -n velero get backup "$BACKUP" \
  -o jsonpath='{.status.phase}{"\n"}'

kubectl -n velero get podvolumebackups \
  -l velero.io/backup-name="$BACKUP" \
  -o custom-columns='VOLUME:.spec.volume,PHASE:.status.phase,BYTES:.status.progress.bytesDone'

You must see Completed, and then one Completed row for each volume. The repository volume must be in the list with a non-zero byte count. An empty volume, such as media on a new install, shows <none> in the BYTES column, and that is correct. If the second command shows no rows, there is nothing to restore.

2. Make sure that exactly one pod-volume-restore plugin config exists. Count it by label, not by name. Velero needs exactly one match on the label pair, and a second match with any name breaks every volume restore:

kubectl get configmap -n velero \
  -l 'velero.io/plugin-config,velero.io/pod-volume-restore=RestoreItemAction'

One row is correct. Two rows is the collision. See On upgrade, remove your own copy. Zero rows means that the chart did not create its copy.

3. Make sure that the resource modifier exists. Check it by name, because the Restore uses its name:

kubectl -n velero get configmap netbox-enterprise-restore-modifiers

The chart creates both ConfigMaps when disasterRecovery.createVeleroSpecs is true. A missing ConfigMap gives a different symptom for each one:

• No plugin config, or two. Velero adds the init container and copies the security context from the first container of the restored pod. Kubelet then refuses it. The init container stays at CreateContainerConfigError, no volume restore runs, and the Restore stays InProgress with every API object restored.
• No modifier. You see no symptom. The PostgresCluster is not paused, PGO makes an empty database over the recovery, the pods become healthy, and the Restore reports Completed.

Then apply the Restore. Set NS to your release namespace. Keep BACKUP from check 1 in the same shell, because the heredoc uses both variables. The kubectl apply runs only when $BACKUP names a Completed backup, so a placeholder or a typo applies nothing. If you see a NotFound error or no output, check the name:

NS=netbox   # your release namespace
[ "$(kubectl -n velero get backup "$BACKUP" -o jsonpath='{.status.phase}')" = Completed ] && kubectl apply -f - <<EOF
apiVersion: velero.io/v1
kind: Restore
metadata:
  # DESTRUCTIVE. Apply this only during a recovery. Velero validates a Restore
  # once and never re-validates it, so a failed attempt cannot be retried under
  # the same name. Use a fresh name each time.
  name: netbox-enterprise-restore-1
  namespace: velero
spec:
  backupName: $BACKUP

  includedNamespaces:
    - $NS
    - velero

  includeClusterResources: true
  restorePVs: true

  # Velero's default. Every object that already exists is SKIPPED with a
  # warning, and the Restore still reports Completed. So on a partly-live
  # cluster, Completed means "restored what was absent", not "restored
  # everything". Read the warning count before you trust it.
  existingResourcePolicy: none

  # REQUIRED. This is what makes the restored PostgresCluster arrive paused, so
  # PGO cannot create an instance and initialize an empty data directory before
  # the pgBackRest restore happens. Without it every pod comes up healthy over
  # an empty database and this Restore still reports Completed.
  #
  # Velero resolves this reference when the Restore is created, and an
  # unresolvable one skips the modifier step with NO error. Confirm the
  # ConfigMap exists before you apply this.
  resourceModifier:
    kind: ConfigMap
    name: netbox-enterprise-restore-modifiers

  orLabelSelectors:
    - matchLabels:
        app.kubernetes.io/managed-by: netbox-operator
    - matchLabels:
        netboxlabs.com/managed-by: netbox-operator
    - matchLabels:
        app.kubernetes.io/part-of: netbox-enterprise
    - matchLabels:
        postgres-operator.crunchydata.com/control-plane: pgo
EOF

Monitor it until it completes:

kubectl -n velero get restore netbox-enterprise-restore-1 \
  -o jsonpath='{.status.phase}  {.status.warnings}  {.status.validationErrors}{"\n"}' -w

FailedValidation with BackupStorageLocation.velero.io "" not found means that Velero could not find the disaster recovery backup when it validated the object. Make sure that the Backup reports Completed. Then apply the manifest again with a new name.

A Completed restore with warnings usually means that existingResourcePolicy: none skipped objects that already existed. Read the warnings before you treat the restore as complete:

velero restore describe netbox-enterprise-restore-1 --details

Completed does not show that your data came back. Go back to Disaster recovery restore procedure. A completed Velero restore always leaves the PostgreSQL data directory empty, and a PostgreSQL restore must rebuild the database.

Create your Schedule again after the recovery. The Schedule has none of the labels that the Backup selects, so the disaster recovery backup does not hold it. A rebuilt cluster makes no more disaster recovery backups until you apply Scheduled backup again.

Restore one resource

You can also restore one named resource into a cluster that still runs. This is not a database recovery, and it is not a disaster recovery restore of the whole namespace.

1. Enable maintenance mode to stop the application:

kubectl -n "$NS" patch netboxenterprise netbox \
  --type merge \
  -p '{"spec":{"maintenanceMode":true}}'

2. Restore the object. Use your backup name and your resource:

RESTORE="netbox-restore-$(date +%Y%m%d%H%M)"

velero restore create "$RESTORE" \
  --from-backup "$BACKUP" \
  --include-namespaces "$NS" \
  --include-resources <resource> \
  --wait

3. Check the restore status, and read the warning count. A non-zero count lists the objects that Velero skipped:

velero restore describe "$RESTORE" --details

4. Disable maintenance mode to start the application again:

kubectl -n "$NS" patch netboxenterprise netbox \
  --type merge \
  -p '{"spec":{"maintenanceMode":false}}'

5. Make sure that your data is present. A healthy pod does not prove it. See Verify.

Manual backups

The manual backup and restore commands in Backup and Restore in Depth also work on Helm. Use Maintenance Mode in place of the admin console steps.

Data preservation

When you delete a NetBoxEnterprise resource, its PersistentVolumeClaims stay. This prevents an accidental loss of data, and you can create the cluster again with the existing volumes.

Next Steps

• PostgreSQL Backups and Disaster Recovery - The terms, PostgreSQL backups, and the restore procedure
• Maintenance Mode - Take NetBox offline during a restore
• Security - Protect the backup credentials
• Velero documentation - The full Velero reference