Backups
Velero backup and restore for NetBox Enterprise Helm deployments
NetBox Enterprise integrates with Velero for Kubernetes-native backup and restore. The operator labels every resource it manages, so Velero can capture the whole deployment with label selectors.
From 2.3.0 the chart renders the Velero objects for you. You no longer write them by hand.
Read this before you rely on a backup
Two prerequisites are properties of your cluster, and neither one warns you at backup time.
- Your storage class must not provision
hostPathPersistentVolumes. Velero skips them without an error, so the backup reportsCompletedwhile it holds no volume data.local-path-provisioner, the default in k3s, k3d, and kind, provisionshostPathvolumes and does not work. - Velero must be 1.13 or newer. An older Velero fails the whole backup at validation.
A third fact decides whether a restore returns your database: a completed Velero restore always leaves the PostgreSQL data volume empty. The database is rebuilt from the pgBackRest repository, and that is a separate step.
All three are covered in PostgreSQL Backups and Disaster Recovery. Read that page before you test a restore.
Embedded Cluster deployments
For Embedded Cluster deployments, Velero is pre-installed and the backup objects are managed for you. Use the Admin Console Backup tab instead of the steps below, and leave disasterRecovery.createVeleroSpecs off. A second copy of these objects in the velero namespace breaks the restore.
Prerequisites
- A running NetBox Enterprise Helm deployment
- A storage class that provisions
local,csi, or network PersistentVolumes -- see Storage requirements - Velero 1.13 or newer
- S3-compatible object storage (AWS S3, MinIO, and similar) for the backup target
- The
veleroCLI installed locally (installation guide)
Install Velero
Install Velero with the node-agent enabled, which is what performs filesystem-level volume backups. The example below uses the AWS S3 provider. See Velero supported providers for the others.
- Create a credentials file for your S3 bucket:
[default]
aws_access_key_id=<YOUR_ACCESS_KEY>
aws_secret_access_key=<YOUR_SECRET_KEY>- Install Velero into the cluster:
velero install \
--provider aws \
--plugins velero/velero-plugin-for-aws:v1.11.1 \
--bucket <BUCKET_NAME> \
--secret-file ./credentials-velero \
--backup-location-config \
region=<REGION>,s3ForcePathStyle="true",s3Url=<S3_ENDPOINT> \
--use-node-agentFor MinIO or another S3-compatible store, set s3Url to the endpoint, such as https://minio.example.com. For AWS S3, omit s3Url and s3ForcePathStyle.
If more than one cluster backs up to the same bucket, give each one its own prefix with --prefix <CLUSTER_NAME>. A shared bucket and prefix puts every cluster's backups in one pool, so velero backup get lists them all with nothing to tell them apart. This matters at restore time: a restore from another cluster's backup completes, reports success, and brings the cluster up over someone else's data.
- Verify that Velero is ready:
velero version
kubectl -n velero get podsEnable the chart's disaster-recovery objects
Set one value:
disasterRecovery:
createVeleroSpecs: trueThe chart then renders four objects into the velero namespace.
| Object | Name | Purpose |
|---|---|---|
Backup | netbox-enterprise-backup | The backup definition, with the label selectors and the volume settings. |
| ConfigMap | netbox-enterprise-backup-resource-filters | The volume policy the Backup references. It skips ephemeral volume types. |
| ConfigMap | netbox-enterprise-pod-volume-restore-config | The security context Velero gives the init container that writes your volumes back. Without it, no volume restore works. |
| ConfigMap | netbox-enterprise-restore-modifiers | Makes a restored PostgresCluster arrive paused, so PostgreSQL cannot initialise an empty database over the recovery. |
If you applied the plugin config by hand, remove your copy
Do this on any upgrade to 2.3.0 or later, whether or not you change your values. An earlier revision of this guide told you to apply netbox-enterprise-pod-volume-restore-config yourself. If you followed it, helm upgrade alone lands a second claimant, Velero's lookup then fails, and every volume restore aborts.
Nothing warns you. Backups keep succeeding, because the plugin config is read only on restore. See On upgrade, remove your own copy.
The chart deliberately ships no Restore. A Backup is a declaration that Velero acts on when the schedule says so. A Restore is an action that runs the moment the object exists, and Helm applies every chart manifest on every install and upgrade. A Restore in the chart would therefore restore over your live cluster as soon as a matching backup existed. See Restoring a pure-Helm install for the manifest to apply by hand at recovery time.
How label selection works
The operator labels every Kubernetes resource it manages. Velero uses these labels with OR semantics to select what to back up:
| Label | What it covers |
|---|---|
app.kubernetes.io/managed-by: netbox-operator | Operator-managed resources (deployments, services, PVCs, jobs) |
netboxlabs.com/managed-by: netbox-operator | Sub-operator resources and chart template resources (Redis, PGO, gateway and ingress) |
app.kubernetes.io/part-of: netbox-enterprise | Helm subchart resources whose charts hardcode managed-by: Helm |
postgres-operator.crunchydata.com/control-plane: pgo | The CrunchyData PGO control plane |
Do not widen these selectors, and do not replace them with a namespace-wide backup. See Do not widen the backup selectors for what breaks.
The PostgreSQL data volume is excluded on purpose. The pgBackRest repository volume carries the label instead, and that repository is the restorable copy of your database. See Why the data volume is not backed up.
Create a backup
Every command below writes netbox as the release namespace. Set NS to yours in the same shell, and the commands then follow it:
NS=netboxA backup whose --include-namespaces matches nothing is not an error to Velero. It reports Completed with zero items.
On-demand backup
BACKUP="netbox-backup-$(date +%Y%m%d%H%M)"
velero backup create "$BACKUP" \
--or-selector "app.kubernetes.io/managed-by=netbox-operator or netboxlabs.com/managed-by=netbox-operator or app.kubernetes.io/part-of=netbox-enterprise or postgres-operator.crunchydata.com/control-plane=pgo" \
--include-namespaces "$NS",velero \
--include-cluster-resources=true \
--default-volumes-to-fs-backup \
--waitIf you set up a schedule, you can also trigger an immediate backup from it:
velero backup create --from-schedule netbox-enterprise-daily --waitScheduled backup
The chart renders a one-shot Backup, not a Schedule. Create a Schedule for recurring backups. This example runs daily at 02:00 UTC with 30-day retention, and it mirrors the chart's own selectors and volume policy:
apiVersion: velero.io/v1
kind: Schedule
metadata:
name: netbox-enterprise-daily
namespace: velero
spec:
schedule: "0 2 * * *"
template:
includeClusterResources: true
includedNamespaces:
- netbox # your release namespace
- velero
defaultVolumesToFsBackup: true
ttl: 720h
resourcePolicy:
kind: ConfigMap
name: netbox-enterprise-backup-resource-filters
orLabelSelectors:
- matchLabels:
app.kubernetes.io/managed-by: netbox-operator
- matchLabels:
netboxlabs.com/managed-by: netbox-operator
- matchLabels:
app.kubernetes.io/part-of: netbox-enterprise
- matchLabels:
postgres-operator.crunchydata.com/control-plane: pgoCheck backup status
Name the backup you just took. $BACKUP from the create step above holds it. A cluster that followed the pre-2.3.0 guidance may still hold an old backup literally named netbox-backup, and velero backup describe on that reports Completed for a backup months old.
velero backup get
velero backup describe "$BACKUP" --details
velero backup logs "$BACKUP"Do not trust Completed on its own. Confirm that volume data was captured:
kubectl -n velero get podvolumebackups \
-l velero.io/backup-name="$BACKUP" \
-o custom-columns='VOLUME:.spec.volume,PHASE:.status.phase,BYTES:.status.progress.bytesDone'Expect one Completed row per persistent volume, each with a non-zero byte count. The pgBackRest repository volume must be among them: it is the only copy of your database in the archive.
An empty result has two causes, and they need different responses. Check the cheap one first: confirm the name against velero backup get, because a name that matches nothing returns exactly the same empty list. Once the name is known good, an empty result means no volume data was backed up at all, and the likely cause is a hostPath storage class.
Confirm separately that pgBackRest holds a backup set. See Verifying backups are working.
Restore from backup
A Velero restore alone does not return your database
The backup excludes the PostgreSQL data volume on purpose, so a completed restore always leaves it empty. The database is rebuilt from the pgBackRest repository, and the operator does that on the restore target.
The full procedure -- the install order on a rebuilt cluster, the Restore manifest to apply, and how to verify the data came back -- is in Restoring a pure-Helm install. Follow it rather than the short form below.
Running 2.2.x with external PostgreSQL?
On the 2.2.x line, deployments using customer-managed (external) PostgreSQL do not detect that a restore has wiped the nbe_ready migration sentinel, so NetBox and the worker can hang on startup. See Known Issues for 2.2.1 for the symptom and recovery steps. Upgrade to 2.3.0 before restoring. Deployments on 2.3.0 or later, and deployments using the built-in PostgreSQL, are not affected.
The short form below recovers a named resource into a cluster that is still running. It is not a database recovery, and it is not a whole-namespace recovery.
Scope the restore, and delete the target first
Velero defaults existingResourcePolicy to none, which skips every object that already exists and still reports Completed. An unscoped restore into a live cluster therefore restores nothing and says it worked. Velero also writes a volume's data only when it creates the pod that mounts it, so a skipped Deployment means its volumes are skipped too.
So name what you are recovering with --include-resources or --selector, and delete that object first so Velero has something to create.
- Enable maintenance mode to stop the application:
kubectl -n "$NS" patch netboxenterprise netbox \
--type merge \
-p '{"spec":{"maintenanceMode":true}}'- Restore the object you are recovering. Substitute the backup name and the resource:
RESTORE="netbox-restore-$(date +%Y%m%d%H%M)"
velero restore create "$RESTORE" \
--from-backup "$BACKUP" \
--include-namespaces "$NS" \
--include-resources <resource> \
--wait- Check restore status, and read the warning count. A non-zero count is Velero listing what it skipped:
velero restore describe "$RESTORE" --details- Disable maintenance mode to resume:
kubectl -n "$NS" patch netboxenterprise netbox \
--type merge \
-p '{"spec":{"maintenanceMode":false}}'- Confirm your data is present. A healthy pod is not evidence. See Verify.
Data preservation
When a NetBoxEnterprise resource is deleted, its PersistentVolumeClaims are preserved rather than deleted. This prevents accidental data loss, and it lets you re-create the cluster and reattach the existing volumes.
Next Steps
- PostgreSQL Backups and Disaster Recovery - pgBackRest, the repository volume, and the restore procedure
- Maintenance Mode - Take NetBox offline during a restore
- Security - Protect backup credentials
- Velero documentation - Full Velero reference