Backup and Restore in Depth
Embedded Cluster only
This page has the full disaster recovery procedures for Embedded Cluster installs, with every check and every edge case. For the short version, see Backup Essentials. For Helm installs, see Helm Backups. The manual backup section applies to both install paths.
This page uses three kinds of backup:
| Term | Meaning | Schedule |
|---|---|---|
| Disaster recovery backup | A Velero backup of the whole install, which you make from the admin console. | None by default. See Schedule disaster recovery backups. |
| PostgreSQL backup | A pgBackRest backup of the built-in database, in a repository volume in the cluster. | On by default. See PostgreSQL backup schedule. |
| Manual backup | A copy of your data that you make with kubectl, such as a database dump. | None. See Manual backup and restore. |
A disaster recovery backup includes the repository volume, so it also holds the PostgreSQL backups. See PostgreSQL Backups and Disaster Recovery. The repository and the other volumes share the disk of the node, and their sizes are not limits. See Volume sizes and disk space.
Like NetBox, NetBox Enterprise uses two main datastores: PostgreSQL and Redis.
PostgreSQL holds the primary NetBox model data, such as sites, facilities, and racks.
Redis holds the internal cache and the task queue.
For each datastore, you can use the built-in deployment, or an external resource that already exists in your environment.
External databases
If you supply your own databases for NetBox Enterprise, you must also supply your own procedures for high availability, backup, and restore. Include the data of the bundled Keycloak and the Platform Authorization Server: see What the dump holds.
Do you run 2.2.x with external PostgreSQL?
On the 2.2.x line, an install with external PostgreSQL does not detect that a restore removed the nbe_ready migration sentinel.
NetBox and the worker can then stop at startup.
See Known Issues for 2.2.1 for the symptom and the recovery steps.
Upgrade to 2.3.0 before you restore.
Installs on 2.3.0 or later, and installs with the built-in PostgreSQL, do not have this problem.
Disaster recovery backups
A disaster recovery backup keeps the state of NetBox Enterprise inside the cluster: its volumes, the in-cluster databases, and the configuration. It does not include media in an external S3 bucket or an external database; back those up with their own provider's tools.
NetBox Enterprise includes this feature. For a KOTS install, you must install some more dependencies in your cluster to enable it.
The database comes back from pgBackRest, not from the volume copy
From 2.3.0, a disaster recovery backup excludes the PostgreSQL data volume on purpose. A completed disaster recovery restore therefore always leaves that volume empty. The operator then rebuilds the database from the PostgreSQL backups in the repository.
Before you use the steps below, know these three facts:
- A disaster recovery backup from before 2.3.0 does not hold the database. After you upgrade, take a full PostgreSQL backup immediately, and then a disaster recovery backup.
- A healthy cluster after a disaster recovery restore does not prove that your data came back. Confirm the row counts.
- If you turn off PostgreSQL backups, disaster recovery backups do not hold the database.
Read PostgreSQL Backups and Disaster Recovery before you test a disaster recovery restore.
Back up the NetBox Enterprise installation
Give each cluster its own backup location
If you have more than one NetBox Enterprise cluster, such as production and staging, give each one a unique Prefix in the backup settings below, or a separate bucket. Clusters that share a bucket and a prefix write their disaster recovery backups into one location. The restore prompt cannot tell these disaster recovery backups apart. See Restore the NetBox Enterprise installation.
- In the Disaster Recovery section of the admin console, click the ^^Backup settings^^ link.

- Enter your S3 bucket, credentials, endpoint, and region.

- Click Update storage settings. The admin console takes a few moments to validate your settings.

- To make a disaster recovery backup now, click the Backups tab, and then click Start a backup.

- Set a schedule for disaster recovery backups. See Schedule disaster recovery backups.
Schedule disaster recovery backups
Disaster recovery backups have no schedule by default. Without a schedule, you have only the disaster recovery backups that you start manually. A disaster recovery restore returns the install at about the time of the disaster recovery backup, so this schedule sets how much data a disaster recovery restore can lose.
PostgreSQL backups are different. They run on a schedule by default, but they stay in the cluster, so they do not protect against the loss of the cluster. See PostgreSQL backup schedule.
- In the admin console, click Disaster Recovery, and then click the Settings & Schedule tab.
- In the Scheduled backups card, select Enable scheduled backups.
- In Schedule, select Hourly, Daily, or Weekly. For a different interval, type a standard five-field cron expression in Cron expression.
- Click Update schedule. The page shows Schedule updated.
The presets use these cron expressions:
| Schedule | Cron expression |
|---|---|
| Hourly | 0 * * * * |
| Daily | 0 0 * * * |
| Weekly | 0 0 * * MON |
The Retention policy card on the same tab sets how long Velero keeps each disaster recovery backup. Enter a number and a unit, and then click Update retention policy. The default is 1 month. The retention applies to disaster recovery backups that you start manually and to scheduled ones. A change applies to the disaster recovery backups that start after the change.
If Velero does not run, or no storage destination is set, the tab shows Scheduling not active.
The admin console runs the schedule itself.
It does not create a Velero Schedule resource, so velero schedule get shows no schedule.
Each scheduled run creates Velero Backup resources in the velero namespace.
S3-compatible stores that reject trailing checksums
Some S3-compatible stores reject the trailing checksum that the Velero AWS plugin adds to each upload.
The plugin sends it as x-amz-content-sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER.
With such a store, every disaster recovery backup fails with this error:
api error InvalidArgument: x-amz-content-sha256 must be UNSIGNED-PAYLOAD, STREAMING-AWS4-HMAC-SHA256-PAYLOAD or a valid sha256 value.Known issue: the workaround does not work on 2.4.0 and later
2.4.0 and 2.5.0 ship Velero AWS plugin v1.14.2, which adds the checksum whatever the value of checksumAlgorithm.
If your store rejects trailing checksums, do not upgrade past 2.3.x until a release with the fix is available.
See Known Issues for 2.5.0.
On 2.3.x and earlier, set checksumAlgorithm: "" in the Velero BackupStorageLocation config to turn off the checksum.
The Backup settings screen has no field for it.
After you save your backup settings, set it from the NetBox Enterprise shell:
/var/lib/embedded-cluster/bin/netbox-enterprise shellkubectl -n velero patch backupstoragelocation default --type merge \
-p '{"spec":{"config":{"checksumAlgorithm":""}}}'Two actions remove the key.
After either one, disaster recovery backups fail until you run the kubectl patch command again:
- You save the Backup settings in the admin console (Update storage settings). The admin console writes the BackupStorageLocation config again with only the fields on that screen.
- You run
netbox-enterprise restore. The restored cluster gets a new BackupStorageLocation without the key.
Restore the NetBox Enterprise installation
This procedure is a disaster recovery restore. It returns the whole install from a disaster recovery backup, in two parts. First, the installer restores the disaster recovery backup. Then the operator does a PostgreSQL restore from the repository. The second part is the same on both install paths. See Disaster recovery restore procedure.
Confirm the installer matches your newest disaster recovery backup
sudo ./netbox-enterprise restore does not offer every disaster recovery backup in the store.
It compares each one with the application version and the Embedded Cluster version in the installer binary on disk.
It removes each one that does not match from the list.
An upgrade through the Admin Console does not replace the installer binary or those versions. On an upgraded cluster, the restore therefore hides every disaster recovery backup taken after the original install.
Embedded Cluster shows its reasons only when no disaster recovery backup passes the filter.
When one old backup passes, the newer ones disappear with no message.
The prompt then shows Found 1 restorable backup!.
If you accept it, the cluster returns to its original-install state, and every later change is lost.
The filter does not limit the list to this cluster. A backup store holds the disaster recovery backups of every cluster that writes to it. The restore lists all of them, offers the newest, and shows a name and a timestamp:
○ Found 2 restorable backups!
Restore from backup "netbox-enterprise-019814bb" (2026-09-01 13:55:40 UTC)? [Y/n]:Neither field identifies the cluster that made the backup. If you accept the disaster recovery backup of another cluster, the restore completes, reports success, and brings the cluster up with the data of the other cluster. No error occurs.
Give each cluster its own bucket, or its own prefix in one bucket. Do this when you configure the backup storage location. At restore time, the store already holds disaster recovery backups from more than one cluster.
If you use a shared store, read the cluster ID and the versions from the disaster recovery backup before you accept it.
One disaster recovery backup is two Velero Backups, instance-<id> and application-<id>, and the store has a folder for each.
Replace <name> with one of these folder names, not with the name in the restore prompt.
The first line of the output is the name that the restore prompt shows, so you can match the two:
set -o pipefail # without this the pipeline reports jq's status, not the copy's
# For an S3-compatible store, also give your endpoint: --endpoint-url <url>
aws s3 ls s3://"$BUCKET"/"$PREFIX"/backups/
aws s3 cp s3://"$BUCKET"/"$PREFIX"/backups/<name>/velero-backup.json - \
| jq -er '.metadata.labels["replicated.com/backup-name"],
.metadata.annotations["kots.io/embedded-cluster-id"],
.metadata.annotations["kots.io/apps-versions"],
.metadata.annotations["kots.io/embedded-cluster-version"]'You must get four lines that are not empty. If you do not, stop.
This command is your only check against a restore of the data of another cluster.
Without the two flags above, the command cannot fail visibly.
A failed aws s3 cp writes to stderr, and the pipeline still exits 0.
A missing annotation makes plain jq -r print null and exit 0.
set -o pipefail shows the copy failure, and jq -e exits with a non-zero status on null.
Blank lines, null, or a non-zero exit status mean that the lookup failed.
They do not mean that the backup has no labels.
Correct the credentials, the bucket, or the object name, and run the command again.
When you have four values, compare the cluster ID with the cluster that you restore. Compare the two versions with your installer binary. If they are different, download the installer that matches the newest disaster recovery backup before you start.
This object path is the internal layout of Velero. NetBox Enterprise does not document it or depend on it. The lookup works, but it is not supported.
Prepare the host
Run the disaster recovery restore on a host that has no cluster.
Make sure that the disaster recovery backup holds data before you reset the host
On a single-node Embedded Cluster, reset deletes the PostgreSQL data volume and the repository.
After the reset, the disaster recovery backup in your object store is the only copy of your database.
A disaster recovery backup can report Completed and hold no volume data.
A hostPath storage class causes this, and it shows no warning.
Confirm both of these items before you reset. Use the name of the disaster recovery backup that the admin console shows:
# The name that the admin console and the restore prompt show, such as
# netbox-enterprise-007caca9. It labels two Velero Backups.
NAME=<name>
# 1. Both Velero Backups completed: instance-<id> and application-<id>.
kubectl -n velero get backup -l replicated.com/backup-name="$NAME" \
-o custom-columns='NAME:.metadata.name,PHASE:.status.phase'
# 2. The application Backup holds volume data: one Completed row per volume,
# and the pgBackRest repository volume (repo1) with a non-zero byte count.
APP_BACKUP="$(kubectl -n velero get backup -l replicated.com/backup-name="$NAME" \
-o name | grep '/application-' | cut -d/ -f2)"
kubectl -n velero get podvolumebackups -l velero.io/backup-name="$APP_BACKUP" \
-o custom-columns='VOLUME:.spec.volume,PHASE:.status.phase,BYTES:.status.progress.bytesDone'One disaster recovery backup is two Velero Backups.
instance-<id> holds the admin console, and application-<id> holds the NetBox Enterprise volumes.
If the first command shows no rows, the name is wrong.
If the second command shows no rows, or no repo1 row, the disaster recovery backup does not hold your database.
Do not reset.
See Verifying backups are working.
To restore onto the same machine, run sudo ./netbox-enterprise reset --yes first.
The reset reboots the host, so your SSH session disconnects, and the command reports a non-zero status.
This status is not a failure.
When the host is up again, connect again, and confirm that the host is empty.
Check the files and services that the installer leaves, not only the binary.
A $PATH check passes after a reset that removed the binary but left the data directories:
for p in /var/lib/embedded-cluster /var/lib/k0s /etc/k0s; do
[ -n "$(sudo ls -A "$p" 2>/dev/null)" ] && echo "not empty: $p"
done
systemctl list-units --all 'k0s*' --no-legend | grep . && echo "k0s units remain"
command -v k0s >/dev/null && echo "k0s binary still on PATH"
echo "check complete"The reset can leave /var/lib/embedded-cluster and /etc/k0s as empty directories.
Empty directories do not stop the restore, so the check reports only a directory that has content.
A line before check complete means that the host is not empty.
Run the reset again, or remove the named path manually.
Run the restore
- If you do not have it yet, download the installer that matches your disaster recovery backup. Use the same instructions as for a new install.
- Run:
sudo ./netbox-enterprise restore - Enter the S3 credentials that you use for disaster recovery backups.

- The installer takes a few minutes to start the node.
When it is ready, it asks you to continue the restore:

- Make sure that the name matches the disaster recovery backup that you recorded. Then enter
Yto continue. The restore starts the cluster. - (Optional) If you want more than one node in the new cluster, add the nodes in the admin console when it asks you. See High-availability restore.
- Enter
continueto start the NetBox Enterprise application.

- After a few more minutes, the installer shows "Application restored!"

NetBox Enterprise and the Admin Console are now restored and available. - Confirm that your data came back. "Application restored!" reports on the Velero restore only. Do the checks in Verify, and compare the values with the baseline that you recorded before the loss.
If the cluster comes up healthy but empty, the disaster recovery restore failed. See Disaster recovery restore procedure for the next steps.
High-availability restore
An HA disaster recovery restore stops and waits for you to add the other nodes. After the Admin Console starts, node 1 shows its URL and then waits:
Type 'continue' when you are done adding nodes: You are restoring a
high-availability cluster, which requires at least 3 controller nodes. You
currently have 1. Please add more controller nodes.The prompt repeats until you answer, so you have time. Add the other nodes while it waits. Get the join command on node 1, then run it on each other node:
# On node 1:
cd && sudo ./netbox-enterprise join print-command
# On each remaining node, using the command it printed:
cd && sudo ./netbox-enterprise join <node-1-ip>:30000 <token> --yesUse --yes.
Each join runs host preflight checks, and without --yes a warning stops at a prompt.
--yes also confirms the HA prompt on the node that completes the quorum.
--yes also accepts every preflight warning and does not show it.
Read the preflight output of each node before you answer continue on node 1.
A node that joins with a warning about disk space, a kernel setting, or storage can fail later with no message.
When three controller nodes are present, answer continue.
The disaster recovery restore then continues to the application.
Manual backup and restore
Also keep manual backups of your data. Use them to examine your data, to restore part of it, or to move it to another system.
These commands also work on a Helm install. On Helm, use Maintenance Mode in place of the admin console steps. Admin Console Configuration applies to Embedded Cluster only.
NetBox Enterprise Namespace
On Embedded Cluster, NetBox Enterprise runs in the kotsadm namespace.
On Helm, it runs in the namespace that you installed it in.
If you set the NETBOX_NAMESPACE variable, the commands below use that namespace.
If it is not set, each command block looks for the namespace that runs NetBox Enterprise.
If it finds no such namespace, or more than one, it lists the namespaces that it found and stops before it runs anything in a pod.
Then set the variable, and run the commands again.
Replace <namespace> with the namespace of your install:
NETBOX_NAMESPACE=<namespace>The variable keeps its value in your shell until you change it.
Before you work on another install, set it again, or run unset NETBOX_NAMESPACE to let the commands look for the namespace again.
Back up your data
You can back up the NetBox Enterprise data manually with kubectl.
Before you start: get access to your cluster
You must have command-line access to the cluster before you can back it up.
See the advanced tools documentation to connect to your NetBox Enterprise cluster.
Media, Scripts, and Reports
The runtime files are on a volume that the NetBox containers can read. To back them up, run this command:
NAMESPACE="${NETBOX_NAMESPACE:-$(kubectl get deployments \
-A -l 'app.kubernetes.io/component=netbox' \
-ojsonpath='{range .items[*]}{.metadata.namespace}{"\n"}{end}' \
| sort -u)}" && \
{ [ "$(printf '%s\n' "${NAMESPACE}" | grep -c .)" -eq 1 ] \
|| { printf '%s\n' 'Set NETBOX_NAMESPACE to the namespace of your install. Namespaces found:' \
"${NAMESPACE:-(none)}" >&2; false; }; } && \
NETBOX_MAIN_POD="$(kubectl get pod \
-o name \
-n "${NAMESPACE}" \
-l 'app.kubernetes.io/component=netbox' \
| head -n 1 \
)" && \
kubectl exec "${NETBOX_MAIN_POD}" \
-n "${NAMESPACE}" \
-c netbox \
-- /bin/sh -c ' \
cd /opt/netbox/netbox && \
find media scripts reports static -type f -not -path "*/.velero/*" > /tmp/files.txt && \
tar -czf - \
--owner=0 \
--group=0 \
-T /tmp/files.txt' > netbox-data.tar.gzBuilt-In PostgreSQL
The CrunchyData Postgres Operator deploys the built-in PostgreSQL.
A dump is for portability, not for disaster recovery
From 2.3.0, a disaster recovery restore rebuilds the built-in database from PostgreSQL backups. PostgreSQL backups run on a schedule, and pgBackRest archives the write-ahead log all the time. A dump is a manual backup: a copy at one point in time, in a portable format. Use a dump to move data to another system, or to examine it. To recover the cluster, use a disaster recovery backup. See PostgreSQL Backups and Disaster Recovery.
The PostgreSQL command-line tools are already in the cluster.
To dump the database, run pg_dump in the correct container.
To dump the database, run these commands:
NAMESPACE="${NETBOX_NAMESPACE:-$(kubectl get deployments \
-A -l 'app.kubernetes.io/component=netbox' \
-ojsonpath='{range .items[*]}{.metadata.namespace}{"\n"}{end}' \
| sort -u)}" && \
{ [ "$(printf '%s\n' "${NAMESPACE}" | grep -c .)" -eq 1 ] \
|| { printf '%s\n' 'Set NETBOX_NAMESPACE to the namespace of your install. Namespaces found:' \
"${NAMESPACE:-(none)}" >&2; false; }; } && \
NETBOX_DATABASE_FILE="netbox-enterprise.pgsql" && \
POSTGRESQL_MAIN_POD="$(kubectl get pod \
-o name \
-n "${NAMESPACE}" \
-l 'postgres-operator.crunchydata.com/role=master' \
| head -n 1 \
)" && \
EXCLUDE_DATABASES="$(kubectl exec "${POSTGRESQL_MAIN_POD}" \
-n "${NAMESPACE}" \
-c database \
-- \
psql -t -c "SELECT CONCAT('--exclude-database=', datname) \
FROM pg_database \
WHERE datname <> ALL ('{netbox,diode,hydra,copilot,platform_auth,platform_auth_fga}')" \
)" && \
kubectl exec "${POSTGRESQL_MAIN_POD}" \
-n "${NAMESPACE}" \
-c database \
-- \
pg_dumpall \
$(printf '%s\n' "${EXCLUDE_DATABASES}") \
> "${NETBOX_DATABASE_FILE}"The commands create a netbox-enterprise.pgsql file in your local directory.
Keep it in a safe location for future restores.
Make sure that the dump is complete. Its last lines must include -- PostgreSQL database cluster dump complete:
tail -n 3 netbox-enterprise.pgsqlA dump without that line was cut short. Make it again. The restore commands refuse such a dump before they drop anything.
What the dump holds
The dump holds each of these databases that your install has:
| Database | Holds |
|---|---|
netbox | NetBox data, and the data of the bundled Keycloak identity provider |
diode | Diode data |
hydra | The OAuth2 clients of Diode |
copilot | Copilot |
platform_auth, platform_auth_fga | The Platform Authorization Server: platform users, connections to NetBox accounts, and platform roles. They exist once the authorization server has been turned on |
The bundled Keycloak has no database of its own: its realm and users are in a schema named keycloak inside the netbox database.
For more information about NetBox database backups, see the official NetBox documentation.
Cluster Secrets
A manual database restore works only if the rest of the cluster can decrypt the restored data.
The commands below back up the secrets that the restored databases depend on.
The names are for a NetBoxEnterprise resource named netbox, as on Embedded Cluster.
Elsewhere they begin with the name of your resource.
netbox-netbox-config: the NetBoxSECRET_KEYand the API token peppernetbox-netbox-superuser: the generated superuser credentials. It exists only when NetBox Enterprise generates the superuser, so Embedded Cluster does not have it.netbox-diode-hydra-secret: the Hydra system secret, which encrypts the Hydra signing keys in thehydradatabasenetbox-diode-auth-oauth2-secret: the Diode OAuth2 client secrets, which match the clients in thehydradatabasenetbox-keycloak-admin: the break-glass Keycloak administrator, whose account is in the restoredmasterrealmnetbox-keycloak-operatorandnetbox-keycloak-pas: the client secrets of the operator and authorization server service accounts, which match the clients in the restored realmnetbox-pas-secrets: the authorization server's identity-mapping pepper and signing keys. The pepper keys the rows in theplatform_auth_fgadatabase.
NetBox Enterprise creates the Keycloak and authorization server secrets when you turn those features on.
The commands do not back up the other secrets in the namespace, and you do not need them.
NetBox Enterprise, the PostgreSQL operator, and your license create them again on the target install.
The restore does not change the PostgreSQL user passwords, so the target install keeps its own pguser secrets, the Keycloak database role password (netbox-keycloak-db), and the Keycloak serving certificate and its CA key.
They also leave out the realm import secret (netbox-keycloak-netbox-realm on Embedded Cluster), which the Keycloak operator creates from the realm import that NetBox Enterprise renders again on every reconcile.
On Embedded Cluster, the secrets of a federated source (keycloak-oidc-federation-client, keycloak-saml-federation-certs, keycloak-ldap-federation-bind) come from the Admin Console configuration, so restore that configuration instead.
On Helm, you create these secrets yourself and name them in spec.idp.oidcFederation.clientSecret, spec.idp.samlFederation[0].signingCertificates, or spec.idp.ldapFederation[0].bindCredential, so the commands below do not back them up.
A Velero backup holds them when they have the label app.kubernetes.io/part-of=netbox-enterprise.
Otherwise, back them up yourself with their labels: NetBox Enterprise does not read an OIDC or LDAP secret without netboxlabs.com/federation-credential=true.
The commands write only the name, labels, type, and data of each secret. They leave out the owner references, which identify the NetBoxEnterprise resource of this install and do not apply to another install.
Run these commands:
NAMESPACE="${NETBOX_NAMESPACE:-$(kubectl get deployments \
-A -l 'app.kubernetes.io/component=netbox' \
-ojsonpath='{range .items[*]}{.metadata.namespace}{"\n"}{end}' \
| sort -u)}" && \
{ [ "$(printf '%s\n' "${NAMESPACE}" | grep -c .)" -eq 1 ] \
|| { printf '%s\n' 'Set NETBOX_NAMESPACE to the namespace of your install. Namespaces found:' \
"${NAMESPACE:-(none)}" >&2; false; }; } && \
NETBOX_CLUSTER="$(kubectl get netboxenterprise \
-n "${NAMESPACE}" \
-ojsonpath='{.items[0].metadata.name}')" && \
SECRET_TEMPLATE='---
apiVersion: v1
kind: Secret
metadata:
name: {{ .metadata.name }}
labels:
{{- range $key, $value := .metadata.labels }}
{{ $key }}: "{{ $value }}"
{{- end }}
type: {{ .type }}
data:
{{- range $key, $value := .data }}
{{ $key }}: "{{ $value }}"
{{- end }}
' && \
: > netbox-enterprise-secrets.yaml && \
chmod 600 netbox-enterprise-secrets.yaml && \
for SECRET in netbox-config netbox-superuser diode-hydra-secret diode-auth-oauth2-secret \
keycloak-admin keycloak-operator keycloak-pas pas-secrets; do \
kubectl get secret "${NETBOX_CLUSTER}-${SECRET}" \
--namespace "${NAMESPACE}" \
--ignore-not-found \
-o go-template="${SECRET_TEMPLATE}"; \
done >> netbox-enterprise-secrets.yamlIf the commands print an error, the file can be incomplete: fix the cause and run them again.
If the error is array index out of bounds, they found no NetBoxEnterprise resource in the namespace, and they did not change an earlier file.
Before you store the file, list the secrets in it:
grep '^ name: ' netbox-enterprise-secrets.yamlThe list must include netbox-netbox-config.
If Diode is turned on, it must also include netbox-diode-hydra-secret and netbox-diode-auth-oauth2-secret.
If the bundled Keycloak is turned on, it must also include netbox-keycloak-admin and netbox-keycloak-operator.
If the authorization server is turned on, it must also include netbox-keycloak-pas and netbox-pas-secrets.
The file holds the secret values, and only your user can read it.
Keep it with your netbox-enterprise.pgsql file for future restores, and protect both as you protect passwords.
Admin Console Configuration
You can export the current Admin Console configuration as a ConfigValues file.
The file holds all your Admin Console settings in portable YAML, such as database connections, TLS configuration, and feature toggles.
Use it when you move to a new host or install again.
From the cluster shell, run:
kubectl exec deploy/kotsadm -n kotsadm -- \
/kots get config --current --decrypt --namespace kotsadm \
> netbox-enterprise-config.yamlThe --decrypt flag writes password fields, such as database credentials, in plain text, not encrypted.
Keep this file in a secure location with your other backups.
To apply a saved configuration to a new install, run:
kubectl exec -i deploy/kotsadm -n kotsadm -- \
/kots set config netbox-enterprise \
--config-file /dev/stdin --deploy --namespace kotsadm \
< netbox-enterprise-config.yamlRestore your backups
Put NetBox Enterprise into maintenance mode before you restore.
Enable and disable maintenance mode
- Go to the Config tab, and select Enable Maintenance Mode.

- Click the "Save config" button at the bottom of the form. When the admin console asks you, click "go to updated version".

- Make sure that the New version available message at the top shows a config change. If it does, click the "Deploy" button.

This stops NetBox but keeps the other NetBox Enterprise infrastructure running, so you can restore safely.
When the restore is complete, do the same steps again: clear Enable Maintenance Mode, and deploy the updated configuration.
Media, Scripts, and Reports
To restore media, scripts, and reports, unpack them into the correct directory in a NetBox container.
The backup instructions above put the media/, scripts/, and reports/ subdirectories into one file.
A backup from a different NetBox instance can have a different tarball name, or need a different path after -C at the end of this command.
In that case, change them so that the backup unpacks into the correct location.
NAMESPACE="${NETBOX_NAMESPACE:-$(kubectl get deployments \
-A -l 'app.kubernetes.io/component=netbox' \
-ojsonpath='{range .items[*]}{.metadata.namespace}{"\n"}{end}' \
| sort -u)}" && \
{ [ "$(printf '%s\n' "${NAMESPACE}" | grep -c .)" -eq 1 ] \
|| { printf '%s\n' 'Set NETBOX_NAMESPACE to the namespace of your install. Namespaces found:' \
"${NAMESPACE:-(none)}" >&2; false; }; } && \
NETBOX_MAINTENANCE_POD="$(kubectl get pod \
-o name \
-n "${NAMESPACE}" \
-l 'maintenance-mode=true' \
| head -n 1 \
)" && \
cat netbox-data.tar.gz | kubectl exec ${NETBOX_MAINTENANCE_POD} \
-n "${NAMESPACE}" \
-i \
-- tar -xvzf - \
--exclude=.velero \
--exclude=reports/__init__.py \
--no-same-owner \
--no-same-permission \
-C /opt/netbox/netboxCluster Secrets
Restore the secrets before the database dump. NetBox needs them to check sessions and API tokens, and Hydra needs them to serve Diode OAuth flows against the restored data.
To restore into a new install, first turn on the same features as the install that you backed up, such as Diode, the bundled Keycloak, and the authorization server, and wait until NetBox Enterprise is ready. NetBox Enterprise then creates each secret, and the restore replaces only its values. If Diode is off on the target, NetBox Enterprise deletes the restored Diode secrets on its next reconcile. When you turn Diode on later, NetBox Enterprise creates new ones, which do not match the restored data. If the bundled Keycloak or the authorization server is off, the database restore fails, because their database roles do not exist (see Built-In PostgreSQL).
The commands apply only the secrets listed in the backup, and only their name, labels, type, and data. They also restore a file made with an earlier version of these instructions, which held more secrets and their owner references.
NAMESPACE="${NETBOX_NAMESPACE:-$(kubectl get deployments \
-A -l 'app.kubernetes.io/component=netbox' \
-ojsonpath='{range .items[*]}{.metadata.namespace}{"\n"}{end}' \
| sort -u)}" && \
{ [ "$(printf '%s\n' "${NAMESPACE}" | grep -c .)" -eq 1 ] \
|| { printf '%s\n' 'Set NETBOX_NAMESPACE to the namespace of your install. Namespaces found:' \
"${NAMESPACE:-(none)}" >&2; false; }; } && \
NETBOX_CLUSTER="$(kubectl get netboxenterprise \
-n "${NAMESPACE}" \
-ojsonpath='{.items[0].metadata.name}')" && \
SECRET_TEMPLATE='---
apiVersion: v1
kind: Secret
metadata:
name: {{ .metadata.name }}
labels:
{{- range $key, $value := .metadata.labels }}
{{ $key }}: "{{ $value }}"
{{- end }}
type: {{ .type }}
data:
{{- range $key, $value := .data }}
{{ $key }}: "{{ $value }}"
{{- end }}
' && \
for SECRET in netbox-config netbox-superuser diode-hydra-secret diode-auth-oauth2-secret \
keycloak-admin keycloak-operator keycloak-pas pas-secrets; do \
kubectl create \
--dry-run=client \
--filename netbox-enterprise-secrets.yaml \
-o go-template="{{ if eq .metadata.name \"${NETBOX_CLUSTER}-${SECRET}\" }}${SECRET_TEMPLATE}{{ end }}"; \
done \
| kubectl apply \
--server-side \
--force-conflicts \
--namespace "${NAMESPACE}" \
--filename -The command prints one serverside-applied line for each listed secret in the file.
If it prints fewer lines than that, an error, or no objects passed to apply, stop, and do not restore the database.
A database restored without its secrets cannot be read with the keys of the target install.
no objects passed to apply means that the file holds no secret for the NetBoxEnterprise resource in this namespace.
The NetBoxEnterprise resource on the target must have the same name as the one that you backed up.
Then restart the pods that use the old secret values, so that they get the restored credentials:
NAMESPACE="${NETBOX_NAMESPACE:-$(kubectl get deployments \
-A -l 'app.kubernetes.io/component=netbox' \
-ojsonpath='{range .items[*]}{.metadata.namespace}{"\n"}{end}' \
| sort -u)}" && \
{ [ "$(printf '%s\n' "${NAMESPACE}" | grep -c .)" -eq 1 ] \
|| { printf '%s\n' 'Set NETBOX_NAMESPACE to the namespace of your install. Namespaces found:' \
"${NAMESPACE:-(none)}" >&2; false; }; } && \
kubectl rollout restart deployment \
--namespace "${NAMESPACE}" \
-l 'app.kubernetes.io/managed-by=netbox-operator'Built-In PostgreSQL
The restore replaces the data of the bundled Keycloak
The restore drops the netbox database, and with it the keycloak schema that holds the realm and users of the bundled Keycloak.
Keycloak gets back only what the dump holds.
That includes access you removed after the dump: a deleted or disabled local account comes back, and so does a person you deactivated in the platform interface, when the dump holds the authorization server's databases.
Remove that access again after the restore.
A dump from an install that never turned on the bundled Keycloak has no keycloak schema.
Such a dump comes from a release before 2.5.0, from a 2.5.0 or later install that has always had the bundled Keycloak off, or from NetBox Community.
Turning the bundled Keycloak off later does not remove the schema, so a dump from such an install still holds it.
After you restore it, nobody can sign in through Keycloak until the NetBox Enterprise operator repairs the schema, within about 10 minutes after you turn off maintenance mode.
Before you restore such a dump, read Restore a dump that has no Keycloak data.
The restored schemas belong to database roles that the install creates when each feature is turned on: keycloak_netbox for Keycloak, and pas-migrate and pas-runtime for the authorization server.
When the dump holds these schemas, restore it into an install that has the same features turned on as the one you backed up.
On a new install, apply your saved configuration first, and wait until NetBox Enterprise is Ready.
If a role is missing, the restore cannot give it the schema, and turning the feature on afterwards does not fix that.
The Keycloak role is named after the NetBoxEnterprise resource: on Embedded Cluster the resource is always netbox.
On Helm, restore into a resource with the same name as the one you backed up.
To restore from a dump file, send the netbox-enterprise.pgsql file from the backup to psql in the PostgreSQL pod:
NAMESPACE="${NETBOX_NAMESPACE:-$(kubectl get deployments \
-A -l 'app.kubernetes.io/component=netbox' \
-ojsonpath='{range .items[*]}{.metadata.namespace}{"\n"}{end}' \
| sort -u)}" && \
{ [ "$(printf '%s\n' "${NAMESPACE}" | grep -c .)" -eq 1 ] \
|| { printf '%s\n' 'Set NETBOX_NAMESPACE to the namespace of your install. Namespaces found:' \
"${NAMESPACE:-(none)}" >&2; false; }; } && \
NETBOX_DATABASE_FILE="netbox-enterprise.pgsql" && \
DIODE_DEPLOYMENT_COUNT="$(kubectl get deployments -n "${NAMESPACE}" -o name | grep -c diode || :)" && \
HYDRA_DEPLOYMENT_COUNT="$(kubectl get deployments -n "${NAMESPACE}" -o name | grep -c hydra || :)" && \
POSTGRESQL_MAIN_POD="$(kubectl get pod \
-o name \
-n "${NAMESPACE}" \
-l 'postgres-operator.crunchydata.com/role=master' \
| head -n 1 \
)" && \
{ grep -q '^-- PostgreSQL database cluster dump complete$' "${NETBOX_DATABASE_FILE}" \
|| { echo "${NETBOX_DATABASE_FILE} is missing or incomplete. Nothing was restored." >&2; false; }; } && \
for DB in netbox diode hydra copilot; do
kubectl exec "${POSTGRESQL_MAIN_POD}" \
-n "${NAMESPACE}" \
-c database \
-- dropdb --if-exists --force "${DB}"; \
done && \
for DB in platform_auth platform_auth_fga; do
if grep -q "^\\\\connect ${DB}\$" "${NETBOX_DATABASE_FILE}"; then \
kubectl exec "${POSTGRESQL_MAIN_POD}" \
-n "${NAMESPACE}" \
-c database \
-- dropdb --if-exists --force "${DB}"; \
fi; \
done && \
( \
if [ "${DIODE_DEPLOYMENT_COUNT}" -gt 0 ]; then \
echo "CREATE DATABASE diode WITH TEMPLATE = template0 ENCODING = 'UTF8';"; \
fi && \
if [ "${HYDRA_DEPLOYMENT_COUNT}" -gt 0 ]; then \
echo "CREATE DATABASE hydra WITH TEMPLATE = template0 ENCODING = 'UTF8';"; \
fi && \
if grep -q '^\\connect copilot$' "${NETBOX_DATABASE_FILE}"; then \
echo "CREATE DATABASE copilot WITH TEMPLATE = template0 ENCODING = 'UTF8';"; \
fi && \
for DB in platform_auth platform_auth_fga; do \
if grep -q "^\\\\connect ${DB}\$" "${NETBOX_DATABASE_FILE}"; then \
echo "CREATE DATABASE ${DB} WITH TEMPLATE = template0 ENCODING = 'UTF8';"; \
fi; \
done && \
echo "CREATE DATABASE netbox WITH TEMPLATE = template0 ENCODING = 'UTF8';"; \
) \
| kubectl exec "${POSTGRESQL_MAIN_POD}" \
-n "${NAMESPACE}" \
-i \
-c database \
-- psql -d template1 -f- && \
grep -v -E '^(ALTER|CREATE|DROP) (DATABASE|ROLE) ' "${NETBOX_DATABASE_FILE}" \
| kubectl exec "${POSTGRESQL_MAIN_POD}" \
-n "${NAMESPACE}" \
-i \
-c database \
-- psql -d netbox -f- 2>&1 \
| tee netbox-enterprise-restore.logpsql reports a failed statement and goes on, so the commands can finish while part of the data did not restore.
Count the errors in the log:
grep -c 'ERROR:' netbox-enterprise-restore.logA clean restore prints 0.
If it prints more, keep maintenance mode on, and read the failed statements with grep 'ERROR:' netbox-enterprise-restore.log.
A role ... does not exist error means the install does not have the features of the one you backed up: see the warning above.
Then run this command to set the correct database permissions:
NAMESPACE="${NETBOX_NAMESPACE:-$(kubectl get deployments \
-A -l 'app.kubernetes.io/component=netbox' \
-ojsonpath='{range .items[*]}{.metadata.namespace}{"\n"}{end}' \
| sort -u)}" && \
{ [ "$(printf '%s\n' "${NAMESPACE}" | grep -c .)" -eq 1 ] \
|| { printf '%s\n' 'Set NETBOX_NAMESPACE to the namespace of your install. Namespaces found:' \
"${NAMESPACE:-(none)}" >&2; false; }; } && \
POSTGRESQL_MAIN_POD="$(kubectl get pod \
-o name \
-n "${NAMESPACE}" \
-l 'postgres-operator.crunchydata.com/role=master' \
| head -n 1 \
)" && \
for DB in $(kubectl exec "${POSTGRESQL_MAIN_POD}" \
-n "${NAMESPACE}" \
-c database \
-- \
psql -t -c "SELECT datname FROM pg_database WHERE datname IN ('netbox', 'hydra', 'diode', 'copilot');"; \
); do \
kubectl exec "${POSTGRESQL_MAIN_POD}" \
-n "${NAMESPACE}" \
-i \
-c database \
-- \
psql --dbname "${DB}" -e -c "\
ALTER DATABASE ${DB} OWNER TO ${DB}; \
GRANT ALL PRIVILEGES ON DATABASE ${DB} TO ${DB}; \
GRANT CREATE ON SCHEMA public TO ${DB}; \
GRANT ALL PRIVILEGES ON ALL FUNCTIONS IN SCHEMA public TO ${DB}; \
GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO ${DB}; \
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO ${DB}; \
"; \
doneThe command leaves the keycloak schema and the authorization server's databases alone.
The dump already sets the owners and grants of their schemas and tables.
If you use the bundled Keycloak and the dump holds the keycloak schema, restart Keycloak.
When the dump's realm differs from the one Keycloak is serving, Keycloak keeps serving the old realm until it restarts.
Until then, sign-in through Keycloak can fail, or can sign people in against the realm from before the restore.
The Keycloak operator owns the Keycloak StatefulSet, so restart Keycloak by deleting its pods, not by changing the StatefulSet:
NAMESPACE="${NETBOX_NAMESPACE:-$(kubectl get deployments \
-A -l 'app.kubernetes.io/component=netbox' \
-ojsonpath='{range .items[*]}{.metadata.namespace}{"\n"}{end}' \
| sort -u)}" && \
{ [ "$(printf '%s\n' "${NAMESPACE}" | grep -c .)" -eq 1 ] \
|| { printf '%s\n' 'Set NETBOX_NAMESPACE to the namespace of your install. Namespaces found:' \
"${NAMESPACE:-(none)}" >&2; false; }; } && \
NETBOX_CLUSTER="$(kubectl get netboxenterprise \
-n "${NAMESPACE}" \
-ojsonpath='{.items[0].metadata.name}')" && \
KEYCLOAK_PODS="app=keycloak,app.kubernetes.io/instance=${NETBOX_CLUSTER}-keycloak,app.kubernetes.io/managed-by=keycloak-operator" && \
kubectl delete pod -n "${NAMESPACE}" -l "${KEYCLOAK_PODS}" && \
kubectl rollout status statefulset "${NETBOX_CLUSTER}-keycloak" \
-n "${NAMESPACE}"If the dump has no keycloak schema, skip this step: the operator restarts Keycloak itself, as Restore a dump that has no Keycloak data describes.
Then sign in with an account from the dump to confirm the restore.
The restore replaces platform_auth and platform_auth_fga only when the dump holds them.
A dump without them leaves the authorization server's data as it was before the restore, so it no longer matches the restored NetBox data.
Connections can then point to NetBox accounts that the restore removed.
A connection names its NetBox account by ID, and the restore also sets back the ID that NetBox gives its next account.
A NetBox account created later can get the ID of a removed one, and with it that account's connection.
Before anyone creates a NetBox account, remove the connections whose NetBox account no longer exists.
User mapping shows only accounts that exist, so use the API: list the connections with ListLinks, and remove each one with DeleteLink.
Restore a dump that has no Keycloak data
After you restore a dump that has no keycloak schema, Keycloak has no tables, and every sign-in through it fails.
The Keycloak logs show errors such as relation "keycloak.realm" does not exist.
NetBox Enterprise still reports Ready.
The NetBox Enterprise operator repairs the schema without your help. It does not check the schema while maintenance mode is on, so the repair starts when you turn off maintenance mode:
- The operator sets the
IdentityProvidercondition of the NetBoxEnterprise resource toFalse, with reasonKeycloakSchemaMissing, and records aKeycloakSchemaMissingwarning event. - At least 5 minutes later, if the schema is still missing, the operator restarts Keycloak and runs the provisioning job again, and records a
KeycloakSchemaReprovisionedevent. The job creates an empty schema, and Keycloak creates its tables when it starts. - The Keycloak operator imports the realm again, and the condition goes back to
True. While Keycloak restarts, the condition can showUnknown.
To follow the repair, read the condition:
NAMESPACE="${NETBOX_NAMESPACE:-$(kubectl get deployments \
-A -l 'app.kubernetes.io/component=netbox' \
-ojsonpath='{range .items[*]}{.metadata.namespace}{"\n"}{end}' \
| sort -u)}" && \
{ [ "$(printf '%s\n' "${NAMESPACE}" | grep -c .)" -eq 1 ] \
|| { printf '%s\n' 'Set NETBOX_NAMESPACE to the namespace of your install. Namespaces found:' \
"${NAMESPACE:-(none)}" >&2; false; }; } && \
NETBOX_CLUSTER="$(kubectl get netboxenterprise \
-n "${NAMESPACE}" \
-ojsonpath='{.items[0].metadata.name}')" && \
kubectl get netboxenterprise "${NETBOX_CLUSTER}" \
-n "${NAMESPACE}" \
-o jsonpath='{.status.conditions[?(@.type=="IdentityProvider")]}'The condition reads True until the operator first checks the schema, about a minute after you turn off maintenance mode.
Wait until it reads False, then until it reads True again.
Then sign in to confirm that the repair is done.
Keycloak now has a new, empty realm, so every person gets a new identity:
- People who sign in through your federated provider get a new Keycloak user at their next sign-in, with a new ID (the
subclaim). - Local Keycloak accounts are gone, and nobody can sign in with one.
Create each account again in the
netboxrealm, as Installs with no identity provider describes. On Embedded Cluster, create an administrator's account with the script in the next item instead. - The platform administrators in Initial Platform Administrators (
pas_initial_admin_subs) lose access, because their IDs no longer exist. If an administrator signs in through your federated provider, have them sign in once first, so that Keycloak has a user for them. On Embedded Cluster, make each administrator again withnbe-bootstrap-admin.sh, as Add the first platform administrator describes. The script finds the person's Keycloak user by email, adds its new ID to Initial Platform Administrators, and deploys. Wait for the deploy to finish before signing in. If the administrator had a local account, the script finds no user and offers to create one: answery. The script keeps the old IDs in the list, where they match nobody. To remove them, see Removing an administrator. On Helm, find each administrator's new ID under Users in the Keycloak admin console, as Read the break-glass Keycloak administrator describes. SetnetboxEnterprise.spec.pas.initialAdminSubsto the new IDs, as Who can administer connections describes. - Connections and platform roles that belong to the old IDs no longer match anyone. Remove the old connections, as described above, then connect people again in User mapping.
Backup Essentials
The short version of backup and restore for an Embedded Cluster install. Set up disaster recovery backups once, check them, and restore on a new host.
PostgreSQL Backups and Disaster Recovery
The built-in PostgreSQL uses pgBackRest for PostgreSQL backups. This page covers disaster recovery, PostgreSQL backups, and how the two work together.