NetBox Enterprise Backups

Embedded Cluster only

This guide covers backup procedures for Embedded Cluster deployments. For Helm deployments, see Helm Backups.

Much like the NetBox software itself, NetBox Enterprise uses 2 main datastores: PostgreSQL, and Redis.

PostgreSQL is used for the primary model data in NetBox, including sites, facilities, racks, and so on.

Redis is used for internal caching and the task queue.

For each type of datastore you can choose to use a built-in deployment, or configure NetBox Enterprise to use an existing external resource already in your environment.

External Databases

If you are providing your own database(s) for use by NetBox Enterprise, it is expected that you have your own processes for high availability, backup, and restore.

Running 2.2.x with external PostgreSQL?

On the 2.2.x line, deployments using customer-managed (external) PostgreSQL do not detect that a restore has wiped the nbe_ready migration sentinel, so NetBox and the worker can hang on startup. See Known Issues for 2.2.1 for the symptom and recovery steps. Upgrade to 2.3.0 before restoring. Deployments on 2.3.0 or later, and deployments using the built-in PostgreSQL, are not affected.

Using Disaster Recovery for Backups

A disaster recovery backup will preserve the complete state of your NetBox Enterprise install, from allocated volumes to databases to custom configuration.

This feature is included in NetBox Enterprise installs, and can be enabled by installing some extra dependencies to your cluster for KOTS installs.

The database comes back from pgBackRest, not from the volume copy

From 2.3.0 the built-in PostgreSQL is backed up by pgBackRest, and the PostgreSQL data volume is excluded from the backup on purpose. A completed restore therefore always leaves that volume empty, and the database is rebuilt from the pgBackRest repository. The operator does that for you on the restore target.

Three consequences matter before you rely on the steps below:

A backup taken before 2.3.0 is not a restore point for the database. Take a full backup immediately after you upgrade.
A healthy cluster after a restore is not evidence that your data came back. Confirm the row counts.
Turning PostgreSQL backups off leaves the database backed up by nothing at all.

Read PostgreSQL Backups and Disaster Recovery before you test a restore.

Backing Up the NetBox Enterprise Installation

Give each cluster its own backup location

If you run more than one NetBox Enterprise cluster -- production and staging, or two sites -- give each one a unique Prefix in the backup settings below, or a separate bucket. Clusters that share a bucket and prefix write into a single pool of backups that the restore prompt cannot tell apart. See Restoring the NetBox Enterprise Installation.

Navigate to the backup configuration by clicking the ^^Backup settings^^ link in the Disaster Recovery section of the admin console.
Input your S3 bucket, credentials, endpoint, and region.
Click Update storage settings -- it will spend a few moments validating that your settings work.
Perform a backup by clicking the Backups tab, and then clicking Start backup.

Restoring the NetBox Enterprise Installation

Known issue: a shared backup location offers other clusters' backups

netbox-enterprise restore offers every backup it finds in the location you point it at, newest first, whether or not this cluster produced it. The prompt identifies a backup only by name and timestamp:

✔  Found 2 restorable backups!
? Restore from backup "instance-zfk59" (2024-09-10 18:33:03 UTC)? (Y/n)

So if two clusters share a bucket and prefix and the other cluster's backup is the newer one, that is what you are offered. Nothing about accepting it looks wrong: the restore completes, reports success, and the cluster comes up healthy holding another cluster's data, with no error to investigate.

Before you rebuild a cluster, note the name and completion time of the backup you intend to restore, from the Backups tab of the admin console while the cluster is still healthy. Continue at the restore prompt only if the name matches. If it does not, answer n, and re-run the restore against a bucket and prefix that hold only this cluster's backups.

The prevention is a unique prefix or bucket per cluster, set under Backing Up the NetBox Enterprise Installation.

Known issue: an upgraded cluster hides its newest backups

netbox-enterprise restore compares each backup against the application version and the Embedded Cluster version compiled into the installer binary on disk, and it drops every backup that differs. An Admin Console upgrade replaces neither the binary nor those versions, so on an upgraded cluster the restore hides every backup taken since the original install.

Embedded Cluster lists its reasons only when no backup survives the filter. When one old backup does survive, the newer ones disappear with no message. The prompt then reads Found 1 restorable backup!, and to accept it restores the cluster to its original-install state.

Before you start, read the versions off the backup you intend to restore and compare them against your installer. See Confirm the installer matches your newest backup.

Download the latest NetBox Enterprise installer following the same instructions you did for a new install, if you haven't already.
Run: ./netbox-enterprise restore
Enter the same S3 credentials you use for backups.
Next, it will take a few minutes to bring up the node. When it's complete, you will be prompted to continue the restore:
Enter Y to continue, and the restore will launch the cluster.
(Optional) If you plan to have more than one node in the new cluster, you can go to the admin console and configure them when prompted.
Enter continue to finish bringing the NetBox Enterprise application up.
It will take a few more minutes to finish bringing the application up, and then you will see "Application restored!"

NetBox Enterprise and the Admin Console should now be completely restored and available as normal.
Confirm your data came back. "Application restored!" reports on the Velero restore only. Run the check below, and compare both numbers against the baseline you recorded before the loss.

NETBOX_NAMESPACE="$(kubectl get deployments \
  -A -l 'app.kubernetes.io/component=netbox' \
  -ojsonpath='{.items[0].metadata.namespace}')" && \
NETBOX_MAIN_POD="$(kubectl get pod \
  -o name \
  -n "${NETBOX_NAMESPACE}" \
  -l 'app.kubernetes.io/component=netbox' \
  | head -n 1 \
  )" && \
kubectl exec "${NETBOX_MAIN_POD}" \
  -n "${NETBOX_NAMESPACE}" \
  -c netbox \
  -- python manage.py shell -c \
  'from core.models import ObjectChange
from dcim.models import Site
print("sites:", Site.objects.count())
print("newest change:", ObjectChange.objects.order_by("-time").values_list("time", flat=True).first())'

The object count proves the database is not empty. The newest change timestamp proves you did not lose the tail -- a recovery that stops at the backup instead of the end of the write-ahead log loses days of edits and leaves the count unchanged. Record both while the cluster is healthy, as part of your backup routine.

A cluster that comes up healthy but empty is a failed restore. See Restore procedure for what to do next.

Restoring a high-availability cluster stops partway

After the Admin Console comes back, node 1 holds and asks you to add the other controller nodes. It re-asks rather than fails, so join the others while it waits. See Restore the cluster.

Manual Backup and Restore

Besides disaster recovery, it is also a good idea to keep backups of your data in case you want to view, partially restore, or move your data to another system.

NetBox Enterprise Namespace

The default namespace is kotsadm.

The instructions below default to kotsadm, but you can change the NETBOX_NAMESPACE variable to match your system.

Backing Up Your Data

Backing up NetBox Enterprise's data manually is reasonably simple, and Kubernetes makes it easy to access them from the command-line.

Before You Back Up: Accessing Your Cluster

Before you can back anything up, you must first make sure you can access the cluster on the command line.

See the advanced tools documentation for details on connecting to your NetBox Enterprise cluster.

Media, Scripts, and Reports

Runtime files are stored in a volume accessible from the NetBox containers. To back them up, you can run this:

NETBOX_NAMESPACE="$(kubectl get deployments \
  -A -l 'app.kubernetes.io/component=netbox' \
  -ojsonpath='{.items[0].metadata.namespace}')" && \
NETBOX_MAIN_POD="$(kubectl get pod \
  -o name \
  -n "${NETBOX_NAMESPACE}" \
  -l 'app.kubernetes.io/component=netbox' \
  | head -n 1 \
  )" && \
kubectl exec "${NETBOX_MAIN_POD}" \
  -n "${NETBOX_NAMESPACE}" \
  -c netbox \
  -- /bin/sh -c ' \
      cd /opt/netbox/netbox && \
      find media scripts reports static -type f > /tmp/files.txt && \
      tar -czf - \
      --owner=0 \
      --group=0 \
      -T /tmp/files.txt' > netbox-data.tar.gz

Built-In PostgreSQL

The built-in PostgreSQL is deployed using the CrunchyData Postgres Operator.

This dump is a portability tool, not the disaster-recovery path

From 2.3.0 the disaster-recovery path for the built-in database is pgBackRest, which runs on a schedule and archives the write-ahead log continuously. A dump is a point-in-time copy in a portable format. Use it to move data to another system, or to inspect it. Use pgBackRest to recover the cluster. See PostgreSQL Backups and Disaster Recovery.

Since the PostgreSQL CLI tools are already available inside the cluster, all we need to do to dump the database is to call into the correct container and run a pg_dump there.

To perform a database dump, run these commands:

NETBOX_NAMESPACE="$(kubectl get deployments \
  -A -l 'app.kubernetes.io/component=netbox' \
  -ojsonpath='{.items[0].metadata.namespace}')" && \
NETBOX_DATABASE_FILE="netbox-enterprise.pgsql" && \
POSTGRESQL_MAIN_POD="$(kubectl get pod \
  -o name \
  -n "${NETBOX_NAMESPACE}" \
  -l 'postgres-operator.crunchydata.com/role=master' \
  | head -n 1 \
  )" && \
EXCLUDE_DATABASES="$(kubectl exec "${POSTGRESQL_MAIN_POD}" \
  -n "${NETBOX_NAMESPACE}" \
  -c database \
  -- \
    psql -t -c "SELECT CONCAT('--exclude-database=', datname) \
      FROM pg_database \
      WHERE datname <> ALL ('{netbox,diode,hydra}')" \
)" && \
kubectl exec "${POSTGRESQL_MAIN_POD}" \
  -n "${NETBOX_NAMESPACE}" \
  -c database \
  -- \
    pg_dumpall \
      $EXCLUDE_DATABASES \
    > "${NETBOX_DATABASE_FILE}"

This will create a netbox-enterprise.pgsql file in your local directory. Save it somewhere safe for future restores.

For more details on backing up NetBox databases, see the official NetBox documentation.

Cluster Secrets

A manual database restore is only useful if the rest of the cluster can decrypt the data and authenticate against the restored databases. Back up:

Operator-managed secrets, including the NetBox SECRET_KEY and API token peppers, the auto-generated superuser credentials, the Diode OAuth2 client secret, and the Hydra system secret.
PGO-managed pguser secrets, which hold the per-user PostgreSQL passwords that match the password hashes inside the pg_dumpall output.

Run this set of commands:

NETBOX_NAMESPACE="$(kubectl get deployments \
  -A -l 'app.kubernetes.io/component=netbox' \
  -ojsonpath='{.items[0].metadata.namespace}')" && \
(
  kubectl get secrets \
    --namespace "${NETBOX_NAMESPACE}" \
    --selector 'app.kubernetes.io/managed-by=netbox-operator' \
    --output name; \
  kubectl get secrets \
    --namespace "${NETBOX_NAMESPACE}" \
    --output name \
    | grep -E '/[^/]+-postgres-pguser-[^/]+$'; \
) | sort -u | while read -r SECRET; do \
  echo "---" && \
  kubectl get "${SECRET}" \
    --namespace "${NETBOX_NAMESPACE}" \
    -o yaml \
  | grep -v -E '^  (creationTimestamp|resourceVersion|uid):'; \
done > netbox-enterprise-secrets.yaml

Save the resulting netbox-enterprise-secrets.yaml alongside your netbox-enterprise.pgsql for future restores. Open the file before stashing it and confirm it contains multiple kind: Secret documents -- an empty or single-document file means the selectors did not match anything in your cluster.

Admin Console Configuration

You can export the current Admin Console configuration as a ConfigValues file. This is useful when migrating to a new host or reinstalling, as it captures all your Admin Console settings (database connections, TLS configuration, feature toggles, etc.) in a portable YAML file.

From within the cluster shell, run:

kubectl exec deploy/kotsadm -n kotsadm -- \
  /kots get config --current --decrypt --namespace kotsadm \
  > netbox-enterprise-config.yaml

The --decrypt flag ensures that password fields (like database credentials) are included in plaintext rather than encrypted. Store this file securely alongside your other backups.

To apply a saved configuration to a new install, run:

kubectl exec -i deploy/kotsadm -n kotsadm -- \
  /kots set config netbox-enterprise \
    --config-file /dev/stdin --deploy --namespace kotsadm \
  < netbox-enterprise-config.yaml

Restoring Your Backups

Restoring is almost as simple as backing up. You just need to put NetBox Enterprise into maintenance mode first.

Enabling and Disabling Maintenance Mode

Put NetBox Enterprise into maintenance mode by going to the Config tab and checking the Enable Maintenance Mode checkbox.
Click the "Save config" button at the bottom of the form, and then when the admin console prompts you, click "go to updated version".
Confirm that the New version available at the top denotes it's a config change, and if so click the "Deploy" button.

This will shut down NetBox but leave the other NetBox Enterprise infrastructure up, so you can safely restore.

When you are done restoring your data, just follow the same steps, unchecking Enable Maintenance Mode and deploying the updated configuration.

Media, Scripts, and Reports

To restore media, scripts, and reports, you just need to unpack them into the correct directory inside a NetBox container.

The backup instructions above back up all three of the media/, scripts/, and reports/ subdirectories in one file.

If you are restoring a backup from another NetBox instance, you might need to change the name of the tarball and the path after the -C at the end of this command to unpack your backup into the right location.

NETBOX_NAMESPACE="$(kubectl get deployments \
  -A -l 'app.kubernetes.io/component=netbox' \
  -ojsonpath='{.items[0].metadata.namespace}')" && \
NETBOX_MAINTENANCE_POD="$(kubectl get pod \
  -o name \
  -n "${NETBOX_NAMESPACE}" \
  -l 'maintenance-mode=true' \
  | head -n 1 \
  )" && \
cat netbox-data.tar.gz | kubectl exec ${NETBOX_MAINTENANCE_POD} \
  -n "${NETBOX_NAMESPACE}" \
  -i \
  -- tar -xvzf - \
    --no-same-owner \
    --no-same-permission \
    -C /opt/netbox/netbox

Cluster Secrets

Restore the secrets before the database dump. The operator and PGO need them in place so that pods can decrypt session data, authenticate to Postgres, and serve Diode OAuth flows against the restored data.

NETBOX_NAMESPACE="$(kubectl get deployments \
  -A -l 'app.kubernetes.io/component=netbox' \
  -ojsonpath='{.items[0].metadata.namespace}')" && \
kubectl apply \
  --server-side \
  --force-conflicts \
  --namespace "${NETBOX_NAMESPACE}" \
  --filename netbox-enterprise-secrets.yaml

After applying, restart any pods that mounted the old secret values so they pick up the restored credentials:

kubectl rollout restart deployment \
  --namespace "${NETBOX_NAMESPACE}" \
  -l 'app.kubernetes.io/managed-by=netbox-operator'

Built-In PostgreSQL

To restore from a dump file, pipe the netbox-enterprise.pgsql created during backup into psql in the PostgreSQL pod:

NETBOX_NAMESPACE="$(kubectl get deployments \
  -A -l 'app.kubernetes.io/component=netbox' \
  -ojsonpath='{.items[0].metadata.namespace}')" && \
NETBOX_DATABASE_FILE="netbox-enterprise.pgsql" && \
DIODE_DEPLOYMENT_COUNT="$(kubectl get deployments -n "${NETBOX_NAMESPACE}" -o name | grep -c diode || :)" && \
HYDRA_DEPLOYMENT_COUNT="$(kubectl get deployments -n "${NETBOX_NAMESPACE}" -o name | grep -c hydra || :)" && \
POSTGRESQL_MAIN_POD="$(kubectl get pod \
  -o name \
  -n "${NETBOX_NAMESPACE}" \
  -l 'postgres-operator.crunchydata.com/role=master' \
  | head -n 1 \
  )" && \
for DB in netbox diode hydra; do
  kubectl exec "${POSTGRESQL_MAIN_POD}" \
    -n "${NETBOX_NAMESPACE}" \
    -c database \
    -- dropdb --if-exists --force "${DB}"; \
done && \
( \
  if [ "${DIODE_DEPLOYMENT_COUNT}" -gt 0 ]; then \
    echo "CREATE DATABASE diode WITH TEMPLATE = template0 ENCODING = 'UTF8';"; \
  fi && \
  if [ "${HYDRA_DEPLOYMENT_COUNT}" -gt 0 ]; then \
    echo "CREATE DATABASE hydra WITH TEMPLATE = template0 ENCODING = 'UTF8';"; \
  fi && \
  echo "CREATE DATABASE netbox WITH TEMPLATE = template0 ENCODING = 'UTF8';"; \
) \
| kubectl exec "${POSTGRESQL_MAIN_POD}" \
  -n "${NETBOX_NAMESPACE}" \
  -i \
  -c database \
    -- psql -d template1 -f- && \
grep -v -E '^(ALTER|CREATE|DROP) (DATABASE|ROLE) ' "${NETBOX_DATABASE_FILE}" \
| kubectl exec "${POSTGRESQL_MAIN_POD}" \
  -n "${NETBOX_NAMESPACE}" \
  -i \
  -c database \
    -- psql -d netbox -f-

Following this run the below to ensure all database permissions are correct:

NETBOX_NAMESPACE="$(kubectl get deployments \
  -A -l 'app.kubernetes.io/component=netbox' \
  -ojsonpath='{.items[0].metadata.namespace}')" && \
POSTGRESQL_MAIN_POD="$(kubectl get pod \
  -o name \
  -n "${NETBOX_NAMESPACE}" \
  -l 'postgres-operator.crunchydata.com/role=master' \
  | head -n 1 \
  )" && \
for DB in $(kubectl exec "${POSTGRESQL_MAIN_POD}" \
  -n "${NETBOX_NAMESPACE}" \
  -c database \
  -- \
    psql -t -c "SELECT datname FROM pg_database WHERE datname IN ('netbox', 'hydra', 'diode');"; \
); do \
  kubectl exec "${POSTGRESQL_MAIN_POD}" \
  -n "${NETBOX_NAMESPACE}" \
  -i \
  -c database \
  -- \
    psql --dbname "${DB}" -e -c "\
      ALTER DATABASE ${DB} OWNER TO ${DB}; \
      GRANT ALL PRIVILEGES ON DATABASE ${DB} TO ${DB}; \
      GRANT CREATE ON SCHEMA public TO ${DB}; \
      GRANT ALL PRIVILEGES ON ALL FUNCTIONS IN SCHEMA public TO ${DB}; \
      GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO ${DB}; \
      GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO ${DB}; \
    "; \
done

On this page