In-Cluster Object Storage

High availability requires that NetBox media be held in shared object storage rather than on a node-local volume (see High Availability). From 2.3.0, that object store can run inside the cluster instead of coming from an external S3 service: NetBox Enterprise provisions and manages an in-cluster, S3-compatible object store (Garage) for NetBox media and scripts.

This removes the external object-storage dependency for availability. It does not, on its own, make the whole deployment highly available — the database and cache tiers must also be HA (see Requirements) — and it is not a backup (see Disaster recovery).

When to use in-cluster storage

In-cluster and external object storage both satisfy the object-storage requirement for HA. The choice is operational.

Requirements

In-cluster storage satisfies only the object-storage leg of HA. All of the following must still hold for a supported highly available deployment — the same conditions the Admin Console HA-readiness checklist enforces:

1. Three or more nodes, all registered as controllers.
2. The multi-node entitlement (included in the Premium tier).
3. An HA-capable database and cache tier. The built-in PostgreSQL and Redis are HA-capable in place on a 3+ node cluster with the multi-node entitlement — PostgreSQL through Patroni quorum synchronous replication, Redis through Sentinel with a one-replica-per-node spread — or you can use external PostgreSQL and Redis. External PostgreSQL is the stronger choice if you need a guarantee that no two database replicas share a node, since built-in replica placement is best-effort. See In-Cluster Database and Cache for what each tier guarantees, and High Availability for the full requirement set.
4. The storage backend set to in-cluster (below).

On a cluster of three or more nodes, a preflight check fails the install while media is still on node-local storage; selecting in-cluster storage (or external S3) satisfies it. The built-in PostgreSQL and Redis do not need to be externalized — at 3+ nodes with the entitlement they run in HA mode in place.

Enable in-cluster storage

On Embedded Cluster, set the storage backend in the Admin Console Config screen. The File Storage Backend setting offers three options:

• Node-Local Storage (default; single-node only) (pvc) — media on a node-local volume.
• External S3-Compatible Storage (external) — an external endpoint you provide.
• In-Cluster Object Storage (Garage; requires 3+ nodes) (in-cluster) — the managed in-cluster store.

Select In-Cluster Object Storage and deploy. NetBox Enterprise then, without further configuration:

• provisions a three-node Garage cluster with replication factor 3, so every object is stored on all three Garage nodes and the store survives the loss of any one;
• creates the netbox bucket and an access key; and
• points NetBox at the in-cluster S3 endpoint automatically.

NetBox does not start serving until Garage is ready — the operator waits for the object store to apply its storage layout and pass an S3 read/write round-trip first, so the application never comes up against storage that cannot yet hold media.

Sizing impact

In-cluster storage adds a Garage object-storage tier to the cluster. Size for it on top of the standard three-node HA profile (8 vCPU / 24 GB RAM / 100 GB SSD per node — see System Requirements).

Notes:

• CPU and memory overhead is modest — roughly half a core and 512Mi per node.
• Disk is the change to plan for. Uploaded media now lives in the Garage data volume on node storage rather than in an external service. The default data volume is 20Gi per pod; size node disk for your expected media growth.
• PostgreSQL and Redis add in-cluster load only if you run them in-cluster (the built-in HA option). At three nodes that is three PostgreSQL instances plus three Redis data pods and three Sentinel pods, each with their own CPU/memory and small data volumes. Built-in PostgreSQL ships with no CPU/memory requests set by default — apply a resource preset in the Admin Console to bound it — while built-in Redis data pods request 250m CPU / 256Mi. Running PostgreSQL and Redis externally leaves Garage as the only added in-cluster load. See High Availability for the full data-tier picture.
• The install preflight still enforces the standard minimums, which are not all per-node: at least one node must have 4Gi allocatable memory, the cluster's total CPU capacity must reach 4 cores, and every node must have 40Gi ephemeral storage. These are floors, not the recommended HA profile above.

Disaster recovery

In-cluster storage keeps NetBox media available through the loss of a node: Garage runs three storage nodes with replication factor 3, so every object is held on all three and a surviving Garage node continues serving media. That is an availability property, not a backup.

Configure the external backup target in the Admin Console Disaster Recovery settings (bucket, credentials, endpoint, region) as described in Backups. The backup location is separate from — and unaffected by — the in-cluster store.

Snapshot the metadata before every backup

Garage keeps its metadata in a SQLite database on each node's meta volume. A filesystem backup of a live SQLite file can capture a torn, mid-write state. The data volume needs no such care, because its blocks are content-addressed and written once.

Run this immediately before you start a backup. One command snapshots the metadata on every Garage node, and Garage writes the snapshot onto the same meta volume that the backup captures:

kubectl exec -n "$NS" "$CLUSTER_NAME-garage-0" -- \
  /garage -c /etc/garage.toml meta snapshot --all

On Embedded Cluster, run it just before you click Start backup in the Admin Console.

If a restored node then fails to start with a SQLite error, recover it from that snapshot:

1. Scale the StatefulSet down, or cordon the node, so the affected pod stops restarting.
2. Replace the live database in the pod's meta volume with the newest directory under /var/lib/garage/meta/snapshots/. See the Garage recovery documentation.
3. Bring the node back. With three replicas, a node rebuilt from a slightly older snapshot re-syncs from its peers.

What comes back, and in what order

Every Garage resource carries the label the backup selects, so the restore returns the GarageCluster resource, the StatefulSet and its volumes, the config Secret, and the S3 credential Secret that NetBox authenticates with. The peer-discovery resources are not backed up and do not need to be, because Garage repopulates them at startup.

The operator sequences the bring-up. Both reconcilers stand down while the restore is active, then resume in dependency order: Garage re-applies its layout and reaches Ready only once a real S3 round-trip succeeds, and only then does NetBox get pointed at the store. No manual ordering is needed. If you are watching, expect Garage to reach Ready first, then NetBox.

Confirm the data came back rather than trust the pods:

kubectl get garagecluster -n "$NS" \
  -o jsonpath='{.items[0].status.conditions[?(@.type=="Ready")].status}{"\n"}'

Then open a NetBox record that has an attached image. A green Ready plus a served object is the proof. A running pod alone is not.

Conditions to watch

The object store is a separate GarageCluster resource with its own conditions. They do not appear on the NetBoxEnterprise resource, so kubectl describe netboxenterprise will not show them.

On Embedded Cluster, get a shell with kubectl on it first — the appliance does not put one on your login PATH:

sudo /var/lib/embedded-cluster/bin/netbox-enterprise shell
kubectl describe garagecluster

Ready is the conjunction of LayoutApplied and Serving. For the full list, see GarageCluster Conditions. Two are worth knowing.

InClusterStorageBelowNodeGate

Reported on the NetBoxEnterprise resource — not on the GarageCluster — when the storage backend is in-cluster but the cluster has fewer than three nodes. All three Garage replicas can then share a node, so the store holds no real redundancy. Expected while you are still adding nodes during a multi-node build-out; it is not a state to leave a deployment in.

GarageSpreadHeld

Two or more Garage pods share a node, and their ReadWriteOnce volumes pin them there. Applying the hard one-pod-per-node spread could not be satisfied without leaving a pod Pending, so it is held as a soft preference instead.

The store serves normally, but it is not protected against losing that node — and this state is permanent until an operator acts. It typically arises on a cluster that started smaller and grew, leaving Garage pods bound to volumes on nodes they were originally scheduled onto.

This is the object-storage counterpart of RedisSpreadHeld, with the same cause and the same shape of fix. Two differences matter in practice:

• Garage is masterless. Every object is held on all three nodes at replication factor 3, so there is no primary to identify and avoid. Any co-located pod can be the one you move, provided the other two are healthy.
• Each pod has two volumes, not one. The meta and data claims must both be deleted, or the surviving one keeps the pod pinned to its original node.

First find which pods share a node:

kubectl get pods -n <namespace> -l app.kubernetes.io/component=garage \
  -o custom-columns='POD:.metadata.name,NODE:.spec.nodeName'

Confirm the other two pods are Ready before touching anything — this deletes one pod's copy of the data, which Garage then re-syncs from its peers.

With a co-located pod chosen:

1. Delete both of its PersistentVolumeClaims — meta-<pod> and data-<pod>.
2. Delete the pod, so it rebinds on a node with no Garage pod.

The operator hardens the spread on its own once the tier is one pod per node.

Limitations

• Object storage only. In-cluster storage makes the media tier HA; the database and cache tiers are configured separately (external, or built-in in HA mode — see Requirements).
• A held spread is permanent until an operator acts. On a cluster that grew into HA, Garage pods can stay co-located — see GarageSpreadHeld.
• HA only. In-cluster storage is intended for three-or-more-node clusters with the multi-node entitlement; it is not a single-node option.
• Switching the backend later does not migrate media automatically. Changing away from in-cluster storage leaves the Garage store in place rather than deleting it; plan a media migration if you move between backends.

Next steps

• Accessing the In-Cluster Object Store — credentials and endpoint, uploading migrated media and plugin wheelhouses, and inspecting what is stored
• High Availability — full HA topology, requirements, and failover behavior
• In-Cluster Database and Cache — the PostgreSQL and Redis tiers
• Storage Options — external S3 providers and setup
• Backups — disaster-recovery and manual backup procedures
• System Requirements — per-node hardware requirements