High Availability Requirements

High availability (HA) in NetBox Enterprise is single-cluster high availability: a multi-node cluster that recovers automatically from the loss of any single fault domain - a node or an entire availability zone - with NetBox staying available and the cluster repairing itself without manual intervention.

The recovery mechanism is the same whether the lost fault domain is a node or an availability zone. The difference is where the nodes sit - a single location or spread across availability zones - and which topology labels group them.

NetBox Enterprise 2.2 required customer-provided external services - object storage, PostgreSQL, and Redis - for data resiliency. From 2.3.0, each of these tiers can instead run in-cluster under NetBox Labs management on a cluster of three or more nodes with the multi-node entitlement; external services remain fully supported. HA uses the Multi-Node Deployment installation procedure.

What HA covers

HA operates within a single cluster: automated failover when a node or an availability zone is lost. It is not cross-site disaster recovery - replicating to an independent deployment elsewhere, with one site taking over for another, is a separate capability. The practical boundary is latency: a single cluster can span nodes, availability zones, or low-latency regions, but not geographically distant high-latency regions, because etcd quorum needs low inter-node latency.

What to expect on node or AZ loss

When a node fails - or an entire availability zone, which takes down every node in that zone - a correctly configured HA deployment recovers automatically through several mechanisms working together:

• Faster failure detection. Node-down detection and pod eviction timeouts are tuned well below Kubernetes defaults, so recovery starts in tens of seconds rather than minutes.
• Spread across fault domains. NetBox web replicas are forced onto distinct nodes (a hard spread) on multi-node installs that use shared object storage - external S3 or in-cluster; other workloads use a softer best-effort spread. Once nodes carry zone labels, the same spread operates at the availability-zone level (see Multi-AZ placement). Either way, the loss of one fault domain never takes down every replica of a service.
• Per-node scaling. Stateful and pipeline components scale their replica counts to match node count (up to 3) automatically as nodes join.
• HA-readiness signal. If a cluster has 3 or more nodes but HA has not been enabled, NetBox Enterprise surfaces a warning condition and reports itself not-ready until HA setup is complete.

After losing a node, a three-node cluster runs at two of three controllers - still functional, but with no remaining fault tolerance. Replace the failed node promptly.

Recovery time

The one-minute figure is convergence - detecting the failure and rescheduling pods. Users stay served throughout, because a surviving replica on another fault domain keeps answering (this is what the hard spread buys). A replacement pod takes longer to fully start on its own - a NetBox web pod cold-starts in roughly 2-4 minutes through its dependency-gated init containers and app boot - but on a single node or zone loss no one sees that, because the survivor is already serving. The cold-start only becomes user-visible if every replica is lost at once, which the spread exists to prevent.

Restoring failover speed after an in-place upgrade

A cluster upgraded in place from a pre-2.2 release does not pick up all of the failover tuning automatically, so it recovers more slowly than a fresh 2.2+ install until you act:

• Node-down detection: run sudo systemctl restart k0scontroller on each node after the upgrade to apply the faster detection settings.
• Pod eviction: the faster eviction timeout is fixed when the cluster is first created and cannot be changed on an existing Embedded Cluster, so eviction stays at the Kubernetes default (about five minutes) until the cluster is reinstalled.

Fresh 2.2+ installs have both already and need no action.

Requirements

A deployment delivers the HA promise only when all of the following hold. Anything less is a single-node (non-HA) configuration, regardless of what the install UI indicates.

1. Three nodes, all registered as controllers, so etcd holds quorum. Losing one leaves two of three - the cluster keeps running but has no remaining fault tolerance, so replace a lost node promptly.
2. The multi-node entitlement (included in the Premium tier). This gates HA itself, whichever way you run the data tiers, and it is what allows an in-cluster tier to scale past a single instance.
3. A highly available object-storage tier for NetBox media and scripts. Without it, media lives on a node-local volume that pins NetBox to one node and breaks recovery. Use external S3-compatible storage, or, from NetBox Enterprise 2.3.0, an in-cluster object store that NetBox Enterprise manages for you.
4. A highly available database tier. External PostgreSQL, or - from 2.3.0 - the built-in PostgreSQL running in HA mode (Patroni quorum synchronous replication, enabled at three or more nodes with the multi-node entitlement). External PostgreSQL is the stronger choice if you need a guarantee that no two replicas share a node, since built-in replica placement is best-effort.
5. A highly available cache tier. External Redis, or - from 2.3.0 - the built-in Redis running in HA mode (Sentinel with a one-replica-per-node hard spread, at three or more nodes with the entitlement).

Each data tier must be highly available, whichever way you run it. For an external database, Redis, or object store, making the service HA is your responsibility - NetBox Enterprise depends on it but does not make it HA. For a tier run in-cluster, NetBox Enterprise manages its HA for you (at three or more nodes with the multi-node entitlement).

If you choose to run a tier externally instead of in-cluster:

External PostgreSQL:

• AWS RDS, Google Cloud SQL, Azure Database for PostgreSQL, or self-hosted PostgreSQL with streaming replication (Patroni, pg_auto_failover)
• Requires three databases: netbox, diode, hydra

External Redis:

• AWS ElastiCache (Redis OSS), Google Memorystore, Azure Cache for Redis, or self-hosted Redis with Sentinel
• Sentinel-based Redis is required for durable workloads (the Diode ingestion queue)

From NetBox Enterprise 2.3.0, you can point NetBox at your Sentinel endpoints directly rather than at a fixed Redis host. In the Admin Console Redis Cache section, with Use Built-In Redis unchecked, fill in External Sentinel (Redis HA Failover):

With Sentinel configured, NetBox discovers the current master through Sentinel instead of connecting to the Redis host above, so a Sentinel-managed failover does not need a configuration change. On Helm, these map onto spec.redis.sentinels and spec.redis.sentinelMasterName.

External object storage: any S3-compatible service reachable from all nodes — a managed cloud bucket, or an endpoint you run yourself (see Self-Hosted S3 for High Availability). See Storage Options and Storage Installation. For the in-cluster alternative, see In-Cluster Object Storage.

See the external database guide for PostgreSQL and Redis configuration details.

How HA readiness is evaluated

Each data tier is assessed independently. A tier counts as highly available when it is external, or when it runs in-cluster on three or more nodes with the multi-node entitlement. An external tier counts regardless of node count, since its availability does not depend on this cluster.

The Admin Console Config screen shows a live High Availability checklist reflecting the current cluster:

• 3+ nodes
• S3 storage (external or in-cluster)
• PostgreSQL (external, or in-cluster with the multi-node entitlement)
• Redis (external, or in-cluster with the multi-node entitlement)

Node-local file storage never counts, so it stays blocked for HA. The install preflight enforces the same per-tier rule at install time and evaluates node count against the live cluster, so a fresh install that begins as a single node is assessed correctly as nodes join.

NetBox Enterprise also reports HA gaps as conditions on the NetBoxEnterprise resource - HANotEnabled when a 3+ node cluster has not had HA enabled, and HAWithoutExternalData naming any tier that cannot survive node loss. See Conditions to watch.

Placement options

Single-location: three nodes in one location. Survives the loss of any single node.

Multi-AZ: three nodes, one per availability zone, in a single region with low-latency inter-AZ networking. Survives the loss of any single availability zone. The deployment is identical to single-location - three nodes, all controllers - but each node is provisioned in a different zone. You provision the hosts across zones yourself; no cloud-provider API integration is required.

Why three availability zones (not two)

The Kubernetes control plane uses etcd, which requires a quorum majority. With two availability zones, one holds more etcd members than the other; if the larger zone fails, the survivors are in the minority and the control plane halts. With three zones (one node each), losing any single zone leaves two of three members - a majority - and the cluster keeps running.

Zone labels on Embedded Cluster

Embedded Cluster has no first-class way to apply per-node topology labels during the join process. To get availability-zone-level spread, label the nodes manually after they join, following Topology-Aware Node Labels to extend the cluster configuration. Until the nodes carry zone labels, spread is node-level only. In the standard one-node-per-zone topology that still places replicas in separate zones, but applying the labels keeps the guarantee correct if a zone ever holds more than one node.

Infrastructure requirements

Node topology

Network

Inter-node network budget: high availability depends on a low-latency, low-loss, low-jitter network between nodes. The following budget is supported, measured on a three-node cluster:

Packet loss and jitter bind sooner than mean latency: the tuned node-down detection (see What to expect on node or AZ loss) reads a lossy or jittery link as a failed node and evicts its pods, well before steady RTT becomes a problem. Co-locate nodes on a LAN or within a single region - a clean 50 ms link is fine, but a 10 ms link with high jitter or loss is not. A single cluster stretched across a high-latency or lossy WAN is not supported; that is disaster recovery, not HA.

Bandwidth: a 1 Gbps or faster NIC is recommended for inter-node traffic (etcd snapshot transfer, image pulls, node-rejoin catch-up, and cross-node pod traffic).

Also ensure: a consistent MTU across all nodes (the overlay network needs headroom, and a mismatch silently breaks pod-to-pod traffic) and clock synchronization (NTP) between nodes (TLS and etcd/Kubernetes leases assume low clock skew).

Required ports (inter-node): open these between all cluster nodes, in addition to the standard single-node ports:

Inter-node security

• Control plane: etcd and the Kubernetes API server communicate over TLS, with certificates managed by k0s.
• Data-plane (pod-to-pod) traffic: carried by the Calico overlay network. Overlay encryption (WireGuard) is not enabled by default, so run all cluster nodes on a trusted private network - for example a dedicated VLAN, VPC, or subnet with security groups - rather than across untrusted links.
• Admin Console (port 30000): TLS; restrict access to administrators only.

Nodes must be reachable from each other by IP address. NAT between nodes is not supported.

Load balancer

A stateless load balancer in front of all nodes provides a single entry point and routes around failed nodes using health checks. The cluster handles failover internally - no failover logic is needed in the load balancer. See Load Balancer Configuration.

Operator high availability

The operator can run multiple replicas with leader election, so the loss of an operator pod does not pause reconciliation and operator upgrades can be zero-downtime. This is separate from the cluster failover described above - NetBox and node or zone recovery do not depend on it - and it is opt-in (off by default).

Pre-deployment checklist

• 3 nodes, each on a separate physical or virtual machine
• All 3 nodes registered as control plane instances
• Each node meets hardware requirements (8 vCPU, 24 GB RAM, 100 GB SSD)
• Database tier is HA - external PostgreSQL reachable from all nodes, or built-in PostgreSQL with the multi-node entitlement
• Cache tier is HA - external Redis (with Sentinel) reachable from all nodes, or built-in Redis with the multi-node entitlement
• Object storage is HA - external S3-compatible storage reachable from all nodes, or in-cluster storage
• Inter-node ports open (6443, 2379-2380, 10250, 4789 UDP, 179 TCP)
• Inter-node network within budget: mean RTT ≤ 50 ms, packet loss < 1%, low/bounded jitter (co-located or intra-region)
• Load balancer configured with health checks against all nodes
• TLS certificate includes SANs for all node hostnames and the load balancer hostname

For multi-AZ, also:

• Nodes provisioned across exactly 3 availability zones (one per zone), same region
• Zone topology labels applied to each node (see Topology-Aware Node Labels)