# Advanced Tools and Troubleshooting (/docs/enterprise/nbe-troubleshooting) This guide provides diagnostic procedures and solutions for issues you may encounter when running NetBox Enterprise. ## Accessing Your Cluster [#accessing-your-cluster] Most troubleshooting procedures require cluster access through either the command line or web console. ### Command-Line Access [#command-line-access] **Prerequisite:** NetBox Enterprise installer deployment. Enter the cluster shell environment to use kubectl and related tools: ```shell /var/lib/embedded-cluster/bin/netbox-enterprise shell ``` This shell session has kubectl preconfigured with cluster credentials and context. All subsequent kubectl commands execute within this environment. **Example session:** ```shell # Enter cluster shell /var/lib/embedded-cluster/bin/netbox-enterprise shell # Now kubectl commands work kubectl -n kotsadm get pods # Exit shell when done exit ``` ### Web Console Access [#web-console-access] Access the admin console in a browser at `https://:30000/` where `` is your server hostname or IP address. The admin console provides: * Configuration management * Log viewing * System status monitoring * Application deployment controls **Note:** The admin console uses a self-signed certificate by default. Your browser will display a security warning on first access. ## Getting Help [#getting-help] If you encounter an issue that is not covered in this guide, contact NetBox Labs support with the following information: * NetBox Enterprise version * Kubernetes cluster environment details * Support bundle (see [Generating a Support Bundle](#generating-a-support-bundle)) * Relevant error messages from logs ## Diagnostic Tools [#diagnostic-tools] ### Support Bundles [#generating-a-support-bundle] Support bundles collect system information, configuration, and logs for troubleshooting. Generate a support bundle when reporting issues to NetBox Labs. **Command-line generation:** 1. [Access your cluster](#command-line-access) from the shell 2. Run the support bundle collector: ```shell kubectl support-bundle /var/lib/embedded-cluster/support/host-support-bundle.yaml --load-cluster-specs ``` The tool runs diagnostic tests and displays results in a TUI interface. Press `s` to save a summary text file or `q` to quit. A `.tar.gz` bundle file is created in the current directory regardless of the option selected. Support Bundle Output ### Log Access [#log-access] View container logs directly using kubectl: ```shell kubectl logs netbox-enterprise- -n kotsadm ``` Replace `` with the actual pod identifier from your cluster. ## Common Issues [#common-issues] ### Git Repository Sync Failures [#git-repository-sync-failures] **Symptoms:** * Large repositories (over 1GB) fail to sync * Sync operations timeout * Worker pods are killed with out-of-memory (OOM) errors * Files larger than 50MB fail to process **Cause:** Default worker pod resource limits are insufficient for large repository operations. **Solution:** Increase worker pod resource allocation through the admin console: 1. Access the admin console at `https://your-cluster-host-or-ip:30000/` 2. Navigate to **Config** section 3. Under **Resources**, select an appropriate preset: * **Regular (up to 1.5Gi mem per replica)**: Standard workloads * **Large (up to 3Gi mem per replica)**: Moderate to heavy workloads * **Extra Large (up to 6Gi mem per replica)**: Very large repositories * **2x Large (up to 12Gi mem per replica)**: Repositories over 1GB 4. Click **Save Config** 5. Deploy the updated configuration **Verification:** Check that worker pods have restarted with new resource limits: ```shell # Verify resource limits kubectl -n kotsadm get deployment netbox-enterprise-worker -o yaml \ | yq -r '.spec.template.spec.containers[] | select(.name == "netbox-worker") | .resources' # Check worker pod status kubectl -n kotsadm get pods -l app=netbox-enterprise-worker # Monitor pod restart kubectl -n kotsadm get pods -w ``` Test repository sync: 1. Navigate to **Operations > Integrations > Data Sources** in NetBox 2. Add or select your Git repository 3. Click **Sync** and monitor progress 4. Verify completion in the NetBox UI or worker logs **Additional steps if issue persists:** * Use `.gitignore` patterns to exclude large binary files not needed for NetBox * Split large monorepos into smaller, focused repositories * Contact NetBox Labs support for specialized configuration ### Excessive Log Output [#excessive-log-output] **Symptoms:** * Log storage fills rapidly * Difficult to locate relevant error messages * Performance degradation due to high I/O **Cause:** Debug logging is enabled in a production environment. **Solution:** Disable debug logging through the admin console: 1. Access the admin console 2. Navigate to **Config** section 3. Find **Enable Debug Logging** toggle 4. Disable the toggle 5. Click **Save Config** 6. Deploy the updated configuration **Important:** Debug logging exposes detailed system information and significantly impacts performance. Only enable debug mode in non-production environments when actively troubleshooting issues. ### Enabling Debug Logging (Non-Production Only) [#enabling-debug-logging-non-production-only] Debug logging provides detailed operational information for diagnosing issues. Enable it only in development or test environments. Never enable debug logging on a production system. It can expose sensitive data to unauthenticated users and impose a substantial performance penalty. When enabled, this setting (`custom_logging_enabled` in the Admin Console) sets Diode components (Ingester, Reconciler, Auth) to DEBUG logging level, providing verbose output useful for diagnosing integration and data ingestion issues. **Steps:** 1. Access the admin console 2. Navigate to **Config** section 3. Find **Enable Debug Logging** toggle 4. Enable the toggle 5. Click **Save Config** 6. Deploy the updated configuration Debug Logging Enabled **View debug logs:** ```shell kubectl logs netbox-enterprise- -n kotsadm ``` Debug Logging Example Disable debug logging after completing troubleshooting to restore normal performance. ### Cluster Access Issues [#cluster-access-issues] **Symptoms:** * Cannot connect to admin console * kubectl commands fail * Cannot access NetBox web interface **Solution:** **Web console access:** The admin console is available at `https://your-cluster-host-or-ip:30000/` in default installations. **Command-line access:** For NetBox Enterprise installer deployments, enter the cluster shell environment: ```shell /var/lib/embedded-cluster/bin/netbox-enterprise shell ``` This provides a shell with kubectl configured to interact with the embedded cluster. ### Manual edits to the NetBoxEnterprise resource are reverted [#manual-edits-to-the-netboxenterprise-resource-are-reverted] **Symptoms:** * Changes made with `kubectl edit netboxenterprises` disappear after the next deploy * Configuration reverts to what the Admin Console shows **Cause:** The Admin Console is the source of truth for the `NetBoxEnterprise` custom resource. Editing the resource directly (for example, `kubectl edit netboxenterprises`) is not supported. The console overwrites the resource on the next deploy, so manual edits are lost. **Solution:** Make configuration changes through the Admin Console at `https://your-cluster-host-or-ip:30000/`, then **Save Config** and deploy. For settings the console does not expose (such as `valueFrom: secretKeyRef` environment variables), apply them through a Helm deployment instead, where the custom resource is authoritative. ### Redis Out-Of-Memory Restart Loop [#redis-out-of-memory-restart-loop] **Symptoms:** * The Redis pod restarts repeatedly under memory pressure * Redis is killed with OOM and re-fills memory on startup, then is killed again * The ingester rejects ingest with `ResourceExhausted` **Cause:** With append-only file (AOF) persistence enabled, a restarted Redis replays its entire write log, re-filling memory immediately and triggering another OOM kill. Without a memory cap, Redis grows until the pod hits its limit. **Solution:** For operator-managed (built-in) Redis, tune memory through the Admin Console under **Config > Show Advanced Settings > Redis Tuning**, or through the custom resource on Helm: 1. Set **AOF (Append-Only File) Mode** to **Disabled**. A restart then loads the compact RDB snapshot instead of replaying every write, so a crashed Redis does not re-fill memory on startup. The NetBox cache is preserved; in-flight Diode stream events are lost on crash. 2. Set **Redis Memory Cap** to a percentage of the pod limit (for example, `80`) and choose an eviction policy such as `allkeys-lru`. This requires a Redis memory limit, set via the Redis Resources preset. The equivalent custom-resource fields (`redis.aofEnabled`, `redis.maxMemoryPercent`, `redis.maxMemoryPolicy`) are documented in [Redis Configuration](/enterprise/helm/configuration/redis/#memory-and-persistence-tuning). Changing these settings rolls the Redis pod once on the next deploy to apply the new configuration. This is expected. ### Deployment Stuck in "Updating" or "Reconciling" [#deployment-stuck-in-updating-or-reconciling] **Symptoms:** * The `NetBoxEnterprise` resource reports `Reconciling` or `ComponentsNotReady` for an extended period * `kubectl get netboxenterprises` shows `READY` as `false` after a config change **Cause:** The operator processes each spec change and waits for every component to become ready before marking the cluster ready. During an image upgrade, migration, or pod roll, the resource stays in a progressing state until the rollout converges. **Solution:** Check whether the operator has caught up with the latest spec change: ```shell kubectl -n netbox get netboxenterprise netbox \ -o jsonpath='gen={.metadata.generation} observed={.status.observedGeneration} converged={.status.convergedGeneration}' ``` * When `convergedGeneration` equals `metadata.generation`, the current configuration has fully rolled out. * When `observedGeneration` is behind `metadata.generation`, the operator has not yet processed the latest change. Inspect component status and operator events to find what is still progressing: ```shell kubectl -n netbox get netboxenterprise netbox -o jsonpath='{.status.components}' | jq kubectl -n netbox get events --field-selector source=nbe-operator ``` See [Status & Conditions](/enterprise/helm/api/status-conditions/) for the full field and condition reference. If a component stays unready, check that pod's logs for the underlying failure. ### Gateway Logs `middleware does not exist` at Startup [#gateway-logs-middleware-does-not-exist-at-startup] On the bundled Traefik gateway, you may see a brief burst of errors like `middleware "-@kubernetescrd" does not exist` in the gateway pod logs each time it starts. This is expected: Traefik's Gateway API provider publishes routes a moment before its CRD provider finishes indexing the request body-size `Middleware`, so the reference is briefly unresolved before it self-heals (typically within a second). A short burst **at startup** is benign. Errors that **persist** past startup indicate the `traefik.io` Middleware CRD is missing. Install the `netbox-enterprise-crds-traefik` chart (Embedded Cluster installs it automatically). See [Prerequisites](/enterprise/helm/installation/prerequisites/). ## Command-Line Tools Reference [#command-line-tools-reference] NetBox Enterprise includes specialized tools for cluster management and diagnostics. All tools require cluster access as described in [Accessing Your Cluster](#accessing-your-cluster). ### kubectl [#kubectl] Standard Kubernetes CLI for interacting with cluster resources. Included with NetBox Enterprise installer. **Common operations:** ```shell # List all pods in the kotsadm namespace kubectl -n kotsadm get pods # View pod logs kubectl logs -n kotsadm # Describe a resource kubectl describe pod -n kotsadm # Get deployment details kubectl -n kotsadm get deployment -o yaml ``` ### preflight [#preflight] Validation tool that checks system requirements before deployment or upgrade. Included with NetBox Enterprise installer. **Manual installation:** ```shell curl https://krew.sh/preflight | bash ``` **Usage:** ```shell # Run preflight checks kubectl preflight # Example with NetBox Enterprise spec kubectl preflight /var/lib/embedded-cluster/support/host-preflight.yaml ``` Use preflight when planning upgrades or diagnosing installation issues. ### support-bundle [#support-bundle] Generates comprehensive diagnostic bundles containing logs, configuration, and cluster state. Included with NetBox Enterprise installer. **Manual installation:** ```shell curl https://krew.sh/support-bundle | bash ``` **Usage:** ```shell # Generate support bundle kubectl support-bundle --load-cluster-specs # Example with NetBox Enterprise spec kubectl support-bundle /var/lib/embedded-cluster/support/host-support-bundle.yaml --load-cluster-specs ``` The command creates a `.tar.gz` file in the current directory. Provide this file to NetBox Labs support when reporting issues. **Output options:** * Press `s` to save a summary text file * Press `q` to quit (bundle file is still created) ### k9s [#k9s] Optional terminal UI for managing and viewing cluster resources interactively. Not included with installer. **Installation:** See installation instructions at [k9scli.io](https://k9scli.io/). **Usage:** ```shell # Launch k9s k9s ``` K9s provides a visual interface for common kubectl operations and real-time resource monitoring. ## Advanced Configuration Reference [#advanced-configuration-reference] The Admin Console exposes additional configuration options under **Show Advanced Settings**. The following settings are available for specialized deployment scenarios. ### Custom Object Types Limit [#custom-object-types-limit] The `max_custom_object_types` setting limits the number of custom object types that can be created using the netboxlabs-netbox-custom-objects plugin. This protects database infrastructure from excessive schema modifications. | Setting | Default | Description | | ------------------------- | ------- | ---------------------------------------------------------------- | | `max_custom_object_types` | 50 | Maximum number of custom object types. Set to `0` for unlimited. | This option is visible in the Admin Console under **Config > Show Advanced Settings**. ### Extra CA Certificates [#extra-ca-certificates] The `extra_ca_certificates` setting allows uploading a custom CA certificate bundle. Use this when NetBox Enterprise needs to trust certificates signed by an internal or private certificate authority. **Common scenarios requiring custom CA certificates:** * Self-signed certificates on external PostgreSQL * Self-signed certificates on external Redis with TLS * Self-signed certificates on S3-compatible storage (e.g., MinIO) * LDAPS connections using an internal CA To configure: 1. Access the Admin Console 2. Navigate to **Config > Show Advanced Settings** 3. Upload your CA certificate bundle in the **Extra CA Certificates** field 4. Click **Save Config** and deploy For MITM proxy environments, use the `--private-ca` installer flag instead. See [Proxy Environments](/enterprise/embedded-cluster/nbe-ec-requirements/#traditional-proxies) for details. ### Internal Proxy for NetBox Containers [#internal-proxy-for-netbox-containers] The `use_proxies_inside_netbox` setting configures proxy settings inside the NetBox containers themselves. By default, proxy configuration only applies to the cluster infrastructure. Enable this option when NetBox needs to route outbound requests (e.g., webhooks, external API calls, data source synchronization) through a proxy. This option is visible in the Admin Console under **Config > Show Advanced Settings** in the Gateway Configuration section. This is separate from the `--http-proxy` and `--https-proxy` installer flags, which configure cluster-level proxy settings. The `use_proxies_inside_netbox` option extends those proxy settings into the NetBox application containers. ## Related Documentation [#related-documentation] * [NetBox Enterprise Installation](/enterprise/embedded-cluster/nbe-ec-installation/) * [NetBox Enterprise Requirements](/enterprise/embedded-cluster/nbe-ec-requirements/) * [NetBox Enterprise Overview](/enterprise/nbe-overview/)