TurboBulk User Guide

Overview

TurboBulk is a high-performance bulk data API for NetBox that achieves massive throughput improvements over the REST API for large-scale data operations.

Data Formats

TurboBulk supports two data formats for bulk operations:

JSON Lines (JSONL) - Default

JSON Lines (JSONL/NDJSON) is the recommended default format for most users:

• Row-oriented: Each line is a complete JSON object
• Easy to create: Just JSON.stringify() each row
• Works everywhere: Any language with JSON support
• Compressed: Automatically gzipped for efficient uploads (.jsonl.gz)

Example JSONL file:

{"name": "site-1", "slug": "site-1", "status": "active"}
{"name": "site-2", "slug": "site-2", "status": "active"}
{"name": "site-3", "slug": "site-3", "status": "planned"}

Creating JSONL in Python:

import gzip
import json

sites = [
    {'name': 'site-1', 'slug': 'site-1', 'status': 'active'},
    {'name': 'site-2', 'slug': 'site-2', 'status': 'active'},
]

with gzip.open('sites.jsonl.gz', 'wt', encoding='utf-8') as f:
    for site in sites:
        f.write(json.dumps(site) + '\n')

Creating JSONL in JavaScript/Node:

const zlib = require('zlib');
const fs = require('fs');

const gzip = zlib.createGzip();
const output = fs.createWriteStream('sites.jsonl.gz');
gzip.pipe(output);

for (const site of sites) {
    gzip.write(JSON.stringify(site) + '\n');
}
gzip.end();

Parquet - High Performance

Apache Parquet is available for maximum performance with large datasets:

• Columnar: Excellent compression, fast for large datasets
• Typed: Strong schema enforcement
• Recommended for: 100K+ rows, repeated operations, maximum throughput

Creating Parquet in Python:

import pyarrow as pa
import pyarrow.parquet as pq

data = {
    'name': ['site-1', 'site-2', 'site-3'],
    'slug': ['site-1', 'site-2', 'site-3'],
    'status': ['active', 'active', 'planned'],
}

table = pa.table(data)
pq.write_table(table, 'sites.parquet')

Choosing a Format

Use JSONL when:

• You want the simplest integration
• Your data source is in any programming language
• You're loading moderate datasets (<100K rows)
• You want human-readable files for debugging

Use Parquet when:

• You're loading very large datasets (100K+ rows) frequently
• Maximum throughput is critical
• You already have PyArrow or similar library available
• You're doing repeated bulk operations

Format Auto-Detection

TurboBulk automatically detects the file format from:

1. File extension (.jsonl, .jsonl.gz, .ndjson, .parquet)
2. Content inspection

When to Use Bulk API vs REST API

When to use TurboBulk:

• Initial data population (>1K objects)
• Regular full syncs from external systems
• Data migration between NetBox instances
• Analytics/reporting exports

When to use the REST API:

• Interactive changes
• Operations on <1000 objects
• Operations requiring custom Django validators

What Gets Bypassed

The bulk API deliberately bypasses certain NetBox features for performance. Understanding these tradeoffs is critical for proper use.

Features NOT Applied During Bulk Operations

What Still Works

Save Hooks (apply_save_hooks)

When enabled (apply_save_hooks=True), TurboBulk applies bulk SQL fixups after the raw SQL merge to replicate Django model save() side-effects. This is off by default for maximum performance.

What save hooks fix:

• Custom field defaults: Populates empty custom fields with their configured defaults
• Location inheritance: Device inherits location from its Rack
• Airflow/platform inheritance: Device inherits airflow and default_platform from DeviceType (insert only)
• Unit normalizations: Computes _abs_weight (grams), _abs_length (meters), _abs_distance (meters) from human-readable values
• Scope cache fields: Populates _region, _site, _site_group, _location on Prefix, CircuitTermination
• CircuitTermination FK: Circuit's termination_a/termination_z set after insert
• Other: VLANGroup _total_vlan_ids, IPAddress dns_name lowercase, IPRange size, WirelessLink device cache

Performance: Save hooks use bulk SQL UPDATE statements, adding near-zero overhead (<100ms regardless of batch size) for all core NetBox models. For plugin models with custom save() overrides not covered by the SQL fixups, it falls back to per-object save() calls.

result = client.load(
    'dcim.device',
    'devices.jsonl.gz',
    apply_save_hooks=True,  # Enable SQL fixups for computed fields
)

Validation Modes

TurboBulk provides flexible validation to balance speed and thoroughness. Understanding what each mode validates is important for maintaining data integrity.

How TurboBulk Validation Differs from REST API

When you create or update objects through NetBox's REST API, data passes through multiple validation layers:

1. Field-level validation - Type checking, format constraints
2. Cross-field validation - Fields that depend on each other
3. Model business logic - NetBox's clean() methods that enforce rules like "device must be in the same site as its rack"
4. Custom validators - User-defined validation rules
5. Database constraints - Foreign keys, unique constraints

TurboBulk bypasses some of these layers by default for performance. You can control this tradeoff with the validation_mode parameter.

Available Modes

Mode: none

What runs:

• PostgreSQL constraints: foreign keys, unique constraints, NOT NULL

What's skipped:

• All application-level validation
• Model business logic

Use when:

• Data comes from a trusted source (another NetBox instance, validated CMDB)
• You've pre-validated data externally
• Maximum speed is critical

Risks:

• Data that satisfies database constraints but violates business rules will be accepted
• Example: A device assigned to a rack in a different site (DB allows it, but it's logically invalid)

Mode: auto (Default)

What runs:

• All database constraints
• IP address hierarchy validation (parent prefix exists, VRF matches)
• Prefix hierarchy validation (parent prefix exists, VRF matches)

What's skipped:

• Django model clean() methods
• Custom validators

Use when:

• Normal bulk operations
• IP address management workflows
• Data from reasonably trusted sources

Risks:

• Model-specific business logic is not checked
• Complex models (cables, devices with rack positions) may have invalid configurations

Mode: full

What runs:

• All database constraints
• Django's full_clean() method on each object
• Model-specific business logic (e.g., Device.clean(), Cable.clean())

What's skipped:

• Custom validators configured in PLUGINS_CONFIG
• DRF serializer validation

Use when:

• Data comes from untrusted or unknown sources
• Complex models with important business rules
• Data integrity is more important than speed

Performance note: Full validation is substantially slower because it runs Python validation code for each row instead of using bulk SQL operations. For large datasets, consider validating a sample first, or using the REST API for complex models.

Models That Benefit from Full Validation

These models have significant business logic in their clean() methods:

What's Always Enforced

Regardless of validation mode, these database constraints are always enforced:

• Foreign key references - Referenced objects must exist
• Unique constraints - No duplicate values in unique fields
• NOT NULL constraints - Required fields must have values
• Check constraints - Database-level value constraints

If any of these fail, the entire operation rolls back with no partial commits.

Examples

Full validation for cables:

from turbobulk_client import TurboBulkClient

client = TurboBulkClient()
result = client.load(
    'dcim.cable',
    'cables.jsonl.gz',
    validation_mode='full'  # Important for cables
)

No validation for trusted migration data:

result = client.load(
    'dcim.site',
    'sites.jsonl.gz',
    validation_mode='none'  # Data already validated
)

Using curl:

# Full validation mode
curl -X POST -H "Authorization: Bearer $TOKEN" \
  -F "model=dcim.cable" \
  -F "mode=insert" \
  -F "validation_mode=full" \
  -F "file=@cables.jsonl.gz" \
  "https://your-instance.cloud.netboxapp.com/api/plugins/turbobulk/load/"

Best Practices

1. Know your data source

   • Trusted source (another NetBox, validated CMDB) → auto or none
   • Untrusted source (user spreadsheets, unknown systems) → full

2. Use full for complex models

   • Cables, devices with rack positions, interfaces with LAG membership

3. Validate before committing large datasets

   # Test with a small sample first
   result = client.load('dcim.device', 'sample.jsonl.gz', validation_mode='full')

4. Consider the REST API for small batches

   • Under 1,000 objects with complex validation needs
   • The REST API provides the most thorough validation

5. Custom validators are not run

   • If you rely on custom validators in your NetBox configuration, those rules are not enforced by TurboBulk
   • Consider pre-validating data or using the REST API for objects subject to custom validation rules

Foreign Key Requirements

Foreign keys must be provided as integer primary keys:

import gzip
import json

# CORRECT: FK as integer PK
device = {
    'name': 'device-1',
    'site_id': 123,           # Site PK
    'device_type_id': 456,    # DeviceType PK
    'role_id': 789,           # Role PK
    'status': 'active',
}

with gzip.open('devices.jsonl.gz', 'wt') as f:
    f.write(json.dumps(device) + '\n')

DO NOT provide nested objects or attribute lookups:

# WRONG: This will not work
device = {
    'name': 'device-1',
    'site': {'name': 'NYC-DC1'},  # WRONG!
}

Recommended Client Workflow

1. Export reference data first:

   sites = client.export('dcim.site')

2. Build PK mapping tables:

   import json
   import gzip
   
   with gzip.open(sites['path'], 'rt') as f:
       site_name_to_pk = {}
       for line in f:
           row = json.loads(line)
           site_name_to_pk[row['name']] = row['id']

3. Transform source data to use PKs:

   source_data['site_id'] = site_name_to_pk[source_data['site_name']]

4. Submit bulk load with resolved PKs

Post-Operation Hooks

TurboBulk provides post-operation hooks to handle necessary updates after bulk operations.

Available Hooks

Configuring Hooks

Using Python client:

from turbobulk_client import TurboBulkClient

client = TurboBulkClient()
result = client.load(
    'dcim.device',
    'devices.jsonl.gz',
    post_hooks={
        'fix_denormalized': True,
        'rebuild_search_index': True,
        'fix_counters': True,
        'rebuild_cable_paths': False,  # Skip for non-cable operations
    }
)

When to Disable Hooks

Operational Runbooks

Initial Data Load Runbook

Goal: Load initial data from an external system into a fresh NetBox instance.

Steps:

1. Export reference data (if migrating from another NetBox)

   client = TurboBulkClient()
   
   # Export sites, manufacturers, device types first
   client.export('dcim.site')
   client.export('dcim.manufacturer')
   client.export('dcim.devicetype')

2. Build PK mapping tables

   • Map source system IDs to NetBox PKs
   • Create lookup dictionaries for FK resolution

3. Transform source data to use PKs

   • Replace string/name references with integer PKs
   • Ensure all required fields are present

4. Load in dependency order

   a. Sites, Regions, Locations
   b. Manufacturers, Device Types, Device Roles
   c. Racks
   d. Devices
   e. Interfaces
   f. IP Addresses
   g. Cables

5. Run post-load validation

   • Verify counts match source
   • Spot-check a sample of devices
   • Verify FK relationships are correct

Ongoing Sync Runbook

Goal: Synchronize data from an external CMDB on a regular schedule.

Steps:

1. Export current state

   result = client.export(
       'dcim.device',
       fields=['id', 'name', 'site_id', 'status']
   )

2. Identify changes

   • Compare exported data with source system
   • Categorize: inserts, updates, deletes

3. Submit bulk operations

   • Upserts for inserts + updates
   • Deletes for removed records

4. Verify job success

   # The client automatically waits for completion
   result = client.load('dcim.device', 'devices.jsonl.gz', mode='upsert')
   print(f"Rows affected: {result['data']['rows_affected']}")

5. Spot-check critical changes

Troubleshooting Guide

Platform Configuration

NetBox Cloud and NetBox Enterprise come with TurboBulk pre-configured with optimized settings:

• Write APIs: Disabled by default (enable_writes: False). When disabled, load and delete endpoints return 403 while export and read endpoints remain available. Set enable_writes: True to enable write operations.
• Maximum upload size: 1GB per file
• Job timeout: 1 hour maximum
• Export caching: Enabled

If you need adjustments to these limits for your use case, contact NetBox Labs support.

Security Considerations

• All operations require authentication via NetBox token
• Standard NetBox model permissions are enforced
• Bulk add requires add permission
• Bulk upsert requires add + change permissions
• Bulk delete requires delete permission
• Bulk export requires view permission
• File uploads are validated for format and size

Changelog Generation

TurboBulk generates ObjectChange records by default, providing full audit trail support in NetBox's changelog system.

Enabling/Disabling Changelogs

# Default - changelogs enabled
result = client.load('dcim.device', 'devices.jsonl.gz')

# Disable for performance-critical operations
result = client.load('dcim.device', 'devices.jsonl.gz', create_changelogs=False)

When to disable changelogs:

• Initial data migrations where audit trail is not required
• Large imports (>100K rows) where changelog table growth is a concern
• Ephemeral or test data
• Performance-critical operations

When to keep changelogs enabled:

• Production data changes that need auditing
• Compliance requirements for change tracking
• Troubleshooting - having before/after data helps debug issues

Event Dispatch

TurboBulk dispatches events (webhooks, event rules) asynchronously after bulk operations complete.

Configuration

# Default - events dispatched
result = client.load('dcim.device', 'devices.jsonl.gz')

# Disable for initial data loads
result = client.load('dcim.device', 'devices.jsonl.gz', dispatch_events=False)

When to disable events:

• Initial data migrations where downstream systems don't need notifications
• Large imports where event volume would overwhelm consumers
• Test data loads

Export Caching

TurboBulk caches export results to improve performance for repeated requests.

How It Works

1. Client requests an export
2. If data hasn't changed since last export → returns cached file immediately
3. Otherwise → creates new export job

Cache-Aware Workflow

# Check if data changed without downloading
result = client.export('dcim.device', check_cache_only=True)

if result.get('cached'):
    print("Data unchanged, using local file")
else:
    # Data changed, get fresh export
    result = client.export('dcim.device')

Force Fresh Export

# Bypass cache
result = client.export('dcim.device', force_refresh=True)

NetBox Branching Integration

TurboBulk integrates with NetBox Branching for branch-aware bulk operations.

Basic Usage

# Load data into a branch
result = client.load(
    'dcim.device',
    'devices.jsonl.gz',
    branch='my-feature-branch'
)

Workflow

1. Create a branch (via NetBox UI or API)
2. Wait for branch to be ready (status becomes ready)
3. Bulk load data to the branch
4. Review changes in NetBox Branch UI
5. Merge the branch when satisfied

For more detailed branching documentation, see Branching Documentation.