Over the last year at NetBox Labs, we’ve had hundreds of conversations with NetBox users, network automation engineers, and many others in our community. One of the most common needs all of them have is making it easier to get data into NetBox’s data model. Whether you’re merging legacy data into a newly deployed NetBox or updating network data in real time, there are real complexities tied to NetBox’s complex data model. NetBox’s data model is a superpower for getting network documentation and automation right, but getting data into it correctly can be challenging as a result. We’ve tackled this need head on, and today we’re excited to share an early preview of the upcoming NetBox ingestion service, codenamed Project Diode. This capability is designed not just to ease but to redefine how data is fed into NetBox at scale, transforming a complex undertaking into a straightforward task. Diode will deepen NetBox’s role as the linchpin of modern network automation architectures by dramatically lowering the bar to connect other data sources with NetBox’s data model.
In this blog post, we will explore the challenges that Diode addresses in some more depth, describe our innovative approach to getting data into NetBox’s model, and walk through a practical example of how Diode can be leveraged to improve your operations.
Common challenges with a network source of truth (such as NetBox)
A very common need with NetBox is consolidating all the disparate sources of network data and leveraging them effectively to keep the NetBox data model updated as the network evolves. A network source of truth’s effectiveness is tied directly to how closely it conforms to the network team’s intended or documented state of the network. Planning network changes or driving network automation using stale data can lead to unexpected, undesirable, or even disastrous results.
A key element of NetBox’s value is that it provides a structured, cohesive and internally consistent data model that can accurately represent the network intent. This enforces completeness and correctness throughout all phases of the network lifecycle, whether you’re designing, planning, building or operating your network, resulting in fewer errors and incidents, better network performance, and unlocking automation. This strictness – while critical – can also be a burden when working with fragmented or incomplete data found in pre-existing sources or specific-purpose systems (such as monitoring or device configuration tools), making the collection and capture of this data in NetBox an effort that requires a detailed understanding of the NetBox data model and its APIs.
Our solution: Diode, a new NetBox ingestion service
Diode is a new NetBox ingestion service that greatly simplifies and enhances the process to add and update network data in NetBox, ensuring your network source of truth is always accurate and can be trusted to power your network automation pipelines. Our guiding principle in designing Diode has been to make it as easy as possible to get data into NetBox, removing as much burden as possible from the user while shifting that effort to technology.
To achieve this, Diode sits in front of NetBox and provides an API purpose built for ingestion of complex network data. Diode eliminates the need to preprocess data to make it conform to the strict object hierarchy imposed by the NetBox data model. This allows data to be sent to NetBox in a more freeform manner, in blocks that are intuitive for network engineers (such as by device or by interface) with much of the related information treated as attributes or properties of these components of interest. Then, Diode takes care of the heavy lifting, automatically transforming the data to align it with NetBox’s structured and comprehensive data model. Diode can even create placeholder objects to compensate for missing information, which means even fragmented information about the network can be captured in NetBox.
Walkthrough: importing data with Diode
Enough discussion – what does Diode really do? Let’s walk through a very common real world example: importing inventory information into NetBox from a simple structured data source, such as a spreadsheet or CSV, JSON, or YAML file. For our walkthrough, let’s use a simple CSV text file that has some inventory information about our environment. Here’s what that might look like:
Diode comes with SDKs for sending data in both Python and Go. In this example, we’ll be using the Diode Python SDK and sending the data with gRPC/Protobuf to the new Diode ingestion service:
Let’s put together a Python script to parse the file content. We’ll first open our inventory file and initialize our entities list variable that will hold our data in preparation to sending it to the ingestion service:
Now, let’s loop through the rows in our inventory file and load the entities list with the data content of each row. Savvy NetBox users will notice that we aren’t breaking apart the data into different object types, but rather treating them like attributes (or properties) of each device.
Finally, we need to pass the entities list to the Diode client for processing by the ingestion service, which will automatically translate and store the data in the NetBox data model:
If we switch over to NetBox, we’ll now see that a number of objects were created based on the content of the inventory file. We now have 6 new devices, 5 device types, 4 sites, 3 manufacturers and 3 roles. All of these were created by simply sending the device information that was available in the inventory file. The Diode ingestion service automatically took care of verifying if related objects (such as sites and device types) already existed, and created new objects if they didn’t.
Comparing Diode to the NetBox REST API
First and foremost, Diode greatly reduces the amount of NetBox specific logic that needs to be embedded in external scripts and integrations that push data to NetBox, and simplifies how much preprocessing of the data needs to be done before it’s sent. Using the NetBox REST API, the inventory import in our example above would have required over 25 API calls to NetBox:
- 5 GETs to retrieve the lists of existing devices, manufacturers, device types, roles and sites to ensure they don’t already exist or to determine if they need to be updated
- 21 POSTs to create each entity that doesn’t already exist (or PUTs to update if they do), all in the correct order:
- Sites (4 API calls)
- Roles (3 API calls)
- Manufacturers (3 API calls)
- Device Types (5 API calls)
- Devices (6 API calls)
The NetBox REST API is extensive and enables a high level of granular control and interaction with the underlying NetBox data model. This can be extremely useful for certain specialized use cases, such as building network automation workflows. But for more generalized use cases such as network data ingestion, reducing the amount of NetBox data model logic that needs to be embedded in external scripts and integrations can drastically simplify these components, accelerate their development and reduce ongoing maintenance burden.
What’s next for Diode?
Diode, the new NetBox ingestion service, is currently an Experimental product as defined in our product development lifecycle. We’re putting finishing touches on the first production ready version of Diode, ahead of releasing it soon for Private Preview by NetBox Labs customers and community members. Ultimately Diode’s SDKs and ingestion service will be available to the open source community. Please contact us if you’d like to experience firsthand how Diode can transform how you integrate NetBox into your network automation stack.
And lastly, with Diode, we’re just getting started on a much broader initiative to make it easier to leverage NetBox as the heart of modern network automation. Stay tuned as we announce many more upcoming capabilities enabled by NetBox and Diode, including data reconciliation workflows to streamline and accelerate management of your network data.
