Network Test Automation with NetBox + pyATS + Genie

In a recent webinar we explored how to get started with network test automation using NetBox, pyATS and Genie, and this blog is a reference to accompany the webinar. In the webinar we covered:

Introduction to NetBox, pyATS and Genie: What each of these amazing tools can do and how they fit together in a modern network automation solution.

Using NetBox as a Dynamic Inventory Source for pyATS: How to integrate NetBox with pyATS for efficient testbed file creation.

Automated Network Testing: How to get started validating your network state by automating some simple pre and post change network testing.

There is also a Git repository which contains all the example code used, so you can follow along on your own laptop.

Benefits of Automated Network Testing

There are so many benefits to be realised as network engineers by adopting automation, and here are just few related specifically to automated network testing:

Easy Pre + Post change validation

Quicker Mean Time to Innocence (prove it’s not the network)

Stateful validation of device operational status

Build a safety-net of scalable, data-driven and reusable tests

Find out what changed when there is an outage

It’s easy to get started and is a lot of fun!

It will make your life easier and save you time (automate the boring stuff)

It will enhance your career (become a Network Engineering rock star!)

Automation Requires a Network Source of Truth (NSoT)

To perform network automation at any sort of scale requires an NSOT, which is a representation of the intended configuration and state of the devices, connections, and services of the network. It captures intended state – different than operational state and uses a structured, cohesive, comprehensive data model for network intent, which forces completeness and correctness in design, planning, configuration management, operations.

NetBox – Your Network Source of Truth

NetBox is your system of record for network management & automation – it allows you to eliminate spreadsheets, break down DCIM & IPAM data silos, and accelerate network automation with a REST API, GraphQL interface, Webhooks, Event Streams etc.

A Modern Network Automation Architecture

Before diving into pyATS, let’s just start by exploring what a modern network automation architecture looks like:

Starting in the center, NetBox as the network source of truth contains the intended state of the network and other tools will interact with the source of truth to either change or extract the Intended state.

When a change to the intended state is made either by an operator directly in NetBox or by another tool such as a CMDB or ticketing system, this will then initiate an update to the network automation tools.

Network automation tools (such as pyATS) query NetBox for the intended state of the network as stored in the NetBox database – this can be for both inventory data and configuration state data. The automation tools then deploy the intended state out to the network devices themselves or run tests against them.

Monitoring, Assurance and Testing tools (such as pyATS) then compare the actual observed state of the network to the intended state of the network as defined in NetBox.

When a delta between states is detected, then action can be taken, either in the form of an alert to a messaging platform so an engineer can take action or directly to automation tooling to remediate and restore the network back to the intended state.

pyATS – Python Automated Test Systems

pyATS is a Python3 based test automation infrastructure, which started as an internal Cisco Engineering tool, and now millions of CI/CD, sanity, regression, scale, HA, and solution tests are run on a monthly basis at Cisco with pyATS. It is now available externally (via Python PyPI), and is compatible with existing tooling and infrastructure. It is suitable for agile, rapid development iterations and can be used in a wide variety of testing scenarios. Read more at https://developer.cisco.com/docs/pyats/api/.

There are many layers to the overall testing framework. pyATS is the foundation for testing where inventories and topologies are defined. Genie sits above it parsing the output of show commands run against network devices into structured data. The top layer includes business logic and integrations with other systems, plus XPRESSO which is a full blown web application that drives your testing and reporting.

The pyATS Testbed File

The pyATS testbed file defines the devices, connection details and topology of the network under test – think Ansible or Nornir inventory files. Here is an example testbed file with a single device in it:

Testbed files can be created manually or automatically and for NetBox integration we create them automatically via the NetBox API using the device data stored in NetBox – so there is no need to manually created your testbed files! One thing to note is that the device name in the testbed yaml file MUST match the configured hostname of the device – I got burned by this and hopefully this warning will prevent the same happening to you!

pyATS and NetBox

With the pyATS testbed file built dynamically from the NetBox device data, the pyATS host can now run automated network tests against the network devices themselves.

Set Up and Installation

Follow these steps to get everything set up:

Clone the Git repo and change into the netbox-pyats-webinar directory:

2. Create and activate Python 3 virtual environment:

3. Install PyATS:

As per the official documentation, there are a options to perform a minimal installation (option 1) or a full installation (opton 2).

Option 1 Minimal install that includes the Genie library and that allows you to use the interactive testbed creation command to create your testbed files from NetBox:

Option 2 Full installation that includes all packages and libraries:

pip install pyats[full]

Note If you are using Zsh on a Mac then you need to quote the install string (this had me stuck for a long time until I figured it out!)

pip install "pyats[full]"

Note if you plan to run the example script ospf_neighbor_table.py then you will also need to install the prettytable library with pip install prettytable or you can simply run the command pip install -r requirements.txt to install this along with pyATS, Genie and the contrib library.

There is also a PyATS Docker Image. This command will pull down the container if you don’t have it locally and drop you into a Bash shell:

docker run -it ciscotestautomation/pyats:latest /bin/bash

Lab Network

Our lab network consists of 2 x Cisco CSR100V routers and they are documented in NetBox under the Site PyATS Webinar and are directly connected to each other over port GigabitEthernet2 on the 192.168.1.0/30 subnet. They are both running OSPF, and you can find the the configuration for this in the initial_device_configs.md file:

Generating The Testbed file Dynamically from NetBox Inventory

Option 1 Use the pyats create testbed netbox command to build your testbed file. Note that where a value is prefixed with os.getenv or %ENV then these values are being pulled in from the local environment variables that you need to set with the export command eg. export NETBOX_URL=https://example.cloud.netboxapp.com/, export DEF_PYATS_USER=admin:

In this example we are generating a testbed file called testbed.yaml and filtering NetBox by the site name pyats-webinar. When you hit enter the output will look like this:

Option 2 Run the generate_testbed_file.py Python script. Note that where a value is prefixed with os.getenv or %ENV then these values are being pulled in from the local environment variables that you need to set with the export command eg. export NETBOX_URL=https://example.cloud.netboxapp.com/, export DEF_PYATS_USER=admin.

In this script we are generating a testbed file called testbed.yaml and filtering NetBox by the site name pyats-webinar, but you could just as easily filter on other fields as in the examples commented out:

The resulting testbed file produced by either option will look something like this, depending on your network. Note that as we included the --topology switch the testbed file output includes the interfaces and connections from NetBox also:

Structured Data Vs Unstructured Data

When you run a command at the CLI of a network device, you get unstructured data back as the response, which is just a blob of text:

This is great for humans, as we can read this, but a computer cannot understand this data. Also if the next version of the OS you are using makes a change to the way that the output is formatted then you will have a problem and have to re-write your scripts to handle this. This is where the Genie Parser comes into play as it will parse the output into structured data. The data is then represented using key/value pairs in JSON format that can be used by a computer:

Running Your First Tests With The Genie CLI

Genie Parsing

There are parsers available for a large number of network OS’s (not just Cisco), and you can view the complete list here In this example we will run the genie parse command to parse the output of the show ip ospf neighbor command and limit it to just device CSR1:

This will output the parsed data to your terminal, but if you want to save the output just append the directory you wish to save it to with the --output switch. This will save both the _console file (the unstructured data) and the _parsed file (the structured data), along with the connection_ log (the raw output of the full connection process) into the chosen directory:

Genie Learning

In this example we will run the genie learn command to learn all about routing and ospf for both devices in our testbed file. We will also save the output into directory called pre-change. Genie will automagically create directories for us if they don’t already exist.

For reference if you run the genie learn all command then for IOS-XE devices the list of features learned is:

acl, arp, bgp, device, dot1x, eigrp, fdb, hsrp, igmp, interface, isis, lag, lisp, lldp, mcast, mld, msdp, nd, ntp, ospf, pim, platform, prefix_list, rip, route_policy, routing, static_routing, stp, terminal, utils, vlan, vrf, vxlan, config

Making a Change to The Network

Let’s make a small change to the CSR2 router in our test network, and remove the network 2.2.2.2 0.0.0.0 area 0 statement from the OSPF configuration:

Next lets re-learn routing and ospf for both devices in our testbed file. We will also save the output into directory called post-change:

Genie Diff

Now let’s use the genie diff command to tell us what changed in our network, and we will store the output in a directory called routing-diff:

There are differences in the outputs of both the routing_iosxe_CSR1_ops.txt and ospf_iosxe_CSR2_ops.txt files and we can view the contents of those files to see what has changed:

As we can see, CSR1 has lost the the route to the 2.2.2.2/32 prefix and on CSR2 the interface lo0 has been removed from OSPF area 0.

pyATS and Python

As you would expect, it is also very easy to work with pyATS and python. Type python in your terminal, and then start by importing the load module from the genie.testbed library and load the testbed file:

Then connect to the CSR1 device:

Then parse the command show ip interface brief and print the output nicely using the pprint library:

Lastly, close the connection to the device:

device.disconnect()

As you can imagine, the possibilities are endless and you can see an example of a script that gets and prints OSPF Neighbor table for a given device in the file ospf_neighbor_table.py. Run the script and you will get a nicely formatted table of the OSPF neighbors for CRS1:

pyATS Test Scripts and Jobs

So far we have only scratched the surface of what you can do with pyATS, and test scripts will take your automated testing to the next level. Test scripts are written and executed with the AEtest Python package

I have included an example test script called connectivity_check.py, which is adapted from the example script in the official documentation. This script tests that both CSR1 and CSR2 can ping each other’s loopback addresses. Test scripts are structured Python files that contain/describe the testing you want to do. A clean, elegant test script is scalable, and generates easy-to-read test results and logs.

To run the example test script, you must also specify the testbed file that is to be used, for example:

python connectivity_check.py --testbed testbed.yaml

Once the test script has been run then the results are displayed in the output:

pyATS Jobs take test scripts to another level and allow the execution of test scripts as tasks. This is beyond the scope of this webinar/repo but you can read all about pyATS jobs in the official docs

If you’ve got this far, first of all well done and thanks for reading! If you have any feedback or suggestions then please feel free raise a Github issue in the repo.

Learn More in our On-Demand Webinar

Hopefully you now have a good understanding of how pyATS and NetBox work together seamlessly to power automated network testing. If you would like to see everything I have covered in this blog in action, simply the register for the on-demand webinar. Lastly, don’t forget to check out the NetBox Learning Git Repo for all the code used in this blog, as well as other great NetBox Learning resources.