The following is a guest blog from Jason Davis, Distinguished Engineer, Cisco DevNet.
As many readers know, Cisco has run its user conference, CiscoLive, since 1998. It exists to provide training, new product and service launches, collaboration, and a twist of fun.
To showcase much of its products, Cisco brings a lot of network infrastructure onsite. The CiscoLive Network Operations Center (NOC) team is empowered to design, implement and monitor the event network. We use our own commercial products, some from partners and a mix of open-source solutions to manage and monitor the event network.
The event is quickly set-up, executed and torn down. We do not have long-term or permanent equipment onsite at the venues we use, so tracking and configuring the thousands of assets we bring – wireless access points, network switches, routers, compute blades and storage nodes, is imperative to a world-class event experience. As somewhat routine in some IT environments, using spreadsheets to track the type, status and whereabouts of assets is problematic and lacks the integration we strive for in achieving a programmable network, especially one as dynamic as CiscoLive.
Our use of NetBox has increased over the years, as it affords us strong asset tracking and extensive APIs for orchestration across our many tools. Our event team is comprised of dedicated staff in our Technology Experiences organization, volunteers from other parts of Cisco in sales, services and engineering, local IT specialists with Tater Networks and students/interns from our Cisco Network Academy program. The experience level of the team ranges from new to networking to a few that have decades of experience. Our tooling and processes must be intuitive and not require a lot of training or prior knowledge.
Not all our available assets are deployed for every event. Some events, like CiscoLive, are large and require a lot of equipment. Other events, like Partner Summit, are much smaller and don’t require as many assets. Over time we transitioned much of our inventory from spreadsheets into NetBox. Our use case is to update device statuses and have that also drive the active monitoring (or deletion) in our other tools by using Webhooks and API integration. We use custom device statuses like:
- In Storage
- In Transit
- Awaiting Deployment
- Deployed
- Operational
- Decommissioned
- Faulty
- Investigate
- Offline
- Unused
We knew the Cisco team would be comfortable interfacing with NetBox’s web UI. However, we believed the external helpers would be more comfortable with a mobile app or something tablet friendly as they assisted us in deployment. We needed those assistants to understand what assets were under their responsibility and which were deployed and ready for service. To that end we built a mobile web-app using Python scripting and the PyWebIO framework. Since we were working with Tater Networks, providing a ‘hand off’ of equipment and because our lead programmer [me] is a bit of a jokester we called the mobile app ‘Hot Potato’.
The Cisco NOC asset manager would use the NetBox web UI to select a device or devices and adjust the status to ‘Awaiting Deployment’. That would trigger the device(s) to show up on a list of available assets for the deployment personnel to pick up and deploy.
The deployment personnel would use the Hot Potato mobile webapp to see what devices were ready for them and they would take responsibility for their deployment. When the device was placed in the venue, connected to power and networking, they would take a picture of the asset’s placement, mark the status as ‘Deployed’ and add an optional deployment comment. We learned over the years that the person deploying the asset may not be the same one reclaiming it. If the device was tucked under a stage, behind a plant or in a closet then we might have a problem finding it later. Taking pictures allows us to understand where the asset is with contextual clues of placement in a room. The image shows up in NetBox and on the Hot Potato webapp, as can be seen with this asset being deployed in a closet behind double-doors and a swing-up trap door.
The device change of status to ‘Deployed’ would trigger the addition of the device into our active monitoring tooling – availability, health checks, configuration compliance, performance and fault monitoring.
When the device is retrieved as an area is decommissioned or with the event closing, it is marked as ‘Decommissioned’ which triggers removing it from active monitoring. As the event closes less and less equipment is seen actively monitored on the dashboards. We eventually would know what assets may have been missed during pick up and could focus our efforts on recovery.
The extensive APIs in NetBox allow us to influence the status bi-directionally and gain better operational insights to what assets are deployed, what state they are in, and what work remains.
In my regular job at Cisco, I lead special projects in our DevNet team, an organization focused on evangelizing network programmability and APIs. To that end, some of our CiscoLive NOC project for importing devices into NetBox has been shared with the open-source community on GitHub at https://github.com/jasoncdavis/import2netbox
If you’re looking to get started with Cisco network programmability, check out DevNet at https://developer.cisco.com our Learning Labs at https://developer.cisco.com/learning/ or our Code Exchange site at https://developer.cisco.com/codeexchange/
