Like many in the network engineering field, Ian Chilton’s path to network engineering was unique. In episode three of NetBox Heroes, Chilton shares with Host Kris Beevers that his journey was different from most traditional network engineers in that he spent much of his career as a software engineer, first at British Telecom, then for ten years at web development agencies. While his work focused on back-end tech for small firms, he was also responsible for managing servers to host the sites and setting up the networks and internet in the agencies’ offices.
The web development roles led to tinkering with servers outside of work, where he gravitated towards Linux admin duties, operations, production environments, and inevitably, fixing things when they break in the middle of the night. (The latter taught him important soft skills like staying calm under pressure when everyone else is panicking.) Despite this hands-on experience, Chilton often compares his background to those of his peers, who spent their whole career in networking, and feels a bit of imposter syndrome.
“I work with networks and switches every day,” Chilton says. But in some ways, “I struggle to call myself a network engineer because I don’t have that background.”
Host Kris Beevers points out that lots of IT professionals find their way to network engineering much the same way.
“In a lot of ways, all roads lead to the network, especially when you’re working on fundamental infrastructure,” says Kris Beevers. “You find yourself really passionate about the systems that connect us all and make the modern network work.”
Following a short stint at a startup, Chilton joined a hosting company that sold virtual machines and dedicated servers to customers as a software engineer. That role gave him exposure to data center work and added responsibility around a cloud hosting virtual machine platform. He quickly was promoted to manage an operations team in charge of that platform, as well as the rest of the company’s internal tools and customer facing portal. After the company was acquired, his role changed and he sought a new position. He joined LONAP, a major internet exchange in London that connects large networks, as a DevOps engineer in 2019.
“DevOps, I feel like it’s almost a bit of a modern buzzword, but it really accurately describes what I do and what my role is,” says Chilton. “I do software engineering, software development, and I write code. But then I’m also heavily involved in the operational side, implementing monitoring, dealing with alerts, debugging problems, and dealing with any outages.”
Chilton’s role also involves applying configuration changes, automation, maintenance, software upgrades, and setting up and managing servers with other services that LONAP uses to manage the network. It includes data center work, such as moving switches around, links, fiber, optical networking, and hardware that both runs and supports the running of the exchange.
You’ll Learn By Doing, Not By Reading
When asked what advice he has for those coming up in the field, Chilton is quick to encourage practical, hands-on experience.
“Find something you’re interested in, find something you’re passionate about and almost get yourself into a position where you have to do that,” says Chilton. “Get yourself in at the deep end.”
Chilton notes that books will only take you so far, but the best training he’s had has been learning on the job, when he needed to solve something and figure it out along the way. Finding something you enjoy will motivate you to learn more and make it happen. New network engineers don’t have to worry about knowing it all.
“None of us know it all either. We make it up day by day. We go and look things up when we need to look them up,” says Chilton. “There’s things we’re used to knowing and can’t remember and we go and look it up. You don’t have to know it all to do this kind of thing. You just have to be technically minded and you work it out as you go along.”
In his current role at LONAP, which is one of the largest internet exchange points in the UK, Chilton is part of the team that oversees eight racks in seven of the major London data centers, with more than 20 switches in production. Traffic peaks above 1 terabit per second. The not-for-profit neutral internet exchange has 250 members who are all equal stakeholders, including Google, BBC. Microsoft, Netflix, Sky, Vodafone, eBay, and Twitter.
One of the biggest challenges in this role is the 24/7 nature of the internet itself. Chilton says, noting that it limits his ability to make changes live because unplugging something isn’t an option. LONAP has had to invest in tools and technologies to evolve the network without any downtime. Among other tools they use, one of Chilton’s colleagues at LONAP developed techniques for BGP Session Culling with peers for this reason.
“Generally, we have to engineer around these things to try and make it as little disruption as possible,” says Chilton. “So basically we get paid to not break the internet.
LONAP is a non-profit, which introduces additional challenges. There are limited budgets, but networking equipment is very expensive, so everyone at LONAP aims to operate as cost-effectively as possible. They also aim to drop prices every year in line with commoditization of internet traffic, and LONAP is wary of spending more money than needed while also ensuring they can manage large traffic spikes.
What Automation Looks LIke at LONAP
Chilton and his team are small but mighty with three technical networking professionals that run the network and act as customer support. In this scenario, it’s critical for the team to automate as much as possible and make management tasks as easy as possible and as quick as possible to do.
Until 2020, the network was configured manually. Switches were provisioned manually by copying and pasting from a template that the team prepared. Member parts were then configured one by one by hand. Chilton was brought in to templatize and automate the network, without disrupting any traffic in production, which has been a success and remains an ongoing effort.
“Nowadays, we still log into switches to look at things, check things. But any configuration changes, other than if it’s specific testing or whatever, are made by running the automation. And that actually applies to the switches.”
Automation has made network changes much more consistent, Chilton notes. It takes away the human element, eliminating manual errors that can be common when a person is involved.
Initially Chilton used SaltStack for managing servers and all the VMs. On the networking side, he implemented a lot of the logic in Jinaj2 templates. This setup worked very well for several years and led to a number of learnings. Recently, Chilton has begun working on in-house tools to render templates and drive configuration changes, after realizing that while SaltStack is a deeply powerful automation tool, it’s complexity introduces unnecessary risks for LONAP’s straightforward automation use case.
Host Kris Beevers notes there is an important lesson for the networking community in LONAP’s automation evolution. No team implements automation and then is done completely, forever.
“The network is almost a living beast and so are your automation goals and the set of tools and the problems or challenges you’re solving around it,” says Beevers. “You’re never done.”
NetBox at LONAP
While at the hosting company, Chilton used in-house software to organize data center racks, track storage in the racks, and manage servers. From this experience, when he learned about NetBox he immediately saw its value. When Chilton joined LONAP in 2019, he was pleased to see them using NetBox.
Currently LONAP uses NetBox for inventory management, as well as using it for VLAN and IP address management. They have more than 800 devices in NetBox, with half of them being optics.
“We have eight full racks of equipment. We need to know, you know, all the time, what is in those racks, where are those devices?” says Chilton. “We might be getting new hardware and we need to know, where can we fit it into that rack? What do we need to move around? And so NetBox really helpfully lets us manage that and lets us visualize it.”
LONAP uses IXP Manager, an open source tool written by their colleagues at INIX, one of the internet exchanges in Ireland. They use the tool as a customer database, storing contact and billing details of all of LONAP’s members and their associated ports. NetBox manages all hardware, optics, and shelves as their Network Source of Truth.
The team also uses lots of scripts, which talk to NetBox via the API, pull inventory data and then feed into their monitoring. They also use the API to put data into NetBox. So for example rather than manually registering a new delivery of optics, the LONAP team deploys the new gear and uses tools they’ve built to query those switches, discover newly deployed, and register them in NetBox with serial numbers, model numbers, manufacturers, etc.
Chilton’s example is very representative of the most common, modern network automation architectures NetBox Labs sees, Beevers mentions. Teams use tools to do discovery and identify differences between the operational state of the network and the intended state as represented in NetBox. Then they use automation tools to converge the operational state and the intended state – much like LONAP’s optics discovery and registration. This is a great example of successful network automation, Beevers notes – automation needn’t be massive, comprehensive, or AI driven to deliver value.
“That’s not really what network automation is. It’s about picking off these repetitive tasks that are otherwise laborious, like sitting on the data center floor and typing in all the details of all the optics you’ve just racked,” says Beevers. “It’s a waste of time and there’s going to be a typo in there and it’s going to be brutal. That’s what you are solving for with this automation.”
Going back to NetBox, Chilton points to NetBox’s reliability as something he really appreciates. He gives major thanks to the developers, and those in the community and industry working on NetBox, for this very useful tool.
“Whether you’re running an internet exchange or a smaller network, or you’re setting up a network in a field, NetBox is a great tool,” says Chilton. “And if that wasn’t around, it would be a lot of development to make something equivalent. So the fact that you guys and everyone is developing this thing just to be used as open source is absolutely brilliant. So thank you very much for all your work.”
I couldn’t agree more. To listen to the full NetBox Heroes episode, visit https://youtu.be/G0J4K0D5_N8.
