IT Team Modifies Open Source NetBox to Help Manage Hybrid Multicloud

Making sense of hybrid multicloud can seem maddening at times. Every cloud provider does things just a bit differently. System admins must navigate a maze of configuration options to account for all the assets in their IT architecture. Individual IT pros tend to specialize in specific cloud platforms, creating knowledge silos that add yet more complexity.

These and many more variables make it tough to answer two questions that bedevil multicloud system admins: What are my cloud assets and which cloud provider hosts them?

Danko Trajkovic is a manager of network operations and Eric Pearce is IT systems architect at Nutanix. They get asked most often: In which cloud account and VPC is this asset part of, and who do I contact?

The two work for a company that pioneered hyperconverged infrastructure (HCI) and a software platform to run hybrid multicloud IT operations. Their team is well-known for getting the business to run on its own software – following their “drink our own champagne” initiative. In that environment, Trajkovic, Pearce and a small band of others teamed up on a recent project to more quickly and easily answer common questions they get about their hybrid multicloud assets. 

They modified the open-source software NetBox to query a database and see a column of servers, instances (VMs) and other cloud assets on a single browser screen. The screen also shows a column identifying each asset’s cloud provider (Amazon Web Services, Microsoft Azure or Google Cloud). 

Redacted screenshot shows the potential of NetBox data. This one page from a list of some 15,000 VMs, showing “Instance Ids” (VM identifiers), “Name” given to each VM, “VM type” (a per-cloud “model type” for the VM), “Primary IP” (the private IP of the VM within the cloud), “Public IP” (the public IP of the VM in cloud public IP space), “VPC ID” (VPC or equivalent), “Subnet” (private network within cloud where the VM resides), “Status” (running status of the VM), “OS” (estimated operating system running on the VM) and “Tags” (metadata for sorting, and searching).

NetBox normalizes all the data from the three cloud providers into a single, uniform pane, explained Pearce. The orange tagging indicates AWS, blue is Azure and green is GCP and you can see all of these presents in the example.  

“You no longer have to visit three separate cloud consoles and manually consolidate information to get a unified view of your cloud assets,” he said.

The VM type field is built from what standard models (or “instance types”) are available from each cloud provider, and any custom VM models from GCP are added to the database as they are encountered, he explained. 

Most of the fields displayed here are dynamic links that lead to more links to the object hierarchy within the cloud model.  

“You can also search on almost every field and/or combination of tags and fields,” Pearce said. “Once you get into the world of tagging,  it is hard to stop creating tags for each use case that you encounter.” 

The interface is good for looking up a single offending IP and finding the VM owner. It allows the IT team to find all VPCs that have a VPN connection to a specific on-prem data center. The tool also allows for analyzing the impact of a connectivity problem on this data center.  

“These assets are all conveniently tagged, so it is simple to search for this tag to produce a report,” Pearce said.

If you are dealing with tens or hundreds of results, NetBox allows the export of the search results to common formats, such as CSV.  This export and report generation capability makes it easy for less technical users to consume NetBox data by importing it into their own tools for auditing, compliance, financial analysis and other purposes.

“If you want to consume NetBox in a continuous, scalable, automated fashion and depend on NetBox as the “source of truth”, you should be using the NetBox REST API interface, as this is the real power of the tool,” said Pearce.  

The GUI is simply a front end that depends on the API, he explained. His team frequently uses Postman to prototype NetBox REST interactions and then turn these into Python code.

“All of this information is available via the NetBox API for other automation to query,” Pearce said, hinting at a desire to find new ways to automate basic tasks. 

The effort shows what can happen when IT pros have the time, motivation, resourcefulness to solve difficult problems, a culture hungry for innovation and support from the CIO and top management. It also demonstrates the value of using modern open-source platforms and APIs to innovate. 

“There's something very inspiring about seeing this information in this format,” said Pearce. “Now, I can tell you how many VMs we have across all clouds.” 

He said there was no way to do this previously.

“You’d have to visit the portals for all three cloud providers and consolidate information by hand.”

Finding Assets to Strengthen Security

Multicloud asset search is not just a nice-to-have perk for Pearce and his colleagues. It’s also a substantial boon for security teams forced to solve riddles based on a single clue: an IP address for a suspicious cloud asset. They need to know everything else — location, memory, software installed, etc.

“When the security team has an incident, they want to quickly find the problem without going through three different cloud administrators who traditionally use manual methods to look for it,” Pearce said.

NetBox makes that information easier to find.

Automating Data Center Inventory

Nutanix’s IT department supports business operations, including product developers. Documenting all the data center equipment (servers, switches, storage, cabling, etc.) and services is essential and non-stop. Even in global enterprises, this kind of information is often stored in spreadsheets. Prior generations probably made lists using pencil and paper.

What IT teams really need is dynamic storage in a database. Indeed, the need is so pressing that an entire community sprang up around NetBox, the open-source software built expressly to create searchable databases of IT asset inventories. In a July 2022 article, Pearce and Trajkovic first explained how their team put NetBox to use tracking data center hardware.

That article spelled out how the NetBox database dramatically reduced the effort required to find specific devices on a network. This comes in extremely handy in security scenario outlined above (IP address but no other data). Filtering tools in the NetBox DB make this data available in seconds. Finding the same data in somebody’s spreadsheet could be precarious at best — easy on Monday during the document owner’s work shift but impossible on Wednesday if the owner took a holiday trek to Everest Base Camp.  

Trajkovic praised NetBox’s power and flexibility in the July article. 

“Out of the box, you’ll get about 90 percent-plus of the capabilities to document a modern IT infrastructure,” he said. “And what's not in there, you can make on your own.” 

These do-it-yourself possibilities sparked a burst of creativity in the Nutanix IT team.

Finding Assets Inside Hybrid Multicloud Systems

The best open-source software platforms use a global developer community to build a core application and a galaxy of plugins that expand functionality.

NetBox followed this route. The platform sprang up organically because saving inventory data in spreadsheets and other static formats had become untenable as networks grew increasingly complex.

Nutanix’s IT team found the NetBox database did an excellent job of documenting their data center assets. This functionality made them eager for more: Why not add multicloud data search and aid the company in the transition to hybrid multicloud?

That functionality wasn’t built into NetBox, so Pearce, Trajkovic and colleagues, including Milan Nikolic and Dusan Cvetković did what developers always do when they don’t feel like waiting around for somebody else to solve their problem. They fixed it themselves — thanks to the plugin capability recently added to the NetBox application.  

They built a collection of plugins that made it intuitive to click through the core categories and subcategories of their multicloud ecosystem:

• Cloud Platform (service providers)
• Cloud Accounts (owner of cloud asset collection)
• Cloud VPCs/Network/VNet (virtual private cloud in a specific region)
• Cloud Subnets (IP networks gained within VPCs)
• Cloud VM Types (Models of virtual hardware that runs cloud VM)
• Cloud VMs (virtual machines our instances running in the VPC)
• Cloud VM Ints (virtual machine network interfaces or NICs)
• Cloud Resource Groups (another way to group cloud assets)

One benefit (and challenge) is getting similar asset types into the same bucket, regardless of cloud provider, said Pearce.

A text search and sort fields allow for granular inquiries. The IT team can also search across cloud assets using tags or labels. 

“The Nutanix GCO operations team tags cloud accounts for owner, contacts and department, making it easy to locate the human behind an asset when there is an issue,” Pearce said.

Using the REST API, the Nutanix team pulled in data feeds from AWS, Azure and GCP, making cloud asset data in all three services sortable and searchable on a single NetBox page.

“We developed a Python crawler scripts that would use API calls for these three platforms,” Trajkovic recalled. “They're very different from one another, but the result is the same.” 

The scripts can collect data, compare it and then populate certain search-result fields based on logic doing things like asking whether something does not exist and then creating it if needed. If something does exist, then the script would compare it to a previous scan and provide updates if anything has changed, Trajkovic explained.

Python crawler scripts can identify all cloud assets in the ecosystem and document their vital data. Pearce cautions that their plugins work for specific needs. 

“The crawl can take an hour or more and it’s not practical for real-time monitoring (which is widely available from clouds themselves or third-party providers in any case),” said Pearce.   

“The accuracy of the cloud object data in NetBox is only as good as the most recent cloud crawl and objects within the cloud are in constant flux, being created, modified and destroyed, much more often than their on-prem equivalent assets already in NetBox. This does not diminish the value of the data.”

It has great potential for detecting unused or underused assets, he said. 

“It could be an incredible cost saver because it can go out and say ‘that environment was provisioned but no one ever used it’ or no one's ever changed this object in a year; perhaps it's dead or abandoned. Obviously, reducing cloud spend is a huge focus for businesses now”. 

Building the NetBox plugins took time and focus. 

“We spent about a month learning the framework,” Trajkovic said. 

While Trajkovic was figuring out how to create database tables and present results on an intuitive GUI then integrate everything with the cloud providers’ APIs, Pearce was developing the Python crawlers. From start to implementation was about six months.

Pearce noted that the project helped him brush up on his Python coding skills. 

“Each cloud has a different Python API, with a different approach or style, and documentation of varying quality,” said Pearce. “Jumping back and forth between three different cloud APIs, trying to accomplish the same task in each,  is challenging.”

He said there are a variety of approaches to producing a cloud asset inventory within a single cloud. Figuring out the best programming approach is not simple.

“For example, we found that one cloud had costly and restrictive limits on a certain API call and had to rewrite the code to avoid using it,” he said.

He said the real game-changer was the REST APIs.

“Once you've got REST APIs, you can talk to any other thing in the world that's got the REST API and send information to it or pull information from it,” Pearce said. “And it just really opens up everything.”

Tom Mangan is a contributing writer. He is a veteran B2B technology writer and editor, specializing in cloud computing and digital transformation. Contact him on his website or LinkedIn.

© 2023 Nutanix, Inc. All rights reserved. For additional legal information, please go here.