Building a Solid Enterprise AI Infrastructure Strategy

It takes power and prudence to make AI work at enterprise scale.

For starters, large language models (LLMs) and other machine-learning (ML) workloads are ravenous consumers of electricity and compute resources. Rising wireless bandwidth opens a raft of opportunities at the network edge. AI governance must ensure that benefits outweigh risks. These are just a few examples of challenges CIOs face as they create and roll out enterprise AI strategies.

At the core of any enterprise AI strategy is a well-designed, thoughtfully executed IT infrastructure. That requires a framework of processes and tools for managing an enterprise IT ecosystem that can scale to meet demands of AI applications and data, according to IT experts interviewed by The Forecast. 

“AI-based solutions and services are not a fad,” said Debojyoti “Debo” Dutta, Chief AI Officer at Nutanix.

He pointed to the meteoric rise of generative AI applications such as ChatGPT that are already making a big impact yet are revealing some big challenges.

“People and businesses will use generative AI to get ahead of those who don't use generative AI,” he said. “But it is such a new technology that many organizations today don't have tools and skills to build and implement these capabilities on their own.”

1. Strategy Aligning AI to Business Needs

The relentless stream of new AI technologies is forcing organizations to separate the truly compelling from the merely interesting. An LLM-driven bot like ChatGPT can answer questions in plain language on just about any topic. But can it help close sales, make transactions frictionless and bolster a company’s bottom line?

Answering such questions requires leaders to craft an AI strategy aligned to specific business needs. 

“Once you start talking about what the business use is, and the value you're trying to extract from any technology, that then is going to drive the technology decisions,” Steve McDowell, chief analyst at NAND Research told The Forecast.

A good strategy pulls things forward and enables the enterprise to stay innovative.

“How am I going to use AI to enable the next generation or the next iteration of digital transformation?” asked McDowell. “But it's also, how do I use it to make my own IT operations more efficient?”

I look at Nutanix’s AI strategy sheds light on how others can build and evolve strategies for their own business. Based in Silicon Valley, Nutanix pioneered virtualization software that emulates compute, storage and networking in a model called hyperconverged infrastructure (HCI). Today their software enables IT teams to manage data center resources across public and private cloud systems, including cloud native technologies used to build and run AI applications.

“We want to help our customers land their AI workloads on the Nutanix platform very easily while maintaining full, enterprise-grade control and managing costs,” said Dutta

He helped the company develop Nutanix Enterprise AI capabilities, including ChatGPT-in-a-Box to give enterprises a fast, simple platform for building productive generative AI (GenAI) use cases.

Nutanix also added GenAI capabilities designed specifically for Kubernetes developers. It allows developers to ask questions that guide them through the often-massive complexities of Kubernetes. The company’s reliability engineers use GenAI internally to make their processes more efficient. And internal teams created SupportGPT, SalesGPT and SEGPT applications to improve productivity for Nutanix customer support, sales and systems engineering teams.  

Dutta noted that Nutanix carefully tracks usage of GenAI to make sure it has measurable data documenting its effectiveness and pointing to opportunities for improvement.

2. Energy-Efficient Compute and Accelerated Storage 

Dutta urged enterprises to map out a five-year plan for developing a cloud-native, GPU-enhanced infrastructure including high-performance compute and accelerated storage. Hardware sales suggest many organizations are already taking this approach.

Gartner reported that data center system sales jumped 34.7% in 2024 and could rise another 15.5% in 2025. 

“GenAI will easily eclipse the effects that cloud and outsourcing vendors had on previous years regarding data center systems," said John-David Lovelock, Distinguished VP Analyst at Gartner.

Meanwhile, data center energy consumption in the U.S. is expected to double or triple by 2028 after tripling in the past decade, according to the U.S. Department of Energy (DOE). It’s not just AI workloads: Manufacturing is on the rise amid the electrification of the nation’s auto fleet, DOE reported.

Technology firms are figuring out ways to use less electricity in data centers: “From air cooling to liquid cooling and beyond, the industry has really stepped up the gas,” Dutta said. 

He said AI developers are building more compact models that distill machine learning into smaller models, reducing compute requirements.

“It’s like a teacher training a student,” Dutta said. These condensed models can run on a smartphone, laptop or any other networked device.

3. Inference at the IT System’s Edge 

The network edge is brimming with phones, laptops, PCs, tablets, cameras and special-purpose sensors. Adding AI-optimized chipsets to these gadgets creates a wealth of opportunities because of inference — the process of a microprocessor trying to figure out what a user wants from an AI application.

Inference is the latter of two key phases of AI. The former is training, when a mathematical model analyzes massive datasets and develops basic cognitive capabilities. Because training takes far more compute power than inference, it typically happens on GPU-accelerated servers or workstations. 

Inference, however, can be powered at the IT network’s edge on devices designed for AI apps. Moreover, AI-optimized devices can run inference workloads locally, easing the burden on remote data centers and reducing latency that degrades the user experience.  

TechTarget notes that inference costs can eventually exceed compute costs if millions of users adopt an AI application. However, running inference at the network edge distributes a portion of the costs among the owners of AI-enhanced devices, reducing a potential hurdle to AI adoption. Dutta noted that Nutanix’s enterprise AI software makes it easy for companies to add inference at the network edge.

4. IT System Virtualization

It’s easy to see the consumer appeal of adding AI chips to laptops or phones that make apps like ChatGPT more effective. AI at the edge is much more convoluted at enterprise scale, however. A wireless provider might have hundreds or even thousands of edge servers to optimize cell-tower data and voice traffic, for instance.

“The reason we push AI to the edge is because that's where the data is,” said NAND analyst McDowell, in a Tech Barometer podcast interview about his 2024 report Taming the AI-enabled Edge with HCI-based Cloud Architectures.

“Deploying new AI capabilities to the edge disrupts traditional edge architectures,” he said. 

“Traditional edge computing involves things like point-of-sale systems in retail. Once we start putting AI in, then suddenly we have processing requirements that can require AI accelerators.”

A conventional enterprise edge network is centralized, while an AI edge network is distributed, he wrote. Enterprise networks are largely homogenous, while AI edge networks have device and application diversity.  

Hardware virtualization via HCI can make AI-enabled edge networks much more manageable. With hypervisor software orchestrating virtual networks, servers and storage from a central console, IT teams can substantially reduce the complexity of their environments.

“An HCI-based platform's centralized management tools allow for easier monitoring and management of resources across multiple edge locations while allowing for remote updates, increased security, and comprehensive data and network protection,” McDowell wrote. Download his report to get the full story.  

5. Interconnected Data Platforms 

AI apps depend on data platforms that store, manage, digest and deliver safe and reliable data. The data can be structured, like numbers and fields in formal databases, and unstructured, like text and video in data lakes. Big cloud services companies like Amazon, Google and Microsoft offer their own data platforms, or IT teams can use the Nutanix Cloud Platform to manage across their owned and rented public cloud services.  

“Data platforms can be used to generate the right type and quantity of data that's needed to fuel the AI models,” Nutanix’s Dutta said. 

“The platforms are relevant because you have to extract, curate and summarize the data using traditional tools and then augment it with large language models.”

Enterprises running multiple AI workloads will most likely need to weave a web of data platforms across their IT ecosystems. Fortunately, data platforms are a mature technology that emerged with the increasing focus on data science and predictive analytics. 

“A lot of the practical AI systems are being built with advances in LLMs and big-data tech,” Dutta added.

6. Safe, Compliant AI Guardrails 

When generative AI apps first hit the market, people noted a disturbing tendency to make up false facts and present them as true. Making these AI hallucinations go away has been a priority of AI developers ever since.   

“We are very early in the process of creating robust systems for AI model safety and governance,” Dutta cautioned. 

He said some apps may deliver content that’s inaccurate or offensive. Fine-tuning AI models and creating standards to improve data quality will be central to the mainstreaming of AI apps.