Understanding Retrieval Augmented Generation (RAG)

What is retrieval augmented generation (RAG)?

Retrieval augmented generation (RAG) is a technique that improves the accuracy and relevance of the outputs of large language models (LLMs). It does this by combining data from external information retrieval systems with LLMs, which can often be significantly out of date. 

For instance, recently ChatGPT’s breadth of knowledge only extended through early 2022. If it wasn’t enhanced by other programs, you wouldn’t be able to rely on its answers to queries if asking about something very recent or new. RAG is one of those programs that helps fill in the gaps for generative AI models.   

RAG is often used to improve relevance and reliability of a variety of LLM outputs, including customer service systems, support chatbots, project management frameworks, and Q&A systems.

How retrieval augmented generation works

In addition to often being out-of-date, LLMs are taught to extrapolate when data isn’t available—this extrapolation is called hallucination, and can result in utterly false information that might look legitimate at first glance. 

Retrieval augmented generation can resolve these issues by boosting an LLM’s knowledge bank with information from external retrieval systems. The external data adds up-to-date context and enables the LLM to develop a deeper understanding of facts around particular topics or queries. RAG makes it possible to get accurate and relevant outputs without having to retrain the LLM. This can be especially helpful when training an LLM on your own organization’s data. 

To illustrate how RAG works, consider a system like ChatGPT that answers user queries. As soon as a user enters a question, the system goes into action. The (simplified) process is as follows:

• Information retrieval – The AI system transmits the text-based query to a model that changes it to machine code, which is numeric and sometimes called a vector. The vector compares its code to a machine-coded, pre-existing index of external information sources, such as web pages, databases, and other existing knowledge banks. 

• Data pre-processing – When the system finds related information on the query topic in those external sources, it retrieves that data, converts it back to text that humans can read, and transmits it back to the LLM. 

• Data integration – The LLM parses the retrieved data and combines it with its own internal knowledge base and forms a complete answer for the user. Some systems will also cite the external sources for some of the data in the answer. 

• Continual updates – Once a vector has been created and it has delivered the relevant external data to the LLM, it can keep working in the background to create continual updates to the index. This helps ensure that the LLM always has access to up-to-date information.

To work as described above, the retrieval augmented generation system must first undergo an ingestion phase. This is where the system goes out and creates the index, or library, of external information sources. While a lot of that is done beforehand, some retrieval augmented generation systems can also find external information and sources in real time—by querying databases, for instance, which are easily searched and analyzed; API calls, which allow the system to access data contained in different applications or platforms; or scraping web pages.

Why retrieval augmented generation is important

Retrieval augmented generation is a critical component to keeping LLM outputs up-to-date and relevant. While LLMs are very good at what they do, they have limitations. These include:

• Making up answers when it lacks the appropriate data. 

• Offering up overly generic or outdated information. 

• Not knowing exactly how to identify reliable information sources. 

• Getting confused over terminology in the training process, as different training systems can use different terminology for the same concepts. 

Without RAG, the LLM’s store of knowledge would be limited to a specific date range or the LLM would need constant retraining, which takes time and is expensive. With RAG, LLMs are more intelligent and versatile, and offer better overall outputs in everything from AI-based content creation to complex virtual assistants to savvy chatbots in the customer service center.

The role of cloud computing in retrieval augmented generation

Retrieval augmented generation wouldn’t be possible without the cloud. The cloud offers vital capabilities and features that enable RAG systems to work their magic. These include: 

• Near-infinite scalability and massive storage capacity – Retrieval augmented generation systems are dependent on very large datasets, such as a company’s entire knowledge base, content from thousands or millions of websites, and huge collections of online documents. Public cloud providers can deliver as much storage and computing capacity as you need at any time, and scaling the system even further can be quick and easy. 

• Distributed databases and search capabilities – A lot of the external data that RAG systems retrieve is unstructured data, meaning it can’t be easily organized into neat spreadsheets. The cloud is an ideal place to store and access large pools of unstructured data. Distributed search systems are also common in the cloud and accelerate and simplify the access and retrieval of information from large datasets. 

• High performance and high availability – The cloud has built-in features that enable it to operate at peak performance and reduced latency. High throughput and low latency are essential to retrieval augmented generation. The cloud also has a wide range of built-in redundancy measures to ensure that operations continue even if a particular node or cluster fails. 

• Deployment and management of LLMs – The cloud is an optimal location for LLM training. Many providers offer managed AI services that make model deployment and management even easier.

Benefits of retrieval augmented generation for enterprises

One of the biggest benefits of RAG is that it effectively and efficiently bridges the gap between the dated nature of LLMs and the constantly evolving state of human language and modern knowledge. Other benefits include: 

• Enhanced accuracy of LLM outputs – Retrieval augmented generation combines the latest information to an LLM’s existing body of knowledge, enabling the most up-to-date and accurate answers or outputs. This also improves user trust in the overall system. 

• Improved relevance – The added current knowledge from RAG helps LLMs to understand sophisticated nuances in language and meaning that might not have been obvious without it. 

• Fact verifiability and transparency – Many RAG systems cite sources of external information, which enables users to verify the data and make corrections if needed. It helps eliminate the “black box” nature of some AI-based systems, where users aren’t sure how answers or other outputs are being generated. 

• Added versatility – Retrieval augmented generation systems can give LLMs increased versatility and enable additional use cases. The more relevant external data fed into the LLM, the more detailed and personalized outputs will be.

Use cases for retrieval augmented generation

• Chatbots – Whether a chatbot for enterprise knowledge management that employees use or a customer-facing support chatbot, RAG systems help ensure that conversations are relevant and detailed. In addition to simply giving answers, these chatbots can also synthesize information and situations and offer up actionable insights. 

• Drafting assistants – Retrieval augmented generation systems can be extremely helpful when it comes to AI-based content creation, such as when a user needs to create a report using company-specific information or a reporter needs to find relevant statistics for an article. This can help users save time and create content more efficiently. 

• Research assistants – LLMs enhanced with RAG can be very helpful during research phases, such as for graduate students’ dissertations, legal cases, medical and clinical research, and more. 

• Knowledge engines – Advanced question-and-answer systems and knowledge bases can be more accurate, relevant, and timely with RAG combined with LLMs. 

• Customized idea generation – Using an LLM with retrieval augmented generation, you can enhance brainstorming with AI-based recommendations, insights into future trends, identification of relevant experts, and more. This also helps improve decision making and enables you to resolve unique challenges more efficiently.

How Nutanix supports retrieval augmented generation

Nutanix understands the challenges of today’s organizations as the hybrid multicloud ecosystem becomes the IT norm. We have always worked hard to simplify operations and management of data and applications in the cloud—and offer a number of solutions, such as Nutanix Cloud Platform, that can help you overcome the common roadblocks that limit your ability to compete.  

As AI and LLMs become more intrinsic to modern business success, Nutanix is finding innovative ways to harness that power and make it work for you. One of those innovations is retrieval augmented generation, and we are currently testing RAG systems on our internal infrastructure so we enhance our understanding of how it can make your organization’s AI-based solutions more efficient and effective.