A language model does not know a company's internal documents, neither the current price list nor the travel expense policy nor the manual for its own software. In addition, its knowledge ends with the cutoff of its training data. Retrieval-augmented generation, meaning generation extended by retrieval, solves this problem without retraining the model. The approach is now one of the most common ways to connect AI assistants with company knowledge.
How does retrieval-augmented generation work?
A RAG system works in two phases.
Preparation: Documents are read from their sources, for example from file shares, wikis, or ticketing systems, and split into smaller chunks. For each chunk, an embedding is calculated, that is, a series of numbers that represents its meaning. This data is stored together with information on source and permissions in a search index, often a vector database.
Query:
- A person asks a question, for example about the procedure for a vacation request.
- The system searches for the chunks that best match in content, often with a combination of semantic and keyword search.
- The chunks found are passed to the language model together with the question.
- On this basis, the model formulates an answer and ideally cites the sources.
Typical examples
- An assistant answers questions about internal policies and processes.
- In the service desk, matching solutions from earlier cases and the documentation are found for a new ticket.
- Technical manuals become searchable via chat for field staff.
What companies should keep in mind
- Carry over permissions: An assistant may only use content that the person asking is allowed to see themselves.
- Maintain the content: Outdated or contradictory documents lead to wrong answers. Changes and deletions must reliably reach the index.
- Test search quality: Whether the right chunks are found determines the quality of the answer. Test questions with known answers make this measurable.
- Know the residual risk: RAG reduces the risk of hallucinations but does not eliminate it. Source citations make checking easier.
- Consider manipulation: Documents can contain hidden instructions intended to influence the model.
How it differs
With fine-tuning, a model is further trained with your own data to adapt style, format, or technical language. For current and frequently changing knowledge, RAG is usually better suited, because documents can be replaced at any time without changing the model.
View AI assistance