What is RAG?
RAG stands for Retrieval-Augmented Generation.
RAG allows an LLM to search relevant information from an external knowledge source and use that information to generate an answer.
Without RAG:
User Question
↓
LLM
↓
Answer based mainly on its trained knowledgeWith RAG:
┌───────────────┐
│ User Question │
└───────┬───────┘
↓
┌─────────────┐
│ Retriever │
└──────┬──────┘
↓
Search Knowledge Base
↓
Relevant Documents
↓
┌──────────────────────┐
│ LLM │
│ Question + Documents │
└──────────┬───────────┘
↓
Answer1. Why do we need RAG?
LLMs have several limitations. An LLM normally doesn’t know your company’s internal documents.
Imagine you have an automotive company’s internal documents:
DMS Documentation
CRM Documentation
Service Manual
Warranty Policy
Customer Policy
Vehicle Manual
API Documentation
Employee HandbookNow a user asks:
“What is the warranty period for a battery replacement?”
A general LLM may not know your company’s specific warranty policy. Even if you provide the model with a huge prompt containing all documents, you have problems with:
- Context size
- Cost
- Latency
- Maintaining documents
- Outdated information
RAG solves this by retrieving only the relevant information.
User
│
│ "What is battery warranty?"
↓
RAG Retriever
│
│ Search company documents
↓
Warranty Policy
│
│ Relevant section
↓
LLM
│
↓
"Battery replacement has a warranty of X months..."2. RAG has two major phases
A typical RAG system has:
Phase 1 — Indexing
Prepare your documents for searching.
Documents
↓
Load
↓
Split into chunks
↓
Create embeddings
↓
Store in Vector DatabasePhase 2 — Retrieval + Generation
When the user asks a question:
Question
↓
Question Embedding
↓
Vector Search
↓
Relevant Chunks
↓
Prompt + Retrieved Context
↓
LLM
↓
AnswerThis distinction is very important when designing RAG systems.
RAG Architecture
A typical RAG system looks like this:
┌──────────────────┐
│ User Question │
└────────┬─────────┘
↓
┌──────────────────┐
│ Query Processing │
└────────┬─────────┘
↓
┌──────────────────┐
│ Retriever │
└────────┬─────────┘
↓
┌─────────────────────────────┐
│ Vector Database │
│ │
│ Documents → Embeddings │
└──────────────┬──────────────┘
↓
Relevant Documents
↓
┌──────────────────┐
│ LLM │
└────────┬─────────┘
↓
┌──────────────────┐
│ Generated Answer │
└──────────────────┘There are actually two major phases in RAG:
- Knowledge ingestion
- Question answering
Let’s understand both.
1. Knowledge Ingestion
Suppose you have this PDF:
company_policy.pdfIt contains 100 pages.
We don’t normally send the entire PDF to the LLM every time a user asks a question.
Instead, we process it.
Step 1: Load documents
PDF
Word
HTML
Database
Web pages
API
CSV
JSON↓
Step 2: Split documents into chunks
A large document is divided into smaller pieces.
For example:
Document
Page 1
Page 2
Page 3
...
Page 100could become:
Chunk 1
Chunk 2
Chunk 3
Chunk 4
...
Chunk 500Why?
Because retrieving a small relevant section is usually more useful than passing an entire 100-page document.
2. Create Embeddings
Each chunk is converted into a numerical representation called an embedding.
For example:
"Employees can take 24 annual leave days."might become conceptually:
[0.21, -0.42, 0.73, 0.15, ...]The actual embedding contains many dimensions.
The important idea is:
Similar meanings produce vectors that are close to each other in vector space.
For example:
"How many vacation days do employees get?"
↓
Embedding
↓
Vector Databasecan find:
"Employees can take 24 annual leave days."even though the words vacation and annual leave are different.
3. Store embeddings
The embeddings are stored in a vector database.
Popular choices include:
- Pinecone
- Weaviate
- Milvus
- Qdrant
- Chroma
- PostgreSQL + pgvector
- Elasticsearch/OpenSearch
Conceptually:
Chunk Embedding
"Annual leave is 24 days" [0.21, 0.42, ...]
"Sick leave is 12 days" [0.72, 0.11, ...]
"Remote work policy..." [0.31, 0.82, ...]4. User asks a question
Now the user asks:
“How many annual leaves can I take?”
The question itself is converted into an embedding.
Question
↓
Embedding Model
↓
Query Vector5.Retrieval
The RAG system compares the question vector with vectors stored in the database.
For example:
Question:
"How many annual leaves can I take?"
↓
Vector Search
↓
Results
1. Annual leave policy 0.94
2. Sick leave policy 0.72
3. Remote work policy 0.31The system retrieves the most relevant chunks.
For example:
Employees can take 24 annual leave days per year.
Employees can take 12 sick leave days per year.6. Augmentation
Now the retrieved information is added to the LLM prompt.
Conceptually:
SYSTEM:
Answer the question using the provided context.
CONTEXT:
Employees can take 24 annual leave days per year.
QUESTION:
How many annual leaves can I take?This is the Augmentation part of RAG.
7. Generation
The LLM receives the question plus retrieved context.
It generates:
“According to the company policy, employees can take 24 annual leave days per year.”
That’s the Generation part.
So:
Retrieval
+
Augmentation
+
Generation
=
RAGComplete RAG Flow
The entire process looks like this:
DOCUMENTS
│
↓
┌─────────────────┐
│ Document Loader │
└────────┬────────┘
↓
Chunking
↓
Embedding Model
↓
┌──────────────────┐
│ Vector Database │
└──────────────────┘
USER
│
│ "How many annual leaves?"
↓
Embedding Model
│
↓
Query Vector
│
↓
Vector Search
│
↓
Relevant Chunks
│
↓
Prompt + Context
│
↓
LLM
│
↓
Generated Answer
RAG Example: Software Company
Since you are working with Java, Spring Boot, microservices and architecture, let’s take a technical example.
Imagine your organization has:
Architecture/
CRM Architecture.pdf
DMS Architecture.pdf
API Guidelines.pdf
Security Guidelines.pdf
API/
Customer API.docx
Vehicle API.docx
Payment API.docx
Database/
Customer DB Schema.pdf
Vehicle DB Schema.pdfYou build an internal AI Architecture Assistant.
A developer asks:
“Which API should I use to retrieve customer vehicle information?”
Without RAG
The LLM might say:
“You can use the Customer Vehicle API.”
But it may be inventing the API.
With RAG
The system searches your internal documentation.
It finds:
Vehicle API Documentation
GET /api/v1/customers/{customerId}/vehiclesThen the LLM generates:
“According to the API documentation, you can retrieve a customer’s vehicles using
GET /api/v1/customers/{customerId}/vehicles.”
This is much more useful because the answer is grounded in your organization’s documentation.
RAG in an Agentic AI System
This becomes especially interesting for Agentic AI.
A simple RAG application:
User
↓
RAG
↓
LLM
↓
AnswerAn agentic system can use RAG as a tool.
Agent
│
┌──────────────┼──────────────┐
↓ ↓ ↓
RAG Tool API Tool Database Tool
│
↓
Knowledge Base
For example:
User:
"Why is customer 123 unable to book a service?"
Agent
↓
Understand problem
↓
Search API documentation
↓
Search troubleshooting knowledge
↓
Call customer API
↓
Check booking status
↓
Analyze results
↓
Answer userHere RAG is one tool available to the agent, rather than the entire application.
RAG Example with Python
A simplified RAG implementation looks like:
documents = [
"Employees receive 24 annual leave days.",
"Employees receive 12 sick leave days.",
"Employees can work remotely two days per week."
]
# 1. Create embeddings
embeddings = embedding_model.embed_documents(documents)
# 2. Store embeddings
vector_db.add_documents(
documents,
embeddings
)
# User question
question = "How many annual leaves do employees get?"
# 3. Retrieve relevant documents
results = vector_db.similarity_search(
question,
k=3
)
# 4. Create context
context = "\n".join(results)
# 5. Send context + question to LLM
prompt = f"""
Answer the question using the following context.
Context:
{context}
Question:
{question}
"""
answer = llm.generate(prompt)
print(answer)Conceptually:
Documents
↓
Embeddings
↓
Vector DB
↓
Retriever
↓
Context
↓
LLM
↓
AnswerIn a production system, you would also consider chunking, metadata, access control, reranking, citations, evaluation, monitoring, and prompt-injection defenses.
Important RAG Problems
RAG is powerful, but it has its own challenges.
Problem 1: Bad chunking
If you split documents incorrectly:
Chunk A:
"Employees can..."
Chunk B:
"take 24 days..."The meaning may become difficult to retrieve.
Problem 2: Wrong retrieval
The system may retrieve irrelevant documents.
Question:
"Annual leave?"
Retrieved:
"Sick leave policy"The LLM then receives bad context.
Garbage in → garbage out.
Problem3 : Too much context
Suppose retrieval returns:
100 chunksSending all of them to the LLM may increase cost and make the answer less focused.
This is why systems often use:
Top-K retrieval
+
Reranking
+
Context compressionProblem 4 : Outdated documents
Your vector database may contain:
API v1
API v2
API v3If the retriever returns v1 when v3 is required, the LLM may provide an outdated answer.
Metadata and version filtering can help.
Problem 5 : Security
This is particularly important in enterprise RAG.
Suppose:
Employee A → allowed to see Finance documents
Employee B → not allowedYour RAG system must enforce authorization before returning restricted chunks.
RAG should not become a way to bypass existing access controls.
Advanced RAG
As you progress toward Agentic AI, you will encounter:
Basic RAG
↓
Hybrid RAG
↓
Advanced RAG
↓
Agentic RAGBasic RAG
Question
↓
Vector Search
↓
Context
↓
LLMAdvanced RAG
Question
↓
Query Transformation
↓
Hybrid Retrieval
↓
Metadata Filtering
↓
Reranking
↓
Context Compression
↓
LLM
↓
Answer + CitationsAgentic RAG
Agent
│
┌───────────┼───────────┐
↓ ↓ ↓
RAG Search API Tool DB Tool
│ │ │
└───────────┼───────────┘
↓
Reasoning
↓
Answer
This is particularly useful when the system needs to decide when and how to retrieve information, rather than blindly performing one search for every question.