RAG — Retrieval Augmented Generation
Give your LLM a memory: retrieve facts from your data, then generate accurate answers.
Interview tip Say: "RAG = retrieve relevant chunks → stuff into prompt → LLM answers with grounded context." Mention vector DB + chunking + evaluation.
① What you must know (30 sec)
RAG augments an LLM at query time by retrieving relevant document chunks from your knowledge base, injecting them into the prompt, and asking the model to answer using only that context. Plain LLMs only know training data; RAG grounds answers in your wiki, tickets, codebase, or policies — reducing hallucinations and enabling fresh, citeable responses.
Analogy: Like an open-book exam: the LLM is the student, your vector DB is the textbook index, and retrieval finds the right pages before answering.
② How it works
RAG pipeline
User question→Embed query→Vector search→Top-k chunks→LLM + prompt→Answer + cites
Ingestion — load docs (PDF, HTML, tickets), clean text, attach metadata
Chunking — split into 300–800 token segments with 50–100 overlap
Embedding — convert chunks to vectors; store in vector DB with IDs
Retrieval — embed query, cosine similarity, optional BM25 hybrid + rerank
Generation — system prompt: "Answer only from context below; cite sources"
Offline indexing (batch) is separate from online query path (real-time). Re-run ingestion when source docs change.
③ Step-by-step (hands-on)
Step 1 — Prepare documents
Collect sources (Notion export, Confluence, GitHub markdown). Strip boilerplate, preserve headings as metadata, and tag by team or product area for filtered retrieval.
Step 2 — Chunk intelligently
Use recursive character splitting on headings first, then paragraphs. Target 400–600 tokens with overlap so sentences are not cut mid-thought. Store source URL, page, and section in metadata.
Step 3 — Embed and index
Batch-embed chunks with text-embedding-3-small (or open alternative). Upsert into Pinecone, pgvector, Chroma, or Milvus. Verify dimension matches model output.
Step 4 — Build retrieval
Embed user query with the same model. Fetch top 5–8 by cosine similarity. Optionally merge BM25 keyword hits and rerank with cross-encoder for precision.
Step 5 — Craft the prompt
System: role + "use only provided context; say I don't know if missing." User: question + numbered context blocks with source IDs. Require citations in the answer format.
Step 6 — Evaluate and iterate
Create 30–50 labeled question→expected-doc pairs. Measure recall@k and answer faithfulness. Tune chunk size, k, and prompts until metrics plateau.
④ Code / config patterns
| Pattern | When to use | Trade-off |
|---|---|---|
| Naive top-k | POC, small corpus | Fast; may miss nuance |
| Hybrid (BM25 + vector) | Technical docs with exact terms | More infra; better recall |
| Reranker (Cohere, bge) | High-stakes answers | +latency, +cost |
| Parent-child chunks | Long docs | Retrieve small, feed large parent |
| Query rewriting | Vague user questions | Extra LLM call |
// Minimal RAG query path
query_vec = embed(user_question)
chunks = vector_db.search(query_vec, top_k=6)
context = format_chunks(chunks) // include source_id per chunk
messages = [
{ role: "system", content: "Answer only from context. Cite [source_id]." },
{ role: "user", content: f"Context:\n{context}\n\nQ: {user_question}" }
]
answer = llm.chat(messages)⑤ Production & pitfalls
| Pitfall | Why it hurts | Fix |
|---|---|---|
| Wrong chunks retrieved | Confident but incorrect answers | Tune chunking, hybrid search, reranker, metadata filters |
| Stale index | Outdated policies or APIs cited | Scheduled re-index; webhook on doc publish |
| Context overflow | Truncated prompt, missed facts | Lower k, summarize chunks, parent-child retrieval |
| No citations enforced | Users cannot verify claims | Require source IDs in prompt + UI links |
| Mixed embedding models | Similarity scores meaningless | Re-embed entire index on model change |
| Ignoring ACLs | Leaked confidential docs in answers | Filter by user permissions at retrieval time |
Production tips:
- Log retrieval IDs and scores per query for debugging
- A/B test chunk sizes on a golden eval set before shipping
- Cache embeddings for frequent queries
- Monitor faithfulness with LLM-as-judge or human spot checks weekly
⑥ Interview / on-the-job Q&A
| Question | Answer |
|---|---|
| What is RAG? | Retrieve relevant docs at query time, inject into prompt, generate grounded answer — not retraining the model. |
| RAG vs fine-tuning? | RAG for fresh facts and citations; fine-tune for style, format, or domain phrasing when examples are abundant. |
| What is chunk overlap? | Repeated tokens between adjacent chunks so sentences split across boundaries are still retrievable. |
| What is hybrid search? | Combine dense vector similarity with sparse keyword (BM25) retrieval, then merge or rerank results. |
| How do you evaluate RAG? | Retrieval metrics (recall@k, MRR) plus answer quality (faithfulness, relevance) on labeled Q&A pairs. |
| Why hallucinate with RAG? | Retrieved context irrelevant or empty; model fills gaps — fix retrieval or say "I don't know." |
⑦ Tools & ecosystem
- Vector DBs: Pinecone, pgvector, Weaviate, Chroma, Milvus, Qdrant
- Frameworks: LangChain, LlamaIndex, Haystack
- Embeddings: OpenAI text-embedding-3, Cohere embed-v3, BGE, Voyage
- Rerankers: Cohere Rerank, cross-encoder models (bge-reranker)
- Eval: RAGAS, TruLens, custom golden sets in spreadsheets
⑧ Revision checklist
- Same embedding model for index and queries
- Chunk size 300–800 tokens with overlap tested on sample docs
- Metadata includes source URL, title, section, access level
- System prompt forbids answering outside context
- Citations visible to end users
- Eval set of 30+ real questions with expected sources
- Re-index pipeline triggered on content updates
- Latency and cost per query logged (embed + LLM tokens)
- Hybrid or rerank considered if naive retrieval fails eval