RAG & RAGAS

Search, then write

A demo that uploads one PDF and asks one question does not say which chunk was used, or what happens on the next fifty questions. An evaluation set does. Metadata filters (tenant, brand, “published”) belong before similarity, or you will retrieve another customer’s document.

Starter

question = "What is the return window?"
chunks = ["Returns: 30 days, unused, with receipt."]
prompt = "Use only the context.\n" + "\n".join(chunks) + "\nQ: " + question
print(prompt)

A RAG app does two jobs. A retriever finds chunks that match the question. A generator (the LLM) writes an answer from those chunks. When the answer is wrong, one of those jobs failed. Looking at the paragraph cannot tell you which.

question
  → embed
  → nearest chunks in a vector store   # retrieval
  → prompt = instructions + chunks + question
  → LLM answer                           # generation

Uploading a paper and asking one question is a demo. It does not say which chunk was used, or what happens on the next fifty questions. An evaluation set does.

Example: an electronics help bot

Replace the keyword overlap with a real embedding model when you leave the tutorial. Keep k small. Add a reranker if the right row is in the top 20 but not the top 3. Tell the model to abstain when the context is silent.

Starter

catalog = ["Returns: 30 days, unused.", "Shipping: free over $50."]

def retrieve(query: str):
    return [row for row in catalog if any(w in row.lower() for w in query.lower().split())]

print(retrieve("return window"))

A shop, call it TechNest, has a short catalog and a few policies. The bot should answer only from those rows.

CATALOG = [
    {"id": "p1", "text": "NovaBuds: wireless earbuds, 24h battery, 1-year warranty."},
    {"id": "p2", "text": "Returns: 30 days, unused, with receipt. Opened earbuds are final sale."},
    {"id": "p3", "text": "Shipping: free over $50, otherwise $6. Delivered in 3 to 5 business days."},
]

def retrieve(query: str, k: int = 3) -> list[str]:
    # Replace this keyword overlap with real embeddings in production.
    scored = []
    words = set(query.lower().split())
    for row in CATALOG:
        overlap = len(words & set(row["text"].lower().split()))
        scored.append((overlap, row["text"]))
    scored.sort(reverse=True)
    return [text for score, text in scored[:k] if score > 0]

def answer(query: str) -> dict:
    chunks = retrieve(query)
    # Real app: send chunks to the model and require it to use only them.
    return {"question": query, "contexts": chunks}

Production retrieval uses an embedding model (Gemini, OpenAI, or a local MiniLM) and a vector store. Add metadata filters (product line, tenant id, “published”) before similarity, or you will retrieve another customer’s document.

Chunking, top-k, and rerankers

Overlap keeps a rule that crosses a boundary. Prefer paragraph or heading boundaries over a hard character cut when the source is markdown. Do not send all 10 hits. Drop weak scores and leave room for the system prompt and the answer.

Starter

text = "Returns: 30 days if unused. Opened earbuds are final sale."
size = 28
chunks = [text[i:i+size] for i in range(0, len(text), size)]
print(chunks)  # the exception may fall into the next chunk; overlap fixes that

• Chunks too big: the right sentence is buried, the prompt is expensive, the model gets distracted.
• Chunks too small: “30 days” is retrieved without “opened earbuds are final sale”.
• No overlap: a rule split across a heading boundary is invisible.
• Top-k too high: you pay to show the model junk. Rerank the top 20 down to the best 4.

RAGAS: a score for each half

Faithfulness: do the claims follow from the chunks? Answer relevancy: does it address the question? Context precision: are useful chunks ranked above noise? Context recall: did search find what the reference answer needed? Answer correctness: does it match the expected answer? A judged score wobbles. A jump from 0.55 to 0.82 matters. A jump from 0.81 to 0.83 might be noise.

Starter

row = {
    "faithfulness": 0.4,   # answer invented facts
    "context_recall": 0.9, # the right chunk was there
}
print("fix the writer" if row["context_recall"] > row["faithfulness"] else "fix search")

RAGAS (Retrieval Augmented Generation Assessment) uses an LLM as a judge. A low number points at the part to fix. Five metrics cover the usual lab:

You can also score without a model: exact match, “does this string appear?”, overlap of chunk ids. Use those for cheap checks. Use the judge when the wording can vary and still be right.

One sample, a judge, an experiment

Cheap checks need no judge: exact match, “does this string appear?”, overlap of chunk ids. Use a judge when the wording can vary. Pin the library version. Import paths move. The judge is another model call, so it costs money and it is not perfectly stable.

Starter

sample = {
    "user_input": "What is the return window for earbuds?",
    "retrieved_contexts": ["Returns: 30 days, unused, with receipt."],
    "response": "Unused earbuds can be returned within 30 days with a receipt.",
    "reference": "30 days, unused, with receipt.",
}
print(sample["reference"][:2], "matches the spirit of the response")

A SingleTurnSample is one question, the chunks that came back, the answer, and the answer you expected.

from ragas import SingleTurnSample
from ragas.metrics import Faithfulness
from ragas.llms import llm_factory

sample = SingleTurnSample(
    user_input="What is the return window for NovaBuds?",
    retrieved_contexts=[
        "Returns: 30 days, unused, with receipt. Opened earbuds are final sale."
    ],
    response="You can return unused NovaBuds within 30 days with a receipt.",
    reference="Unused NovaBuds can be returned within 30 days with a receipt.",
)

# llm_factory wraps a hosted model so it can judge.
# judge = llm_factory("groq/openai/gpt-oss-120b")
# score = Faithfulness(llm=judge).score(sample)

The names in RAGAS 0.4: llm_factory for the judge, Dataset plus @experiment for a saved run over a file of goldens, and the older evaluate() one-call API you will see in most existing blog posts. Pin the version. The import paths have moved.

Labs often use one key for the answering model (Groq) and another for embeddings (Gemini or a local model). The judge is a second model call. It is not free, and it is not perfectly stable. Read scores as a direction. A move from 0.55 to 0.82 matters. A move from 0.81 to 0.83 might be noise.

How to read a bad run

Sort questions by correctness. Low context recall means the generator never saw the fact. High recall and low faithfulness means the instruction is loose. Low precision with high recall means you retrieved the answer plus junk: lower k or rerank. Do not tune on the only test set you have.

Starter

recall, faith = 0.55, 0.90
if recall < 0.7:
    print("fix chunking, embeddings, or filters")
elif faith < 0.7:
    print("tighten the prompt and add a refusal example")

1. Sort questions by answer correctness.
2. If context recall is low, stop. The generator never saw the fact. Inspect the chunk and the embedding of that question.
3. If recall is high and faithfulness is low, the prompt is letting the model improvise. Tighten “use only the context” and show a refusal example.
4. If precision is low but recall is high, you retrieved the answer plus a lot of junk. Lower k or rerank.

Practice

• Add a golden whose answer is “I don’t know” because the catalog never mentions gift cards.
• Break retrieval on purpose (k = 1 and a vague chunk) and predict which metric falls.
• Write one sentence each for faithfulness vs answer relevancy, using the earbuds policy.
• Name one metadata filter you would add if the catalog had two brands.