Runs in your browser — no install. (Works once the repo is public.)
# --- Colab setup (skipped when running locally) ---
import os, sys
if 'google.colab' in sys.modules and not os.path.exists('data/sample_corpus.json'):
!git clone -q https://github.com/mdhabibi/llm-search-handbook.git
%cd llm-search-handbook
!pip -q install -r requirements.txt
Chapter 10 — RAG: Retrieval-Augmented Generation¶
Retrieve relevant passages, then have an LLM answer from them — grounded, current, citable. This assembles the whole course into one pipeline.
Retrieval + prompt assembly run and are tested without a model. Generation downloads a small model on first run, or plug in your own
generate().
Setup¶
pip install sentence-transformers transformers
# Bootstrap: locate repo root and import shared helpers
import sys, os, json
d = os.getcwd()
while d != os.path.dirname(d) and not os.path.exists(os.path.join(d, 'data', 'sample_corpus.json')):
d = os.path.dirname(d)
ROOT = d; sys.path.insert(0, os.path.join(ROOT, 'src'))
import numpy as np
from corpus import load_corpus, tokenize
1. Retrieve relevant passages¶
Reuse the Chapter 5 dense retriever (swap in hybrid+rerank for production).
from semantic_search import SemanticSearch
from sentence_transformers import SentenceTransformer
docs = load_corpus()
bi = SentenceTransformer('all-MiniLM-L6-v2')
engine = SemanticSearch(bi.encode).index(docs)
def retrieve(question, k=3):
return [d for _,_,d in engine.search(question, k=k)]
for d in retrieve('what is the largest animal that has ever lived?'):
print(' -', d['title'])
2. Augment: build a grounded prompt¶
The prompt enforces grounding (use ONLY the context), allows refusal, and tags passages with ids for citation.
def build_prompt(question, passages):
context = '\n'.join(f"[{p['id']}] {p['text']}" for p in passages)
return (
'Answer the QUESTION using ONLY the CONTEXT below.\n'
'If the context does not contain the answer, say '
'"I don\'t know based on the provided context."\n'
'Cite the passages you used by their [id].\n\n'
f'CONTEXT:\n{context}\n\n'
f'QUESTION: {question}\n\nANSWER:'
)
q = 'what is the largest animal that has ever lived?'
print(build_prompt(q, retrieve(q)))
3. Generate an answer¶
A small open-source LLM reads the prompt. generate() is the only thing to change to use a bigger model or a hosted API.
from transformers import pipeline
llm = pipeline('text2text-generation', model='google/flan-t5-base')
def generate(prompt, max_new_tokens=128):
return llm(prompt, max_new_tokens=max_new_tokens)[0]['generated_text']
def rag_answer(question, k=3):
passages = retrieve(question, k=k)
prompt = build_prompt(question, passages)
answer = generate(prompt)
cited = [p['id'] for p in passages]
return answer, cited
ans, cited = rag_answer('what is the largest animal that has ever lived?')
print('ANSWER:', ans)
print('grounded in passages:', cited)
4. Grounding demo: refusing the unanswerable¶
Ask something the corpus can't answer. A grounded system should decline, not hallucinate.
ans, cited = rag_answer('who won the 2019 Nobel Prize in Physics?')
print('ANSWER:', ans)
print('(Corpus has nothing on this -> the model should refuse rather than invent.)')
5. Answer length: what max_new_tokens really does¶
That parameter is a hard cap on how many tokens the model may emit. Hit the cap and the answer simply stops -- often mid-sentence. No error, no warning. Let's see it deliberately.
q = 'what is the largest animal that has ever lived?'
passages = retrieve(q, k=3)
prompt = build_prompt(q, passages)
for cap in (8, 24, 128):
out = generate(prompt, max_new_tokens=cap)
print(f'max_new_tokens={cap:>4} -> {out!r}\n')
The short caps don't produce a worse answer -- they produce a truncated one. If your RAG output ever stops mid-word, check this setting before you blame the model or the retrieval.
Both halves of the budget matter: the prompt (question + passages) and the answer share the model's context window. More passages means less room to answer in.
6. The same prompt, several times¶
Retrieval is deterministic -- rerun a query and the ranking is identical. Generation usually is not: most LLMs sample the next token from a probability distribution, so the same prompt can yield different wording each time.
Running one prompt several times is the quickest way to see whether an instruction reliably works, or whether a good result was luck.
def generate_n(prompt, n=3, max_new_tokens=128, **kw):
"""Sample the same prompt n times (do_sample=True makes the variation visible)."""
outs = llm(prompt, max_new_tokens=max_new_tokens,
do_sample=True, num_return_sequences=n, **kw)
return [o['generated_text'] for o in outs]
print('Same prompt, 3 samples (temperature=1.0):')
for i, g in enumerate(generate_n(prompt, n=3, temperature=1.0), 1):
print(f' {i}. {g}')
print('\nSame prompt, 3 samples (temperature=0.2 - much more consistent):')
for i, g in enumerate(generate_n(prompt, n=3, temperature=0.2), 1):
print(f' {i}. {g}')
Temperature sets how adventurous the sampling is. Near 0 the model nearly always takes
the most likely token -- repetitive but stable. Higher values flatten the distribution: more
varied, more creative, more prone to drifting off the passages.
For factual RAG you usually want low temperature: the job is to report the retrieved evidence faithfully, not to improvise.
Why this matters for evaluation. A RAG answer scored from a single run partly measures luck. Retrieval metrics (Chapter 9) are deterministic and cheap, which is why they're the dependable half of RAG evaluation -- and why you should sample generation several times before concluding a prompt change helped.
Note: with do_sample=False (the default in our generate()), the model uses greedy decoding
and repeated runs agree -- the variation above is opt-in, so our earlier results stay
reproducible.
Takeaway & exercises¶
RAG = retrieve → augment → generate. Retrieval quality (Chapters 2–9) is what makes the answers trustworthy. Chapter 11 makes the ingestion side production-ready (chunking, pipelines).
Exercises
- Replace
retrievewith hybrid + cross-encoder re-rank. Do answers improve? - Vary
k(1, 3, 6). When does more context help vs. hurt? - Swap
generate()for a larger model or an API. Keep everything else identical. - Add answer-faithfulness checking: does the answer only use facts from the cited passages?
- Shrink
max_new_tokensuntil the answer truncates, then raise it. What is the smallest cap that still answers fully? - Sample one prompt 5 times at
temperature=1.0. Do all five stay faithful to the passages, or does one drift off them?