Runs in your browser — no install. (Works once the repo is public.)

In [ ]:

# --- Colab setup (skipped when running locally) ---
import os, sys
if 'google.colab' in sys.modules and not os.path.exists('data/sample_corpus.json'):
    !git clone -q https://github.com/mdhabibi/llm-search-handbook.git
    %cd llm-search-handbook
    !pip -q install -r requirements.txt

# --- Colab setup (skipped when running locally) --- import os, sys if 'google.colab' in sys.modules and not os.path.exists('data/sample_corpus.json'): !git clone -q https://github.com/mdhabibi/llm-search-handbook.git %cd llm-search-handbook !pip -q install -r requirements.txt

Chapter 10 — RAG: Retrieval-Augmented Generation¶

Retrieve relevant passages, then have an LLM answer from them — grounded, current, citable. This assembles the whole course into one pipeline.

Retrieval + prompt assembly run and are tested without a model. Generation downloads a small model on first run, or plug in your own generate().

Setup¶

pip install sentence-transformers transformers

In [ ]:

# Bootstrap: locate repo root and import shared helpers
import sys, os, json
d = os.getcwd()
while d != os.path.dirname(d) and not os.path.exists(os.path.join(d, 'data', 'sample_corpus.json')):
    d = os.path.dirname(d)
ROOT = d; sys.path.insert(0, os.path.join(ROOT, 'src'))
import numpy as np
from corpus import load_corpus, tokenize

# Bootstrap: locate repo root and import shared helpers import sys, os, json d = os.getcwd() while d != os.path.dirname(d) and not os.path.exists(os.path.join(d, 'data', 'sample_corpus.json')): d = os.path.dirname(d) ROOT = d; sys.path.insert(0, os.path.join(ROOT, 'src')) import numpy as np from corpus import load_corpus, tokenize

1. Retrieve relevant passages¶

Reuse the Chapter 5 dense retriever (swap in hybrid+rerank for production).

In [ ]:

from semantic_search import SemanticSearch
from sentence_transformers import SentenceTransformer
docs = load_corpus()
bi = SentenceTransformer('all-MiniLM-L6-v2')
engine = SemanticSearch(bi.encode).index(docs)

def retrieve(question, k=3):
    return [d for _,_,d in engine.search(question, k=k)]

for d in retrieve('what is the largest animal that has ever lived?'):
    print(' -', d['title'])

from semantic_search import SemanticSearch from sentence_transformers import SentenceTransformer docs = load_corpus() bi = SentenceTransformer('all-MiniLM-L6-v2') engine = SemanticSearch(bi.encode).index(docs) def retrieve(question, k=3): return [d for _,_,d in engine.search(question, k=k)] for d in retrieve('what is the largest animal that has ever lived?'): print(' -', d['title'])

2. Augment: build a grounded prompt¶

The prompt enforces grounding (use ONLY the context), allows refusal, and tags passages with ids for citation.

In [ ]:

def build_prompt(question, passages):
    context = '\n'.join(f"[{p['id']}] {p['text']}" for p in passages)
    return (
        'Answer the QUESTION using ONLY the CONTEXT below.\n'
        'If the context does not contain the answer, say '
        '"I don\'t know based on the provided context."\n'
        'Cite the passages you used by their [id].\n\n'
        f'CONTEXT:\n{context}\n\n'
        f'QUESTION: {question}\n\nANSWER:'
    )

q = 'what is the largest animal that has ever lived?'
print(build_prompt(q, retrieve(q)))

def build_prompt(question, passages): context = '\n'.join(f"[{p['id']}] {p['text']}" for p in passages) return ( 'Answer the QUESTION using ONLY the CONTEXT below.\n' 'If the context does not contain the answer, say ' '"I don\'t know based on the provided context."\n' 'Cite the passages you used by their [id].\n\n' f'CONTEXT:\n{context}\n\n' f'QUESTION: {question}\n\nANSWER:' ) q = 'what is the largest animal that has ever lived?' print(build_prompt(q, retrieve(q)))

3. Generate an answer¶

A small open-source LLM reads the prompt. generate() is the only thing to change to use a bigger model or a hosted API.

In [ ]:

from transformers import pipeline
llm = pipeline('text2text-generation', model='google/flan-t5-base')
def generate(prompt, max_new_tokens=128):
    return llm(prompt, max_new_tokens=max_new_tokens)[0]['generated_text']

def rag_answer(question, k=3):
    passages = retrieve(question, k=k)
    prompt = build_prompt(question, passages)
    answer = generate(prompt)
    cited = [p['id'] for p in passages]
    return answer, cited

ans, cited = rag_answer('what is the largest animal that has ever lived?')
print('ANSWER:', ans)
print('grounded in passages:', cited)

from transformers import pipeline llm = pipeline('text2text-generation', model='google/flan-t5-base') def generate(prompt, max_new_tokens=128): return llm(prompt, max_new_tokens=max_new_tokens)[0]['generated_text'] def rag_answer(question, k=3): passages = retrieve(question, k=k) prompt = build_prompt(question, passages) answer = generate(prompt) cited = [p['id'] for p in passages] return answer, cited ans, cited = rag_answer('what is the largest animal that has ever lived?') print('ANSWER:', ans) print('grounded in passages:', cited)

4. Grounding demo: refusing the unanswerable¶

Ask something the corpus can't answer. A grounded system should decline, not hallucinate.

In [ ]:

ans, cited = rag_answer('who won the 2019 Nobel Prize in Physics?')
print('ANSWER:', ans)
print('(Corpus has nothing on this -> the model should refuse rather than invent.)')

ans, cited = rag_answer('who won the 2019 Nobel Prize in Physics?') print('ANSWER:', ans) print('(Corpus has nothing on this -> the model should refuse rather than invent.)')

Takeaway & exercises¶

RAG = retrieve → augment → generate. Retrieval quality (Chapters 2–9) is what makes the answers trustworthy. Chapter 11 makes the ingestion side production-ready (chunking, pipelines).

Exercises

1. Replace retrieve with hybrid + cross-encoder re-rank. Do answers improve?
2. Vary k (1, 3, 6). When does more context help vs. hurt?
3. Swap generate() for a larger model or an API. Keep everything else identical.
4. Add answer-faithfulness checking: does the answer only use facts from the cited passages?