Curriculum

Week 4 — Retrieval-Augmented Generation

July 14–17 · Embeddings · Vector search · RAG in ComplianceGPT · Project work begins

W4

Retrieval-Augmented Generation

Jul 14–17  ·  RAG concepts + ComplianceGPT RAG strategy  ·  Deliverable: mid-point pitch (5 min) + preliminary results

Learning Goals

Also this week — Contextual Integrity & Compliance (slides + notebooks/contextual_integrity.ipynb): the theory this whole system is built on — Nissenbaum's CI 5-tuple, why it alone isn't enough (with a real example from this lab's own data), and how to reframe your project's research question in CI terms. Worth doing before your mid-point pitch — it'll sharpen how you describe your own results.

Day 1 — Monday, July 14: What is RAG?

🔍 Topics

The Problem RAG Solves

  • LLMs have a knowledge cutoff — they don't know your documents
  • Fine-tuning is expensive and doesn't always generalize
  • Solution: give the model relevant context at query time
  • RAG = Retrieval + Augmented Generation

The RAG Pipeline

  1. Chunk the knowledge base into segments
  2. Embed each chunk as a vector
  3. Store vectors in a vector database
  4. Query: embed the question, find nearest chunks
  5. Inject retrieved chunks into the prompt
  6. Generate: LLM answers with context

Analogy

RAG is like an open-book exam. Without RAG, the model takes a closed-book exam — it has to rely on what it memorized during training. With RAG, you hand it the relevant textbook pages before it answers. The model doesn't need to have memorized HIPAA — it gets the relevant section retrieved and injected into its prompt at runtime.

🎯 Day 1 Task

Open notebooks/week4_rag_intro.ipynb. Walk through: embedding a sentence using sentence-transformers, computing cosine similarity between two sentences, finding the most similar HIPAA section from a list of 5. Run all cells. Modify one cell to test a sentence of your own.

Day 2 — Wednesday, July 16: RAG in ComplianceGPT + Mid-Point Pitches

📡 Topics

How ComplianceGPT Uses RAG

  • Strategy: "rag" and "rag_bm25" in batch_runner.py
  • Retrieval corpus: HIPAA sections, preamble, enforcement guidance
  • Query: the last 3 sentences of the scenario (not the full text)
  • Why 3 sentences: full scenario floods BM25 with unrelated terms
  • Retrieved sections injected into LLM1 extraction prompt

Vector Search vs. BM25

  • BM25: keyword-based scoring (TF-IDF variant). Fast, interpretable
  • Vector search: semantic similarity via embeddings. Better for paraphrase
  • Hybrid: rank-bm25 combines both signals
  • ComplianceGPT finding: BM25 outperforms pure vector on this corpus
  • Your project might test: why? What scenarios does each strategy handle better?
🎤 Mid-Point Pitches

Each student presents a 5-minute update. Structure:

  1. What I've done — experiments run, data collected so far
  2. What I've found — even preliminary: trends, surprises, anything
  3. What's blocked — any technical issues, unclear methodology, missing data
  4. Plan for weeks 5–6 — what experiments remain, what you'll write up
🎯 Day 2 Task

Run the same 20 scenarios with --strategy formal and --strategy rag_bm25. Compare accuracy. Are there rows where RAG helps? Where does it hurt? Write 3 sentences summarizing what you observe.

Day 3 — Friday, July 17: Chunking, Embedding Models, and Limits

📦 Topics

Chunking Strategies

  • Fixed-size: split every N tokens. Simple but breaks sentences
  • Sentence-based: split on punctuation. Cleaner but variable size
  • Semantic: group sentences by topic. Best quality, slower
  • For legal text: section-based is often best (HIPAA § per chunk)

Embedding Model Choices

  • all-MiniLM-L6-v2: fast, small, good for short text
  • all-mpnet-base-v2: slower, higher quality
  • text-embedding-3-small (OpenAI): very good, not free
  • Legal-specific models: legal-bert, law-ai/InLegalBERT

Week 4 Deliverable

Due: Wednesday, July 16 (pitch) + Friday, July 18 (written)

1. Mid-point pitch (5 min) — Wednesday morning. See structure above.

2. Experiment log + preliminary results — Friday. A structured log of every experiment you've run:

  • Date, experiment name, configuration (model, strategy, rows)
  • Result (accuracy or relevant metric)
  • Observation (1 sentence: what did you learn?)

Also: at least 1 result table or chart — even if preliminary. Something visual showing a comparison between conditions.

Recommended Reading This Week