Week 3 — ComplianceGPT Lab REU 2026

Prompt Engineering

Jul 7–10 · Prompting strategies for extraction · Deliverable: project proposal presentation (5 min)

Learning Goals

Understand zero-shot, few-shot, and chain-of-thought prompting
Know why structured output (JSON) is hard to get reliably
Modify a prompt and measure whether it helps or hurts
Present a clear, scoped research proposal to the group

Never prompted an LLM deliberately before? Do Prompting 101 first — it's generic (no HIPAA, no repo), covers zero/few-shot and chain-of-thought from scratch, and everything below will make more sense after it.

Two bonus slide decks this week — optional but recommended before/after the core material: LLM Foundations in Practice (decoding, instruction-tuning, scale — the bridge from Week 2's transformer math to prompting) and Agentic Systems (reflection vs. tool-calling, using our own 4-pass extractor as the case study, plus a bonus assignment — see Assignments).

How This Week Is Taught

Every teaching day this week follows the same pattern: generic 101 concept first, then notebook practice to apply it. Day 2 is the one exception — it's presentations, not a lecture day.

Day 1 — Monday, July 7: LLMs & Prompting Foundations (self-paced / flexible)

Step	Material	What it does
1. Generic 101	slides-prompting-101.html (21 slides)	Universal prompting literacy: what an LLM/prompt is, task+context+format+constraints, zero/few-shot, CoT, role prompting, format control, failure modes, debugging checklist. Zero HIPAA content.
2. Generic-but-deeper 101	slides-llm-foundations.html (18 slides)	The "why" behind step 1's rules: decoding (temperature/top-k/top-p), instruction-tuning/RLHF, context windows, in-context learning, real scaling data (RQ5 table). Deliberately no-code — pre-reading, not a lab.
3. Applied recap	slides-week3.html, slides 1–9	Same zero/few-shot/CoT ideas, now on a real failing HIPAA case (row_id 40) — the bridge from generic to applied.
4. Notebook practice	`notebooks/week3_prompting.ipynb`	Hands-on: write zero-shot/few-shot/CoT prompts against the real case via local Ollama. Exercise 5 ("Does Temperature Matter?") is the coding payoff for step 2's decoding slide — temp=0 vs 0.9, 3 runs each.

Deliverable: prompting exercises notebook, flexible deadline (see Assignments).

Day 2 — Wednesday, July 9: Project Proposals (on-campus, live — not a 101+notebook day)

This is presentation day, not lecture day — don't force new teaching content into it. If you want to use the 10:00–10:20 mini-lecture slot for something, options are:

Close out slides-week3.html's "Why Extraction Is Hard" / "Prompt Engineering Levers" / live-coding on connector/llm1_extractor.py (lines 87–270), or
A 5-minute teaser of slides-agentic-systems.html slides 1–3 (the agency spectrum) to prime Friday.

Then 10:50-onward is the real activity: 5-minute oral proposal + 2-minute Q&A per student. That's today's "practice" — no notebook needed.

Day 3 — Friday, July 10: Extraction Depth & Agentic Systems

Step	Material	What it does
1. Applied bridge (if not done Wed)	slides-week3.html extraction-is-hard / levers / live-coding slides	Closes the loop on why prompting alone isn't enough for messy legal text.
2. Generic 101	slides-agentic-systems.html (19 slides)	The agency spectrum (single-shot → reflection → tool-calling → multi-agent), grounded in `extract_with_reflection()` and the real Qwen-vs-Gemma ablation (RAG regression + reflection recovery).
3. Notebook practice	`notebooks/teach_07_agentic.ipynb`	Part 1 runs cold on real CSVs, proves the regression/recovery row-by-row. Part 2 (optional, live) builds a one-step reflective wrapper on the same row_id 40 case from Day 1 — a deliberate callback.

Deliverable: agentic mini-lab (flexible, see Assignments) + written project proposal (hard deadline, Fri).

Two things worth knowing, not fixing: Day 1's notebook doesn't drill context windows / instruction-tuning / scaling hands-on — that's intentional, those are conceptual, not code exercises (temperature is the one LLM-Foundations concept that does get coded, in Exercise 5). And Day 2 genuinely has no room for new teaching — if that ever feels thin, the Agentic teaser is the cheapest thing to slot in without cutting proposal time.

Day 1 — Monday, July 7: Prompting Strategies

💬 Topics

Zero-Shot

Give the model only instructions, no examples. Fastest to iterate. Works surprisingly well for familiar tasks. Fails on novel formats or edge cases.

Extract from this scenario:
{scenario}
Return JSON with:
sender_role, receiver_role,
purpose, phi_involved

Few-Shot

Provide 2–5 worked examples before the query. Dramatically improves consistency of format. Key: examples must cover edge cases you care about.

Example 1:
Scenario: ...
JSON: {...}

Example 2:
Scenario: ...
JSON: {...}

Now extract:
{scenario}

Chain-of-Thought

Ask the model to reason step-by-step before answering. Adds tokens but reduces errors on complex reasoning. "Let's think step by step: first identify who is sending..."

Think step by step:
1. Who is the sender?
2. What is the purpose?
3. Is PHI involved?
Then output JSON.

🎯 Day 1 Task

Take 1 wrong prediction from your Week 2 batch. Write 3 different prompts (zero-shot, few-shot with 2 examples, chain-of-thought). Run all 3 using notebooks/week3_prompting.ipynb. Does any version fix the error? Record your findings.

Day 2 — Wednesday, July 9: Project Proposals

🎤 Proposal Presentations

Each student presents a 5-minute project proposal. This is informal — slides are optional. The goal is to get early feedback before you commit to a direction.

Proposal structure (5 minutes):

Problem (1 min): What question are you investigating? Why is it interesting?
Method (2 min): What experiments will you run? What variables will you change? What will you measure?
Expected result (1 min): What do you expect to find?
Scope (1 min): What can realistically be done in 3 weeks? What's your Week 5 deliverable?

Feedback norms

After each proposal: 2 minutes of questions. We ask "is this feasible in 3 weeks?" and "what's the one most important thing to nail?" Criticism is direct but not harsh. The goal is to help, not to impress.

Day 3 — Friday, July 10: The Extraction Task in Depth

🔍 Topics

Why Extraction is Hard

Legal text is dense and ambiguous by design
Role ambiguity: "the hospital's social worker" — is the social worker a provider or an individual?
Implicit purpose: "called to discuss the situation" — what is the purpose?
Hallucination: model invents fields that aren't in the scenario
Format failures: model outputs prose instead of JSON

Prompt Engineering Levers

Temperature — lower = more consistent, higher = more creative
System prompt — instructions before the scenario
Output format constraints — JSON schema, strict mode
Negative examples — "do NOT put X in extra_facts"
Reasoning traces — ask model to explain its choices

Live coding: Open connector/llm1_extractor.py (prompt section only — lines 87–270). We'll read the existing rules together and discuss which ones address which failure modes.

Week 3 Deliverable

Due: Wednesday, July 9 (proposal) + Friday, July 11 (written)

1. Proposal presentation (5 min) — Wednesday morning. See structure above.

2. Written proposal (1 page) — Due Friday. Structure:

Research question — one clear sentence ending in a question mark
Background — 2 sentences on what's already known
Method — what experiments, what variables, what you'll measure
Expected results — your hypothesis
Timeline — Week 4, Week 5, Week 6 milestones

Project Options

Choose one — or propose your own with my approval. See Student Projects for full descriptions.

LLM Benchmark Comparison

Compare Gemma3, Llama, Qwen on the same 137-row HIPAA benchmark. Analyze where each model fails. What failure modes are model-specific vs. universal?

Adversarial Scenarios

Write 30 edge cases that are legally permitted but sound like violations (or vice versa). Test whether models can handle them. Analyze failure patterns.

Prompt Engineering Study

Systematically test 5 prompting strategies (zero-shot, few-shot, CoT, system prompts, schema constraints) on 30 scenarios. Which strategy maximizes extraction accuracy?

Explanation Quality Study

ComplianceGPT generates explanations for its verdicts. Are they correct? Faithful? Useful? Build a rubric and rate 50 explanations. What makes a good AI explanation?

GDPR Generalization Study

Does prompting strategy transfer from HIPAA to GDPR? Run the same 5 prompts on both regulations. Measure whether cross-regulation transfer holds or breaks.

Error Taxonomy

Categorize all wrong predictions from 3 models into a taxonomy (role error, purpose error, hallucination, etc.). Is there a systematic pattern? Does it predict which prompts will fix it?

Week 3 — Prompt Engineering