Can you fix a wrong prediction just by changing the prompt? No new code, no new model.
No new theory today. You already know how the model works inside (Week 2). This week is 100% practical: three prompting strategies, one real failing case, and your own Ollama instance running the exact model (gemma3:4b) from the real 94.2% experiment.
Every one of those 8 errors happened with the same code, same model, same weights. The only thing that changes today is the prompt.
"If I write a smarter prompt — more examples, more structure, more reasoning steps — does the answer change? And if it does, why didn't the production system already do that?"
Give the model only the instructions and the schema. No worked examples. Fastest to write.
The task is close to something the model has seen constantly in training — simple, familiar formats.
Novel schemas, edge cases, or anything requiring the model to not follow its strongest prior (e.g. "mentions a lawsuit" ≠ "has a court order").
Show 2–5 worked (scenario → correct JSON) examples before the real question. The model pattern-matches against your examples, not just its training data.
The one rule that matters: your examples must cover the edge case you actually care about. A few-shot prompt with two easy examples won't teach the model to handle a hard one — it has to see something shaped like the failure you're trying to fix.
Ask the model to reason step-by-step before committing to an answer. More tokens, more time — but it can catch ungrounded inferences a one-shot answer would miss.
The real payoff isn't the reasoning text itself — it's forcing the model to justify each field individually instead of pattern-matching the whole scenario in one leap. "Is there explicit evidence of X?" is a different, harder question than "does this scenario feel like X?"
Case row_id 40 — real false positive, ground truth DENIED. Ran all three prompts today against gemma3:4b on a simplified 6-field schema.
| Strategy | Field that misfired | Simplified verdict | Correct? |
|---|---|---|---|
| Zero-shot | is_required_by_law = true | PERMITTED | ✗ |
| Few-shot (2 examples) | is_required_by_law = true | PERMITTED | ✗ |
| Chain-of-thought | is_business_associate = true | PERMITTED | ✗ |
None of the three naive strategies fixed it. Chain-of-thought didn't repeat the exact same mistake — it made a different one. That's the honest result you'll get in the notebook too. Hold that thought for slide 8.
Not model laziness — these are genuinely ambiguous even for a careful human reader.
Every lever above is a way of narrowing the space of plausible-sounding-but-wrong answers. None of them add new knowledge to the model — they only change which of its existing patterns gets activated by your specific wording.
Try this yourself in Exercise 5 of today's notebook: run the identical zero-shot prompt 3 times at temperature=0, and 3 times at temperature=0.9.
Always picks the single highest-probability next token. Deterministic — same input, same output, every run. This is what reproducible research requires.
Samples from the probability distribution. Same scenario can flip between PERMITTED and DENIED on different runs — your accuracy number stops meaning anything.
If your extraction pipeline doesn't set temperature=0 explicitly, you don't have one accuracy number — you have a different one every time you run it.
Today's notebook uses a 6-field toy schema. The production system (connector/llm1_extractor.py) uses 40 fields and roughly 2,000 lines of instructions.
6 fields, one line of description each. Enough to learn zero-shot vs few-shot vs CoT — not enough to actually beat the real system.
40 fields, 14 numbered rules, explicit trigger words per oracle, worked edge cases, and instructions like "a lawsuit filing, subpoena, or legal dispute alone ≠ court order" — added specifically because a real case broke the simpler version.
Every one of those 2,000 lines exists because someone ran the exact exercise you just ran, found a hallucination, and wrote one more instruction to stop it. That is quite literally what prompt engineering research looks like at scale.
Friday, July 10: open connector/llm1_extractor.py, lines 87–270, together.
"…CRITICAL: 'lawsuit', 'legal dispute', 'legal battle', 'malpractice case'… → instrument_type='none' → DENIED unless you separately find one of the specific instrument keywords above."
This is exactly the failure mode from case 40 — a legal/billing-sounding context with no actual court order — encoded as an explicit negative rule instead of hoped-for good judgment.
We'll match 3–4 more numbered rules to the failure modes on slide 7. Bring your Exercise 1–4 results — you'll recognize your own hallucination in there.
Runs entirely on your laptop — gemma3:4b via local Ollama, no cluster, no API key. If Ollama isn't running: open the Ollama app, or run ollama serve in a terminal.
5 minutes each, informal, slides optional. Goal: get early feedback before you commit to a direction for the rest of the summer.
2 minutes of questions after each proposal. We ask: "is this feasible in 3 weeks?" and "what's the one most important thing to nail?" Direct, not harsh. The goal is to help, not to impress.
Choose one, or propose your own with approval. Full descriptions on the Student Projects page.
Gemma3 vs Llama vs Qwen on the same 137-row benchmark. Model-specific vs universal failure modes?
Write 30 edge cases that sound like violations but are permitted (or vice versa). Analyze failure patterns.
Systematically test 5 strategies (today's 3 plus system prompts and schema constraints) on 30 scenarios.
Are ComplianceGPT's generated explanations correct, faithful, and useful? Build a rubric, rate 50 of them.
Does a prompting strategy that helps HIPAA also help GDPR? Run the same 5 prompts on both.
Categorize wrong predictions from 3 models into a taxonomy. Does the category predict which prompt fixes it?
5-minute pitch: problem, research question (1 sentence), method, how you'll know if it worked.
Graded on specificity. "I will run experiments" is not a milestone.
Read the abstract and Section 2 — the original definition of few-shot learning, from the paper that made it a standard technique.
Short paper. Read the introduction and Figure 1 — the original chain-of-thought result on grade-school math word problems.
| Day | Focus | Deliverable |
|---|---|---|
| Mon Jul 7 | Zero/few-shot/CoT + week3_prompting.ipynb | Notebook, flexible timing |
| Wed Jul 9 | Project proposal presentations (on campus) | 5-min oral pitch |
| Fri Jul 10 | Why extraction is hard + live code read of the real prompt | — |
| Fri Jul 11 | — | 1-page written proposal due |
Same model (gemma3:4b), same weights, same tokenizer, same attention mechanism. Nothing about the network changed.
Only the text you send it. And that alone was enough to change the answer on case 40 — three different ways, none of them fully correct.
The takeaway: prompting is real engineering, with real trade-offs and real failure modes — not guessing at magic words. That's what today's exercises, and Friday's read of the real 2,000-line prompt, are meant to prove to you directly.
Proposal presentations are Wednesday. Look back at your Week 2 project selection message and the six options on slide 14 — come Monday with a rough direction so today's exercises can double as pilot evidence for your proposal.