Why this exists: it's easy to run a script, get a CSV back, and never actually see what happened in between. This page has you do by hand, in the terminal — with a real row from the real dataset — exactly what batch_runner.py automates for every row. Once you've done it manually once, the script stops being magic.
0. Setup 1 min
Same as Ollama Playground — any small local model works:
ollama run llama3.21. A Real Row, As-Is 2 min
This is row_id 1 from data/goldcoin_hhs_merged.csv — a real HHS OCR enforcement case, word for word. No prompt engineering yet. Just paste this into your running ollama run session and see what comes back:
Read what you get. Is it a clear PERMITTED/DENIED, or does it hedge? Does it cite anything specific, or just talk generally about "privacy concerns"?
2. Now Ask It Like the Real Pipeline Does 5 min
batch_runner.py doesn't ask "is this legal?" directly. It asks the model to first extract five specific fields — the CI 5-tuple you saw in the Contextual Integrity slides — before any verdict is decided. Try this, same scenario:
Compare this to Step 1's answer. Notice: extraction doesn't decide PERMITTED or DENIED at all — it just pulls out facts. The verdict comes after, from a completely separate, non-AI system (the Datalog verifier). This is the trust boundary from the Contextual Integrity slides, now something you triggered yourself.
3. Check Yourself Against the Real System 3 min
Before you look — write down (mentally or on paper) what you think sender_role, receiver_role, and purpose should be for this scenario. Then reveal:
Reveal the real extraction for row_id 1
| sender_role | group-health-plan |
| receiver_role | individual |
| subject | city_employees |
| attribute | medical-record |
| purpose | law-enforcement |
| ground truth verdict | DENIED |
Notice purpose = law-enforcement — Michelson is a private citizen making a public-records request, not law enforcement. If your model (or you) picked something like "transparency" or "public interest" instead, that's a real, common extraction disagreement — the kind your project might actually be measuring.
4. Your Turn — No Peeking First 6 min
New scenario, row_id 66. Run it through Step 2's extraction prompt yourself before revealing anything:
Run the Step 2 extraction prompt on this. Then guess the verdict yourself before revealing.
Reveal the real extraction + verdict for row_id 66
| sender_role | hospital |
| receiver_role | individual | (the private investigator)
| attribute | psychotherapy-notes |
| purpose | legal-defense |
| ground truth verdict | PERMITTED |
If that surprised you, good — say so out loud. Psychotherapy notes obtained without consent, by a private investigator, sounds like it should be DENIED. This is real HHS OCR case data, so the actual determination reflects facts and legal nuance (specific procedural history, how the request was actually made) that this short summary doesn't fully capture. Bring this exact case to your mentor if you want the real reasoning — it's a genuinely good example of why extraction from narrative text is hard, not a trick question.
5. The Bridge to the Real Script
What you just did by hand — paste scenario text, extract structured fields, compare to a verdict — is exactly the loop batch_runner.py runs automatically for every row in a CSV, at whatever scale you point it at. The only things it adds: it reads the scenario from a file instead of your clipboard, it uses the real 40-field extraction prompt (not the simplified 5-field one above), and it feeds the extracted facts into the actual Datalog verifier instead of you eyeballing the answer. Same idea, more fields, automated. You're ready for Project Navigator.