Why this exists: it's easy to run a script, get a CSV back, and never actually see what happened in between. This page has you do by hand, in the terminal — with a real row from the real dataset — exactly what batch_runner.py automates for every row. Once you've done it manually once, the script stops being magic.

0. Setup 1 min

0

Same as Ollama Playground — any small local model works:

ollama run llama3.2

1. A Real Row, As-Is 2 min

1

This is row_id 1 from data/goldcoin_hhs_merged.csv — a real HHS OCR enforcement case, word for word. No prompt engineering yet. Just paste this into your running ollama run session and see what comes back:

In the City of Plainfield, William H. Michelson, a concerned citizen, submitted a detailed request for access to government records. His request targeted the health insurance benefits available to city employees, officials, and their dependents over recent years. Michelson sought comprehensive details, including descriptions of health plans, costs, participant names, and claims experience, to scrutinize the city's handling of health insurance and possibly uncover inefficiencies or fraud. The City, tasked with managing employee health benefits and ensuring privacy, faced a dilemma. Sharing detailed health plan information, especially about individual claims experience and the identities of covered dependents, could infringe on the privacy of city employees and their families. Such disclosure risked violating the expectations of privacy guaranteed under state laws and potentially breaching federal regulations, which protect personal health information from unauthorized access. Despite Michelson's intention to promote transparency and accountability, the City had to navigate the complex interplay of public right to information and individual privacy rights. The City's response to Michelson, limiting the scope of disclosed information due to privacy concerns, sparked a legal challenge. Michelson argued for broader access under public records laws, while the City defended its stance by invoking privacy protections, setting the stage for a legal examination of the balance between public transparency and the safeguarding of personal health information.
>>> [paste the scenario above] Is this disclosure permitted under HIPAA?

Read what you get. Is it a clear PERMITTED/DENIED, or does it hedge? Does it cite anything specific, or just talk generally about "privacy concerns"?

2. Now Ask It Like the Real Pipeline Does 5 min

2

batch_runner.py doesn't ask "is this legal?" directly. It asks the model to first extract five specific fields — the CI 5-tuple you saw in the Contextual Integrity slides — before any verdict is decided. Try this, same scenario:

>>> Extract the following fields from this HIPAA scenario as JSON: sender (who is disclosing), receiver (who receives it), subject (whose information it is), attribute (what type of information), purpose (why the disclosure would happen). Respond with ONLY the JSON object. [paste the same scenario from Step 1]

Compare this to Step 1's answer. Notice: extraction doesn't decide PERMITTED or DENIED at all — it just pulls out facts. The verdict comes after, from a completely separate, non-AI system (the Datalog verifier). This is the trust boundary from the Contextual Integrity slides, now something you triggered yourself.

3. Check Yourself Against the Real System 3 min

3

Before you look — write down (mentally or on paper) what you think sender_role, receiver_role, and purpose should be for this scenario. Then reveal:

Reveal the real extraction for row_id 1
sender_rolegroup-health-plan
receiver_roleindividual
subjectcity_employees
attributemedical-record
purposelaw-enforcement
ground truth verdictDENIED

Notice purpose = law-enforcement — Michelson is a private citizen making a public-records request, not law enforcement. If your model (or you) picked something like "transparency" or "public interest" instead, that's a real, common extraction disagreement — the kind your project might actually be measuring.

4. Your Turn — No Peeking First 6 min

4

New scenario, row_id 66. Run it through Step 2's extraction prompt yourself before revealing anything:

In the midst of a contentious legal battle, Ross Cashen found himself charged with serious offenses stemming from a domestic dispute with Jane Doe. As part of his defense, Cashen planned to argue self-defense, necessitating a deep dive into Doe's past, specifically her mental health history. Cashen's legal team decided that understanding Doe's psychological background could be crucial to their case, revealing patterns of behavior or credibility issues that could sway the jury in Cashen's favor. To this end, Cashen hired a private investigator who successfully obtained Doe's mental health records from a hospital and a medical office. This action was taken without Doe's consent, sparking significant legal and ethical questions. The records uncovered detailed Doe's long history of mental health challenges, including diagnoses of posttraumatic stress disorder, anxiety, depression, and her experiences in therapy since a young age. Further, Doe's own deposition testimony, where she admitted to past abusive relationships and impulsive reactions, only added layers to the narrative Cashen hoped to construct for his defense. This pursuit of Doe's private health information without her consent, aimed at undermining her credibility and highlighting a propensity for violence, set the stage for a complex legal debate. It questioned the balance between a defendant's right to a thorough defense and a victim's right to privacy over her mental health records. The court was tasked with navigating this delicate balance, ensuring the legal proceedings respected both the letter and spirit of privacy laws while allowing for a fair and just trial.

Run the Step 2 extraction prompt on this. Then guess the verdict yourself before revealing.

Reveal the real extraction + verdict for row_id 66 (the private investigator)
sender_rolehospital
receiver_roleindividual
attributepsychotherapy-notes
purposelegal-defense
ground truth verdictPERMITTED

If that surprised you, good — say so out loud. Psychotherapy notes obtained without consent, by a private investigator, sounds like it should be DENIED. This is real HHS OCR case data, so the actual determination reflects facts and legal nuance (specific procedural history, how the request was actually made) that this short summary doesn't fully capture. Bring this exact case to your mentor if you want the real reasoning — it's a genuinely good example of why extraction from narrative text is hard, not a trick question.

5. The Bridge to the Real Script

What you just did by hand — paste scenario text, extract structured fields, compare to a verdict — is exactly the loop batch_runner.py runs automatically for every row in a CSV, at whatever scale you point it at. The only things it adds: it reads the scenario from a file instead of your clipboard, it uses the real 40-field extraction prompt (not the simplified 5-field one above), and it feeds the extracted facts into the actual Datalog verifier instead of you eyeballing the answer. Same idea, more fields, automated. You're ready for Project Navigator.