Assignments

Part 1 — Pipeline run: Run batch_runner.py on 5 rows from the GoldCoin dataset. Save as results/[yourname]_week1.csv and commit to GitHub.

Part 2 — Reflection (1 page, ~1 paragraph per question):

• What surprised you most about how ComplianceGPT works?
• Did any of your 5 results surprise you? Why?
• What question about AI and compliance are you most curious about?

Submit: Post CSV filename + Google Doc link in Slack #deliverables.

Part 1 — Notebook (all code + reflection cells):

• Ex 5 — Token counting: Count tokens for 3 scenarios, compute % of Gemma3's 8,192 context window, reflect on truncation risk
• Ex 6 — Precision / Recall / F1: Implement from scratch (no sklearn), pass all 3 test cases, write interpretation of always-DENIED model result
• Ex 7 — Batch error analysis: Group errors by purpose category using pandas groupby, identify worst-performing category
• Ex 8 — Oracle predicate analysis: Use Counter to find which oracle predicate over-fires in FP rows
• Ex 9 — Cluster extraction run: Write SLURM job script, submit on cluster, download CSV, run Soufflé locally, compute accuracy

Part 2 — Cluster run evidence (post in Slack #deliverables):

• Output of sacct -u yournetid — screenshot or paste showing ≥1 completed job
• Filename of your downloaded extraction CSV (e.g. extractions_48305.csv)
• One sentence: "I ran X scenarios with [model], got Y% accuracy, most common error was [oracle predicate]."

Part 3 — Error analysis (1 page): Pick 2 wrong predictions from your cluster run. For each:

• 1-sentence scenario summary
• Model's verdict vs. correct answer
• Which oracle predicate is wrong and what it should be
• Root cause: hallucination / under-extraction / role confusion / purpose OOV
• One proposed fix

Part 1 — Prompting notebook (no hard deadline, complete before Friday):

• Ex 1 — Zero-shot (worked example): run it, read the real hallucination it produces
• Ex 2 — Few-shot: write 2 examples of your own, shaped like the failing case
• Ex 3 — Chain-of-thought: write your own 4–5 step reasoning instructions
• Ex 4 — Compare: build the 3-strategy comparison table; answer the reflection question
• Ex 5 — Temperature (bonus): temp=0 vs temp=0.9, 3 runs each
• Ex 6 — Your own case: repeat Exercises 1–4 on a wrong prediction from your Week 2 cluster run

Runs entirely on your laptop via local Ollama (gemma3:4b) — no cluster, no API key. Submit in Slack #deliverables when complete.

Part 2 — Oral proposal (5 min) — Wednesday July 9: In-class pitch. Cover:

1. The problem (why does this matter?)
2. Your specific research question (one sentence)
3. Your method (what experiments, data, metric)
4. How you'll know if it worked

Part 3 — 1-page written summary — due Friday July 11:

• Research question — 1 sentence, specific and answerable
• Background — 2–3 sentences: gap and motivation
• Method — bullet list of experiments, data, and metric
• Expected result — hypothesis with reasoning
• Milestones — what will you have by Jul 18? By Jul 25?

Graded on specificity. "I will run experiments" is not a milestone.

Companion to Agentic Systems slides and teach_07_agentic.ipynb (watch it built live in class first, then adapt it yourself).

1. Pick 15 GoldCoin-HHS scenarios — include the wrong prediction you used in the Week 3 prompting exercise.
2. Run each through baseline extract() — one pass, no reflection.
3. Build your own one-step reflection wrapper: re-prompt the model with its own extracted facts and ask it to double-check the riskiest field(s), the same pattern shown live in the notebook.
4. Run all 15 through your wrapper.

Report: accuracy before vs. after, and for every case that flipped (right→wrong or wrong→right) one sentence on why — grounded in the actual scenario text, not a guess.

This is a small-scale rehearsal of the real Qwen2.5:72B ablation shown in class, where the same reflection pattern recovered 19 of 21 scenarios that raw RAG had broken. Directly useful groundwork if your project is Prompt Engineering Study or Error Taxonomy.

Full brief: Compliance QA Playground. Build a Streamlit app answering HIPAA compliance questions three ways — Baseline (raw LLM), RAG (retrieval over the real eCFR text), and Agentic (a chained pipeline of specialized prompts) — in both Single Q&A and Chat modes. Pure LLM/NLP: no formal verifier involved anywhere in this one. Every answer must show a verdict + full citation(s), and you'll build your own 15–20 question test set from the real HHS HIPAA FAQ. HIPAA only for now — see the brief for the multi-regulation parity requirement if you extend it.

Submit: link to your app's repo + a short (½–1 page) written comparison of how the three frameworks answered the same test questions. Post in Slack #deliverables when done.

Mid-point pitch (5 min) — Wednesday July 16: Show actual numbers. Structure:

1. Done — what experiments ran?
2. Found — the numbers (table or chart)
3. Blocked — what's not working?
4. Plan — exactly what happens in Weeks 5–6?

Experiment log — due Friday July 18: Running log of every experiment in Weeks 3–4. Per experiment:

• Date · configuration (model, prompt version, data slice)
• Numeric result
• 1-sentence observation
• 1-sentence next step

Minimum 5 experiments logged. At least 1 summary table or chart.

Poster draft: All sections filled (not placeholders). Main result visible as table or chart. Feedback returned by Mon Jul 28.

2-page write-up draft: Introduction · Background · Method · Results · Discussion/Conclusion. References required.

Final results CSV: Commit results/[yourname]_final.csv to GitHub. No new experiments after this date.

Final poster (print-ready PDF) — Thursday July 30: 48×36 in · min 24pt body · min 56pt title · all required sections · at least 1 figure · SBU affiliation line.

Final write-up — Thursday July 30: Polished 2-page research summary with all feedback incorporated.

Closing Ceremony — Thursday July 31: Stand at your poster and present to faculty and guests. Prepare a 2-minute summary you can give without notes.