Curriculum

Week 5 — ComplianceGPT Deep Dive

July 21–24 · Error analysis · Final experiments · Poster draft · Writing up

W5

ComplianceGPT Deep Dive

Jul 21–24  ·  Final experiments + results analysis  ·  Deliverable: poster draft + 2-page write-up draft

Learning Goals

Day 1 — Monday, July 21: Error Analysis Deep Dive

🔬 Topics

Systematic Error Analysis

  • Filter all wrong rows: df[df["verdict"] != df["ground_truth"]]
  • Group by error type: role error, purpose error, hallucination, format fail
  • Measure error rate per HIPAA category (LE, judicial, TPO, research...)
  • Look for patterns: does the same model fail on the same types?

The LLM Extraction Layer (Black Box Boundary)

  • LLM1: extracts structured JSON from scenario text
  • LLM2: generates natural language explanation for the verdict
  • Postprocessing: normalizes roles, injects oracle predicates
  • Compliance engine: applies HIPAA rules (you don't see the internals)
  • Your research: understand extraction quality, not engine internals

What extraction fields matter most?

Rank the fields by their impact on verdict accuracy: sender_role and purpose together account for ~70% of extraction errors. has_authorization booleans account for most of the rest. Understanding this priority order will help you write a clear "most impactful finding" for your poster.

🎯 Day 1 Task

Run your final experiments today. All data collection should be complete by end of Day 1 so you have Tue–Fri to analyze and write. Finalize your results CSV. Compute your key metric(s). Make at least 1 table and 1 chart.

Day 2 — Wednesday, July 23: Poster Draft Workshop

🖼️ Topics

Poster Structure (6-foot rule)

Your poster will be viewed by visitors who walk past. They spend 30 seconds on first pass. Structure for that scan:

  1. Title — one clear claim or question (large, top)
  2. Problem + Motivation — why does this matter? (left column)
  3. Method — what did you do? One diagram helps
  4. Results — the key table or chart (center, large)
  5. Conclusion — one sentence takeaway (right bottom)

Design Rules

  • Minimum font: 24pt body, 36pt headers, 56pt title
  • Maximum text per panel: 5 sentences. Cut the rest.
  • Every figure needs a caption with 1 takeaway sentence
  • Use your main result as the visual anchor — big, centered
  • Tools: PowerPoint, Google Slides, Canva, LaTeX beamerposter
  • Size: 48×36 inches (standard research poster)
🎯 Day 2 Task

Sketch a poster layout on paper or in slides. Bring it to Friday's session. It doesn't need to be polished — just have the structure: title, boxes for each section, placeholder for your main result figure.

Day 3 — Friday, July 25: 2-Page Write-Up Workshop

✍️ Topics

2-Page Research Summary Structure

  • Introduction (3–4 sentences): what problem, why it matters, what you did
  • Background (3–4 sentences): what's already known, what gap you fill
  • Method (4–6 sentences): experiments, data, metrics — enough to reproduce
  • Results (4–6 sentences): main table or figure, your key finding
  • Discussion (3–4 sentences): what does it mean, why, what's next
  • References (3–5 citations): papers you built on

Writing rule: claim first, evidence second

Every paragraph starts with a claim: "Few-shot prompting increased extraction accuracy by 8%." Then the evidence: "Table 2 shows accuracy improved from 74% to 82% across 137 scenarios when 3 examples were added to the prompt." Then analysis: "The improvement was largest for law enforcement scenarios (from 62% to 79%), suggesting few-shot examples help most when the exception category is rare in training data."

Week 5 Deliverable

Due: Friday, July 25 (end of day)

1. Poster draft — Does not need to be final. Must have: title, all sections sketched, your main result visible (even as a rough table). Submit as PDF or Google Slides link.

2. 2-page write-up draft — Does not need to be polished. Must have all 5 sections (intro, background, method, results, discussion). I will give detailed written feedback by Monday morning.

3. Results CSV — Final, complete experiment results in results/[yourname]_final.csv.

Recommended Reading This Week