From hand-coded rules to Claude and Gemma — the progression that made ComplianceGPT possible
1956
AI is born
──
1986
Backprop
──
2012
Deep learning
──
2013
Word2Vec
──
2017
Transformer
──
2020
GPT-3
──
2022
ChatGPT
Every model you're working with this summer — Gemma3, Llama3, Claude Sonnet — is the product of 70 years of incremental ideas. Understanding the story helps you understand why these models succeed and fail the way they do.
O2
Origins · Machine Learning
Two Ways to Teach a Computer
This distinction — rules vs. learning — is the single most important idea in modern AI.
Problem: misses "the bench ordered disclosure." Catches "the officer mentioned a possible order." Never finishes. Breaks on every edge case.
Approach 2: Machine Learning
Show the system thousands of labeled examples:
"The judge signed a court order" → True
"The bench ordered production" → True
"He mentioned it was ordered" → False
"Acting per a judicial mandate" → True
The model learns the pattern — including edge cases it's never seen. DATA replaces LOGIC.
The insight that took 40 years to prove
Writing rules was the wrong direction. The right direction: collect enough labeled examples, let a learning algorithm figure out the rules itself. Simple in hindsight. Required decades of compute, data, and algorithms to make work.
O3
Origins · Deep Learning
Neural Networks — The Engine
Inspired by the brain, backpropagation-trained networks finally proved themselves in 2012. Three ingredients came together: data, GPUs, and better algorithms.
What a neural network does
Input layer — receives raw numbers (pixels, token IDs, embeddings)
Hidden layers — each layer transforms the representation. Deeper = more abstract features. Layer 1 might detect edges; Layer 10 might detect "court order context."
Output layer — class probabilities, next-token scores, or a regression value
Weights — the numbers connecting every node to every node in the next layer. Training adjusts these.
Backpropagation (1986) — compute how wrong the output is, propagate error backward, nudge each weight slightly in the right direction. Repeat millions of times.
The 2012 ImageNet moment
The ImageNet challenge: classify 1.2 million images into 1,000 categories. Best 2011 system: 26% error. AlexNet (deep convolutional net, trained on GPUs, 2012): 15.3% error. Next best competitor: 26.2%.
A 10-point gap, not 1. Research capital flooded into deep learning overnight. Everything changed.
Data
The internet created billions of labeled examples that didn't exist before
GPUs
Built for gaming — massively parallel — perfect for training neural nets
O4
Origins · NLP
The Language Problem — Words as Geometry
Images have pixels — easy to feed to a neural net. Text has words. The breakthrough was encoding meaning as geometry.
Semantically identical words are geometrically unrelated. No context.
Word2Vec — Mikolov et al. (2013)
Train a neural net to predict: "The [blank] is a medical facility." The hidden layer weights become the word embeddings. Similar words end up geometrically close.
sim("hospital", "clinic") → 0.87✓ close
sim("hospital", "lawsuit") → 0.12✓ far
king − man + woman ≈ queen ✓ compositionality
Semantic meaning encoded as geometry. A revolution.
Why this matters for your research
Modern LLMs use embeddings 10,000× richer than Word2Vec. When your model "knows" that "court order" and "judicial mandate" are related, it's because they're close in embedding space. This is also why adversarial prompts work: "acting pursuant to what might be a court order" embeds dangerously close to "court order."
O5
Origins · Sequence Models
The Sequential Problem — RNNs and LSTMs
Word2Vec gave words meaning. But language is ordered — word N depends on words 1 through N-1. We needed models that process sequences.
Recurrent Neural Networks (RNNs, mid-2010s)
Process one token at a time, passing a "hidden state" forward — a running memory of what came before. Sequential: token 1 → hidden state → token 2 → hidden state → …
The vanishing gradient problem: information from early tokens fades. By token 100, the model has effectively forgotten token 1.
LSTMs — Long Short-Term Memory (1997)
Added explicit memory gates to control what to remember and forget. Better at long sequences. But still sequential — can't parallelize on GPUs.
The legal text problem
"The patient, who was admitted last year following a workplace injury treated by a physician who had previously disclosed records in an unrelated matter, filed for damages claiming the hospital had improperly shared information without court authorization."
The critical fact — "without court authorization" — is at the end. The setup that makes it matter is at the beginning. By the time an LSTM reaches the end of that sentence, the beginning is attenuated. This is exactly why RNNs fail on long legal narratives.
The Transformer (2017) solved this by eliminating the sequential bottleneck entirely.
O6
Origins · The Transformer (2017)
Attention Is All You Need
Vaswani et al., Google Brain, 2017. Eight pages. Every word attends to every other word simultaneously — no sequential bottleneck, fully parallelizable on GPUs.
Self-Attention in one example
For each token, compute: how much should I attend to every other token?
"records" → 0.61 (object confirms production context)
"He" → 0.03 (pronoun, low relevance)
→ "order" = court order, not medical order ✓
Encoder vs. Decoder
Encoder — reads full input bidirectionally, builds contextual representations. Used for understanding / extraction. Model: BERT (2018).
Decoder — generates tokens left-to-right, attends to what's been generated. Used for text generation. Model: GPT (2018→).
Enc-Dec — reads input, generates output. Used for translation / summarization.
Why transformers dominate
Fully parallelizable — train on thousands of GPUs simultaneously
No vanishing gradient — all tokens equidistant in attention
Scales predictably with data and compute
O7
Origins · The LLM Era
The Scaling Hypothesis — Bigger Models Learn More
Once transformers worked, researchers discovered: just make them bigger. More parameters + more data + more compute → qualitatively new capabilities emerge.
GPT-1 — 117M params · Decoder-only · Predicts next token · Language generation begins
2019
GPT-2 — 1.5B params · "Too dangerous to release" → then released · Coherent paragraph generation · First emergence
2020
GPT-3 — 175B params · Few-shot learning: show 3 examples in the prompt, no fine-tuning needed. First model that could "read instructions."
2022
InstructGPT + RLHF — Human feedback trains the model to be helpful, not just fluent. The key step from "impressive demo" to "useful product."
Nov 2022
ChatGPT — 100 million users in 60 days. Fastest consumer product adoption in history. The world changed in 10 weeks.
2023–25
Claude, Gemini, Llama, Gemma — the models you're using this summer. 4B params today matches 175B from 2020.
O8
Origins → Your Research
Why This History Is Your Research
LLMs know HIPAA — sort of
Gemma3 was trained on trillions of tokens including legal text, HIPAA summaries, law review articles. It has a statistical representation of compliance law. It can explain HIPAA in plain English better than most non-lawyers.
But: it predicts the most likely next token, not the most legally correct one. These are not the same.
Your model in numbers
Gemma3:4B — 4 billion parameters
~10 trillion tokens of training data
8,192 token context window
Runs on a single GPU
Achieves 94.2% on GoldCoin with the right extraction layer
The gap your research closes
What LLMs do naturally
Summarize a medical case. Identify parties. Explain what HIPAA says about law enforcement disclosures.
What they fail at without engineering
Reliably extract has_court_order = False when the text says "the officer implied there might be a warrant." The word "warrant" activates; "implied" and "might" don't cancel it.
Your research question
Can we engineer LLMs precisely enough to support formal legal compliance reasoning?
1
ComplianceGPT Lab · AI Innovation & Diffusion REU 2026
Week 2 — AI Fundamentals
ML · DL · NLP · Transformers · LLMs · GPU Cluster · First Real Experiments
Monday Jun 29
Origin story · tokenizer demo · cluster email
Tuesday Jun 30
LLMs in practice · cluster setup · test job
Wednesday Jul 1
First GPU experiment · error analysis
This week is entirely AI. No regulations. Just the models, the math, and the machines. By Wednesday you will have run a real experiment on a GPU cluster and explained why a model got a prediction wrong at the level of individual embedding geometry.
2
Recap
What You Did Last Week
You ran the pipeline end-to-end
Loaded a real HIPAA court case scenario
Passed it to Gemma3 or Llama3 via the API
Got back a structured JSON extraction
Fed that JSON to the formal reasoning engine
Got a verdict: PERMITTED or DENIED
Found at least one wrong prediction
The question you should have
"The pipeline got the wrong answer. Which step failed? The LLM? The JSON? The reasoning engine? And why — at the level of how the model works?"
This week answers that question. By understanding how the model reads text, you can predict where it will fail — and design fixes before you even run the experiment.
137
GoldCoin court cases
94.2%
Best accuracy (Gemma3)
5.8pp
Gap to perfect — all extraction failures
3
How LLMs Read Text · Part 1
Tokenization — LLMs Don't Read Words
LLMs process tokens — subword units, not words. This matters for compliance because legal language tokenizes differently than casual text.
Real example: a HIPAA scenario sentence
"The hospital disclosed the patient's anti-inflammatory prescription to a law enforcement officer."
The hospital disclosed the patient's anti-inflammatory prescription to law enforcement officer
Every token is mapped to a high-dimensional vector. Semantically similar tokens are geometrically close. This is how the model "knows" that "hospital" and "clinic" are related even if the exact word doesn't appear.
Healthcare cluster (blue) vs. legal-process cluster (red). The model knows these are different concept groups without being told explicitly.
What this means for extraction
"Doctor shared records" and "Physician disclosed PHI" embed similarly → model correctly recognizes both as disclosure events
"Court order" and "judicial mandate" embed closely → model correctly sets has_court_order=true for both
The hallucination surface
"The officer said the records were needed" embeds too close to "court order" — model sets oracle True. But no court order exists. This is the mechanism behind false positives. Understanding embeddings is understanding why your FPs happen.
5
How LLMs Read Text · Part 3
Attention — Context Determines Meaning
Self-attention lets each token "look at" all other tokens when computing its meaning. This is why transformers handle legal text better than RNNs: the word "order" means something completely different depending on context.
The "order" disambiguation — live example
"The judge signed an order requiring the hospital to produce records."
→ Attends to "judge", "signed", "requiring" → court order → has_court_order = true
"The doctor placed an order for a CT scan and sent results to the insurer."
→ Attends to "doctor", "CT scan", "results" → medical order → has_court_order = false
Where attention succeeds
Disambiguating legal terms from context
Identifying direction of disclosure (sender → receiver)
Linking pronouns across a long narrative
Recognizing when a conditional clause negates permission
Where attention fails — your research
Adversarial framing: "acting pursuant to what might be a court order" — attended tokens suggest legal mandate
Role confusion: "the attorney's client" — who is the sender?
Long-range dependencies: key negation at sentence 1, permission granted at sentence 8
Legal boilerplate that resembles enabling language
6
LLMs in Practice · Training
How LLMs Are Built — The 3-Stage Recipe
Every modern LLM you use — Gemma3, Llama3, Claude — was built through the same three stages. Understanding them explains why models behave the way they do.
Stage 1
Pretraining
Train on 10+ trillion tokens from the internet: web pages, books, code, legal documents, Wikipedia. Task: predict the next token.
Result: a model that knows about the world — factual knowledge, language patterns, writing styles. It can complete sentences in any register.
⚠ At this stage the model will happily complete "How do I make a bomb..." — no safety filters yet.
Stage 2
Instruction Fine-Tuning (SFT)
Train on thousands of (instruction, good response) pairs. Task: given a user request, generate a helpful, relevant response.
Result: a model that understands what it's supposed to do when given a task. It follows instructions, uses the right format, stays on topic.
This is what makes GPT-3 → InstructGPT. Same weights, radically different behavior.
Stage 3
RLHF
Human raters rank model outputs for quality. A reward model is trained on these rankings. The LLM is fine-tuned to maximize reward.
Result: a model optimized for what humans actually prefer — accurate, helpful, harmless, honest. This is what makes ChatGPT feel so different from raw GPT-3.
RLHF also introduces subtle biases: the model may prefer confident-sounding wrong answers over uncertain-sounding correct ones.
Why this matters for extraction
RLHF trains models to sound confident. Confident-sounding wrong extractions (e.g., asserting has_court_order: true) score well on fluency but are wrong. The model is optimized for human approval, not for legal precision.
7
LLMs in Practice · Inference
What Happens at Inference
Every time you call an LLM, it's doing the same thing: predicting the next token, one at a time, until it decides to stop. Understanding this explains most of the failure modes you'll see.
The generation loop
Input tokens are processed by all transformer layers in parallel
The final layer produces a score for every token in the vocabulary (50,000+ scores)
Softmax converts scores to probabilities
One token is sampled according to those probabilities
That token is appended to the input, and the loop repeats
Continues until a special end-of-sequence token is generated
Context window = your budget
Input tokens + output tokens must fit in 8,192 tokens for Gemma3:4B. A 400-token prompt + 300-token scenario + 200-token extraction = 900 tokens. Fine. But add few-shot examples and you shrink the budget fast.
Temperature — controlling randomness
temp = 0
Always picks the highest-probability token. Deterministic. Good for structured extraction — you want the same JSON every time.
temp = 0.7
Default for most APIs. Some randomness — good for creative tasks, bad for boolean oracles.
temp = 1.5
High randomness. Produces varied, sometimes incoherent outputs. Useful for brainstorming, terrible for compliance extraction.
Implication for your experiments
Always set temperature=0 for extraction experiments. A non-zero temperature means the same scenario can give different oracle values on repeated runs — your results become non-reproducible.
8
LLMs in Practice · Failure Modes
Hallucination — Why It Happens
Hallucination is not a bug or a random glitch. It's a predictable consequence of how LLMs work. Understanding the mechanism helps you design prompts that prevent it.
Mechanism 1: Embedding proximity
Adversarial language activates nearby oracle predicates in embedding space:
"acting pursuant to what might be a court order"
↓ embeds close to ↓
"court order"
Model sets has_court_order = True ← WRONG
The model is pattern-completing, not fact-verifying. No court order exists. But the text sounds like court-order text.
Mechanism 2: No internal verification
LLMs have no access to ground truth. They cannot check whether "has_court_order = true" is actually evidenced in the text. They predict the token that, statistically, tends to appear after prompts like this one.
If training data contains many examples where judicial-purpose scenarios end with True oracles, the model will predict True for any judicial-purpose scenario — even when no instrument exists.
FP — Oracle hallucination
Oracle predicate set True when no evidence → formal engine fires exception → PERMITTED when correct answer is DENIED → PHI leak
FN — Oracle under-extraction
Oracle predicate left False even though evidence is present → formal engine cannot fire → DENIED when correct answer is PERMITTED → over-blocking
9
From Text to Structure
Information Extraction — What LLM₁ Must Do
ComplianceGPT's LLM₁ performs structured information extraction: convert an unstructured narrative into a precise JSON object. One wrong boolean and the verdict flips.
Input: natural language narrative
"A hospital received a civil subpoena from an attorney representing the plaintiff in a malpractice case. The attorney provided a written certification that the patient had been served with notice and the time to object had passed without objection."
A model that sets has_court_order=true here gets the right verdict PERMITTED — but for the wrong reason. That's an oracle hallucination that happens to cancel out. The next scenario it may not.
10
ComplianceGPT's Extraction Schema
The CI 5-Tuple + Oracle Predicates
Every scenario maps to a 5-tuple of structural facts plus 24 boolean oracle predicates. The formal engine cannot grant PERMITTED without the right oracles set to True.
The CI 5-Tuple (structural facts)
sender
Who is disclosing PHI (hospital, physician, health plan…)
receiver
Who receives PHI (law enforcement, court, researcher…)
subject
The patient whose PHI is being disclosed
phi_type
Type of health information (diagnosis, billing, lab…)
purpose
Why the disclosure happens (treatment, law enforcement, research…)
Oracle Predicates (enabling conditions)
Boolean flags. The formal engine cannot grant PERMITTED unless the right ones are True.
If an oracle predicate is False, the exception cannot fire — regardless of how the narrative is phrased. There is no "probably has a court order." It's binary. This is what makes the system verifiable.
11
Stony Brook AI Cluster
Your Research Infrastructure
The NSF-funded Stony Brook AI Cluster gives you access to H100s, V100s, and Quadro RTX 8000s. Running 137 scenarios on Gemma3 takes ~1 hour on your laptop CPU. On one V100, it takes under 5 minutes. Cluster = LLM extraction only. Soufflé stays on your laptop.
Home dir shared across all nodes — code lives in one place
DGX A100 — If needed
8× A100 (40GB each)
Login: 130.245.162.235:130
Docker containers only
Use only if training from scratch
Most of you won't need this.
HGX H100 — Restricted
8× H100 (80GB each)
Requires separate form request
High demand, limited access
For very large models only
You won't need this.
Your laptop write job script
→
Submit node sbatch myjob.sh
→
SLURM queue waits for GPU
→
GPU node runs your code
→
Output file results CSV
You never SSH into the GPU node directly. You submit jobs and read output files. The submit node is the only node you log into.
12
Cluster · SLURM
SLURM — 4 Steps to Run a GPU Job
SLURM is the job scheduler. You describe what you need. It queues your job and runs it when a GPU is free. Complete guide at cluster.html.
The job script (run_extraction.sh)
Extraction only — no Soufflé here
#!/bin/bash
#SBATCH --job-name=cg_extract#SBATCH --output=/home/netid/logs/extract_%j_out.txt#SBATCH --error=/home/netid/logs/extract_%j_err.txt#SBATCH --time=0-04:00# 4-hour max#SBATCH --mem=32000# 32 GB RAM#SBATCH --gres=gpu:1# 1 GPU
source ~/miniconda3/etc/profile.d/conda.sh
conda activate compliancegpt
# LLM extraction → CSV with scenario_json# Soufflé runs locally AFTER you download this
python ~/compliancegpt/connector/llm1_extractor.py \
--model"google/gemma-3-4b-it" \
--output"results/extractions_$SLURM_JOB_ID.csv"
The commands
sbatch myjob.shSubmit — get a job ID back
squeue -u yournetidSee your jobs (PD=pending, R=running)
tail -f logs/run_*_out.txtWatch output update live
scancel JOBIDCancel a job
sacct -u yournetidJob history
sinfoSee all nodes and their status
Request guidelines for ComplianceGPT inference
--gres=gpu:1 — one GPU is enough
--mem=32000 — 32GB RAM for 4B model
--time=0-04:00 — 4 hrs covers 137 scenarios with buffer
Do not request 8 GPUs for inference. Other users need them.
13
Live Exercise
Extract the CI Fields — Do It Before Looking
Read the scenario. Before the answer appears: write down sender, receiver, purpose, and which oracle predicates you think are True. Then we'll compare.
"A hospital received a civil subpoena for a patient's medical records. The requesting attorney provided the hospital with a written certification stating that the patient had been served with notice of the subpoena, that the time permitted for the patient to raise objections had passed without objection, and that the information requested was needed for the malpractice litigation. The hospital released the records."
Step 1 — The 5-tuple
Sender: hospital
Receiver: attorney
Subject: patient
PHI type: medical records
Purpose: judicial / malpractice
Step 2 — Oracle predicates
{
"has_court_order": false,
"has_lawful_process_ with_assurance": true,
// ^ written certification"made_reasonable_ effort_to_notify": true// ^ patient served + no obj
}
Step 3 — The trap
A model that extracts has_court_order=true will still return PERMITTED — but for the wrong reason.
That's an oracle hallucination that cancels out here. In a DENIED case with similar language, it would cause a false positive. Getting the right answer for the wrong reason is dangerous.
14
Research Skill
How to Read a Wrong Prediction
When the pipeline returns the wrong verdict, your job is to determine exactly which layer failed and why. Four steps — always in this order.
Step 1
Check the ground truth
What was the correct verdict? What enabling condition makes it PERMITTED or DENIED? What would need to be True for the correct answer to be reached?
Step 2
Inspect the JSON extraction
Wrong sender_role or receiver_role?
Wrong purpose string?
Oracle predicate True when it shouldn't be?
Oracle predicate False when text clearly states it?
Step 3
Trace to root cause
Oracle hallucination invented enabling fact
Under-extraction fact present, model missed it
Role confusion sender/receiver swapped
Purpose OOV purpose not in schema vocab
Format failure invalid JSON
Step 4
Propose the fix
A prompt instruction guarding against this error type?
A few-shot example showing the correct extraction?
A schema change to add a missing predicate?
A post-processing rule?
This is your project — do this systematically, measure whether your fix works.
15
Live Coding
Inspecting Extraction Outputs in Python
Open your results CSV. We find a wrong prediction and trace it together.
# Load your batch resultsimport pandas as pd, json
df = pd.read_csv("results/yourname_batch.csv")
# Find wrong predictions
wrong = df[df["match"] == "N"]
print(f"Wrong: {len(wrong)}/{len(df)}")
# Pick the first one
row = wrong.iloc[0]
print(row["ground_truth"], "→", row["verdict_norm"])
# Inspect the JSON extraction
sj = json.loads(row["scenario_json"])
# Print CI fieldsfor f in ["sender_role","receiver_role","purpose"]:
print(f"{f}: {sj.get(f)}")
# Which oracles are True?
true_oracles = [k for k,v in sj.items()
if v isTrue]
print("True oracles:", true_oracles)
# Print the scenarioprint(row["question"][:500])
What to look for
Does the scenario text mention a court order? If yes — is has_court_order True? If no — is it correctly False?
Does the sender/receiver match who's actually disclosing in the narrative?
Is the purpose string a valid schema vocabulary term, or something invented?
Are there True oracle predicates that the text does not actually support?
16
Python Skills · Exercise 6
Precision, Recall, F1 — Why Accuracy Isn't Enough
A model that always outputs DENIED gets 90% accuracy on a dataset that's 90% DENIED — but it's useless. It catches nothing. Build these metrics from scratch.
# Implement from scratch — no sklearndef precision_recall_f1(pred, gt):
tp = sum(p=="PERMITTED"and g=="PERMITTED"for p,g in zip(pred,gt))
fp = sum(p=="PERMITTED"and g=="DENIED"for p,g in zip(pred,gt))
fn = sum(p=="DENIED"and g=="PERMITTED"for p,g in zip(pred,gt))
precision = tp / (tp + fp) if (tp+fp) else0
recall = tp / (tp + fn) if (tp+fn) else0
f1 = 2*precision*recall/(precision+recall) \
if (precision+recall) else0return precision, recall, f1
What each metric means
Precision
Of cases we said PERMITTED, what fraction was correct? Low precision = PHI leaks.
Recall
Of truly PERMITTED cases, what fraction did we catch? Low recall = over-blocking.
F1
Harmonic mean. Punishes models that sacrifice one metric for the other.
The always-DENIED model trap
If 90% of cases are DENIED, a model that always outputs DENIED gets 90% accuracy. Precision: undefined (0 PERMITTED predictions). Recall: 0%. F1: 0%. This is why you must report all three metrics.
17
Exercise 9 · Wednesday
Cluster Extraction → Local Soufflé — Together
Two phases. Phase 1: cluster runs LLM extraction, outputs a CSV of JSON extractions. Phase 2: you download the CSV and run Soufflé locally to get verdicts.
Phase 1 — On the cluster
ssh netid@submit.ai.stonybrook.eduConnect
mkdir -p ~/compliancegpt/logs resultsCreate dirs
nano run_extraction.shPaste the job script from Ex 9
sbatch run_extraction.shSubmit — note the job ID
squeue -u yournetidWatch PD → R → done
tail -f logs/extract_*_out.txtLive output
Output CSV has scenario_json for each row — no verdicts yet. That's expected. Soufflé is not on this machine.
# Step 2: Run Soufflé locally → get verdicts# Open week2_nlp.ipynb → Exercise 9 → Step 3# hipaa_connector.py feeds each scenario_json# to local Soufflé → adds verdict_norm column
Wednesday success criteria
squeue shows job R then completes
Extraction CSV downloaded, has scenario_json
Soufflé run locally → verdict_norm column added
Accuracy computed in Exercise 9 Step 4
One wrong prediction found, oracle named
18
Week 2 Tasks
What You're Doing This Week
Monday Jun 29 — On Campus
Origin story lecture (Slides O1–O8)
Live tokenizer demo
Tonight: 3Blue1Brown videos 1–2
Tonight: email rt@cs.stonybrook.edu for cluster access
Tuesday Jun 30 — On Campus
LLMs in practice (Slides 3–10)
Cluster setup — everyone connects and submits test job before leaving
Tonight: Exercises 5–6 in notebook
Tonight: 3Blue1Brown videos 3–4
Wednesday Jul 1 — On Campus
First real GPU experiment (together)
Exercises 7–8 with your own results
Error analysis method walkthrough
Find your first interesting research finding
Thursday Jul 2 — Remote
Finish notebook (all 4 exercises + reflections)
Second cluster run: different model (Llama3 vs Gemma3)