The progression that made modern AI possible — and why you need to understand each step
Each step solved a problem the previous step couldn't. By the end you'll know exactly why LLMs are necessary for HIPAA compliance extraction — not because someone told you, but because you'll have hit every wall yourself.
Every approach before ML required someone to write the logic. ML inverts this: you supply examples, the algorithm extracts the logic itself.
Misses "judicial mandate". Catches "the officer mentioned a possible order". Never finished. Breaks on every edge case.
The algorithm finds its own rules from data. Works on synonyms it's seen in examples. Improves with more data.
The key shift: The programmer no longer decides what matters. The data decides. A feature that predicts PERMITTED well gets a high weight automatically. One that doesn't gets downweighted. The machine learns the decision boundary.
Three concepts you need to understand everything that follows.
Numeric representations of your input. For HIPAA scenarios:
You choose what features to extract. Bad features → bad model, regardless of algorithm. Day 1 notebook: you hand-crafted these.
The answer for each training example.
Labels come from human experts — lawyers or annotators who read each case. This is the GoldCoin dataset you're using this summer.
The algorithm sees (features, label) pairs and adjusts internal parameters to minimize mistakes.
After training: show the model a new scenario it has never seen. It predicts PERMITTED or DENIED based on learned patterns.
Hand-crafted boolean features gave you ~90% accuracy — on clean, pre-extracted features. The hard part is getting those features from raw text in the first place. That's the extraction problem, and it's what the rest of this week is about.
You used all three in Day 1. Here's the intuition behind each.
Learns a weight for each feature. PERMITTED probability = sigmoid(w₁×f₁ + w₂×f₂ + ... + b).
Think: a weighted checklist. High weight on has_court_order means that feature pushes strongly toward PERMITTED.
Best for: understanding which features matter. Weights are directly interpretable.
Limitation: only learns linear decision boundaries. Can't capture "court order AND NOT officer request."
Learns a sequence of if/else rules. Splits data at each node to maximize separation between PERMITTED and DENIED.
Best for: explainability — you can read the tree and verify it against HIPAA rules.
Trains 100 decision trees, each on a random subset of data and features. Final prediction = majority vote.
Each tree is slightly different. Their errors are uncorrelated. Average them → much lower error than any single tree.
Best for: accuracy. Almost always outperforms a single tree.
Limitation: less interpretable than a single tree. But feature importances still tell you what matters.
The shared limitation of all three: They classify based on features you provide. Change the features, change the model. For text, what features do you provide? This is the next problem.
In Day 1, you engineered features by hand. Deep learning eliminates that step: layers of the network learn to extract features automatically from raw input.
A neuron receives inputs, multiplies each by a learned weight, adds a bias, and passes the sum through a non-linear function (ReLU, sigmoid):
Stack millions of these in layers and the network learns arbitrarily complex functions from data.
Multiple hidden layers. Each layer learns increasingly abstract features:
No human labeled these levels. The network figured them out from the training signal alone.
Training a neural network is an iterative loop. The same loop runs billions of times.
Input flows through the network layer by layer. Each neuron computes its output. The final layer produces a prediction: e.g., 72% PERMITTED, 28% DENIED.
Compare prediction to the true label. The loss function (e.g., cross-entropy) gives a single number measuring how wrong the prediction was. High loss = very wrong.
Compute how much each weight contributed to the error (gradient). Chain rule propagates error backward from the output layer to the input layer.
Gradient descent: nudge every weight slightly in the direction that reduces the loss. Learning rate controls the step size. Repeat from Step 1.
ImageNet challenge: classify 1.2M images into 1,000 categories. Best 2011 approach: 26% error. AlexNet (deep network on GPUs, 2012): 15.3% error. Next best: 26.2%.
A 10-point gap, not 1. Everyone in AI pivoted overnight. GPUs became research infrastructure.
The backprop loop runs the same operation (matrix multiply) millions of times. GPUs have thousands of cores designed for exactly this — they run these operations in parallel.
Neural networks process numbers. Text is not numbers. Every NLP approach is an answer to: how do we represent text as vectors? Each approach captures more meaning than the last.
Count how many times each word appears. Vocabulary of 10,000 words → 10,000-dimensional vector.
Problem: no word order. "court dismissed order" = "order court dismissed". No semantic meaning.
Weight rare words more. "subpoena" appears in 3 scenarios → very informative. "the" appears in all → useless.
Better, but: "court order" and "judicial mandate" have 0% word overlap → near-zero similarity. Semantically blind.
Train a network to predict word context. Words used in similar contexts end up close in vector space.
Captures meaning! But still fails on adversarial context: "implied there might be a court order" also scores ~0.80.
Two lines of sklearn. You used these in Day 2's notebook. Here's what they actually do.
TF-IDF cosine similarity: "court order" vs "judicial mandate" ≈ 0.08. Zero word overlap = near-zero similarity. Legally they enable the same HIPAA exception. TF-IDF doesn't know that. This is the wall TF-IDF hits.
Word2Vec (2013): train a network to predict word context. The byproduct is a 300-dimensional vector for every word where meaning is encoded as position in space.
Given "The [BLANK] is a medical facility" — predict the blank. Network learns to represent words that fill similar blanks similarly.
The hidden layer weights become the word vectors. Hospital and clinic end up geometrically close. Lawsuit ends up far away.
king − man + woman ≈ queen
Vector arithmetic preserves relationships. This proved that meaning is encoded geometrically, not symbolically.
Word2Vec embeds individual words. Sentence transformers embed entire sentences — capturing context, not just vocabulary.
| court order ↔ judicial mandate | 0.87 ✓ FIXED |
| court order ↔ implied court order | 0.79 ✗ STILL CLOSE |
| court order ↔ officer request | 0.52 |
"Implied there might be a court order" is semantically about court orders — it just doesn't constitute one. An embedding model captures topic similarity. It cannot detect the legal distinction between mentioning a court order and having one.
This is the exact gap LLMs try to close. A model that reads the full sentence in context can attend to "implied" and "might be" and understand they are hedging qualifiers.
Vaswani et al., 2017. The paper that changed everything. Instead of processing one word at a time (RNNs), every word attends to every other word simultaneously.
RNNs process tokens sequentially. Token 1 → hidden state → Token 2 → hidden state → ... By token 50, information from token 1 has faded (vanishing gradient).
"The patient, who was admitted after a workplace injury treated by a physician who had previously disclosed records in an unrelated matter, filed for damages claiming the hospital had shared information without court authorization."
Key fact ("without court authorization") is at the end. Subject ("patient") is at the start. RNN has trouble connecting them over 40 tokens.
For each token, compute: how much should I attend to every other token?
Reads the entire input bidirectionally. Every token can attend to every other token in both directions simultaneously.
Training task: Masked Language Model — randomly mask 15% of tokens and predict them from context.
Use for: classification, extraction, understanding. BERT representations are the foundation of ComplianceGPT's extraction layer.
Generates tokens left-to-right. Each token can only attend to previous tokens — never future ones (causal masking).
Training task: Next token prediction — given all previous tokens, predict the next one. Trained on trillions of tokens from the internet.
Use for: text generation, completion, conversation. Gemma3 and Llama3 are decoder-only LLMs.
The practical difference for you: When Gemma3 extracts oracle predicates, it's doing next-token prediction on your prompt. It predicts "has_court_order": true because that's the most statistically likely completion of your prompt — not because it verified the text.
Everything you learned about text representations (Parts 1–3) comes together here.
This happens for every token in parallel, across 32+ transformer layers, for every scenario in your batch.
Gemma3:4B can process 8,192 tokens at once. Everything outside the window is invisible to the model. A long court case narrative can fill 30–50% of that window before your prompt instructions even start.
temp=0: always picks the highest-probability next token. Deterministic — same input, same output every time. Always use temp=0 for extraction experiments — otherwise you can't reproduce results.
Train on 10+ trillion tokens: web pages, books, code, legal documents. Task: predict the next token.
Result: a model that has absorbed statistical patterns of human knowledge. It can complete sentences in any domain — including legal text.
Train on thousands of (instruction, good-response) pairs. Task: generate the helpful response given a user request.
Result: model understands what it's supposed to do. Follows instructions, stays on topic, uses correct format.
Human raters rank model outputs. A reward model is trained on rankings. LLM fine-tuned to maximize reward.
Result: optimized for what humans prefer — helpful, harmless, honest.
RLHF optimizes for human approval of fluency, not legal precision. A model that confidently writes "has_court_order": true sounds good to a human rater. But it may be wrong. You can't distinguish confidence from correctness in LLM output — you have to verify against ground truth.
Not a random bug. A predictable consequence of how LLMs work. Understanding it lets you design prompts that prevent it.
When generating "has_court_order": , the model predicts whether true or false is the most likely next token given the prompt and everything generated so far.
It cannot look back at the scenario text and verify. It predicts what statistically tends to follow prompts like this one.
Oracle set True without evidence → formal engine fires exception → verdict PERMITTED → PHI leak
Most common error in ComplianceGPT FPs: has_court_order hallucinated in law-enforcement scenarios
Oracle left False even though scenario text clearly states it → exception cannot fire → verdict DENIED → over-blocking
Happens when the enabling fact is stated indirectly or with unusual phrasing
What you built this week. Each row: what it can do and what it can't.
| Approach | What you built | Accuracy | Wall hit |
|---|---|---|---|
| Hand-crafted rules | if "court order" in text |
~60% | Misses synonyms. Catches false positives. Never finished. |
| ML + hand features | Day 1 — sklearn on boolean features | ~90% | Who extracts the features from raw text? |
| Bag of Words + ML | Day 2 — CountVectorizer + LogReg | ~70% | Word order ignored. No semantic meaning. |
| TF-IDF + ML | Day 2 — TfidfVectorizer + LogReg | ~80% | "court order" ↔ "judicial mandate" similarity = 0.08 |
| Sentence Embeddings | Day 3 — sentence-transformers | ~87% | "implied court order" still embeds near real court order |
| LLM Extraction | Exercise 9 — Gemma3 on cluster | 94.2% | Still hallucinates has_court_order on adversarial phrasing |
The research frontier: That last 5.8% gap. Each error is an oracle hallucination or under-extraction that LLMs still make despite reading in full context. Closing that gap — through better prompts, schema design, few-shot examples, or model selection — is your project this summer.
Every technique solved something the previous one couldn't. The story is cumulative, not replaceable.
You've now built every step of the 70-year progression by hand. Here's where you stand.
has_court_order=true when text only implies possible authorityEvery project attacks the same 5.8% gap from a different angle. See projects.html for full descriptions. Project selection due Friday Jul 3.
day1_sklearn.ipynb
day2_nlp.ipynb
day3_embeddings.ipynb
week2_nlp.ipynb Exercises 5–9week2_nlp.ipynb Exercises 5–9