Curriculum

Week 2 — AI Fundamentals

June 29 – July 3 · ML → DL → NLP → Transformers → LLMs · Cluster access · First GPU experiments

W2

AI Fundamentals

Jun 29 – Jul 3  ·  Mon/Tue/Wed on campus · Thu/Fri remote  ·  Deliverable: notebook exercises + first cluster run

This Week's Story

Why do LLMs like Gemma and Claude exist? Why do they work? And why do they still fail 5.8% of the time on HIPAA? This week you learn the 70-year progression from hand-coded rules → machine learning → deep learning → transformers → LLMs. On Tuesday you get access to GPUs. By Wednesday you have run a real LLM extraction experiment on the Stony Brook AI Cluster. Regulations are next week — this week is entirely AI.

The Two-Machine Workflow — Critical Architecture

The Soufflé formal reasoning engine cannot run on the AI Cluster — it is a locally compiled binary that requires system permissions the submit nodes do not grant. Your research workflow is therefore always split:

AI Cluster (GPU)
LLM extraction only
llm1_extractor.py
→
scp / VS Code download
extractions_JOBID.csv
→
Your Laptop (local)
Soufflé engine + analysis
hipaa_connector.py

This is standard HPC practice: run the expensive part (GPU inference) on the cluster, run everything else locally. Soufflé takes milliseconds — it doesn't need a GPU.

Learning Goals

Monday, June 29 — The 70-Year Story

On campus · NCS 242 · 10am–noon

🧠 Lecture — How We Got to LLMs (Slides O1–O8)

The origin story

  • 1956 — AI is born. Hand-coded rules. Why they fail at scale.
  • 1986 — Backpropagation: neural networks can finally learn
  • 2012 — AlexNet / ImageNet: deep learning actually works. The three ingredients: data, GPUs, algorithms.
  • 2013 — Word2Vec: words as geometric vectors. "hospital" and "clinic" are close. "hospital" and "lawsuit" are far. Meaning encoded as geometry.
  • ~2015 — RNNs, LSTMs: processing sequences. The vanishing gradient problem.

The modern era

  • 2017 — "Attention Is All You Need": the Transformer. Every word attends to every other word in parallel. No sequential bottleneck. Encoder vs. Decoder.
  • 2018 — BERT (encoder) and GPT-1 (decoder): the two paths from the Transformer
  • 2020 — GPT-3 (175B params): few-shot learning. Show 3 examples in the prompt → model gets the task. No fine-tuning.
  • 2022 — RLHF + ChatGPT: 100M users in 60 days. The world changes.
  • Now — Claude, Gemma3, Llama3: the models you're using this summer

Bridge: Why This Matters for Your Research

Gemma3:4B has read more text than any human could in a million lifetimes. It knows HIPAA — statistically. But it predicts the most likely next token, not the most legally correct one. These are not the same. That 5.8% error rate is the gap your research lives in.

💻 Live Demo — Tokenization in Action

We'll run tiktoken on real HIPAA scenario text live. Students will see:

  • How "anti-inflammatory" splits into 3 tokens, "§164.512(e)(1)(ii)" into 8+
  • How a 500-word scenario fills 12% of Gemma3's context window before the prompt even starts
  • Why truncation is a silent failure: the model gets a verdict, but the key fact was cut off
📓 Monday Assignment — Day 1: Scikit-learn ML

Download day1_sklearn.ipynb and work through it tonight or during independent time. You will:

  • Load a simulated HIPAA feature matrix (hand-crafted boolean features)
  • Train logistic regression, decision tree, and random forest classifiers
  • Read what the model actually learned (feature weights and tree rules)
  • Hit the wall: why does clean extracted data get 90% but raw text is the hard part?

Also tonight: Watch 3Blue1Brown — Neural Networks, videos 1–2 (~40 min). Email rt@cs.stonybrook.edu for cluster access — include your NetID. Must go out tonight so it's ready for Tuesday's setup session.

No hard deadline — submit day1_sklearn.ipynb in Slack #deliverables when complete.

Tuesday, June 30 — LLMs in Practice + Cluster Setup

On campus · NCS 242 · 10am–noon

🧠 Lecture — Tokens, Embeddings, Attention (Slides 1–7)

How LLMs actually read text

  • Tokens: subword units, not words. Context window limits.
  • Embeddings: each token → a point in high-dimensional space. Geometric meaning.
  • Self-attention: "He signed the order" — how the model resolves "order" to court order vs. medical order based on surrounding context
  • Where attention succeeds: long-range legal dependencies
  • Where attention fails: adversarial framing, role confusion, legal boilerplate that looks like enabling language

What ComplianceGPT's LLM₁ must extract

  • The CI 5-tuple: sender, receiver, subject, phi_type, purpose
  • Oracle predicates: 24 boolean flags, each True only if explicitly evidenced
  • Why hallucination happens: "acting pursuant to what might be a court order" embeds close to "court order" → model sets has_court_order = True
  • Why this matters: a single wrong oracle predicate changes the verdict
🖥️ Cluster Setup — Everyone Gets a GPU

Full step-by-step instructions: cluster.html. We do this together in class. Do not skip any step.

  1. Verify access email arrived. If not, we troubleshoot now.
  2. Connect via SSH — Mac: Terminal. Windows: VS Code + Remote SSH (recommended) or PuTTY.
  3. First login: ssh yournetid@submit.ai.stonybrook.edu — home directory setup takes 1–2 min the first time.
  4. Load modules: module load slurm python3
  5. Install Miniconda and create the compliancegpt conda environment.
  6. Install packages: PyTorch, transformers, pandas, tiktoken.
  7. Submit the test job from cluster.html — verify you see CUDA: True and a GPU name in the output.

Goal by end of Tuesday

Every student has submitted a SLURM job, seen it run on a GPU node, and read the output file. If your test_out.txt shows a GPU name and CUDA: True — you're done. If not, we fix it before you leave.

📓 Tuesday Assignment — Day 2: NLP Representations

Download day2_nlp.ipynb. This is the bridge between Day 1 (hand-crafted features) and Day 3 (embeddings). You will:

  • Represent the same scenarios as a Bag-of-Words matrix using CountVectorizer
  • Switch to TF-IDF — watch accuracy improve as rare legal terms get upweighted
  • Compute cosine similarity between "court order" and "judicial mandate" in TF-IDF space
  • Find the semantic gap: TF-IDF gives them ~0.1 similarity even though they're legally identical

Also tonight: Watch 3Blue1Brown videos 3–4 (backpropagation, training). Come Wednesday knowing: what happens when the model is wrong, and how weights update.

No hard deadline — submit in Slack #deliverables when complete.

Wednesday, July 1 — First Real GPU Experiment

On campus · NCS 242 · 10am–noon

🧠 Mini-Lecture — The Scaling Story + What Breaks (15 min)

A fast recap of the model lineage (Slides O7–O8): BERT → GPT-3 → ChatGPT → Gemma. Then: what specifically goes wrong in legal text extraction. We look at one real FP and one real FN from the GoldCoin results and trace each back to an oracle predicate failure.

🔬 Hands-on — Submit Your First Real Experiment (Exercise 9)

Using the job template from Exercise 9 in notebooks/week2_nlp.ipynb, each student submits an LLM extraction job on the cluster. Together we watch the queue, monitor output, download results, and run Soufflé locally.

On the cluster

  1. Edit run_extraction.sh with your NetID
  2. sbatch run_extraction.sh — get a job ID
  3. squeue -u yournetid — watch PD → R → done
  4. tail -f logs/extract_*_out.txt — live output
  5. Download extractions_JOBID.csv to your laptop

On your laptop (after download)

  1. Open Exercise 9 in week2_nlp.ipynb
  2. Update EXTRACTION_CSV path
  3. Run Step 3: Soufflé connector adds verdict_norm
  4. Run Step 4: compute accuracy, find first wrong prediction
  5. The scenario_json column shows what the LLM extracted — find the oracle that's wrong

Remember: Soufflé runs on your laptop, not the cluster. The cluster output is extractions only — no verdicts yet.

💻 Live Coding — Exercises 7 & 8 (together in class)

We work through Exercises 7 (group errors by category) and 8 (oracle predicate analysis) from notebooks/week2_nlp.ipynb together using the results you just downloaded. The pattern will be clear: has_court_order fires in most FP rows, and it fires wrong in the judicial pathway cases.

📓 Wednesday Assignment — Day 3: Embeddings + LLM Motivation

Download day3_embeddings.ipynb. This is the payoff of the 3-day story. You will:

  • Install sentence-transformers and embed HIPAA phrases with a pre-trained model
  • See "court order" and "judicial mandate" finally score ~0.85 similarity
  • Find the adversarial failure: "implied there might be a court order" also scores ~0.80 near a real court order
  • See why embeddings alone can't solve legal text — and why LLMs reading in context are the next step

Also complete Exercises 7–8 in week2_nlp.ipynb using your downloaded cluster results from Exercise 9.

No hard deadline — submit in Slack #deliverables when complete.

Thursday July 2 — Remote Work

📋 Agenda
  • By noon: Slack update in #daily-updates — what you're working on, any blockers
  • Finalize and clean up notebooks/week2_nlp.ipynb (all 4 exercises completed, reflections written)
  • Run a second cluster extraction with a different model (Llama3:8B instead of Gemma3). Download it, run Soufflé locally, compare accuracy.
  • Watch Andrej Karpathy — "The State of GPT" (2023) on YouTube — 45 min. Pretraining, instruction tuning, RLHF. This is the background for Week 3 (prompt engineering).
  • Read Jay Alammar — "The Illustrated Transformer" blog post. The best visual explanation of attention.

Friday July 3 — Deliverable Due

📋 Agenda
  • By noon: Slack update
  • Write the 1-page error analysis (see deliverable below)
  • Clean up and submit everything by 5pm

Week 2 Deliverable

Due: Friday, July 3 at 5pm

1. Completed notebook — notebooks/week2_nlp.ipynb with all reflection cells filled in:

  • Exercise 5 — Token counts for 3 scenarios, reflection on truncation risk
  • Exercise 6 — Precision/recall/F1 implemented from scratch, interpretation written
  • Exercise 7 — Error grouped by purpose category; which category is worst?
  • Exercise 8 — Which oracle predicate over-fires in your FP rows?
  • Exercise 9 — Cluster extraction run complete: job script, downloaded CSV, Soufflé run locally, accuracy computed, reflection answered

2. Cluster run evidence — Post in #deliverables on Slack:

  • Output of sacct -u yournetid showing at least 1 completed job
  • The results CSV filename from your batch run (e.g. gemma3_run_48305.csv)
  • One sentence: "I ran X scenarios, got Y% accuracy, the most common error was [oracle predicate]."

3. Error analysis (1 page) — Pick 2 wrong predictions from your cluster run. For each write:

  • 1-sentence summary of the scenario
  • What the model predicted vs. the correct answer
  • Which exact oracle predicate is wrong in the JSON, and what it should be
  • Root cause: hallucination, under-extraction, role confusion, or purpose OOV?
  • One proposed fix (prompt instruction, few-shot example, or schema change)

Submit: committed notebook + Google Doc link, both posted to Slack #deliverables by 5pm.

Resources This Week

Required

  • 3Blue1Brown — Neural Networks, videos 1–4 (YouTube) — ~80 min total. Watch all four this week.
  • Andrej Karpathy — "The State of GPT" (2023) (YouTube, 45 min) — pretraining, instruction tuning, RLHF explained clearly.
  • Jay Alammar — "The Illustrated Transformer" (jalammar.github.io) — the best visual explanation of self-attention on the internet.
  • Cluster setup guide: cluster.html — step-by-step for Mac and Windows.

Highly Recommended

  • Vaswani et al. — "Attention Is All You Need" (2017) (arxiv 1706.03762) — abstract, Section 1, Figure 1 only. You don't need the math.
  • Andrej Karpathy — "Let's Build GPT from Scratch" (YouTube, 2 hrs) — if you watch one optional video this summer, make it this one. You will genuinely understand transformers afterward.
  • GoldCoin paper (arxiv 2405.15175) — abstract + Section 2. This is the benchmark you're working with all summer.
  • AI Cluster docs: ai.stonybrook.edu/research/computingresources — official hardware specs and usage policy.