Foundations

John McCarthy coins the term "artificial intelligence." Early vision: write enough rules, and computers will think. Programs play chess, prove theorems. The dominant approach for 30 years: hand-coded rules.

The problem: rules don't scale. HIPAA alone has hundreds of exceptions, sub-conditions, and edge cases. Maintaining rules for every possible patient scenario is impossible.

Rumelhart, Hinton, and Williams publish the backpropagation algorithm. Neural networks (inspired by biological neurons) can now be trained on data by adjusting millions of numerical weights to minimize prediction error. The idea is elegant: show the network labeled examples, compute how wrong it is, propagate the error backward, adjust every weight slightly. Repeat millions of times.

The bottleneck: not enough data, not enough compute. The method worked in theory but not at scale — yet.

The ImageNet Large Scale Visual Recognition Challenge asked models to classify 1.2 million images into 1,000 categories. The best systems had ~26% error. AlexNet — a deep convolutional neural network trained on GPUs — achieved 15.3%. The next-best competitor: 26.2%. A 10-point gap, not 1 point.

Three things converged: (1) data — the internet generated billions of labeled examples; (2) compute — GPUs built for gaming are massively parallel and perfect for training; (3) algorithms — ReLU activations and dropout made deep networks trainable. Research investment in deep learning exploded immediately.

Tomas Mikolov at Google trains a neural network on billions of words with one task: given context words, predict the missing word. The hidden layer weights — never the intended output — turn out to be extraordinarily useful: they are word embeddings.

"hospital" and "clinic" end up close in the 300-dimensional vector space. "hospital" and "lawsuit" are far apart. The famous demonstration: king − man + woman ≈ queen. Meaning encoded as geometry. This is how modern LLMs "know" that "judicial mandate" and "court order" are related — their embeddings end up close.

Language is ordered: "the dog bit the man" ≠ "the man bit the dog." Recurrent Neural Networks (RNNs) process tokens one at a time, passing a hidden state forward — a memory of what came before. Long Short-Term Memory networks (LSTMs, 1997, Hochreiter & Schmidhuber) added gates to control what to remember and forget.

The fundamental problem: sequential processing is slow (can't parallelize on GPUs), and information from early tokens fades over long sequences. A legal narrative might have the crucial fact in sentence 1 and the disclosure in sentence 20. By sentence 20, the LSTM has lost the thread. This is the failure mode that the Transformer fixed.

Vaswani et al. at Google Brain publish an 8-page paper. The core idea: self-attention. For every token, compute how much it should attend to every other token when determining its meaning. "He signed the order" — when computing the meaning of "order," the model attends to "signed" (action) and "judge" (context) with high weight, resolving the ambiguity toward "court order."

Crucially: every token attends to every other token in parallel. No sequential bottleneck. The whole sentence is processed at once. This allows training on 10,000 GPUs simultaneously — and the models can be made enormously larger.

Architecture split: Encoder (reads text → builds contextual representations → used for understanding) and Decoder (generates tokens one at a time → used for text generation). Or both: encoder-decoder for translation and summarization.

BERT (Google, 2018) — encoder-only, 110M parameters. Pre-trained to predict masked words bidirectionally (it can see context on both sides). Fine-tuned on specific tasks: question answering, classification, named entity recognition. Dominated NLP benchmarks immediately. Still used in search engines today.

GPT-1 (OpenAI, 2018) — decoder-only, 117M parameters. Pre-trained to predict the next token, left-to-right. Designed for generation. The beginning of the GPT lineage.

GPT-2 (1.5B parameters, 2019): OpenAI decided not to release it, worried about misuse. When they did, it could write convincing paragraphs, complete stories, fake news. The first glimpse of emergent generation capability.

GPT-3 (175B parameters, 2020): the model that proved the scale hypothesis — bigger models, trained on more data, develop qualitatively new capabilities. GPT-3 demonstrated few-shot learning: show it 3 examples in the prompt, and it generalizes to new examples — without any fine-tuning. For the first time, you could "program" a model with natural language instructions.

InstructGPT (OpenAI, 2022): GPT-3 could generate text but wasn't reliably helpful or safe. Researchers applied Reinforcement Learning from Human Feedback (RLHF): human raters rank model outputs, a reward model is trained on those rankings, and the LLM is fine-tuned to maximize reward. The result: a model that tries to give correct, helpful answers rather than just plausible-sounding ones.

ChatGPT (November 2022): 100 million users in 60 days. The fastest consumer product adoption in history. The general public discovered LLMs existed. Every company reallocated resources. Universities rewrote academic integrity policies. Regulatory bodies scrambled. The world changed in about 10 weeks.

The models you're working with. Gemma3:4B is a 4-billion-parameter model from Google DeepMind — "small" by 2025 standards, runs locally on a laptop, but achieves 94.2% on the HIPAA GoldCoin benchmark with the right extraction layer. Claude Sonnet is Anthropic's frontier model, trained with Constitutional AI (a variant of RLHF). Llama 3 is Meta's open-weight model. All of them descend from the 2017 Transformer.

The gap between what these models know and what they can operationalize precisely is exactly what your research is about.