From next-token predictor to research instrument — the bridge between Week 2's transformer math and Week 3's prompting work
You already know how a transformer computes one forward pass. Today is about what happens around that forward pass to turn it into something you can talk to.
The forward pass you studied in Week 2 never changes. What changes across these stages is what the model was trained to predict.
Predict the next token over trillions of words of raw internet/book/code text. No notion of "question" or "answer" — just completion.
"The capital of France is" → "Paris"
Fine-tune on (instruction, good response) pairs so the model completes instructions, not just text, and adopts a chat format.
"Summarize this." → does it, not "Summarize what?"
Humans (or a reward model) rank multiple candidate responses; the model is nudged toward the ranked-higher style — helpful, harmless, on-format.
Same facts, safer / more useful phrasing
Every model we benchmark — Gemma, Llama, Qwen, GPT-OSS, Claude — has been through some version of stages 2 and 3. That is why you can prompt them in plain English at all.
There is no "system / user / assistant" data structure inside the model. It's still one flat token sequence — with special marker tokens the instruction-tuned model was trained to respect.
The model then does exactly the next-token prediction from Week 2 — it just happens to have been trained so that the highest-probability continuation after <start_of_turn>model looks like a helpful reply instead of more internet text.
Every step, the model outputs one probability over the entire vocabulary (~150k tokens for Gemma/Qwen). Decoding is the separate algorithm that turns that distribution into an actual chosen token.
Always pick the single highest-probability token. Deterministic — same input, same output, every time.
Used for: extraction, classification, anything you need to reproduce and audit.
Draw randomly from the distribution instead of always taking the max. Same prompt can give different answers on different runs.
Used for: creative writing, brainstorming, chat — anywhere variety beats reproducibility.
| Parameter | What it does | Effect at extremes |
|---|---|---|
| temperature | Rescales the distribution before sampling. Divides logits by T before softmax. | T→0: greedy. T→∞: uniform random token. |
| top_k | Only sample from the k highest-probability tokens. | k=1: greedy. k=vocab size: no restriction. |
| top_p | Sample from the smallest set of tokens whose probabilities sum to p ("nucleus"). | p=1.0: no restriction. p small: very conservative. |
We need the same scenario to produce the same extracted facts every time we re-run an experiment — otherwise "accuracy" isn't measurable and a bug fix can't be verified. Reproducibility is a research requirement, not a preference. This is exactly what week3_prompting.ipynb asks you to confirm empirically on Wednesday.
Between API calls, the model remembers nothing. Every fact it uses — your system prompt, the scenario, few-shot examples, prior turns — must physically fit inside one token sequence, every single call.
When you show the model 2 worked examples before your real question (few-shot), nothing in the network's parameters changes. The model is pattern-matching the shape of the prompt itself, at inference time.
This is the mechanism behind everything you'll do in Wednesday's few-shot exercises — you are not teaching the model new facts, you are teaching it the format and granularity of a good answer.
Real accuracy on our 137-row GoldCoin-HHS benchmark, same prompt/strategy, five models spanning two orders of magnitude in parameter count.
| Model | Params | Accuracy | Macro-F1 | Rank |
|---|---|---|---|---|
| Gemma2:2B | 2B | 78.2% | 0.778 | 5th |
| Llama3.1:8B | 8B | 84.7% | 0.839 | 4th |
| GPT-OSS:20B | 20B | 90.0% | 0.897 | 3rd |
| Qwen2.5:72B | 72B | 92.7% | 0.919 | 2nd |
| Claude-Sonnet-4.6 | ~200B* | 94.2% | 0.938 | 1st |
Roughly monotonic here — but notice the curve is flattening: 2B→8B gained 6.5pp, 20B→72B gained only 2.7pp. *Proprietary size estimated. This is exactly the kind of data your own project could extend or complicate.
This is a real engineering decision every project makes: your Week 4–5 experiment plan has to budget for the fact that a 137-row run on Qwen takes roughly 2.5 hours of wall-clock GPU time, not seconds.
The model was trained to always produce a fluent next token. It has no built-in "I don't know" state — it will complete a JSON field even when the scenario never mentions it.
Week 2's analysis already showed this on our own data: the model's stated confidence does not reliably predict whether the verdict is right. A wrong answer can be delivered exactly as fluently as a right one.
| Open-weight (Gemma, Llama, Qwen, GPT-OSS) | Proprietary API (Claude) | |
|---|---|---|
| Where it runs | Your GPU / rented GPU, via Ollama | Someone else's servers |
| Reproducibility | Full — same weights forever | Provider can silently update the model |
| Cost model | Compute time (rent a GPU) | Per-token API billing |
| Inspectability | Can quantize, fine-tune, inspect activations | Black box — text in, text out |
We evaluate both for a reason: open-weight models are what a compliance officer could actually self-host and audit; the proprietary model is our accuracy ceiling reference point.
You cannot open the model up and fix a wrong weight. The prompt — system instructions, examples, formatting constraints, reasoning scaffolding — is the only lever you have to shape behavior from outside the black box.
Every technique today assumed one call to the model. Our actual production pipeline calls the LLM up to four times per scenario, checking its own work in between.
How connector/llm1_extractor.py's extract_with_reflection() turns a single extraction call into a four-pass self-checking loop — and why that loop helps a small model enormously but can backfire on a large one if you're not careful with retrieval.
No notebook exercise today. Instead:
notebooks/week3_prompting.ipynb — you'll write zero-shot, few-shot, and chain-of-thought prompts against that exact failure case.
Architecture (Week 2) + generation mechanics (today) = everything that happens before a single prompting choice you make on Wednesday even matters.