⌗ LECTURE 12 · DECODER-ONLY MODELS — GPT & AUTOREGRESSIVE LANGUAGE MODELING
The Decoder, Alone
Lecture 11 ended with three descendants of the Transformer. This lecture builds the one that became the large language model — from the objective up, not the architecture down. What does it mean to model language? One product of conditional probabilities. How do you learn it? By counting, then by a network you will train in this browser. What architecture computes it? The 2017 decoder with its encoder deleted — masked attention all the way down. By §08 you will have watched every mechanism of GPT run live, at doll scale.
WATCH A LANGUAGE MODEL WRITE
The machine: one token at a time. At every step the model exposes a full probability distribution over its vocabulary; one token is drawn, appended, and the whole thing repeats. That loop — and nothing else — is how GPT writes anything: essays, code, dialogue, this sentence’s next word.
This toy was trained live when the page loaded (18 sentences, 1,159 parameters, deterministic seed). §04 replays the entire training run; §08 hands you the loop’s controls.
CONTEXT (last 2 tokens)
P( NEXT TOKEN | CONTEXT ) — LIVE
That was generation. But every honest mechanism lecture starts one step earlier: before a model can produce language, we need a mathematical object called “the probability of a sentence.” Defining it is the whole game — and it factors.
02 LANGUAGE MODELING, DEFINED
What is the probability of a sentence?
You play this game daily: someone starts a sentence and you finish it. Claude Shannon — the father of information theory — turned it into a number in 1948: score every possible sentence, higher for the ones people actually write. One application of conditional probability later, the whole task collapses into guess the next token, repeat. Below, a real sentence is minted factor by factor; every probability is on the screen.
THE SENTENCE, MINTING — each token’s conditional probability appears beneath it
P( NEXT | CONTEXT ) — THE FACTOR BEING MINTED
So a language model is any function that outputs those factors. The 1950s answer — and still a shockingly good one — needs no learning at all: count.
03 THE COUNTING ANSWER
N-gram models — count, normalize, done
Want P(next | “the cat”)? Find every “the cat” in your corpus, count what followed, divide. No gradients, no GPU — the entire model is a hash map. It works… until it doesn’t: the table’s emptiness grows exponentially with context length. Every number below is counted live from the page’s 18-sentence corpus.
THE WALL — slide the context length and watch coverage collapse
Time to make the bet yourself. Below is a complete language model — embeddings, hidden layer, softmax — with 1,159 parameters and the exact GPT objective. Press TRAIN and watch it learn the corpus from §03, live, in your browser.
04 THE NEURAL BET, RUN LIVE
Train a language model — right here
The architecture of every neural LM since Bengio (2003): embed the context tokens, mix them in a hidden layer, softmax over the whole vocabulary. Below it trains with full-batch gradient descent on cross-entropy — the identical objective GPT-1 trained on, at 1/100,000th the scale. Watch the loss fall from ln(15) = 2.71 (know-nothing uniform) toward the entropy the data genuinely contains.
THE MACHINE — shapes on every wire (Lecture 10 style)
1,159 parameters · GPT-1: 117,000,000 · GPT-3: 175,000,000,000 — same objective, same loss, same softmax. Only the mixer between embedding and logits grows.
CROSS-ENTROPY LOSS — live, full-batch gradient descent
PROBE THE MODEL — click a context, watch its distribution sharpen as training runs
The learner works — now scale the picture. Keep the objective (§02) and the training loop you just watched (§04); swap the one tanh layer for Lecture 11’s decoder blocks. The architecture part of “decoder-only” is pure subtraction.
05 ARCHITECTURE SURGERY
One 2017 machine, three destinies — GPT keeps one half
Lecture 11’s closing table showed encoder-only, encoder–decoder and decoder-only as three families. Here is the surgery at sublayer granularity: which stack each keeps, which mask, which training task — and then the piece every diagram forgets: the LM head, one matrix that turns a hidden state into a distribution over the entire vocabulary.
THE LM HEAD, LIVE — hidden state → logits → next-token distribution, one dot product per word
GPT-1 (2018) vs ITS 2017 PARENT vs THIS PAGE’S TOY
The stack computes one hidden state per position — but a next-token model must never let position t’s state be influenced by token t+1. That constraint has a name, a shape, and a starring role in the single most important trick of the whole architecture.
06 THE CAUSAL MASK
The arrow of time, as a matrix
Lecture 11 §08 introduced the mask. Here it is the whole story: the lower-triangular “allowed to look” pattern that makes prediction honest, lets every position train in parallel, and doubles as the definition of “autoregressive.” Then the second half of the trick everyone gets wrong: the one-position label shift.
ROW’S ATTENTION WEIGHTS — softmax over what it can see
THE LABEL SHIFT — inputs on top, labels one step left, hₜ predicts wₜ₊₁ (hover any input)
Mask + shift + the loss you watched fall in §04 — assemble them and “training a GPT” is one sentence of math. Time to total the books, position by position.
07 TRAINING = TEACHER FORCING
T predictions, one pass, one number
Feed the real sentence in (the “teacher”), demand a next-token distribution at every position, score each against the truth that actually followed. The mean of those −log p’s is the training loss — and its exponential, perplexity, is the number the whole field chased for decades. Every value below is computed live from the same corpus counts of §03.
THE TARGETS — click any position (or STEP) to score it
P( NEXT | CONTEXT ) FROM THE COUNTS — truth marked
THE LOSS LEDGER — one bar per position (−ln p)
SELF-SUPERVISION — slide the window; the labels write themselves
No annotators, no task labels: the text is the answer key. Each window position yields one (context → next-token) example the moment it slides past — this is where GPT’s training set comes from.
THE WINDOW
EXAMPLES THIS WINDOW YIELDS — context → target
Training is done — weights frozen forever. Now run the machine the other way: no more teacher, the model eats its own output. Five little stages, forever, until </s>. This loop is every chat box you have ever used.
08 GENERATION
The autoregressive loop, on your marks
The trained §04 model (identical weights, frozen) writes left to right: read context → forward pass → distribution → pick → append → repeat. Greedy always takes the top token; sampling draws from the bars. Flip between them and watch what repetition does — and what variety costs.
CONTEXT
P( NEXT | CONTEXT )
GENERATED STREAM
THE LOOP’S LOG
Notice what you never touched in §08: the weights. The only thing you changed was the context — and the output changed completely. Hold that thought; it has a name, and it ate the world.
09 THE INTERFACE
The context is the program
A decoder-only model has no buttons, no modes, no API but one: the prefix. Same frozen weights — different context, different continuation. GPT-2 (2019) showed this could go absurdly far: translate, summarize, answer — all by prompting, zero weight updates. Build a prefix below and steer the model yourself.
YOUR PREFIX (click tokens to extend, ⌫ to remove)
P( NEXT | YOUR PREFIX ) — live
CONTINUATION — greedy, weights frozen
SAME WEIGHTS, THREE FUTURES — the verb distributions the trained model gives each subject
Nothing about the model differs between these three columns — only two tokens of context. Now imagine those two tokens are two thousand, and the “verbs” are “answer in Urdu” / “write Python” / “summarize this document”. That is prompting; that is few-shot learning; that is the entire user interface of every LLM product since.
One objective, half an architecture, a loop, and a prompt — that is the entire mechanism. The last honest question: why did this half of the 2017 machine inherit the earth, and what did it pay?
10 THE HONEST LEDGER
Why decoder-only won
Not because it is the best reader (BERT reads better) or the natural translator (the 2017 original translates better). It won on economics: one objective that never needs labels, one architecture that gets better with nothing but more data and more parameters, and output that IS the product. Pick a task and ask each family for its honest verdict.
PICK A TASK — who should you hire?
GPT-1: pretrain ONE decoder on books (next-token, §04’s loss), then bolt a tiny head per task and fine-tune — 9 of 12 benchmarks at state of the art. The decoder’s representation, learned from guessing the next word, already understood sentiment, entailment, similarity.
GPT-3: skip the fine-tuning entirely — 175B parameters, tasks solved inside the prompt (§09, scaled a million times over). Few-shot learning with frozen weights proved the objective alone had been the point all along.
Left-context only — never a right neighbor. O(n²) attention over the context. Generation is sequential: one token per pass. Exposure bias: trained on real text, deployed on its own. Every one of those bills is a research field now — none of them overturned the verdict.
11 CODE LAB
The whole lecture, in runnable code
Three programs, era-accurate: the counting LM of §03 in thirty lines of NumPy; GPT-1’s decoder in 2018 PyTorch — the Lecture 11 block plus the causal mask, the weight-tied LM head and the shift; and the generation loop of §08, verbatim.
12 MASTERY CHECK