⌗ GPT — GENERATIVE PRETRAINED TRANSFORMERS · Radford et al. 2018 → Brown et al. 2020 → ChatGPT 2022
GPT — the decoder, alone
Lecture 11 ended with three descendants of the Transformer. This lecture chases the one that became the large language model: keep only the decoder half, train it on one absurdly simple objective — predict the next token — on a slice of the whole internet, and scale it 1,500×. Every mechanism below runs live: a toy GPT writing token by token, a genuine byte-pair tokenizer you train by hand, the pre-LN block, the scaling law, the samplers behind every chat box, the KV cache, and the RLHF recipe that turned GPT-3 into ChatGPT.
THE WHOLE SHOW IN ONE LOOP
This is generation, the entire mechanism of every GPT product: look at what exists, predict a distribution over the next token, draw one sample, append it, repeat. Below, a tiny hand-crafted distribution table plays the role of the 96-block model — the loop, the probabilities and the sampling are the real thing.
Then walk the lecture: what GPT deleted (§02), the objective (§03), the tokenizer (§04), the block (§05), the scale story (§06), how answers are sampled (§07), how generation is made affordable (§08), and how GPT-3 became an assistant (§10–11).
THE TEXT SO FAR
P(next | entire prefix)MODEL’S NEXT-TOKEN DISTRIBUTION — computed live
Lecture 11 built the full machine and met its three descendants. GPT’s first move was subtraction — press the button and delete half the architecture.
02 FROM TRANSFORMER TO GPT
The decoder, alone
2017 built two stacks: an encoder to read a source, a decoder to write a target, cross-attention bridging them. GPT’s question: if pretraining teaches a decoder to predict the next token of ordinary text, what is the encoder still for? Answer: nothing. One stack remains — and with it, every task in the world becomes a text-in-text-out problem.
THE 2017 MACHINE vs THE GPT MACHINE
THE GPT STACK — CLICK ANY BLOCK TO OPEN ITS INNER MACHINERY
sample → append → loopHover to trace a block’s wires · click to open its inner machinery.
One stack remains. Now the question that made the decade: what exactly is it trained to do? The answer is embarrassingly small — one prediction per token.
03 ONE OBJECTIVE
Predict the next token
Every position plays the same game: given everything to its left, guess the next token. The causal mask — Lecture 08’s −∞ wall — is what makes this legal for the whole sequence at once: position t can never peek at the answer it is asked to predict. Click any position and read its loss; the average over billions of positions is the entire pretraining loss.
THE SENTENCE — CLICK ANY WORD TO SEE WHAT THE MODEL HAD TO PREDICT FOR IT
MODEL’S DISTRIBUTION AT THAT POSITION (hand-set toy)
THE CAUSAL MASK — WHY ALL POSITIONS TRAIN AT ONCE
Row = the position predicting; column = what it may see. × = −∞: the answer is hidden from itself. One forward pass, six supervised predictions, zero labels.
Before the model can predict tokens, text must become tokens. GPT-2’s answer — byte-level BPE — runs for real, right here, one merge at a time.
04 BYTE-LEVEL BPE
A tokenizer you can train yourself
Word vocabularies hit an out-of-vocabulary wall (Lecture 04). GPT-2’s fix: start from 256 bytes and repeatedly merge the most frequent adjacent pair. Below is a genuine BPE learner on a doll-scale corpus — every merge is the real argmax over real counts, and the probe words re-tokenize live as the vocabulary grows.
MERGE RULES — most frequent pair first
THE CORPUS — tokenized with the vocabulary so far
THE REAL NUMBERS
Byte-level means no <UNK> can ever occur — any string in any language decomposes: frequent words become one or two pieces, rare or misspelled words cost a few more. cl100k (GPT-3.5/4-era) pushed the same idea to ~100k tokens for better multilingual compression. Watch it happen above: “low” and “est” merge early because your corpus uses them often.
Tokens become vectors, vectors ride the stack. But a 96-layer stack needs one quiet architectural change before it can train at all — the block had to be rewired.
05 THE PRE-LN BLOCK
Depth without collapse
The 2017 block normalized after adding the sublayer output (post-LN). Gradient-wise that puts a potentially shrinking factor on every level of the stack — fine at 6 layers, fatal at 96. GPT-2’s fix: move LayerNorm inside the branch (pre-LN), keep the residual rail pure, and add one final LayerNorm after the last block. Drag the depth and watch what GPT-3’s 96 layers demanded.
2017 · POST-LN (GPT-1 kept this)
GPT-2 → · PRE-LN
RELATIVE GRADIENT MAGNITUDE AT INIT, TOP OF STACK → BOTTOM (illustrative per-layer factors: post 0.62 · pre 1.00)
THE REWIRED BLOCK, IN ONE LINE
At initialization this makes the whole stack a near-identity map — gradients reach the bottom almost undamped, and the Noam warmup of Lecture 08 becomes unnecessary. Two more GPT signatures in the same block: GELU instead of ReLU in the FFN, and learned positional embeddings instead of fixed sin/cos waves.
The block trains. Now the story of the decade: what happens when you make it bigger — 12 layers, then 48, then 96 — and feed it the web?
06 SCALE
117M → 1.5B → 175B
Kaplan et al. (2020) measured what everyone suspected: validation loss falls as a smooth power law in parameters, data and compute — straight lines on log-log axes, over many orders of magnitude. That predictability is why labs bet billions on “just make it bigger.” Drag the slider along the law; the three GPTs sit exactly on it.
Training is only half the product. The other half is the moment a user presses enter: how do you turn 50,257 logits into ONE next token? That choice — the sampler — shapes the personality of every model you have ever chatted with.
07 SAMPLING
Temperature · top-k · top-p
The model outputs a distribution; the sampler turns it into a token. Temperature rescales the logits before softmax (T<1 sharper, T>1 flatter). Top-k keeps the k most likely tokens. Top-p — the nucleus — keeps the smallest set whose cumulative probability reaches p, so it adapts: confident steps keep few candidates, uncertain steps keep many. All three, live, on one distribution:
NEXT WORD AFTER “THE CAPITAL OF FRANCE IS…” (toy logits)
IN ONE PLACE — THE ORDER MATTERS
Defaults of the era: GPT-2 papers used pure sampling or top-k=40; ChatGPT-class products typically run temperature ≈ 0.7 with top-p ≈ 0.9–1.0. T→0 collapses to greedy — deterministic, safe, and famously prone to looping (“of the French Republic. … of the French Republic.”). You just watched why in the code lab.
Sampling makes generation creative — and expensive. Each new token re-feeds the whole prefix… unless you stop recomputing what you already know. Enter the cache that makes chat instant.
08 THE LOOP & THE KV CACHE
Don’t recompute the past
Step t feeds tokens 1…t through the stack — but tokens 1…t−1 already produced their Keys and Values last step, and attention is the only thing that reads them. The KV cache stores those matrices once; each new step computes exactly one new K and V and attends against the stored past. Step the loop and watch the two bills diverge.
THE CACHE — one cell per past token’s K·V, filled once
The cache holds the past — but only as much of it as the window allows. What does a model actually “see” when it answers you?
09 THE CONTEXT WINDOW
Working memory, not knowledge
Everything the model can attend to lives inside one fixed-length token window: your prompt, its own previous words, any documents you paste. Beyond the edge, attention scores are exactly zero — those words are not forgotten, they are never seen. Drag the window and feel the cliff.
The weights hold what training put there — facts, style, skills. The window holds this conversation. Long-context tricks (rope scaling, sliding windows) all negotiate the same budget you just dragged.
512 → 1024 → 2048 tokens. Each doubling changed what fits: a paragraph, a page, a few pages — and with it, whole new use-cases (“paste the document”).
Knowledge in weights is frozen at training time and expensive to update; the window is small but live. Retrieval splits the job: fetch fresh text into the window, reason with frozen weights.
Scale made the model capable. The next two sections made it useful — first a prompting trick that needs no training at all, then the alignment recipe that turned a text completer into an assistant.
10 ZERO-SHOT → FEW-SHOT → FINE-TUNE
Learning inside the window
GPT-1 fine-tuned a separate head per task. GPT-2 dropped the head and asked zero-shot. GPT-3’s discovery: put worked examples in the prompt and the same frozen weights get dramatically better — few-shot, or in-context learning. Compose the prompt below and watch the format teach what no gradient updated.
TASK
EXAMPLES IN THE PROMPT
THE PROMPT — everything the model sees
MODEL COMPLETION (same weights every time)
Powerful ≠ helpful. Left alone, a next-token completer answers questions with more questions and politely continues scams. The fix — three stages, one of them with you in the annotator seat.
11 RLHF
From GPT-3 to ChatGPT
Pretraining optimizes plausibility, not usefulness. InstructGPT (2022) bolted on three stages: imitate human demonstrators, learn human preferences from comparisons, then optimize the policy against that learned reward — tethered to the pretrained model by a KL penalty. That recipe, wrapped in a chat format, is ChatGPT.
THE THREE-STAGE PIPELINE
STAGE 2, HANDS-ON — YOU ARE THE ANNOTATOR
pick the better responseTOY REWARD-MODEL CONFIDENCE (grows with each preference pair)
50%
Before the quiz, the ledger every fan and every skeptic should read: what this architecture genuinely cannot do — demonstrated, not asserted.
12 THE HONEST LEDGER
Confidence is not correctness
The training objective rewards plausible next tokens — never true ones. Every limitation below is a direct corollary of that one sentence. Start with the demo: ask about a book that does not exist and watch the model answer fluently anyway.
PROMPT: “Who wrote ‘The Moon and the Sixth Continent’?”
Fluent fabrication is not a bug in the sampler — it is the objective doing its job. Grounding (RAG, tools, citations) works by moving the evidence into the window, where copying beats inventing.
Web-scale training memorizes some of its data verbatim — extraction attacks proved it. Deduplication, and later alignment, reduce but never fully remove it.
Everything after the training cutoff, and everything outside the window, is equally invisible. The model has no lookup reflex — it has a distribution over plausible continuations.
Training runs of this class cost millions of dollars and megawatts; inference multiplies it by every user and every token. Efficiency research — quantization, distillation, speculative decoding — is the other half of the GPT story.
13 CODE LAB
The lecture, in runnable code
Three tabs: the pre-LN block and tied head in PyTorch (§02+§05, with the chance-level loss you can verify — ln 50257 ≈ 10.82), generation with the real sampler settings (§07), and the BPE encode loop (§04).
14 MASTERY CHECK