CAS logo CAS NLP LABInteractive Tutorial Series ← Series

⌗  GPT — GENERATIVE PRETRAINED TRANSFORMERS · Radford et al. 2018 → Brown et al. 2020 → ChatGPT 2022

GPT — the decoder, alone

Lecture 11 ended with three descendants of the Transformer. This lecture chases the one that became the large language model: keep only the decoder half, train it on one absurdly simple objective — predict the next token — on a slice of the whole internet, and scale it 1,500×. Every mechanism below runs live: a toy GPT writing token by token, a genuine byte-pair tokenizer you train by hand, the pre-LN block, the scaling law, the samplers behind every chat box, the KV cache, and the RLHF recipe that turned GPT-3 into ChatGPT.

RAW TEXT→ BYTE BPE→ EMBED + POS→ MASKED DECODER ×12…×96→ LOGITS→ SAMPLE→ APPEND→ LOOP
14Sectionsintuition → math → code
14Live Demosincl. the clickable GPT stack
1,500×GPT-1 → GPT-3117M → 175B parameters
8Quiz Checksthe mastery gate
THE ONE-SENTENCE SUMMARY

A GPT is a causal-masked Transformer decoder trained to predict the next token on web-scale text — one loss, no labels, every position at once. Everything the world now calls an LLM — few-shot learning, chat, code generation — emerged from that single objective plus scale, and (for chat) one alignment stage on top: RLHF.

THE WHOLE SHOW IN ONE LOOP

This is generation, the entire mechanism of every GPT product: look at what exists, predict a distribution over the next token, draw one sample, append it, repeat. Below, a tiny hand-crafted distribution table plays the role of the 96-block model — the loop, the probabilities and the sampling are the real thing.

Then walk the lecture: what GPT deleted (§02), the objective (§03), the tokenizer (§04), the block (§05), the scale story (§06), how answers are sampled (§07), how generation is made affordable (§08), and how GPT-3 became an assistant (§10–11).

THE TEXT SO FAR

P(next | entire prefix)

MODEL’S NEXT-TOKEN DISTRIBUTION — computed live

Lecture 11 built the full machine and met its three descendants. GPT’s first move was subtraction — press the button and delete half the architecture.

02 FROM TRANSFORMER TO GPT

The decoder, alone

2017 built two stacks: an encoder to read a source, a decoder to write a target, cross-attention bridging them. GPT’s question: if pretraining teaches a decoder to predict the next token of ordinary text, what is the encoder still for? Answer: nothing. One stack remains — and with it, every task in the world becomes a text-in-text-out problem.

THE 2017 MACHINE vs THE GPT MACHINE

deletedencoder stack · cross-attention · the second sequence · sinusoidal positions (GPT learns them) · post-LN (moved inside)
keptmasked self-attention · FFN · residuals · softmax head — now tied to the input embedding
addedbyte-level BPE tokenizer · pre-LN blocks + final LayerNorm · one objective for everything
213M → 1.5B → 175BTransformer-big (2017) → GPT-2 → GPT-3: the same decoder, 800× bigger, trained on the open web

THE GPT STACK — CLICK ANY BLOCK TO OPEN ITS INNER MACHINERY

sample → append → loop

Hover to trace a block’s wires · click to open its inner machinery.

The bet that paid off a trillion times: training on raw next-token prediction needs no labels — the text labels itself. Data ceiling moves from “thousands of annotated pairs” (Lecture 07) to “however much text exists.” Scale became the story (§06), and everything else in this lecture is what that scale required.

One stack remains. Now the question that made the decade: what exactly is it trained to do? The answer is embarrassingly small — one prediction per token.

03 ONE OBJECTIVE

Predict the next token

Every position plays the same game: given everything to its left, guess the next token. The causal mask — Lecture 08’s −∞ wall — is what makes this legal for the whole sequence at once: position t can never peek at the answer it is asked to predict. Click any position and read its loss; the average over billions of positions is the entire pretraining loss.

THE SENTENCE — CLICK ANY WORD TO SEE WHAT THE MODEL HAD TO PREDICT FOR IT

MODEL’S DISTRIBUTION AT THAT POSITION (hand-set toy)

THE CAUSAL MASK — WHY ALL POSITIONS TRAIN AT ONCE

Row = the position predicting; column = what it may see. × = −∞: the answer is hidden from itself. One forward pass, six supervised predictions, zero labels.

L = − (1/n) Σᵢ log P(tokeni+1 | tokens≤i)

Why such a dumb objective learns so much: to predict the next token well you need grammar (agreement, subcategorization), facts (“the capital of France is →”), style, and — on code and math — step-by-step reasoning. The task is shallow; the best possible solution to it is deep. Nothing about translation or question-answering was ever programmed in — they fell out as side-effects, which is exactly what GPT-2’s zero-shot results showed.

Before the model can predict tokens, text must become tokens. GPT-2’s answer — byte-level BPE — runs for real, right here, one merge at a time.

04 BYTE-LEVEL BPE

A tokenizer you can train yourself

Word vocabularies hit an out-of-vocabulary wall (Lecture 04). GPT-2’s fix: start from 256 bytes and repeatedly merge the most frequent adjacent pair. Below is a genuine BPE learner on a doll-scale corpus — every merge is the real argmax over real counts, and the probe words re-tokenize live as the vocabulary grows.

vocabulary: —

MERGE RULES — most frequent pair first

THE CORPUS — tokenized with the vocabulary so far

THE REAL NUMBERS

GPT-2: 256 bytes + 50,000 learned merges + 1 special = 50,257 tokens · trained on WebText

Byte-level means no <UNK> can ever occur — any string in any language decomposes: frequent words become one or two pieces, rare or misspelled words cost a few more. cl100k (GPT-3.5/4-era) pushed the same idea to ~100k tokens for better multilingual compression. Watch it happen above: “low” and “est” merge early because your corpus uses them often.

Tokens become vectors, vectors ride the stack. But a 96-layer stack needs one quiet architectural change before it can train at all — the block had to be rewired.

05 THE PRE-LN BLOCK

Depth without collapse

The 2017 block normalized after adding the sublayer output (post-LN). Gradient-wise that puts a potentially shrinking factor on every level of the stack — fine at 6 layers, fatal at 96. GPT-2’s fix: move LayerNorm inside the branch (pre-LN), keep the residual rail pure, and add one final LayerNorm after the last block. Drag the depth and watch what GPT-3’s 96 layers demanded.

2017 · POST-LN (GPT-1 kept this)

GPT-2 → · PRE-LN

STACK DEPTH (blocks)12
6 · GPT-112 · GPT-1/GPT-2-small48 · GPT-296 · GPT-3

RELATIVE GRADIENT MAGNITUDE AT INIT, TOP OF STACK → BOTTOM (illustrative per-layer factors: post 0.62 · pre 1.00)

post-LN
—
pre-LN
—

THE REWIRED BLOCK, IN ONE LINE

x = x + Attn(LN(x))  ·  x = x + FFN(LN(x))  ·  output = LN_f(x)  — LN moves inside; the rail stays clean

At initialization this makes the whole stack a near-identity map — gradients reach the bottom almost undamped, and the Noam warmup of Lecture 08 becomes unnecessary. Two more GPT signatures in the same block: GELU instead of ReLU in the FFN, and learned positional embeddings instead of fixed sin/cos waves.

The block trains. Now the story of the decade: what happens when you make it bigger — 12 layers, then 48, then 96 — and feed it the web?

06 SCALE

117M → 1.5B → 175B

Kaplan et al. (2020) measured what everyone suspected: validation loss falls as a smooth power law in parameters, data and compute — straight lines on log-log axes, over many orders of magnitude. That predictability is why labs bet billions on “just make it bigger.” Drag the slider along the law; the three GPTs sit exactly on it.

PARAMETERS (log)—
—predicted loss (L = (8.8×10¹³/N)^0.076, Kaplan et al. 2020)

The scaling law’s fine print: smooth loss → smooth capability? Not quite — many behaviors (arithmetic, translation quality) improve gradually, but the 2022 “emergent abilities” literature documented thresholds where certain benchmarks jump. And Chinchilla (2022) corrected the recipe: for a fixed compute budget, parameters and training tokens must scale together (~20 tokens per parameter) — GPT-3 was trained on fewer tokens than its size would later demand.

Training is only half the product. The other half is the moment a user presses enter: how do you turn 50,257 logits into ONE next token? That choice — the sampler — shapes the personality of every model you have ever chatted with.

07 SAMPLING

Temperature · top-k · top-p

The model outputs a distribution; the sampler turns it into a token. Temperature rescales the logits before softmax (T<1 sharper, T>1 flatter). Top-k keeps the k most likely tokens. Top-p — the nucleus — keeps the smallest set whose cumulative probability reaches p, so it adapts: confident steps keep few candidates, uncertain steps keep many. All three, live, on one distribution:

NEXT WORD AFTER “THE CAPITAL OF FRANCE IS…” (toy logits)

TEMPERATURE T0.70
TOP-K10
TOP-P0.90

IN ONE PLACE — THE ORDER MATTERS

probs = softmax(logits / T) → keep top-k → keep smallest set with Σp ≥ p → renormalize → sample

Defaults of the era: GPT-2 papers used pure sampling or top-k=40; ChatGPT-class products typically run temperature ≈ 0.7 with top-p ≈ 0.9–1.0. T→0 collapses to greedy — deterministic, safe, and famously prone to looping (“of the French Republic. … of the French Republic.”). You just watched why in the code lab.

Sampling makes generation creative — and expensive. Each new token re-feeds the whole prefix… unless you stop recomputing what you already know. Enter the cache that makes chat instant.

08 THE LOOP & THE KV CACHE

Don’t recompute the past

Step t feeds tokens 1…t through the stack — but tokens 1…t−1 already produced their Keys and Values last step, and attention is the only thing that reads them. The KV cache stores those matrices once; each new step computes exactly one new K and V and attends against the stored past. Step the loop and watch the two bills diverge.

THE CACHE — one cell per past token’s K·V, filled once

WITHOUT CACHE — total K·V units computed
0
WITH CACHE
0

The cache’s own bill: memory grows linearly with context × layers × heads — at GPT-3 scale, megabytes per token of conversation. This is the engineering heart of the context-window arms race: KV compression, paged attention, and sliding-window caches all exist to shrink the strip above while pretending it never ends.

The cache holds the past — but only as much of it as the window allows. What does a model actually “see” when it answers you?

09 THE CONTEXT WINDOW

Working memory, not knowledge

Everything the model can attend to lives inside one fixed-length token window: your prompt, its own previous words, any documents you paste. Beyond the edge, attention scores are exactly zero — those words are not forgotten, they are never seen. Drag the window and feel the cliff.

WINDOW SIZE (tokens of this strip)26

Window = state

The weights hold what training put there — facts, style, skills. The window holds this conversation. Long-context tricks (rope scaling, sliding windows) all negotiate the same budget you just dragged.

GPT-1 → GPT-3

512 → 1024 → 2048 tokens. Each doubling changed what fits: a paragraph, a page, a few pages — and with it, whole new use-cases (“paste the document”).

Why RAG exists

Knowledge in weights is frozen at training time and expensive to update; the window is small but live. Retrieval splits the job: fetch fresh text into the window, reason with frozen weights.

Scale made the model capable. The next two sections made it useful — first a prompting trick that needs no training at all, then the alignment recipe that turned a text completer into an assistant.

10 ZERO-SHOT → FEW-SHOT → FINE-TUNE

Learning inside the window

GPT-1 fine-tuned a separate head per task. GPT-2 dropped the head and asked zero-shot. GPT-3’s discovery: put worked examples in the prompt and the same frozen weights get dramatically better — few-shot, or in-context learning. Compose the prompt below and watch the format teach what no gradient updated.

TASK

EXAMPLES IN THE PROMPT

THE PROMPT — everything the model sees


          

MODEL COMPLETION (same weights every time)

Few-shot vs fine-tuning — the honest split: few-shot is instant, per-request, and dies with the context; fine-tuning changes weights permanently, costs a training run, and wins when the task is fixed and you own the data. GPT-3’s paper measured the gap closing as models grew — 175B few-shot beating 175B zero-shot by wide margins, and beating fine-tuned smaller models on many tasks.

Powerful ≠ helpful. Left alone, a next-token completer answers questions with more questions and politely continues scams. The fix — three stages, one of them with you in the annotator seat.

11 RLHF

From GPT-3 to ChatGPT

Pretraining optimizes plausibility, not usefulness. InstructGPT (2022) bolted on three stages: imitate human demonstrators, learn human preferences from comparisons, then optimize the policy against that learned reward — tethered to the pretrained model by a KL penalty. That recipe, wrapped in a chat format, is ChatGPT.

THE THREE-STAGE PIPELINE

STAGE 2, HANDS-ON — YOU ARE THE ANNOTATOR

pick the better response

TOY REWARD-MODEL CONFIDENCE (grows with each preference pair)

50%

The punchline of the InstructGPT paper: a 1.3B RLHF model was preferred over the 175B GPT-3 — alignment bought more usefulness than 134× more parameters. Alignment also buys the failure modes of §12: the model learns to be confidently helpful, and sometimes confidently wrong.

Before the quiz, the ledger every fan and every skeptic should read: what this architecture genuinely cannot do — demonstrated, not asserted.

12 THE HONEST LEDGER

Confidence is not correctness

The training objective rewards plausible next tokens — never true ones. Every limitation below is a direct corollary of that one sentence. Start with the demo: ask about a book that does not exist and watch the model answer fluently anyway.

PROMPT: “Who wrote ‘The Moon and the Sixth Continent’?”

The right answer: the book does not exist. The toy distribution spends 62% inventing an author — because “[plausible name]” is a perfectly likely continuation of the text. RLHF reduced this behavior but cannot eliminate it: the objective still has no term for truth.

Hallucination

Fluent fabrication is not a bug in the sampler — it is the objective doing its job. Grounding (RAG, tools, citations) works by moving the evidence into the window, where copying beats inventing.

Memorization vs generalization

Web-scale training memorizes some of its data verbatim — extraction attacks proved it. Deduplication, and later alignment, reduce but never fully remove it.

The window is not knowledge

Everything after the training cutoff, and everything outside the window, is equally invisible. The model has no lookup reflex — it has a distribution over plausible continuations.

The bill

Training runs of this class cost millions of dollars and megawatts; inference multiplies it by every user and every token. Efficiency research — quantization, distillation, speculative decoding — is the other half of the GPT story.

13 CODE LAB

The lecture, in runnable code

Three tabs: the pre-LN block and tied head in PyTorch (§02+§05, with the chance-level loss you can verify — ln 50257 ≈ 10.82), generation with the real sampler settings (§07), and the BPE encode loop (§04).

GPT · THREE WAYS

14 MASTERY CHECK

Eight questions — one per mechanism.

YOUR SCORE

0 / 8