CAS · L32
← GPT-2
Brown et al. · OpenAI · May 2020 · arXiv:2005.14165 · Lecture 32 (Directly After L15 GPT-2)

GPT-3 — Scaling Autoregressive Transformers to 175B Parameters & Few-Shot In-Context Learning

Scaling GPT-2’s decoder-only architecture by 116x (from 1.5B to 175B parameters across 96 layers × 12,288 width) on 300B weighted tokens unlocked a fundamentally new machine learning paradigm: In-Context Learning. Without any task-specific datasets or gradient updates, GPT-3 absorbs zero-shot instructions and K-shot demonstrations directly through its 2,048-token prompt window — scoring 86.4% on LAMBADA, 71.2% closed-book TriviaQA, 71.8 SuperGLUE, and 80.4% 3-digit addition.

175B
96L × 12,288 · 96 Heads (128d)
300B / 2,048
Weighted Tokens · Prompt Ctx
3.14e23
Pretraining Compute FLOPs
86.4% / 71.2%
LAMBADA & TriviaQA Few-Shot
0 Updates
Pure Activation Conditioning

Interactive Breakthrough Explorer: Three Pillars of GPT-3

02

Fine-Tuning vs Zero-Shot, One-Shot & Few-Shot In-Context Learning

Contrast the classic BERT-era outer-loop fine-tuning workflow against GPT-3’s frozen-weight forward conditioning inside the 2,048-token prompt.

03

Interactive 12-Block GPT-3 Architecture Diagram (125M to 175B)

Inspect how GPT-3 extends GPT-2’s Pre-LN Transformer decoder with a doubled 2,048-token window, alternating dense and locally-banded sparse causal attention, depth-scaled residual init, and 4x GELU FFN blocks.

04

Table 2.1: Complete 8-Model Parameter Scaling Spectrum (125M → 175B)

Select any scale chip below to highlight exact layer depth, width, head count, batch size (in millions of tokens), learning rate schedule, and pretraining FLOPs from OpenAI Table 2.1.

05

Alternating Dense & Locally-Banded Sparse Causal Attention

Switch between GPT-3 even layers (full lower-triangular causal attention) and odd layers (Sparse Transformer locally-banded causal window) on the prompt ["25", "+", "46", "="].

06

Exact 6-Decimal Pure-JS Forward Trace (["25", "+", "46", "="] → "71")

Follow token t=3 ("=") step by step through Pre-LN, Dense vs Banded causal softmax, depth-scaled 0.5 residual addition, 4x GELU MLP, and next-token probability P("71") = 0.537971 (L_CLM = 0.619950, PPL = 1.858835).

STEP 1 / 6
07

Table 2.2: Quality-Weighted 300B-Token Training Mixture

GPT-3 is trained on 300 billion tokens sampled proportionally to quality rather than raw byte volume — upsampling WebText2 and Wikipedia ~3 epochs while sub-sampling filtered CommonCrawl once.

08

Interactive K-Shot Prompt Formatter & 2,048-Token Budget Lab

Toggle shot count K = 0, 1, 10, or 32 inside n_ctx = 2,048 to inspect prompt structure, context consumption, and zero-gradient task conditioning.

09

Emergent Scaling Phase Transitions in Multi-Digit Arithmetic

Inspect how symbolic math accuracy sits near 0–5% through 13B before jumping sharply at 175B (100% 2-digit, 80.4% 3-digit addition few-shot).

10

LAMBADA, Closed-Book TriviaQA & SuperGLUE Few-Shot Results

Compare 125M, 13B, and 175B across OpenAI May 2020 evaluation suites.

11

Data Contamination Audit, Bidirectional Deficits & Broader Limitations

Brown et al. devote entire sections to honest weaknesses: bidirectional comparison gaps (WiC/ANLI), 13-gram test overlap audits, and compute/safety bounds.

12

Autoregressive GPT Dynasty (GPT-1 → GPT-2 → GPT-3 → InstructGPT)

13

Pure-JS vs Python FP64 Verification Ledger

Every scalar and tensor shown in Section 06 runs live in Pure JavaScript inside your browser and matches calibrated Python FP64 outputs to within 1e-12.

14

4 Production PyTorch & Few-Shot Prompting Recipes

15

8-Question GPT-3 Mastery Check

MASTERY SCORE 0 / 8