GPT-3 — Scaling Autoregressive Transformers to 175B Parameters & Few-Shot In-Context Learning
Scaling GPT-2’s decoder-only architecture by 116x (from 1.5B to 175B parameters across 96 layers × 12,288 width) on 300B weighted tokens unlocked a fundamentally new machine learning paradigm: In-Context Learning. Without any task-specific datasets or gradient updates, GPT-3 absorbs zero-shot instructions and K-shot demonstrations directly through its 2,048-token prompt window — scoring 86.4% on LAMBADA, 71.2% closed-book TriviaQA, 71.8 SuperGLUE, and 80.4% 3-digit addition.
Interactive Breakthrough Explorer: Three Pillars of GPT-3
Fine-Tuning vs Zero-Shot, One-Shot & Few-Shot In-Context Learning
Contrast the classic BERT-era outer-loop fine-tuning workflow against GPT-3’s frozen-weight forward conditioning inside the 2,048-token prompt.
Interactive 12-Block GPT-3 Architecture Diagram (125M to 175B)
Inspect how GPT-3 extends GPT-2’s Pre-LN Transformer decoder with a doubled 2,048-token window, alternating dense and locally-banded sparse causal attention, depth-scaled residual init, and 4x GELU FFN blocks.
Table 2.1: Complete 8-Model Parameter Scaling Spectrum (125M → 175B)
Select any scale chip below to highlight exact layer depth, width, head count, batch size (in millions of tokens), learning rate schedule, and pretraining FLOPs from OpenAI Table 2.1.
Alternating Dense & Locally-Banded Sparse Causal Attention
Switch between GPT-3 even layers (full lower-triangular causal attention) and odd layers (Sparse Transformer locally-banded causal window) on the prompt ["25", "+", "46", "="].
Exact 6-Decimal Pure-JS Forward Trace (["25", "+", "46", "="] → "71")
Follow token t=3 ("=") step by step through Pre-LN, Dense vs Banded causal softmax, depth-scaled 0.5 residual addition, 4x GELU MLP, and next-token probability P("71") = 0.537971 (L_CLM = 0.619950, PPL = 1.858835).
Table 2.2: Quality-Weighted 300B-Token Training Mixture
GPT-3 is trained on 300 billion tokens sampled proportionally to quality rather than raw byte volume — upsampling WebText2 and Wikipedia ~3 epochs while sub-sampling filtered CommonCrawl once.
Interactive K-Shot Prompt Formatter & 2,048-Token Budget Lab
Toggle shot count K = 0, 1, 10, or 32 inside n_ctx = 2,048 to inspect prompt structure, context consumption, and zero-gradient task conditioning.
Emergent Scaling Phase Transitions in Multi-Digit Arithmetic
Inspect how symbolic math accuracy sits near 0–5% through 13B before jumping sharply at 175B (100% 2-digit, 80.4% 3-digit addition few-shot).
LAMBADA, Closed-Book TriviaQA & SuperGLUE Few-Shot Results
Compare 125M, 13B, and 175B across OpenAI May 2020 evaluation suites.
Data Contamination Audit, Bidirectional Deficits & Broader Limitations
Brown et al. devote entire sections to honest weaknesses: bidirectional comparison gaps (WiC/ANLI), 13-gram test overlap audits, and compute/safety bounds.
Autoregressive GPT Dynasty (GPT-1 → GPT-2 → GPT-3 → InstructGPT)
Pure-JS vs Python FP64 Verification Ledger
Every scalar and tensor shown in Section 06 runs live in Pure JavaScript inside your browser and matches calibrated Python FP64 outputs to within 1e-12.