Multi-Query Attention (MQA) — One Write-Head Is All You Need
Standard Multi-Head Attention (MHA) stores H separate Key caches and H separate Value caches — forcing GPUs to reload H duplicated archives across slow memory for every generated token. Multi-Query Attention keeps all H independent Query heads so every head still asks a different question, while sharing ONE single Key and ONE single Value shelf across them all — cutting KV VRAM and decode HBM traffic by H times.
01 SIX VISUAL IDEAS THAT MAKE MQA CLICK
One shared Key/Value shelf, H specialized Query hunters
02 THE H× MEMORY WALL
From O(H) Duplicate Caches to O(1) Shared Write-Head
Pick a model architecture or drag the Query Heads (H) slider below. Watch standard MHA's Key-Value cache climb linearly as O(H) while MQA stays completely flat at 1 single shared KV head — turning 568 KB/tok into 8 KB/tok on Falcon-7B!
1 · O(H) → O(1) KV Memory Scaling Curve
Toggle Linear vs Log10 scale and click any head count H along the curve2 · Visual Proof: Collapse H Libraries → 1 Shelf
Step or Play to watch H duplicate KV columns merge into one shared write-head03 INTERACTIVE MQA VISUAL LEARNING STUDIO
Inspect Every Head, Broadcast Ray & Shared Shelf Live
Switch between Toy 4-Head, Shazeer 8-Head, StarCoder/PaLM 48-Head, and Falcon-7B 71-Head configurations. Explore how 1 shared Key-Value shelf serves every Query head differently!
04 INTUITION PROOF
Why Sharing Keys & Values Works — While Keeping H Queries
Plain-English visual answers to every student's #1 question: "If all heads share ONE Key and Value matrix, won't all attention heads become identical clones?" Spoiler: No — here is visually why!
05 SHAZEER (2019) & POPE ET AL. (MLSYS 2023) · SYSTEMS REALITY
Breaking the HBM Memory-Bandwidth Wall in Incremental Decode
Autoregressive text generation is NOT limited by how fast tensor cores multiply — it stalls waiting for past KV caches to travel across slow HBM memory wires. Moving 1 shared shelf instead of H duplicate shelves delivers a 12.1× speedup.
06 HAND-CALCULATED 6-DECIMAL VERIFICATION
Two Query Heads Attending to 1 Shared KV Cache on "The chef baked bread"
Pre-calibrated in FP64 (d_model=8, H_q=2 query heads, H_kv=1 shared write-head, d_h=4). Step through tokens 1→4 to verify that Head 0 and Head 1 read the exact same shared KV rows yet compute completely different attention patterns and outputs!
07 SINGLE MQA DECODER BLOCK PIPELINE
Tutorial Schematic: H Query Streams, 1 Shared Vault & Zero-Copy Expand
Pick a model scale (Shazeer 2019, Falcon-7B, or PaLM 540B), filter by stage, or click any block inside the schematic to inspect exact tensor dimensions and the upper Residual Bypass Rail.
08 INTERACTIVE MEMORY SIZER
MHA vs MQA VRAM Cache Calculator (FP16 / BF16)
Exact KV cache byte count: 2 · L · B · n_kv_heads · S · d_head · 2 bytes. Replacing n_kv_heads = H (MHA) with n_kv_heads = 1 (MQA) slashes VRAM by exactly H×!
09 EMPIRICAL RESULTS & DEBUGGING LAB
Shazeer (2019) Verified Benchmarks & 4 Real-Code MQA Pitfalls
Verified numbers directly from https://arxiv.org/abs/1911.02150 and https://arxiv.org/abs/2211.05102 — plus the 4 broadcasting and RoPE bugs that quietly break MQA inference in real codebases.
| Benchmark / Metric | Standard MHA | Multi-Query Attention (MQA) | Verified Impact | Primary Source |
|---|
Interactive MQA Production Bug Inspector
10 ARCHITECTURE LEDGER
Standard MHA vs Shrunk MHA vs Multi-Query Attention
Side-by-side summary of memory scaling, latency, translation quality, and real LLM deployments. (Historical note: Ainslie et al. 2023 later introduced Grouped-Query Attention GQA with 1 < H_kv < H to interpolate smoothly between MQA and MHA.)
| Engineering Property | Standard MHA (H_kv = H) | Shrinking Heads / d_k | Multi-Query Attention (H_kv = 1) |
|---|
11 PYTORCH & KERAS 3 CODE LAB
Write Clean Multi-Query Attention in PyTorch & Keras 3
Compare PyTorch zero-copy .expand(), backend-agnostic Keras 3 ops.einsum, static cache pre-allocation, and real-world LLM config flags.
12 8-QUESTION MASTERY QUIZ
Verify Your Multi-Query Attention Mastery
Test your intuition on query-key asymmetry, H× cache reduction, Shazeer's 46 µs → 3.8 µs TPUv2 decode gain, and zero-copy broadcasting.