CAS logo CAS NLP LABInteractive Tutorial Series ← Series
CAS-NLP LECTURE 33 (NEXT TO KV CACHE) · SHAZEER 2019 · ARXIV:1911.02150

Multi-Query Attention (MQA) — One Write-Head Is All You Need

Standard Multi-Head Attention (MHA) stores H separate Key caches and H separate Value caches — forcing GPUs to reload H duplicated archives across slow memory for every generated token. Multi-Query Attention keeps all H independent Query heads so every head still asks a different question, while sharing ONE single Key and ONE single Value shelf across them all — cutting KV VRAM and decode HBM traffic by H times.

H× Smaller KVMemory & Read Traffic8× (T5) to 71× (Falcon-7B)
46 µs → 3.8 µsTPUv2 Decode Step12.1× faster (Shazeer Table 2)
28.5 vs 28.4WMT14 En→De BLEUQuality preserved (Beam-4)
H Queries StayZero Q CollapseEvery head extracts distinct clues

01 SIX VISUAL IDEAS THAT MAKE MQA CLICK

One shared Key/Value shelf, H specialized Query hunters

02 THE H× MEMORY WALL

From O(H) Duplicate Caches to O(1) Shared Write-Head

Pick a model architecture or drag the Query Heads (H) slider below. Watch standard MHA's Key-Value cache climb linearly as O(H) while MQA stays completely flat at 1 single shared KV head — turning 568 KB/tok into 8 KB/tok on Falcon-7B!

Query Heads (H_q)8 Query Heads
Context Length (S tokens)2,048 tokens
MHA KV Cache O(H): H shelves
—
MQA KV Cache O(1): 1 shared shelf
—

1 · O(H) → O(1) KV Memory Scaling Curve

Toggle Linear vs Log10 scale and click any head count H along the curve

2 · Visual Proof: Collapse H Libraries → 1 Shelf

Step or Play to watch H duplicate KV columns merge into one shared write-head

03 INTERACTIVE MQA VISUAL LEARNING STUDIO

Inspect Every Head, Broadcast Ray & Shared Shelf Live

Switch between Toy 4-Head, Shazeer 8-Head, StarCoder/PaLM 48-Head, and Falcon-7B 71-Head configurations. Explore how 1 shared Key-Value shelf serves every Query head differently!

Scale

04 INTUITION PROOF

Why Sharing Keys & Values Works — While Keeping H Queries

Plain-English visual answers to every student's #1 question: "If all heads share ONE Key and Value matrix, won't all attention heads become identical clones?" Spoiler: No — here is visually why!

05 SHAZEER (2019) & POPE ET AL. (MLSYS 2023) · SYSTEMS REALITY

Breaking the HBM Memory-Bandwidth Wall in Incremental Decode

Autoregressive text generation is NOT limited by how fast tensor cores multiply — it stalls waiting for past KV caches to travel across slow HBM memory wires. Moving 1 shared shelf instead of H duplicate shelves delivers a 12.1× speedup.

06 HAND-CALCULATED 6-DECIMAL VERIFICATION

Two Query Heads Attending to 1 Shared KV Cache on "The chef baked bread"

Pre-calibrated in FP64 (d_model=8, H_q=2 query heads, H_kv=1 shared write-head, d_h=4). Step through tokens 1→4 to verify that Head 0 and Head 1 read the exact same shared KV rows yet compute completely different attention patterns and outputs!

07 SINGLE MQA DECODER BLOCK PIPELINE

Tutorial Schematic: H Query Streams, 1 Shared Vault & Zero-Copy Expand

Pick a model scale (Shazeer 2019, Falcon-7B, or PaLM 540B), filter by stage, or click any block inside the schematic to inspect exact tensor dimensions and the upper Residual Bypass Rail.

08 INTERACTIVE MEMORY SIZER

MHA vs MQA VRAM Cache Calculator (FP16 / BF16)

Exact KV cache byte count: 2 · L · B · n_kv_heads · S · d_head · 2 bytes. Replacing n_kv_heads = H (MHA) with n_kv_heads = 1 (MQA) slashes VRAM by exactly H×!

MHA KV Cache: 100% Baseline MQA KV Cache (1 / H of MHA)
Layers (L)32 layers
Query Heads (H_q)32 Q heads
Head Dimension (d_head)128 d_head
Sequence Length (S)4,096 tokens
Batch Size (B)8 batch

09 EMPIRICAL RESULTS & DEBUGGING LAB

Shazeer (2019) Verified Benchmarks & 4 Real-Code MQA Pitfalls

Verified numbers directly from https://arxiv.org/abs/1911.02150 and https://arxiv.org/abs/2211.05102 — plus the 4 broadcasting and RoPE bugs that quietly break MQA inference in real codebases.

Benchmark / Metric Standard MHA Multi-Query Attention (MQA) Verified Impact Primary Source

Interactive MQA Production Bug Inspector

10 ARCHITECTURE LEDGER

Standard MHA vs Shrunk MHA vs Multi-Query Attention

Side-by-side summary of memory scaling, latency, translation quality, and real LLM deployments. (Historical note: Ainslie et al. 2023 later introduced Grouped-Query Attention GQA with 1 < H_kv < H to interpolate smoothly between MQA and MHA.)

Engineering Property Standard MHA (H_kv = H) Shrinking Heads / d_k Multi-Query Attention (H_kv = 1)

11 PYTORCH & KERAS 3 CODE LAB

Write Clean Multi-Query Attention in PyTorch & Keras 3

Compare PyTorch zero-copy .expand(), backend-agnostic Keras 3 ops.einsum, static cache pre-allocation, and real-world LLM config flags.

12 8-QUESTION MASTERY QUIZ

Verify Your Multi-Query Attention Mastery

Test your intuition on query-key asymmetry, H× cache reduction, Shazeer's 46 µs → 3.8 µs TPUv2 decode gain, and zero-copy broadcasting.

Live Score: 0 / 8