Grouped-Query Attention (GQA)
Standard MHA gives every chef a massive private fridge (peak quality 47.2, sluggish 1.51 s T5-XXL decode). Extreme MQA forces all 64 queries to cram into one mini-fridge (fast 0.24 s, drops to 46.6). Grouped-Query Attention (GQA) splits queries into G = 8 team prep stations — slashing inference latency 5.4× to 0.28 s while hitting 47.1 quality and powering Llama-2/3, Mistral-7B, Gemma-2, and Qwen2.5.
Six Pillars of Grouped-Query Attention
Multi-Head Attention gives 64 separate fridges. Multi-Query squeezes everyone into 1. What happens when you smoothly sweep G from 64 down to 1?
↓ 02 · THE G = 1..H_q SPECTRUM02 THE MHA ↔ GQA ↔ MQA SPECTRUM
Sliding G Between Full Quality & Peak Speed
Start with concrete hardware intuition: each GPU decode step must haul every cached Key and Value vector out of slow High-Bandwidth Memory (HBM). Keeping G = H_q copies forces huge memory traffic; shrinking G to 1 (MQA) sacrifices cross-head subspace diversity. Pick a model below and sweep G to watch why G = 8 dominates modern LLM architecture design.
Interactive G = 1..H_q Knee
Watch Queries Partition Into Groups
How can 4 or 8 distinct Query heads look at the exact same Key/Value shelf inside a group without turning into copies of each other?
↓ 03 · 4-LENS GQA VISUAL LEARNING STUDIO03 VISUAL LEARNING STUDIO
Four Interactive Lenses on Grouped Sharing
Switch between MHA-vs-GQA team partitioning, live attention weight specialization inside a single group, checkpoint mean-pooling strategies, and multi-layer KV bank footprints across Toy, Mistral-7B, Llama-70B, and Gemma-2 scales.
Now let us step through the exact kitchen analogy, checkpoint mean-pooling proof, and hardware sweet spot in wide, textbook-clear diagrams.
↓ 04 · INTUITION BEFORE EQUATIONS04 WHY GROUPS WORK
From Kitchen Analogies to Tensor-Parallel GPUs
Three crisp mental models explaining why partitioning H_q queries into G = 8 groups keeps MHA expressivity, converts pretrained weights without full retraining, and fits 8-GPU nodes natively.
Nobody wants to retrain a 70B LLM from scratch just to add GQA. Ainslie et al. discovered a surgical 2-step conversion that takes only 5% compute.
↓ 05 · MEAN-POOLING + 5% UPTRAINING05 CHECKPOINT SURGERY · AINSLIE ET AL.
MHA → GQA in Two Steps: Mean-Pool + 5% Uptraining
Existing Multi-Head Attention checkpoints already contain rich semantic features in all H_q heads. Instead of discarding heads or randomizing KV projections, mean-pool each group of g adjacent K/V matrices and run 5% continued pretraining.
Let us freeze all abstractions and trace real FP64 numbers by hand across 4 tokens, 4 Query heads, and 2 KV groups.
↓ 06 · HAND-CALIBRATED NUMERICAL TRACE06 BY HAND ON 4 TOKENS
Exact Numbers: 4 Queries, 2 KV Groups (g = 2)
Sentence: "The chef baked bread" (T = 4, d_model = 8, H_q = 4, G = 2, d_head = 2). Click each decoding step to verify every vector down to 4 decimal places — plus exact 1.11e-16 parity against repeat_interleave and the 1.033 error when .repeat() is misused!
Now zoom into an entire Transformer decoder attention block to watch Residual Bypass, G-Group KV Vaults, and zero-copy 5D pairing work together.
↓ 07 · SCHEMATIC GQA DECODER PIPELINE07 SINGLE DECODER LAYER PIPELINE
Schematic GQA Attention Block & Tensor Shapes
Toggle between Mistral-7B (4:1), Llama-70B (8:1), and Gemma-2 9B (2:1), isolate individual pipeline stages, and click any module on the diagram to inspect its exact tensor formula and hardware role.
How many gigabytes of HBM VRAM does switching from MHA to GQA save on an 80-layer 70B model serving 8 concurrent requests?
↓ 08 · 3-WAY PRODUCTION VRAM CALCULATOR08 PRODUCTION VRAM CALCULATOR
3-Way KV Memory: MHA vs GQA vs MQA
KV cache VRAM scales linearly with layers L, active KV heads G, head width d_head, sequence length S, batch size B, and 2 bytes per FP16 scalar (times 2 for Key + Value). Test real frontier configs below.
KV_bytes = 2 (K+V) * L * G * d_head * S * B * 2 bytes (FP16)
Now verify the exact published Table 1 benchmarks across summarization, translation, and QA — plus the 4 bugs engineers hit most in production.
↓ 09 · AINSLIE TABLE 1 & BUG PITFALLS09 BENCHMARKS & BUG PITFALLS
Table 1 Verification & 4 GQA Code Traps
Exact TPUv4 latency and benchmark scores reported by Ainslie et al. (EMNLP 2023 Table 1), alongside an interactive bug debugger highlighting why .repeat() quietly scrambles attention heads.
| Model | TPUv4 T_infer | Avg | CNN | arXiv | PubMed | MediaSum | MultiNews | WMT | TriviaQA |
|---|
Put MHA, GQA, and MQA side by side in a single reference sheet for rapid architecture reviews.
↓ 10 · MHA vs GQA vs MQA LEDGER10 ENGINEERING LEDGER
MHA vs GQA vs MQA Side-by-Side
Compare KV heads, group sharing ratios, VRAM compression, Table 1 speedups, checkpoint conversion rules, and open-weight LLM adoption in one table.
| Property | Standard MHA | Grouped-Query GQA | Multi-Query MQA |
|---|
Ready to ship GQA cleanly? Switch between PyTorch, backend-agnostic Keras 3, Ainslie mean-pooling conversion, and SDPA native kernels.
↓ 11 · PYTORCH & KERAS 3 CODE LAB11 CODE LAB · PYTORCH & KERAS 3
Clean PyTorch & Keras 3 Implementations
Five production modules covering zero-copy 5D PyTorch GQA (.train(False)), Keras 3 ops.einsum (training=False), Ainslie mean-pool checkpoint surgery, PyTorch SDPA enable_gqa=True parity checks, and HuggingFace frontier configs.
Test your intuition on G-group partitioning, Table 1 tradeoffs, mean-pool uptraining, and the repeat_interleave bug.
↓ 12 · MASTERY QUIZ12 MASTERY CHECK
GQA Engineering Quiz
Eight scenario questions covering group ratios, Ainslie Table 1 results, checkpoint mean-pooling, tensor parallelism, and silent broadcasting bugs.