CAS Logo CAS-NLP · L34 Grouped-Query Attention ← Hub
CAS-NLP SERIES · LECTURE 34 · AINSLIE ET AL. (EMNLP 2023) · ARXIV:2305.13245

Grouped-Query Attention (GQA)

Standard MHA gives every chef a massive private fridge (peak quality 47.2, sluggish 1.51 s T5-XXL decode). Extreme MQA forces all 64 queries to cram into one mini-fridge (fast 0.24 s, drops to 46.6). Grouped-Query Attention (GQA) splits queries into G = 8 team prep stations — slashing inference latency 5.4× to 0.28 s while hitting 47.1 quality and powering Llama-2/3, Mistral-7B, Gemma-2, and Qwen2.5.

H_q → G = 8 Default Grouping Powers Llama, Mistral, Qwen
1.51 s → 0.28 s 5.4× T5-XXL Speedup TPUv4 infer (vs 0.24 s MQA)
47.1 vs 47.2 Near-Exact Quality Beats MQA-XXL (46.6 avg)
alpha = 0.05 5% Uptrain Budget Mean-pool existing MHA weights
4× – 8× Cut KV VRAM Savings 1 KV head / GPU on 8-GPU TP
WHAT YOU WILL MASTER IN THIS LECTURE

Six Pillars of Grouped-Query Attention

Multi-Head Attention gives 64 separate fridges. Multi-Query squeezes everyone into 1. What happens when you smoothly sweep G from 64 down to 1?

↓ 02 · THE G = 1..H_q SPECTRUM

02 THE MHA ↔ GQA ↔ MQA SPECTRUM

Sliding G Between Full Quality & Peak Speed

Start with concrete hardware intuition: each GPU decode step must haul every cached Key and Value vector out of slow High-Bandwidth Memory (HBM). Keeping G = H_q copies forces huge memory traffic; shrinking G to 1 (MQA) sacrifices cross-head subspace diversity. Pick a model below and sweep G to watch why G = 8 dominates modern LLM architecture design.

1 · CHOOSE A FRONTIER ARCHITECTURE PRESET
Query Heads (H_q) 64 Query Heads
Sequence Context (S tokens) 2,048 tokens
Key/Value Groups G (all integer divisors of H_q) G = 8 KV Groups (8:1)
MHA Baseline (G = H_q)
128.0 MB
Active GQA Setting (G Groups)
16.0 MB
MQA Floor (G = 1 KV)
2.0 MB
SCALING CURVE (QUALITY vs KV TRAFFIC)

Interactive G = 1..H_q Knee

GEOMETRIC PARTITION STEPPER

Watch Queries Partition Into Groups

How can 4 or 8 distinct Query heads look at the exact same Key/Value shelf inside a group without turning into copies of each other?

↓ 03 · 4-LENS GQA VISUAL LEARNING STUDIO

03 VISUAL LEARNING STUDIO

Four Interactive Lenses on Grouped Sharing

Switch between MHA-vs-GQA team partitioning, live attention weight specialization inside a single group, checkpoint mean-pooling strategies, and multi-layer KV bank footprints across Toy, Mistral-7B, Llama-70B, and Gemma-2 scales.

SELECT MODEL SCALE
Active Group 0 / 1

Now let us step through the exact kitchen analogy, checkpoint mean-pooling proof, and hardware sweet spot in wide, textbook-clear diagrams.

↓ 04 · INTUITION BEFORE EQUATIONS

04 WHY GROUPS WORK

From Kitchen Analogies to Tensor-Parallel GPUs

Three crisp mental models explaining why partitioning H_q queries into G = 8 groups keeps MHA expressivity, converts pretrained weights without full retraining, and fits 8-GPU nodes natively.

Nobody wants to retrain a 70B LLM from scratch just to add GQA. Ainslie et al. discovered a surgical 2-step conversion that takes only 5% compute.

↓ 05 · MEAN-POOLING + 5% UPTRAINING

05 CHECKPOINT SURGERY · AINSLIE ET AL.

MHA → GQA in Two Steps: Mean-Pool + 5% Uptraining

Existing Multi-Head Attention checkpoints already contain rich semantic features in all H_q heads. Instead of discarding heads or randomizing KV projections, mean-pool each group of g adjacent K/V matrices and run 5% continued pretraining.

Primary Source · Section 3.3 & Figure 4: https://arxiv.org/abs/2305.13245 ↗

Let us freeze all abstractions and trace real FP64 numbers by hand across 4 tokens, 4 Query heads, and 2 KV groups.

↓ 06 · HAND-CALIBRATED NUMERICAL TRACE

06 BY HAND ON 4 TOKENS

Exact Numbers: 4 Queries, 2 KV Groups (g = 2)

Sentence: "The chef baked bread" (T = 4, d_model = 8, H_q = 4, G = 2, d_head = 2). Click each decoding step to verify every vector down to 4 decimal places — plus exact 1.11e-16 parity against repeat_interleave and the 1.033 error when .repeat() is misused!

Now zoom into an entire Transformer decoder attention block to watch Residual Bypass, G-Group KV Vaults, and zero-copy 5D pairing work together.

↓ 07 · SCHEMATIC GQA DECODER PIPELINE

07 SINGLE DECODER LAYER PIPELINE

Schematic GQA Attention Block & Tensor Shapes

Toggle between Mistral-7B (4:1), Llama-70B (8:1), and Gemma-2 9B (2:1), isolate individual pipeline stages, and click any module on the diagram to inspect its exact tensor formula and hardware role.

SELECT PRODUCTION MODEL ARCHITECTURE
HIGHLIGHT PIPELINE STAGE

How many gigabytes of HBM VRAM does switching from MHA to GQA save on an 80-layer 70B model serving 8 concurrent requests?

↓ 08 · 3-WAY PRODUCTION VRAM CALCULATOR

08 PRODUCTION VRAM CALCULATOR

3-Way KV Memory: MHA vs GQA vs MQA

KV cache VRAM scales linearly with layers L, active KV heads G, head width d_head, sequence length S, batch size B, and 2 bytes per FP16 scalar (times 2 for Key + Value). Test real frontier configs below.

1 · FRONTIER MODEL PROFILES
2 · Select KV Group Count G G=8 KV (8:1)
FP16 KV Cache Formula: KV_bytes = 2 (K+V) * L * G * d_head * S * B * 2 bytes (FP16)
MHA (G = H_q)100% Baseline
GQA (Active G Groups)g× Smaller
MQA Floor (G = 1)Maximum Cut
Layers (L)80 layers
Query Heads (H_q)64 Q heads
Head Dimension (d_head)128 d_head
Sequence Length (S)4,096 tokens
Concurrent Batch Size (B)8 batch

Now verify the exact published Table 1 benchmarks across summarization, translation, and QA — plus the 4 bugs engineers hit most in production.

↓ 09 · AINSLIE TABLE 1 & BUG PITFALLS

09 BENCHMARKS & BUG PITFALLS

Table 1 Verification & 4 GQA Code Traps

Exact TPUv4 latency and benchmark scores reported by Ainslie et al. (EMNLP 2023 Table 1), alongside an interactive bug debugger highlighting why .repeat() quietly scrambles attention heads.

Verified Paper Table 1: https://arxiv.org/abs/2305.13245 ↗
Model TPUv4 T_infer Avg CNN arXiv PubMed MediaSum MultiNews WMT TriviaQA
INTERACTIVE GQA BUG DEBUGGER · CLICK A PITFALL TO INSPECT

Put MHA, GQA, and MQA side by side in a single reference sheet for rapid architecture reviews.

↓ 10 · MHA vs GQA vs MQA LEDGER

10 ENGINEERING LEDGER

MHA vs GQA vs MQA Side-by-Side

Compare KV heads, group sharing ratios, VRAM compression, Table 1 speedups, checkpoint conversion rules, and open-weight LLM adoption in one table.

Property Standard MHA Grouped-Query GQA Multi-Query MQA

Ready to ship GQA cleanly? Switch between PyTorch, backend-agnostic Keras 3, Ainslie mean-pooling conversion, and SDPA native kernels.

↓ 11 · PYTORCH & KERAS 3 CODE LAB

11 CODE LAB · PYTORCH & KERAS 3

Clean PyTorch & Keras 3 Implementations

Five production modules covering zero-copy 5D PyTorch GQA (.train(False)), Keras 3 ops.einsum (training=False), Ainslie mean-pool checkpoint surgery, PyTorch SDPA enable_gqa=True parity checks, and HuggingFace frontier configs.

Test your intuition on G-group partitioning, Table 1 tradeoffs, mean-pool uptraining, and the repeat_interleave bug.

↓ 12 · MASTERY QUIZ

12 MASTERY CHECK

GQA Engineering Quiz

Eight scenario questions covering group ratios, Ainslie Table 1 results, checkpoint mean-pooling, tensor parallelism, and silent broadcasting bugs.

CURRENT SCORE
0 / 8