RoPE — encoding position by rotating space
Adding a position vector x_m + p_m pollutes token semantics and corrupts vector norms. Rotary Positional Embedding (RoPE, Su et al. 2021) takes a radically cleaner path: it slices every Query and Key head into 2D planes and rotates token m by angle mθ_i. Because the dot product of two rotated vectors depends only on the angle between them, ⟨R_m q, R_n k⟩ = qᵀ R_{n−m} k — turning absolute rotations into exact relative attention.
Six Core Insights that power every modern LLM
Filter by mathematical or systems domain, or click any card to jump directly to its interactive simulator below.
THE HISTORICAL ARC (2017–2026) — CLICK ANY MILESTONE TO INSPECT ITS POSITIONAL BREAKTHROUGH
Why do Transformers need positional encoding at all — and why did adding sin/cos vectors to embeddings fall short?
02 THE PERMUTATION EQUIVARIANCE TRAP & ADDITIVE POLLUTION
Why pure self-attention is order-blind — and why adding position pollutes meaning
Unlike RNNs that process words sequentially, a Transformer’s self-attention layer computes softmax(QKᵀ/√d)V across all tokens in parallel. Without positional information, self-attention is permutation-equivariant: swapping words in the sentence simply swaps the output rows, producing identical pairwise scores for "The dog bit man" and "The man bit dog". Toggle the three regimes below to see why RoPE’s multiplicative rotation cleanly beats additive encodings:
How can multiplying each token separately by its own absolute position m produce a dot product that depends only on relative distance m − n?
03 FIRST-PRINCIPLES DERIVATION · SU ET AL. (2021) FUNCTIONAL EQUATION
The 2D Complex-Plane Proof: why absolute rotations cancel into relative distance
Su et al. (2021) posed a fundamental question: can we find a transformation q̃_m = f(q, m) and k̃_n = f(k, n) applied independently to each token such that their inner product depends only on the vectors q, k and their relative offset m − n? In the 2D complex plane ℂ, multiplying by Euler’s unit phasor e^{imθ} = cos(mθ) + i·sin(mθ) rotates a vector by angle mθ. When we take the inner product Re[q̃_m · conj(k̃_n)], the conjugate flips the sign of the key’s angle to −nθ, yielding e^{i(m−n)θ}! Shift both tokens forward below to see the cancellation live:
1. SELECT BASE TOKEN PAIR (m₀, n₀)
2. SHIFT BOTH TOKENS FORWARD BY +Δ POSITIONS (WATCH GLOBAL ANGLES SPIN WHILE DOT PRODUCT STAYS CONSTANT!)
A single attention head has d_head = 64 or 128 dimensions, not 2. How do we generalize 2D rotation to d dimensions without an O(d²) matrix multiply?
04 FROM 2D TO d_head · BLOCK-DIAGONAL SPARSITY & EQ. (13)
The Block-Diagonal Matrix & the fast O(d) rotate_half trick
To generalize 2D rotation to an even head dimension d, RoPE partitions the vector into d/2 independent 2D subspaces, each rotated by its own frequency θ_i via a block-diagonal matrix R_{Θ,m}^d. Because 1 − 2/d of that matrix is zero (98.4% zeros at d = 128!), multiplying by the full matrix wastes O(d²) FLOPs. Instead, Su et al. Eq. (13) rewrites the product as two elementwise vector operations in O(d) time:
8×8 BLOCK-DIAGONAL ROTATION MATRIX R_{Θ,m}^8 (FOUR 2×2 ORTHOGONAL BLOCKS ON DIAGONAL · 75% ZEROS)
EQUIVALENT O(d) FAST ELEMENTWISE KERNEL: R_{Θ,m} x = x ⊙ cos(mΘ) + rotate_half(x) ⊙ sin(mΘ)
Let’s verify every single decimal by hand on a concrete 4-dimensional attention head across four tokens.
05 HAND-COMPUTED 4D TRACE · SIX-DECIMAL PARITY
Four tokens, two 2D rotation planes, every number on screen
Trace ["The", "cat", "sat", "down"] at positions m = 0, 1, 2, 3 with d_head = 4: Subspace 0 (dims 0,1) spins fast at θ₀ = 1.00 rad/tok, while Subspace 1 (dims 2,3) spins 100× slower at θ₁ = 0.01 rad/tok. Step through each token to verify both norm preservation and exact relative inner-product parity:
Why do we assign a geometric progression of frequencies θ_i = base^{−2i/d} across the d/2 planes — and how does it create automatic distance decay?
06 GEOMETRIC FREQUENCY SCHEDULE · WAVELENGTHS & ABEL DECAY
The Multi-Scale Clock Bank: from 6.28-token seconds hands to 62,832-token century hands
If you only had a single rotation frequency θ, tokens separated by Δ = 2π/θ would wrap 360° and collide with distance 0! RoPE solves this by giving each of the d/2 pairs a geometric frequency θ_i = base^{−2i/d} (wavelength λ_i = 2π · base^{2i/d}). Even better: by Abel summation, summing cos(Δ·θ_i) across all 64 frequencies creates a smooth long-term relative distance decay without any hard window cutoff:
1. SELECT ROPE BASE FREQUENCY θ_base (ROFORMER 10K → LLAMA-3 500K → QWEN 1M)
When converting weights between Meta’s official LLaMA repo and HuggingFace transformers, why do unpermuted RoPE weights produce complete gibberish?
07 SYSTEMS IMPLEMENTATION TRAP · META LLAMA VS HUGGINGFACE GPT-NEOX
Contiguous Pairs vs Split-Half Pairing: the #1 checkpoint conversion bug
There are two ways to pair d features into d/2 2D planes: (1) Contiguous / Interleaved (x_{2i}, x_{2i+1}) used in Su et al. (2021) and Meta’s official LLaMA codebase, and (2) Split-Half (x_i, x_{i + d/2}) used in GPT-NeoX and HuggingFace transformers for coalesced GPU memory reads. Toggle between both conventions below to see how the wires cross — and why converting checkpoints requires permuting the rows of W_Q and W_K:
Since sin(mθ) and cos(mθ) are defined for any integer m, why does a 4K-trained RoPE model collapse into gibberish at token 4,200?
08 THE EXTRAPOLATION WALL · CHEN ET AL. (2023) POSITION INTERPOLATION
Why RoPE breaks past L_train — and the Position Interpolation tradeoff
During 4,096-token pretraining, high-frequency clocks spin hundreds of full 360° circles, so the model knows every angle on the circle. Slowly rotating clocks, however, only sweep a narrow arc [0, 4096·θ_slow] (e.g., 0° to 93.9°). At token 8,192, those slow vectors enter the negative-cosine quadrant for the first time in their lives — blowing up attention logits! Position Interpolation (PI, Chen et al. 2023) rescales m′ = m / s to pull slow angles back inside the training sector, at the cost of crowding local tokens:
How do we keep high frequencies sharp while interpolating only the slow out-of-distribution frequencies? Enter NTK-Aware Scaling and YaRN.
09 LONG-CONTEXT BREAKTHROUGH · NTK-BY-PARTS, YaRN & LLAMA-3.1 SCALING
Never stretch high frequencies: NTK-Aware Scaling, YaRN & Softmax Temperature
Neural Tangent Kernel (NTK) theory shows that neural networks struggle to learn high-frequency information when inputs are compressed. YaRN (Peng et al. 2023) and LLaMA-3.1 classify every RoPE dimension pair i by its wavelength ratio r(i) = L_train / λ_i: high-frequency local clocks (r > 32) are kept 100% unscaled (γ = 1), slow global clocks (r < 1) are interpolated by 1/s (γ = 0), and middle clocks follow a smooth ramp — combined with attention logit temperature scaling √(1/t) = 0.1·ln(s) + 1:
How does RoPE interact with the GPU KV Cache — and why did DeepSeek-V2/V3 have to invent "Decoupled RoPE" for Multi-head Latent Attention?
10 SYSTEMS CO-DESIGN · KV CACHE REROTATION & DEEPSEEK DECOUPLED RoPE
RoPE in the KV Cache & DeepSeek MLA’s 64-dim Decoupled RoPE trick
In standard GQA/MHA models, Keys are cached after applying R_n because token n’s position never changes during append-only decoding. However, when DeepSeek-V2/V3 compressed all KV heads into a joint 512-dim latent vector c_t^{KV}, applying position-dependent R_t blocked absorbing the decompression matrix W^{UK} into W^Q! Toggle below to see how DeepSeek solved this by splitting content (512-dim NoPE latent) from position (64-dim shared RoPE key):
Text is 1D, but images are 2D (row, column) and video is 3D (time, row, column). How does RoPE rotate across multiple spatial dimensions?
11 MULTIMODAL GENERALIZATION · 2D AXIAL RoPE & QWEN2-VL 3D M-RoPE
Rotating across space and time: 2D Axial Vision RoPE & 3D M-RoPE
Flattening an image into a 1D sequence destroys vertical adjacency: patch (r, c) and patch (r+1, c) are 1 pixel apart vertically but W tokens apart in 1D! 2D Axial RoPE (used in ViT, Pixtral, and LLaMA-3.2 Vision) splits the head’s d/2 rotation planes in half: the first d/4 planes rotate by row coordinate y, and the remaining d/4 planes rotate by column coordinate x. 3D M-RoPE (Qwen2-VL) splits planes into (temporal t, height h, width w). Click any patches on the 8 × 8 grid below:
INTERACTIVE 8×8 IMAGE PATCH GRID — CLICK ANY CELL TO MOVE Q OR K
Can we push context windows to 10,000,000 tokens without ever worrying about slow RoPE wavelengths wrapping around? Enter LLaMA-4 iRoPE.
12 THE 2025–2026 FRONTIER · LLAMA-4 iRoPE & QK-NORM STABILITY
Beyond pure RoPE: Interleaved RoPE/NoPE (iRoPE) & QK-RMSNorm
At 10 million tokens (LLaMA-4 Scout), even θ_base = 500,000 suffers from slow-angle phase drift. Meta’s iRoPE (Interleaved RoPE) architecture solves this by alternating 3 local RoPE layers (which handle sharp syntax within chunked windows) with 1 global NoPE layer (No Positional Embedding at all, relying on causal masking + inference-time temperature scaling for infinite-length global retrieval):
Every positional encoding architecture from 2017 to 2026 compared side by side on one engineering ledger.
13 THE ENGINEERING SCOREBOARD
Master comparison ledger — Sinusoidal, Learned, ALiBi, RoPE, YaRN & iRoPE
Compare how every major positional encoding scheme handles relative distance, length extrapolation, and fused FlashAttention / KV Cache compatibility:
| Scheme & Paper | Mathematical Mechanism | Relative Property | Length Extrapolation | FlashAttn / KV Cache & Models |
|---|
Every mathematical identity above implemented in clean, runnable PyTorch code.
14 CODE LAB · META COMPLEX RoPE, HUGGINGFACE SPLIT-HALF, YaRN & 2D RoPE
RoPE in production PyTorch
Four complete, self-contained implementations: (1) Meta LLaMA Complex-Plane RoPE using torch.polar and torch.view_as_complex; (2) HuggingFace Fast O(d) Split-Half rotate_half plus the exact weight permutation between Meta and HF; (3) YaRN & LLaMA-3.1 Wavelength Ramp; and (4) 2D Axial Vision RoPE.
15 MASTERY CHECK · 8 QUESTIONS
Test your mastery of Rotary Positional Embedding
Eight questions spanning SO(2) relative rotation proofs, O(d) rotate_half complexity, multi-scale wavelengths, the Meta-vs-HuggingFace weight permutation, YaRN wavelength bands, DeepSeek MLA decoupled RoPE, and 2D/3D M-RoPE.
Click any option to check your answer and reveal the mathematical derivation.