# LECTURE 22 · ROTARY POSITIONAL EMBEDDING (RoPE) · Su et al. (RoFormer 2021) → LLaMA 1–3 → YaRN (2023) → DeepSeek MLA & Qwen2-VL M-RoPE → LLaMA-4 iRoPE (2025–2026)

RoPE — encoding position by rotating space

Adding a position vector x_m + p_m pollutes token semantics and corrupts vector norms. Rotary Positional Embedding (RoPE, Su et al. 2021) takes a radically cleaner path: it slices every Query and Key head into 2D planes and rotates token m by angle mθ_i. Because the dot product of two rotated vectors depends only on the angle between them, ⟨R_m q, R_n k⟩ = qᵀ R_{n−m} k — turning absolute rotations into exact relative attention.

● LIVE 2D COMPLEX-PLANE ROTATION SIMULATOR · SHIFT INVARIANCE PROVER
Query m = 3 · Key n = 1 · Relative Distance m − n = +2
PROJECT q_m, k_n → SLICE INTO d/2 2D PAIRS → ROTATE BY m·θ_i & n·θ_i IN O(d) → DOT PRODUCT ⟨R_m q, R_n k⟩ = qᵀ R_{n-m} k → VALUES V UNTOUCHED
qᵀ R_{n−m} k Exact Relative Inner Product Absolute rotation → relative score
‖R_m q‖ = ‖q‖ Norm-Preserving Orthogonal det(R_m) = +1 · zero V pollution
O(d) Fast Kernel x⊙cos + rotate_half(x)⊙sin 64× fewer ops than dense d×d
4K → 10M Context Scaling Horizon Via YaRN, Decoupled RoPE & iRoPE
LEARNING OBJECTIVES 1 Why pure self-attention is order-blind 2 2D Complex Euler derivation 3 Block-diagonal O(d) rotate_half 4 Wavelength clocks & θ_base 5 Meta vs HuggingFace pairing trap 6 YaRN, MLA Decoupled RoPE & M-RoPE
★ EXECUTIVE ENGINEERING BRIEFING · THE 6 PILLARS OF RoPE

Six Core Insights that power every modern LLM

Filter by mathematical or systems domain, or click any card to jump directly to its interactive simulator below.

THE ONE-EQUATION INTUITION

In 2D, rotating vector q by angle mθ and vector k by angle nθ changes both vectors in global coordinates, but the angle between them becomes mθ − nθ = (m − n)θ. Since the dot product ⟨u, v⟩ = ‖u‖‖v‖cos(Δangle) depends only on lengths and the angle between the vectors, rotating each token by its own absolute position index m automatically makes every attention score a pure function of relative distance m − n — without ever materializing an N × N relative position matrix!

THE HISTORICAL ARC (2017–2026) — CLICK ANY MILESTONE TO INSPECT ITS POSITIONAL BREAKTHROUGH

Why do Transformers need positional encoding at all — and why did adding sin/cos vectors to embeddings fall short?

02 THE PERMUTATION EQUIVARIANCE TRAP & ADDITIVE POLLUTION

Why pure self-attention is order-blind — and why adding position pollutes meaning

Unlike RNNs that process words sequentially, a Transformer’s self-attention layer computes softmax(QKᵀ/√d)V across all tokens in parallel. Without positional information, self-attention is permutation-equivariant: swapping words in the sentence simply swaps the output rows, producing identical pairwise scores for "The dog bit man" and "The man bit dog". Toggle the three regimes below to see why RoPE’s multiplicative rotation cleanly beats additive encodings:

How can multiplying each token separately by its own absolute position m produce a dot product that depends only on relative distance m − n?

03 FIRST-PRINCIPLES DERIVATION · SU ET AL. (2021) FUNCTIONAL EQUATION

The 2D Complex-Plane Proof: why absolute rotations cancel into relative distance

Su et al. (2021) posed a fundamental question: can we find a transformation q̃_m = f(q, m) and k̃_n = f(k, n) applied independently to each token such that their inner product depends only on the vectors q, k and their relative offset m − n? In the 2D complex plane ℂ, multiplying by Euler’s unit phasor e^{imθ} = cos(mθ) + i·sin(mθ) rotates a vector by angle mθ. When we take the inner product Re[q̃_m · conj(k̃_n)], the conjugate flips the sign of the key’s angle to −nθ, yielding e^{i(m−n)θ}! Shift both tokens forward below to see the cancellation live:

1. SELECT BASE TOKEN PAIR (m₀, n₀)

2. SHIFT BOTH TOKENS FORWARD BY +Δ POSITIONS (WATCH GLOBAL ANGLES SPIN WHILE DOT PRODUCT STAYS CONSTANT!)

A single attention head has d_head = 64 or 128 dimensions, not 2. How do we generalize 2D rotation to d dimensions without an O(d²) matrix multiply?

04 FROM 2D TO d_head · BLOCK-DIAGONAL SPARSITY & EQ. (13)

The Block-Diagonal Matrix & the fast O(d) rotate_half trick

To generalize 2D rotation to an even head dimension d, RoPE partitions the vector into d/2 independent 2D subspaces, each rotated by its own frequency θ_i via a block-diagonal matrix R_{Θ,m}^d. Because 1 − 2/d of that matrix is zero (98.4% zeros at d = 128!), multiplying by the full matrix wastes O(d²) FLOPs. Instead, Su et al. Eq. (13) rewrites the product as two elementwise vector operations in O(d) time:

TOKEN POSITION m (INSPECT BLOCK-DIAGONAL MATRIX R_{Θ,m}^8) Position m = 2

8×8 BLOCK-DIAGONAL ROTATION MATRIX R_{Θ,m}^8 (FOUR 2×2 ORTHOGONAL BLOCKS ON DIAGONAL · 75% ZEROS)

EQUIVALENT O(d) FAST ELEMENTWISE KERNEL: R_{Θ,m} x = x ⊙ cos(mΘ) + rotate_half(x) ⊙ sin(mΘ)

Let’s verify every single decimal by hand on a concrete 4-dimensional attention head across four tokens.

05 HAND-COMPUTED 4D TRACE · SIX-DECIMAL PARITY

Four tokens, two 2D rotation planes, every number on screen

Trace ["The", "cat", "sat", "down"] at positions m = 0, 1, 2, 3 with d_head = 4: Subspace 0 (dims 0,1) spins fast at θ₀ = 1.00 rad/tok, while Subspace 1 (dims 2,3) spins 100× slower at θ₁ = 0.01 rad/tok. Step through each token to verify both norm preservation and exact relative inner-product parity:

Why do we assign a geometric progression of frequencies θ_i = base^{−2i/d} across the d/2 planes — and how does it create automatic distance decay?

06 GEOMETRIC FREQUENCY SCHEDULE · WAVELENGTHS & ABEL DECAY

The Multi-Scale Clock Bank: from 6.28-token seconds hands to 62,832-token century hands

If you only had a single rotation frequency θ, tokens separated by Δ = 2π/θ would wrap 360° and collide with distance 0! RoPE solves this by giving each of the d/2 pairs a geometric frequency θ_i = base^{−2i/d} (wavelength λ_i = 2π · base^{2i/d}). Even better: by Abel summation, summing cos(Δ·θ_i) across all 64 frequencies creates a smooth long-term relative distance decay without any hard window cutoff:

1. SELECT ROPE BASE FREQUENCY θ_base (ROFORMER 10K → LLAMA-3 500K → QWEN 1M)

2. DRAG RELATIVE TOKEN DISTANCE Δ = |m − n| (WATCH THE 4 CLOCK HANDS & DECAY CURVE) Relative Distance Δ = 12 tokens

When converting weights between Meta’s official LLaMA repo and HuggingFace transformers, why do unpermuted RoPE weights produce complete gibberish?

07 SYSTEMS IMPLEMENTATION TRAP · META LLAMA VS HUGGINGFACE GPT-NEOX

Contiguous Pairs vs Split-Half Pairing: the #1 checkpoint conversion bug

There are two ways to pair d features into d/2 2D planes: (1) Contiguous / Interleaved (x_{2i}, x_{2i+1}) used in Su et al. (2021) and Meta’s official LLaMA codebase, and (2) Split-Half (x_i, x_{i + d/2}) used in GPT-NeoX and HuggingFace transformers for coalesced GPU memory reads. Toggle between both conventions below to see how the wires cross — and why converting checkpoints requires permuting the rows of W_Q and W_K:

Since sin(mθ) and cos(mθ) are defined for any integer m, why does a 4K-trained RoPE model collapse into gibberish at token 4,200?

08 THE EXTRAPOLATION WALL · CHEN ET AL. (2023) POSITION INTERPOLATION

Why RoPE breaks past L_train — and the Position Interpolation tradeoff

During 4,096-token pretraining, high-frequency clocks spin hundreds of full 360° circles, so the model knows every angle on the circle. Slowly rotating clocks, however, only sweep a narrow arc [0, 4096·θ_slow] (e.g., 0° to 93.9°). At token 8,192, those slow vectors enter the negative-cosine quadrant for the first time in their lives — blowing up attention logits! Position Interpolation (PI, Chen et al. 2023) rescales m′ = m / s to pull slow angles back inside the training sector, at the cost of crowding local tokens:

DRAG TOKEN POSITION m (PRETRAINED MAX L_train = 4,096) Token Position m = 8,192 (L_train = 4,096)

How do we keep high frequencies sharp while interpolating only the slow out-of-distribution frequencies? Enter NTK-Aware Scaling and YaRN.

09 LONG-CONTEXT BREAKTHROUGH · NTK-BY-PARTS, YaRN & LLAMA-3.1 SCALING

Never stretch high frequencies: NTK-Aware Scaling, YaRN & Softmax Temperature

Neural Tangent Kernel (NTK) theory shows that neural networks struggle to learn high-frequency information when inputs are compressed. YaRN (Peng et al. 2023) and LLaMA-3.1 classify every RoPE dimension pair i by its wavelength ratio r(i) = L_train / λ_i: high-frequency local clocks (r > 32) are kept 100% unscaled (γ = 1), slow global clocks (r < 1) are interpolated by 1/s (γ = 0), and middle clocks follow a smooth ramp — combined with attention logit temperature scaling √(1/t) = 0.1·ln(s) + 1:

CONTEXT EXTENSION SCALE FACTOR s = L_target / L_train Scale Factor s = 16× (4K → 64K Context)

How does RoPE interact with the GPU KV Cache — and why did DeepSeek-V2/V3 have to invent "Decoupled RoPE" for Multi-head Latent Attention?

10 SYSTEMS CO-DESIGN · KV CACHE REROTATION & DEEPSEEK DECOUPLED RoPE

RoPE in the KV Cache & DeepSeek MLA’s 64-dim Decoupled RoPE trick

In standard GQA/MHA models, Keys are cached after applying R_n because token n’s position never changes during append-only decoding. However, when DeepSeek-V2/V3 compressed all KV heads into a joint 512-dim latent vector c_t^{KV}, applying position-dependent R_t blocked absorbing the decompression matrix W^{UK} into W^Q! Toggle below to see how DeepSeek solved this by splitting content (512-dim NoPE latent) from position (64-dim shared RoPE key):

Text is 1D, but images are 2D (row, column) and video is 3D (time, row, column). How does RoPE rotate across multiple spatial dimensions?

11 MULTIMODAL GENERALIZATION · 2D AXIAL RoPE & QWEN2-VL 3D M-RoPE

Rotating across space and time: 2D Axial Vision RoPE & 3D M-RoPE

Flattening an image into a 1D sequence destroys vertical adjacency: patch (r, c) and patch (r+1, c) are 1 pixel apart vertically but W tokens apart in 1D! 2D Axial RoPE (used in ViT, Pixtral, and LLaMA-3.2 Vision) splits the head’s d/2 rotation planes in half: the first d/4 planes rotate by row coordinate y, and the remaining d/4 planes rotate by column coordinate x. 3D M-RoPE (Qwen2-VL) splits planes into (temporal t, height h, width w). Click any patches on the 8 × 8 grid below:

INTERACTIVE 8×8 IMAGE PATCH GRID — CLICK ANY CELL TO MOVE Q OR K

Can we push context windows to 10,000,000 tokens without ever worrying about slow RoPE wavelengths wrapping around? Enter LLaMA-4 iRoPE.

12 THE 2025–2026 FRONTIER · LLAMA-4 iRoPE & QK-NORM STABILITY

Beyond pure RoPE: Interleaved RoPE/NoPE (iRoPE) & QK-RMSNorm

At 10 million tokens (LLaMA-4 Scout), even θ_base = 500,000 suffers from slow-angle phase drift. Meta’s iRoPE (Interleaved RoPE) architecture solves this by alternating 3 local RoPE layers (which handle sharp syntax within chunked windows) with 1 global NoPE layer (No Positional Embedding at all, relying on causal masking + inference-time temperature scaling for infinite-length global retrieval):

Every positional encoding architecture from 2017 to 2026 compared side by side on one engineering ledger.

13 THE ENGINEERING SCOREBOARD

Master comparison ledger — Sinusoidal, Learned, ALiBi, RoPE, YaRN & iRoPE

Compare how every major positional encoding scheme handles relative distance, length extrapolation, and fused FlashAttention / KV Cache compatibility:

Scheme & Paper Mathematical Mechanism Relative Property Length Extrapolation FlashAttn / KV Cache & Models

Every mathematical identity above implemented in clean, runnable PyTorch code.

14 CODE LAB · META COMPLEX RoPE, HUGGINGFACE SPLIT-HALF, YaRN & 2D RoPE

RoPE in production PyTorch

Four complete, self-contained implementations: (1) Meta LLaMA Complex-Plane RoPE using torch.polar and torch.view_as_complex; (2) HuggingFace Fast O(d) Split-Half rotate_half plus the exact weight permutation between Meta and HF; (3) YaRN & LLaMA-3.1 Wavelength Ramp; and (4) 2D Axial Vision RoPE.

ROTARY POSITIONAL EMBEDDING · PYTORCH REFERENCE IMPLEMENTATIONS

15 MASTERY CHECK · 8 QUESTIONS

Test your mastery of Rotary Positional Embedding

Eight questions spanning SO(2) relative rotation proofs, O(d) rotate_half complexity, multi-scale wavelengths, the Meta-vs-HuggingFace weight permutation, YaRN wavelength bands, DeepSeek MLA decoupled RoPE, and 2D/3D M-RoPE.

Lecture 22 Mastery Quiz

Click any option to check your answer and reveal the mathematical derivation.

SCORE: 0 / 8