DeBERTa — Decoding-enhanced BERT with Disentangled Attention
Instead of mixing content and absolute position once at Layer 0, DeBERTa represents each token with disentangled content H and relative position P across three cross-terms (C2C + C2P + P2C scaled by 1 / sqrt(3d)), injects absolute positions late inside the Enhanced Mask Decoder (EMD), and fuses ELECTRA RTD + GDES in DeBERTa-v3.
From Entangled Layer-0 Sums to Disentangled Attention
When BERT sums token content and position at input Layer 0, deep layers progressively entangle them. Keeping H and P cleanly separated preserves explicit cross-interactions at every layer.
Positional Encoding Lineage Across Transformers
Interactive DeBERTa + EMD Block Diagram
Click any of the 12 blocks across Pure Content/Relative Inputs, Three-Way Disentangled Attention, Encoder Stack, Enhanced Mask Decoder (late Iabs), and SiFT / v3 GDES — across Base, Large, 1.5B XL, and v3-Base presets.
Decomposing C2C + C2P + P2C & 1/√(3d) Scaling
Inspect each pairwise matrix individually or summed and divided by sqrt(3d) (3 for d=3) on ["new", "[MASK]", "mall"]. Notice how P2C reverses indices δ(j, i)!
Computing δ(i, j) vs δ(j, i) Buckets
Slide the query position i and max distance span k to observe how C2P indexes forward offset δ(i, j) while P2C reverses direction δ(j, i).
6-Decimal Disentangled Attention + EMD Trace
Walk step-by-step through exact live matrix math on ["new", "[MASK]", "mall"] (d = 3, k = 2), verifying C2C/C2P/P2C scores, 1/√9 scaling, encoder FFN, late EMD Iabs injection, and prediction "store" (79.57%, LMLM = 0.228582).
Why Absolute Positions Belong in Enhanced Mask Decoder
Pure relative encoding is translation-invariant and struggles when two masked slots share nearly identical local neighbors. EMD supplies absolute indices only at the 2-layer decoder top.
Proving Every Component Counts: -EMD, -C2P, -P2C
Dropping any single term — EMD, C2P, or swapped P2C — degrades MNLI, SQuAD 1.1/2.0, and RACE compared to Full DeBERTa-base (71.7 RACE, 82.5 SQuAD 2.0 F1).
SiFT & Surpassing Humans on SuperGLUE (89.9)
Normalizing token embedding vectors before injecting virtual adversarial perturbations prevents norm variance from destabilizing 1.5B fine-tuning — beating T5-11B (89.3) and Human (89.8).
78GB DeBERTa Beats 160GB RoBERTa & XLNet
Trained on half the text corpus of RoBERTa-Large (78GB vs 160GB), DeBERTa-Large pushes MNLI to 91.1, SQuAD v2.0 F1 to 90.7 (+2.3), and RACE to 86.8 (+3.6).
DeBERTa-v3: ELECTRA RTD + Stop-Grad GDES
Combining DeBERTa with ELECTRA's 100% RTD hurts Generator embeddings under naive sharing ("tug-of-war"). GDES parameterizes ED = stop_grad(EG) + Δ, lifting Base MNLI to 89.3 and SQuAD 2.0 to 87.2 F1.
The Positional & Discriminative Dynasty
How DeBERTa relates to T5 scalar biases, RoPE decoder rotations, and DeBERTa-v3's modern dominance in NLU & retrieval cross-encoders.
RoBERTa vs DeBERTa-v1 vs DeBERTa-v3 Ledger
Side-by-side summary of positional formulation, scaling factor, vocabulary, objective, and downstream yield.
PyTorch Disentangled Attention, GDES & SiFT Recipes
Inspect how gather builds C2P and transposed P2C in O(n·d + 2k), load HuggingFace microsoft/deberta-v3-base, stop gradients in GDES, and normalize embeddings for SiFT.
8-Question DeBERTa Mastery Quiz
Verify your understanding of C2C/C2P/P2C symmetry, 1/√(3d) variance control, Enhanced Mask Decoder, Table 4 ablations, SiFT 1.5B, and GDES stop-gradients.