CAS · L27
← ELECTRA
ICLR 2021 & 2023 · Microsoft Dynamics & MSR · Lecture 27

DeBERTa — Decoding-enhanced BERT with Disentangled Attention

Instead of mixing content and absolute position once at Layer 0, DeBERTa represents each token with disentangled content H and relative position P across three cross-terms (C2C + C2P + P2C scaled by 1 / sqrt(3d)), injects absolute positions late inside the Enhanced Mask Decoder (EMD), and fuses ELECTRA RTD + GDES in DeBERTa-v3.

C2C+C2P+P2C
3-way disentangled attention
1 / √(3d)
Unit-variance logit scaling
89.9 > 89.8
1.5B + SiFT beats Human SuperGLUE
78GB → 91.1
Large MNLI beats 160GB RoBERTa
89.3 / 87.2
DeBERTa-v3 Base MNLI · SQuAD2
Four Pillars of the DeBERTa Family
02 Why Adding H + P Falls Short

From Entangled Layer-0 Sums to Disentangled Attention

When BERT sums token content and position at input Layer 0, deep layers progressively entangle them. Keeping H and P cleanly separated preserves explicit cross-interactions at every layer.

Positional Encoding Lineage Across Transformers

03 Architecture Blueprint

Interactive DeBERTa + EMD Block Diagram

Click any of the 12 blocks across Pure Content/Relative Inputs, Three-Way Disentangled Attention, Encoder Stack, Enhanced Mask Decoder (late Iabs), and SiFT / v3 GDES — across Base, Large, 1.5B XL, and v3-Base presets.

Model Scale & Version
Stage Focus
04 Three Cross-Interaction Matrices

Decomposing C2C + C2P + P2C & 1/√(3d) Scaling

Inspect each pairwise matrix individually or summed and divided by sqrt(3d) (3 for d=3) on ["new", "[MASK]", "mall"]. Notice how P2C reverses indices δ(j, i)!

05 Clamped Relative Mapping

Computing δ(i, j) vs δ(j, i) Buckets

Slide the query position i and max distance span k to observe how C2P indexes forward offset δ(i, j) while P2C reverses direction δ(j, i).

Query Position i Query i = 3
Max Span k k = 3
06 Live Pure-JS Tensor Verification

6-Decimal Disentangled Attention + EMD Trace

Walk step-by-step through exact live matrix math on ["new", "[MASK]", "mall"] (d = 3, k = 2), verifying C2C/C2P/P2C scores, 1/√9 scaling, encoder FFN, late EMD Iabs injection, and prediction "store" (79.57%, LMLM = 0.228582).

STEP 1 / 6
07 Resolving Syntactic Symmetry

Why Absolute Positions Belong in Enhanced Mask Decoder

Pure relative encoding is translation-invariant and struggles when two masked slots share nearly identical local neighbors. EMD supplies absolute indices only at the 2-layer decoder top.

08 Paper Table 4 Ablations

Proving Every Component Counts: -EMD, -C2P, -P2C

Dropping any single term — EMD, C2P, or swapped P2C — degrades MNLI, SQuAD 1.1/2.0, and RACE compared to Full DeBERTa-base (71.7 RACE, 82.5 SQuAD 2.0 F1).

09 Scale-Invariant Fine-Tuning

SiFT & Surpassing Humans on SuperGLUE (89.9)

Normalizing token embedding vectors before injecting virtual adversarial perturbations prevents norm variance from destabilizing 1.5B fine-tuning — beating T5-11B (89.3) and Human (89.8).

10 Sample-Efficient Scaling

78GB DeBERTa Beats 160GB RoBERTa & XLNet

Trained on half the text corpus of RoBERTa-Large (78GB vs 160GB), DeBERTa-Large pushes MNLI to 91.1, SQuAD v2.0 F1 to 90.7 (+2.3), and RACE to 86.8 (+3.6).

11 ICLR 2023 Evolution

DeBERTa-v3: ELECTRA RTD + Stop-Grad GDES

Combining DeBERTa with ELECTRA's 100% RTD hurts Generator embeddings under naive sharing ("tug-of-war"). GDES parameterizes ED = stop_grad(EG) + Δ, lifting Base MNLI to 89.3 and SQuAD 2.0 to 87.2 F1.

12 Modern Context

The Positional & Discriminative Dynasty

How DeBERTa relates to T5 scalar biases, RoPE decoder rotations, and DeBERTa-v3's modern dominance in NLU & retrieval cross-encoders.

13 Architecture Comparison

RoBERTa vs DeBERTa-v1 vs DeBERTa-v3 Ledger

Side-by-side summary of positional formulation, scaling factor, vocabulary, objective, and downstream yield.

14 Implementation

PyTorch Disentangled Attention, GDES & SiFT Recipes

Inspect how gather builds C2P and transposed P2C in O(n·d + 2k), load HuggingFace microsoft/deberta-v3-base, stop gradients in GDES, and normalize embeddings for SiFT.

15 Self-Check

8-Question DeBERTa Mastery Quiz

Verify your understanding of C2C/C2P/P2C symmetry, 1/√(3d) variance control, Enhanced Mask Decoder, Table 4 ablations, SiFT 1.5B, and GDES stop-gradients.

Current Score
0 / 8