CAS · L31
← MODERNBERT
Le Breton et al. (Feb 2025, arXiv:2502.19587) & Boizard et al. (Mar 2025, arXiv:2503.05500) · Lecture 31

NeoBERT & EuroBERT — Deep-Narrow 250M, Multilingual 2.1B GQA & Modern Encoder Scaling

Following ModernBERT, two landmark 2025 encoder families complete the modernization of bidirectional transformers using Pre-RMSNorm, SwiGLU, FlashAttention, and high-base RoPE. NeoBERT (250M) proves an optimal 28L × 768 deep-and-narrow shape reaches 89.0 GLUE and 250M MTEB SOTA as a plug-and-play WordPiece drop-in up to 4,096 tokens, while EuroBERT (210M / 610M / 2.1B) pairs 3:1 Grouped-Query Attention (18Q:6KV), RoPE theta = 250,000 at 8,192 context, LLaMA-3 128K BPE, and 5T multilingual + code + math tokens.

28L × 768
NeoBERT 250M Optimal Ratio
18Q : 6KV
EuroBERT 610M / 2.1B GQA
4K / 8K
RoPE Context (10K vs 250K Base)
2.1T / 5T
RefinedWeb & Multilingual+Code
89.0 GLUE
NeoBERT Beats RoBERTa-Large

Interactive Architecture Spotlight: NeoBERT vs EuroBERT

02

Why Deep-and-Narrow (28L × 768) Beats Wide-Shallow Encoders

Le Breton et al. swept depth-to-width allocations at a fixed 250M budget and discovered that stacking 28 narrower layers outperforms wider 12L or 16L encoders on semantic GLUE and MTEB tasks while preserving full BERT-Base 768-dim checkpoint compatibility.

03

Interactive 12-Block Architecture Diagram (NeoBERT 250M & EuroBERT 210M–2.1B)

Select any scale (NeoBERT 250M, EuroBERT 210M, 610M, or 2.1B), filter by stage, and click any block inside the SVG pipeline to examine shapes, equations, and GQA/RoPE parameters.

04

Grouped-Query Attention (18Q : 6KV) in EuroBERT-610M & 2.1B

Click any KV Group below to inspect how EuroBERT ties 3 distinct Query heads onto each shared Key-Value head — cutting KV weights and memory bandwidth by 3x compared to standard Multi-Head Attention.

05

Pre-RMSNorm & Bias-Free SwiGLU Activation Lab

Both 2025 encoders adopt the LLaMA-style Pre-RMSNorm + SwiGLU pairing: normalizing purely by root-mean-square energy prior to each sub-layer and gating MLP channels via SiLU(x * W_gate) * (x * W_up).

06

Exact 6-Decimal Pure-JS Forward Trace (Token 1: "code" at pos=13)

Walk through every exact numerical step verified against Python FP64: Pre-RMSNorm, Dual-RoPE frequency divergence (10K vs 250K), 2:1 GQA shared Key-Value attention, SwiGLU gating, and MLM cross-entropy loss (L_MLM = 0.144502).

STEP 1 / 6
07

RoPE 4K / 8K Extension & Annealed Masking Rate Tradeoffs

Compare masking-rate regimes reported by NeoBERT (fixed 20%) and EuroBERT (15%–50% annealing ablations): higher masking biases encoders toward holistic retrieval (BEIR/MIRACL), whereas lower masking preserves sharp local NLI discrimination.

08

Multi-Trillion-Token Pretraining Curricula (2.1T & 5.0T)

Inspect how NeoBERT’s 2.1T RefinedWeb two-stage schedule and EuroBERT’s 4.8T + 200B annealing curriculum expand bidirectional capacity by over 600x relative to BERT’s original 3.3B words.

09

Complete Architecture & Scale Comparison Matrix

Full side-by-side comparison across 2018–2019 legacy encoders, ModernBERT, NeoBERT, and all three EuroBERT parameter tiers.

10

GLUE, MTEB Contrastive Embeddings & Multilingual Retrieval

Switch between NeoBERT’s English GLUE & MTEB SOTA leaderboards and EuroBERT’s multilingual + retrieval scaling across XLM-R baselines.

11

15+ Multilingual Coverage, Code & Math in 128K BPE

Explore EuroBERT’s cross-lingual and STEM domains enabled by adopting the 128,256-token LLaMA-3 vocabulary alongside 8,192 RoPE context.

12

Selecting Between NeoBERT, EuroBERT & ModernBERT

13

Pure-JS vs Python FP64 Verification Ledger

Every vector displayed in Section 06 is computed live in Pure JavaScript inside your browser and checked continuously against calibrated IEEE-754 Python FP64 reference tensors.

14

4 Production PyTorch & HuggingFace Transformers Recipes

15

8-Question NeoBERT & EuroBERT Mastery Check

MASTERY SCORE 0 / 8