CAS · L29
← DEBERTA-V3
Nomic 2024 · Jina AI 2024 · arXiv:2402.01613 & 2409.10173 · Lecture 29

Nomic Embed & Jina-v3 — 8K Long-Context Bi-Encoders, Matryoshka & Task LoRA

Legacy 512-token bi-encoders truncate RAG documents and freeze fixed vector widths. Nomic Embed (137M) rebuilt BERT with RoPE 8,192 + SwiGLU + FlashAttention + MRL (62.39 MTEB, 85.5 LoCo), while Jina-Embeddings-v3 (570M, 89 langs) added 5 Rank-4 Task LoRA adapters and nested Matryoshka slicing down to 32 dims — hitting 65.52 English MTEB.

8,192
Native RoPE Context
65.52
Jina-v3 MTEB English Avg
62.39
Nomic-v1 MTEB · 85.5 LoCo
r = 4
5 Task LoRAs (<3% params)
32x
MRL + 1-Bit RAM Shrink
INTERACTIVE SCENARIO EXPLORER · SWITCH MECHANISM
02 · WHY LEGACY BI-ENCODERS HIT A WALL

From 512-Token Truncation & Fixed Widths to Resizable 8K Vectors

Compare how classic Sentence-BERT/E5 chunking fails long documents vs modern RoPE + MRL + LoRA bi-encoders.

03 · ARCHITECTURE ANATOMY

Interactive 12-Block Nomic & Jina-v3 Embedding Diagram

Switch between Nomic-Embed-v1.5 (137M NomicBERT) and Jina-Embeddings-v3 (570M XLM-R + Task LoRA), filter by stage, and click any block.

MODEL FAMILY PRESET
STAGE FOCUS FILTER
04 · CONTEXT LENGTH SCALING

Extending Bi-Encoders to 8,192 Tokens via Dynamic NTK RoPE

Drag the document length slider to compare 512-token fragmentation against single-pass 8K inference and Dynamic NTK phase stretch.

4096 tokens
05 · PARAMETER-EFFICIENT MULTI-TASKING

Jina-v3 5 Task LoRAs (r=4) vs Nomic Instruction Prefixes

Inspect how five lightweight rank-4 LoRA adapters adapt the frozen 570M XLM-RoBERTa backbone cleanly without prompt-token interference.

06 · VERIFIED PURE-JS WALKTHROUGH

6-Decimal RoPE + Task-LoRA + MRL Forward Trace

Step through exact tensor arithmetic (d = 4, r = 2 LoRA, tau = 0.10): RoPE rotation, frozen W_0 vs LoRA shift, mean pooling, 4D to 2D MRL slice, and 1-bit sign compression.

STEP 1 / 6
07 · NESTED EMBEDDING SLICE

Matryoshka Representation Learning (1024d down to 32d)

Select any target slice dimension to inspect Jina-v3 Retrieval nDCG@10, STS text-matching stability, and Nomic v1.5 MTEB retention.

08 · CONTRASTIVE OPTIMIZATION

235M Curated Pairs & InfoNCE Temperature Lab

Adjust temperature tau to observe how tightly InfoNCE penalizes hard negatives versus easy in-batch distractors.

tau = 0.05
09 · BENCHMARK STANDINGS

MTEB Leaderboard: Jina-v3 (65.52) & Nomic v1 (62.39)

Compare open-weight 137M and 570M long-context encoders directly against OpenAI ada-002, text-embedding-3-small/large, and multilingual-e5-large-instruct.

10 · VECTOR DATABASE SCALE

Combining MRL + INT8 / 1-Bit Binary Quantization

Inspect per-passage bytes and cluster RAM across 10 million documents when stacking 256d Matryoshka slicing with 1-bit sign compression.

11 · ENGINEERING PLAYBOOK

Production RAG Traps: Prefixes, LayerNorm & Hot-Swaps

Three high-impact deployment details that frequently trip up RAG indexing pipelines with Nomic v1.5 and Jina-v3.

12 · HISTORICAL EVOLUTION

From Word2Vec to Multi-Task 8K Bi-Encoders

Click any milestone along the embedding encoder timeline.

13 · SIDE-BY-SIDE LEDGER

Architecture & Capability Comparison Ledger

Full specification comparison across Nomic-Embed-v1.5, Jina-v3, multilingual-e5-large-instruct, and text-embedding-3-large.

14 · PYTHON & PYTORCH RECIPES

Four Drop-In Production Recipes

Nomic LayerNorm+MRL slicing, Jina Task LoRA swapping, PyTorch nested MRL InfoNCE loss, and 1-bit binary packing.

15 · MASTERY VERIFICATION

8-Question Nomic & Jina-v3 Check

Test your understanding of Dynamic NTK 8K RoPE, Rank-4 Task LoRAs, MRL prefix training, LayerNorm pre-slicing, InfoNCE, MTEB scores, and 1-bit quantization.

Current Score
0 / 8