CAS Logo CAS-NLP · L36 Decoding & Sampling ← Hub
LECTURE 36 · GREEDY · TEMPERATURE (T) · TOP-K · TOP-P (NUCLEUS) · COMBINED PIPELINE

Generative Decoding & Sampling Strategies

A transformer forwards a score (logit) for all 50,000+ words on every single generated token — yet Greedy argmax traps models in robotic loops while Pure sampling leaks nonsense tail tokens. Master how Temperature (T), Top-K, and Nucleus (Top-P) reshape and slice distributions step-by-step, and how all three snap together in exact production order: Logits → Divide by T → Top-K → Top-P → Softmax → Sample.

1.48 vs 12.38BEAM vs HUMAN PPLGreedy/Beam loops (28.94% rep)
q_i ~ exp(z_i/T)TEMPERATURE (T)Hinton et al. (arXiv:1503.02531)
2 : 1 WinTOP-K STORY GENFan et al. k=10 human eval
13.13 / 0.97NUCLEUS P=0.95Matches human PPL & HUSE
z/T → K → PCOMBINED ORDERCanonical HuggingFace warpers
INTERACTIVE LEARNING MAP

Six Visual Ideas to Master Every Decoding Knob

Start with the dual failure modes discovered by Holtzman et al. (ICLR 2020): maximization collapses into loops on the left, while untrimmed random tails drift into gibberish on the right.

↓ 02 · THE DOUBLE DEGENERATION TRAP & TAIL SLICER

02 LOOP TRAP & UNRELIABLE TAIL

Why Pure Greedy Fails — And Pure Sampling Leaks Garbage

Human text walks an unexpected sweet spot (~12.38 Perplexity on WebText). Picking the highest-probability token every step drives Perplexity unnaturally low (1.48) and locks models in boring repetition loops (28.94%), whereas sampling the full long tail pushes Perplexity too high (22.73).

Empirical benchmark: https://arxiv.org/abs/1904.09751 (Holtzman et al., ICLR 2020 Table 1)
HOLTZMAN TABLE 1 CURVE

Perplexity vs Repetition% Trade-off

GEOMETRIC PROOF STEPPER

Watch All Three Filters Slice Together

Move from static theory to hands-on experimentation: twist Temperature, Top-K, and Top-P together below and watch live token probabilities reshape, mask, and re-normalize in real time.

↓ 03 · INTERACTIVE 4-LENS DECODING STUDIO

03 VISUAL LEARNING STUDIO

Live Greedy, Temperature, Top-K & Top-P Simulator

Switch between a Confident factual prompt, a Creative flat prompt, and an Ambiguous technical prompt. Drag all three knobs together and click Sample Next Token to observe real multinomial draws!

Temperature (T)0.70
Top-K (0 = Off)4
Top-P Nucleus0.90

Before diving into equations, ground each decoding rule with concrete analogies and geometric comparisons showing exactly why static Top-K breaks where dynamic Top-P succeeds.

↓ 04 · INTUITION FIRST: GREEDY, T, TOP-K & TOP-P

04 INTUITION BEFORE EQUATIONS

How Greedy, Temperature, Top-K & Top-P Differ Visually

Explore three storytelling diagrams: the everyday restaurant-menu analogy, why static Top-K fails on extreme distributions, and why Temperature transforms logit gaps exponentially inside softmax.

Now combine all three knobs together. Ordering them wrong silently shifts your nucleus budget or triggers NaN crashes — here is the canonical pipeline order.

↓ 05 · COMBINING TEMPERATURE + TOP-K + TOP-P SAFELY

05 ALL THREE TOGETHER · EXECUTION ORDER

Why z/T → Top-K → Top-P → Softmax Order Matters

In production frameworks (Hugging Face _get_logits_warper), Temperature scales logits before truncation so cumulative probability mass reflects the sharpened or softened shape before Top-K and Top-P cut the tail.

What Breaks With Naive Ordering

COMMON BUGS

Running Top-P before dividing by T ignores your Temperature setting when deciding where the nucleus boundary falls. Worse, checking cumsum <= p without protecting rank 1 masks out every word whenever the top choice already beats p!

Why T → Top-K → Top-P Harmonizes

CANONICAL PIPELINE

First z / T adjusts confidence smoothly (and catches T = 0 as Greedy). Second, Top-K bounds worst-case noise in O(V) selection. Third, Top-P tightens or relaxes dynamically using normalized remaining probabilities while protecting at least min_tokens_to_keep = 1.

Follow the exact floating-point arithmetic verified in Python FP64 (to 6 decimal places) for both our Peaked and Creative test sentences.

↓ 06 · STEP-BY-STEP 6-DECIMAL NUMERICAL TRACE

06 BY-HAND MATRIX WALKTHROUGH

Exact 6-Decimal Trace: Single Filters vs Combined

Every probability below is calibrated to < 1e-6 FP64 precision against scratch/calibrate_decoding.py. Watch Combined (T=0.7 → Top-K=4 → Top-P=0.90) turn completely deterministic (100% "Paris") on confident trivia while preserving 4 rich story nouns on creative prose!

Zoom out to the full token generation step inside an LLM server — click each pipeline block or the safety bypass rail to inspect data shapes and failure guards.

↓ 07 · DECODING PIPELINE ARCHITECTURE DIAGRAM

07 FULL INFERENCE ARCHITECTURE

Interactive Logit Warper & Safety Bypass Schematic

An autoregressive decoder does not replace KV caching or attention — sampling sits directly on top of the final vocabulary logits head, guarded by a dedicated safety bypass rail.

Quantify exactly how your parameter combination alters Shannon Entropy H(P), effective candidate count 2^H, and removed tail probability mass.

↓ 08 · COMBINED ENTROPY & TAIL CUT SANDBOX

08 ENTROPY & TAIL MASS SIZER

Dial Temperature, Top-K & Top-P Jointly

Observe the exact surviving candidate pool, Shannon entropy H(P) in bits, perplexity 2^H (effective number of unbiased coin choices), and percentage of noisy tail mass eliminated.

Temperature T0.70
Top-K (0 = Off)4
Top-P Nucleus0.90
Pipeline(...)
Teal: Kept Probability Mass (Renormalized to 100%) Rose: Pruned Unreliable Tail Mass
KEPT TOKENS4 / 6 tokens
SHANNON ENTROPY H(P)1.959 bits
EFFECTIVE CHOICES (2^H)3.89 choices
TAIL MASS PRUNED14.6% cut

Review published WebText measurements from Holtzman et al. (ICLR 2020) and examine the four silent implementation bugs that derail real LLM deployments.

↓ 09 · BENCHMARKS & PRODUCTION PITFALLS

09 EMPIRICAL BENCHMARKS & TRAPS

Holtzman et al. (ICLR 2020) Table 1 & Four Silent Pitfalls

Measured on GPT-2 Large generating WebText continuations (arXiv:1904.09751), alongside Fan et al. (arXiv:1805.04833) showing k=10 sampling wins 2-to-1 over baselines on story prompts.

Decoding Strategy Perplexity Self-BLEU Repetition % Human / HUSE Paper Takeaway
Production Decoding Traps Toggle:

Side-by-side cheat sheet comparing every decoding method by mechanism, determinism, tail handling, and ideal task fit.

↓ 10 · DECODING STRATEGY LEDGER

10 ARCHITECTURE & TUNING LEDGER

All Strategies Compared At a Glance

Match your task — factual extraction, coding, open storytelling, or balanced chat — to the right combination of knobs.

Method Core Formula / Rule Adaptive Width? Primary Risk Best Recommended Task
Greedy (T = 0) argmax_i z_i No (Always 1) 28.94%-class repetition loops in long text Math answers, JSON extraction, unit-test assertions
Beam Search (b) Track top b sequence log-probs Fixed b beams High repetition (28.94% at b=16), bland phrasing Constrained translation / exact formatting
Temperature (T) softmax(z_i / T) No hard zeroing High T lifts weird tail words without truncation Combined with Top-K / Top-P (0.2 code, 0.7 chat)
Top-K (Fan 2018) Keep top K highest logits Static K slots Too wide on peaked facts, cuts options on flat prompts Coarse upper bound (K=40..50) ahead of Top-P
Top-P (Holtzman 2020) Smallest prefix CumSum >= p Yes (1 to N+) Needs min_tokens_to_keep=1 on extreme peaked prompts Natural dialogue, prose, summarization (p=0.90..0.95)
Combined (T→K→P) z/T → Top-K → Top-P → Sample Bounded Dynamic Requires do_sample=True + T=0 greedy branch Modern production LLM default (T=0.7, K=40, P=0.90)

Drop directly into production Python code across PyTorch, Keras 3, HuggingFace transformers, pure FP64 NumPy, and boundary-guarded samplers.

↓ 11 · PYTORCH & KERAS 3 CODE LAB

11 PRODUCTION IMPLEMENTATION

Clean PyTorch, Keras 3 & HuggingFace Code Lab

Five runnable modules enforcing canonical ordering (z/T → Top-K → Top-P → Softmax), PyTorch .train(False), Keras 3 training=False, and zero-division guards.


          

Test your command of Holtzman Table 1 metrics, Temperature geometry, static-K traps, combined pipeline ordering, and silent production bugs.

↓ 12 · 8-ITEM MASTERY QUIZ

12 MASTERY CHECK

Decoding & Sampling Mastery Quiz

Eight diagnostic scenarios validating your intuition and production engineering mastery.

CURRENT SCORE
Score: 0 / 8