⌗ TEXT REPRESENTATION — How Words Become Math
From Tokens to Vectors
Every NLP model you have met so far — the RNN of Lecture 02, the LSTM of Lecture 03 — silently assumed its input was already numbers. This lecture fills in that assumption: the eight rungs of the ladder that turn raw text into geometry a network can learn from. Each rung fixes exactly one wound of the rung before it — and keeps one of its own.
ONE SENTENCE, EIGHT LIVES
“The keys to the cabinet are on the table.” — the same probe sentence from Lecture 03. This time we won't feed it to a memory cell; we'll watch it get encoded, rung by rung, and feel what each encoding preserves and what it throws away.
Nothing here is a screenshot. Every table, grid and map on this page is computed live from the text in front of you — edit any input and the math follows.
RUNG 1 · WORD TOKENS ↓THE LADDER'S LOGIC
text → ids → sparse → counts → weighted → denseThe whole history of pre-transformer NLP is these four pairs. Keep this map in your head all the way down.
Before any math, a decision that seems trivial and isn't: what is one "unit" of language? Letters? Words? Syllables? Rung 1 makes the call that everything else inherits.
02 RUNG 1 · WORD TOKENS
Break language into units
A model never sees a sentence — it sees a list. Tokenization is the act of chopping raw text into pieces called tokens, and the simplest useful chop is by words. Type below; the chips update live.
TOKEN STREAM · 0 tokens
Notice three things: “The” and “the” — case would double-count them, so real pipelines lowercase (this demo does). Punctuation — sometimes a token, sometimes noise; toggle it and watch the count change. Words like “don't” — the apostrophe rule here keeps it whole, but there is no single correct answer. Tokenization is a design choice, and modern subword tokenizers (Lecture 06) revisit it entirely.
Tokens are still strings — and strings can't be multiplied by weights. Next: give every word a number. Any number. It will matter that it doesn't matter which.
03 RUNG 2 · TOKEN IDS
Words become numbers — meaningless ones
Build a vocabulary: every unique word gets an integer ID. The IDs are arbitrary labels — shuffle them and nothing about the language changes. That arbitrariness is the whole point of this rung.
VOCABULARY · built from the probe sentence
THE SENTENCE, ENCODED
Shuffle the IDs: the encoding changes completely, yet carries exactly the same information. Conclusion: the integer itself holds no meaning — it is a name tag, not a measurement.
type a word…
The cure for fake ordering is brutally democratic: give every word its own axis, equally far from all others. Democratic — and ruinously expensive.
04 RUNG 3 · ONE-HOT ENCODING
Every word an island
A one-hot vector is as long as the vocabulary, all zeros except a single 1 at the word's ID. No word is “between” two others anymore — every pair is exactly the same distance apart. Pick a word and see its vector.
VOCABULARY AXES · one column per word
DISTANCE LAB · how similar are two one-hot words?
Words are vectors now — but documents are what we actually classify, search and compare. The cheapest way to vectorize a document: pour its words into a bag and count.
05 RUNG 4 · BAG OF WORDS
Documents as counts — order dies here
Bag-of-Words represents a document as the sum of its words' one-hot vectors — one vocabulary-length row of counts. The classic demo of the price: two sentences, opposite meanings, identical vectors. Edit them and watch.
COUNT MATRIX · shared vocabulary
And yet — BoW powered decades of spam filters, search engines and topic classifiers, and still wins benchmarks as a baseline. Counts carry topic even when they lose meaning. Never skip the baseline.
THE WORKED EXAMPLE, BY HAND
A tiny corpus of three documents — D1 = "I love machine learning", D2 = "I love deep learning", D3 = "machine learning is powerful". Tokenize all three, collect the shared vocabulary [I, love, machine, learning, deep, is, powerful], then count how many times each vocabulary word occurs in each document:
| Document | I | love | machine | learning | deep | is | powerful |
|---|---|---|---|---|---|---|---|
| D1 · I love machine learning | 1 | 1 | 1 | 1 | 0 | 0 | 0 |
| D2 · I love deep learning | 1 | 1 | 0 | 1 | 1 | 0 | 0 |
| D3 · machine learning is powerful | 0 | 0 | 1 | 1 | 0 | 1 | 1 |
It is called a bag of words because we behave as though we threw all the words into a bag: we know which words are present and how many times — but not their original positions.
WHY BOW WON DECADES ANYWAY
Consider 10,000 movie reviews. A computer cannot train on "This movie was absolutely fantastic" — but BoW converts every review into a fixed-length numeric vector: text → R^V, where V is the vocabulary size. With V = 20,000, every document — 5 words or 500 — becomes a 20,000-dimensional vector, and classical machine learning can finally eat text: Logistic Regression · Naive Bayes · SVM · Random Forest.
Order matters — at least locally. “not good” ≠ “good”. The patch: tokenize into short word sequences instead of single words. It works. It also blows up.
06 RUNG 5 · N-GRAMS
Stealing back local order
An n-gram is a window of n consecutive tokens. Bigrams catch “New York” and “not good”; trigrams catch “United States of”. Slide n and watch the units change — then look at what n does to the vocabulary.
N-GRAM STREAM · 0
VOCABULARY EXPLOSION · unique n-grams in a 22-word corpus
On real corpora a 50k-word vocabulary holds ~2.5 billion possible bigrams — almost all rare or unseen. More features, less data per feature: the curse of dimensionality in its natural habitat.
ONE SENTENCE, FOUR CUTS
An n-gram is a contiguous sequence of n tokens. One sentence — “I love machine learning” — chopped four ways:
| n | Name | The units | Count |
|---|---|---|---|
| 1 | unigrams | I · love · machine · learning | 4 |
| 2 | bigrams | I love · love machine · machine learning | 3 |
| 3 | trigrams | I love machine · love machine learning | 2 |
| 4 | 4-gram | I love machine learning | 1 |
Traditional word-level BoW is simply the n = 1 case — unigram features.
THE DOG-BITES-MAN FIX
Return to the wound of §05: D1 = "dog bites man" vs D2 = "man bites dog". Their unigram features are identical — both contain {dog, bites, man}. But look at the bigram features:
| Document | Unigram features | Bigram features |
|---|---|---|
| D1 · dog bites man | dog · bites · man | dog bites · bites man |
| D2 · man bites dog | dog · bites · man — identical | man bites · bites dog — different! |
The central intuition: BoW with unigrams knows words; n-grams let the model know local sequences of words.
THE DISTINCTION EVERYONE GETS WRONG
N-grams did not replace BoW — you can build a Bag of N-grams. When sklearn writes CountVectorizer(ngram_range=(1, 2)) it means: build count-based features from unigrams AND bigrams together. For “I love NLP” the feature set becomes:
Two orthogonal decisions — not a ladder of replacements. Hold that thought: it becomes essential at the very next rung, TF-IDF.
Counting treats every word as equally informative. But “the” appears everywhere and says nothing, while “saturn” appears twice and says everything. Time to weigh the counts.
07 RUNG 6 · TF-IDF
Which words matter?
Term Frequency rewards words used often in this document; Inverse Document Frequency punishes words used in every document. Their product finds the words that characterize a text. Three mini-documents below — every number is hand-checkable.
THE MATH, ONE PIECE AT A TIME
TF(AI, D1) = 2/7 ≈ 0.286. High TF = frequent in THIS document. But “the” occurring 20 times is still meaningless — TF alone is not enough.DF(the) = 990 — everywhere, undiscriminating. DF(glioblastoma) = 5 — rare, and potentially decisive.IDF(t) = log(N / DF(t)). So IDF(the) = log(1000/990) ≈ 0.004 — nearly no weight; IDF(glioblastoma) = log(1000/5) = log(200) ≈ 5.30 — heavy. Common everywhere → low IDF; rare across documents → high IDF.THE PRODUCT'S INTUITION
| Word | TF in document | IDF | TF-IDF |
|---|---|---|---|
| the | high | very low | low |
| patient | medium | medium | medium |
| glioblastoma | high | high | high |
TF-IDF effectively asks: is this word unusually important to THIS document compared with the rest of the corpus? — a far better question than raw counting's how many times did it occur? BoW count = how often · TF-IDF = how informative here.
THE MISCONCEPTION, CORRECTED — AND THE COMBINATION THAT WORKS
TF-IDF does not solve word order: “dog bites man” vs “man bites dog” — same words, same frequencies, same unigram TF-IDF vector. N-grams address local sequence; TF-IDF addresses feature weighting. Different problems — so they combine. TfidfVectorizer(ngram_range=(1, 2)) on “The movie is not good” yields:
Every feature — unigram AND bigram — receives a TF-IDF weight instead of a raw count: n-gram features + TF-IDF weighting. Far more precise than treating the three techniques as mutually exclusive.
CAPSTONE · THE COUNTING TRILOGY, WHOLE
| Property | BoW | N-grams | TF-IDF |
|---|---|---|---|
| Counts words / features | ✓ | ✓ | weighted, not raw counts |
| Fixed numeric representation | ✓ | ✓ | ✓ |
| Simple | ✓ | ✓ | ✓ |
| Captures local order | ✗ | ✓ | ✗ by itself |
| Captures phrases | ✗ | ✓ | only with n-grams |
| Reduces common-word importance | ✗ | ✗ | ✓ |
| Importance weighting | ✗ | ✗ | ✓ |
| Understands synonyms | ✗ | ✗ | ✗ |
| Learns semantics | ✗ | ✗ | ✗ |
| Sparse | yes | very | yes |
| Vocabulary explosion | medium | high | depends on features |
| Context-dependent meaning | ✗ | ✗ | ✗ |
DON'T TEACH THEM AS REPLACEMENTS — TWO ORTHOGONAL DECISIONS
TEXT → tokenization → vocabulary — then two independent choices: what are my features × how do I weight them:
| Feature choice ↓ · weighting → | COUNT | TF-IDF |
|---|---|---|
| Unigrams | traditional Bag of Words | unigram TF-IDF |
| Unigrams + bigrams | Bag of N-grams | n-gram TF-IDF |
N-grams change WHAT the features are; TF-IDF changes HOW MUCH each feature counts. Mix freely — the trilogy is a toolkit, not a timeline.
Every method so far counts. The revolution of 2013: stop counting and learn — let a small network discover that words used in similar contexts should live at similar coordinates.
08 RUNG 7 · WORD2VEC & GLOVE
Meaning as geometry
“You shall know a word by the company it keeps” (Firth, 1957). Word2Vec (Mikolov et al., 2013) trains a tiny network to predict context words — and the learned weights turn out to place similar words at similar coordinates. GloVe (2014) reaches the same geometry by factorizing global co-occurrence counts. Below: a hand-placed 2D map that behaves like the real 300-D space.
THE EMBEDDING MAP · hover or tap any word — its 3 nearest neighbors light up
Hover a word to inspect its neighborhood…
HOW SKIP-GRAM LEARNS · slide the center word — pairs (center, context) are the training data
The network adjusts the center word's vector to better predict each context word. Words that predict the same contexts — “cat” and “dog” both sit near “pet”, “feed”, “barked” — get pulled to the same region. The task is an excuse; the vectors are the treasure.
THE FAMOUS ARITHMETIC · relationships are directions
The solid arrow and the dashed arrow are the same displacement — gender, royalty, capital-of all become consistent directions. This is the property no counting method ever had: structure you can do algebra on.
Word2Vec treats each word as an indivisible atom. But words aren't atoms — they're built from pieces, and the pieces carry meaning. FastText (2016) splits the atom.
09 RUNG 8 · FASTTEXT
Words are made of pieces
FastText (Bojanowski et al., 2016) represents each word as the sum of its character n-grams — “playing” becomes <pla, play, layi, ying, ing>, …. A word's vector is built from its subwords, so unseen, rare and misspelled words still get meaningful vectors. Type two words; shared subwords highlight.
SUBWORDS OF WORD 1 · character n-grams, n = 3…5, with < > boundaries
SUBWORDS OF WORD 2 · highlighted = shared with word 1
The < and > boundary markers let the model distinguish the prefix “un-” in “<un” from the “un” inside “bundle”. Morphology — prefixes, suffixes, roots — becomes shared structure: “playing” and “playful” borrow each other's vectors, and even “kingdoms” (never seen in training) inherits from “kingdom” + “-s”.
10 SUMMARY
The eight-step ladder, whole
Every rung fixed exactly one wound and kept one of its own. Read the table top to bottom — it is the entire pre-transformer history of text representation in eight rows.
| # | Representation | What it adds | What it still misses |
|---|---|---|---|
| 1 | Word tokens | Break language into units | Still text — no math possible |
| 2 | Token IDs | Convert words to numbers | IDs have no meaning — fake ordering |
| 3 | One-hot | Remove artificial ordering | Huge + sparse — no similarity at all |
| 4 | BoW | Represent documents | Loses order |
| 5 | N-grams | Capture local order | Vocabulary explosion |
| 6 | TF-IDF | Identify important words | Weak semantics — synonyms never meet |
| 7 | Word2Vec / GloVe | Learn semantic vectors | One vector per word — no polysemy, OOV fatal |
| 8 | FastText | Learn subword information | Still mostly static — context comes next |
Each representation is a repair of the previous one. When you meet a new encoding, ask two questions: what did it fix, and what did it keep broken?
Rungs 1–6 count; rungs 7–8 learn. Counting is transparent and free; learning discovers structure no one counted — at the price of opacity and training data.
One-hot/BoW/TF-IDF vectors are vocabulary-long and mostly zeros. Embeddings are 100–1000 dense floats where every dimension works. Dense is what neural networks were built to eat.
In a trained embedding space, similarity is distance and relationships are directions. king − man + woman ≈ queen is not a party trick — it is evidence the space has structure.
FastText's character n-grams handle rare words, typos and morphology. Modern tokenizers (BPE, WordPiece, SentencePiece — Lecture 06) are this idea industrialized.
All eight rungs produce static vectors. The transformer makes each vector a function of its sentence — “bank” by a river ≠ “bank” in a vault. The ladder continues in Lecture 06.
11 CODE LAB
Every rung, in runnable Python
Four tabs: the whole ladder in twelve lines of standard library, TF-IDF the way it's done at work, Word2Vec + FastText training on a toy corpus — including the OOV trick Word2Vec can't do — and Keras, where one TextVectorization layer reproduces rungs 2, 4 and 6 of this very ladder.
12 MASTERY CHECK