⌗ TEXT REPRESENTATION — How Words Become Math
From Tokens to Vectors
Every NLP model you have met so far — the RNN of Lecture 02, the LSTM of Lecture 03 — silently assumed its input was already numbers. This lecture fills in that assumption: the eight rungs of the ladder that turn raw text into geometry a network can learn from. Each rung fixes exactly one wound of the rung before it — and keeps one of its own.
ONE SENTENCE, EIGHT LIVES
“The keys to the cabinet are on the table.” — the same probe sentence from Lecture 03. This time we won't feed it to a memory cell; we'll watch it get encoded, rung by rung, and feel what each encoding preserves and what it throws away.
Nothing here is a screenshot. Every table, grid and map on this page is computed live from the text in front of you — edit any input and the math follows.
RUNG 1 · WORD TOKENS ↓THE LADDER'S LOGIC
text → ids → sparse → counts → weighted → denseThe whole history of pre-transformer NLP is these four pairs. Keep this map in your head all the way down.
Before any math, a decision that seems trivial and isn't: what is one "unit" of language? Letters? Words? Syllables? Rung 1 makes the call that everything else inherits.
02 RUNG 1 · WORD TOKENS
Break language into units
A model never sees a sentence — it sees a list. Tokenization is the act of chopping raw text into pieces called tokens, and the simplest useful chop is by words. Type below; the chips update live.
TOKEN STREAM · 0 tokens
Notice three things: “The” and “the” — case would double-count them, so real pipelines lowercase (this demo does). Punctuation — sometimes a token, sometimes noise; toggle it and watch the count change. Words like “don't” — the apostrophe rule here keeps it whole, but there is no single correct answer. Tokenization is a design choice, and modern subword tokenizers (Lecture 05) revisit it entirely.
Tokens are still strings — and strings can't be multiplied by weights. Next: give every word a number. Any number. It will matter that it doesn't matter which.
03 RUNG 2 · TOKEN IDS
Words become numbers — meaningless ones
Build a vocabulary: every unique word gets an integer ID. The IDs are arbitrary labels — shuffle them and nothing about the language changes. That arbitrariness is the whole point of this rung.
VOCABULARY · built from the probe sentence
THE SENTENCE, ENCODED
Shuffle the IDs: the encoding changes completely, yet carries exactly the same information. Conclusion: the integer itself holds no meaning — it is a name tag, not a measurement.
type a word…
The cure for fake ordering is brutally democratic: give every word its own axis, equally far from all others. Democratic — and ruinously expensive.
04 RUNG 3 · ONE-HOT ENCODING
Every word an island
A one-hot vector is as long as the vocabulary, all zeros except a single 1 at the word's ID. No word is “between” two others anymore — every pair is exactly the same distance apart. Pick a word and see its vector.
VOCABULARY AXES · one column per word
DISTANCE LAB · how similar are two one-hot words?
Words are vectors now — but documents are what we actually classify, search and compare. The cheapest way to vectorize a document: pour its words into a bag and count.
05 RUNG 4 · BAG OF WORDS
Documents as counts — order dies here
Bag-of-Words represents a document as the sum of its words' one-hot vectors — one vocabulary-length row of counts. The classic demo of the price: two sentences, opposite meanings, identical vectors. Edit them and watch.
COUNT MATRIX · shared vocabulary
And yet — BoW powered decades of spam filters, search engines and topic classifiers, and still wins benchmarks as a baseline. Counts carry topic even when they lose meaning. Never skip the baseline.
Order matters — at least locally. “not good” ≠ “good”. The patch: tokenize into short word sequences instead of single words. It works. It also blows up.
06 RUNG 5 · N-GRAMS
Stealing back local order
An n-gram is a window of n consecutive tokens. Bigrams catch “New York” and “not good”; trigrams catch “United States of”. Slide n and watch the units change — then look at what n does to the vocabulary.
N-GRAM STREAM · 0
VOCABULARY EXPLOSION · unique n-grams in a 22-word corpus
On real corpora a 50k-word vocabulary holds ~2.5 billion possible bigrams — almost all rare or unseen. More features, less data per feature: the curse of dimensionality in its natural habitat.
Counting treats every word as equally informative. But “the” appears everywhere and says nothing, while “saturn” appears twice and says everything. Time to weigh the counts.
07 RUNG 6 · TF-IDF
Which words matter?
Term Frequency rewards words used often in this document; Inverse Document Frequency punishes words used in every document. Their product finds the words that characterize a text. Three mini-documents below — every number is hand-checkable.
Every method so far counts. The revolution of 2013: stop counting and learn — let a small network discover that words used in similar contexts should live at similar coordinates.
08 RUNG 7 · WORD2VEC & GLOVE
Meaning as geometry
“You shall know a word by the company it keeps” (Firth, 1957). Word2Vec (Mikolov et al., 2013) trains a tiny network to predict context words — and the learned weights turn out to place similar words at similar coordinates. GloVe (2014) reaches the same geometry by factorizing global co-occurrence counts. Below: a hand-placed 2D map that behaves like the real 300-D space.
THE EMBEDDING MAP · hover or tap any word — its 3 nearest neighbors light up
Hover a word to inspect its neighborhood…
HOW SKIP-GRAM LEARNS · slide the center word — pairs (center, context) are the training data
The network adjusts the center word's vector to better predict each context word. Words that predict the same contexts — “cat” and “dog” both sit near “pet”, “feed”, “barked” — get pulled to the same region. The task is an excuse; the vectors are the treasure.
THE FAMOUS ARITHMETIC · relationships are directions
The solid arrow and the dashed arrow are the same displacement — gender, royalty, capital-of all become consistent directions. This is the property no counting method ever had: structure you can do algebra on.
Word2Vec treats each word as an indivisible atom. But words aren't atoms — they're built from pieces, and the pieces carry meaning. FastText (2016) splits the atom.
09 RUNG 8 · FASTTEXT
Words are made of pieces
FastText (Bojanowski et al., 2016) represents each word as the sum of its character n-grams — “playing” becomes <pla, play, layi, ying, ing>, …. A word's vector is built from its subwords, so unseen, rare and misspelled words still get meaningful vectors. Type two words; shared subwords highlight.
SUBWORDS OF WORD 1 · character n-grams, n = 3…5, with < > boundaries
SUBWORDS OF WORD 2 · highlighted = shared with word 1
The < and > boundary markers let the model distinguish the prefix “un-” in “<un” from the “un” inside “bundle”. Morphology — prefixes, suffixes, roots — becomes shared structure: “playing” and “playful” borrow each other's vectors, and even “kingdoms” (never seen in training) inherits from “kingdom” + “-s”.
10 SUMMARY
The eight-step ladder, whole
Every rung fixed exactly one wound and kept one of its own. Read the table top to bottom — it is the entire pre-transformer history of text representation in eight rows.
| # | Representation | What it adds | What it still misses |
|---|---|---|---|
| 1 | Word tokens | Break language into units | Still text — no math possible |
| 2 | Token IDs | Convert words to numbers | IDs have no meaning — fake ordering |
| 3 | One-hot | Remove artificial ordering | Huge + sparse — no similarity at all |
| 4 | BoW | Represent documents | Loses order |
| 5 | N-grams | Capture local order | Vocabulary explosion |
| 6 | TF-IDF | Identify important words | Weak semantics — synonyms never meet |
| 7 | Word2Vec / GloVe | Learn semantic vectors | One vector per word — no polysemy, OOV fatal |
| 8 | FastText | Learn subword information | Still mostly static — context comes next |
Each representation is a repair of the previous one. When you meet a new encoding, ask two questions: what did it fix, and what did it keep broken?
Rungs 1–6 count; rungs 7–8 learn. Counting is transparent and free; learning discovers structure no one counted — at the price of opacity and training data.
One-hot/BoW/TF-IDF vectors are vocabulary-long and mostly zeros. Embeddings are 100–1000 dense floats where every dimension works. Dense is what neural networks were built to eat.
In a trained embedding space, similarity is distance and relationships are directions. king − man + woman ≈ queen is not a party trick — it is evidence the space has structure.
FastText's character n-grams handle rare words, typos and morphology. Modern tokenizers (BPE, WordPiece, SentencePiece — Lecture 05) are this idea industrialized.
All eight rungs produce static vectors. The transformer makes each vector a function of its sentence — “bank” by a river ≠ “bank” in a vault. The ladder continues in Lecture 05.
11 CODE LAB
Every rung, in runnable Python
Three tabs: the whole ladder in twelve lines of standard library, TF-IDF the way it's done at work, and Word2Vec + FastText training on a toy corpus — including the OOV trick Word2Vec can't do.
12 MASTERY CHECK