CAS logo CAS NLP LABInteractive Tutorial Series ← Series

⌗ LECTURE 18 · SINGULAR VALUE DECOMPOSITION · every number computed live

Every matrix is
spin · stretch · spin

Lecture 17 showed that a matrix is a motion of space. SVD makes a bold promise about that motion: no matter how messy the matrix is, it is secretly just three simple moves — a rotation, a stretch along the axes, and another rotation. Learn to see those three moves and you can compress images, find hidden topics in text, and understand why low-rank tricks power modern AI.

BIG IDEA→ CIRCLE TEST→ THREE MOVES→ BY HAND→ COMPRESSION→ LSA · NLP

THE PREVIEW — one matrix, taken apart into its three moves, on a loop

3MOVESU · Σ · Vᵀ
8LIVE DEMOSnothing pre-rendered
1HAND EXAMPLEstep by step
~45MINUTESwith the quiz

Before any formulas: what does it actually mean to “decompose” a matrix? Start with a picture you can hold in your head.

02 THE BIG IDEA

One equation, three simple moves

Decomposing a number means writing it as simpler pieces: 12 = 2 · 2 · 3. SVD does the same for a matrix — it factors any matrix A into three matrices, each of which does only one easy thing:

A= U· Σ· Vᵀ
read it right → left, like function calls: first Vᵀ spins · then Σ stretches · then U spins again
Vᵀ — the first spin

A rotation (it may also mirror). It turns the input so that the two most important input directions, v₁ and v₂, line up with the x and y axes. Rotations never change lengths or angles.

Σ — the stretch

A diagonal matrix: it only stretches along the axes. The numbers on its diagonal, σ₁ ≥ σ₂ ≥ … ≥ 0, are the singular values — how strongly each direction is amplified, sorted biggest first.

U — the second spin

Another rotation. It turns the stretched shape into its final orientation. Its columns u₁, u₂ are the output directions — where the most important directions end up.

AN EVERYDAY PICTURE · ROLLING DOUGH

Imagine you are told to transform a round ball of pizza dough into a particular oval, lying at a particular angle on the counter. You would never do it in one confusing motion. You would (1) turn the dough so the direction you want to roll lines up with your rolling pin, (2) roll it — stretching it a lot in one direction and a little in the other, and (3) turn the oval into its final position. Turn, roll, turn. That is exactly Vᵀ, Σ, U — and SVD proves that every linear transformation can be done this way.

WHY STUDENTS SHOULD CARE · SINGULAR VALUES ARE AN IMPORTANCE RANKING

Because Σ sorts the stretches from biggest to smallest, SVD automatically tells you which directions of your data matter most. Keep the top few, throw away the rest, and you have a smaller, cleaner version of the matrix. That single idea is behind image compression (§07), topic discovery in text (§08), PCA, recommender systems, and LoRA fine-tuning of large language models (§09).

Nice story — but how do we find the special directions v₁ and v₂? Put a circle through the matrix and look.

03 THE CIRCLE TEST

Every circle becomes an ellipse

Take every arrow of length 1 — together they form the unit circle. Push all of them through A. Because A is linear, the circle always comes out as an ellipse (possibly squashed flat). The ellipse’s longest semi-axis is σ₁, its shortest is σ₂. Turn the white arrow v and watch how much A stretches it:

INPUT · unit circle and arrow v

OUTPUT · ellipse A·(circle) and arrow A·v

THE MATRIX A

ANGLE OF v30°

STRETCH |A·v| AS v GOES ALL THE WAY ROUND — the peak is σ₁, the valley is σ₂

|A·v| = —stretch of the current arrow
σ₁ = —biggest possible stretch — direction v₁ (blue dashed)
σ₂ = —smallest possible stretch — direction v₂ (red dashed)

The key sentence

A·v₁ = σ₁·u₁ and A·v₂ = σ₂·u₂. Each special input direction vᵢ lands on an ellipse axis uᵢ, stretched by σᵢ. That is all SVD is.

Always perpendicular

The peak and the valley of the stretch curve are always exactly 90° apart. So v₁ ⟂ v₂, and the ellipse axes u₁ ⟂ u₂. That is why V and U are rotations.

σ₂ = 0 means squashed

Try the SQUASH preset: the ellipse collapses into a line segment. One direction is completely destroyed — the matrix has rank 1. Count the non-zero σ’s and you have the rank.

We found v₁, v₂, σ₁, σ₂ and u₁, u₂ by looking. Now replay the matrix as the three separate moves — one button per matrix.

04 THREE MOVES, ANIMATED

Watch Vᵀ, then Σ, then U

Press the moves in order. The face shows you rotation (it turns), stretching (it gets wider and thinner) and mirroring (it flips). The faint gold outline at the end is A applied directly — the three moves land exactly on it. Pick another matrix and replay.

THE THREE MATRICES — computed live for the chosen A (the active move lights up)

Pictures convinced? Good. Now do it with a pencil — six small steps, every number shown.

05 BY HAND

A 2×2 SVD, one step at a time

The trick: we cannot find σ directly, but the symmetric matrix AᵀA hides them as its eigenvalues (σ²). Start with A = [[3, 0], [4, 5]] — chosen because the answer comes out beautifully — then type your own numbers and every step recomputes.

A =

The recipe to remember: AᵀA → eigenvalues λ → σ = √λ → eigenvectors give V → U = A·V / σ. For big matrices nobody does this by hand (and computers use more stable algorithms), but this is exactly what they compute.

So far: 2×2. Real data is tall, wide and huge. What do the pieces look like then — and why does that let us compress?

06 ANATOMY

A matrix is a stack of simple layers

SVD works for any m×n matrix. Multiply it out and you get a second, very useful reading: A is a sum of rank-1 layers, each one a column u times a row vᵀ, scaled by σ. The layers come sorted — the first one carries the most, the last ones the least. Keep only the first k and you have the best possible rank-k approximation (the Eckart–Young theorem).

—numbers stored by A itself (m·n)
—numbers stored by k layers: k·(m + n + 1)
—size of the factors relative to A

Time to see it on something real. A grayscale picture is just a matrix of brightness values…

07 IMAGE COMPRESSION LAB

Keep the loud layers, drop the whispers

Each picture below is a 96×96 matrix. Your browser just ran a full SVD on it (one-sided Jacobi, in plain JavaScript). Drag k to rebuild the picture from only its first k layers. Then compare the four pictures: structured images compress beautifully; random noise refuses.

ORIGINAL · 96×96 = 9 216 numbers

RANK-k APPROXIMATION

k = LAYERS KEPT8
—storage (— numbers)
—“energy” kept: Σσᵢ² of the kept layers ÷ total
—relative error ‖A − Aₖ‖ / ‖A‖
—layers needed for 99% energy

THE SINGULAR-VALUE “SCREE” PLOT — teal bars are the layers you kept

THE FIRST SIX LAYERS σᵢuᵢvᵢᵀ ON THEIR OWN (gray = 0, white = +, black = −) — notice each is just “a column times a row”

Why it works

Real images repeat themselves: neighbouring rows look alike. That repetition piles up in a few big singular values. The scree plot drops fast — a few layers carry almost everything.

Why noise fails

Random pixels share nothing. Apart from σ₁ (which only captures the average grey level), the energy is spread almost evenly over the remaining σ’s: the bars stay flat, and you need dozens of layers for 99%. Compressibility = structure.

The plaid trick

The PLAID image is built from a few row patterns times column patterns — it is almost exactly low-rank, so a handful of layers rebuild it perfectly. Find the k where it becomes exact.

This is an NLP series. So point the same machine at words — a matrix of which word appears in which document.

08 SVD READS TEXT

Latent Semantic Analysis: synonyms for free

In 1990, Deerwester et al. applied truncated SVD to a term–document count matrix and called it Latent Semantic Analysis. Below are eight tiny documents. Look closely: “car” and “automobile” never appear together, and neither do “cat” and “kitten”. A keyword search cannot know they are related. SVD can — because they keep the same company.

THE CORPUS

THE TERM × DOCUMENT MATRIX X (10 × 8)

Singular values of X:

CONCEPT SPACE — words (dots) and documents (squares) on the top-2 singular directions

Dashed rays run from the origin to each word. Words on the same ray point the same way → cosine 1, even if one sits further out. Words on exactly the same spot share one label — look for “car = automobile”.

k = CONCEPTS KEPT2

WORD SIMILARITY (cosine) — raw counts vs. after truncated SVD

pairraw countsLSA, top-k concepts

Try k = 2 (synonyms merge), then k = 8 (all layers kept = the raw matrix again: synonyms separate). Truncation is what creates the meaning.

SEARCH — click words to build a query

The bridge to word embeddings: LSA was the first “words as vectors from co-occurrence” method — the ancestor of the embeddings in Lecture 06. Levy & Goldberg (2014) even showed that word2vec’s skip-gram with negative sampling is implicitly factorizing a (shifted PMI) word–context matrix — the same family of idea as this demo.

09 WHERE SVD LIVES

The most useful factorization in data science

Once you can see “rotate · stretch · rotate, sorted by importance”, you will find SVD everywhere:

PCA

Center your data matrix and take its SVD: the rows of Vᵀ are the principal components, and σᵢ² / (n − 1) are the variances along them. PCA is an SVD.

Recommender systems

A users × movies rating matrix is approximately low-rank: a few hidden “taste” factors explain most ratings. Matrix-factorization methods in this family were central to the Netflix Prize.

LoRA for LLMs

Low-Rank Adaptation fine-tunes a giant model by learning a weight update ΔW = B·A of small rank r — betting, as §07 showed for pictures, that the important change lives in a few directions.

Denoising & least squares

Dropping tiny σ’s removes noise; inverting only the big ones gives the pseudo-inverse A⁺ = VΣ⁺Uᵀ, the stable way to solve least-squares problems.

Eigen-decomposition A = PΛP⁻¹SVD A = UΣVᵀ
Which matrices?Square only — and not all of themEvery matrix, any shape
Numbers on the diagonalEigenvalues — can be negative or complexSingular values — always real and ≥ 0
The direction matricesP — not orthogonal in generalU and V — always orthogonal (pure rotations/mirrors)
Input vs output directionsThe same set (vectors that stay on their own line)Two sets: v’s in, u’s out
A pure rotationComplex eigenvalues — no real pictureσ₁ = σ₂ = 1 — “no stretching”, perfectly real
When are they the same?For symmetric matrices with non-negative eigenvalues (try the SYMMETRIC preset in §04)

10 CODE LAB

Every demo, in a few lines of Python

NumPy reproduces the §05 hand calculation, a rank-k compression loop, and scikit-learn’s TruncatedSVD rediscovers the §08 synonyms. The terminal outputs are real runs.

SVD · THREE WAYS

11 MASTERY CHECK

Eight questions — on spin, stretch, spin.

YOUR SCORE

0 / 8