⌗ LECTURE 18 · SINGULAR VALUE DECOMPOSITION · every number computed live
Every matrix is
spin · stretch · spin
Lecture 17 showed that a matrix is a motion of space. SVD makes a bold promise about that motion: no matter how messy the matrix is, it is secretly just three simple moves — a rotation, a stretch along the axes, and another rotation. Learn to see those three moves and you can compress images, find hidden topics in text, and understand why low-rank tricks power modern AI.
THE PREVIEW — one matrix, taken apart into its three moves, on a loop
Before any formulas: what does it actually mean to “decompose” a matrix? Start with a picture you can hold in your head.
02 THE BIG IDEA
One equation, three simple moves
Decomposing a number means writing it as simpler pieces: 12 = 2 · 2 · 3. SVD does the same for a matrix — it factors any matrix A into three matrices, each of which does only one easy thing:
A rotation (it may also mirror). It turns the input so that the two most important input directions, v₁ and v₂, line up with the x and y axes. Rotations never change lengths or angles.
A diagonal matrix: it only stretches along the axes. The numbers on its diagonal, σ₁ ≥ σ₂ ≥ … ≥ 0, are the singular values — how strongly each direction is amplified, sorted biggest first.
Another rotation. It turns the stretched shape into its final orientation. Its columns u₁, u₂ are the output directions — where the most important directions end up.
AN EVERYDAY PICTURE · ROLLING DOUGH
Imagine you are told to transform a round ball of pizza dough into a particular oval, lying at a particular angle on the counter. You would never do it in one confusing motion. You would (1) turn the dough so the direction you want to roll lines up with your rolling pin, (2) roll it — stretching it a lot in one direction and a little in the other, and (3) turn the oval into its final position. Turn, roll, turn. That is exactly Vᵀ, Σ, U — and SVD proves that every linear transformation can be done this way.
WHY STUDENTS SHOULD CARE · SINGULAR VALUES ARE AN IMPORTANCE RANKING
Because Σ sorts the stretches from biggest to smallest, SVD automatically tells you which directions of your data matter most. Keep the top few, throw away the rest, and you have a smaller, cleaner version of the matrix. That single idea is behind image compression (§07), topic discovery in text (§08), PCA, recommender systems, and LoRA fine-tuning of large language models (§09).
Nice story — but how do we find the special directions v₁ and v₂? Put a circle through the matrix and look.
03 THE CIRCLE TEST
Every circle becomes an ellipse
Take every arrow of length 1 — together they form the unit circle. Push all of them through A. Because A is linear, the circle always comes out as an ellipse (possibly squashed flat). The ellipse’s longest semi-axis is σ₁, its shortest is σ₂. Turn the white arrow v and watch how much A stretches it:
INPUT · unit circle and arrow v
OUTPUT · ellipse A·(circle) and arrow A·v
THE MATRIX A
STRETCH |A·v| AS v GOES ALL THE WAY ROUND — the peak is σ₁, the valley is σ₂
A·v₁ = σ₁·u₁ and A·v₂ = σ₂·u₂. Each special input direction vᵢ lands on an ellipse axis uᵢ, stretched by σᵢ. That is all SVD is.
The peak and the valley of the stretch curve are always exactly 90° apart. So v₁ ⟂ v₂, and the ellipse axes u₁ ⟂ u₂. That is why V and U are rotations.
Try the SQUASH preset: the ellipse collapses into a line segment. One direction is completely destroyed — the matrix has rank 1. Count the non-zero σ’s and you have the rank.
We found v₁, v₂, σ₁, σ₂ and u₁, u₂ by looking. Now replay the matrix as the three separate moves — one button per matrix.
04 THREE MOVES, ANIMATED
Watch Vᵀ, then Σ, then U
Press the moves in order. The face shows you rotation (it turns), stretching (it gets wider and thinner) and mirroring (it flips). The faint gold outline at the end is A applied directly — the three moves land exactly on it. Pick another matrix and replay.
THE THREE MATRICES — computed live for the chosen A (the active move lights up)
Pictures convinced? Good. Now do it with a pencil — six small steps, every number shown.
05 BY HAND
A 2×2 SVD, one step at a time
The trick: we cannot find σ directly, but the symmetric matrix AᵀA hides them as its eigenvalues (σ²). Start with A = [[3, 0], [4, 5]] — chosen because the answer comes out beautifully — then type your own numbers and every step recomputes.
So far: 2×2. Real data is tall, wide and huge. What do the pieces look like then — and why does that let us compress?
06 ANATOMY
A matrix is a stack of simple layers
SVD works for any m×n matrix. Multiply it out and you get a second, very useful reading: A is a sum of rank-1 layers, each one a column u times a row vᵀ, scaled by σ. The layers come sorted — the first one carries the most, the last ones the least. Keep only the first k and you have the best possible rank-k approximation (the Eckart–Young theorem).
Time to see it on something real. A grayscale picture is just a matrix of brightness values…
07 IMAGE COMPRESSION LAB
Keep the loud layers, drop the whispers
Each picture below is a 96×96 matrix. Your browser just ran a full SVD on it (one-sided Jacobi, in plain JavaScript). Drag k to rebuild the picture from only its first k layers. Then compare the four pictures: structured images compress beautifully; random noise refuses.
ORIGINAL · 96×96 = 9 216 numbers
RANK-k APPROXIMATION
THE SINGULAR-VALUE “SCREE” PLOT — teal bars are the layers you kept
THE FIRST SIX LAYERS σᵢuᵢvᵢᵀ ON THEIR OWN (gray = 0, white = +, black = −) — notice each is just “a column times a row”
Real images repeat themselves: neighbouring rows look alike. That repetition piles up in a few big singular values. The scree plot drops fast — a few layers carry almost everything.
Random pixels share nothing. Apart from σ₁ (which only captures the average grey level), the energy is spread almost evenly over the remaining σ’s: the bars stay flat, and you need dozens of layers for 99%. Compressibility = structure.
The PLAID image is built from a few row patterns times column patterns — it is almost exactly low-rank, so a handful of layers rebuild it perfectly. Find the k where it becomes exact.
This is an NLP series. So point the same machine at words — a matrix of which word appears in which document.
08 SVD READS TEXT
Latent Semantic Analysis: synonyms for free
In 1990, Deerwester et al. applied truncated SVD to a term–document count matrix and called it Latent Semantic Analysis. Below are eight tiny documents. Look closely: “car” and “automobile” never appear together, and neither do “cat” and “kitten”. A keyword search cannot know they are related. SVD can — because they keep the same company.
THE CORPUS
THE TERM × DOCUMENT MATRIX X (10 × 8)
Singular values of X:
CONCEPT SPACE — words (dots) and documents (squares) on the top-2 singular directions
Dashed rays run from the origin to each word. Words on the same ray point the same way → cosine 1, even if one sits further out. Words on exactly the same spot share one label — look for “car = automobile”.
WORD SIMILARITY (cosine) — raw counts vs. after truncated SVD
Try k = 2 (synonyms merge), then k = 8 (all layers kept = the raw matrix again: synonyms separate). Truncation is what creates the meaning.
SEARCH — click words to build a query
KEYWORD MATCH (cosine on raw counts)
LSA MATCH (cosine in concept space)
09 WHERE SVD LIVES
The most useful factorization in data science
Once you can see “rotate · stretch · rotate, sorted by importance”, you will find SVD everywhere:
Center your data matrix and take its SVD: the rows of Vᵀ are the principal components, and σᵢ² / (n − 1) are the variances along them. PCA is an SVD.
A users × movies rating matrix is approximately low-rank: a few hidden “taste” factors explain most ratings. Matrix-factorization methods in this family were central to the Netflix Prize.
Low-Rank Adaptation fine-tunes a giant model by learning a weight update ΔW = B·A of small rank r — betting, as §07 showed for pictures, that the important change lives in a few directions.
Dropping tiny σ’s removes noise; inverting only the big ones gives the pseudo-inverse A⁺ = VΣ⁺Uᵀ, the stable way to solve least-squares problems.
| Eigen-decomposition A = PΛP⁻¹ | SVD A = UΣVᵀ | |
|---|---|---|
| Which matrices? | Square only — and not all of them | Every matrix, any shape |
| Numbers on the diagonal | Eigenvalues — can be negative or complex | Singular values — always real and ≥ 0 |
| The direction matrices | P — not orthogonal in general | U and V — always orthogonal (pure rotations/mirrors) |
| Input vs output directions | The same set (vectors that stay on their own line) | Two sets: v’s in, u’s out |
| A pure rotation | Complex eigenvalues — no real picture | σ₁ = σ₂ = 1 — “no stretching”, perfectly real |
| When are they the same? | For symmetric matrices with non-negative eigenvalues (try the SYMMETRIC preset in §04) | |
10 CODE LAB
Every demo, in a few lines of Python
NumPy reproduces the §05 hand calculation, a rank-k compression loop, and scikit-learn’s TruncatedSVD rediscovers the §08 synonyms. The terminal outputs are real runs.
11 MASTERY CHECK