cat → 7. IDs keep the input compact, but ID 7 is not “more cat” than ID 6.
LECTURE 06 · FROM TEXT PREPROCESSING TO LEARNED MEANING
Words become
coordinates.
An embedding is a learned lookup table: give it a token ID, get back a dense vector. The interesting question is how those rows get their geometry. Meet the Keras layer that stores them, then compare Word2Vec, GloVe, and FastText — three ways to teach words where to live.
OPEN THE EMBEDDING LAB ↓illustrative coordinates
Same output shape.
Different learning stories.
A matrix with one row per token.
Token IDs are addresses, not magnitudes. The Keras Embedding layer uses each integer to select a trainable row from a matrix of shape (vocabulary size, embedding dimension).
Think of a numbered shelf of word cards
The word cat becomes an ID, such as 7. The ID is only an address. Row 7 holds a short list of learned numbers; nearby values emerge through training, not from the number 7 itself.
If E has shape (12, 4), E[7] returns four numbers. A six-word sentence becomes a 6 × 4 grid.
The sentiment classifier makes a prediction, computes loss, and backpropagation adjusts the rows used by the sentence. Keras Embedding stores these trainable rows.
Try it: change the sentence below and click a token. Watch which table row is selected; then change the dimension to see the output shape change.
This demo uses a deterministic teaching matrix. A real Keras layer starts with initialized weights and learns them from the task loss.
output = E[token_id] — no multiplication by a one-hot vector is needed in practice; it is the same row selection computed efficiently.Keras Embedding: complete IMDb sentiment lab
IMDb is already integer-tokenized by Keras. The script pads/truncates sequences, splits the official training set, learns an Embedding layer, uses early stopping, evaluates the untouched test set, prints precision/recall/F1 and ROC AUC, and saves a confusion matrix.
pip install tensorflow keras numpy scikit-learn matplotlib
python keras_embedding_imdb.pySHOW COMPLETE RUNNABLE SCRIPT ↕
"""Train and evaluate a Keras Embedding sentiment model on IMDb.
Run: python keras_embedding_imdb.py
Install: pip install tensorflow keras numpy scikit-learn matplotlib
Keras downloads its integer-tokenized IMDb dataset on first run.
"""
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
import numpy as np
import keras
from keras import layers
from keras.datasets import imdb
from keras.utils import pad_sequences
from sklearn.metrics import (ConfusionMatrixDisplay, classification_report,
confusion_matrix, roc_auc_score)
from sklearn.model_selection import train_test_split
SEED = 42
VOCAB = 12000
LENGTH = 220
keras.utils.set_random_seed(SEED)
# IMDb ships as token IDs; 0 is padding, 1 is start, 2 is OOV.
(train_sequences, train_labels), (test_sequences, test_labels) = imdb.load_data(
num_words=VOCAB, seed=SEED)
X_all = pad_sequences(train_sequences, maxlen=LENGTH, padding="post",
truncating="pre")
X_test = pad_sequences(test_sequences, maxlen=LENGTH, padding="post",
truncating="pre")
y_all = np.asarray(train_labels, dtype="int32")
y_test = np.asarray(test_labels, dtype="int32")
# Split only the official training partition; keep the official test untouched.
train_ids, val_ids = train_test_split(
np.arange(len(X_all)), test_size=0.2, random_state=SEED,
stratify=y_all)
X_train, y_train = X_all[train_ids], y_all[train_ids]
X_val, y_val = X_all[val_ids], y_all[val_ids]
print("train / validation / test:", len(X_train), len(X_val), len(X_test))
model = keras.Sequential([
keras.Input(shape=(LENGTH,), dtype="int32"),
layers.Embedding(input_dim=VOCAB, output_dim=64, mask_zero=True,
name="learned_word_vectors"),
layers.GlobalAveragePooling1D(),
layers.Dense(64, activation="relu"),
layers.Dropout(0.3),
layers.Dense(1, activation="sigmoid"),
])
model.compile(optimizer=keras.optimizers.Adam(1e-3),
loss="binary_crossentropy",
metrics=[keras.metrics.BinaryAccuracy(name="accuracy"),
keras.metrics.AUC(name="auc")])
model.summary()
model.fit(X_train, y_train, validation_data=(X_val, y_val),
batch_size=128, epochs=15, verbose=2,
callbacks=[keras.callbacks.EarlyStopping(
monitor="val_loss", patience=2, restore_best_weights=True)])
print("Untouched test metrics:", model.evaluate(X_test, y_test,
return_dict=True, verbose=0))
probabilities = model.predict(X_test, batch_size=256, verbose=0).ravel()
predictions = (probabilities >= 0.5).astype("int32")
print(classification_report(y_test, predictions,
target_names=["negative", "positive"],
zero_division=0))
print("Test ROC AUC:", round(roc_auc_score(y_test, probabilities), 4))
cm = confusion_matrix(y_test, predictions, labels=[0, 1])
ConfusionMatrixDisplay(cm, display_labels=["negative", "positive"]).plot(
cmap="Blues", colorbar=False)
plt.title("IMDb held-out test · Keras Embedding")
plt.tight_layout()
plt.savefig("embedding_imdb_confusion.png", dpi=160)
plt.close()
# Inspect learned rows after training; ID 0 is the padding row.
weights = model.get_layer("learned_word_vectors").get_weights()[0]
print("Embedding matrix:", weights.shape)
print("Example learned vector for token ID 10:", np.round(weights[10, :8], 3))
print("First 5 test probabilities:", np.round(probabilities[:5], 3))
model.save("embedding_imdb.keras")
print("Saved embedding_imdb.keras and embedding_imdb_confusion.png")
Teach a word by asking
who stands near it.
Word2Vec learns vectors from local context prediction. Choose the direction: predict the center word from its neighbors (CBOW), or predict surrounding words from the center (Skip-gram).
Learn a word from the company it keeps
Imagine repeatedly covering one word in a sentence. To recover it, a learner must notice that cat and dog often appear beside similar words. Their vectors become useful because they help solve the same prediction task.
In “the cat sat near the window,” a window of 1 around cat sees the and sat. Each position creates a new training example.
Skip-gram: cat → the, sat. CBOW: the + sat → cat. Both learn weights by making context predictions.
After many examples, the prediction weights carry useful word relationships. The highlighted pairs below are inputs to training, not proof that vectors have already learned.
Try it: switch Skip-gram and CBOW, then widen the window. Say aloud what is the input and what is the label each time.
These highlighted pairs are the training signal, not trained vectors. A real Word2Vec run optimizes prediction weights over many corpus windows (often with negative sampling).
Count the whole corpus.
Fit the relationships.
GloVe starts with a word–word co-occurrence matrix. Counts summarize how often words appear near each other across the corpus; training fits vector dot products to the logarithm of those counts, with a weighting function.
Replace millions of windows with a count map
Word2Vec learns while walking through examples. GloVe first counts how often each word appears near every other word across the corpus. Its training then asks vectors to explain those counts.
Every nearby pair increments a cell X[i,j]. Rows summarize each word’s surroundings over the whole corpus.
If cat and dog both co-occur with pet and plays, their rows share a pattern. A distinctive neighbor such as bark still separates them.
GloVe adjusts word and context vectors so their dot product plus biases approximates log(X[i,j]). Weighting keeps extremely common pairs from dominating.
Try it: read the cat and dog rows below. Which columns are shared, and which column distinguishes them?
Find the pattern. Rows for cat and dog share contexts like pet and plays; dog has extra signal from bark.
This fixed count grid is a small, hand-built illustration. Real GloVe training uses large corpora and a weighted least-squares objective; co-occurrence counts alone are not the final vectors.
Train GloVe vectors, then test them in a classifier
This Keras file builds a sparse co-occurrence table from training reviews only, fits the weighted log-count objective, checks held-out co-occurrence error and nearest neighbors, then freezes the vectors for IMDb sentiment classification. The classifier produces test accuracy, AUC, precision/recall/F1, and a confusion matrix.
pip install tensorflow keras numpy scikit-learn matplotlib
python keras_glove_imdb.pyGloVe itself has no class labels, so a confusion matrix belongs to the downstream sentiment classifier. This bounded educational run is not a benchmark for large pretrained GloVe vectors.
SHOW COMPLETE RUNNABLE SCRIPT ↕
"""Train small GloVe vectors, then test them in an IMDb classifier.
Run: python keras_glove_imdb.py
Install: pip install tensorflow keras numpy scikit-learn matplotlib
This teaching run uses a subset of TRAINING reviews for co-occurrence counts.
"""
from collections import Counter
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
import numpy as np
import keras
from keras import layers, ops
from keras.datasets import imdb
from keras.utils import pad_sequences
from sklearn.metrics import ConfusionMatrixDisplay, classification_report, confusion_matrix
from sklearn.model_selection import train_test_split
SEED = 42
VOCAB = 3000
DIM = 50
LENGTH = 160
WINDOW = 2
keras.utils.set_random_seed(SEED)
(train_reviews, train_labels), (test_reviews, test_labels) = imdb.load_data(
num_words=VOCAB, seed=SEED)
train_ids, val_ids = train_test_split(
np.arange(len(train_reviews)), test_size=5000, random_state=SEED,
stratify=train_labels)
X_train = pad_sequences([train_reviews[i] for i in train_ids], maxlen=LENGTH,
padding="post", truncating="pre")
X_val = pad_sequences([train_reviews[i] for i in val_ids], maxlen=LENGTH,
padding="post", truncating="pre")
X_test = pad_sequences(test_reviews, maxlen=LENGTH, padding="post",
truncating="pre")
y_train = np.asarray(train_labels)[train_ids]
y_val = np.asarray(train_labels)[val_ids]
y_test = np.asarray(test_labels)
# Count local context across training text only. IDs 0, 1, 2 are special.
counts = Counter()
for review in X_train[:1200]: # bounded so the first run stays approachable
words = [int(w) for w in review if 2 < w < VOCAB]
for position, center in enumerate(words):
for offset in range(-WINDOW, WINDOW + 1):
neighbor = position + offset
if offset and 0 <= neighbor < len(words):
counts[(center, words[neighbor])] += 1 / abs(offset)
print("Nonzero co-occurrence cells:", len(counts))
# X_ij is sparse: train only on observed pairs, never on zero counts.
items = counts.most_common(120000)
ids_i = np.asarray([pair[0][0] for pair in items], dtype="int32")
ids_j = np.asarray([pair[0][1] for pair in items], dtype="int32")
values = np.asarray([pair[1] for pair in items], dtype="float32")
target_log_count = np.log(values).reshape(-1, 1)
weights = np.minimum((values / 100.0) ** 0.75, 1.0)
perm = np.random.default_rng(SEED).permutation(len(values))
cut = max(1, round(0.1 * len(values)))
fit_ids, holdout_ids = perm[cut:], perm[:cut]
word_id = keras.Input(shape=(), dtype="int32")
context_id = keras.Input(shape=(), dtype="int32")
word_vectors = layers.Embedding(VOCAB, DIM, name="word_vectors")
context_vectors = layers.Embedding(VOCAB, DIM, name="context_vectors")
word_bias = layers.Embedding(VOCAB, 1, name="word_bias")
context_bias = layers.Embedding(VOCAB, 1, name="context_bias")
score = (ops.sum(word_vectors(word_id) * context_vectors(context_id),
axis=-1, keepdims=True)
+ word_bias(word_id) + context_bias(context_id))
glove = keras.Model([word_id, context_id], score)
glove.compile(optimizer=keras.optimizers.Adam(0.01), loss="mse")
glove.fit([ids_i[fit_ids], ids_j[fit_ids]], target_log_count[fit_ids],
sample_weight=weights[fit_ids], batch_size=4096, epochs=12,
validation_data=([ids_i[holdout_ids], ids_j[holdout_ids]],
target_log_count[holdout_ids], weights[holdout_ids]),
callbacks=[keras.callbacks.EarlyStopping(
monitor="val_loss", patience=2, restore_best_weights=True)], verbose=2)
print("Held-out observed-pair weighted MSE:", glove.evaluate(
[ids_i[holdout_ids], ids_j[holdout_ids]], target_log_count[holdout_ids],
sample_weight=weights[holdout_ids], verbose=0))
# GloVe commonly combines the two learned vector tables.
embeddings = (word_vectors.get_weights()[0] +
context_vectors.get_weights()[0]).astype("float32")
embeddings[0] = 0 # padding stays inert downstream
word_index = imdb.get_word_index()
id_to_word = {index + 3: word for word, index in word_index.items()
if index + 3 < VOCAB}
normalized = embeddings / np.maximum(np.linalg.norm(embeddings, axis=1,
keepdims=True), 1e-8)
for query in ("great", "bad", "movie"):
token_id = word_index.get(query, -99) + 3
if token_id not in id_to_word:
continue
similarity = normalized @ normalized[token_id]
neighbors = [id_to_word[i] for i in np.argsort(-similarity)
if i in id_to_word and i != token_id][:5]
print(f"Nearest to {query}: {neighbors}")
# Extrinsic evaluation: freeze GloVe and train a sentiment classifier.
classifier = keras.Sequential([
keras.Input(shape=(LENGTH,), dtype="int32"),
layers.Embedding(VOCAB, DIM, weights=[embeddings], trainable=False,
mask_zero=True, name="frozen_glove"),
layers.GlobalAveragePooling1D(),
layers.Dense(64, activation="relu"),
layers.Dropout(0.3),
layers.Dense(1, activation="sigmoid"),
])
classifier.compile(optimizer="adam", loss="binary_crossentropy",
metrics=[keras.metrics.BinaryAccuracy(name="accuracy"),
keras.metrics.AUC(name="auc")])
classifier.fit(X_train, y_train, validation_data=(X_val, y_val),
batch_size=128, epochs=12, verbose=2,
callbacks=[keras.callbacks.EarlyStopping(
monitor="val_loss", patience=2, restore_best_weights=True)])
print("Untouched sentiment test:", classifier.evaluate(X_test, y_test,
return_dict=True, verbose=0))
probabilities = classifier.predict(X_test, batch_size=256, verbose=0).ravel()
predictions = (probabilities >= 0.5).astype("int32")
print(classification_report(y_test, predictions,
target_names=["negative", "positive"],
zero_division=0))
cm = confusion_matrix(y_test, predictions, labels=[0, 1])
ConfusionMatrixDisplay(cm, display_labels=["negative", "positive"]).plot(
cmap="Blues", colorbar=False)
plt.title("IMDb test · frozen GloVe vectors")
plt.tight_layout()
plt.savefig("glove_imdb_confusion.png", dpi=160)
plt.close()
glove.save("glove_cooccurrence.keras")
classifier.save("glove_imdb_classifier.keras")
print("Saved both .keras models and glove_imdb_confusion.png")
A word is also its parts.
FastText represents a word using character n-grams as well as a whole-word vector. Related spellings can share subword parameters, and unseen forms can still be represented by their n-grams.
Give a word reusable spelling pieces
A single row per word cannot help much when a spelling never appeared in training. FastText learns vectors for character fragments, so a new word can borrow information from fragments seen inside other words.
Wrap playful as <playful> so pieces at the start and end are distinguishable.
For length 3, fragments include <pl, pla, and lay. play and playful share several fragments.
FastText sums the word vector and its n-gram vectors. For an unseen word, known n-gram vectors can still produce a representation; quality depends on what the corpus taught them.
Try it: type playfully and compare its pieces with playful; then type a made-up word and see what can be reused.
FastText surrounds the word with boundary markers before extracting character n-grams.
play, plays, and playful share pieces. A rare or out-of-vocabulary spelling can borrow signal from known character n-grams.vector("playful") ≈ vector(<playful>)
+ Σ vector(character n-gram)The original FastText formulation combines the word vector with its subword vectors. Exact implementation details depend on the training setup.
Same question,
different strengths.
Pick the property you care about. See what each approach contributes — and what it does not give you for free.
Can you explain the mechanism?
Answer the eight checks. You can reset and try again.