CAS NLP SERIES← SERIES

LECTURE 06 · FROM TEXT PREPROCESSING TO LEARNED MEANING

Words become
coordinates.

An embedding is a learned lookup table: give it a token ID, get back a dense vector. The interesting question is how those rows get their geometry. Meet the Keras layer that stores them, then compare Word2Vec, GloVe, and FastText — three ways to teach words where to live.

OPEN THE EMBEDDING LAB ↓
01 Keras lookup layer03 vector learning methods05 interactive labs
A TINY LEARNED SPACE 2D PROJECTION
king
queen
prince
apple
nearby ≈ similar context
illustrative coordinates
THE FOUR LENSES

Same output shape.
Different learning stories.

01 / KERAS LAYER

A matrix with one row per token.

Token IDs are addresses, not magnitudes. The Keras Embedding layer uses each integer to select a trainable row from a matrix of shape (vocabulary size, embedding dimension).

BEFORE THE LAB

Think of a numbered shelf of word cards

The word cat becomes an ID, such as 7. The ID is only an address. Row 7 holds a short list of learned numbers; nearby values emerge through training, not from the number 7 itself.

01 · Give every word an address

cat → 7. IDs keep the input compact, but ID 7 is not “more cat” than ID 6.

02 · Look up a row

If E has shape (12, 4), E[7] returns four numbers. A six-word sentence becomes a 6 × 4 grid.

03 · Let the task edit the rows

The sentiment classifier makes a prediction, computes loss, and backpropagation adjusts the rows used by the sentence. Keras Embedding stores these trainable rows.

Try it: change the sentence below and click a token. Watch which table row is selected; then change the dimension to see the output shape change.

This demo uses a deterministic teaching matrix. A real Keras layer starts with initialized weights and learns them from the task loss.

EMBEDDING TABLE shape: 12 × 4LIVE LOOKUP
SELECTED OUTPUT
Read the operationoutput = E[token_id] — no multiplication by a one-hot vector is needed in practice; it is the same row selection computed efficiently.
COPY · TRAIN · EVALUATE

Keras Embedding: complete IMDb sentiment lab

IMDb is already integer-tokenized by Keras. The script pads/truncates sequences, splits the official training set, learns an Embedding layer, uses early stopping, evaluates the untouched test set, prints precision/recall/F1 and ROC AUC, and saves a confusion matrix.

DOWNLOAD .PY ↓
pip install tensorflow keras numpy scikit-learn matplotlib
python keras_embedding_imdb.py
01 · padding + split02 · fit learned vectors03 · held-out test04 · confusion matrix
SHOW COMPLETE RUNNABLE SCRIPT ↕
"""Train and evaluate a Keras Embedding sentiment model on IMDb.
Run: python keras_embedding_imdb.py
Install: pip install tensorflow keras numpy scikit-learn matplotlib
Keras downloads its integer-tokenized IMDb dataset on first run.
"""
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
import numpy as np
import keras
from keras import layers
from keras.datasets import imdb
from keras.utils import pad_sequences
from sklearn.metrics import (ConfusionMatrixDisplay, classification_report,
                             confusion_matrix, roc_auc_score)
from sklearn.model_selection import train_test_split

SEED = 42
VOCAB = 12000
LENGTH = 220
keras.utils.set_random_seed(SEED)

# IMDb ships as token IDs; 0 is padding, 1 is start, 2 is OOV.
(train_sequences, train_labels), (test_sequences, test_labels) = imdb.load_data(
    num_words=VOCAB, seed=SEED)
X_all = pad_sequences(train_sequences, maxlen=LENGTH, padding="post",
                      truncating="pre")
X_test = pad_sequences(test_sequences, maxlen=LENGTH, padding="post",
                       truncating="pre")
y_all = np.asarray(train_labels, dtype="int32")
y_test = np.asarray(test_labels, dtype="int32")

# Split only the official training partition; keep the official test untouched.
train_ids, val_ids = train_test_split(
    np.arange(len(X_all)), test_size=0.2, random_state=SEED,
    stratify=y_all)
X_train, y_train = X_all[train_ids], y_all[train_ids]
X_val, y_val = X_all[val_ids], y_all[val_ids]
print("train / validation / test:", len(X_train), len(X_val), len(X_test))

model = keras.Sequential([
    keras.Input(shape=(LENGTH,), dtype="int32"),
    layers.Embedding(input_dim=VOCAB, output_dim=64, mask_zero=True,
                     name="learned_word_vectors"),
    layers.GlobalAveragePooling1D(),
    layers.Dense(64, activation="relu"),
    layers.Dropout(0.3),
    layers.Dense(1, activation="sigmoid"),
])
model.compile(optimizer=keras.optimizers.Adam(1e-3),
              loss="binary_crossentropy",
              metrics=[keras.metrics.BinaryAccuracy(name="accuracy"),
                       keras.metrics.AUC(name="auc")])
model.summary()
model.fit(X_train, y_train, validation_data=(X_val, y_val),
          batch_size=128, epochs=15, verbose=2,
          callbacks=[keras.callbacks.EarlyStopping(
              monitor="val_loss", patience=2, restore_best_weights=True)])
print("Untouched test metrics:", model.evaluate(X_test, y_test,
                                                return_dict=True, verbose=0))

probabilities = model.predict(X_test, batch_size=256, verbose=0).ravel()
predictions = (probabilities >= 0.5).astype("int32")
print(classification_report(y_test, predictions,
                            target_names=["negative", "positive"],
                            zero_division=0))
print("Test ROC AUC:", round(roc_auc_score(y_test, probabilities), 4))
cm = confusion_matrix(y_test, predictions, labels=[0, 1])
ConfusionMatrixDisplay(cm, display_labels=["negative", "positive"]).plot(
    cmap="Blues", colorbar=False)
plt.title("IMDb held-out test · Keras Embedding")
plt.tight_layout()
plt.savefig("embedding_imdb_confusion.png", dpi=160)
plt.close()

# Inspect learned rows after training; ID 0 is the padding row.
weights = model.get_layer("learned_word_vectors").get_weights()[0]
print("Embedding matrix:", weights.shape)
print("Example learned vector for token ID 10:", np.round(weights[10, :8], 3))
print("First 5 test probabilities:", np.round(probabilities[:5], 3))
model.save("embedding_imdb.keras")
print("Saved embedding_imdb.keras and embedding_imdb_confusion.png")
02 / WORD2VEC

Teach a word by asking
who stands near it.

Word2Vec learns vectors from local context prediction. Choose the direction: predict the center word from its neighbors (CBOW), or predict surrounding words from the center (Skip-gram).

BEFORE THE LAB

Learn a word from the company it keeps

Imagine repeatedly covering one word in a sentence. To recover it, a learner must notice that cat and dog often appear beside similar words. Their vectors become useful because they help solve the same prediction task.

01 · Slide a window

In “the cat sat near the window,” a window of 1 around cat sees the and sat. Each position creates a new training example.

02 · Choose prediction direction

Skip-gram: cat → the, sat. CBOW: the + sat → cat. Both learn weights by making context predictions.

03 · Keep the learned vectors

After many examples, the prediction weights carry useful word relationships. The highlighted pairs below are inputs to training, not proof that vectors have already learned.

Try it: switch Skip-gram and CBOW, then widen the window. Say aloud what is the input and what is the label each time.

Skip-gramcenter → context words. Often useful for learning representations of less frequent words, given enough training.
CBOWcontext words → center. Combines neighbors and predicts the missing middle token.

These highlighted pairs are the training signal, not trained vectors. A real Word2Vec run optimizes prediction weights over many corpus windows (often with negative sampling).

03 / GLOVE

Count the whole corpus.
Fit the relationships.

GloVe starts with a word–word co-occurrence matrix. Counts summarize how often words appear near each other across the corpus; training fits vector dot products to the logarithm of those counts, with a weighting function.

BEFORE THE LAB

Replace millions of windows with a count map

Word2Vec learns while walking through examples. GloVe first counts how often each word appears near every other word across the corpus. Its training then asks vectors to explain those counts.

01 · Fill the count table

Every nearby pair increments a cell X[i,j]. Rows summarize each word’s surroundings over the whole corpus.

02 · Compare patterns, not raw IDs

If cat and dog both co-occur with pet and plays, their rows share a pattern. A distinctive neighbor such as bark still separates them.

03 · Fit vectors to log counts

GloVe adjusts word and context vectors so their dot product plus biases approximates log(X[i,j]). Weighting keeps extremely common pairs from dominating.

Try it: read the cat and dog rows below. Which columns are shared, and which column distinguishes them?

CO-OCCURRENCE COUNTS toy corpus · window = 2GLOBAL VIEW

Find the pattern. Rows for cat and dog share contexts like pet and plays; dog has extra signal from bark.

wᵢ · w̃ⱼ + bᵢ + b̃ⱼ ≈ log(Xᵢⱼ)
Global vs localWord2Vec creates examples from moving windows; GloVe aggregates them into global counts first. Both learn dense vectors from co-occurrence information.

This fixed count grid is a small, hand-built illustration. Real GloVe training uses large corpora and a weighted least-squares objective; co-occurrence counts alone are not the final vectors.

GLOVE · FROM COUNTS TO EVALUATION

Train GloVe vectors, then test them in a classifier

This Keras file builds a sparse co-occurrence table from training reviews only, fits the weighted log-count objective, checks held-out co-occurrence error and nearest neighbors, then freezes the vectors for IMDb sentiment classification. The classifier produces test accuracy, AUC, precision/recall/F1, and a confusion matrix.

DOWNLOAD .PY ↓
pip install tensorflow keras numpy scikit-learn matplotlib
python keras_glove_imdb.py
01 · training-only counts02 · weighted GloVe loss03 · vector neighbors04 · downstream confusion matrix

GloVe itself has no class labels, so a confusion matrix belongs to the downstream sentiment classifier. This bounded educational run is not a benchmark for large pretrained GloVe vectors.

SHOW COMPLETE RUNNABLE SCRIPT ↕
"""Train small GloVe vectors, then test them in an IMDb classifier.
Run: python keras_glove_imdb.py
Install: pip install tensorflow keras numpy scikit-learn matplotlib
This teaching run uses a subset of TRAINING reviews for co-occurrence counts.
"""
from collections import Counter

import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
import numpy as np
import keras
from keras import layers, ops
from keras.datasets import imdb
from keras.utils import pad_sequences
from sklearn.metrics import ConfusionMatrixDisplay, classification_report, confusion_matrix
from sklearn.model_selection import train_test_split

SEED = 42
VOCAB = 3000
DIM = 50
LENGTH = 160
WINDOW = 2
keras.utils.set_random_seed(SEED)

(train_reviews, train_labels), (test_reviews, test_labels) = imdb.load_data(
    num_words=VOCAB, seed=SEED)
train_ids, val_ids = train_test_split(
    np.arange(len(train_reviews)), test_size=5000, random_state=SEED,
    stratify=train_labels)
X_train = pad_sequences([train_reviews[i] for i in train_ids], maxlen=LENGTH,
                        padding="post", truncating="pre")
X_val = pad_sequences([train_reviews[i] for i in val_ids], maxlen=LENGTH,
                      padding="post", truncating="pre")
X_test = pad_sequences(test_reviews, maxlen=LENGTH, padding="post",
                       truncating="pre")
y_train = np.asarray(train_labels)[train_ids]
y_val = np.asarray(train_labels)[val_ids]
y_test = np.asarray(test_labels)

# Count local context across training text only. IDs 0, 1, 2 are special.
counts = Counter()
for review in X_train[:1200]:  # bounded so the first run stays approachable
    words = [int(w) for w in review if 2 < w < VOCAB]
    for position, center in enumerate(words):
        for offset in range(-WINDOW, WINDOW + 1):
            neighbor = position + offset
            if offset and 0 <= neighbor < len(words):
                counts[(center, words[neighbor])] += 1 / abs(offset)
print("Nonzero co-occurrence cells:", len(counts))

# X_ij is sparse: train only on observed pairs, never on zero counts.
items = counts.most_common(120000)
ids_i = np.asarray([pair[0][0] for pair in items], dtype="int32")
ids_j = np.asarray([pair[0][1] for pair in items], dtype="int32")
values = np.asarray([pair[1] for pair in items], dtype="float32")
target_log_count = np.log(values).reshape(-1, 1)
weights = np.minimum((values / 100.0) ** 0.75, 1.0)
perm = np.random.default_rng(SEED).permutation(len(values))
cut = max(1, round(0.1 * len(values)))
fit_ids, holdout_ids = perm[cut:], perm[:cut]

word_id = keras.Input(shape=(), dtype="int32")
context_id = keras.Input(shape=(), dtype="int32")
word_vectors = layers.Embedding(VOCAB, DIM, name="word_vectors")
context_vectors = layers.Embedding(VOCAB, DIM, name="context_vectors")
word_bias = layers.Embedding(VOCAB, 1, name="word_bias")
context_bias = layers.Embedding(VOCAB, 1, name="context_bias")
score = (ops.sum(word_vectors(word_id) * context_vectors(context_id),
                 axis=-1, keepdims=True)
         + word_bias(word_id) + context_bias(context_id))
glove = keras.Model([word_id, context_id], score)
glove.compile(optimizer=keras.optimizers.Adam(0.01), loss="mse")
glove.fit([ids_i[fit_ids], ids_j[fit_ids]], target_log_count[fit_ids],
          sample_weight=weights[fit_ids], batch_size=4096, epochs=12,
          validation_data=([ids_i[holdout_ids], ids_j[holdout_ids]],
                           target_log_count[holdout_ids], weights[holdout_ids]),
          callbacks=[keras.callbacks.EarlyStopping(
              monitor="val_loss", patience=2, restore_best_weights=True)], verbose=2)
print("Held-out observed-pair weighted MSE:", glove.evaluate(
    [ids_i[holdout_ids], ids_j[holdout_ids]], target_log_count[holdout_ids],
    sample_weight=weights[holdout_ids], verbose=0))

# GloVe commonly combines the two learned vector tables.
embeddings = (word_vectors.get_weights()[0] +
              context_vectors.get_weights()[0]).astype("float32")
embeddings[0] = 0  # padding stays inert downstream
word_index = imdb.get_word_index()
id_to_word = {index + 3: word for word, index in word_index.items()
              if index + 3 < VOCAB}
normalized = embeddings / np.maximum(np.linalg.norm(embeddings, axis=1,
                                                   keepdims=True), 1e-8)
for query in ("great", "bad", "movie"):
    token_id = word_index.get(query, -99) + 3
    if token_id not in id_to_word:
        continue
    similarity = normalized @ normalized[token_id]
    neighbors = [id_to_word[i] for i in np.argsort(-similarity)
                 if i in id_to_word and i != token_id][:5]
    print(f"Nearest to {query}: {neighbors}")

# Extrinsic evaluation: freeze GloVe and train a sentiment classifier.
classifier = keras.Sequential([
    keras.Input(shape=(LENGTH,), dtype="int32"),
    layers.Embedding(VOCAB, DIM, weights=[embeddings], trainable=False,
                     mask_zero=True, name="frozen_glove"),
    layers.GlobalAveragePooling1D(),
    layers.Dense(64, activation="relu"),
    layers.Dropout(0.3),
    layers.Dense(1, activation="sigmoid"),
])
classifier.compile(optimizer="adam", loss="binary_crossentropy",
                   metrics=[keras.metrics.BinaryAccuracy(name="accuracy"),
                            keras.metrics.AUC(name="auc")])
classifier.fit(X_train, y_train, validation_data=(X_val, y_val),
               batch_size=128, epochs=12, verbose=2,
               callbacks=[keras.callbacks.EarlyStopping(
                   monitor="val_loss", patience=2, restore_best_weights=True)])
print("Untouched sentiment test:", classifier.evaluate(X_test, y_test,
                                                         return_dict=True, verbose=0))
probabilities = classifier.predict(X_test, batch_size=256, verbose=0).ravel()
predictions = (probabilities >= 0.5).astype("int32")
print(classification_report(y_test, predictions,
                            target_names=["negative", "positive"],
                            zero_division=0))
cm = confusion_matrix(y_test, predictions, labels=[0, 1])
ConfusionMatrixDisplay(cm, display_labels=["negative", "positive"]).plot(
    cmap="Blues", colorbar=False)
plt.title("IMDb test · frozen GloVe vectors")
plt.tight_layout()
plt.savefig("glove_imdb_confusion.png", dpi=160)
plt.close()
glove.save("glove_cooccurrence.keras")
classifier.save("glove_imdb_classifier.keras")
print("Saved both .keras models and glove_imdb_confusion.png")
04 / FASTTEXT

A word is also its parts.

FastText represents a word using character n-grams as well as a whole-word vector. Related spellings can share subword parameters, and unseen forms can still be represented by their n-grams.

BEFORE THE LAB

Give a word reusable spelling pieces

A single row per word cannot help much when a spelling never appeared in training. FastText learns vectors for character fragments, so a new word can borrow information from fragments seen inside other words.

01 · Mark the boundaries

Wrap playful as <playful> so pieces at the start and end are distinguishable.

02 · Extract character n-grams

For length 3, fragments include <pl, pla, and lay. play and playful share several fragments.

03 · Combine learned pieces

FastText sums the word vector and its n-gram vectors. For an unseen word, known n-gram vectors can still produce a representation; quality depends on what the corpus taught them.

Try it: type playfully and compare its pieces with playful; then type a made-up word and see what can be reused.

FastText surrounds the word with boundary markers before extracting character n-grams.

Why it helpsplay, plays, and playful share pieces. A rare or out-of-vocabulary spelling can borrow signal from known character n-grams.
CONCEPTUAL COMPOSITION
vector("playful") ≈ vector(<playful>)
                 + Σ vector(character n-gram)

The original FastText formulation combines the word vector with its subword vectors. Exact implementation details depend on the training setup.

05 / CHOOSE A LENS

Same question,
different strengths.

Pick the property you care about. See what each approach contributes — and what it does not give you for free.

One important boundaryThese are static word representations: one token type has one learned vector. Contextual models such as transformers produce representations that can change with the sentence.
06 / MASTERY CHECK

Can you explain the mechanism?

Answer the eight checks. You can reset and try again.

SCORE 0 / 8