⌗ TEXT PREPROCESSING — Raw Text Made Model-Ready
Text Preprocessing
Models do not read text — they read numbers. Everything between the wild sentence a user types and the clean token stream a model eats is preprocessing: cleaning, tokenizing, normalizing, featurizing. This lecture walks the whole pipeline — and teaches the one rule that decides every step: remove only what your task cannot use.
ONE REVIEW, SIX LIVES
“I REALLY loved this movie!!! 😍 It wasn’t boring at all. Visit https://abc.com” — straight from the wild. Watch it flow through every stage of the pipeline and notice what each stage keeps, what it throws away, and what it would have destroyed had we been careless.
Every transform below is computed live in JavaScript — a real Porter stemmer, a real lemmatizer, a real tokenizer. Edit anything, break everything.
THE PIPELINE, LIVE
click any stageBefore any algorithm, a knife: what can this task not use? Cut that first — and nothing else.
02 TEXT CLEANING
Cut what the task cannot use
Cleaning removes information that is usually irrelevant: extra spaces, HTML tags, URLs, email addresses, stray symbols — sometimes punctuation. Toggle each operation and watch the same messy review get cleaned, piece by piece.
OPERATIONS — each one is a bet
AFTER CLEANING
THE HTML LINE EVERY PIPELINE USES
Web data arrives wrapped in markup — <p>This movie was <b>excellent</b>.</p>. BeautifulSoup strips it to This movie was excellent. in one call:
Whitespace gets the same treatment with plain Python: text = " ".join(text.split()) — NLP is very interesting becomes NLP is very interesting.
The cheapest normalization in all of NLP is one call: .lower(). It merges three words into one — and sometimes merges two meanings into one.
03 LOWERCASING
Cheap compression, real risk
Lowercasing collapses Apple, APPLE and apple into one type — three rows in the vocabulary become one. Models train on less data per feature. Watch the merge happen live, then meet the words it should never have touched.
OUTPUT
MERGED GROUPS — same word, different surfaces
The second merge is the wound: US (the country) and us (you and me) are now the same token. Capitalization was information. In entity work, apple the fruit and Apple the company meet the same fate.
Clean text is still one long string. Models eat sequences. Tokenization is the act of chopping — and the size of the chop is one of the biggest design decisions in NLP.
04 TOKENIZATION
Words, sentences, subwords
A token is a unit of text — the atom your model will actually see. Word tokenization breaks on words, sentence tokenization on sentences, and modern models split words into subword pieces so nothing is ever out-of-vocabulary. All three run live below.
WHY MODERN MODELS SPLIT THE WORD
A word-level vocabulary can never cover every word: misspellings, brand names, new slang — all become <UNK> and lose their meaning. Subword tokenizers (BPE, WordPiece, SentencePiece) decompose rare words into known pieces instead:
## marks a piece that continues a word — WordPiece notation, the style BERT uses. BERT, GPT and every modern LLM tokenize this way, not with text.split().
Next two cuts seem trivial: delete punctuation, delete tiny words. Each one has quietly destroyed a sentiment model somewhere. Watch them work — and misbehave.
05 PUNCTUATION
Signal, not noise
Traditional pipelines strip punctuation with one line of Python. But Great., Great!!! and Great? are three different sentences — statement, shout, doubt. Type below and watch a transparent toy model read the marks as intensity.
WITH PUNCTUATION
STRIPPED VIEW · —
THE ONE-LINE STRIP — AND WHEN TO USE IT
Reasonable: topic classification, document clustering, search indexes — punctuation rarely carries topic. Reckless: sentiment, emotion, dialogue analysis, authorship attribution — and any text heading into an LLM, which was trained on natural punctuation.
Stopwords — the, is, a, of, to — fill maybe 40% of any English text while carrying almost no topic. Removing them is the classic win. Unless the meaning lives in one of them.
06 STOPWORDS
Small words, big consequences
Stopwords are the very common words — the, is, a, of, in, to, and. Removing them shrinks Bag-of-Words and TF-IDF features dramatically. Toggle removal below — and watch what happens when not lands on the wrong side of the filter.
TOKENS · —
teal content words · grey stopwords · rose negators
THE CLASSIC NLTK PATTERN
clean = [w for w in words if w not in stop_words]
“The cat is sitting on the table” → ['cat', 'table']. This powered decades of BoW, TF-IDF, Naive Bayes and information retrieval — less noise, smaller vectors, faster training. The lesson of the demo stands: subtract the negators from the stop list when meaning matters, and skip the step entirely for transformers.
playing, played, plays, player — four vocabulary rows, one concept. Stemming merges them with an axe. Lemmatization does it with a scalpel. Two sections, one goal.
07 STEMMING
Chopping endings, mechanically
A stemmer strips prefixes and suffixes by rules, not by knowledge — think mechanically chopping word endings. It is fast and crude, and it does not care whether the result is a real word. Below is a genuine Porter stemmer running in your browser.
Read the surprises: studies → studi and relational → relat are not words — and Porter does not care. player survives whole because chopping -er would leave a stem too small (the algorithm's measure guard). And was → wa: function words get butchered — one reason stemming always follows stopword removal.
IN CODE — THREE LINES OF NLTK
stemmer.stem("playing") # → play
stemmer.stem("studies") # → studi
The intuition to keep: stemming trades linguistic correctness for speed and vocabulary compression. For counting features (BoW, TF-IDF), studi is perfectly fine. For anything a human reads — or a language model consumes — it is not.
The axe leaves splinters: studi. Lemmatization asks a different question — not “what can I cut?” but “what is the dictionary form of this word?” — and answers with real words.
08 LEMMATIZATION
The dictionary base form
A lemma is the word you would look up: cars → car, studies → study, better → good. Unlike a stemmer, a lemmatizer uses vocabulary and grammar — which means the same word can have different lemmas depending on its part of speech. Try it:
POS PLAYGROUND — the same word, three lemmas
STEM vs LEMMA — computed live, same words, different philosophies
| Property | Stemming | Lemmatization |
|---|---|---|
| Method | Rule / chopping based | Linguistically informed — dictionary + POS |
| Speed | Faster | Usually slower |
| Output | May create invalid words — studies → studi | Always dictionary words — studies → study |
| Irregular forms | Blind — geese → geese | Handled — geese → goose, better → good |
Cleaned, cased, tokenized, stemmed — and still the text says @user #Wow https://abc.com 😍 soooo good!!!. Social media has its own species of noise, and its own toolbox.
09 THE SOCIAL-MEDIA TOOLBOX
Contractions, URLs, hashtags, emoji — normalized, not deleted
Real text is full of structured noise: contractions hiding negations, URLs, emails, @mentions, #hashtags, emoji that are the sentiment, stretched letters, typos. Nine normalizations, one tweet — every switch computed live.
NORMALIZATIONS — toggle each one
OUTPUT · —
<URL> <EMAIL> <USER> <NUMBER> — replacing preserves the information that something was here, which deletion throws away.
#MachineLearning carries a whole topic — split it into Machine Learning, never delete it.:smiling_face_with_heart_eyes:) rather than strip.One tiny word deserves its own section — because it can invert your entire model's output while looking completely harmless.
10 NEGATION
The word that flips everything
A Bag-of-Words model scores “The movie is not good” mostly by the word good — the not is just another feature sitting beside it. The classic fix: mark the negation scope so not good becomes a feature of its own. Toggle the two repairs below and watch the verdict flip.
FEATURES THE MODEL SEES
The toy sentiment meter is a transparent lexicon — you can read every weight it applies. Real BoW models learn weights from data; what they cannot do is see a phrase. Marking and bigrams give the phrase a fighting chance.
Some junk has no letters at all: zero-width characters, look-alike alphabets, ligatures, non-breaking spaces. You cannot fix what you cannot see — so let's see it.
11 UNICODE
The characters you cannot see
Visually identical characters can be encoded differently — ligatures (fi), full-width letters (NLP), circled numbers (①), non-breaking spaces. Unicode normalization (NFKC) folds them into one canonical form. Your browser does this natively — watch it:
OUTPUT · —
WHAT CHANGED — character by character, with codepoints
ONE LINE, AND WHY MULTILINGUAL NLP NEEDS IT
Without it, file and file are different tokens; searching NLP misses NLP; the same German word with composed vs decomposed accents occupies two vocabulary rows. The Greek ο and ω in the demo are the deliberate exception: they are different letters that merely look like Latin ones — NFKC (correctly) leaves them, and confusable-letter attacks need separate handling.
Every step so far treated words as opaque bags. Two classic annotations look inside the sentence — what role each word plays, and which words name things.
12 POS & NER
Knowing what words are
POS tagging labels each word's grammatical role — The → DET · cat → NOUN · eats → VERB. Named-entity recognition finds the things: Elon Musk → PERSON, United States → LOCATION. Together they sharpen lemmatization and power anonymization. A tiny lexicon-driven tagger runs live:
TOKENS + TAGS
ENTITIES
SENTENCE OUT
The ambiguity demo: “The working prototype was working at Google.” — same word, two tags, two different lemmas downstream. This is exactly why §08 lemmatizers accept a pos argument. Try “ambiguous” in the presets.
You now own every tool. The final question is architectural: which tools, in which order — and the answer depends entirely on which era of model is waiting at the end of the pipe.
13 TWO PIPELINES
Traditional vs Transformer
For classical ML you assemble the full line: lowercase → strip → tokenize → filter → stem → vectorize. For BERT, GPT and friends, the pretrained tokenizer does nearly everything, and hand-processing hurts. Same probe sentence, two philosophies, both live:
THE STEPS — toggle and watch the features change
FINAL FEATURES
A PADDED BATCH — toy WordPiece IDs computed live
ATTENTION MASKS — 1 = real token · 0 = padding, never attended
So which operations survive which task? Here is the whole lecture compressed into one decision matrix — choose a task, read your pipeline.
14 THE GOLDEN RULE
Preprocessing is task-dependent
There is no universal clean_text(). Pick a task below and the matrix tells you which of the lecture's operations help, which need care, and which will quietly destroy signal.
Text Preprocessing is the gate at the very start of the NLP ladder: Preprocessing → Tokenization → One-Hot → BoW → N-grams → TF-IDF → Word2Vec → GloVe → FastText → contextual embeddings. Lecture 04 covered the middle rungs; this lecture owns the gate.
Everything before the tokenizer serves classical ML; everything after belongs to the model's own tokenizer. When you switch eras, re-decide every step — do not copy old pipelines forward.
The honest way to decide: train with the step, train without it, compare. Preprocessing choices are empirical questions — the matrix above is prior knowledge, not a substitute for measurement.
15 CODE LAB
Every pipeline, in runnable Python
Four tabs: the classic NLTK line with negators protected, the regex toolbox for social-media text, the modern transformers way — where the tokenizer does the work — and Keras, where preprocessing becomes a TextVectorization layer that lives inside the model.
16 MASTERY CHECK