CAS logo CAS NLP LABInteractive Tutorial Series ← Series

⌗  TEXT PREPROCESSING — Raw Text Made Model-Ready

Text Preprocessing

Models do not read text — they read numbers. Everything between the wild sentence a user types and the clean token stream a model eats is preprocessing: cleaning, tokenizing, normalizing, featurizing. This lecture walks the whole pipeline — and teaches the one rule that decides every step: remove only what your task cannot use.

RAW TEXT→ CLEANING→ TOKENIZATION→ NORMALIZATION→ FEATURES→ MODEL
21Operationsevery classic step, live
16Sectionsintuition → math → code
15Live Demostype, toggle, click — nothing is a picture
8Quiz Checksone per pipeline stage
THE ONE-SENTENCE SUMMARY

Text preprocessing cleans and transforms raw text into a form an NLP model can understand — but there is no universal clean_text(). For sentiment, NOT, !!! and 😍 are the signal. For topic classification they may be noise. The art is knowing which is which — and this page makes you touch every decision.

ONE REVIEW, SIX LIVES

“I REALLY loved this movie!!! 😍 It wasn’t boring at all. Visit https://abc.com” — straight from the wild. Watch it flow through every stage of the pipeline and notice what each stage keeps, what it throws away, and what it would have destroyed had we been careless.

Every transform below is computed live in JavaScript — a real Porter stemmer, a real lemmatizer, a real tokenizer. Edit anything, break everything.

START WITH CLEANING ↓

THE PIPELINE, LIVE

click any stage

Before any algorithm, a knife: what can this task not use? Cut that first — and nothing else.

02 TEXT CLEANING

Cut what the task cannot use

Cleaning removes information that is usually irrelevant: extra spaces, HTML tags, URLs, email addresses, stray symbols — sometimes punctuation. Toggle each operation and watch the same messy review get cleaned, piece by piece.

OPERATIONS — each one is a bet

AFTER CLEANING

0characters before
0characters after
0%of the text removed

THE HTML LINE EVERY PIPELINE USES

Web data arrives wrapped in markup — <p>This movie was <b>excellent</b>.</p>. BeautifulSoup strips it to This movie was excellent. in one call:

BeautifulSoup(html, "html.parser").get_text()

Whitespace gets the same treatment with plain Python: text = " ".join(text.split()) — NLP    is  very interesting becomes NLP is very interesting.

But cleaning is task-dependent. For sentiment, good and good!!! carry different emotional intensity — stripping the marks loses it. And I like it. vs I don't like it. differ by four characters that flip the meaning. Remove only what you are confident is irrelevant — we will build that judgment section by section.

The cheapest normalization in all of NLP is one call: .lower(). It merges three words into one — and sometimes merges two meanings into one.

03 LOWERCASING

Cheap compression, real risk

Lowercasing collapses Apple, APPLE and apple into one type — three rows in the vocabulary become one. Models train on less data per feature. Watch the merge happen live, then meet the words it should never have touched.

OUTPUT

0unique types before
0unique types after
0vocabulary rows saved

MERGED GROUPS — same word, different surfaces

The second merge is the wound: US (the country) and us (you and me) are now the same token. Capitalization was information. In entity work, apple the fruit and Apple the company meet the same fate.

The modern rule: don't decide by hand — follow the pretrained tokenizer. Models like bert-base-uncased expect lowercase (and lowercase for you); cased models expect original case. Match your preprocessing to the checkpoint, not to habit.

Clean text is still one long string. Models eat sequences. Tokenization is the act of chopping — and the size of the chop is one of the biggest design decisions in NLP.

04 TOKENIZATION

Words, sentences, subwords

A token is a unit of text — the atom your model will actually see. Word tokenization breaks on words, sentence tokenization on sentences, and modern models split words into subword pieces so nothing is ever out-of-vocabulary. All three run live below.

WHY MODERN MODELS SPLIT THE WORD

A word-level vocabulary can never cover every word: misspellings, brand names, new slang — all become <UNK> and lose their meaning. Subword tokenizers (BPE, WordPiece, SentencePiece) decompose rare words into known pieces instead:

unbelievable → un + ##believ + ##able  ·  playing → play + ##ing

## marks a piece that continues a word — WordPiece notation, the style BERT uses. BERT, GPT and every modern LLM tokenize this way, not with text.split().

The design tension: words give meaning but explode the vocabulary and strand rare words; characters give coverage but make sequences long and meaning scarce. Subwords are the treaty between the two — which is why the field moved to them.

Next two cuts seem trivial: delete punctuation, delete tiny words. Each one has quietly destroyed a sentiment model somewhere. Watch them work — and misbehave.

05 PUNCTUATION

Signal, not noise

Traditional pipelines strip punctuation with one line of Python. But Great., Great!!! and Great? are three different sentences — statement, shout, doubt. Type below and watch a transparent toy model read the marks as intensity.

WITH PUNCTUATION

STRIPPED VIEW · —

THE ONE-LINE STRIP — AND WHEN TO USE IT

text.translate(str.maketrans("", "", string.punctuation))

Reasonable: topic classification, document clustering, search indexes — punctuation rarely carries topic. Reckless: sentiment, emotion, dialogue analysis, authorship attribution — and any text heading into an LLM, which was trained on natural punctuation.

Do not remove punctuation blindly. Great? questions what Great. asserts; no!!! is not the same word as no. The marks are among the cheapest sentiment features that exist.

Stopwords — the, is, a, of, to — fill maybe 40% of any English text while carrying almost no topic. Removing them is the classic win. Unless the meaning lives in one of them.

06 STOPWORDS

Small words, big consequences

Stopwords are the very common words — the, is, a, of, in, to, and. Removing them shrinks Bag-of-Words and TF-IDF features dramatically. Toggle removal below — and watch what happens when not lands on the wrong side of the filter.

TOKENS · —

teal content words · grey stopwords · rose negators

THE CLASSIC NLTK PATTERN

stop_words = set(stopwords.words("english"))
clean = [w for w in words if w not in stop_words]

“The cat is sitting on the table” → ['cat', 'table']. This powered decades of BoW, TF-IDF, Naive Bayes and information retrieval — less noise, smaller vectors, faster training. The lesson of the demo stands: subtract the negators from the stop list when meaning matters, and skip the step entirely for transformers.

playing, played, plays, player — four vocabulary rows, one concept. Stemming merges them with an axe. Lemmatization does it with a scalpel. Two sections, one goal.

07 STEMMING

Chopping endings, mechanically

A stemmer strips prefixes and suffixes by rules, not by knowledge — think mechanically chopping word endings. It is fast and crude, and it does not care whether the result is a real word. Below is a genuine Porter stemmer running in your browser.

PORTER STEM …

Read the surprises: studies → studi and relational → relat are not words — and Porter does not care. player survives whole because chopping -er would leave a stem too small (the algorithm's measure guard). And was → wa: function words get butchered — one reason stemming always follows stopword removal.

IN CODE — THREE LINES OF NLTK

stemmer = PorterStemmer()
stemmer.stem("playing")  # → play
stemmer.stem("studies")  # → studi

The intuition to keep: stemming trades linguistic correctness for speed and vocabulary compression. For counting features (BoW, TF-IDF), studi is perfectly fine. For anything a human reads — or a language model consumes — it is not.

The axe leaves splinters: studi. Lemmatization asks a different question — not “what can I cut?” but “what is the dictionary form of this word?” — and answers with real words.

08 LEMMATIZATION

The dictionary base form

A lemma is the word you would look up: cars → car, studies → study, better → good. Unlike a stemmer, a lemmatizer uses vocabulary and grammar — which means the same word can have different lemmas depending on its part of speech. Try it:

POS PLAYGROUND — the same word, three lemmas

STEM vs LEMMA — computed live, same words, different philosophies

wordporter stemlemma · nounlemma · verbnote
PropertyStemmingLemmatization
MethodRule / chopping basedLinguistically informed — dictionary + POS
SpeedFasterUsually slower
OutputMay create invalid words — studies → studiAlways dictionary words — studies → study
Irregular formsBlind — geese → geeseHandled — geese → goose, better → good

POS makes it precise. working as a verb lemmatizes to work; as a noun it stays working. That is why real pipelines run POS tagging (§12) before lemmatization — and why NLTK's lemmatize(word, pos=...) takes a part of speech at all.

Cleaned, cased, tokenized, stemmed — and still the text says @user #Wow https://abc.com 😍 soooo good!!!. Social media has its own species of noise, and its own toolbox.

09 THE SOCIAL-MEDIA TOOLBOX

Contractions, URLs, hashtags, emoji — normalized, not deleted

Real text is full of structured noise: contractions hiding negations, URLs, emails, @mentions, #hashtags, emoji that are the sentiment, stretched letters, typos. Nine normalizations, one tweet — every switch computed live.

NORMALIZATIONS — toggle each one

OUTPUT · —

<URL> <EMAIL> <USER> <NUMBER> — replacing preserves the information that something was here, which deletion throws away.

#Hashtags#MachineLearning carries a whole topic — split it into Machine Learning, never delete it.
😍 EmojiOften the strongest sentiment token in the post — map to words (:smiling_face_with_heart_eyes:) rather than strip.
soooo → sooRepeated letters encode intensity; collapse to a bounded form and the emotion survives in a countable token.
recieve → receiveSpelling correction de-fragments the vocabulary — but must not touch product names, usernames, technical terms or slang.

One tiny word deserves its own section — because it can invert your entire model's output while looking completely harmless.

10 NEGATION

The word that flips everything

A Bag-of-Words model scores “The movie is not good” mostly by the word good — the not is just another feature sitting beside it. The classic fix: mark the negation scope so not good becomes a feature of its own. Toggle the two repairs below and watch the verdict flip.

FEATURES THE MODEL SEES

The toy sentiment meter is a transparent lexicon — you can read every weight it applies. Real BoW models learn weights from data; what they cannot do is see a phrase. Marking and bigrams give the phrase a fighting chance.

Some junk has no letters at all: zero-width characters, look-alike alphabets, ligatures, non-breaking spaces. You cannot fix what you cannot see — so let's see it.

11 UNICODE

The characters you cannot see

Visually identical characters can be encoded differently — ligatures (fi), full-width letters (NLP), circled numbers (①), non-breaking spaces. Unicode normalization (NFKC) folds them into one canonical form. Your browser does this natively — watch it:

OUTPUT · —

WHAT CHANGED — character by character, with codepoints

ONE LINE, AND WHY MULTILINGUAL NLP NEEDS IT

unicodedata.normalize("NFKC", text)

Without it, file and file are different tokens; searching NLP misses NLP; the same German word with composed vs decomposed accents occupies two vocabulary rows. The Greek ο and ω in the demo are the deliberate exception: they are different letters that merely look like Latin ones — NFKC (correctly) leaves them, and confusable-letter attacks need separate handling.

Every step so far treated words as opaque bags. Two classic annotations look inside the sentence — what role each word plays, and which words name things.

12 POS & NER

Knowing what words are

POS tagging labels each word's grammatical role — The → DET · cat → NOUN · eats → VERB. Named-entity recognition finds the things: Elon Musk → PERSON, United States → LOCATION. Together they sharpen lemmatization and power anonymization. A tiny lexicon-driven tagger runs live:

TOKENS + TAGS

ENTITIES

SENTENCE OUT

The ambiguity demo: “The working prototype was working at Google.” — same word, two tags, two different lemmas downstream. This is exactly why §08 lemmatizers accept a pos argument. Try “ambiguous” in the presets.

You now own every tool. The final question is architectural: which tools, in which order — and the answer depends entirely on which era of model is waiting at the end of the pipe.

13 TWO PIPELINES

Traditional vs Transformer

For classical ML you assemble the full line: lowercase → strip → tokenize → filter → stem → vectorize. For BERT, GPT and friends, the pretrained tokenizer does nearly everything, and hand-processing hurts. Same probe sentence, two philosophies, both live:

CLASSIC ERATraditional ML pipelineBoW · TF-IDF · Naive Bayes · SVM
Raw text↓ Lowercase↓ Remove HTML / URLs↓ Tokenize↓ Remove punctuation↓ Stopwords↓ Stem / lemmatize↓ BoW / TF-IDF → ML model

THE STEPS — toggle and watch the features change

FINAL FEATURES

TRANSFORMER ERAPretrained tokenizerBERT · GPT · modern LLMs
Raw text↓ Basic normalization↓ Pretrained tokenizer → subword IDs↓ Attention mask → Transformer

A PADDED BATCH — toy WordPiece IDs computed live

ATTENTION MASKS — 1 = real token · 0 = padding, never attended

Never feed a pretrained transformer stemmed, stopword-stripped, punctuation-stripped text. The model was trained on text exactly as its own tokenizer produces it — “the”, “!”, and word endings are load-bearing. At most: light basic normalization, then trust the tokenizer. In code it is one call: AutoTokenizer.from_pretrained("bert-base-uncased")(text) — out come token IDs and the attention mask.

So which operations survive which task? Here is the whole lecture compressed into one decision matrix — choose a task, read your pipeline.

14 THE GOLDEN RULE

Preprocessing is task-dependent

There is no universal clean_text(). Pick a task below and the matrix tells you which of the lecture's operations help, which need care, and which will quietly destroy signal.

THE RULE THAT SURVIVES EVERY TASK

Remove only information that you are confident is irrelevant to your prediction task.

Every deletion is a bet. Sentiment needs NOT, !!! and 😍. NER needs capitalization. Language models want natural text. When in doubt, keep — and let the model learn what matters.

Where you are on the map

Text Preprocessing is the gate at the very start of the NLP ladder: Preprocessing → Tokenization → One-Hot → BoW → N-grams → TF-IDF → Word2Vec → GloVe → FastText → contextual embeddings. Lecture 04 covered the middle rungs; this lecture owns the gate.

The two-era split

Everything before the tokenizer serves classical ML; everything after belongs to the model's own tokenizer. When you switch eras, re-decide every step — do not copy old pipelines forward.

When in doubt, ablate

The honest way to decide: train with the step, train without it, compare. Preprocessing choices are empirical questions — the matrix above is prior knowledge, not a substitute for measurement.

15 CODE LAB

Every pipeline, in runnable Python

Four tabs: the classic NLTK line with negators protected, the regex toolbox for social-media text, the modern transformers way — where the tokenizer does the work — and Keras, where preprocessing becomes a TextVectorization layer that lives inside the model.

TEXT PREPROCESSING · FOUR WAYS

16 MASTERY CHECK

Eight questions — one per pipeline stage.

YOUR SCORE

0 / 8