A few years ago, people noticed that ChatGPT was strangely bad at counting the letters in a word, or at simple tasks like reversing a string. How could a model that writes working code fail at something a child can do?The answer is buried in a step most people never think about. The model doesn't see letters. It doesn't even see words. Before a single neuron fires, your text gets chopped into chunks called tokens, and the model only ever sees those chunks. "Unbelievable" might arrive as three pieces: un, belie, vable. "Strawberry" might arrive as str, aw, berry which is exactly why the model struggled to count its r's. It never saw individual r's in the first place.Tokenization is the invisible lens through which the model views all language. Get the lens wrong, and everything downstream is blurry.This article is about that lens, and about the unglamorous but critical work of preparing data before any training begins. We'll go slowly, because this is the stage where most real-world projects quietly fail.Tools Used in This ArticleEvery tool we touch in this article, what it does, and who maintains it. Refer back here whenever a name shows up later.tiktoken → OpenAI's fast tokenizer library. It implements the exact tokenization used by GPT-3.5 and GPT-4. We use it to see how a production tokenizer behaves.HuggingFace Transformers → provides model architectures and, for our purposes, ready-made tokenizers for thousands of published models. We use its AutoTokenizer to load and compare tokenizers side by side.HuggingFace Tokenizers → a separate, lower-level library (written in Rust for speed) for training your own tokenizer from scratch. Different thing from Transformers, confusingly similar name.HuggingFace Datasets → loading and transforming large datasets without exhausting your memory. We use it for the data pipeline.SentencePiece → Google's tokenizer library, used by LLaMA and many others. Mentioned for comparison.MinHash (concept) → a technique for estimating how similar two documents are, used to find near-duplicates quickly. Implemented in libraries like datasketch.KenLM → a small, fast language model toolkit built on n-grams (statistics of short fixed-length word sequences), by Kenneth Heafield. We use it to score and filter low-quality text.1. Tokens: The Lego Bricks of LanguageWhat is a token?Let us define it plainly first, then build intuition.A token is a chunk of text that the model treats as one indivisible unit. Crucially, it is not the same thing as a word. A token might be:a whole word → cata piece of a word → un (as in "unusual")a punctuation mark → .a space followed by a word → the (the leading space matters!)or even a single character → xThis matters because the model literally cannot see anything smaller than a token. If the tokenizer hands it straw + berry, the model has no direct way to know there are three r's in there. Same way you can't taste the individual eggs in a finished cake.The Lego analogyThink of tokens as Lego bricks. The complete set of available brick shapes is the vocabulary, the full list of every token the tokenizer knows about. A typical modern vocabulary has between 30,000 and 130,000 tokens. The machine that snaps a stream of text apart into bricks is the tokenizer.The analogy earns its keep because of a real constraint: just like a Lego set has a fixed inventory of pieces, the vocabulary is fixed. If the shape you need isn't in the box, you approximate it by combining smaller pieces, and the result is clunkier. That clunkiness is exactly what hurts a model on rare or specialized words.Why the choice of bricks matters, a MedBot exampleSuppose MedBot's tokenizer was trained mostly on general web text. When it hits "myocardial" (meaning "of the heart muscle"), it might chop it into my, o, card, ial, four pieces that, individually, mean nothing related to the heart. The model now has to learn, from scratch, that these four fragments in this specific order signal "heart muscle."Now suppose instead the tokenizer was trained on medical text and has seen "myocardial" tens of thousands of times. It keeps myocardial as a single token. One clean brick. The model picks up quickly that this brick is a medical adjective that almost always sits right before "infarction."Same model architecture, same training compute, different lens, very different outcome.How many tokens are in a word?A useful rule of thumb for English: 1 token ≈ 0.75 words, or about 4 characters. So:Keep this rule handy, we'll use it constantly to estimate costs and memory.2. Byte Pair Encoding (BPE): How a Tokenizer Learns Its BricksSo where does the vocabulary come from? Nobody sits down and writes a list of 50,000 useful word-pieces by hand. The tokenizer learns its vocabulary from data, using an algorithm called Byte Pair Encoding (BPE), which repeatedly merges the most frequent pair of adjacent symbols in a training corpus into a new, single symbol.BPE is the algorithm behind GPT-2, GPT-3, GPT-4, and LLaMA. It sounds technical, but the idea is something you'd probably invent yourself if you sat with the problem long enough.The intuition first (no math)Imagine you're inventing shorthand for a notebook. You start by writing everything out letter by letter, slow and tedious. After a while you notice you keep writing "th" over and over. So you invent a single squiggle that means "th." Faster.Then you notice the squiggle for "th" is almost always followed by "e." So you invent a new squiggle for "the." Even faster.You keep doing this: find the pair of symbols you use most often, glue it into a new single symbol, repeat. Over time your most common words become single squiggles, while rare words still get spelled out from smaller pieces.That's BPE. The "symbols" start as individual characters, and each merge glues the most frequent adjacent pair into a new symbol.Watch it happen: a complete BPE walkthrough, every single stepWe'll run BPE by hand on a tiny corpus, and this is the important part, we'll show every merge, not just the first few, until nothing useful is left to merge.Our toy corpus is five words, each with a frequency:Step 0 → Initialize. Split every word into individual characters, plus a special end-of-word marker so the algorithm knows where words stop (this prevents it from gluing the end of one word to the start of another):Right now every "brick" is a single character. The vocabulary is just {l, o, w, e, r, n, s, t, i, d, }. Now we start merging.Merge 1 → Count every adjacent pair across the whole corpus, weighted by frequency.The full count:Three-way tie at 8: w e, n e, and e w. BPE breaks ties deterministically. Our rule (here and in the code in Part 7): highest count first, and on a tie, alphabetically first pair, which picks e w.Merge e + w → ew. The corpus becomes:The new token ew joins the vocabulary.Merge 2 → Pair counts shift because e and w are now partly consumed. The big change:n ew appears in "newest" (×6) + "new" (×2) = 8, now the sole leader.Merge n + ew → new: Merge 3 → l o and o w tied at 7. Tie-break picks l o. Merge → lo: Merge 4 →lo w now appears 5 + 2 = 7, top of the board. Merge → low:And there it is: the common word "low" is now a single token. That's the payoff.Merge 5 →Four-way tie at 6 among new e, e s, s t, t . Tie-break picks e s. Merge → es:Merge 6 →es t wins the tie-break among the remaining sixes. Merge → est:Merge 7 →est wins over new est on tie-break. Merge → est:Merge 8 → "newest" is now just two symbols, new + est, appearing 6 times. Clear leader. Merge → newest:The most frequent word in our corpus, six letters, is now a single token.Merge 9 → Three pairs tied at 5: low , e r, and r . Tie-break picks e r. Merge → er:Merge 10 → er (5) ties with low (5); tie-break picks er . Merge → er:We could keep going, Merge 11 would glue low into low, and then we'd start eating into "wider" but you've now seen the complete mechanism. Common words become single tokens, rare words stay assembled from smaller pieces.The crucial takeaway: we stop when we hit our target vocabulary size (say, 30,000 tokens). The exact list of merge rules, the ordered sequence of "glue this pair, then that pair" gets saved to a file. At inference time, we replay those same rules, deterministically. That replayable rule list is the trained tokenizer.3. What Happens When the Tokenizer Sees a Word It Has Never Seen?This question trips up almost everyone, so it's worth answering carefully.Suppose MedBot's tokenizer was trained before "tirzepatide" entered the medical literature. The word isn't in the vocabulary as a single token. What happens?A modern BPE tokenizer doesn't crash, doesn't return an error, and doesn't emit an [UNKNOWN] placeholder. It falls back to smaller pieces it does know. This is one of BPE's quiet superpowers, there's no such thing as a word it can't represent at all.For "tirzepatide," the process looks like:Step 1: Start with characters (or bytes):Step 2: Apply the saved merge rules, in order, wherever they match.Maybe "ti" was a learned merge → ti. Maybe "de" was learned → de. Maybe "pa" was learned → pa.Step 3: Keep applying until no more rules match. Whatever's left stays as-is.Final tokens might be: ti r ze pa ti deSo "tirzepatide" becomes six clunky pieces instead of one clean one. The model can process it, nothing breaks, but it has to work harder to figure out that these six fragments together name a single diabetes medication. There's no built-in signal connecting them.Two nuances worth knowing:Byte-level fallback. Modern tokenizers (GPT-2 onward) operate on bytes, not just characters. Since every possible piece of text is made of bytes, and all 256 byte values are in the vocabulary, any string, emoji, Chinese characters, random binary, can always be tokenized. There is genuinely no "unknown word" failure case. The worst case is just "many tiny tokens."The older [UNK] token. Some older tokenizers (like the original BERT's WordPiece) do have a special [UNK] token for things they can't represent. When a model emits or ingests [UNK], information is permanently lost, you can never recover what the original text was. This is strictly worse than byte-level fallback, which is why modern LLMs avoid it.For MedBot, new medical terms will always appear after the tokenizer is trained. Byte-level BPE guarantees they'll at least be representable. But "representable" isn't "well-represented." If your domain has lots of specialized vocabulary, training the tokenizer on domain text, so important terms become single tokens, pays off in both quality and efficiency.4. How "myocardial infarction" Tokenizes Across Real TokenizersTheory is nice. Let's watch real tokenizers in action. We'll run three popular ones on the phrase "myocardial infarction" and see how differently they slice it. Two words become six or seven tokens and not one of the pieces means anything medical. That's tolerable but wasteful. Compare to a tokenizer trained on a large medical corpus (like Bio+Clinical BERT's): it's seen "myocardial" so many times that it keeps the whole word as one token, preserving the meaning as a single clean unit.The takeaway for MedBot: either start with a medical-domain tokenizer, or extend an existing one with medical vocabulary before pretraining. We won't regret it.5. The Data Pipeline: From Raw Internet to Training-ReadyRaw text is never ready to train on. Garbage in, garbage out is a cliché because people keep learning it the hard way. Every serious LLM project has a data pipeline, a sequence of cleaning and formatting steps that turns messy raw text into something you'd actually want a model to learn from.Here's what that pipeline looks like, with a caption for each stage:The two stages people most often get wrong deserve a closer look.Deduplication (and why duplicates are poison)Finding and removing documents that are exact or near-exact copies of one another. If the same article appears 10,000 times in your data, the model doesn't learn "language" from it, it memorizes that specific article. Memorization looks fine on your training metrics but fails on anything new. A study of C4, the corpus behind Google's T5 model, found a single 61-word sentence repeated more than 60,000 times (Lee et al., 2022). Sixty thousand.For exact duplicates, you hash each document (compute a short digital fingerprint of its exact contents) and drop repeats. For near-duplicates, same article with minor edits, a slightly different header, one paragraph added, exact hashing fails. That's where MinHash comes in: it produces a small "fingerprint" of a document such that similar documents get similar fingerprints. Compare fingerprints, flag any pair above ~80% similarity. Removing duplicates reliably improves model quality; there's no real debate about this.Quality FilteringNot all text deserves to be learned from. Common filters:Language identification: keep only text in your target language (English, for MedBot).Perplexity filter: run a small, fast language model (KenLM, trained on clean Wikipedia) over each document and measure how "surprised" it is. Wildly surprising text is usually gibberish or a wrong language, discard it.Length and punctuation heuristics: drop documents shorter than ~200 tokens, or where more than 30% of characters are punctuation (a sign of tables, code dumps, or scraping artifacts).Toxicity filter: run a toxicity classifier and remove flagged documents, so offensive text never enters training.Note what that last filter does not catch: patient-identifying information. For MedBot, de-identification, scrubbing names, dates, and record numbers from clinical notes, is a separate, mandatory pass. A toxicity classifier will sail right past a patient's name without blinking.Real corpus sizes (for perspective) 6. Token Count Math (Do This Before You Start Training)Before you spend money on GPUs, you should be able to answer one basic budgeting question. Here it is, in MedBot terms:"If MedBot trains on 50 billion tokens, using a batch of 1,024 sequences where each sequence is 2,048 tokens long, how many training steps does one full pass take?"A batch is the group of examples the model looks at before updating its weights once.How many steps to cover all 50 billion tokens once (one "epoch")?One pass over MedBot's corpus is about 24,000 training steps. Most LLMs do roughly one pass over their full corpus, with extra passes only on the highest-quality subset. This single number, 24,000, will let you estimate how long training takes and what it costs, once we know the speed per step in Article 03.7. Building a Tiny BPE Tokenizer From ScratchNow we turn the hand-walkthrough from Part 2 into real, runnable code. This isn't optimized for speed (production tokenizers use Rust under the hood), but every line maps directly to the algorithm you just traced by hand.In pseudocode, the whole thing is four lines:Here it is as real Python:Because the tie-break is deterministic, the printed output traces exactly the same merges we did by hand, check any line against Part 2:And now the real thing: training a production tokenizer with HuggingFaceYou'll rarely write BPE by hand in practice. Here's how you train a real tokenizer using HuggingFace's tokenizers library:8. Data Prep Code: The End-to-End PipelineFinally, here's a complete data pipeline using HuggingFace Datasets, scrape-to-split for PubMed abstracts. Every step has a comment explaining why it's there. Fair warning: the real pipeline runs against the full PubMed corpus (~36 million abstracts). As printed, it processes a 100,000-abstract slice so you can actually run it on a laptop, the comments tell you what changes for the real thing.Common Mistakes (What Can Go Wrong)Tokenization and data prep are unglamorous, which is exactly why they're where most projects quietly break. The failure modes worth watching for:1. Using a general tokenizer for a specialized domain. Feed MedBot a Reddit-trained tokenizer and "tachycardia" (fast heart rate) shatters into ta, chy, card, ia, four meaningless fragments. The model can learn to reassemble them, but it costs training compute and data you may not have. Fix: train or extend your tokenizer on domain text.2. Silently wasting your context budget. That same shattering means a 300-word clinical note that should fit in ~400 tokens balloons to 600+. With a fixed context window, the model now sees 40% less of each patient note before hitting the limit. You lose information, and there's no error message telling you so.3. Skipping deduplication. Leave duplicates in and the model memorizes repeated documents instead of generalizing. The tell-tale sign is cruel: your training loss looks great, while held-out performance quietly stagnates. You won't see it until it's expensive to fix.4. Train/validation leakage. If a near-duplicate of a validation document sneaks into the training set, your validation score becomes a lie, the model has effectively seen the test. Always deduplicate before splitting, and split by document, not by sentence.5. Forgetting that whitespace is part of the token. In many tokenizers, the (with a leading space) and the (without) are different tokens. Inconsistent whitespace in your data fragments the model's learning across near-identical tokens. Normalize whitespace during the formatting step. It's a two-line fix that saves weeks of confusion later.Next UpIn Article 02 Transformer Architecture, we take the token IDs we just produced and feed them into the model itself. We will build a complete Transformer from scratch embeddings, positional encoding, the attention mechanism, multi-head attention, the works and we will compute *every cell* of a 4×4 attention matrix by hand on a toy sentence. By the end, the black box will not be a black box anymore.