Tokenization

In the previous chapter we saw that language models are, at their core, next-token predictors. But what exactly is a token? And, above all, how do you get those tokens from human-language text?

A language model cannot actually “read” text. It can only do mathematical operations on numbers. Before a sentence can enter the model, we have to turn it into a sequence of small text fragments, and then turn those fragments into numeric identifiers (IDs). We call those fragments tokens.

Why not just letters, or whole words?

Using letters (or characters) is wonderfully simple: any text can be broken down into its characters. But the problem is that the sequences would be far too long. Even a short paragraph would become hundreds or thousands of steps for the model, which slows down both training and generating answers.

Using whole words sounds appealing — it is the most natural option for us. But the number of possible words is enormous: tens of thousands in Spanish alone, and far more if we count proper names, slang, and typos. The model would have to learn a distinct token for each one, and every time a new word appeared (like a proper name), it would not know how to handle it.

Tokens are a smart compromise between both extremes. A tokenized sentence can have more pieces than the same sentence counted in words, but the reusable vocabulary can be much smaller than the set of all possible words. Common words become a single token; rare or new words get split into pieces the model already knows.

Analogy: think of tokens as Lego bricks. You have a limited number of pieces (the vocabulary), but with them you can build virtually any word. Some common words are large pieces (like a 4×2 brick), while rare words are built from several small pieces combined.

	Characters	Whole words	Tokens (subwords)
Vocabulary size	Very small (~100-300 symbols)	Enormous (tens of thousands, not counting proper names and slang)	Intermediate (typically 30,000-100,000)
Resulting sequence length	Very long	Short	Intermediate
Handles never-seen words	Yes, always — any text breaks down into known characters	No — it fails or needs a special “unknown” token	Yes — it splits into pieces already known
Computational cost	High, because of very long sequences	Low per sequence, but the giant vocabulary is expensive to maintain and train	Balanced — the middle ground used in practice

BPE: building a tokenizer by merges

The most widespread technique for creating these tokens is called Byte Pair Encoding (BPE). The idea, in four steps:

Start by splitting the text into individual characters (or even bytes).
Count which pairs of adjacent characters appear most often.
Merge the most frequent pair into a new symbol.
Repeat the process until you reach a desired number of merges (or until no pair exceeds a minimum threshold).

The result is a set of “building blocks” the tokenizer learns automatically from the data. Common words merge fast (because their pairs appear a lot); rare words stay as several small pieces, and in the worst case you can always fall back to the character level.

Manual example: the word “bajo” repeated 3 times

Take the text: bajo bajo bajo. Let’s apply BPE step by step.

Step 0 (start): each character is its own token — b-a-j-o (×3), ignoring spaces for simplicity.

We count adjacent pairs, weighted by frequency:

Pair	Frequency
(b,a)	3
(a,j)	3
(j,o)	3

All tied — we pick the first: (b,a) → we merge it into ba.

Step 1: tokens: ba-j-o (×3). We recount: (ba,j)=3, (j,o)=3. We pick (ba,j) → merge into baj.

Step 2: tokens: baj-o (×3). We recount: (baj,o)=3 (the only one). Merge → bajo.

Step 3: tokens: bajo (×3). After 3 merges, the complete word became a single token. The algorithm decided on its own that “bajo” deserves to be a unit, based solely on its frequency.

The compression, at a glance:

Merges applied	Token shape	Total number of tokens
0	`b-a-j-o` (×3)	12
1	`ba-j-o` (×3)	9
2	`baj-o` (×3)	6
3	`bajo` (×3)	3

From 12 loose symbols to 3 complete tokens, in just 3 merges — this is, in miniature, exactly the compression BPE achieves at real scale over corpora of billions of words.

Example with a more natural sentence

Now let’s use: the little cat played under the warm sun

At 0 merges, each character is its own token — inefficient and very long. As the tokenizer applies merges (learned from a large corpus, not just from this sentence), it discovers larger and larger blocks: first it merges very frequent pairs like (t,h) → th, then larger pieces, until it forms complete words like ·cat.

With enough merges, the final result might look like this: [·the, ·little, ·cat, ·played, ·under, ·the, ·warm, ·sun] — depending on the final learned vocabulary.

The mystery of the little dot (·)

In many BPE tokenizers a special marker is added at the start of each word — in our examples we use · (some tokenizers use _ or <w> instead, but the idea is the same). This is crucial for two reasons:

It distinguishes the start of a word. The token ·cat is different from cat without the marker. The first indicates that the word starts there; the second would appear only as a continuation inside a longer word. Without this marker, the model would have no way to know whether cat is a complete word or part of another.
It lets you tokenize without ambiguity. By adding · before each word, the space stops being treated as just another character and becomes the signal that triggers the insertion of that marker.

In practice, the tokenizer learns that ·cat is a very common token (because “cat” appears often as a standalone word), while cat without the marker might be much rarer, or not exist at all in the final vocabulary.

The algorithm, step by step

def step(words):
    # 'words' is a list of lists of symbols, e.g. [['b','a','j','o'], ...]
    pairs = count_adjacent_pairs(words)   # dict of (a,b) → frequency
    best = max(pairs, key=pairs.get)            # the most frequent pair
    if pairs[best] < 2:
        return None                              # nothing worth merging

    for i, word in enumerate(words):
        words[i] = merge(word, best[0], best[1])

    vocabulary.add(best[0] + best[1])
    merges.append(best)
    return words

After k merges, the vocabulary is the original (characters) plus the k new symbols: |V_k| = |V_0| + k.

To tokenize new text, the same merges are applied, in the same order they were learned:

def encode_word(word):
    symbols = ["·"] + list(word)    # word-start marker
    for rule in merges:              # applied in the order they were learned
        symbols = apply_merge(symbols, rule[0], rule[1])
    return symbols

BPE merge counter (the lab)

The “bajo” example can be followed by hand. On a longer sentence, the same step() algorithm — counting adjacent pairs and merging the most frequent — is easier to see live. Each click applies one merge; there is no training and no gradient descent. Because BPE only merges pairs that appear at least twice, this short sentence barely has any merges: “the” repeats, the rest do not. Paste the text several times, or the example bajo bajo bajo, to see the token count drop.

Text

Tokens remaining: 41Merges learned: 0Vocabulary size: 17 (|V₀| + merges applied)

Current token stream (· marks the start of each word)

·the·little·cat·played·under·the·warm·sun

Learned merges (newest first)
Merged pair	Frequency at merge time
No merges yet. Run the next step to see the first one.

The “real” tokenizer

In current models, the BPE tokenizer does not start from Unicode characters directly, but from raw bytes (the UTF-8 representation of each character). This has an enormous advantage: any text, in any language or with any symbol, can be represented as a sequence of bytes, and the tokenizer never runs into a genuinely “unknown” character — in the worst case, it can always fall back to the individual byte.

The tokenizers of GPT and many other modern models use this strategy. Other families exist with similar principles, but different variants in how they choose what to merge or split:

Family	Used in	How it decides merges/splits
BPE (byte-level)	GPT-2, GPT-3/4, Llama, and most current LLMs	Repeatedly merges the most frequent adjacent pair — the algorithm we just practiced
WordPiece	BERT	Similar in spirit to BPE, but chooses the merge that maximizes the probability of the training corpus, not just the most frequent one
Unigram	SentencePiece, T5 family	Starts with a large vocabulary and removes the symbols that contribute least, until it reaches the desired size — the reverse process of BPE

In production, BPE tokenizers are usually trained with tens of thousands of merges over huge amounts of text, which produces vocabularies between 30,000 and 100,000 tokens. But the central idea — learning building blocks from co-occurrence frequency — is still exactly the one we practiced here with “bajo bajo bajo”.

And after tokenizing?

Once we have the tokens, each one is assigned an integer — its ID in the vocabulary. That sequence of IDs is what actually feeds the model. Later, in the Embeddings chapter, we will see how those IDs become numeric vectors the model can process with real mathematical operations.

For now, it is worth locking this in: the tokenizer is the model’s front door. If we tokenize badly — with a poorly chosen vocabulary, or with too few merges — the model will struggle to learn linguistic patterns, no matter how good the rest of the system’s architecture is. Tokenization is a step that looks small, but it is critical to the success of any language model.

Chapter summary

Tokens are intermediate text fragments between characters and whole words.
BPE learns by merging the most frequent pairs in a corpus, in a purely statistical way.
Common words become a single token; rare ones are split into known pieces.
A word-start marker (·) distinguishes where each word begins.
Modern tokenizers start from bytes so they can represent any text, with no exceptions.

In the next chapter we will see how those numeric IDs are turned into vectors, and how the model learns to relate them to each other.