Training data: the fuel of learning

We have seen the transformer architecture: a machine designed to process sequences and predict the next token. But a machine, on its own, knows nothing. It needs to learn from examples. And to learn, it needs data. A lot of data.

In this chapter we will see how the dataset that feeds a language model is built: where the text comes from, how it is cleaned, how it is prepared for training, and how much you actually need. As the old saying in AI goes: “data is the fuel; the model is the engine”. A powerful engine with dirty fuel is not going to run well.

The real scale: how much text does a model need?

The short answer is: a lot. Far more than any human could read in several lifetimes.

For a small model (say, 100 million parameters, similar to small GPT-2), you need hundreds of millions of tokens (a few tens of gigabytes of text). For a large model (like GPT-3, with 175 billion parameters), you need hundreds of billions of tokens — several terabytes of text.

Where does all that text come from? Typical sources include:

• Web crawls: Common Crawl is the backbone of almost every model — an organization that has been downloading and storing web pages monthly since 2008. Massive, but also very noisy (spam, HTML markup, duplicated text).
• Wikipedia: about 6 million articles in English, well structured, in relatively formal language. Small compared to Common Crawl, but high quality.
• Digitized books: Project Gutenberg, Google Books, and others — quality narrative and expository text.
• Academic papers: arXiv, PubMed, and so on — scientific and technical text.
• Source code: GitHub and other repositories — essential for models that generate code.
• News, forums, social media: Reddit, Hacker News, and so on — conversational, up-to-date language.

The mix matters: the proportion of each source shapes the model’s behavior. A model trained on too many forums can turn colloquial and informal; one trained on too much Wikipedia can be stiff and encyclopedic. The The Pile team (EleutherAI) published open documentation detailing exactly which sources they mixed and why — a good example of uncommon transparency in this space.

How much is enough? The Chinchilla rule

During the first years of transformers, the trend was clear: models kept getting larger, trained on datasets that were not growing at the same pace. People thought model size was the most important factor for performance.

But in 2022, researchers at DeepMind published “Training Compute-Optimal Large Language Models” (the Chinchilla paper), and that view changed. They trained more than 400 models of different sizes and data volumes, and found an empirical relationship: for every parameter in the model, you need approximately 20 training tokens for optimal use of a fixed compute budget.

The big lesson: it is not optimal to have a giant model trained on little data — it is better to have a smaller model trained on more data. In fact, Chinchilla (70B parameters, trained on 1.4 trillion tokens) beat GPT-3 (175B parameters, trained on only ~300 billion tokens) on most benchmarks, despite having less than half the parameters — because GPT-3 was, in retrospect, significantly undertrained for its size.

Applied to our small model: 8 blocks, width 256 ≈ 9 million parameters → the 20-token rule asks for ~180 million tokens. Default training uses 20,000 steps × 32 windows of 512 tokens ≈ 330 million tokens, or ~37 per parameter — almost double Chinchilla’s recommendation. Training past the Chinchilla point is fine: that figure describes the best use of a fixed budget; a small model keeps improving past that point, just more slowly.

Calculator and chart: where is the Chinchilla point for this model?

The line is tokens = 20 × parameters. The points are not invented: GPT-3, Chinchilla, and Llama 3 70B use published figures; the small point is this chapter’s example. It is arithmetic, not training.

An important note the chart is going to reveal: not every real model follows the rule to the letter, and for good reasons. The Chinchilla rule optimizes training cost — but if you are going to serve the model to millions of users, inference cost also matters, and there it sometimes pays to train longer.

The Llama 3 case: Meta trained its 70B-parameter model on 15 trillion tokens — almost 11 times more than the Chinchilla-optimal point for that size (1.4T). According to Meta’s own blog, the model’s performance kept improving log-linearly even far past the point Chinchilla would mark as “optimal”. The reason: a smaller model, trained for longer, ends up more efficient and cheaper to serve afterward — a higher training cost once, in exchange for a lower inference cost multiplied by every future query. This is exactly the tension we already saw in The inference trilemma in Part III: there is no configuration that optimizes training and inference at the same time — you have to choose, deliberately, which one to prioritize for the use case.

Cleaning the data

A “messy” dataset usually has HTML tags, broken encodings, repeated paragraphs, navigation menus, and nonsense text. Cleaning (data curation) involves:

• Extracting the clean text: strip HTML tags and keep only the textual content.
• Normalizing: unify encodings, handle whitespace, convert curly quotes to straight ones.
• Filtering for quality: drop pages with very little text, a high proportion of repeated characters, or offensive language.
• Removing duplicates: at the document level and also at the paragraph level.

A cleaning example:

Before:

<story>
  <p>The little cat found a red ball.</p>
  <p>BUY NOW BUY NOW BUY NOW</p>
  <p>The little cat found a red ball.</p>
</story>

After:

The little cat found a red ball.

The HTML markup, the spam, and the duplicated paragraph are gone. In a typical cleaning process, you keep approximately 60% of the original bytes — the remaining 40% is junk that contributes nothing, and that could even hurt learning.

Deduplication: quality beating quantity

If the model sees the same sentence hundreds of times, it memorizes that specific sentence instead of learning a general pattern — and at generation time, it will tend to repeat it (the famous “parroting”). The paper “Deduplicating Training Data Makes Language Models Better” (2022) showed that removing duplicates improves the model with less data — training on 100GB of unique data beats training on 200GB where half is duplicated.

Deduplication happens at several levels: at the document level (the same page under several URLs), at the paragraph or line level, and sometimes at the n-gram level (overly common fragments). A frequent method uses approximate hashing (like MinHash) to find duplicates that are not identical byte for byte, but are very similar.

The data mix: the art of balance

Not all text is equal. A model trained exclusively on books would generate formal, academic text; one trained on Reddit would be colloquial and full of slang. The trick is the mix (data mix): choosing proportions of each domain so the model is versatile.

An illustrative example of a typical mix:

This mix is not fixed — every team adjusts it to their goals (a code-specialized model, like CodeLlama, uses a much higher proportion of source code). An important detail: the mix is applied during sampling of the data, not by simply concatenating files — if a domain is smaller (like Wikipedia), it gets more weight in the sampling so the model sees enough examples of it.

From clean text to training files

The final step turns text into token identifiers, split into a training set and a validation set. Weight updates use only the training side; validation is reserved to check generalization to text never seen.

data = encode_documents_to_array(tokenizer, documents, dtype=np.uint16)
n_val = max(1, min(int(len(data) * 0.1), 2_000_000))
train_ids, val_ids = data[:-n_val], data[-n_val:]

np.save("cache-tokens/train.npy", train_ids)
np.save("cache-tokens/val.npy", val_ids)

train = np.load("cache-tokens/train.npy", mmap_mode="r")
i = np.random.randint(len(train) - block_size - 1)
x = train[i : i + block_size]              # the context
y = train[i + 1 : i + 1 + block_size]      # shifted by one = the targets

The central trick: shift by one. At each position, the next token is the correct answer. With no human labeling, we turn text into millions of self-labeled prediction problems — this is self-supervised learning at massive scale: the text itself contains the answers.

Validation: how do you know the model is learning?

We set aside a small percentage of the data (5-10%) as a validation set, which is never used to update weights — it is only evaluated periodically to measure performance on data the model has not seen. If training loss keeps falling but validation loss starts to rise, the model is overfitting: it memorized the training data instead of learning general patterns.

The practical cap of 2 million tokens in the code above exists because you do not need a giant validation set — a representative sample is enough to detect overfitting.

The final balance: model, data, and compute

Training a model means deciding how to spend a limited budget across model size (larger = more capacity, but slower and more expensive), dataset size (more data = more examples, but more expensive to process), and training duration (more steps = better fit, with diminishing returns).

The Chinchilla rule says: with a fixed budget, ~20 tokens per parameter is optimal — but that is optimal only for training cost. If the model is going to be deployed at scale (and inference cost matters), it can be better to train on much more data even if the model is smaller:

Look at the comparison between the two 70B-parameter rows: same model size, but Llama 3 paid a much higher training cost in exchange for a model that is markedly cheaper and more efficient to serve afterward — a deliberate decision, not a miscalculation.

Chapter summary

Training a language model goes through several phases:

• Acquisition: collect text from multiple sources (web, books, code, and so on).
• Cleaning: remove HTML, spam, duplicates, and nonsense text.
• Mixing: balance the proportions of each domain to get versatility.
• Tokenization: turn the clean text into arrays of IDs.
• Split: separate into training (~90-95%) and validation (~5-10%).
• Self-supervision: create (input, output) pairs by shifting the sequence one position.

Training data is the most decisive factor in the model’s final behavior — an excellent architecture with mediocre data produces a mediocre model; a modest architecture with excellent data produces a surprisingly good one.

In the How models learn chapter we will see how models actually learn: the gradient descent algorithm, the loss function (cross-entropy), and the backpropagation that turns data into useful weights.

If you ever work on preparing data for a model of your own, keep the real effort split in mind: most of the time goes into cleaning and curating the data, not into designing the architecture. It is almost a cliché in machine learning, but it is true — data is the heart of learning; without it, the model is just an empty shell.