What is a language model?

Have you ever used your phone’s autocomplete and watched it suddenly suggest the entire sentence you were about to write? That little everyday bit of magic is, at its core, the same idea driving the most advanced artificial intelligence of our time.

At their core, both the giant models leading the current AI boom and the more modest ones we study in this compendium are simple — but enormously powerful — next-token predictors. Given a sequence of text fragments, the model computes a probability for every possible fragment that could come next. We call those fragments tokens. And note: they are not always complete words; they can be subwords, single characters, or even punctuation marks. For example, the word independently might split into [in, de, pendent, ly]. That lets the model handle terms it has never seen (like proper names) by combining known pieces, and it also makes working with languages very different from English much easier.

To generate text, the process is a loop that repeats over and over:

1. The model receives a sequence of tokens.
2. It computes a probability distribution over every token in its vocabulary to decide what comes next.
3. One is chosen (not always the most probable — we will come back to this when we talk about sampling, temperature, top-k, and top-p).
4. That token is appended to the end of the sequence.
5. The cycle repeats with the sequence now extended.

This mechanism — repeating a prediction token by token — is, surprisingly, all it takes to generate entire paragraphs, code, poetry, or complete conversations. There is no prior “plan” of the full text; the model literally does not know how the sentence will end when it writes the first word.

Not a new idea

It’s worth demystifying this: the central idea did not start with ChatGPT. One of the first ways most people experienced something like it was autocomplete on keyboards or search engines: you typed “world wide” and it suggested “web” as the likely continuation. That same principle — computing what is likely to come next, given what has already been written — is exactly the mechanism behind today’s LLMs.

What changed drastically between that simple autocomplete from years ago and today’s capabilities was not the task itself (it is still “predict the next token”), but:

• Scale (parameters, data, compute) by several orders of magnitude.
• The architecture used to make that prediction (from simple statistical models like n-grams, to recurrent networks, to today’s transformers).
• The quality and diversity of the training data.
• Later techniques such as fine-tuning, which shape how the model uses that predictive capacity to behave like a conversational assistant instead of just “continuing text”.

The difference between your phone’s autocomplete and a model like GPT-4 is not qualitative (both predict the next token), but quantitative. It is like the difference between having a notebook with 100 stock phrases and being able to look up, in milliseconds, a library that holds virtually all the public text on the internet. The leap in the amount of data and parameters is what produces the appearance of “understanding”.

The key idea we want to make clear: an LLM does not “know” things or “reason” in the human sense. Instead, it is solving a statistical problem of enormous dimensionality: “given this pattern of tokens, what is the most likely pattern that follows?” The complexity of that problem is so vast that its solutions look like reasoning, but they are, in essence, a statistical reflection of the data the model was trained on. Everything else (that it seems to reason, that it seems to know facts, that it seems to have a personality) emerges from that mechanism applied at massive scale over massive data.

Let’s build a simple statistical model

Before we touch transformers or neural networks, we are going to build something much humbler: a statistical n-gram model. The point is to show, in a tangible way, that “predict the next token” does not require magic — you can do it with simple counting.

The training corpus:

The little cat found a red ball. The little cat pushed the ball.
The little dog found the red ball. The little dog chased the ball.
The little bird saw the blue sky. The little bird flew toward the sky.
The little cat saw the little bird. The little dog saw the little cat.
The red ball rolled down the hill. The blue sky turned pink at night.
Each little friend wanted to play. Each little friend ran home at night.

How this model is “trained”: the process is pure counting, with no gradients and no backpropagation. We walk through the text word by word, and for each word we record which word(s) came before it and which word came after. That frequency record — how many times we saw each continuation given a certain context — is everything the model will remember. There is no “understanding” of any kind, just a table of counts.

We distinguish three levels of context, from least to most specific:

The prediction strategy: from specific to general. When the model needs to predict the next word, it follows a backoff strategy:

1. Try the trigram first. If it has already seen those two previous words together, use that specific continuation.
2. If it has never seen that trigram, fall back to the bigram. Use only the last word as context.
3. If it has never seen that bigram either, fall back to raw frequency (unigram). In the worst case, it picks the most common word in the entire corpus, regardless of context.

The model is asking, in essence: “Do I know this trigram? If not, do I know this bigram? If not, what is the most common word overall?”

This pattern of preferring the richest context available, and degrading toward something more generic when information is missing, is an idea that will show up again when we get to attention mechanisms in transformers.

A complete manual example: counting by hand

Before we jump into the giant tables, let’s work the exercise on a tiny sentence so the algorithm is clear step by step. We’ll use: the cat runs . the dog runs .

Walking through word by word:

• i=0 (“the”): nothing previous → unigram: the = 1
• i=1 (“cat”): previous “the” → unigram: cat = 1; bigram: the → cat = 1
• i=2 (“runs”): previous “cat”; pair “(the, cat)” → unigram: runs = 1; bigram: cat → runs = 1; trigram: (the, cat) → runs = 1
• i=3 (”.”): previous “runs”; pair “(cat, runs)” → unigram: . = 1; bigram: runs → . = 1; trigram: (cat, runs) → . = 1
• i=4 (“the”): previous ”.”; pair “(runs, .)” → unigram: the = 2; bigram: . → the = 1; trigram: (runs, .) → the = 1
• i=5 (“dog”): previous “the”; pair ”(., the)” → unigram: dog = 1; bigram: the → dog = 1; trigram: (., the) → dog = 1
• i=6 (“runs”): previous “dog”; pair “(the, dog)” → unigram: runs = 2; bigram: dog → runs = 1; trigram: (the, dog) → runs = 1
• i=7 (”.”): previous “runs”; pair “(dog, runs)” → unigram: . = 2; bigram: runs → . = 2; trigram: (dog, runs) → . = 1

Final tables:

• Unigram: the=2, runs=2, .=2, cat=1, dog=1
• Bigram: the→cat=1, the→dog=1, cat→runs=1, runs→.=2, .→the=1, dog→runs=1
• Trigram: (the,cat)→runs=1, (cat,runs)→.=1, (runs,.)→the=1, (.,the)→dog=1, (the,dog)→runs=1, (dog,runs)→.=1

Predicting with backoff:

• Prompt ends in “runs .”: trigram (runs, .) exists → 100% “the”.
• Prompt ends only in “the” (1 word, not enough for a trigram): bigram the → ? gives 50% “cat”, 50% “dog” (tied).
• Prompt ends in “dog cat” (never seen): no trigram → falls back to bigram cat → ? → 100% “runs”. The model does not understand that “dog cat” is a strange combination; it only looks at the last word and repeats what it has seen.

This by-hand example prepares us to understand the widget that comes next, where the same algorithm is applied to the full corpus.

N-gram counter (the lab)

I put together this little lab so you can see the model in action. Type or paste text, and the counter tokenizes by spaces and punctuation, builds the unigram, bigram, and trigram tables, and predicts the next word with backoff: trigram → bigram → unigram. It is pure counting, not training.

The limitations this model already teaches us:

• The vocabulary of contexts grows exponentially. With trigrams alone, the number of possible combinations of 2 words in a row is enormous, and the vast majority will never appear in any corpus, no matter how large.
• There is no semantic generalization. The model does not know that “dog” and “cat” are both animals that could appear in similar contexts. To it, they are unrelated distinct symbols.
• Memory is literal, not learned. There are no weights that get adjusted gradually; there is only a fixed table of counts. If it has not seen an exact sequence, it cannot invent a plausible continuation.

From statistical models to neural networks: the categorical leap

Before we go on, it is worth stopping on something we have left implicit until now. The n-gram counter we just built is not a neural network — it is, literally, a table of counts. There are no artificial neurons, no weights adjusted by gradient descent, nothing that “learns” in the sense we use that word in machine learning. It is pure, direct statistics.

Everything from here on — RNN, LSTM, and the transformer we will take apart in detail later — is a neural network. And this is, strictly, the minimum definition of what that means: a system made of many simple interconnected units (artificial “neurons”, organized in layers), where each one performs a basic computation (a weighted combination of its inputs, followed by a simple transformation), and where the system’s knowledge does not live in count tables or hand-written rules, but in the weights of those connections — millions of numbers that are adjusted gradually during training, through the gradient descent algorithm we will take apart in the How models learn chapter.

This connects directly to the definition we gave in the Introduction: “an LLM is, in its most precise definition, a neural network trained to predict text”. This is where that claim starts to take concrete shape — we are not going to take apart the full architecture yet (that happens in the dedicated chapter, Anatomy of a transformer), but it is worth marking this clearly: from this point in the compendium onward, when we say “the model”, we mean a neural network — an object categorically different from the count table we just built by hand, even though both share the same goal (predict the next token).

What exactly are weights?

When we say a neural network “learns by adjusting its weights”, it is worth being precise about what a weight is: it is a number that determines how strongly one node’s output influences another node’s input. In essence, weights are the connections between the nodes of the network — organized in matrices, where each entry in the matrix is the strength of a specific connection between a node in one layer and a node in the next.

This applies to virtually every parameter of a transformer: the attention matrices, the MLP matrices, and even the embeddings table (which, even if we think of it as a dictionary, is mathematically a connection matrix like any other — if you represented each token as a vector with a single 1 in the position of its ID and zeros everywhere else, multiplying it by the embeddings table would give exactly the same result as “looking up the row”). The only real exception is the normalization parameters (RMSNorm, which we will see in the Anatomy of a transformer chapter), which rescale each dimension on its own, without connecting two distinct nodes — but they represent such a small fraction of the total parameters (recalling the breakdown we compute later: Attention ~31%, MLP ~62%, Embeddings ~6%) that, in practice, almost everything we call “the model’s weights” is, effectively, its connections.

Temperature and sampling

Earlier language models (before transformers)

The counter we just built by hand is not an invented toy — it is a real language model, based on an idea from the 1950s: the n-gram model. For decades it was the state of the art in language processing.

The problem: the model only looks back a handful of words. Consider the sentence: “The boy who grew up in France speaks perfect…” — any speaker would complete it with “French”. An n-gram model, however, only sees something like “speaks perfect…” — it has no window back to “grew up in France”, because that fell outside the reach of its memory.

And this is not just a memory problem. Even if we widened the window to 10 words, the number of possible combinations (“The boy who grew up in the little…”) is so astronomical that we would need a corpus the size of the universe to have enough examples of each one. That is why the model cannot generalize the idea that “France” implies “French”.

This limitation drove the next generation of models (1990s–2000s): recurrent neural networks (RNNs) and their most famous variant, the LSTM.

RNNs and LSTMs

Instead of a fixed window, these networks read the text one word at a time, keeping a hidden state: a running summary that gets updated at each step.

The two problems that, in practice, were never fully solved:

1. Long-range memory was still fragile. Compressing an entire paragraph into a fixed-size summary means early details blur or get lost as the reading goes on.
2. They were inherently slow to train. Because each word depends on the hidden state of the previous word, processing is necessarily sequential: you cannot compute word 100 until you have finished words 1–99. There is no way to parallelize that computation.

The two problems to remember: n-grams only see a small, fixed window. RNN/LSTMs carry the entire past, but they compress it through a sequential state, which makes them slow and still prone to losing long-range detail.

2017: the transformer attacks both problems at once

And then, in 2017, the transformer arrived. This was not an evolution. It was a paradigm shift. It attacked both problems at the root:

• With the attention mechanism, it built a highway of information between every word and every other word, eliminating the curse of the fixed window. Any token can “look” directly at any earlier token, no matter the distance. That solves the long-range memory problem.
• In doing so, it allowed training to be fully parallelized: every token in a sequence is processed at the same time, not one after another. What used to take weeks could now be done in days or even hours.

This double victory is why today, virtually every conversational AI model (ChatGPT, Claude, Gemini, and the rest) has a transformer at its core.

Research did not stop there — there are active lines exploring modern recurrent alternatives and state-space models — but transformers still completely dominate the current AI boom.

tokens = tokenize(text)          # words and punctuation, separately
for i, w in tokens:
    unigram[w] += 1
    if i >= 1: bigram[tokens[i-1]][w] += 1
    if i >= 2: trigram[(tokens[i-2], tokens[i-1])][w] += 1

if trigram[w1, w2] exists:   P(w | w1, w2) = count / sum
else if bigram[w2] exists:   P(w | w2)     = count / sum
else:                        P(w)          = unigram[w] / N

adjusted_weight(w) = P(w) ^ (1 / T)
P'(w)              = adjusted_weight(w) / Σ adjusted_weight

Chapter summary

We have seen that:

• A language model is, at its core, a next-token predictor.
• We can build one with simple counts (n-grams) and understand its limitations: a fixed window and no generalization.
• Later attempts (RNN/LSTM) improved memory, but introduced slowness and fragility.
• The transformer revolutionized the field by offering parallelizable attention, solving both problems.

In the next chapter we will open up the transformer and see how attention actually works. But first, make sure you have internalized this fundamental idea: everything starts with guessing the next word.