How models learn behavior: from text predictor to useful assistant

Pretraining learns a single objective: given text, guess what comes next. A capable base model can answer questions and follow instructions from the prompt, but it has not learned to behave like an assistant reliably.

A base model is not an assistant

A base model is, at its core, a talented mimic with no sense of occasion. It has seen billions of tokens of every kind — books, forums, code, poetry — and learned to imitate all those styles. The problem is that it has no notion of when to use each one.

A practical example — the prompt: "What is the capital of France?" A base model might answer, with similar probability across the options:

• “Paris.” (correct, but dry, with no style judgment).
• “Did you know that Paris is the capital of France?” (answers with another question — it saw many dialogues where characters take turns asking).
• “The capital of France is Paris, situated on the banks of the Seine…” (continues as if it were an encyclopedia article).
• “User: What is the capital of France? Assistant: Paris.” (imitates the full format of a dialogue it saw in its data, including the labels, as part of its own reply).

The model does not “understand” that a direct, useful answer is expected — it simply continues the most statistically likely pattern, and that pattern can be a question, a label, or a ramble, depending on what it saw most often in its data. Pretraining gives the model knowledge — it does not give it behavior. That is taught in the phase that follows.

Supervised fine-tuning (SFT): teaching the desired format

SFT uses exactly the same next-token prediction as pretraining, aimed at a different kind of example: complete conversations in assistant format.

User: What is the capital of France?
Assistant: Paris is the capital of France.

User: How do you make a chocolate cake?
Assistant: To make a chocolate cake, you need the following ingredients...

The model sees thousands of examples like this. Little by little it learns that when it sees the pattern User: question..., what should come next is Assistant: useful answer... — and not only the format, but also the style: concise, polite, direct.

The key: we only score the assistant’s reply. The model sees the full exchange (user + assistant), but we only update the weights based on the tokens of the assistant’s reply. The user’s tokens are context the model must learn to follow, not to predict — if we also “got the user’s question right” as part of the loss, we would be teaching it to predict questions, not to answer them.

Under the hood (simplified)

tokens = tokenizer(full_conversation) # [USER, ASSISTANT, ...]
mask = [0, 0, ..., 1, 1, 1] # 1 only for the assistant tokens
logits = model(tokens)
loss = cross_entropy(logits, tokens, mask=mask) # Gradients from user tokens are ignored

How many examples do you need? Far fewer than in pretraining. While pretraining uses hundreds of billions of tokens, SFT usually works with tens of thousands of conversations (typically 10k-100k examples) — the model already knows language, it only needs to learn format and style. But the quality of those examples is critical: if they are poor (incorrect, not very useful answers), the model will learn exactly that behavior with the same diligence it would learn a good one.

Analogy: imagine an actor who has read complete encyclopedias (pretraining) — they know a lot of facts, but they do not know how to act in an interview. SFT is like a director who hands them scripts and says “this is how we want you to answer”. The actor learns to use their knowledge in the desired format, but they do not acquire new facts just by reading scripts.

What SFT can and cannot do

Very good at teaching forms: formats (lists, tables, code), styles (formal, friendly, technical), and routines (how to ask for clarification, how to say goodbye). But SFT cannot reliably teach new knowledge — if the model never saw in pretraining that “the capital of Australia is Canberra”, no SFT example is going to teach it that. It can only teach it to express the knowledge it already has, not to acquire new knowledge. And if the training examples contradict each other, the model gets confused instead of learning a clean pattern.

The next frontier: human preferences

SFT produces a model that follows the desired format — but it still does not know which answers are “better” than others among several that are equally correct in format.

The historical lineage:

• 2017: researchers at OpenAI and DeepMind taught a simulation agent to do a backflip using purely human comparisons between video clips — nobody defined mathematically what a “good” backflip was, but humans recognized it when they saw it. First demonstration that comparative feedback could teach complex behaviors without explicit labels.
• 2022: the same principle is applied to text — OpenAI trains InstructGPT with RLHF (Reinforcement Learning from Human Feedback): humans compare answers, a reward model is trained to score them, and RL is used to optimize the model against that score. First step toward modern chatbots — but a heavy pipeline (reward model, reference model, RL that is delicate to stabilize).
• 2023: a Stanford group derives that, under certain assumptions, the desired policy can be trained with a single classification-style loss, directly on the comparisons — no separate reward model, no RL. That is DPO (Direct Preference Optimization). It is not mathematically interchangeable with every form of RLHF, but it is a simpler path to the same destination.

DPO: direct preference optimization

What it does, exactly: two copies of the model are kept — the policy (the one being trained) and a frozen reference (usually the model already tuned with SFT). For each preference pair, both copies are asked: how likely would you be to produce the chosen answer, and the rejected one? The loss pushes on the gap between the two, rewarding the policy for favoring the chosen answer more than the reference already did.

A hypothetical example, to see it in numbers: suppose the question "How do you make a chocolate cake?", with two candidate answers:

• A (chosen): “First, mix the flour, the cocoa, and the sugar. Then add the eggs and the milk. Bake at 180°C for 30 minutes.”
• B (rejected): “Chocolate cake is delicious. You need chocolate, flour, and eggs.”

Humans prefer A (clear, actionable instructions) over B (vague). Suppose the reference model (SFT, before DPO) assigned, for some reason, more probability to B than to A — perhaps because B is more generic and that kind of answer is more common in the data. DPO training pushes the policy so that the probability of A goes up relative to the reference, and that of B goes down relative to the reference, widening the gap in favor of A. The model does not learn a simple rule like “prefer long answers” — it learns the specific human judgment behind that preference, which in other cases might favor exactly the opposite (a short, direct answer).

The beta parameter: it controls how strongly the objective is anchored to the reference. A small beta (~0.1) leaves the policy freer to deviate — it learns preferences fast, but can diverge more. A large beta (~1.0) keeps it closer to the reference — it learns more slowly, but more stably. In practice, beta is usually between 0.1 and 0.5.

The canonical formula:

$$\mathcal{L}_{DPO} = -\log \sigma\left(\beta \left[\log\frac{\pi_\theta(y_w|x)}{\pi_{ref}(y_w|x)} - \log\frac{\pi_\theta(y_l|x)}{\pi_{ref}(y_l|x)}\right]\right)$$

Where π_θ is the policy, π_ref the frozen reference, y_w the chosen answer (win), y_l the rejected one (lose). The loss is small when the policy prefers the chosen answer much more than the reference does, and large when it prefers the rejected one — it is a binary classification loss, where the “class” is which answer is better.

Under the hood — a length-normalized variant: the formula above compares summed log-probabilities of each complete answer — but that has a side effect: a longer answer accumulates more negative log-probability simply by having more tokens, so the preference margin can turn partly into a length contest, not a quality one. A more careful variant normalizes by the number of tokens, comparing how natural each answer is per token, instead of in total:

token_count = mx.maximum(mask.sum(axis=-1), 1)
response_logprob = -(token_losses * mask).sum(axis=-1) / token_count

# how much more the policy likes each answer, compared to the reference
chosen_gap = policy_chosen_logprob - reference_chosen_logprob
rejected_gap = policy_rejected_logprob - reference_rejected_logprob

loss = -log_sigmoid(beta * (chosen_gap - rejected_gap))

If the policy prefers the chosen answer more strongly than the reference, the sigmoid saturates and the loss falls toward zero. The reference weights stay frozen at all times — only the policy learns.

Advantages of DPO over RLHF: simplicity (no separate reward model, no unstable RL algorithms), efficiency (a single loss, optimizable with the same gradient descent as always), and stability (none of the typical RL divergence problems). Limitations: it assumes that human preferences are transitive and modelable with a simple utility function (not always true), and it does not work well for tasks that require active exploration, because DPO is an offline method — it uses a fixed set of comparisons, it does not generate new ones on the fly.

Evaluating behavior: how do you know it works?

Evaluating behavior is harder than evaluating validation loss — a numeric metric is not enough; you need human judgment or automated evaluation with other models. This is exactly the territory of LLM judges and evals, which we already covered in depth in Part II: pairwise comparison between two versions of a model, single-answer scoring on a scale, or rubric-guided evaluation — the same three techniques, applied here to decide whether a post-trained model actually improved, not just whether its loss went down.

An additional pattern common in this specific context: the win rate — what percentage of the comparisons the new model wins against a base version or against a competitor (e.g. “the model with DPO wins 65% of the comparisons against the SFT-only model”). It is, in essence, an aggregation of many pairwise comparisons into a single summary number.

The full post-training flow

The usual path to produce an assistant model:

1. Pretraining — massive text, next-token predictor → base model.
2. SFT — user↔assistant conversation examples → a model that follows the format.
3. DPO (or RLHF) — human comparisons → a model that also prefers higher-quality answers.

Closing analogy: think of a medical student. Pretraining is reading every medical textbook — they acquire raw knowledge. SFT is learning to structure diagnoses and prescriptions in the correct format. DPO is working side by side with an expert physician who points out, again and again, when one diagnosis is better than another — until that judgment is internalized.

Chapter summary

• A base model is a mimic with no sense of occasion — it has knowledge, but not behavior.
• SFT teaches format and style with conversation examples, scoring only the assistant’s reply.
• DPO teaches preferences through human comparisons, without needing an explicit “correct answer” — only which of two options is better.
• Neither post-training technique teaches new knowledge or reasoning the model did not develop in pretraining — they only shape how it uses what it already has.
• Evaluating whether post-training worked requires the same LLM-judge tools we saw in Part II, not just looking at the loss.

In the Base-model limits chapter we will see the limits of base models — what they can do, what they cannot, and where they fail systematically.