Why (Usually) NOT to Fine-Tune

Fine-tuning is taking an already pretrained model and further adjusting its weights with gradient descent and backpropagation — the same apparatus from Chapter 6 of Part I — using your own data, so the model behaves in a specific way. This general definition says nothing about whether every weight is updated or only a fraction — we will see that in the next chapter. What we can preview here is the most direct, intuitive version of doing it: full fine-tuning, where you literally update every weight in the network.

The scenario: a customer-support company wants its AI assistant to always answer in the specific tone of its brand — warm but direct, never generic. Someone on the team proposes: “let’s give the model hundreds of real conversations from our best agents, and have it learn to talk like that.” We are going to see why that intuition, though reasonable, almost always leads to more problems than it solves.

It is expensive and slow — and not just because of the training itself

The common intuition is to think of fine-tuning cost as “the price of running the training”. But remember Chapter 3: to know whether something improved, you need evals — a judge, a set of test cases, a systematic way to compare before and after. Without that, “it feels better” is exactly the trap we already identified as insufficient.

The real cost of our customer-support scenario includes, at a minimum:

Curating the training dataset — “hundreds of conversations” is not enough; they have to be genuinely representative of the desired tone, with no examples that contradict each other.
The training itself — the step most people do anticipate.
Building the eval — a set of test conversations, with a calibrated judge, so you can claim with evidence that the tuned model actually beat the alternative of not tuning.
Iterating — the first run is almost never the final one; you have to repeat the previous steps several times.

Step 3 is the one almost nobody budgets at the start, and it is precisely the one that makes “just train” underestimate the real cost by a considerable margin.

Cost comparison, in relative terms (not specific figures — see the note below):

Approach	Relative implementation cost	Recurring usage cost
Prompting / RAG, training nothing	Low — only prompt engineering or a retrieval pipeline	Standard per-token API cost, plus the infrastructure and extra tokens retrieval requires
PEFT (LoRA/QLoRA) — see the next chapter	Medium — requires a curated dataset and an efficient training run	Generally similar to standard inference cost, sometimes with a small surcharge
Full fine-tuning via a provider’s API	High — everything above, without PEFT’s efficiencies	Usually more expensive than using the base model — the inference surcharge is paid on every call

Why this table uses relative terms and not specific dollars: we actively researched concrete pricing figures, and found a genuinely volatile picture — different sources from 2026 (March, June, August) report different prices for supposedly comparable models. The most striking example is that OpenAI began the phasing out of its self-serve fine-tuning platform in May 2026: it stopped accepting new users, later restricted inactive organizations, and announced that active customers could create new jobs only until January 6, 2027. It was not a complete shutdown in May — inference of already-tuned models continues as long as their base models remain available — but it was a material change in availability with concrete deadlines. Any specific figure we had put here would have carried a real risk of describing a service that had already changed or no longer existed by the time you read this. What does stay more stable over time is the approximate cost-and-effort relationship among the approaches — that is why that is what we document.

The deeper lesson, beyond cost: this gradual retirement is, in itself, an additional reason not to commit a production architecture to one provider’s specific offering at a given moment — AI services change in price, in availability, and sometimes disappear, more often than you would expect when designing a system meant to last years. This is not a particular criticism of OpenAI: it is a real example of market volatility. It is always worth verifying the current offering directly in the provider’s documentation before committing — never trust a figure you saw in an article, however recent it seems.

The Market’s Speed Catches Up With You

By the time you finish tuning your model, it is common for the lab behind the base model to have already shipped a new version that beats your tuned model with no effort on your part. This applies especially when you tune to improve something that is also a general improvement target for the lab — following instructions, sounding more natural, holding a consistent tone. If what you were after, at bottom, was for the model to “better understand what you ask”, the next version of the base model may achieve that same improvement without you having invested anything.

It applies much less when you tune for something genuinely specific to your domain, that no lab has a reason to improve in its general model — for example, classifying tickets according to your company’s exact internal taxonomy. That distinction is the seed of the exceptions we will see at the end.

The Danger of Overfitting

Here it is worth going back explicitly to the “learning, memorization, and validation” chapter of Part I: we saw that a model can have low training loss by learning reusable patterns, by memorizing specific passages, or by a mix of both — and that training loss alone, without separate validation, cannot tell you which of the two is happening.

Fine-tuning is, mechanically, the same cycle from that chapter — the difference is that now the “training text” is your own curated examples. And the same risk applies: any accidental regularity in your data can be memorized with the same fidelity as the pattern you actually wanted to teach.

Case 1 — Ross Lazerowitz’s “Slack Fine-Tuning”: Ross trained an LLM on his 140 thousand Slack messages, hoping it would talk like him. When he asked it “write a 500-word blog post on prompt engineering”, the model — having learned his real communication habits — answered: “I’ll work on that in the morning.” When he insisted the next morning: “I’m writing it right now, it’s 6:30 a.m. here, please.”

The model did not fail in the sense of breaking or generating nonsense — it learned a genuine, consistent pattern from the data, just the wrong one. Ross wanted tone and communication style; the model learned, with the same fidelity, human procrastination habits — because both patterns were present in the data, and gradient descent does not distinguish “this is what I want you to learn” from “this is simply what repeats most”.

Case 2 — a recurring pattern in customer-support fine-tuning: a problem documented frequently in teams that tune models on real support transcripts: if a notable fraction of the training conversations end with the human agent escalating the case to a supervisor (because, naturally, the hardest conversations are the ones that get documented and reviewed most carefully), the tuned model can learn to escalate more often than desired — even on simple cases it could resolve on its own — simply because “escalate” was overrepresented in the training data relative to the real proportion of cases that truly warrant it. It is the same mechanism as Case 1: the model learns the statistical distribution of the data, not the intention behind why those examples were selected.

When Does It Make Sense, at Real Production Scale?

The title says “usually” on purpose. There are genuine cases where fine-tuning is the right decision, and it is worth being specific about when:

Very specific, structured tasks where prompting is already exhausted and still does not achieve enough consistency — for example, classifying documents into a very particular internal taxonomy that no base model knows out of the box, and where neither RAG (Chapter 5) solves the problem because it is not a matter of information but of consistent behavior.
Cost reduction at scale, using a small model tuned for a narrow task instead of paying repeatedly for a large model via API — exactly the logic of From 10 users to 1,500 salespeople in Part IV, where a small model tuned to evaluate customers turns out cheaper at high volume than a large generic model for that same repetitive task. The concrete signal that you have already reached this point: when the accumulated cost of repeated prompt tokens (the same lengthy context, over and over, on every call) exceeds, projected over a horizon of months, what it would cost to train and then serve a smaller, specialized model.
Behavior that must be “baked in” consistently, without depending on the correct context always being present in the prompt — unlike RAG, which depends on retrieval working well on every call.

The question is not “fine-tuning yes or no” in the abstract — it is “have I already exhausted prompting (Chapter 2) and RAG (Chapter 5), and I still do not have the consistency I need, at a volume where the cost already justifies it?” If the answer is yes on all three parts, it is worth it — but, as we will see next, almost never with full fine-tuning.