Base-model limits: why LLMs need engineering around them

When we use a language model directly — the way you do when you open a chat and ask it something, with no extra layer between you and the model — we are using what is known as a base model (or a “vanilla” model). It is the model that comes out of pretraining and, perhaps, the post-training (SFT + DPO) we saw in Part I, but with no extra customization for your domain or specific use case.

These models are incredibly capable — they hold fluent conversations, summarize documents, write code, answer questions across a huge range of topics. But using them like this, directly, in the real world, has serious limits that are worth understanding in depth.

The reason this chapter comes now — before we talk about prompting, RAG, agents, or fine-tuning — is that each of those techniques exists specifically to solve one of these limits. If you do not first understand what fails, you will not really understand why the solutions work.

Lack of domain knowledge

The model was trained on a huge slice of the public internet, books, articles, and code. But it does not know your company’s private information, or the hyper-specialized vocabulary of very specific sectors, or the internal data that was never published on the web.

The problem in detail: a base model knows general medicine because it read millions of medical articles — but it does not know your hospital’s specific protocol for administering a given drug, because that protocol was never on the internet. It knows agriculture because it read agronomy books, but it does not know what a specific disease looks like on the crop in your particular region, because that does not get published at scale.

A concrete example: a team designing an autonomous agricultural vehicle with cameras to detect diseased plants has thousands of labeled images of “Cercospora leaf spot on soybean variety X”. That information is in no public corpus. The base model cannot recognize that disease because it never saw it — it is not that it “reasons poorly”; the knowledge simply does not exist in its parameters, for the reasons we already saw in the Training data chapter of Part I.

Why more pretraining does not fix this? Because pretraining is static and general — including domain knowledge would mean gathering every internal document, continuing pretraining on them (expensive), and updating the model every time the information changes (constantly). It is not practical for most organizations. That is why the typical solution is not to retrain, but to inject knowledge at query time — what we will see as RAG later in this part.

An important nuance: the base model can have general knowledge of the domain (it knows what soybeans are, what a plant disease is, what Cercospora is in general terms) — what it does not have are the specific details of your particular case. That distinction between general and specific knowledge is the key to understanding what a base model can and cannot solve on its own.

Lack of up-to-date information

Models have a knowledge cutoff — they know nothing about what happened after that date, and each model has its own.

Why aren’t they updated constantly? Retraining a model with hundreds of billions of parameters costs tens of millions of dollars and months of compute. Nobody is going to do that every week just to fold in the news. Techniques for continual learning exist, but they are still expensive and can degrade performance on earlier tasks — so-called “catastrophic forgetting”.

Concrete examples: ask a model about an event after its cutoff date, and it simply will not know. A curious historical case: in 2017, a Donald Trump tweet with the misspelled word “covfefe” drove several automated systems of the time crazy, because that character sequence existed in no prior corpus. The same thing happens constantly with new slang, recent product names, or regulatory changes after the cutoff — the model will give plausible but outdated information, without knowing it.

How is it solved? By querying external sources in real time — connecting the model to a search API, a news database, or an up-to-date internal system. This is RAG with live sources, or more generally, tool use.

Hard to control

It is extremely difficult to predict or tightly constrain what an LLM will answer, especially under deliberate provocation or in ambiguous contexts.

The Tay case (2016): Microsoft launched a Twitter bot designed to learn from real conversations in real time, mimicking users’ language. In less than 16 hours, malicious users bombarded it with offensive messages, and Tay started replicating that behavior — Microsoft had to shut it down. The problem: Tay learned from everything that reached it, with no safety filter, and attackers exploited exactly that. Modern models have guardrails, but it is still possible to jailbreak them to ignore their safety instructions under the right pressure.

The unpredictability problem, even without attacks: the same question can get slightly different answers on two attempts (because of temperature and sampling, Part I), and a prompt worded slightly differently can change the answer dramatically. This is a serious problem in legal systems where every word matters, or medical ones where the tone has to be precise.

How is it addressed? Structured prompting (clear instructions about expected format and content), guardrails (systems that filter answers or detect jailbreak attempts), and fine-tuning (reinforcing desired behaviors with specific examples).

Lack of precision and consistency

In sensitive industries — legal, medical, financial — every word counts. A slightly wrong tone or an inconsistent format can ruin a contract or create confusion in a diagnosis.

The variability problem: without forcing determinism (temperature = 0), the model generates slightly different answers every time you ask the same thing. Even at temperature = 0, some implementations are not fully deterministic because of parallelism and non-commutative floating-point operations.

The factual-precision problem — hallucinations: models have no fact-checking mechanism; they generate what is statistically likely given their training. If there is a gap in their knowledge, they fill it with what they think “should” go there, not necessarily with the truth. Example: ask a base model “what is the capital of an invented country called ‘Zarlandia’?” — it is quite likely to invent a plausible answer like “Zarlandia’s capital is Zarcity”, instead of recognizing that the country does not exist. It is a problem of overconfidence, not a lack of intelligence.

How is it addressed? Structure the output (force JSON or specific formats instead of free text), combine the model with verification systems (RAG with source checking), and use evals (Part II) to systematically detect when the model drifts from the expected format or content.

Context-window limits

Models can “read” the equivalent of a couple of entire books. But their attention degrades on gigantic texts, even when the text technically fits in the context window.

The classic exercise — “needle in a haystack”: a random, irrelevant sentence is inserted in the middle of a huge document (“Arun and Max were drinking coffee at Blue Bottle”), and the model is asked what Arun and Max were drinking. Even when the context window is technically enough, the attention mechanism sometimes fails to find that “needle” inside the haystack.

This is not just an anecdote — it is measured and documented. The paper “Lost in the Middle: How Language Models Use Long Contexts” (Liu et al., Transactions of the Association for Computational Linguistics, 2024) tested this systematically with multi-document question-answering tasks: they placed the document with the correct answer at different positions inside the context (beginning, middle, end), surrounded by distractor documents, and measured the resulting accuracy. The central finding: a U-shaped accuracy curve — accuracy is highest when the relevant information is at the beginning or the end of the context, and drops sharply when it is in the middle. This pattern repeated consistently across several models, including models designed explicitly for long context.

Chart: the “Lost in the Middle” effect

Why it happens: the attention mechanism is parallel and dense — every token can attend to every other — but in practice, with thousands of tokens in between, that attention becomes diffuse: the “signal” of the relevant information gets diluted into the “noise” of the rest of the context. Models also tend to pay more attention to tokens at the beginning (primacy bias) and at the end (recency bias) of the context, relatively neglecting the center. This connects to something we already saw: the transformer architecture gives direct access to any part of the context through attention, but that does not guarantee that training taught the model to use that connection well when the relevant information is buried under thousands of irrelevant tokens.

How is it addressed? Chunking (splitting the text into more manageable fragments), retrieval architectures like RAG (instead of stuffing the entire text into the context, retrieve only the relevant parts), and context management in agents (keep only the most relevant information in each turn, discarding what is redundant).

Compute and latency limits

Generating a long answer can take several seconds even on expensive hardware, and every generated token has a real economic cost. Attention requires memory that grows with the square of the context length — we already covered all of this in depth in Part III (How many tokens per second you need, Inference optimization, KV cache, quantization, and the latency/throughput/cost trilemma). It is worth mentioning here as a base-model limit because it is, together with the rest of this list, one of the reasons “using the model directly” is not enough for production — but we are not going to repeat that full discussion here; it will be developed with the rigor it deserves later in the compendium.

Multi-step reasoning limits

Although models are excellent at recognizing statistical patterns, they fail more often on long-range reasoning tasks that require chaining several logical steps.

Example: “If Alice has 3 apples and Bob gives her 2 more, and then Alice eats one, how many apples does Alice have?” — a typical model solves it with no problem, it is simple. But a problem that requires 10 chained reasoning steps can make the model “lose the thread” at some intermediate point.

Models do not have a persistent “working buffer” the way human reasoning does — their ability to chain steps emerges from attention and the MLP layers, which have a practical limit on the depth of relations they can capture in a single pass. That is why specific techniques exist for this — like Chain-of-Thought in Basic prompt engineering in this same part (force the model to write its reasoning step by step, instead of jumping straight to the answer), or agents that break a large problem into more manageable subtasks (which we will see later).

Why this chapter comes first in Part II

Limit	Technique that attacks it
Lack of domain knowledge	RAG
Lack of up-to-date information	RAG with real-time sources, tool use
Hard to control	Structured prompting, guardrails, fine-tuning
Lack of precision and consistency	Structured prompting, fine-tuning, evals
Context-window limits	Chunking, RAG, context management in agents
Compute and latency limits	Smaller/specialized models, KV cache, quantization (Part III)
Multi-step reasoning	Chain-of-Thought, agents with planning

When you see someone using an LLM in production in a serious way, they are not “using the base model” — they are using a stack of techniques that compensate for these limits. The base model is the engine, but the complete vehicle needs steering, brakes, and a body. Without those layers, the engine is just a noisy block of metal.

The most common trap when adopting AI: see an impressive demo of a base model, assume it solves your problem directly, try it on a real case, watch it fail, and blame “AI” for not being smart enough. But the problem is almost never that the model is “dumb” — it is that it was asked to do something it is, structurally, not equipped to do without help: it does not have your internal data, it does not know what happened yesterday, it cannot reason stably across 10 steps, and it is not deterministic. Asking it to do all of that with no engineering layer around it is like asking an engine to take you to work without wheels.

Chapter summary

The limits of a base model are not fixable “defects” — they are constraints inherent to its architecture and training method. AI engineering is not about eliminating them (many are fundamental), but about designing around them — as in civil engineering, where you do not eliminate gravity, you build bridges that withstand it.

Each of the chapters that follow in this part of the compendium is, precisely, one of those pieces of engineering: prompt engineering to give better steering, RAG to inject external and up-to-date knowledge, agents to give planning capacity and tools, fine-tuning to adapt behavior, and evaluation to measure whether all of the above actually works.