Basic prompt engineering

We already saw in the previous chapter that a base model has no “sense of the occasion” — it continues whatever pattern the prompt suggests, with no guarantee that it is the one you want. The fastest, cheapest way to improve a model’s behavior — without touching a single weight, with none of gradient descent — is to design better instructions. That is called prompt engineering, the first and most accessible layer of all the techniques in this part.

Why prompt engineering is the first line of defense

Back to the limits table from the previous chapter: RAG, fine-tuning, agents, and prompt engineering do not cost the same, are not implemented at the same speed, and are not undone with the same ease if something goes wrong. Before you scale up to any of the heavier techniques in this part of the compendium, it is worth understanding precisely why prompt engineering should almost always be tried first — and also, with the same precision, when it stops being the right answer.

Cost: what you have to build for each option

Changing a prompt requires building nothing. Compare that with what each alternative demands:

RAG needs a vector database, an embeddings pipeline, a chunking strategy, and a process to keep that data up to date — real infrastructure you have to design, deploy, and maintain (RAG and Vector databases).
Fine-tuning needs compute (GPU), a curated dataset of quality examples, and an evaluation process to confirm that something actually improved (Why (usually) not to fine-tune and LoRA vs. QLoRA).
Prompt engineering needs you, literally, to change the text of a variable in your code. Zero new infrastructure, zero training compute, zero data pipeline.

Iteration speed: minutes vs. weeks

Technique	Typical time to see the first result
Prompt engineering	Minutes — you change the text, call the API again, see the result immediately
RAG	Days — you have to prepare documents, generate embeddings, stand up the vector database, and test retrieval
Fine-tuning (even with LoRA/QLoRA)	Days to weeks — collect and curate data, train, evaluate, iterate if the result does not convince

This speed difference is not a minor detail — it is the main reason a good engineer tries first whatever is fastest to try, not whatever is most “sophisticated” or whatever occurs to them first.

Reversibility: the sunk cost you do not want to pay twice

If a prompt change does not work, you undo it with no loss — you never spent compute, you never committed infrastructure. If a fine-tuning goes wrong (poor-quality data, overfitting, an unwanted behavior that got “baked” into the weights), you already spent the training compute, and fixing it means repeating that cost, not simply undoing a text change.

A worked example: where prompt engineering works, and where it hits its ceiling

Let’s take the “lack of precision and consistency” limit from the previous chapter, with a concrete case: a system that must extract, from customer reviews, whether the product had a manufacturing defect.

Attempt 1 (naive prompt): “Did this product have a manufacturing defect? Review: [text]” — the model answers with long explanations, sometimes “yes”, sometimes “it seems so”, sometimes an entire paragraph with no clear answer. Inconsistent, hard to process automatically.

Attempt 2 (improved prompt, with format and explicit criteria): “Answer ONLY with ‘YES’ or ‘NO’. Treat it as a ‘manufacturing defect’ only if the review explicitly says the product arrived broken, did not work from first use, or has a problem present in multiple units reported by other customers in the same text. Review: [text]” — with explicit criteria and a forced format, consistency improves dramatically. This is prompt engineering working exactly as it should.

Where it hits its ceiling: if you ask the same system to identify whether the defect belongs to a specific product model that was discontinued before the model was trained, no prompt — however well designed — can solve that. The knowledge simply is not in the model’s weights. Here the problem is no longer instructions, it is domain knowledge — and that is exactly the limit RAG exists to solve, not prompt engineering. No amount of instruction engineering substitutes for information the model never saw.

A nuance almost nobody mentions: prompt engineering is not always free at scale

Here is the truly important part, and the reason “first line of defense” does not mean “always the best option”: an elaborate few-shot prompt, with several examples and lengthy instructions, consumes more tokens on every call — and that cost is paid over and over, on every request, forever.

Compare that with fine-tuning: the training cost is high, but it is paid once. After that, the tuned model can answer correctly with a much shorter prompt — because the behavior is already “baked” into the weights, it does not need you to explain it again on every call.

That means the equation changes with volume: for a system that makes a thousand calls a day, the long prompt is indisputably cheaper than fine-tuning. But for a system that makes millions of calls a day, that recurring per-token cost, multiplied by every call, can end up far exceeding the training cost — which was paid once. This is, precisely, one of the legitimate reasons to scale up to fine-tuning that we already saw in Chapter 4: not because prompting “does not work”, but because at enough scale, it stops being the cheapest option. It also connects directly to the latency/throughput/cost trilemma in Part III — the same kind of engineering decision, applied here to prompt design instead of inference infrastructure.

The full decision criterion

Is the problem unclear instructions or inconsistent format? → Prompt engineering. Start here, almost always.
Is the problem knowledge the model does not have? → RAG, no matter how well you write the prompt.
Does the problem persist after a well-designed prompt, AND is call volume high? → Then yes, evaluate fine-tuning — both for behavioral consistency and for the recurring-cost savings at that scale.

The shot spectrum: zero-shot vs. few-shot

Zero-shot: you ask the model to do something directly, with no examples.

Classify this product review as positive, negative, or neutral:
"The laptop is fine, but I expected more."

The problem: this is genuinely subjective. For some companies it is neutral, for others negative — the model has to guess your criteria without you having communicated them.

Few-shot: you give explicit examples, calibrating its criteria before the real question.

Example 1: "It exceeded my expectations" -> Positive
Example 2: "It's fine, but it's missing features" -> Negative
Now classify: "The laptop is fine, but I expected more."

The model now answers “Negative” with more confidence — not because it “understood” your policy in the abstract, but because the second example sets a textual pattern similar to the real question.

Why it works, connecting to Part I: no weight changes between zero-shot and few-shot. The only thing that changed is which tokens are in the context. The examples become additional context that attention can consult directly — the same causal attention from the anatomy of the transformer, with more information available so the query at the final position can “find” a relevant pattern among the keys of the examples. This is called in-context learning — radically different from fine-tuning: no backpropagation, no weight update.

When to use each?

	Zero-shot	Few-shot
When it makes sense	The task is common and unambiguous (“translate this to English”)	The criteria are subjective, specific to your company, or the output format is unusual
Cost	None extra	Each example occupies tokens in the context — it costs money and space on every call
Risk if you skip it	The model may guess an ambiguous criterion badly	—
How many examples to use	—	A few well-chosen examples usually suffice; adding many more rarely helps as much as people think, and consumes valuable context that could be used for the rest of the prompt

Advanced reasoning techniques

Chain of Thought (CoT)

Force the model to write its reasoning step by step before the final answer.

Why it works mechanically: an LLM generates one token at a time, and each generated token becomes context for the next. If the model jumps straight to the answer, it has to produce it in a single compute step. If it generates intermediate steps, each step becomes additional context that later steps can consult — more compute space spread across more tokens, instead of compressing all the reasoning into a single answer token.

A worked example, showing the real difference:

Question: “A train leaves at 14:20 and the trip lasts 3 hours and 45 minutes, but it stops for 20 minutes at an intermediate station that is not included in the trip duration. What time does it arrive?”

Direct answer (no CoT): a model may answer hastily “18:05” — it added 3h45m to 14:20 but forgot the 20-minute stop, a common error when you ask for “just the answer”.
Answer with CoT (“think step by step”): “First I add the trip duration: 14:20 + 3:45 = 18:05. Then I add the 20-minute stop, which is not included: 18:05 + 0:20 = 18:25. Answer: 18:25.”

By forcing the model to expose each step, each intermediate sum is “anchored” in the context before calculating the next — reducing the probability of skipping a step, exactly as a person would when solving the problem on paper instead of trying to do it all in their head.

Two variants, with names of their own:

Zero-shot CoT: simply add a trigger phrase like “Think step by step and do not skip any step”, with no example of how to reason. Kojima et al. (2022, “Large Language Models are Zero-Shot Reasoners”) showed that this single instruction, with no examples, already improves reasoning-task performance notably compared with asking for the direct answer.
Few-shot CoT: besides asking for the reasoning, you show one or two complete examples with their chain of thought already written (like the train example above, used as an example inside the prompt). It usually works better on harder tasks, because the model has a complete pattern of “what good reasoning looks like” to imitate, not just the instruction to reason.

Prompt chaining

Split a giant task into a sequence of smaller prompts, where the output of one feeds the next.

Example — a customer-support email:

Prompt 1: extract the key problems from the customer’s message.
Prompt 2: generate a structured response outline, based on those problems.
Prompt 3: write the final email in the company’s tone.

This lets you debug the system with precision — if the final email comes out wrong, you can see which link the problem happened at, something impossible with a monolithic prompt. This decomposition principle shows up again, more forcefully, in agentic flows: an agent is, in large part, a more dynamic way of chaining prompts, where the model itself decides when to move to the next step.

Chapter summary

Zero-shot works well for common, unambiguous tasks; few-shot calibrates the model’s criteria with examples, at the cost of extra tokens on every call.
Neither technique changes the model’s weights — both are in-context learning, taking advantage of attention over what is already in the prompt.
Chain-of-Thought (zero-shot or few-shot) improves multi-step reasoning by exposing the intermediate work instead of jumping straight to the answer — the concrete technique that solves the “multi-step reasoning” limit we saw in the previous chapter.
Prompt chaining decomposes complex tasks into separately verifiable steps — the same principle that reappears, at larger scale, in agentic flows.

In the next chapter we will see how to know, systematically and not just by intuition, whether a prompt change actually improved something: LLM judges and prompt evals.