LLM Inference Optimization: Key Techniques

You have chosen the model, designed the agentic flow, and calibrated the evals. Production arrives, and the question shows up: “why is it so slow? why is it so expensive?”

Inference on large models is expensive and slow by nature. But there are four main levers to speed it up and cheapen it — and the best part is they do not compete with each other. You can combine all of them.

Distillation: Train a Small Model to Imitate a Large One

Distillation trains a smaller model (the “student”) to imitate the behavior of a larger, more accurate one (the “teacher”). The student learns not only the correct answers, but the teacher’s full probability distribution over the vocabulary — so-called soft knowledge.

How it works: the teacher generates logits for each token; a high temperature is applied to smooth that distribution (the same temperature concept from Part I); the student is trained to minimize the difference between its own distribution and the teacher’s, on top of the standard supervised loss. The result: the student learns not only which answer is correct, but why the teacher preferred it.

Real case — the DeepSeek-R1 family:

Model	Parameters	Approximate VRAM	Precision (MATH-500)
DeepSeek-R1	671B	~1,543 GB	97.3%
Distill-Llama-70B	70B	~181 GB	94.5%
Distill-Qwen-32B	32B	~82 GB	92.0%
Distill-Qwen-14B	14B	~36 GB	88.0%
Distill-Qwen-7B	7B	~18 GB	83.9%
Distill-Qwen-1.5B	1.5B	~3.9 GB	72.0%

The 70B version keeps ~97% of the original accuracy with 8.5× less memory. The 7B version keeps ~86% of the accuracy with 85× less memory — but look at the chart: the drop accelerates noticeably between 7B and 1.5B. Distillation has clearly diminishing returns, not a linear loss.

When to use it: the base model is too large for your hardware, you need to cut inference costs, and you have access to a good teacher model. When not to: you need maximum accuracy on very complex tasks, you don’t have a good teacher available, or the cost of training the student exceeds the expected savings on inference.

Quantization: Less Numerical Precision, Same Architecture

It reduces the numerical precision of the weights — from 32 or 16 bits down to 8 or 4 bits — shrinking size and speeding up inference. It is like converting a high-resolution photo to a lower-resolution one: you lose detail, but it weighs much less and processes faster.

Types: PTQ (post-training, fast and simple, with pre-quantized checkpoints frequently available), dynamic quantization (applied at inference time, more accurate but slower), and QAT (the model is trained with quantization in mind from the start — maximum accuracy, more expensive and slower).

Popular formats: GGUF (llama.cpp, efficient on CPU/GPU, limited hardware), GPTQ (optimized for NVIDIA GPUs), AWQ (higher accuracy than GPTQ at comparable size), bitsandbytes/NF4 (used in QLoRA, LoRA vs. QLoRA).

Real benchmarks (deepsense.ai):

Model	Hardware	Base Speed (FP16)	Quantized Speed (AWQ)	Gain
DeepSeek 7B	RTX 4090 (24GB)	52 tok/s	130 tok/s	2.5×
DeepSeek 32B	AWS g5.12xlarge	22 tok/s	50 tok/s	2.3×
Mistral 7B	AWS g5.xlarge	28 tok/s	88 tok/s	3.1×
Llama 3.3 70B	AWS g5.48xlarge	23 tok/s	46 tok/s	2.0×

Small models gain 2.5-3× in speed; large models gain ~2×. At 8 bits, the accuracy loss is usually minimal; at 4 bits, a bit larger but generally acceptable for most tasks, with a memory saving of about 75%.

When to use it: almost always, unless you need maximum numerical precision (finance, complex math) or the model already fits comfortably in VRAM with enough speed in FP16. In practical terms, 4 bits is usually a good starting point for most cases; if consistent failures show up on specific tasks, it is worth stepping up to 6 or 8 bits.

Continuous batching: maximize GPU use with multiple users

Instead of processing one request at a time, you group several to process them in parallel, using the GPU’s capacity.

Static batching (the problem): fixed batch size — you have to wait for the longest sequence in the batch to finish before continuing, a real bottleneck. Continuous batching (the solution): batch size is decided on every iteration; as soon as a sequence finishes, a new one is inserted in its place. Frameworks such as vLLM and TGI handle this internally.

Analogy: a chef cooking 4 dishes at once. With static batching, you wait until you have 4 orders before you start — if one is complex, the other 3 wait. With continuous batching, as soon as one dish is ready, you start the next without waiting for the others.

The trade-off, with an illustrative example (not measured figures, only to show the direction of the effect): without batching, low latency but limited throughput (few requests per second); with continuous batching, individual latency rises somewhat, but total throughput grows much more — the GPU, which has a fixed cost, is used much better.

When to use it: multiple concurrent users, throughput matters more than individual latency, you want to maximize use of a GPU whose cost is already fixed. When not to: latency of each request is critical (live chat with a single user), or you have very few concurrent users, where the overhead of coordinating the batching may not pay off.

KV Cache: avoid recomputing what you already computed

An LLM generates token by token; without optimization, each step would recompute attention scores over the entire accumulated context — prohibitively slow. The KV Cache stores and reuses the attention computations (keys and values) of previous tokens, instead of recalculating them.

Advanced techniques:

Technique	What it does	Benefit
PagedAttention (vLLM)	Stores the KV cache in non-contiguous memory pages, like an operating system	Reduces fragmentation, uses memory more efficiently
Quantized KV Cache	Stores keys and values at lower precision (Q4, Q8)	Less memory, slight accuracy loss
Chunked Prefill	Splits prompt processing into chunks, run alongside generation	Improves throughput under high concurrency
Prefix Caching	Caches and reuses the computation of prefixes shared across prompts (e.g. the system prompt)	Cuts TTFT dramatically in systems with long, repeated prompts

When to use it: practically always — it is standard in every modern inference framework, and especially valuable with long text (>512 tokens) or extensive context (>8K). The trade-off is extra memory in exchange for speed — if VRAM is scarce, it can become the bottleneck itself, and then it is worth reducing the maximum context or quantizing the cache.

Combining everything: why the techniques do not compete

Technique	What it solves	Cost (trade-off)
Distillation	Model too large/slow in general	Accuracy loss (variable)
Quantization	Memory use and compute speed	Accuracy loss (smaller than distillation)
Continuous batching	Low throughput with multiple users	Higher latency per individual request
KV Cache	Redundant recomputation on long generation	Higher memory use

A typical deployment combines: a distilled and/or quantized model, served with a framework (vLLM, TGI) that implements continuous batching and KV cache automatically, with PagedAttention and chunked prefill to maximize memory efficiency, and prefix caching if the system prompt is long and repeated — optionally, with speculative decoding to speed things up even more.

Illustrative configuration example, for an HR chatbot (reference numbers, not measured on a real deployment): a medium-size distilled model, quantized to 4 bits, on a single high-end consumer GPU, with 16K context (enough for HR policies), continuous batching, and KV cache at 8 bits. Under this setup it would be reasonable to expect a substantial improvement in both speed and cost per request, compared with relying exclusively on an external API for every call — the exact magnitude depends entirely on your model, your real traffic, and your specific hardware, and must be measured, not assumed from a generic table.

The decision framework

Question	If YES…	If NO…
Does the model not fit in VRAM?	Quantization or distillation	Quantization is still recommended, even if not urgent
Is it >70B and very slow?	Distillation (to 32B/14B) + quantization	You can prioritize other techniques
Do you have >10 concurrent users?	Continuous batching (vLLM/TGI)	Batching may not be necessary yet
Do you generate long text (>512 tokens)?	KV Cache with PagedAttention	KV cache is still useful, even if less critical
Is the cost per token too high?	Distillation + quantization	You can prioritize speed over cost
Is latency (TTFT) the problem?	Prefix caching, or reduce the context	The bottleneck is generation (tok/s), not startup
Do you need maximum accuracy?	8-bit quantization or no quantization; avoid distillation	4 bits is usually enough

A reasonable starting point: quantization is usually the cheapest optimization to implement with the best benefit/cost ratio, so start there. If the model is still too slow or too large, distillation is the next logical step. If there are multiple concurrent users, continuous batching becomes relevant. And KV cache, in practice, should almost always be on — it is a standard optimization, not a case-by-case decision.

Connection with Part II and with security

With agent architecture (Part II): a ReAct agent that does 5 turns generates several times more tokens than a simple answer — optimizing each turn matters more than optimizing a single isolated response. A coordinator adds an extra call just to classify intent, with its own latency and token cost — if that classification is slow, the whole system suffers.

With security and observability: quantization can affect model accuracy on sensitive tasks (classifying a complaint as urgent, for example) — if the model loses too much precision, it can fail to detect something important. Continuous batching adds latency per request, which can matter in systems that need to respond fast to a risk situation. Monitoring these metrics in production is central to detecting performance degradation over time — something we will cover in depth in the LLM observability and EDD chapter later in this part.

Chapter summary

Distillation: a small model imitates a large one — trade-off between size and accuracy, with clearly diminishing returns at the smallest sizes.
Quantization: less numerical precision, less memory, more speed — almost always worth applying to some degree.
Continuous batching: processes multiple requests in parallel — improves throughput, at the cost of some individual latency.
KV Cache: reuses attention computations already done — saves time, consumes memory.
The four techniques do not compete — a real production deployment combines them.

When a company asks “why is our chatbot so expensive or so slow?”, the answer is almost never “you need a larger model” — more often it is “you need to apply these four levers, in the order your specific situation justifies”.