The Inference Trilemma: Latency, Throughput, and Cost — The Art of Choosing What to Sacrifice

In the previous chapters we saw how to optimize inference (distillation, quantization, batching, KV cache) and how to manage demand in production (latency, throughput, rate limits). But there is a background problem that no optimization solves on its own: you cannot optimize all three at once.

The trilemma: you cannot optimize all three at once

Dimension	What It Means	What Happens If You Prioritize It
Low Latency	Each individual request is fast	You need to over-provision capacity → high cost
High Throughput	Many requests per second	Aggressive batching adds latency → high latency
Low Cost	Little spend on hardware/inference	You accept lower throughput, higher latency, or both

Illustrative example (typical qualitative pattern of a 70B model on high-end GPUs, not measured figures from a specific deployment):

Configuration	Latency (p95)	Throughput (req/s)	Relative Cost
Batch=1, unoptimized	Low	Low	Baseline
Batch=64, optimized	High	High	Same (same hardware)
2× GPUs, batch=1	Low	Medium	Double
Smaller model	Very low	Very high	Much lower (but with lower quality)

No configuration wins on all three dimensions at once. The trilemma is real: you always have to choose what to sacrifice.

Why LLM inference breaks normal software paradigms

They are stateful: in a typical REST API, each request is independent. In an LLM, context grows with every token generated — the KV cache (Inference optimization) stores the attention state for all previous tokens. That means you cannot load-balance by sending each request to any replica indistinctly (each replica’s cache is different), and switching replica mid-generation means losing that state.

They are memory-bound, not compute-bound: in most software, the bottleneck is compute. In LLMs, it is memory bandwidth — the model weighs gigabytes, and every generated token requires moving all those weights from memory to the processor. The time to move the weights dominates the cost, not the math operations themselves. That is why a GPU that is faster in FLOPS helps little if memory bandwidth does not improve, quantization speeds things up by reducing how much you have to move, and batching works by sharing that fixed cost across several requests.

It depends intimately on the physical hardware: performance is not just “GPU A is faster than GPU B” — it depends on interconnect topology (NVLink, PCIe), memory bandwidth (HBM vs. GDDR), L2 cache size, and the CUDA/driver version. Two GPUs of the same model can perform differently depending on the motherboard and the cooling.

The four engineering decisions that set the performance ceiling

Model architecture: dense vs. MoE

Type	Description	Implication for Inference
Dense	All parameters activate per token (70B → move ~140GB of weights in FP16 per step)	Bottleneck = memory bandwidth; simpler to deploy
MoE (Mixture of Experts)	Only a fraction activates per token (e.g. a 671B total model with 37B active)	Fewer active weights per token, but requires communication between GPUs — more complex

A dense 70B model does not fit in FP16 on a single 80GB GPU (it needs ~140GB) — you need quantization or multiple GPUs. A large MoE needs several GPUs from the start, but each token activates only a fraction of the total parameters, which can be faster on certain tasks despite the larger total size. With a single GPU, a quantized dense model is usually the simpler option; with a cluster available, an MoE can be more efficient.

Quantization, going deeper

Precision	Relative size (e.g. 70B model)	Relative speed	Accuracy loss (order of magnitude)
FP16/BF16	140GB (baseline)	1×	None
FP8	~70GB	1.5-2×	Very small
INT4 (Q4)	~35GB	2-3×	Perceptible on demanding tasks
INT4 + adapter trained with QLoRA	~35GB	2-3×	Variable; the serving format and the fine-tune must be evaluated separately

Note on the accuracy-loss figures: exact percentages vary widely by model, task, and specific quantization method — there is no reliable universal number. The order relationship (FP16 no loss < FP8 minimal loss < INT4 more noticeable loss) is a consistent, well-documented pattern.

FP8 is usually the reasonable starting point for deployments on compatible production hardware — a good balance between memory and accuracy. INT4 makes sense when hardware is limited, always testing before trusting it on critical tasks.

Parallelism strategy

Strategy	What it does	Best for	Effect on latency
Tensor parallelism	Splits the model layers across GPUs	Minimizing latency on models that do not fit on a single GPU	Reduces latency per request
Expert parallelism	Distributes the “experts” of an MoE across different GPUs	Large-scale MoE deployment	High efficiency, less unnecessary communication
Data parallelism	Full replicas of the model, each processing different requests	Maximizing total throughput	Does not speed up individual latency

Illustrative example (typical pattern, not a specific measurement): a 70B model on 2 GPUs with tensor parallelism tends to give lower individual latency but more modest throughput; the same model with data parallelism tends to give noticeably higher throughput at the cost of higher individual latency. The practical choice: tensor parallelism when latency matters more, data parallelism when throughput matters more, and expert parallelism specifically when you deploy an MoE.

Batching and the “elbow of the curve”

Loading the model weights is a fixed cost on every generation step — batching amortizes it by spreading that cost across several requests processed together.

Illustrative example (typical behavior pattern, not a measurement from a specific benchmark):

Batch size	p50 latency	Throughput (req/s)	Zone
1	Very low	22	Memory-bound
4	Low	83	Memory-bound
8	Low	154	Good balance
16	Moderate	258	The elbow begins
32	Medium-high	376	Throughput peak
64	High	356	Throughput no longer grows — it starts to drop
128	Very high	284	Compute-bound — throughput falls below the peak

An important nuance this table reveals, and that is worth highlighting: past the elbow, throughput does not just stop growing — in this pattern, it falls. It is not only “diminishing returns”; it is directly counterproductive to keep raising the batch past a certain point. Finding the elbow of the curve (the point where throughput stops improving) is the optimal batch configuration for your specific hardware — and confirming where that point sits requires measuring on your own system, not assuming a generic value.

The four real dimensions of cost

The cost of an AI system is not just the price of the GPUs:

Dimension	What it is	The hidden risk
CapEx	Hardware cost	GPUs with fast interconnect are usually sold in fixed blocks — you cannot buy “half a unit”
OpEx	Energy and cooling	You pay the hardware bill even when it is unused — the “idle tax”
Opportunity cost	Idle capacity on intermittent traffic	Every minute without processing tokens is capacity already paid for and wasted
Engineering cost	Talent to configure and maintain the system	Expensive time hunting for the optimal configuration — the “benchmarking tax”

Illustrative order-of-magnitude example (not verified figures from a real case — the structure of the argument is what matters, not the exact numbers): a hosted API can have zero hardware cost but a notable integration cost; a small self-hosted deployment with a few GPUs has moderate hardware and operations cost; a large self-hosted deployment has high hardware cost, and the engineering cost to run it well can end up comparable across all three options, even when the hardware cost is very different. The real lesson, independent of the exact figures: the cheapest hardware on paper is rarely the most profitable once you factor in the talent needed to operate it well — sometimes the API turns out cheaper in the long run, even if the cost per individual token is higher.

Two worlds, two completely different strategies

Latency-sensitive world (chatbots, real-time agents): the priority is that the user does not wait. Key metrics: TTFT and inter-token latency. Strategies: tensor parallelism, small batches (1-8), speculative decoding, chunked prefill.

Throughput-sensitive world (massive offline generation, batch summaries): the priority is to process the maximum amount of data at minimum cost. Key metric: tokens per dollar. Strategies: large batches, data parallelism, treat queuing as an asset (keeping a constant queue ensures maximum GPU utilization), not as a failure.

7-step framework for choosing the right configuration

1. Characterize the load: how many requests per second at peak? what is the latency SLO? what size are typical prompts and answers?
2. Choose model and quantization: the smallest model that does the job; FP8 as the standard when the hardware supports it, INT4 only if the hardware requires it and you already tested that the loss is acceptable.
3. Measure on real hardware: your own benchmarks on at least two configurations — never trust published numbers without verifying on your own system.
4. Locate the elbow of the curve: the point where the system goes from memory-bound to compute-bound — the real useful limit of your configuration.
5. Size with margin: replicas for the expected peak, with extra buffer for unexpected peaks.
6. Calculate the real total cost: CapEx + OpEx + engineering cost, compared against the cost of a hosted API if the alternative exists.
7. Autoscale with judgment: reduce replicas in off-peak hours, accepting the cold-start latency when scaling up.

Connection with Part II: agents multiply the trilemma

Architecture (Part II)	Impact on the trilemma	Recommendation
Single agent	Fewer model calls → lower cost and latency	Use it when the task allows
ReAct (multiple turns)	Each turn adds latency and cost — the trilemma multiplies	Limit the number of turns; use a fast model for them
Coordinator	Adds an extra call to classify	Use a cheap model specifically for that classification
Multi-agent (home automation)	Multiple agents = multiple calls, each one affecting the trilemma	Run in parallel when possible; vary model size by each agent’s task
Human-in-the-loop	Human wait dominates total latency, not inference	Allows more aggressive batch configurations, because inference latency stops being the critical bottleneck

The 1,500-salesperson case, revisited: the fast client-evaluation model prioritizes throughput and cost (self-hosted, large batch in off-peak hours); the capable email-drafting model prioritizes quality and latency, because the salesperson is waiting for the answer (via API, or self-hosted with tensor parallelism). Each piece of the system sits at a different point on the same triangle, according to what that specific piece needs.

The consulting value

When someone asks “which model is better?”, the honest answer is usually: it is not a model question, it is a trilemma question. The productive conversation is: “given your real traffic pattern, your latency tolerance, and your budget, where on the trilemma should we stand, and which specific configuration gets us there?”

Quick diagnosis: if the system is slow, measure TTFT and tok/s — you are probably missing inference optimization. If it is expensive, measure cost per request — distillation or a smaller model is probably worth it. If it does not scale, measure the rate limit and batching behavior — you probably need continuous batching or more replicas. There is no universal answer — there is a diagnostic process that depends on the specific case.

Chapter summary

• The trilemma is real: latency, throughput, and cost do not get optimized all three at once — you have to decide consciously what to sacrifice.
• LLMs are different from traditional software: they have state, they are limited by memory (not by pure compute), and they depend intimately on the physical hardware.
• Four decisions set the performance ceiling: model architecture (dense vs. MoE), quantization, parallelism strategy, and the optimal batching point.
• Real cost has four dimensions, not just the price of the GPUs — and the cheapest hardware on paper is not always the most profitable in practice.
• There are two worlds with opposite strategies: optimize for latency (interactive) or optimize for throughput (massive processing) — rarely both at once.
• A 7-step framework helps choose the right configuration: characterize, choose, measure, locate the elbow, size, calculate total cost, and autoscale.
• Part II agents multiply the trilemma’s impact — more autonomy almost always means more calls, and therefore more pressure on all three corners of the triangle.

The final question is never “which configuration is best?” in the abstract — it is “which configuration is best for my use case, with my budget and latency constraints?” The answer is always a conscious balance, not a perfect solution.