How Many Tokens Per Second You Actually Need — And Why That Is the Wrong Question

You have designed an agentic system. The prompt is calibrated, the evals are running, and the agent flow works on your machine. Production arrives, and the first question is: “what hardware do I need?” And the second: “how many tokens per second do I need?”

The short answer: it depends on your use case. The long answer, which is the one that matters: tokens per second is only one metric, and not the most important one.

The minimum number depends entirely on the use case

There is no universal tokens/second (tok/s) threshold — it depends on whether a human is waiting or not.

Use Case	Typical Usable Range	Why
Interactive chat	10–20 tok/s	The speed at which a person can read while the answer is being generated
Interactive programming	30–40 tok/s	Code has a lot of structural repetition; faster feels more fluid
Intensive Agentic Programming	60–100+ tok/s	Multiple tool calls pile up latency; every extra tok/s shows
Unattended Agent / Overnight Runs	Less than 1 tok/s can be enough	With no human waiting, cost and reliability matter, not raw speed
RAG With Many Documents	>10 tok/s	The prefill (processing the documents) usually dominates total latency
Batch Processing	Depends on Volume	Total speed (tokens/hour) matters more than speed per request

The practical takeaway from this table: if a human is waiting, aiming above 20 tok/s makes sense; if it is a background process, it is worth more to optimize for cost than for speed.

Tokens per second is not enough: other metrics that matter as much or more

A model can generate at 50 tok/s and still feel unbearably slow if it takes 30 seconds to produce the first token.

Time to First Token (TTFT): the time from when you send the prompt to the first token of the answer. For long prompts (a 10,000-token document), TTFT can be 5-10 seconds — the user experiences that as “slowness”, even if generation is fast afterward. You optimize it with prefix caching (see below).

Prefill — the speed of processing the prompt: before generating, the model has to compute the attention keys/values for the entire input prompt. In large models, this can dominate total latency. Models with Flash Attention (Part I, Anatomy of a transformer) cut prefill time dramatically.

KV cache and long context: the model stores the keys and values of previous tokens so it does not recompute them — but the cache grows with context, and generation slows down. A model can produce 50 tok/s with an almost empty context, and slow down dramatically after 50,000 accumulated tokens — reporting tok/s without specifying context length can be misleading. A quantized KV cache (Q8, Q4) reduces memory and speeds things up on long contexts.

Prefix caching: if many prompts share the same prefix (a long system prompt, repeated on every call), that computation can be cached and reused — cutting TTFT and cost dramatically. Services such as Anthropic and OpenAI, and backends such as vLLM, support it. If your system reuses the same system prompt across many queries, this is one of the highest-ROI optimizations available.

Tool-call latency (agentic flows): an agent that calls tools via MCP adds network time, processing, and waiting on every call. Token generation can be excellent, but if each tool call takes 500ms, the overall experience is slow. You mitigate this by running tools in parallel when possible (parallel pattern, Agentic architecture patterns) and by choosing low-latency tools.

Total time to complete the task: in the end, what matters is how long the whole task takes, not each individual token.

Worked example: a ReAct agent does 4 turns (4 model calls), each generating 200 tokens. That is 800 tokens total — at 50 tok/s, that is 16 seconds of pure generation. If each tool call takes 300ms and there are 4 calls, that adds 1.2 extra seconds. Total task time is ~17 seconds, even with a generation speed that, in isolation, looks excellent. The lesson: do not optimize tok/s in isolation — measure end-to-end time of your full agentic flow.

The inference engine matters as much as the hardware

Backend	What it is	When to use it
GGUF (llama.cpp)	Efficient quantization and execution on CPU and GPU	Large models on limited hardware; broad support
MLX (Apple Silicon)	Backend optimized for Apple M chips	Native and fast on Mac
vLLM	Inference server with batching and prefix caching	Production at scale, multiple concurrent users
TGI (Hugging Face)	Similar to vLLM, integrated into the HF ecosystem	Alternative to vLLM with good support for popular models
Speculative decoding / MTP	A small model generates candidates, a large one verifies them	Can speed things up 2-3×, with extra complexity

Real example: the same RTX 3090 can deliver ~70 tok/s with a 27B model + MTP on an optimized backend, versus ~40 tok/s with the same model on another backend without that optimization. It is worth trying different backends with your own model and use case before investing in new hardware — sometimes the cheapest improvement is software, not silicon.

The hardware spectrum

Consumer GPU (RTX 3090/4090/5090): runs quantized 27B-35B models with moderate context. RTX 3090 (24GB): ~70 tok/s with a 27B + MTP. RTX 4090 (24GB): ~40 tok/s with a 36B Q4 GGUF. RTX 5090 (32GB): >100 tok/s with optimized models. Advantage: best performance/price ratio for interactivity. Disadvantage: limited VRAM — models >70B need aggressive quantization or simply do not fit.

Apple Silicon (Mac Studio/Pro, M4/M5 Max): runs larger models (70B+) thanks to 64-128GB of unified memory. M5 Max: ~40 tok/s at 8-bit precision via MLX. Advantage: models that do not fit on a 24GB GPU, with no copy between CPU/GPU. Disadvantage: slower than a dedicated GPU for models of comparable size.

Multi-GPU (server workstations): giant models (70B+, 100B+) with long context — 4-8 high-end cards. Advantage: maximum capacity. Disadvantage: cost and complexity — it stops being a “gaming” setup and becomes a real AI server.

Specialized secondary models — the most actionable pattern: combine several models instead of hunting for one perfect one. A fast model (~27B) for implementation and routine changes; a large one (~70B+) for planning and hard debugging; small specialized models (~7B) for summarization, speech-to-text, or classification. Example: use a mid-size model for 80% of the tasks (fast and cheap), reserving a large model only for the 20% that actually needs it.

The Q4 vs. Q6 debate

Quantization shrinks the model at the cost of some precision — the debate over which level to use is constant in the community. In relative terms: Q4 compresses more and is faster, with a bit more precision loss; Q6 compresses less, is a bit slower, and preserves more fidelity; Q8 approaches full precision, at the cost of more memory and reduced speed. The exact figures for “how much” precision you lose vary widely by specific model and task — there is no reliable universal percentage that applies to every case.

Is Q4 enough? It depends on the task: for chat, summarization, and general reasoning, it is usually practically indistinguishable from full precision. For mathematical code or complex reasoning, Q6 can make a real difference. For tasks where numerical precision is critical (financial calculations), full precision or Q8 is the safer choice.

The practical decision: on a 24GB GPU, Q4 can let the entire model fit in VRAM, while Q6 might force a smaller context or offload to CPU. There is no universal answer — trying both on your own specific tasks is still the only reliable way to decide. In practice, Q4 is usually a reasonable starting point for most cases; if consistent failures show up on certain tasks, it is worth stepping up to Q6 or 8 bits.

Cloud vs. local: the debate nobody wants to have

Option	Advantages	Disadvantages
API (OpenAI, Anthropic)	No maintenance, automatic scale, latest-generation models	Cost per token, privacy considerations, provider lock-in
Managed cloud (Bedrock, Vertex)	More control than a pure API, less than local; integration with the rest of your cloud infrastructure	Cost, configuration complexity, provider lock-in
Local (your own server)	Full privacy, predictable cost if you already have the hardware	Maintenance, scaling, need for infrastructure expertise

The practical decision: for small teams or prototypes, an API is usually cheaper than buying hardware and faster to ship. For medium scale or sensitive data, a managed cloud or a small local server (1-2 GPUs) balances control and cost. For large scale or highly sensitive data, a local multi-GPU server gives full control. In general, do not invest in your own hardware until you confirm — with measured real usage — that the cost justifies it; starting with an API and deciding later is usually the lower-risk path.

The complete decision framework

Question	If YES…	If NO…
Is a human waiting for the answer?	Aim for >20 tok/s, prioritize low TTFT	Prioritize cost per token, not speed
Does the model need long context (>32K)?	Consider Apple Silicon or multi-GPU (more memory)	A 24GB GPU may be enough
Does the model have more than 70B parameters?	Multi-GPU or Apple Silicon (unified memory)	A 24GB GPU with quantization may be enough
Does the task require maximum precision?	Use Q6/Q8 or full precision	Q4 is usually enough
Do you have your own infrastructure team?	Local can give more control and lower long-term cost	Cloud or API means less maintenance
Is your data highly sensitive?	Local or private cloud	An API can be acceptable with adequate confidentiality contracts
Do you expect to scale to hundreds of users?	Dedicated server (vLLM, TGI) with batching	A single GPU may be enough for a few users

Connection with Part II: the same tension, applied to performance

The fundamental tension of Part II — autonomy and verification grow together — shows up here in performance terms: a high-autonomy agent (ReAct, coordinator) makes more model calls, multiplying latency and cost — every reasoning turn adds TTFT and generation. A low-autonomy agent (sequential, parallel) is more predictable and faster, but less flexible.

Returning to the worked example above: a 5-turn ReAct agent can take 20 seconds even if it generates at 50 tok/s. Is the flexibility worth it? It depends on the case — if the user is waiting live, a faster sequential pipeline may be preferable; if the task is genuinely complex and there is no alternative, ReAct’s flexibility becomes necessary despite the time cost.

In Part III we will keep seeing how to monitor these metrics in production, detect performance drift, optimize cost with batching and prefix caching, and keep security even when performance degrades.

Chapter summary

• There is no magic tok/s number — it depends on whether a human is waiting and on the complexity of the task.
• Tok/s is not the only metric: TTFT, prefill, long context, tool calls, and total task time matter as much or more.
• Software matters as much as hardware — trying different backends before buying is almost always worth it.
• The hardware spectrum is wide: consumer GPU (fast, limited memory), Apple Silicon (more memory, slower), multi-GPU (maximum capacity).
• Combining models — a fast one for most tasks, a large one only when needed — is usually more efficient than hunting for a single perfect model.
• Starting with an API and migrating to your own infrastructure only when cost or privacy justifies it reduces the risk of over-investing early.

The right question is not “which model produces tokens fastest?” — it is “which setup completes the work accurately, reliably, and fast enough that you are not spending the session waiting or fixing what it broke?”