Latency, Throughput, Rate Limits, and Batch — How They Relate in Production

These four concepts get confused constantly because they are entangled, but each one answers a different question.

The four questions

Concept	Question It Answers	Who Controls It?
Latency	How long does one individual request take to complete?	Outcome — you do not tune it directly
Throughput	How many requests does the system process in total per unit of time?	Outcome — you do not tune it directly
Batch size	How many requests are processed together in a single pass?	You configure it (only if you self-host inference)
Rate limit	What is the maximum ceiling allowed in a time window?	A third party imposes it (the provider), or you do internally

Latency and throughput are measurements. Batch size and rate limit are levers and constraints.

Batch size: the lever that moves throughput and latency in opposite directions

A GPU is massively parallel; processing a single request wastes capacity. Grouping requests into a batch uses that idle capacity. Mechanical trade-off:

• Raise the batch → throughput goes up (less idle capacity)
• Raise the batch → individual latency goes up (each request shares the compute pass)

This knob only exists if you control the inference layer (self-hosted). With a hosted API, the provider does batching internally by mixing traffic from thousands of clients.

Illustrative example (not real): batch 1 → 9 req/s at 107ms; batch 8 → 51 req/s at 156ms; batch 64 → 117 req/s at 548ms. Throughput grows with diminishing returns (the GPU approaches its compute limit); latency grows steadily and accelerates toward the end.

Rate limit: it is not the same as throughput, it is its ceiling

Throughput is a measurement. Rate limit is a policy. The rate limit puts a ceiling on throughput — but real throughput can sit well below the limit or be pinned exactly against it.

It is like a speed limit on a highway: the sign does not say how fast you are going, it only puts a ceiling.

There are almost always two limits: RPM (requests per minute) and TPM (tokens per minute) — you can hit either one first. Long requests can pin TPM long before you get close to the RPM limit.

Example with concrete numbers: rate limit of 1,000 requests/min. Demand per hour: 8h: 600, 9h: 1,500, 10h: 900, 11h: 500, 12h: 300, 13h: 250, 14h: 400, 15h: 600, 16h: 450, 17h: 300, 18h: 150. At 9am, demand (1,500) exceeds the limit (1,000) — the 500 excess requests queue or get rejected, and whoever sent their request in that peak experiences more latency than someone who sent it at 11am.

How the four concepts connect, end to end

1. Demand is what the business generates — you don’t control it directly, you only forecast it.
2. The rate limit is the ceiling imposed on that demand.
3. When demand ≤ rate limit → throughput simply equals demand.
4. When demand > rate limit → the excess queues, and that queue time becomes extra latency.
5. Batch size (if self-hosted) decides how much real capacity the system has to absorb peaks before it needs to queue.

Designing against this: real options

• Queue with priority: synchronous requests (user waiting) first; asynchronous ones (overnight processing) after.
• Buy more rate limit: enterprise tiers usually offer much higher RPM/TPM.
• Flatten demand: stagger notifications or cache recent results to lower the peak without touching infrastructure.
• Raise your own batch (only if self-hosted): raises the real absorption ceiling, at the cost of latency.

An applied case: from 10 users to 1,500 salespeople

With 10 users evaluating clients and drafting emails: the challenge is judgment and quality, not throughput. Recommended architecture: Internal app (CRM/email plugin) → Orchestration (builds the prompt with rules) → branches to Client data (CRM/DB) and AI model (hosted API) → Human review before sending. There is no reason to self-host anything — a hosted API already solved inference at a scale 10 users would never reach.

With 1,500 salespeople, the problem changes nature: throughput, cost, and governance become real.

• Human review as a separate approver no longer scales — it becomes an automatic filter (guardrails) + the salesperson’s own auto-approval before sending.
• Provider rate limits get hit at peaks (e.g. everyone reviewing their list at 9am).
• Cost per token, multiplied by 1,500 daily users, becomes a real budget line.
• You cannot audit every interaction by hand — you need real observability.

Scaled architecture: App/CRM → AI gateway (limit, cache, routing — the new piece that was not needed before) → branches to Fast model (evaluates clients) and Capable model (drafts emails) → Guardrails and review (automatic check + salesperson approval).

New pieces and why they exist:

• AI gateway: rate limiting (queue/space out requests), prompt caching (reuse the instruction block shared by the 1,500 salespeople instead of reprocessing it every time), routing between models.
• Two model tiers: evaluating clients (a more structured task) can be solved with a cheap/fast model, possibly fine-tuned specifically; drafting emails (tone/quality matter for reputation) justifies a more capable model.
• Automatic guardrails: review is no longer “someone else approves” — it is an automatic filter (does it mention improper data? did it invent a figure?) followed by the salesperson themselves approving before sending.

What is missing from the diagram but matters just as much: observability and costs per team/salesperson; audit by sampling (not full review) instead of reviewing every email; redundancy against API provider outages.

Synchronous vs. asynchronous, the distinction that changes the design:

	Live evaluation (salesperson waiting)	Overnight load of new leads
Pattern	Synchronous	Asynchronous
Latency tolerated	Low (seconds)	High
Priority	Low latency for the fast model	Maximum throughput
Batch config (if self-hosted)	Small/moderate batch	Large batch

There is not a single batch configuration in the system — the same fast model could run with a conservative batch during business hours and an aggressive batch overnight.

Prompt caching / KV cache landed here: the 1,500 salespeople share the same base instructions — identical context repeated 30,000 times a day. With prompt caching (hosted API) or a reused KV cache (self-hosted), that block is processed once and reused, cutting cost and latency directly.

When to reconsider self-hosting: the right question is real volume, not user count — how many calls per day, of what size? If the most frequent task (evaluation) generates tens of thousands of daily calls with a small model, the cost math can tilt toward self-hosting that specific model (e.g. with vLLM), while email drafting (less frequent, needs more quality) stays on the hosted API. Hybrid pattern: each task with the infrastructure its volume and quality need justify.

---