Case Study: Scaling an AI Agent to 1,500 Sales Reps

The business problem (told by the person who lives it)

“I’ve been in sales for 15 years. My job is to know the client, understand what they need, and convince them we’re the right solution. But every new lead means hours of research, email drafting, and follow-up. It is not that I do not want to do it — it is that every hour I spend drafting is an hour I do not spend in the field, meeting another client.”

— Carlos, Sales Director, technology consulting firm

Carlos did not ask for an AI system. Carlos asked for more time to sell. His team of 10 salespeople was overwhelmed: every new lead required researching the company, understanding its industry, finding the right angle, and drafting a personalized email that did not sound like a template. The process worked, but it did not scale. And the company had just closed a funding round that was going to triple the sales team in 18 months.

The challenge was clear: automate the research and the initial draft without losing the human touch that closed the deals. But, as we will see, the solution was not a single system — it was an architecture that would evolve with the business.

The central principle that guides this case

Before we get into the technical details, it is worth remembering a lesson that has shown up again and again in this compendium: the right architecture is a function of volume and of the real usage pattern — not of a generic template for “how you build an AI system”.

What is over-engineering with 10 users (AI gateway, two separate models, automatic guardrails, self-hosting) becomes necessary with 1,500. And what is enough with 10 users (a single model, manual review, no prompt caching) becomes a bottleneck that can take the system down at scale.

This case is an exercise in evolutionary architecture: we do not design for the future, we design for the present, with a clear roadmap to the next step. Each stage of the journey has its own optimal architecture — and the transition between stages is a conscious decision, not an emergency.

Stage 1: Ten business users (the beginning)

“I need my 10 salespeople to be able to evaluate leads and send personalized emails. Fast. We do not have time to build infrastructure.”

— Ana, Head of Product, 3 weeks before launch

Diagnosis before designing

Dimension	Analysis	Architectural Implication
Concurrency	10 internal users, probably not all writing at the same time	It’s not a throughput problem — it’s individual latency, because each person waits live for their answer
Nature of the Task	Two things mixed: evaluating clients (analysis/classification) and generating emails (text with a specific tone)	We need two distinct capabilities, but we can start with a model that does both
Data Sensitivity	“Evaluating new clients” implies personal or commercially sensitive data	Privacy isn’t negotiable — if we use an external API, we need confidentiality contracts
Consequence of Error	A badly drafted email has a direct reputational impact	This asks for a human-review layer, not blind automation

The most important decision: do not self-host anything

With only 10 users, building your own inference infrastructure (vLLM, GPUs, quantization, continuous batching) is using a cannon to kill a fly. Everything we saw about inference optimization in Part III matters when you are responsible for serving thousands of concurrent requests efficiently. With 10 users, a hosted API provider already solved that problem at a scale that would never be reached here.

The cost math was blunt:

Hosted API: ~$0.02 per request (evaluation + email). 10 users × 20 requests/day × 22 days = 4,400 requests/month → ~$88/month.
Self-hosted (one GPU): $3,000 in hardware + $200/month in electricity + maintenance → more than $5,000 in the first year.

Conclusion: paying per token was radically cheaper than keeping a GPU server running 24/7 for this volume.

The initial architecture

What each layer solves:

1. Internal app — the principle of least friction: we did not build a new app. We built a plugin inside the CRM the salespeople already used. Less friction (they do not learn a new interface), more adoption (if it is where they already work, they will use it), and integrated context (the plugin automatically extracts the lead data from the CRM). The lesson: the best interface is the one that does not look like a new interface.

2. Orchestration layer — the brain of the system: this is where all the structured prompt engineering from Part II is applied — system role, hard constraints, company examples, output format, and data sanitization.

The system prompt (simplified):

You are a sales assistant specialized in technology consulting.
Your mission: help the salesperson evaluate leads and draft personalized emails.

STRICT RULES (you cannot violate them):
1. Do not invent information about the client. Use only the data available.
2. Do not promise discounts, features, or timelines that are not in the catalog.
3. Do not use sexist, discriminatory, or inappropriate language.
4. Do not copy templates word for word — personalize every email.

GOLD-STANDARD EMAIL EXAMPLES:
[3 examples of successful company emails, anonymized]

OUTPUT FORMAT:
EVALUATION: [2-3 paragraphs about the client and their industry]
EMAIL: [full email text, ready to send]

Client context:
{crm_data}

Salesperson instruction:
{salesperson_input}

3. Client data — RAG + MCP, the distinction made live: general knowledge (RAG) — the sales playbook, success cases by industry, product catalog, vectorized once and queried by semantic similarity (RAG, Part II). If a lead is in the pharmaceutical industry, the system retrieves pharmacy success cases, not retail. Live data (MCP) — CRM queries about the specific lead (is there already an open opportunity? what was the result of the last call?), queried in real time via MCP (MCP, Part II), exactly like the vacation balance of the HR assistant.

4. AI model — a single model for both tasks: with 10 users, there is no need to split models. A capable model can do both the evaluation and the drafting in the same call.

5. Human review — the red line: an email going out to a real client should not auto-send without human confirmation — the same logic as the vacation agent from Part II: if something goes wrong, the human is responsible, not the model. The flow: the model generates the email → the system shows it to the salesperson in the CRM → the salesperson reads, edits if needed, and clicks “Send” → the system logs the send and updates the CRM. If the salesperson does not review, the email simply does not go out — there is no auto-approval.

What agentic architecture pattern is this?

With the vocabulary of Agentic architecture patterns (Part II), this is the sequential pattern: evaluate the client first, use that result to inform the tone and content of the email, in a fixed order that does not change from one client to another. Like every sequential pattern, the orchestration is code, not an LLM decision — no model needs to “reason” about whether to evaluate before or after drafting; that order is already decided at design time.

This marks a clear limit of when this design would stop being enough: if in the future a salesperson could write the system open, varied messages (“how did Juan do with the proposal?”, “put together a success case for this sector”) instead of a fixed flow, then you would need a coordinator with an LLM dynamically deciding which sub-flow to route each message to. As long as the task stays the fixed one of “evaluate, then draft”, sequential is the right choice.

The first 30 days: what they learned

Learning	What they did	Impact
Prompts need iteration	The first version produced emails that were too long and formal	They added more diverse examples and adjusted the tone
Human review is more valuable than they thought	Salespeople edited ~30% of the emails (small adjustments)	They decided to keep review in the next phase, not automate it
CRM data was inconsistent	Some leads had incomplete information	They added logic to “ask the salesperson if critical information is missing”
Latency matters	If the model took more than 5 seconds, salespeople got impatient	They optimized the prompt to reduce output tokens
RAG worked better than expected	Emails that cited relevant success cases had a higher response rate	They expanded the RAG knowledge base

The “aha” moment: a salesperson, after using the system for a week, said: “It does not take work away from me — it gives me time to do more work. I used to spend 2 hours a day drafting emails. Now I spend 20 minutes reviewing and sending. The rest of the time I spend talking to clients.” The system had not replaced the salesperson. It had amplified them.

Stage 1.5: The intermediate step (50 users)

“The pilot worked. The salespeople want more. The team is going to grow to 50 people next quarter. What do we have to change?”

The jump from 10 to 50 users is not as dramatic as from 10 to 1,500, but it introduces new considerations:

Dimension	Change	Implication
Volume	10 → 50 users (5×)	The API bill starts to be visible
Variety	More industries, more client types	RAG needs more documents and better segmentation
Concurrency	Sharper peaks	The API rate limit starts to matter
Personalization	Salespeople with different styles	The prompt needs to capture “brand style” more explicitly

The Stage 1.5 decisions:

Keep the hosted API — with 50 users the cost was still acceptable (~$500/month), and self-hosting was still more expensive.
Add prompt caching — since the system prompt was the same for everyone, they enabled prefix caching on the API. Savings: ~30-40% in cost and prefill latency.
Segment RAG by industry — instead of a single vector store, they created one per industry (pharmacy, retail, banking, manufacturing), limiting the search to the lead’s industry and improving retrieval accuracy.
Add a “tone filter” at the end of the pipeline — another step, with a cheaper model, verified that the tone matched the brand voice, suggesting corrections if it did not.
Collection of usage metrics — they started recording which salespeople used the system more, which industries generated more emails, and what the most common edits were — data that would be crucial for the 1,500 stage.

Stage 2: Scaling to 1,500 salespeople (the quantum leap)

“We have grown. 1,500 salespeople. 30,000 requests a day. The system that worked with 10 users is now the system the whole company uses. And it is starting to break.”

— Carlos, now VP of Sales

At this scale, the problem changes nature. With 10 users, the challenge was judgment and quality. With 1,500 salespeople, throughput, cost, and governance become real engineering problems.

What stops working as-is

Component	Problem at scale	Why it breaks
Centralized human review	It is not viable for one person to review the emails of 1,500 salespeople	The volume is too high
Direct API calls	Traffic peaks hit the provider’s rate limits	1,500 salespeople → 30,000 requests/day, peaks at 9am
A single model for everything	Cost per token becomes a real budget line	Evaluating clients is cheaper than drafting emails — paying the same for both is waste
Manual audit	You cannot manually audit every interaction	You need real observability (LLM observability and EDD, Part III)
No traffic control	A storm of requests from 300 simultaneous salespeople collapses the API	The provider’s rate limit is hit in minutes

The scaled architecture

The new pieces, and why each one exists:

1. AI gateway — the doorman. With 10 users, orchestration was simple: build the prompt and call the API. With 1,500, that single point needs three new responsibilities:

Rate limiting (queue and space out): the gateway keeps a request queue per salesperson and per time window. If a salesperson exceeds the limit (e.g. 30 requests/hour), their requests are spaced out; if the global system exceeds the provider’s limit, requests are queued and processed in order.
Prompt caching (reuse the compute): the system prompt (5,000 tokens) is processed once; the next 1,499 calls reuse that prefill, paying only for the new tokens — the prompt caching API providers offer.
Routing between models: if the task is “evaluate client” → fast model; if it is “draft email” → capable model. The routing is a fixed code rule, not an LLM deciding dynamically — we are still inside the sequential pattern, only now the decision is automated at the infrastructure level.

2. Two model tiers, not one. Evaluating a new client is a more structured task, closer to classification — it can be solved with a cheaper, faster model, possibly a small one fine-tuned specifically for that task (this is where LoRA/QLoRA from Part II stops being an educational exercise and becomes a real cost decision). Drafting the email, where tone and quality matter for reputation, justifies paying for a more capable model. If evaluation is 10× more frequent than drafting, and can be done with a 10× cheaper model, the savings are significant.

3. Automatic guardrails before human review. Review is no longer “someone else approves” — it is an automatic filter that checks: does the email mention data that should not be there?, does it use forbidden language?, did it invent a figure or a promise that is not in the client data?, does the tone match the brand voice? Only if the guardrail passes does the system show the email to the salesperson to review and approve. The AI never “owns” the final send — the salesperson remains responsible.

4. Recording the action via MCP. After the salesperson approves and sends, the system updates the CRM (marks the email as sent, logs the follow-up date) — a real-time action, on the MCP side, not RAG.

The infrastructure decisions (applying the trilemma)

Demand estimate: each salesperson generates ~20 requests/day → 30,000 requests/day in total. If 40% concentrates in the first 2 hours of the morning → 12,000 requests in 2 hours → ~1.7 requests/second on average. The real problem is the peaks: if 300 salespeople fire an evaluation in the same minute, you have to absorb ~300 concurrent requests in a short window — that peak number is the real throughput requirement, not the average.

Where each concept lives:

Concept	Where it applies	Who decides
Batch size	Inside the “Fast model” — only if it is self-hosted	The infra team, if it is self-hosted
Batching (invisible)	Inside the “Capable model” — the hosted API does it internally	The API provider
Throughput	The AI gateway — must absorb the peak without collapsing	The team, via gateway design and/or quota with the provider
Latency	What each salesperson experiences waiting for their evaluation or email	Result of the two previous decisions
Rate limit	The ceiling imposed by the provider (RPM and TPM)	The provider (negotiable on enterprise tiers)

The question that decides whether self-hosting is worth it: with 1,500 users, the “do not self-host anything” argument is no longer so obvious — but it also does not automatically become “yes, self-host everything”. The right question is real volume: how many model calls are generated per day, and of what size?

Client evaluation: frequent task (~80% of the calls), small model (7B-13B), tens of thousands of daily calls. Annual API cost: ~$20,000-$30,000. Self-hosting (a 24GB GPU, quantized model, vLLM) can cost ~$5,000 in hardware + ~$2,000/year in electricity → significant savings from the first year.
Email drafting: less frequent task (~20% of the calls), large model (70B+), needs quality. The API cost is acceptable because the volume is lower; self-hosting would require several GPUs or a smaller model with quality loss — better to keep it on a hosted API.

The final decision: hybrid pattern. Fast model (evaluation) self-hosted with vLLM, quantized, on a single 24GB GPU. Capable model (drafting) via hosted API, with prompt caching enabled. AI gateway self-hosted on a small server. RAG and MCP self-hosted (vector databases + MCP servers).

Observability and security at this scale

With 10 users, an occasional error was caught by the salesperson themselves when they read the email before sending. With 1,500, you need the same discipline we built for the HR assistant in Part III.

Golden datasets, separated by task: evaluation dataset (test clients with an already-known “correct” rating, run weekly to detect drift), drafting dataset (client profiles with “gold standard” emails, to validate that the model does not degrade), and guardrails dataset (examples of emails that should be blocked, to verify the guardrails do not weaken). Mixing these datasets would make it impossible to diagnose whether a quality drop comes from evaluation or from drafting — separate, they let you isolate the problem.

Agentic tracing: if a salesperson reports that the generated email attributed a need to a client that they do not have, you need to reconstruct the full chain — which chunks did the playbook RAG retrieve?, what data did the MCP query bring from the CRM?, what final prompt was passed to the drafting model? Without that trace, “the email came out wrong” does not say whether the problem was retrieval, live data, or the generation itself.

Red-teaming, before the scale launch: the team tested deliberate attacks — indirect injection in lead-capture forms with hidden malicious text, attempts to extract other clients’ data by manipulating the evaluation model, and tone manipulation via malicious prompts in the client data. Result: they found a real vulnerability in the processing of free-text CRM fields, and closed it before the general launch.

The real impact: what the numbers say (and what the salespeople feel)

After 6 months with the scaled system, the company measured the impact:

Metric	Before (no AI)	After (with AI)	Change
Evaluation time per lead	15-20 min	2-3 min	~80% less
Emails sent per salesperson/day	8-10	18-22	~2× more
Email response rate	12%	18%	+50%
Lead conversion rate	8%	11%	+37%
Time spent reviewing emails	2 min/email	30 sec/email	~75% less
Salesperson satisfaction (1-10)	6.2	8.7	+2.5 points

The testimony that matters:

“The system gives me superpowers. It is not that it does my job for me — it is that it lets me do more work, better work, and work that matters. I used to spend the day drafting emails. Now I spend the day talking to clients. And my numbers reflect it: I have closed 40% more deals this quarter than last.”

— Laura, salesperson, 3 years at the company

What the system did NOT do

It is worth noting what the system did not achieve (and that the company did not expect it to achieve):

It did not replace the salespeople. No salesperson was fired — on the contrary, the team grew from 10 to 1,500 in 18 months.
It did not eliminate the need for training. New salespeople still need to learn how to sell; the system gives them a base, not a shortcut.
It did not solve data problems. The system is only as good as the CRM data — if the data is bad, the emails are bad.
It did not eliminate human review. It is still the salesperson who approves and sends; the system is an assistant, not a substitute.

General lessons from the case

1. The right architecture is a function of volume. What is over-engineering with 10 users (gateway, two model tiers, automatic guardrails) becomes necessary with 1,500. And what is enough with 10 users (a single model, manual review, no cache) becomes a bottleneck. The key decision: knowing when it is time to scale the architecture, and doing it before the system breaks on its own.

2. Human review does not disappear when you scale — it transforms. With 10 users, a dedicated approver reviews everything. With 1,500, automatic guardrails filter, and the actor themselves reviews before acting. The principle: automation does not eliminate human responsibility — it redirects it to where it matters most.

3. Inference cost is an architecture decision, not an implementation detail you add later — it is decided from design, considering the real traffic pattern (synchronous vs. asynchronous, peaks vs. average). The hybrid pattern: each task with the infrastructure its volume and quality need justify.

4. No infrastructure concept lives in isolation. Latency, throughput, batch, rate limit, prompt caching, and model selection are interconnected decisions — changing one affects the others. Example: adding prompt caching reduces prefill latency, which allows larger batches without degrading the user experience.

5. Autonomy and verification grow together. With 10 users, there was almost no automatic verification — the salesperson themselves was the quality control. With 1,500, every new capability we added (RAG, MCP, two models, a gateway) came with a new control mechanism (guardrails, golden datasets, tracing, red-teaming). This case is the practical demonstration of the idea that closed Part II: every layer of autonomy requires an additional layer of verification.

Closing the journey (and what this means for you)

We have walked a path that started in Part I with a token predictor, went through Part II with agents and RAG, crossed Part III with infrastructure and observability, and now ends here: with a real system, running in production, used by 1,500 people, that generates tangible business value.

What this case demonstrates:

The principles of the compendium are not theory. Every chapter you have read — from embeddings to guardrails — has a place in this system. No piece is optional.
Evolution is inevitable. The system that works with 10 users is not the system that works with 1,500. Designing for the present, with a roadmap toward the future, is the key skill.
AI does not replace people — it amplifies them. The salespeople did not lose their jobs. They gained time to do more of what they do best: sell.
Verification is the price of autonomy. Every time capacity was added (RAG, MCP, agents), control was added (guardrails, evals, observability). That is not an extra — it is the cost of having a system that actually works.

For you, who have made it this far: you are no longer just someone who “uses AI”. You are someone who designs systems with AI. You know the right architecture depends on volume, not on a template. You know autonomy without verification is risk, not power. You know observability is not an extra — it is the difference between an experiment and a product.

The next step: apply all of this to your own context. The company in this case had 10 salespeople and reached 1,500. You have your own problem, your own volume, your own usage pattern. The tools and the principles are here. Now it is your turn.

End of the compendium.

Author’s note: this case is a didactic construction based on real scaling patterns of AI systems in consulting and technology companies. It does not correspond to a specific company, but the architectural decisions, the trade-offs, and the lessons are real and applicable. The journey of Carlos, Ana, and Laura is the journey of any team that goes from a prototype to a production system at scale.