LLM Observability and Evaluation-Driven Development

“The gateway reports 99.9% availability, response time is excellent, and cost is under control. But the coordinator is misclassifying 15% of the sensitive complaints as ‘policy question’, and nobody knows.”

This is the central problem of observability in AI systems. Traditional metrics — latency, throughput, rate limits, uptime — tell you whether the server is working. They do not tell you whether the agent is doing its job correctly.

This chapter answers questions that server logs cannot answer: is the coordinator routing correctly? is the vacation agent hallucinating dates? is RAG retrieving the right documents? did a silent prompt change break complaint classification? The answer lives in the interaction data, and in a new paradigm: Evaluation-Driven Development (EDD).

The paradigm shift: from systems monitoring to agentic monitoring

Metric	What It Measures	What It Does NOT Measure
Uptime	Is the server running?	Whether the agent answers correctly
Latency	How fast does it respond?	Whether the answer is useful
Error rate	Are there 500 errors?	Whether the agent is hallucinating
Throughput	How many requests does it handle?	Whether the routing is correct

The gateway can report 99.9% availability, answering instantly — and still, the coordinator could be misclassifying 15% of the sensitive complaints as routine queries. That is exactly the risk we identified in Agentic architecture patterns (Part II): a harassment complaint that never reaches a human because the system classified it wrong. Business reliability depends on whether the answer is correct, not only on whether the service responded.

This demands a transition from systems observability (traditional SRE) to agentic observability. With it comes the role of the AI Engineer — distinct from the ML Researcher (trains models and publishes weights) and the Full Stack Engineer (builds APIs and frontends). The AI Engineer orchestrates prompts, context, agents, and tools — they are responsible for the agent being reliable and useful in production, not just for the server being up.

The broken development cycle: the “black hole” of LLMs

Traditional software development is deterministic: code → unit test → CI → production, with fast, predictable feedback. LLM development is stochastic and opaque — the same prompt can produce different results, and it is not always clear why.

Problem	Description	Example (Our Coordinator)
Non-deterministic logic	The model can answer differently to the same prompt	It classifies “I want to report harassment” correctly today; tomorrow, with the same phrase, it can fail
The “Wow Factor”	The demo impresses on a few cases, but fails on real variety	It works on 5 test examples, fails on the 500 real ways employees write complaints
Lost feedback	Fixing one case silently breaks another	You improve complaint detection, but now it misclassifies vacation questions
Orchestration opacity	Intermediate tool calls are not captured	The gateway only sees “request in, response out” — not the intermediate reasoning

The solution: turn evaluation into the quality gate of AI development — EDD.

Evaluation-Driven Development (EDD): the quality gate of AI

EDD is the step from “experimentation by intuition” to a structured process where every change is validated against predefined success criteria before it reaches production.

Recalling LLM judges and evals (Part II), where we built an LLM judge with five test opinions: EDD is exactly that same practice, taken to the scale of a full team. Instead of “I tried the new prompt on three examples and it seemed to work better”, the correct practice is “I ran the new prompt against the golden dataset of 500 labeled messages, and accuracy on sensitive complaints rose from 88% to 93% without degrading other categories”. The difference is evidence, not intuition.

Evals as Code — integration into CI/CD: if someone adjusts the coordinator prompt, that change is submitted as a Pull Request, same as any code change — CI automatically runs the evaluation against the golden dataset, compares with the previous version, and verifies there are no regressions before merging.

# tests/test_coordinador.py
import pytest
from deepeval import assert_test
from deepeval.metrics import AccuracyMetric
from deepeval.test_case import LLMTestCase

DATASET = [
    {"input": "I want to take vacation next week", "expected_route": "vacaciones"},
    {"input": "I need to report a harassment problem", "expected_route": "queja_sensible"},
    {"input": "How can I reimburse an expense?", "expected_route": "gastos"},
    # ... 500 more cases
]

@pytest.mark.parametrize("test_case", DATASET)
def test_coordinador_enruta_correctamente(test_case):
    resultado = coordinador(test_case["input"])
    metric = AccuracyMetric()
    assert_test(
        LLMTestCase(
            input=test_case["input"],
            actual_output=resultado.route,
            expected_output=test_case["expected_route"]
        ),
        metrics=[metric]
    )

The key discipline: the coordinator prompt, and the set of messages that validates it, live versioned in Git. Nobody edits the prompt “by hand” directly in production without going through this process.

The three pillars of agentic observability

1. Agentic tracing

Capture every observable step of the agent, not just the input and the output. For our vacation flow (Agentic flows, Part II):

Why it is crucial: without tracing, if an employee complains that the agent gave them incorrect information, there is no way to know whether the problem was the MCP server (wrong data), the policy RAG (wrong citation), or the LLM’s reasoning (it combined the data badly). The trace says exactly where the system failed.

What matters to record is the observable contract: inputs, outputs, tools invoked, arguments, results, latency, and errors. A useful trace does not need — and should not require — the model’s private chain of thought. In many deployments that chain is not available, and in others it is not wise to persist it.

2. Golden datasets

A set of labeled test cases that represent the expected behavior — it is not a training dataset, it is a validation dataset.

Dataset	Purpose	Typical size
Coordinator	Validate routing across domains	200-500 cases
Vacations	Validate the request flow	200-400 cases
Expenses	Validate the expense pipeline	200-400 cases
Policies (RAG)	Validate policy retrieval	100-200 cases
Security	Validate detection of sensitive complaints	100-200 cases

Why separate? If you mix them all into one, you cannot diagnose which specific agent a regression happened in. Separate datasets let you isolate the problem.

How it is built: initial collection of real production messages, human labeling by a domain expert, expansion with edge cases and linguistic variations, and periodic review adding new cases that have caused problems.

The over-specificity trap: if your dataset only has complaints written very directly (“I want to report harassment”), the coordinator can overfit to that wording and fail on a more indirect complaint (“I feel uncomfortable with my boss’s behavior”). The solution is deliberate variety: synonyms, paraphrases, different levels of formality.

3. Continuous monitoring in production

Golden datasets are excellent for pre-deployment tests, but they do not capture everything that happens in real production. The cycle: random sampling of conversations (1-5% of interactions), automatic evaluation with an LLM judge against a rubric, drift detection (compare this week’s accuracy against last week’s — if it drops noticeably, alert), and periodic human annotation to confirm the judge is still correctly calibrated.

Quality metrics, beyond accuracy

Metric	Definition	Example (coordinator)
Routing accuracy	% of messages routed correctly	Did “I want to report harassment” go to “queja_sensible” or to “política”?
Faithfulness	Is the answer backed by the context?	Did the vacation agent cite the correct policy?
Hallucination	% of answers with false information	Did the agent invent a vacation balance?
Consistency	Does the same question get the same answer?	Repeat the same question several times and measure variation
Functional latency	Total time, including tools	How long does it take to look up balance + calendar + respond?
Escalation rate	% of cases escalated to a human	What percentage of sensitive complaints reach a human?
User satisfaction	Was the answer useful?	Post-interaction survey, or sentiment analysis

No single metric tells the full story: coordinator accuracy can be 95%, but if the escalation rate of sensitive complaints is 0% (because they never reach a human), the system is failing exactly at its most critical function — you have to look at the set, not a single isolated figure.

EDD in practice: integration with CI/CD

# .evals/config.yaml
golden_datasets:
  coordinador: "datasets/coordinador.jsonl"
  vacaciones: "datasets/vacaciones.jsonl"
  seguridad: "datasets/seguridad.jsonl"

metrics:
  - type: accuracy
    threshold: 0.92
  - type: hallucination
    threshold: 0.05
  - type: escalation_rate
    threshold: 0.95

alerting:
  - metric: accuracy
    drop_threshold: 0.05
  - metric: latency_p95
    threshold_ms: 2000

The lifecycle of a change: Ana (AI Engineer) adjusts the coordinator prompt to improve detection of sensitive complaints. She creates a branch, opens a PR. CI automatically runs the 500 cases of the golden dataset — accuracy rises from 92% to 94%, with no regressions on the other datasets — and the PR is approved and deployed. If instead the change improved complaints (+2%) but degraded vacations (-3%), Ana would receive that exact report and decide whether to adjust again or accept the trade-off consciously. Without CI, that trade-off would have reached production without anyone noticing.

The tooling ecosystem

We already documented this ecosystem with verified figures in LLM judges and evals (Part II) — DeepEval (pytest integration, more than 50 prebuilt metrics), RAGAS (the 4 metrics specific to RAG: faithfulness, answer relevance, context precision and recall), LangSmith, Langfuse, Braintrust, Arize Phoenix, and Garak for red-teaming. The natural progression is still the same: DeepEval for evals as code in CI/CD, adding production tracing with Langfuse or LangSmith, RAGAS if there are RAG systems, and Garak if security is critical.

The next frontier: “AI for AI”

Prompt auto-optimization: instead of a human manually adjusting the prompt every time the golden dataset detects a regression, a system can generate prompt variants, evaluate each one against the same dataset, and use the best as the base for the next round — speeding up iteration without yet replacing the final human judgment.

Judge auto-recalibration: when a human corrects an LLM judge’s evaluation during their periodic review, that correction can automatically feed an adjustment of the judge itself — closing the loop so it improves over time without constant manual intervention on every case.

Drift detection: drift does not only happen in the input data (how employees ask changes over time) but in the model’s own behavior, which can degrade without a single line of code changing. Running the golden dataset automatically every week, and alerting if accuracy drops more than a defined threshold, lets you detect this before it becomes a visible problem for users.

Connection with Part II and Part III

Pattern (Part II)	Observability challenge	Solution
Single agent	Monitor reasoning and tools in a single flow	Full agent trace
Sequential	Know which pipeline step failed	Trace each step separately
Parallel	Reconcile contradictory results	Trace each agent and compare outputs
Coordinator	Detect wrong classifications	Coordinator golden dataset
ReAct	Detect unproductive loops	Count turns, iteration limit
Multi-agent	Implicit trust between agents (the Finnbot risk)	Cross traces, confidence thresholds

With Part III: functional latency (including tools) is a critical metric in its own right; the latency/throughput/cost choice of the trilemma affects how many evals you can afford to run in production; batch size even affects how long evaluations take inside the CI pipeline itself.

The fundamental tension stays intact: every layer of autonomy requires an additional layer of verification. Observability is not an extra — it is the price you pay for having a system that actually works in the real world.

Best practices for launching with observability

Ship small, ship early: do not wait for a “perfect” golden dataset with a thousand cases imagined in advance — launch to a pilot group of 50 employees, and use their real messages (including the ones the system classified wrong) as the first real cases in the dataset.

Evaluate end to end, not every micro-step: do not force the agent to always look up the balance before the calendar — what matters is whether the final answer is correct, leaving the medium autonomy we already defined to decide the order. Measure the final result; use the trace only when you are debugging a specific failure.

Avoid over-specificity: add deliberate variety to the dataset — synonyms, paraphrases, different registers — so the system generalizes instead of memorizing the exact wording of your test examples.

Observability is an investment, not a cost: fewer unnecessary escalations to humans, fewer user complaints, fewer security incidents (avoiding the Finnbot case itself), and more trust that accelerates real adoption of the system.

Chapter summary

Systems monitoring (uptime, latency) is not enough — you need agentic observability, which answers whether the agent is doing its job well, not only whether the server responds.
EDD turns evaluation into the quality gate of AI development — every change is validated with evidence, not intuition.
The three pillars: tracing (every observable step), golden datasets (labeled cases per agent), and continuous monitoring (sampling and drift detection in production).
No quality metric is enough on its own — accuracy, faithfulness, hallucination, consistency, functional latency, escalation rate, and satisfaction must be seen together.
Observability is not a passive accessory — it is what separates a prototype that impressed in the demo from a system thousands of people can trust every day.

The Finnbot case, which we will see in detail in the next chapter on Security in agentic systems, is a direct reminder of what happens when autonomy outruns verification. Observability is the safety net that keeps errors from becoming disasters.