LLM judges and prompt evaluation

Let’s pick up something from the previous chapter. We built two versions of a review classifier (zero-shot and few-shot) and said few-shot “works better” — but we made that claim with a single example. Here is the uncomfortable question this chapter exists to answer: how do you know you did not just get lucky with that example?

The underlying problem, and why you already know it

This should sound familiar. In the loss chapter of Part I we said something almost identical: one prediction is not enough to judge a model — it may have gotten it right by pure luck. That is why we run the model over a batch of real text and average. Evaluating prompts has exactly the same problem, at another scale: if you compare two prompts with a single test case, you do not know whether the difference you see is real or noise. You need many test cases and something that scores them consistently.

The difference is that now we do not have a “true token” against which to compute cross-entropy — the correct answer to “is this review negative?” does not come with a clean mathematical label. We need something that does that scoring work for us: a judge. And that judge, more and more often, is another LLM.

The three ways to evaluate, with the same worked example

Let’s stay with the sentiment classifier. Imagine five test reviews:

#	Review
1	“The laptop is fine, but I expected more.”
2	“I had never had a product this bad.”
3	“It does exactly what it promises, no surprises.”
4	“The battery life is short, though the screen is excellent.”
5	“It exceeded all my expectations, I recommend it completely.”

Pairwise comparison: we run the five reviews through both versions of the prompt, and ask an LLM judge to compare, case by case, which classification seems more aligned with the business criteria.

#	Zero-shot says	Few-shot says	Judge preference
1	Neutral	Negative	Few-shot
2	Negative	Negative	Tie
3	Positive	Positive	Tie
4	Positive	Negative	Few-shot
5	Positive	Positive	Tie

Few-shot wins 2, they tie 3, zero-shot wins none. This is a real comparison — not the single-case claim from the previous chapter, but a verifiable count over several cases.

Single-response scoring (1 to 5): here we score a single output against an absolute scale, without comparing two versions. Useful when you don’t have a “previous version” to compare against. For case 4 (“the battery life is short, though the screen is excellent” → few-shot says “Negative”), a judge might score that classification 3/5: technically reasonable, but the review is mixed and “negative” loses the positive nuance about the screen.

Rubric-guided evaluation: adds precision on top of the 1–5 score — instead of letting the judge freely decide what “3” or “5” means, you give explicit rules:

“Score 5: the classification matches exactly what a human analyst at the company would assign. Score 3: the classification is defensible but loses nuance (such as mixed sentiment). Score 1: the classification clearly contradicts the sentiment expressed.”

With this rubric, case 4 is still a 3 — but now anyone who reads the rubric understands why, and if you run the same evaluation tomorrow with the same judge, it is much more likely to land on the same number.

Common traps when using an LLM judge

An LLM judge is not a perfect neutral referee — it inherits real biases:

Position bias: in pairwise comparisons, some judges systematically prefer the answer shown first, regardless of which is better. Standard mitigation: run the comparison twice, reversing the order, and only trust the result if the judge is consistent in both directions.
Verbosity bias: judges tend to prefer longer answers, even when the short one is equally correct.
Self-preference bias: a judge tends to score higher the answers that look like how it itself would respond, artificially favoring a model from the same family as the judge.

Just as we do not trust the loss of a single prediction (Part I), we also do not trust the verdict of a single comparison here.

How well does an LLM judge agree with humans?

So far we have explained how to use an LLM judge, but not how reliable it is compared with the standard it is trying to replace: human judgment. Here is a real figure worth knowing, with precision about what exactly was measured.

The paper “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena” (Zheng et al., 2023) tested this directly: using the MT-Bench benchmark (80 multi-turn conversation questions, covering writing, reasoning, math, code, and more), they measured how often GPT-4, acting as a judge in pairwise comparison mode, agreed with the verdict of expert human evaluators — and compared that against how often humans themselves agree with each other on the same task.

The result: GPT-4 agreed with humans 85% of the time — above the 81% agreement that the human evaluators themselves had with each other. Put another way: on this specific task, GPT-4 as a judge was as reliable (or more) than asking a second human to review the work of a first.

Precision matters here — what this result does NOT say: this figure is specific to pairwise comparison, on MT-Bench, with GPT-4 as judge. It is not a universal guarantee that “any LLM as a judge, on any task, will match humans” — remember the biases we already saw (position, verbosity, self-preference), which still apply and can degrade this agreement on different tasks or setups. What it does demonstrate, concretely and measured, is that a well-configured LLM judge can reach a reliability level comparable to a second human reviewer — it is not a crude approximation, it is a tool that, used with judgment, genuinely competes with the standard it replaces.

Choosing the right method: a quick guide

With the three techniques already covered (pairwise comparison, single-response scoring, rubric-guided), here is a quick decision guide:

Your situation	Recommended method
I want to know if the new prompt is better than the previous one	Pairwise comparison
I want an absolute quality score, with no previous version to compare against	Single-response scoring (1-5)
I need the evaluation to be reproducible and well defined across different runs	Rubric-guided
I am going to run this hundreds of times in CI/CD, I need consistency	Rubric-guided — the explicit criteria reduce variance across runs
I only need a quick, exploratory review, not automated yet	Pairwise comparison — the simplest to improvise with no prior preparation

Should the judge be the same model that generates, or a different one?

Technically it can be the same model — there is no restriction that prevents it, and in practice it is simpler to integrate. But there is a real reason to prefer a different one, and it connects directly to self-preference bias: if the same model generates and evaluates, any systematic blind spot it has leaks into both sides of the process. If the model has a systematic tendency to classify any mixed review as “neutral” regardless of the relative weight, that same tendency will be present when it evaluates whether “neutral” was the correct classification — the judge and the generator share the same bias, with no genuine second perspective reviewing the work.

A very common pattern in industry: use a larger, more expensive model as judge, to evaluate the outputs of a smaller, cheaper model that is the one that actually runs in production. It makes economic sense because generating happens constantly, while evaluating usually happens much less often — on a test set, during development, or as periodic sampling over production. You can afford to pay the price of an expensive model for the judge precisely because you do not call it on every interaction, only when you evaluate.

It is also worth noting that, now and then, a human reviews a sample of the judge’s verdicts to confirm it is scoring well — if it is systematically wrong on a certain kind of case, that feedback lets you recalibrate it, something we will see formalized as “judge auto-tuning” in the observability chapter of Part III.

Where all of this lives: rubrics, code, and tools

The rubric itself is always plain text — no matter which tool you use afterward, it is natural language that becomes part of the prompt you send the judge. In its rawest form, with no tool in between:

judge_prompt = f"""
You are an evaluator.
Score 5: the classification matches exactly what a human analyst would assign.
Score 3: it is defensible but loses nuance.
Score 1: it contradicts the sentiment expressed.

Original question: {review}
Response to evaluate: {model_classification}

Give your score (1-5) and explain your reasoning.
"""
judge_response = call_judge_model(judge_prompt)

That already is an “LLM as a judge” working, with no YAML and no external platforms. Tools like PromptFoo (which uses YAML config files) automate repeating this pattern over many test cases at once, without you having to write the loop yourself — the llm-rubric field in its config is exactly our text rubric. Platforms like LangSmith change the format (datasets and evaluator functions in Python instead of YAML), but internally they do the same: they build a prompt to the judge with the rubric embedded, and they also handle tracing — saving every run, comparing versions over time.

Asking the judge to explain its reasoning before giving the verdict is, literally, applying chain-of-thought (Chapter 2) to the act of evaluating itself — and it is not just an extra: that explanation is what lets you detect the biases above. If the judge says “I prefer answer B because it is longer and more detailed”, right there you see verbosity bias operating, instead of receiving a blind verdict with no justification.

The tool ecosystem, beyond PromptFoo

In-code testing frameworks, for active development:

DeepEval (from Confident AI): integrates natively with pytest-style tests — the real-tool version of “evals as code” that we will see formalized in the observability chapter of Part III: each evaluation becomes a test that runs in your CI pipeline, just like any software unit test. It offers more than 50 ready-made metrics (faithfulness, hallucination, relevance, bias, toxicity, tool-selection accuracy, among others) — rubrics already written and calibrated by others, so you do not have to draft yours from scratch every time.
RAGAS: specialized specifically in evaluating RAG systems (Chapter 5) — something a generic judge would not know how to evaluate well out of the box. Its four central metrics cleanly separate the retrieval problem from the generation problem: faithfulness (is the answer actually supported by the retrieved context, or did the model hallucinate anyway?), answer relevancy (does the answer actually address the question?), context precision (is what was retrieved relevant?), and context recall (was everything needed to answer well retrieved?). This separation tells you at which stage your RAG system failed — whether the problem is retrieval or generation.

Production observability platforms — they go beyond “evaluate before launch”, corresponding to the “from the platform” evaluation we will see in the EDD chapter (Part III): automatic evaluations over real production traces, not just over a fixed test dataset.

LangSmith (commercial, from LangChain): tracing, debugging, and evaluation automation.
Langfuse: open-source alternative to LangSmith, for teams that prefer to self-host their own infrastructure.
Braintrust: aimed at teams that have already moved past local tests and need to evaluate and monitor agents in production — it connects to the “agentic tracing” of Part III, which insists on capturing not only the final answer but the tool calls and intermediate transformations of the agent.
Arize Phoenix: open source, with anomaly detection included — the piece that reappears as a detective strategy in the agentic security chapter (Part III).

Specialized security tools:

Garak: focused on red-teaming — testing models against a library of known attacks and vulnerabilities, instead of evaluating quality on normal cases. We develop this technique in depth in the agentic security chapter of Part III, together with the Finnbot case.

The full progression: prototype with PromptFoo/DeepEval → specialize for RAG with RAGAS → take to production with LangSmith/Langfuse/Braintrust/Phoenix → harden against attacks with Garak.

Why this is the foundation of everything that comes after

Without systematic evals, “improving a prompt” reduces to “I tried a couple of examples and it looked better” — an intuition that does not hold when the prompt runs thousands of times a day against inputs you never saw. Evals turn that subjective judgment into a number you can compare objectively across versions of a prompt, across different models (for example, comparing one model’s performance against another on the same task), or across different tool descriptions in an agentic flow.

This is the first step toward something much more complete that we will see in Part III: turning evaluation into a systematic engineering practice, not an occasional manual review.