Using a model as a judge

Some qualities cannot be checked with a string comparison. Is this summary faithful to the source? Is this answer actually responsive? For those, you can ask a model. It works, with known caveats.

When to reach for it

Use a judge only where a deterministic check genuinely cannot express what you mean.

Faithfulness to a source, tone, whether an explanation is complete, whether two answers mean the same thing. These are real criteria with no simple test.

Do not use a judge for anything a string match can do. It is slower, costs money, and adds noise.

Grade one thing at a time

The most common mistake is asking for an overall quality score.

> Rate this answer from 1 to 10.

You get a number between 6 and 8 for almost everything, and it does not move when quality does.

Instead ask something specific and binary:

> Does the answer contain any claim not supported by the context? > Reply with exactly YES or NO, then one sentence of justification.

Binary questions with a clear definition are dramatically more consistent, and you can aggregate them into a score yourself.

Known biases

Position bias. When comparing two answers, judges favour the one shown first. Correct by running each comparison twice with the order swapped, and discarding disagreements.

Length bias. Longer answers score higher, whether or not they are better. If length is not a criterion, say so explicitly, or normalise for it.

Self preference. A model tends to prefer text written by itself or its own family. Where it matters, judge with a different model family than the one generating.

Style over substance. Confident, well formatted, correctly structured wrong answers score well. This is the most dangerous one, because it is exactly the failure mode you are trying to catch.

Give it the answer key

A judge marking against a reference is far more reliable than one marking on general impression:

> Reference answer: Refunds take five working days. > Given answer: [text] > Does the given answer convey the same fact? YES or NO.

Now it is comparing, not evaluating. Comparison is a much easier task and the results are far more stable.

Validate the judge

This is the step that gets skipped, and it is the one that makes the whole thing trustworthy.

Grade thirty cases yourself. Have the judge grade the same thirty. Compare.

If they agree on twenty seven, the judge is usable. If they agree on nineteen, your judge prompt needs work, and any number it produces is currently meaningless.

Re-check whenever you change the judge prompt or model.

Keep it cheap

A judge runs on every case on every run. Use a small fast model. You are checking a specific narrow property, not asking for deep reasoning, and small models do this well.

What it cannot do

A judge cannot tell you whether the answer is factually true about the world. It can tell you whether the answer is consistent with the context you gave it.

That distinction matters. For factual correctness there is no substitute for a reference answer written by someone who knows.