Comparing two prompt versions

You changed a prompt. The new one looks better. Here is how to find out whether it is.

Change one thing

If you edit the prompt and switch model in the same step, a better result tells you nothing about which change caused it. This sounds obvious and is violated constantly, usually because you were changing the prompt anyway when a new model came out.

Version the prompt like code. Record which version produced which run.

Run the same cases against both

Take your regression set and run every case through A and through B, in the same session, with the same settings.

Do not compare today's B run against last week's A run. Providers update models underneath you, and the comparison quietly stops being like for like.

Repeat, because output varies

Run each case at least three times per version. A single run per case produces differences that are entirely noise.

If A passes a case twice out of three and B passes it three out of three, that is not yet evidence. It is one extra success.

Know what a real difference looks like

With thirty cases, a change of one or two passes is noise. Four or five starts to be interesting. Ten is real.

You do not need statistics for this, but you do need the honesty to say "no measurable difference" when the numbers barely move. Most prompt changes do not measurably help, and knowing that saves you from a prompt that has accumulated twenty tweaks nobody can justify.

Look at what changed, not just the total

Two versions can score twenty four out of thirty while failing completely different cases.

If B fixes four cases and breaks four others, the totals say "no change" and the reality is that you have traded one set of problems for another. Compare per case, not just in aggregate.

This is also how you notice that the fix for empty input broke the refusal behaviour.

Watch cost and latency too

A version that scores one point higher while using twice the tokens is usually not an improvement. Record token usage and response time alongside the pass rate, so the trade is visible rather than discovered on the bill.

The token cost calculator turns a token difference into a monthly figure.

Keep the loser

Store both versions and both result sets. When the new one turns out to have a problem in production two weeks later, you want to be able to go back and compare properly rather than reconstructing what the old prompt said.

The habit

Change one thing. Run both against the same fixed cases, several times each. Compare per case. Record cost. Write down the result even when the result is "no difference".

That last part is what stops a prompt turning into folklore.