Building a regression set

A regression set is a fixed list of inputs with expectations, run after every change. It is the smallest thing that turns guessing into measuring.

The format

Plain data, in version control:

[
  {
    "id": "refund-timing",
    "input": "How long do refunds take?",
    "expect_contains": ["five working days"],
    "expect_cites": true
  },
  {
    "id": "unknown-topic",
    "input": "What is your position on quantum computing?",
    "expect_refusal": true
  },
  {
    "id": "exact-code",
    "input": "What does error E4021 mean?",
    "expect_contains": ["rate limit"]
  }
]

JSON, YAML, or a spreadsheet all work. What matters is that it is checked in, so a change to the suite shows up in review alongside the change to the prompt.

What to include

Your real failures. The highest value cases. Anything that went wrong once belongs here permanently.

The obvious happy paths. The five things users ask most. These catch the catastrophic regressions.

Boundaries. Empty input, very long input, ambiguous questions, questions your system genuinely cannot answer.

Cases that should refuse. Easy to forget, and the ones that break silently when you loosen a prompt to fix something else.

Three levels of checking

Deterministic. String contains, valid JSON, correct tool called, under a length limit. Cheap, fast, unambiguous. Use these wherever the check can be expressed this way.

Model graded. Ask a model whether the answer meets a criterion. Necessary for anything about quality. Covered in LLM as judge.

Human. You read it. Reserve for a handful of cases, because it does not scale and it is the thing you will stop doing.

Push as much as possible down to the first level. Most useful checks turn out to be expressible deterministically once you decide what you actually care about.

Run it the same way every time

Fix the model, the temperature, and the prompt version. Record all three with the results.

A pass rate is only comparable against another run with the same setup. Changing the model and the prompt together and seeing an improvement tells you nothing about which one caused it.

Run each case more than once

Because output varies, a single run per case is noisy. Three runs and a majority is far more stable, and it exposes cases that pass only sometimes.

Those flaky cases are interesting. They are usually where your prompt is genuinely ambiguous.

Keep it fast

If the suite takes twenty minutes, you will stop running it before merging. Thirty to fifty cases running in a couple of minutes is a suite that survives contact with a deadline.

Use a cheap model for the graded checks. You are testing your prompt, not the judge.

When to run it

Before merging any prompt change, any model change, and any change to retrieval. Also on a schedule, because providers update models underneath you and your prompt can regress without you touching it.

That last one surprises people, and a scheduled run is the only way to catch it.