How do you stop a prompt change from breaking something else?

Answer

Prompt changes have non local effects. Adding an instruction to fix one behaviour routinely changes unrelated behaviour, because you have altered the input distribution for every request.

The discipline mirrors software testing.

Version prompts in git, not in a database or a UI where changes are untracked. You need to see the diff and revert it.

Keep a fixed regression set of real cases with known expected outcomes. Include the cases that motivated past fixes; those are exactly what regresses.

Run the whole set on every change, not just the case you were fixing. The failure mode is fixing case A and silently breaking case B, and you only find out from users.

Run each case multiple times and compare success rates, since a single sample cannot distinguish a real change from nondeterminism.

Gate on aggregate, not anecdote. "This output looks better" is not evidence. A success rate moving from 70 to 85 percent across the set is.

Handle flakiness explicitly. If a case passes 60 percent of the time, it is not a pass or a fail, it is a measurement. Either track its rate or fix the underlying instability rather than rerunning until it goes green.

Treat the prompt as code that ships. It has the same blast radius and deserves the same review.

Common wrong answer

Testing only the case you were trying to fix, confirming it now works, and shipping. This is the single most common way prompt systems degrade over months: a series of local improvements that each broke something nobody checked.