How do you detect an agent failing silently?

Answer

A silent failure is the agent reporting success when the work is wrong or incomplete. It is the most dangerous failure mode because nothing signals it, and the agent's own confidence is uncorrelated with correctness.

Never trust the agent's self report. "I fixed the bug and the tests pass" is a claim, not a result. Run the tests yourself, in your own pipeline. Any verification the agent performs and reports on is inside the same system that may be wrong.

Check the end state against the requirement. Does the file exist, does it compile, does the endpoint return the expected shape, does the row appear in the database. These are facts independent of the narrative.

Watch for the signature patterns:

• Claims a test passes when no test file was created
• Says a change was made to a file that is unmodified in the diff
• Reports a command succeeded when its exit code was non zero
• Produces a summary describing more than the diff contains
• Marks a task complete with a caveat buried in the middle

Log every tool call, its arguments, and its raw result. When a silent failure surfaces later, that log is the only way to reconstruct what happened. Reasoning text alone is not enough.

Diff review is the practical control. For coding agents, reading the actual diff catches most silent failures cheaply, which is why the small reversible change workflow matters.

Common wrong answer

Adding a step where the agent checks its own work. It helps a little and is not a control, because the same reasoning that produced the error produces the verification. Self checking catches slips, not misunderstandings. The check has to come from outside the agent.