Tracing and observability
Someone reports that the agent gave a wrong answer. If all you have is the answer, you cannot do anything about it. Observability is deciding in advance what you will need.
Log the whole request, not just the output
For every interaction, record:
- The full prompt sent, including the system prompt and retrieved context
- The model and version
- Every tool call, with arguments and results
- The final output
- Token counts, split into input and output
- Latency, and time to first token if you stream
- A trace id linking it all together
The prompt is the item people omit, usually because it is large. It is also the one you always need, because "what did the model actually see" is the first question in every investigation.
Why the assembled prompt matters
Your prompt is built at runtime from a template, retrieved chunks, conversation history, and user input.
Logging the template tells you what you intended. Logging the assembled prompt tells you what happened. When retrieval returns the wrong chunks, only the second one shows it.
Group by conversation
A single agent task is many model calls. Logging them individually gives you a pile of fragments.
Give each conversation a trace id and attach it to every call, so you can read the whole sequence in order. Most agent failures only make sense as a sequence: the third tool call failed, so the fifth answer was wrong.
Redact at the point of logging
Prompts contain user data. Logs get shipped to third party services, kept for months, and read by people who were not in the original conversation.
Strip credentials, tokens, and personal data as you write the log, not later. Retrofitting redaction after a leak is not a recovery.
Also make sure API keys never reach logs through error messages, which is a common accidental route.
Track cost per conversation, not per call
Per call costs look trivially small. Per conversation costs are what appear on the bill.
Sum tokens across a whole trace. Then look at the distribution rather than the average: a small number of very long conversations usually dominates total spend, and the average hides them.
Alert on the things that move first
Token usage per conversation. Rising means context is bloating or the agent is thrashing.
Tool call count. Rising means it is struggling before it starts failing.
Refusal and error rates. Sudden movement usually means something upstream changed.
Latency at the ninety fifth percentile. Averages hide the experience that makes people complain.
Sample if volume is high
Full logging of every request gets expensive. Log everything for errors and slow requests, and sample the rest.
Keep the sample rate high enough to be useful for debugging. Two percent is not enough to find a rare failure.
The test
Pick a bad answer from last week. Can you reconstruct exactly what the model saw, what tools it called, and what came back?
If not, you are missing something, and the time to find out is before the incident rather than during it.