What is prompt caching and how do you design for it?

Answer

Prompt caching lets a provider reuse computation for a context prefix that has not changed between requests, charging substantially less for the cached portion and returning it faster.

The mechanic that matters: it is a prefix cache. It matches from the beginning of the context and stops at the first difference. Everything after a change is uncached.

That dictates the design. Order context from most stable to most volatile:

1. System prompt
2. Tool definitions
3. Long lived reference material and project instructions
4. Conversation history
5. The current request

The common expensive mistake is putting something variable early, such as a timestamp or a session id in the system prompt. That invalidates the entire cache on every request, so nothing after it is ever cached.

It suits agents particularly well, because an agent loop sends a growing context repeatedly where only the tail changes. Without caching you pay full price for the same prefix on every iteration.

Practical points: caches have a time to live, typically minutes, so benefit depends on request frequency. There is often a minimum length below which caching does not apply. Writing to the cache can cost slightly more than an uncached request, so it pays off on reuse, not on a single call.

Common wrong answer

Believing the cache stores responses, so an identical question returns an identical cached answer. It caches the processed input prefix, not the output. The model still generates a fresh response.

Also common is assuming caching is automatic and free. Some providers need explicit cache markers, and a badly ordered prompt gets no benefit at all.