Rate limits and scaling

Working locally, you are one user making one request at a time. Under real traffic, the constraints are completely different.

You are limited on two axes

Providers limit both requests per minute and tokens per minute, and you can hit either.

Many small requests exhaust the request limit while barely touching the token limit. A few large ones do the reverse. Agent workloads are usually token heavy, because every turn resends the conversation, so the token limit is the one that bites first.

Check which one you are hitting before optimising, because the fixes differ.

Limits are per organisation

Everything sharing a key shares a limit. Your production traffic, your staging environment, your batch jobs, and your own terminal all draw from the same pool.

A developer running a large local job can rate limit production. Separate keys per environment, both for isolation and so you can see where usage comes from.

Queue instead of failing

For work that does not need an immediate answer, put requests in a queue with a worker pool sized to your limit.

This turns rate limiting from an error the user sees into a delay they mostly do not. It also smooths bursts, which is what usually causes the limit to be hit at all.

Retry properly

Exponential backoff with jitter. The jitter matters: without it, everything that failed together retries together and fails together again.

Respect retry-after when the provider sends it. It is better information than your backoff calculation.

Batch what can wait

Batch APIs typically cost half as much and have separate, far higher limits.

Anything not interactive belongs there: nightly summarisation, bulk classification, backfills. This removes load from your interactive limit at the same time as halving the cost, which is the rare optimisation with no downside.

Concurrency needs a ceiling

Without a cap, a traffic spike opens hundreds of simultaneous requests, hits the rate limit, and everything fails at once.

A bounded worker pool means a spike becomes a queue instead of an outage.

Long tasks need to survive restarts

An agent task running for several minutes across many model calls will eventually be interrupted by a deploy or a crash.

Persist progress as you go. Restarting a ten step task from step one wastes the money already spent on the first nine, and doing that repeatedly during a bad deploy gets expensive quickly.

Falling back to another provider

Multi provider failover sounds attractive and is more work than it looks. Prompts do not transfer perfectly, tool calling formats differ, and quality varies.

If you need it, test the fallback path regularly against your regression set. An untested fallback usually fails at exactly the moment you need it.

Ask for more headroom

Provider limits rise with usage tier, and can often be raised on request for a legitimate workload.

Do this before you need it. Requesting an increase during an incident is slow.