Classic rate limiting has one big assumption baked into it: requests are roughly interchangeable. A token bucket doesn't care whether the call it just admitted was a health check or a full-text search, because in most systems the difference between your cheapest and most expensive request is maybe one or two orders of magnitude, and capacity planning absorbs that.
LLM traffic breaks the assumption twice.
First, the spread is enormous. A short completion against a small model and a long-context request to a frontier model are both "one request", and the difference in cost between them can be four orders of magnitude. If your limit counts requests, a user who sends a hundred tiny calls and a user who sends a hundred maximum-context calls look identical to the limiter and completely different to the finance team.
Second — and this is the part that actually changes the architecture — you don't know what a request cost until it's finished. Output tokens are priced too, and the model decides how many of those there will be, not the caller. With streaming you're in the strange position of having admitted, priced and started delivering a response whose final bill is still unknown.
Admit on an estimate, settle on the truth
The pattern that falls out of this is the same one card payments use: an authorisation hold, then a settlement.
At admission time you estimate. Input tokens you can count exactly; output you guess from a cap or from history. You reserve that estimate against the quota and let the request through. When the response finishes — or dies halfway, which streams love to do — you settle: replace the estimate with the actual usage, and give back or take the difference.
Most of the interesting bugs live in the gap between those two moments.
- A stream that dies mid-response still consumed real tokens. If your settlement path only runs on success, every provider hiccup quietly leaks quota.
- Estimates drift. If you always guess high, you starve users whose actual usage was modest. Always guess low and a burst of long responses blows through a budget that the limiter swears is intact.
- Two requests racing on the same nearly-empty budget will both pass an optimistic check. You need the reservation to be atomic, which is annoying precisely because this is the hottest path you have.
None of this is exotic computer science. It's bookkeeping. But it has to be bookkeeping that runs inside the request path, at streaming speeds, across every replica of the gateway — and "distributed, fast, correct bookkeeping" is one of those phrases that looks calm and isn't.
Requests, tokens and money are three different limits
The other thing I keep re-learning: teams say "rate limit" and mean three unrelated protections.
Requests per second protects your infrastructure. Tokens per minute mostly protects your standing with the model provider, whose own limits are shaped that way. Spend per month protects the budget owner. They have different windows, different owners and different failure conversations — an engineer cares about the first one, a director about the third.
So the limit can't be a single number on an API key. It ends up being a small matrix: dimension (requests, tokens, cost) × scope (user, agent, team, model, provider) × window. That sounds heavy, and honestly the first version I'd sketch on a whiteboard wouldn't have it — but every one of those axes got added because somebody needed exactly that cut and couldn't approximate it with the others.
One quiet detail that pays for itself: when you reject, say why and for how long. A 429 with the dimension that tripped and a reset time turns an angry "the AI is down" ticket into a client that backs off on its own. The status code is the same; the operational experience is not.
If you're building anything in front of model providers right now, I'd spend your effort here, in this unglamorous layer, before spending it almost anywhere else. The models will keep changing. The bill arrives either way.