3 min read
Rate limiting LLM traffic is a cost problem wearing a traffic costume
Request counting assumes requests are roughly interchangeable. LLM calls are not — one can cost ten thousand times more than another, and you don't know the price until it's over.
- ai-infrastructure
- rate-limiting