Rate limit
- Standard term
A rate limit is a cap a service sets on how much you may ask of it in a stretch of time, such as requests per minute, emails per hour, or AI tokens (the word pieces AI services count) per day.
A service shares its capacity among many customers, and a cap keeps one busy account, or one runaway loop, from slowing everyone else. Requests over the cap usually get an error straight away and are dropped, so whatever sent them has to decide when to try again.
Where you’ll meet it
You ask an agent to email a follow-up to the 500 people who signed up for your company's webinar. Halfway through, the email service starts refusing messages, and the agent retries each one the instant it fails.
What to do
- Ask the agent to stop and list who has already received the email.
- Have it look up your plan's sending limit and send the rest in batches under it.
- After any refusal, have it wait as long as the error says before trying again.
Easy to mix up
| Rate limit | Quota |
|---|---|
| When its window resets, anywhere from a minute to a day | Until the plan renews, or you buy more |
Check yourself
To finish a job sooner, your agent splits it across ten copies of itself, each sending one request a second. Your plan with the service allows 300 requests a minute. How much faster does the job go?
Go deeperEach service counts differently, and retries must be safe to repeat
A fixed window resets the count each minute, hour, or day, while a “token bucket” (a different sense of token) refills a steady allowance and lets a saved-up burst through. Past the cap, a web service usually replies with error code 429, the standard code for “Too Many Requests.” The same code can also mean a used-up quota, which the reply's text spells out.
Programs that fail together and wait the same short time retry together, so they use exponential backoff, doubling the wait after each failure and adding some randomness. A retry can also repeat a send that went through but timed out before the reply arrived. An idempotency key, a unique label on each request, lets a service that supports it spot the repeat and drop it.