The token bucket algorithm is built around a simple mechanic: a bucket holds tokens, tokens refill at a fixed rate up to some maximum, and every request has to spend one token to proceed. If the bucket has tokens saved up, a burst of requests can go through immediately, one token at a time, until the bucket runs dry — at which point requests are limited to however fast the bucket refills. That burst allowance is what separates it from stricter rate-limiting approaches, and it is close to the default choice for API throttling today.
What changed in 2026
- It became close to a default for AI and LLM API throttling, where allowing a short burst of requests — a batch job kicking off, several parallel calls from a fan-out pattern — matters more than enforcing a perfectly flat rate.
- Distributed implementations got simpler, with shared-counter libraries built on fast key-value stores making a consistent token bucket across many servers straightforward instead of a custom project.
- Burst-friendly limits became the expected norm for API design, replacing harsh hard caps that reset abruptly at a fixed clock boundary, largely because the token bucket already solves that problem cleanly.
The token-and-bucket model
Tokens accumulate in a bucket at a steady rate, for example ten per second, up to a maximum capacity, for example fifty. A request that arrives when the bucket has at least one token consumes it and proceeds immediately; a request that arrives when the bucket is empty is delayed or rejected until the next token refills. Because tokens can accumulate during idle periods, a client that has been quiet can send a burst up to the bucket's full capacity the moment it becomes active, then settles into the steady refill rate once that reserve is spent.
The two parameters that define behavior
| Parameter |
Controls |
Effect of increasing it |
| Refill rate |
The sustained long-run request rate |
Higher steady-state throughput allowed |
| Bucket capacity |
The maximum burst size |
Larger bursts tolerated before throttling kicks in |
Tuning these independently is the whole point: a low refill rate with a large capacity allows occasional large bursts but a low average rate, while a high refill rate with a small capacity keeps throughput high but smooths out bursts almost entirely.
Pseudocode
class TokenBucket:
def __init__(self, capacity, refill_rate):
self.capacity = capacity
self.refill_rate = refill_rate # tokens per second
self.tokens = capacity
self.last_check = time.now()
def allow(self):
now = time.now()
elapsed = now - self.last_check
self.tokens = min(self.capacity, self.tokens + elapsed * self.refill_rate)
self.last_check = now
if self.tokens >= 1:
self.tokens -= 1
return True
return False
Where it is used
API gateways throttling clients while still allowing short bursts, such as a dashboard loading several endpoints at once; network routers performing traffic policing, where a burst up to a committed size is allowed before excess traffic is marked or dropped; and distributed rate limiters coordinating a shared limit across many server instances via a common token store.
Token bucket vs leaky bucket, briefly
Both use a bucket metaphor, but they answer different questions: a token bucket asks "how much can this client burst right now," while a leaky bucket asks "what is the maximum constant rate this should ever produce." See rate limiting strategies for how the two compare directly against fixed-window and sliding-window approaches.
Common pitfalls
Setting capacity too high relative to what downstream can absorb. A generous burst allowance is only safe if whatever is behind the limiter can actually handle that burst; otherwise the limiter has just moved the overload problem one layer down.
Forgetting refill happens continuously, not in ticks. An implementation that only refills at fixed intervals rather than continuously introduces boundary effects similar to a fixed-window counter.
Running independent buckets per server without coordination. In a multi-server deployment, uncoordinated local buckets can let a client exceed the intended aggregate limit by simply spreading requests across servers.
FAQ
Why allow bursts at all instead of a flat rate?
Real traffic is bursty by nature — a user loading a page, a batch job starting up. A flat-rate limiter penalizes normal usage patterns; a token bucket accommodates them without abandoning a long-run limit.
How is this different from a fixed-window rate limit?
A fixed-window counter resets sharply at a clock boundary, which can allow two bursts back-to-back right at the reset. A token bucket refills continuously, avoiding that boundary spike.
What happens when the bucket is empty?
The request is either rejected, queued until a token becomes available, or delayed, depending on the implementation. Which behavior is right depends on whether the caller can tolerate waiting.
Can a token bucket be shared across multiple servers?
Yes, using a centralized or replicated token store, though this adds coordination overhead compared to a single-process implementation.
Where to go next