{"post":{"slug":"what-is-a-token-bucket-algorithm-2026","title":"What Is the Token Bucket Algorithm in 2026?","excerpt":"The token bucket algorithm allows controlled bursts instead of forcing a perfectly flat rate. Here is how tokens, refill rate, and capacity work together, with pseudocode and where it shows up in real systems.","category":"code","date":"July 21, 2026","author":"ByteLedger Team","content":"<span class=\"dropcap\">T</span>he token bucket algorithm is built around a simple mechanic: a bucket holds tokens, tokens refill at a fixed rate up to some maximum, and every request has to spend one token to proceed. If the bucket has tokens saved up, a burst of requests can go through immediately, one token at a time, until the bucket runs dry — at which point requests are limited to however fast the bucket refills. That burst allowance is what separates it from stricter rate-limiting approaches, and it is close to the default choice for API throttling today.\n\n## What changed in 2026\n\n- **It became close to a default for AI and LLM API throttling**, where allowing a short burst of requests — a batch job kicking off, several parallel calls from a [fan-out pattern](/blog/what-is-a-fan-out-fan-in-pattern-2026) — matters more than enforcing a perfectly flat rate.\n- **Distributed implementations got simpler**, with shared-counter libraries built on fast key-value stores making a consistent token bucket across many servers straightforward instead of a custom project.\n- **Burst-friendly limits became the expected norm for API design**, replacing harsh hard caps that reset abruptly at a fixed clock boundary, largely because the token bucket already solves that problem cleanly.\n\n## The token-and-bucket model\n\nTokens accumulate in a bucket at a steady rate, for example ten per second, up to a maximum capacity, for example fifty. A request that arrives when the bucket has at least one token consumes it and proceeds immediately; a request that arrives when the bucket is empty is delayed or rejected until the next token refills. Because tokens can accumulate during idle periods, a client that has been quiet can send a burst up to the bucket's full capacity the moment it becomes active, then settles into the steady refill rate once that reserve is spent.\n\n## The two parameters that define behavior\n\n| Parameter | Controls | Effect of increasing it |\n| --- | --- | --- |\n| Refill rate | The sustained long-run request rate | Higher steady-state throughput allowed |\n| Bucket capacity | The maximum burst size | Larger bursts tolerated before throttling kicks in |\n\nTuning these independently is the whole point: a low refill rate with a large capacity allows occasional large bursts but a low average rate, while a high refill rate with a small capacity keeps throughput high but smooths out bursts almost entirely.\n\n## Pseudocode\n\n```python\nclass TokenBucket:\n    def __init__(self, capacity, refill_rate):\n        self.capacity = capacity\n        self.refill_rate = refill_rate  # tokens per second\n        self.tokens = capacity\n        self.last_check = time.now()\n\n    def allow(self):\n        now = time.now()\n        elapsed = now - self.last_check\n        self.tokens = min(self.capacity, self.tokens + elapsed * self.refill_rate)\n        self.last_check = now\n        if self.tokens >= 1:\n            self.tokens -= 1\n            return True\n        return False\n```\n\n## Where it is used\n\nAPI gateways throttling clients while still allowing short bursts, such as a dashboard loading several endpoints at once; network routers performing traffic policing, where a burst up to a committed size is allowed before excess traffic is marked or dropped; and distributed rate limiters coordinating a shared limit across many server instances via a common token store.\n\n## Token bucket vs leaky bucket, briefly\n\nBoth use a bucket metaphor, but they answer different questions: a token bucket asks \"how much can this client burst right now,\" while a [leaky bucket](/blog/what-is-a-leaky-bucket-algorithm-2026) asks \"what is the maximum constant rate this should ever produce.\" See [rate limiting strategies](/blog/rate-limiting-strategies-2026) for how the two compare directly against fixed-window and sliding-window approaches.\n\n## Common pitfalls\n\n**Setting capacity too high relative to what downstream can absorb.** A generous burst allowance is only safe if whatever is behind the limiter can actually handle that burst; otherwise the limiter has just moved the overload problem one layer down.\n\n**Forgetting refill happens continuously, not in ticks.** An implementation that only refills at fixed intervals rather than continuously introduces boundary effects similar to a fixed-window counter.\n\n**Running independent buckets per server without coordination.** In a multi-server deployment, uncoordinated local buckets can let a client exceed the intended aggregate limit by simply spreading requests across servers.\n\n## FAQ\n\n**Why allow bursts at all instead of a flat rate?**\nReal traffic is bursty by nature — a user loading a page, a batch job starting up. A flat-rate limiter penalizes normal usage patterns; a token bucket accommodates them without abandoning a long-run limit.\n\n**How is this different from a fixed-window rate limit?**\nA fixed-window counter resets sharply at a clock boundary, which can allow two bursts back-to-back right at the reset. A token bucket refills continuously, avoiding that boundary spike.\n\n**What happens when the bucket is empty?**\nThe request is either rejected, queued until a token becomes available, or delayed, depending on the implementation. Which behavior is right depends on whether the caller can tolerate waiting.\n\n**Can a token bucket be shared across multiple servers?**\nYes, using a centralized or replicated token store, though this adds coordination overhead compared to a single-process implementation.\n\n## Where to go next\n\n- [What is a leaky bucket algorithm in 2026](/blog/what-is-a-leaky-bucket-algorithm-2026)\n- [What is a fan-out/fan-in pattern in 2026](/blog/what-is-a-fan-out-fan-in-pattern-2026)\n- [Rate limiting strategies in 2026](/blog/rate-limiting-strategies-2026)","hue":320,"metaTitle":"Token Bucket Algorithm Explained in 2026","metaDescription":"What is the token bucket algorithm in 2026? How refill rate and bucket capacity allow controlled bursts, with pseudocode and real-world uses.","keywords":["token bucket algorithm 2026","token bucket vs leaky bucket","token bucket rate limiting","api throttling token bucket","token bucket burst capacity","token bucket pseudocode","network traffic policing","token bucket implementation"],"tldr":["**The model: tokens refill at a fixed rate up to a cap, and each request spends one** — if tokens are available, the request goes through immediately.","**Burst tolerance is the defining trait** — a full bucket lets a client send a quick burst all at once, unlike an algorithm that enforces a flat rate at every instant.","**Two numbers define all of its behavior** — the refill rate sets the sustained long-run limit, and the capacity sets the maximum burst size.","**It is close to the default for API throttling in 2026** — most rate-limited APIs, including AI inference endpoints, use some form of this rather than a hard per-window cap.","**Skip:** setting the bucket capacity so large that it stops functioning as a limit at all — a cap with no practical ceiling defeats the purpose of having one."],"readTime":"4 min read","wordCount":852},"related":[{"slug":"what-is-a-leaky-bucket-algorithm-2026","title":"What Is the Leaky Bucket Algorithm in 2026?","excerpt":"The leaky bucket algorithm turns bursty, unpredictable traffic into a smooth, constant output rate. Here is the bucket-and-hole model, how to implement it, and when it is the wrong tool for the job.","category":"code","date":"July 19, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":280},{"slug":"api-rate-limiting-2026","title":"API rate limiting in 2026: algorithms, patterns, and implementation","excerpt":"Rate limiting protects your API from abuse and keeps costs predictable, but the wrong algorithm causes thundering-herd failures. Here is the 2026 guide.","category":"code","date":"March 24, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":306},{"slug":"rate-limit-algorithms-2026","title":"Rate Limit Algorithms: Picking the Right One","excerpt":"Fixed windows allow double bursts at the boundary; token buckets allow bursts by design. Here is how each algorithm behaves and which to use where.","category":"code","date":"July 24, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":132}],"sameCategory":[{"slug":"bazel-build-explained-2026","title":"Bazel Explained: What Hermetic Builds Actually Cost","excerpt":"Bazel delivers reproducible, precisely cached builds and demands that you describe everything. Here is what you get, what it costs, and who should say no.","category":"code","date":"August 19, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":145},{"slug":"graphql-federation-2026","title":"GraphQL Federation: One Graph, Many Teams","excerpt":"Federation composes independently-owned subgraphs into a single API. Here is what it solves, the operational cost it adds, and when a simpler gateway is enough.","category":"code","date":"August 19, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":300},{"slug":"hermetic-builds-explained-2026","title":"Hermetic Builds Explained: Same Inputs, Same Output","excerpt":"A hermetic build depends only on declared inputs. Here is what that property gives you, the common leaks that break it, and how to move toward it incrementally.","category":"code","date":"August 19, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":130},{"slug":"monorepo-tooling-compared-2026","title":"Monorepo Tooling Compared: What You Actually Need","excerpt":"Monorepo tools solve task orchestration and caching, not repository layout. Here is what each tier provides and how to tell when you have outgrown the simple option.","category":"code","date":"August 19, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":150}],"newer":{"slug":"what-is-a-thunderbolt-5-2026","title":"What is Thunderbolt 5 in 2026: speed, ports, and who needs it","excerpt":"Thunderbolt 5 roughly doubles the bandwidth of Thunderbolt 4 and adds smarter dynamic allocation. Here is what it actually changes for real laptops and docks in 2026.","category":"tech","date":"July 21, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":240},"older":{"slug":"what-is-a-trie-2026","title":"What Is a Trie? The Prefix Tree Explained for 2026","excerpt":"A trie stores strings character by character so shared prefixes share memory and search time. Here is how the prefix tree works and why autocomplete and routing tables rely on it in 2026.","category":"code","date":"July 21, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":200}}