The leaky bucket smooths traffic into a steady drip. But sometimes you want to allow bursts — a user who's been idle for an hour should be able to fire off 10 requests immediately, not be forced to wait one-per-second. That's the token bucket's job.

How it works: A bucket holds up to N tokens. Tokens refill at a steady rate (say, 10/sec). Each request consumes one token. If the bucket is empty, the request is rejected (or queued). If it's full, extra refill tokens are discarded.

Two knobs control behavior:

• Bucket capacity — the maximum burst size. A capacity of 100 means a fully-idle client can fire 100 requests instantly.
• Refill rate — the sustained long-term rate. 10 tokens/sec means over any long window, the client averages 10 req/sec.

Real-world example: AWS API Gateway uses token buckets for throttling. A default account gets a burst of 5000 and a steady rate of 10000 req/sec. If you've been quiet, you can hammer the API with 5000 requests in one go. Then you're capped at 10000/sec sustained. Stripe's API works similarly — you get room to retry a batch after a network hiccup without being punished for the burst.

The efficient implementation trick: Don't run a background thread that adds a token every 100ms. Instead, store last_refill_timestamp and tokens. On each request, compute lazily:

tokens = min(capacity, tokens + (now - last_refill) × rate)

Then decrement and update the timestamp. This is O(1) per request, no timers, and works perfectly in Redis with a Lua script for atomicity.

Rule of thumb for sizing: Set capacity to refill_rate × acceptable_burst_seconds. If you allow 100 req/sec sustained and want to tolerate 3-second bursts of double traffic, capacity ≈ 300. Too small and legitimate retry storms get rejected; too large and a bad actor can overwhelm your downstream with one shot.

Token bucket vs leaky bucket: Both enforce the same average rate. The difference is shape. Leaky bucket flattens output — good when the downstream system genuinely cannot handle spikes (a legacy database). Token bucket permits spikes — good when downstream can absorb bursts and you care more about fairness over time than instantaneous smoothness (most user-facing APIs).

One trap: if you share one bucket across all users, a noisy neighbor starves everyone. Bucket per API key or per IP — not per service.