Token Bucket

A token bucket controls how much work a system admits over time. The bucket holds up to a configured number of tokens, and tokens replenish at a configured rate. A request proceeds only when enough tokens are available, consuming the required amount. Tokens added to an already full bucket are discarded, so unused capacity cannot accumulate indefinitely. When tokens are insufficient, the implementation may reject the request, delay it, or place it in a separate queue.

The two main settings are the refill rate and bucket capacity. For example, an API limiter might replenish 100 tokens per second and hold at most 200 tokens, with each request consuming one token. After sufficient idle time, a full bucket can admit a burst of 200 requests, subject to other system limits. Further requests depend on replenishment. In an ideal implementation, admitted work over an interval of t seconds cannot exceed the tokens available at the beginning plus 100 times t. This does not mean that every one-second window is limited to exactly 100 requests.

In cloud architectures, token buckets commonly appear at API gateways, service endpoints, application middleware, and network traffic controls. An API limiter typically measures requests, whereas a network limiter may charge tokens according to packet size or byte count. Weighted request costs are also possible: an expensive report-generation operation could consume more tokens than a lightweight metadata lookup. Always establish the unit being limited before interpreting a configuration.

For cost-optimized designs, place admission control before expensive backend processing where practical. Suppose a public API invokes compute functions that query a database. A token bucket at the gateway can reduce the amount of excess traffic reaching those components, limiting unnecessary compute, database activity, and downstream network traffic. However, rejected requests can still incur charges at the layer receiving them. Throttling is therefore a workload-control mechanism, not a guaranteed spending cap or a substitute for budgeting and security controls.

Bucket capacity is a deliberate trade-off. A larger bucket accommodates brief legitimate spikes but can overwhelm a backend that lacks burst capacity. A smaller bucket protects the backend more tightly but can reject useful requests even when average demand is acceptable. Choose the refill rate from sustainable downstream throughput, and size the burst allowance against available headroom and acceptable latency. A request-rate limit does not directly limit concurrency: slow requests can accumulate even when their arrival rate remains within the configured rate.

Token-bucket throttling is not automatically traffic smoothing. A rejecting limiter admits an allowed burst immediately; delaying traffic requires a shaping or queuing mechanism. A fixed-window counter instead counts requests within defined windows and can allow large spikes around window boundaries. When an exam scenario requires a controlled sustained rate with occasional bursts, token-bucket throttling is an appropriate match. When every accepted job must eventually execute, use durable buffering and controlled consumers rather than relying on rejection alone.

Clients should handle throttling responses, commonly HTTP 429, with bounded exponential backoff and jitter, respecting retry guidance when supplied. Immediate retries can amplify overload and costs. In distributed deployments, independent local buckets multiply the effective allowance unless coordinated or assigned portions of a shared limit. Managed-service throttles may also be best-effort targets rather than exact ceilings, so combine them with backend safeguards and monitor throttled requests, latency, errors, and resource utilization.