Rate Limiting

Without limits, a single client can:

• Burn your database to the ground in 30 seconds
• Run up a $20K cloud bill before you wake up
• Starve every other user of API capacity
• Probe for vulnerabilities at 10K req/s

Rate limits are how you set boundaries — per user, per endpoint, per IP, per API key — that protect the service from any one caller. They don't replace authentication or DDoS protection; they sit alongside both. A well-limited API can survive bad clients, runaway scripts, and accidental loops without paging anyone.

The simplest algorithm: count requests in fixed time windows (e.g. per minute). When the count exceeds the limit, reject.

key = f"rate:{user_id}:{now.minute}"
count = redis.incr(key)
if count == 1:
    redis.expire(key, 60)
if count > 100:
    return 429

The problem: boundary effect. A user can send 100 requests at 11:59:59 and 100 more at 12:00:01 — 200 requests in 2 seconds, technically within the rules. Fine for very rough quotas, bad for protecting against bursts. Don't ship this for anything that needs real safety.

Fix the boundary problem by tracking exactly when each request happened. Two variants:

• Sliding window log: store a timestamp per request; on each new request, count timestamps inside the trailing window. Correct, but memory grows with request rate.
• Sliding window counter: weighted average of the current and previous fixed windows based on how far into the current one you are. Approximate but uses constant memory.

Most production limiters use the counter variant: 10–20% inaccurate at burst boundaries, but rock-solid memory characteristics. The log variant is reserved for small-scale or strict-correctness needs.

Token bucket is the most flexible algorithm. Two parameters:

• Bucket size (max tokens): the burst allowance.
• Refill rate (tokens/sec): the sustained rate.

Each request consumes a token. Tokens regenerate over time. An empty bucket means rejection (or waiting). A full bucket means a burst is available.

def allow(user_id, max_tokens=100, refill_per_sec=10):
    now = time.time()
    bucket = redis.hgetall(f"tb:{user_id}")
    tokens = float(bucket.get("tokens", max_tokens))
    last = float(bucket.get("last", now))
    tokens = min(max_tokens, tokens + (now - last) * refill_per_sec)
    if tokens < 1:
        return False
    redis.hset(f"tb:{user_id}", mapping={"tokens": tokens - 1, "last": now})
    return True

Great default. Behaviour is intuitive: "you can burst up to 100, sustained 10/s". Used by AWS, GitHub, Stripe, almost everyone.

Tokens drip in; requests drink them.

Leaky bucket is similar in mental model but inverted: requests fill a bucket, the bucket leaks at a constant rate, overflowing requests get rejected.

Key behavioural difference vs token bucket: leaky bucket does not allow bursts. The serving rate is constant. Useful when downstream can't tolerate spikes (a 3rd-party API that itself rate-limits you, or a slow database).

Most teams default to token bucket because real users come in bursts and a small burst allowance is friendly. Choose leaky bucket only when you specifically need to smooth traffic — when downstream limits force you to.

With multiple API servers, the limit needs to be per-user across all servers. Two patterns:

1. Central counter (Redis): every server INCRs a shared key. Trivial to implement, adds a round-trip per request. Latency hit on the hot path. Use a Redis cluster or a single-shard fast Redis to keep this manageable.
2. Local-with-sync: each server keeps a local counter with a small allowance and periodically syncs to a central store. Lower latency, slightly looser limit enforcement. Used at very high scale (Cloudflare, AWS) where a Redis hop is too expensive.

For most teams, Redis is fine — the round-trip is sub-millisecond on a co-located cluster. Premature optimisation in this space is common; build the simple thing first, optimise if you actually hit Redis as a bottleneck.

When you reject a request, the client deserves to know:

• Use HTTP 429 Too Many Requests (not 503; reserved for actual outages).
• Include Retry-After: <seconds> so the client knows when to retry.
• Send X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset headers on every response (not just errors) so clients can self-pace.
• Document the limits prominently.
• For repeat offenders, escalate: temporary block, then exponential cooldown, then ban with a contact link.

Bad rate-limit UX (cryptic 500s, no documentation, no Retry-After) drives users to retry harder, which makes the problem worse. Good rate-limit UX makes clients well-behaved automatically.