SYSTEM DESIGN:Lesson 6: Building a Distributed Rate Limiter

The per-pod lie

Your team shipped rate limiting last sprint. Each Express middleware checks an in-memory Map: 100 requests per minute per API key. QA passes. You deploy to production with 12 pods behind an AWS ALB.

A power user with one API key now gets 1,200 requests per minute — 100 on each pod. Your limiter is working perfectly on every machine and failing completely as a system.

This chapter is about building a distributed rate limiter: shared counters, atomic updates, where to put the check in your request path, and what happens when the limiter itself breaks.

Twelve pods, one API key, twelve times the limit

Each pod thinks it is doing its job. The system disagrees.

Centralized counter with Redis

The standard production pattern: one fast, in-memory data store that all app servers talk to before serving a request. Redis is the usual pick — sub-millisecond reads, atomic commands, TTL support.

Distributed rate limiter architecture

Request flow:

Client → Load Balancer → API Gateway → Rate Limiter Check → App Server → Database
                              ↓
                           Redis

Fixed window with Redis (production-ready version):

-- Atomic Lua script: INCR + set TTL in one round trip
local key = KEYS[1]
local limit = tonumber(ARGV[1])
local window = tonumber(ARGV[2])
local current = redis.call("INCR", key)
if current == 1 then
  redis.call("EXPIRE", key, window)
end
if current > limit then
  return 0  -- rejected
end
return 1  -- allowed

Why Lua? Two separate commands (INCR then EXPIRE) can race across network partitions. A script executes atomically on the Redis server. One round trip, no race.

For token bucket, store {tokens, last_refill} as a Redis hash and run the refill math inside a Lua script. Same principle — read-modify-write must be atomic.

Race conditions without atomicity

Here is the bug that ships in code review when someone writes:

# BUG: not atomic — two pods can both read count=99 and both allow request 100 and 101
count = redis.get(key) or 0
if count < 100:
    redis.incr(key)
    return allow()
return reject()

Under 1,000 concurrent requests, this overshoots your limit regularly. The fix is always one of:

• Redis INCR (single command, atomic by design)
• Lua script (multiple operations, one atomic unit)
• Redis transaction with WATCH (optimistic locking — slower, rarely needed here)

I have debugged a production incident where a fintech's UPI webhook endpoint allowed 3× the intended rate because of a read-then-write race. Switching to INCR fixed it in one deploy. Atomic ops are not optional.

Where to place the limiter

You have four realistic options. Each shifts cost and coverage.

1. API Gateway (Kong, AWS API Gateway, Envoy)

• Pros: blocks bad traffic before it hits app servers; centralized config; no code change per service
• Cons: gateway becomes a dependency; custom logic (per-endpoint limits, cost-based GraphQL) is harder
• When: multi-service architecture, standard per-key limits, you already run a gateway

2. Service mesh sidecar (Istio, Linkerd)

• Pros: per-service limits without app code; mTLS and observability bundled
• Cons: operational complexity; team needs mesh expertise
• When: Kubernetes at scale, platform team owns infra

3. Application middleware

• Pros: full control — different limits per route, custom error messages, business logic (trial users vs paid)
• Cons: every service implements it (or shares a library); traffic already reached the app
• When: single monolith or small number of services, complex limit rules

4. CDN / edge (Cloudflare Rate Limiting, Fastly)

• Pros: blocks traffic at the edge, closest to the attacker; protects origin entirely
• Cons: less granular per-user logic unless you pass auth to edge; vendor lock-in
• When: DDoS protection, public endpoints, global traffic

My opinion for most startups: API gateway + Redis for standard limits, app middleware for business-specific rules (e.g., "free tier gets 10 exports/day"). You do not need a service mesh on day one.

Layer them if needed: Cloudflare blocks 10,000 req/s from a bot IP at the edge; your gateway enforces 100/min per API key; your app rejects expensive operations separately.

Multi-region reality

Redis in Mumbai and Redis in Virginia are two separate stores unless you run active-active replication (hard). Options:

Regional limits (most common): each region has its own Redis. A user gets 100/min in ap-south-1 AND 100/min in us-east-1. Total could be 200/min globally. Acceptable for most products.

Global limits: route all rate limit checks to one Redis cluster (cross-region latency — adds 50–150ms per request). Only worth it for strict quotas (billing, compliance).

Approximate global: use a CRDT or gossip protocol between regional counters. Complex. Mention it in interviews to show depth; do not build it unless you have to.

During IPL final traffic spikes, Indian users hit ap-south-1. Regional Redis handles it fine. Do not over-engineer global consistency for a rate limiter.

Fail open vs fail closed

Redis goes down. What does your API do?

There is no universal right answer. It depends on what breaks worse.

Fail open if you are a consumer app (Swiggy, Zomato). A 5-minute Redis blip should not stop people from ordering food. You accept some abuse risk and alert ops.

Fail closed if you are a payment API or SMS gateway. Letting unlimited requests through during an outage could cost real money (SMS charges, fraud). Better to return 503 and tell clients to retry.

Local fallback is the pragmatic middle path many teams use:

try:
    allowed = redis_rate_limit_check(user_id)
except RedisConnectionError:
    allowed = local_rate_limit_check(user_id, limit=20)  # stricter local cap

Document the decision in your design doc. Interviewers ask this explicitly — "Redis is down, what happens?" Have an answer with reasoning, not a shrug.

Key naming and sharding Redis

Keep Redis keys predictable and short:

ratelimit:{user_id}:{window_start}     # fixed window
ratelimit:tb:{user_id}                 # token bucket hash

At 1 million active users with fixed-window keys, you hold ~1M keys per minute window. Redis handles this, but set TTL on every key so expired windows clean up automatically.

For very large deployments, shard rate limit data across multiple Redis nodes by hashing user_id % N. Each shard owns a slice of users. Same pattern as database sharding — just for counters.

Performance numbers that matter

• Redis GET/INCR: ~0.5–1 ms on same AZ
• Cross-AZ Redis: +1–3 ms
• Lua script: similar to INCR, one round trip
• Target: rate limit check adds < 5 ms p99 to request latency

At 10,000 req/s, Redis handles rate limit keys easily — it is built for this. The bottleneck is usually your app, not the counter.

Optimizations when you hit scale:

• Local cache with sync — check local cache first, sync to Redis every N requests (approximate, reduces Redis load)
• Batch checks — for internal service-to-service calls, check every 10th request
• Separate Redis instance — do not share the rate limit Redis with session cache or pub/sub

Observability

Ship metrics from day one:

• rate_limit_allowed_total — counter by endpoint, tier
• rate_limit_rejected_total — counter; spike here means abuse or misconfigured client
• rate_limit_redis_latency_ms — histogram; p99 > 10ms means trouble
• rate_limit_redis_errors_total — triggers fail-open/closed path

Log rejected requests with user_id, IP, endpoint, and current count. When a customer says "I got 429," you need to answer in 5 minutes, not 5 hours.

Interview follow-ups worth preparing

"How do you rate-limit WebSocket connections?" — limit connection count per user at handshake time, then message rate per connection. Different lifecycle than HTTP.

"How do you handle rate limit bypass via multiple accounts?" — that is fraud detection, not rate limiting. Mention device fingerprinting, CAPTCHA, anomaly detection as separate layers.

"Can you rate limit without Redis?" — yes: Nginx limit_req, Envoy local rate limit, or centralized DB (too slow). Redis is the default for a reason.

"What about sticky sessions so in-memory works?" — sticky sessions help with session state, not rate limiting. Users reconnect, pods scale up/down, stickiness breaks. Shared store wins.

Put it together

A solid interview answer in 5 minutes:

1. Pick token bucket or sliding window (previous chapter)
2. Store state in Redis with atomic Lua scripts
3. Check at API gateway for standard limits, app middleware for custom rules
4. Fail open for consumer apps, fail closed for billing APIs
5. Emit metrics on allowed/rejected/error paths

That is a complete, production-aware answer. Not a textbook — a system someone could actually deploy.

Practice the full flow in System Design Challenges — wire up the limiter and watch what happens when traffic spikes.