SYSTEM DESIGN:Lesson 22: A Hundred Million Messages a Day

100 million people, one family group

Lesson 21 got a Diwali group working on one server. Now scale it. WhatsApp has 500+ million users in India alone. On Diwali eve, message rate spikes 10×. Every user holds a WebSocket open. Your cousin in Toronto and your nani in Varanasi must see the same message in the same order — mostly. Gateway servers die and reconnect. You need a plan.

This chapter: partition by conversation, WebSocket gateway scaling, presence in Redis, group fan-out, offline sync, napkin math, and interview traps around ordering and encryption.

Millions of WebSocket connections tangled like spaghetti

100 million users. 100 million open connections. One gateway crying.

Chat system architecture

Scaling chat is scaling connections, not just queries.

Napkin math — 100M DAU

Assumptions:

• 100M daily active users
• Average 50 messages sent per user per day (India skews high — festivals, groups)
• 60% of messages in groups (avg 20 members), 40% in 1:1
• Peak hour (8–10 PM IST): 5× average traffic

Total messages per day:

100M × 50 = 5 billion messages/day
5B / 86,400 ≈ 58,000 messages/sec average
Peak (5×): ~290,000 messages/sec

WebSocket connections (concurrent):

100M DAU, ~30% concurrently online at peak = 30M open connections
Each connection ≈ 4–8 KB memory → 120–240 GB RAM across gateway fleet

Storage per day:

5B messages × ~500 bytes (body + metadata) ≈ 2.5 TB/day raw
With indexes + delivery rows: ~5 TB/day
Retention 1 year → plan petabyte-scale (tiered: hot 30 days, cold archive)

State the message volume first. Interviewers care that you know chat is write-heavy, not read-heavy like URL shorteners.

Partition by conversation_id

Shard messages table by conversation_id — all messages in one conversation live on one shard.

Why conversation, not user?

• Message history query is always scoped to one conversation
• No cross-shard joins for "load last 50 messages"
• Group fan-out reads participant list once, writes messages to one shard

def shard_for_conversation(conversation_id: str) -> int:
    return consistent_hash(conversation_id) % NUM_SHARDS

Use consistent hashing so adding shards does not reshuffle every conversation (Lesson 9).

User's conversation list spans shards — denormalize user_conversations table keyed by user_id, or maintain inbox index per user.

-- Inbox index (separate from message shards)
CREATE TABLE user_inbox (
  user_id           BIGINT NOT NULL,
  conversation_id   UUID NOT NULL,
  last_message_at     TIMESTAMP,
  last_message_preview TEXT,
  unread_count        INT DEFAULT 0,
  PRIMARY KEY (user_id, conversation_id)
);

WebSocket gateway scaling

One server cannot hold 30M connections. You need a gateway tier — stateful servers that maintain WebSocket connections.

Client ──WS──► Gateway (holds connection)
                    │
                    ├──► Chat Service (business logic)
                    ├──► Message Store (sharded DB)
                    └──► Redis (connection registry + presence)

Problem: User A connects to Gateway-3. User B connects to Gateway-7. A sends message to B. How does Gateway-7 receive it?

Option 1: Sticky sessions (load balancer affinity)

Load balancer routes user_id → same gateway every time
On send to B: look up B's gateway from Redis, forward via internal RPC

Pros: simple mental model

Cons: rebalance pain when gateway dies; uneven load if power users cluster

Option 2: Connection registry in Redis (recommended)

On connect:  SET user:{uid}:gateway = "gateway-7"  EX 3600
On message:  gw = Redis GET user:{recipient_id}:gateway
             internal_publish(gw, { type: "deliver", message: ... })
On disconnect: DEL user:{uid}:gateway

Gateway fleet subscribes to a pub/sub channel per gateway ID. Message arrives on correct node, pushes to WebSocket.

def deliver_to_user(recipient_id: int, payload: dict):
    gateway_id = redis.get(f"user:{recipient_id}:gateway")
    if gateway_id:
        redis.publish(f"gw:{gateway_id}", json.dumps({
            "recipient_id": recipient_id,
            "payload": payload
        }))
    else:
        # offline — queue for sync + push notification
        offline_queue.enqueue(recipient_id, payload)
        notification_service.send_push(recipient_id, payload)

Heartbeat every 30 sec refreshes Redis TTL. Missed heartbeat → mark offline, clear registry.

Presence: online / offline / last seen

WhatsApp green dot. "Last seen today at 9:14 PM."

Redis key:  presence:{user_id}
Value:      { "status": "online", "gateway": "gw-7", "last_seen": 1730474040 }
TTL:        60 sec (refreshed by heartbeat)

Online: key exists, status=online

Offline: key expired or status=offline

Last seen: persist to DB on disconnect; show when offline

Do not query DB for presence on every message route — Redis sub-millisecond lookup.

Privacy: let users hide last seen (product feature, but mention in interview — "we respect presence visibility settings").

Group chat fan-out at scale

256-member office group. One message = 255 deliveries.

At 290k messages/sec peak, ~40% groups, avg 20 members:
fan-out deliveries ≈ 290k × 0.4 × 19 ≈ 2.2M delivery ops/sec

Do not synchronously loop 255 WebSocket pushes in the request path.

def send_group_message(conversation_id, sender_id, body):
    msg = message_store.insert(conversation_id, sender_id, body)
    recipients = get_participants(conversation_id) - {sender_id}
    # Async fan-out via queue
    for rid in recipients:
        delivery_queue.publish({
            "message_id": msg.id,
            "recipient_id": rid,
            "payload": serialize(msg)
        })
    ack_sender(sender_id, msg)
    return msg

Delivery workers consume queue, route via connection registry. Sender gets "sent" immediately; deliveries complete in milliseconds.

Same pattern as news feed fan-out — push to N recipients async. Festival spike = scale delivery workers horizontally.

Offline message sync

User offline for 6 hours — 400 messages across 12 conversations.

On reconnect:
  1. Client: { "type": "sync", "last_ids": { "conv_a": 991, "conv_b": 442, ... } }
  2. Server: batch-fetch messages WHERE id > last_id per conversation
  3. Return compact payload; client merges
  4. Client sends delivery ACKs in batch

Sync API is HTTP (or WebSocket after connect) — not real-time path. Paginate: 100 messages per conversation per sync batch.

Offline queue in Redis:

LPUSH offline:{user_id}  serialized_message
On connect: LRANGE + DEL

Short-term buffer for messages sent while user was briefly disconnected (< 24h). Long-term truth stays in sharded message DB.

Push via notification system

User fully offline — app killed, no WebSocket. FCM/APNs push required.

Chat Service → Notification Service → FCM/APNs → device

Notification payload: sender name + message preview (not full E2E ciphertext if encrypted). Tap opens app → sync API fetches full history.

Decouple chat from push delivery. Notification system handles retries, device tokens, rate limits — Notification System. Chat publishes event; notification service consumes.

def notify_offline_user(recipient_id, message):
    if not is_online(recipient_id):
        notification_service.send({
            "user_id": recipient_id,
            "channel": "push",
            "title": message.sender_name,
            "body": truncate(message.body, 100),
            "data": { "conversation_id": message.conversation_id }
        })

Interview pushback — prepare answers

"How do you guarantee message ordering?"

Per conversation: monotonic message_id (Snowflake or DB sequence per shard). Clients sort by ID.

Across conversations: no global order needed.

Caveat: clock skew between servers can cause rare out-of-order IDs — use server-assigned IDs, not client timestamps. Mention Snowflake IDs.

Group edge case: two members send simultaneously — both get sequential IDs on same shard; order is total within conversation.

"Exactly-once delivery?"

True exactly-once is impossible over unreliable networks. Aim for effectively-once:

1. Client generates client_msg_id (UUID)
2. Server deduplicates: UNIQUE(client_msg_id, sender_id)
3. Retries return same message_id, not duplicate row

At-least-once transport + idempotent server = effectively-once from user's perspective.

"What about end-to-end encryption?"

WhatsApp/Signal encrypt on device; server stores ciphertext blobs.

• Server cannot read message body — no ranking, no abuse scanning on content
• Key exchange via Signal protocol (X3DH + Double Ratchet) — mention, do not implement
• Delivery metadata (who, when) still visible to server
• Push notification shows generic "New message" unless client decrypts locally before backgrounding

Interview line: "E2E is a product/security requirement that changes our threat model. I would flag it early and scope server-side features accordingly."

"Gateway dies mid-conversation?"

• Client detects WS close → exponential backoff reconnect
• New gateway registers in Redis
• Sync API fills gaps since last_ack_message_id
• Brief duplicate delivery possible — client dedupes by message_id

Architecture at scale (verbal)

Clients → LB → Gateway fleet ↔ Redis (registry + presence)
              → Chat Service → Kafka → Sharded Message DB
              → Notification Service (FCM/APNs)

The 5-minute interview answer

1. Clarify: 1:1 + group, delivery states, 100M DAU, 50 msg/user/day
2. WebSocket gateways + Redis connection registry
3. Shard messages by conversation_id; inbox index per user
4. Group fan-out async via queue — 255 deliveries per message
5. Presence in Redis with heartbeat TTL
6. Offline: sync API + notification service for push
7. Ordering per conversation via server-assigned IDs; idempotent send with client_msg_id
8. Napkin math: ~290k msg/sec peak, 30M concurrent connections