SYSTEM DESIGN:Lesson 18: Ten Million Push Notifications

IPL final over — ten million phones buzz at once

CSK wins. Cricbuzz taps "Send score alert." Ten million users opted in for IPL notifications. Sixty seconds later, every phone should show "CSK win by 6 wickets." Not six minutes. Not "we will retry tomorrow."

One worker calling FCM one-by-one would take hours. This is fan-out — one event becomes ten million delivery jobs. Queue partitioning, worker armies, provider rate limits, retries, dead-letter queues, and dedup keys at scale.

This chapter extends Lesson 17: same pipeline, different magnitude.

Ten million phones buzz for IPL score alert

One event. Ten million FCM calls. One very stressed worker pool.

Notification system architecture at scale

Fan-out — one event, ten million deliveries

Two fan-out patterns:

Pre-computed fan-out (pull model for feed)

Twitter timeline: write post to every follower's feed at publish time. Read is fast; write is expensive. Not ideal for push notifications.

On-demand fan-out (push model)

Event arrives → expand to recipient list → enqueue one job per user (or per batch) → workers deliver.

IPL alert event
    -> Fan-out service reads 10M user IDs (from "ipl_subscribers" segment)
    -> Enqueues 10M messages to partitioned queue
    -> 500 workers drain queue -> FCM batch API

For IPL alerts, on-demand fan-out is correct. You are not pre-writing ten million inbox rows — you are pushing once.

Architecture at scale

Event API -> Fan-out service -> Kafka (partitioned) -> Worker pool -> FCM / SMS / SES
                  |                    |                    |
                  |                    +-> DLQ              +-> Delivery status (Cassandra)
                  +-> Segment store (10M user IDs)
                  +-> Idempotency store (Redis)

Queue partitioning — parallelize safely

One queue for 10M messages creates a single consumer bottleneck. Partition:

Kafka topic: notifications_delivery
Partitions: 100
Key: hash(user_id) % 100
Each partition consumed by one worker in the group

100 partitions → up to 100 parallel workers draining simultaneously. Add partitions before adding workers (Kafka rule: partitions ≥ consumers).

def enqueue_delivery(user_id: str, campaign_id: str, payload: dict):
    kafka.produce(
        topic="notifications_delivery",
        key=user_id,  # same user always same partition — ordering per user
        value={
            "user_id": user_id,
            "campaign_id": campaign_id,
            "idempotency_key": f"{campaign_id}-{user_id}",
            "payload": payload,
        },
    )

Partition by user_id preserves order per user — two alerts for the same person arrive in sequence, not reversed.

Fan-out service — expand without blocking

Do not load 10M user IDs into memory at once. Stream from segment store:

def fan_out_campaign(campaign_id: str, event_type: str, payload: dict):
    cursor = segment_store.stream_users("ipl_subscribers")
    batch = []
    for user_id in cursor:
        batch.append(build_delivery_job(user_id, campaign_id, payload))
        if len(batch) >= 1000:
            kafka.produce_batch(batch)
            batch = []
    if batch:
        kafka.produce_batch(batch)

Fan-out itself is async — API returns 202 Accepted with campaign_id; expansion runs in background. Cricbuzz UI shows "sending..." while jobs enqueue over 30–60 seconds.

Segment store options:

• Precomputed list in S3 (CSV of user IDs) — fast for known broadcasts
• Query Postgres/Redis set — SMEMBERS ipl_subscribers if stored as Redis set
• Real-time query — "all users in Karnataka who opted in" (slow; avoid for 10M)

Worker scaling — queue lag is your signal

Workers are stateless containers. Autoscale on:

• Kafka consumer lag — messages waiting > threshold
• Queue depth — SQS ApproximateNumberOfMessages
• Age of oldest message — SLA breach predictor

Target: drain 10M jobs in 60 seconds
Required throughput: 10M / 60 ~ 167,000 deliveries/sec

That number is impossible for one FCM connection. You need batching and many workers — see napkin math below.

Same horizontal scaling playbook as Scaling Basics — stateless workers behind lag-based autoscaling.

Third-party provider rate limits

You do not control FCM's capacity. Google publishes limits (vary by tier):

• FCM HTTP v1: batch up to 500 messages per request
• Effective throughput depends on project quota — typically thousands to hundreds of thousands per minute for large apps
• SMS gateways (MSG91, Twilio): 100–1000/sec per account unless negotiated

Rate limiter per provider — mandatory:

fcm_limiter = TokenBucket(rate=5000, per="second")  # tune to quota

def send_batch_to_fcm(messages: list):
    fcm_limiter.acquire(len(messages))
    return fcm.send_each(messages)  # batch API

Apply Rate Limiter Architecture at the provider boundary — workers produce jobs faster than FCM accepts them without a limiter.

Multi-provider fallback: if FCM throttles (429), backoff exponentially. SMS fallback for critical alerts only — at 10M users SMS cost is ₹20 lakh per blast. Push-first is a financial decision, not just technical.

Delivery guarantees — at-least-once

Perfect exactly-once delivery to a phone is impossible — networks drop packets, users uninstall apps mid-flight. Design for at-least-once with idempotent consumers.

Producer -> Queue (persisted) -> Worker -> Provider
              |                    |
              |                    +-> may retry on timeout
              +-> message not acked -> redelivered

Duplicate delivery happens. User might get the same IPL alert twice if worker crashes after FCM accept but before ack. Mitigate with dedup keys.

Dedup keys at scale

Per-user per-campaign idempotency:

idempotency_key = "{campaign_id}-{user_id}"
Example: "ipl-final-2026-usr_abc123"

Store in Redis with TTL (campaign window + 24h):

def should_send(idempotency_key: str) -> bool:
    # SET NX — only first worker wins
    return redis.set(f"sent:{idempotency_key}", "1", nx=True, ex=86400)

def process_job(job: dict):
    key = job["idempotency_key"]
    if not should_send(key):
        return  # duplicate — skip silently
    send_push(job["user_id"], job["payload"])

Redis SET NX is atomic — two workers racing on the same retry both call FCM; only one passes dedup. Same pattern as Notification System Core, backed by Redis instead of Postgres for 10M keys/minute.

Retry and dead-letter queue (DLQ)

Transient failures — FCM 503, network timeout — deserve retry. Permanent failures — invalid token, user deleted — do not.

Attempt 1 -> fail (503) -> retry after 1 sec
Attempt 2 -> fail (503) -> retry after 5 sec
Attempt 3 -> fail (503) -> retry after 30 sec
Attempt 4 -> fail -> move to DLQ

MAX_RETRIES = 3

def handle_failure(job: dict, error: Exception):
    job["retry_count"] = job.get("retry_count", 0) + 1
    if job["retry_count"] > MAX_RETRIES:
        dlq.publish(job)
        return
    delay = 2 ** job["retry_count"]  # exponential backoff
    retry_queue.schedule(job, delay_sec=delay)

DLQ jobs get manual review or automated replay after provider outage clears. Alert on DLQ depth — 50K dead letters after IPL final is a pager event.

Separate retry topic in Kafka avoids blocking fresh deliveries behind retries.

Batching FCM — the only way to hit 60-second SLA

Individual FCM calls: ~100–200 ms each. 10M × 200 ms = 23 days. Unacceptable.

FCM sendEach or legacy multicast: up to 500 tokens per HTTP request.

def worker_loop():
    batch = consume_up_to(500)  # from partition
    tokens = [get_token(j["user_id"]) for j in batch]
    results = fcm.send_each(build_messages(batch, tokens))
    for job, result in zip(batch, results):
        record_status(job, result)
        if result.error == "NotRegistered":
            prune_token(job["user_id"])

500 messages per 200 ms ~ 2,500/sec per worker. 167,000/sec target ÷ 2,500 ~ 67 workers minimum — plus headroom for rate limits.

Napkin math — 10M users in 60 seconds

State these on the whiteboard:

Target throughput:

10M notifications / 60 sec ~ 167,000 deliveries/sec

With FCM batch size 500:

167,000 / 500 = 334 batch requests/sec
If each batch takes ~200 ms: 334 x 0.2 = 67 concurrent batch workers
Add 2x headroom for rate limits and retries: ~150 workers

Fan-out enqueue time:

10M jobs / 1000 per Kafka batch = 10,000 produce calls
At 1000 batches/sec: ~10 seconds to enqueue all jobs
Remaining 50 seconds for workers to drain

**Storage for status (optional analytics):

10M rows x ~200 bytes = 2 GB per campaign
Cassandra TTL 30 days — auto-expire old campaigns

**SMS fallback cost (if you did this — do not):

10M x Rs 0.20 = Rs 20,00,000 per blast
Push cost: ~Rs 0 marginal

Compare with Back-of-Envelope Estimation — round numbers, correct shape.

Priority queues — not all notifications are equal

IPL marketing alert and UPI "payment received" should not share one queue.

Topic: notifications_critical  (OTP, payment failed) — dedicated workers, no delay
Topic: notifications_transactional (order delivered) — standard workers
Topic: notifications_marketing (IPL alert) — lower priority, rate-limited harder

During IPL blast, critical queue stays empty. Marketing consumes spare FCM quota. Never let a million-user promo delay OTP delivery — that is how users lose money and trust.

Interview pushback — what senior interviewers ask

"What if FCM is down for 10 minutes?"

• Retry with exponential backoff; jobs stay in queue (Kafka retention)
• Alert ops; pause new marketing campaigns
• SMS fallback for critical only — not for 10M marketing
• Communicate "delayed notifications" in app if outage extends

"User opts out mid-campaign?"

• Check preferences at worker time, not fan-out time
• Fan-out enqueues all subscribers; worker skips if opt-out happened in last 5 minutes
• Trade-off: some wasted queue jobs vs complex cancellation protocol

"How do you avoid notifying deleted users?"

• FCM returns NotRegistered → delete token immediately
• Nightly job purges tokens with no successful delivery in 90 days
• Segment store refreshed before large campaigns

"Exactly-once delivery?"

• Honest answer: at-least-once with dedup keys; exactly-once to device is not achievable
• Idempotency key + Redis SET NX limits duplicates to edge cases
• Product tolerance: duplicate IPL alert rare and low harm; duplicate OTP is high harm — separate critical pipeline

"10M users but only 100K online?"

• Push still goes to all registered device tokens — "online" is irrelevant for FCM
• Offline devices receive when they reconnect (FCM queues briefly)
• Do not conflate "active users" with "reachable devices"

Pushback separates candidates who built this from candidates who watched a YouTube diagram once.

Monitoring at scale

• Campaign progress — sent / total / failed counts in real time
• Consumer lag — per partition; lag > 60 sec = SLA breach
• FCM error rate — 429 spikes = rate limit; 5xx = provider outage
• DLQ depth — alert threshold per campaign
• Dedup skip rate — high rate = retry storm or duplicate fan-out bug
• P99 delivery latency — event timestamp to FCM accept
• Cost dashboard — SMS sends × rate (prevent accidental SMS blast)

The 5-minute interview answer

1. Clarify: broadcast to 10M, push primary, 60-second target, at-least-once OK with dedup
2. Fan-out service streams segment → enqueues per-user jobs to Kafka (100 partitions)
3. Worker pool (150+) batches 500 FCM calls each; autoscale on lag
4. Rate limiter at FCM boundary; separate critical vs marketing queues
5. Idempotency: Redis SET NX on {campaign_id}-{user_id}
6. Retry 3x exponential → DLQ; prune dead tokens on NotRegistered
7. Napkin math: 167K/sec, 334 batch req/sec, ~150 workers

Draw event → fan-out → partitioned queue → workers → FCM. Label dedup and DLQ. Mention SMS cost if they push "what about fallback at 10M scale."

Wrap-up checklist

Scaling checklist (verbal):

• [ ] Fan-out async; API returns campaign_id immediately
• [ ] Partitioned queue; workers scale on lag
• [ ] FCM batching (500); rate limiter on provider
• [ ] Dedup keys in Redis; at-least-once assumed
• [ ] Retry + DLQ; separate critical queue
• [ ] Monitor lag, DLQ depth, FCM 429 rate, SMS cost

Ship transactional notifications first. Add broadcast fan-out when product proves subscribers — not the other way around.

Practice under pressure

Run a timed mock: Mock Interview — design a notification system for 10M users in 45 minutes.

Browse follow-up drills: Interview Prep.