SYSTEM DESIGN:Lesson 16: Crawling Billions of Pages

One billion pages — your laptop gave up at page 50,000

The core crawler from the previous lesson works. One process, polite, ~86K pages per day. Then your PM says: "Index the entire Indian news web before next IPL season." That is roughly a billion URLs, refreshed weekly.

You cannot scale by buying a bigger CPU. You need a coordinator, a worker pool, a shared frontier, and dedup that survives restarts. Same crawl loop — different infrastructure.

This chapter takes Lesson 15 and distributes it: Redis/Kafka frontier, Bloom filter + URL store at scale, per-domain politeness queues, napkin math, and the pushback interviewers save for senior rounds.

Crawler bot army vs one smoking laptop

Your laptop tapped out at page 50,000. The coordinator did not.

Web crawler architecture at scale

Distributed architecture overview

                    +------------------+
                    |   Coordinator    |
                    | (seeds, metrics) |
                    +--------+---------+
                             |
              +--------------+--------------+
              v              v              v
        +-----------+  +-----------+  +-----------+
        |  Worker 1 |  |  Worker 2 |  |  Worker N |
        | fetch+parse|  | fetch+parse|  | fetch+parse|
        +-----+-----+  +-----+-----+  +-----+-----+
              |              |              |
              +--------------+--------------+
                             |
              +--------------v--------------+
              |  Shared URL Frontier        |
              |  (Redis ZSET or Kafka)    |
              +--------------+--------------+
                             |
        +--------------------+--------------------+
        v                    v                    v
  Bloom filter cluster   URL store (Cassandra)   Document store (S3)

Workers are stateless. All coordination state lives in shared services. Kill a worker, spawn another — crawl continues.

Coordinator — brains, not muscle

The coordinator does not fetch pages. It:

• Loads seed URLs and sitemaps into the frontier on schedule
• Monitors crawl progress (pages/hour, error rate, frontier depth)
• Rebalances workers when queues stall
• Triggers re-crawl of stale high-priority domains

During IPL, the coordinator boosts priority for cricket domains and injects fresh seed URLs every hour. Workers do not need to know why — they just dequeue and fetch.

One coordinator is enough with leader election (ZooKeeper, etcd). Two coordinators fighting over seeds is worse than one coordinator with a cold.

Shared URL Frontier — Redis or Kafka

In-memory heap dies at millions of URLs. Move the frontier to shared storage.

Option A: Redis sorted set (ZSET)

• Score = priority (higher = fetch sooner)
• Member = normalized URL
• ZPOPMAX atomically grabs highest-priority URL — safe for concurrent workers

def dequeue_url() -> str | None:
    result = redis.zpopmax("frontier:global", count=1)
    if not result:
        return None
    url, score = result[0]
    return url

Option B: Kafka topic

• Partition by hash(domain) % N — all URLs for one domain land in one partition
• Consumer group = worker pool; each partition processed by one worker at a time
• Natural fit for per-domain politeness (see below)

Redis ZSET is simpler for interviews. Kafka wins when you already have it for other pipelines and need replay on failure.

Worker pool — horizontal fetch power

Each worker runs the same loop from Lesson 15:

1. Dequeue URL from shared frontier
2. Check Bloom filter + URL store (dedup)
3. Wait for domain politeness slot
4. Fetch, parse, store
5. Enqueue discovered links back to frontier

Scale workers on CPU and outbound bandwidth:

10 workers x 1 req/sec/domain (round-robin) ~ 10 concurrent fetches
100 workers ~ 100 concurrent fetches (watch per-domain caps!)

Workers are stateless containers behind an autoscaler. Queue depth in Redis/Kafka is your scaling signal — same pattern as Scaling Basics.

Dedup at scale — Bloom filter + URL store

A billion URLs cannot live in one Bloom filter on one machine. Two-tier dedup:

Tier 1: Distributed Bloom filters

• Partition URLs by hash(url) % num_filters
• Each filter shard holds ~100M URLs with 0.1% false positive rate
• False positive = skip a URL you never crawled (rare, acceptable)

def bloom_shard(url: str) -> int:
    return hash(url) % NUM_BLOOM_SHARDS

def probably_seen(url: str) -> bool:
    shard = bloom_shard(url)
    return bloom_client[shard].contains(url)

Tier 2: URL store (Cassandra / ScyllaDB)

Bloom filter says "maybe new" → check durable store:

-- Partition key = hash(url) for even spread
CREATE TABLE url_seen (
    url_hash   BIGINT PRIMARY KEY,
    url        TEXT,
    first_seen TIMESTAMPTZ,
    status     TEXT  -- queued | fetched | failed
);

Insert-before-fetch prevents two workers from crawling the same URL. Use conditional write or "insert if not exists" semantics.

Politeness at scale — per-domain queues

Global rate limiting is wrong. Cricbuzz and a tiny cricket blog should not share one token bucket.

Per-domain queue pattern:

frontier:cricbuzz.com   -> ZSET (priority queue, max 1 pop/sec)
frontier:espncricinfo.com -> ZSET
frontier:timesofindia.com -> ZSET

Workers round-robin across domain queues. Each domain enforces its own delay (from robots.txt Crawl-delay or your default).

DOMAIN_QUEUES = redis.keys("frontier:*")

def fair_dequeue() -> str | None:
    for queue_key in round_robin(DOMAIN_QUEUES):
        domain = queue_key.split(":")[1]
        if not rate_limiter.allow(domain):
            continue
        url = redis.zpopmax(queue_key, count=1)
        if url:
            return url[0][0]
    return None

Per-domain token buckets map directly to Rate Limiter Architecture — one bucket per hostname, stored in Redis.

Napkin math — 1 billion pages

State these numbers on the whiteboard before drawing boxes.

Crawl throughput:

Target: 1B pages crawled in 7 days (weekly refresh)
1B / 7 days / 86,400 sec ~ 1,650 pages/sec cluster-wide
Peak (2x): ~3,300 pages/sec

**Workers (assume 5 concurrent fetches per worker, avg 2 sec/page including politeness wait):

Each worker: ~2.5 pages/sec
3,300 / 2.5 ~ 1,320 workers at peak
Off-peak: autoscale down to ~400 workers

Storage:

1B pages x 50 KB avg HTML = 50 TB raw HTML (S3)
Parsed metadata ~2 KB/page = 2 TB in Cassandra
URL store ~100 bytes/URL x 1B = 100 GB
Bloom filters: ~1.2 GB per 1B URLs at 0.1% FP (10 shards x 120 MB)

Bandwidth:

3,300 pages/sec x 50 KB = ~165 MB/sec outbound (~1.3 Gbps)
Manageable with regional egress and compression

Round aggressively. Interviewers want the shape: thousands of workers, tens of TB, politeness is the real bottleneck — not CPU.

Document store at scale

• Raw HTML → S3/GCS with path s3://crawl/{hash[0:2]}/{hash[2:4]}/{url_hash}.html
• Metadata → Cassandra partitioned by url_hash for O(1) lookup
• Change detection → compare content_hash; skip re-index if unchanged

Search indexers consume a Kafka topic pages_fetched — decouple crawl from index pipeline. Crawler publishes event; indexer subscribes. Crawler does not wait for Elasticsearch.

Same async boundary as click analytics in URL Shortener Core — never block the hot path on downstream slowness.

Consistent hashing — where URLs live

When you shard Bloom filters, URL store, or document metadata across nodes, hash(url) % N breaks every time N changes.

Use consistent hashing so adding shard 11 moves ~1/11 of keys, not everything:

• URL store shards on a hash ring
• Bloom filter shards as virtual nodes on the same ring
• Workers unchanged — they talk to a routing layer

Full treatment: Consistent Hashing Basics and Consistent Hashing in Production. Mention this when the interviewer asks "how do you add capacity without reshuffling?"

Failure handling

Crawlers are messy. Design for retry and eventual completeness, not perfect real-time coverage.

Interview pushback — senior round traps

"What about duplicate content?"

Same article syndicated across Times of India, Navbharat Times, and a partner site. Answer:

• Normalize to canonical URL from <link rel="canonical">
• Content fingerprint (SimHash or shingle hash) — cluster near-duplicates
• Index one representative; store alternates for redirect or attribution

"What about JavaScript-rendered pages?"

React SPAs with empty <body> until JS runs. Options:

• Headless browser farm (Chromium pool) — 10× cost, 5× latency
• Render-on-demand — crawl HTML first; queue JS render only for known SPA domains
• Partnership — some sites expose server-rendered or API feeds for crawlers

Say: "HTML-first for MVP; headless render as a separate tier for flagged domains."

"Crawler traps — infinite URL spaces"

/page/1, /page/2, ... forever. Or calendar links generating /2026/05/26, /2026/05/27, ...

• Max depth from seed (e.g., 10 hops)
• Max URLs per domain (e.g., 500K)
• URL pattern blocklist (/calendar/*, ?page= with page > 1000)
• Detect duplicate page content via SimHash — stop expanding that branch

"How fresh is news during IPL?"

• Priority boost for domains + URL patterns matching live scores
• Re-crawl high-priority URLs every 60 seconds during events
• Separate fast lane frontier (Redis ZSET frontier:fast) consumed first

Pushback questions test whether you have operated a crawler, not just read a blog post. Admit trade-offs honestly.

Monitoring at scale

• Pages fetched/sec — cluster throughput vs target
• Frontier depth — per domain and global; growing frontier = need more workers
• Dedup hit rate — Bloom filter blocking repeats (expect > 90%)
• Politeness wait time — p99 delay before fetch; spikes = too aggressive
• HTTP error rate by domain — 403/429 = you are being blocked
• Storage growth — S3 bytes/day vs budget

The 5-minute interview answer

1. Clarify: discover + fetch + parse + store, billions of URLs, politeness required
2. Components: coordinator, stateless workers, shared frontier (Redis/Kafka)
3. Dedup: distributed Bloom filter + Cassandra URL store, insert-before-fetch
4. Politeness: per-domain queues + token bucket rate limits
5. Storage: S3 for HTML, Cassandra for metadata, Kafka to indexers
6. Napkin math: ~1,650 pages/sec for 1B/week, ~1,300 workers peak, ~50 TB HTML
7. Pushback: canonical URLs, crawler traps, JS rendering as optional tier

Draw coordinator on top, workers in the middle, shared frontier + dedup + storage at the bottom. Label the politeness layer — that is what separates your answer from "just spawn more threads."

Wrap-up checklist

You do not launch 1,300 workers on day one. Say what you ship at 1M pages/day vs 1B — interviewers want evolution.

What comes next

Crawlers collect data. Notification systems push data out — the opposite fan direction. Swiggy telling ten million users their order arrived uses the same queue-and-worker thinking, different SLA.

Continue here: Notification System Core.