SYSTEM DESIGN:Lesson 1: When One Server Isn't Enough
Mastering lesson 1: when one server isn't enough concepts and implementation.
Practice this chapter
Read the theory, then wire up the architecture yourself. These labs match what you just learned.
The moment your app breaks on a good day
"We launched a flash sale at 8 PM. By 8:03 PM, Swiggy-style notifications were going out, traffic spiked 40x, and our single server was gasping like it had run a marathon in a wool sweater."
That is not a failure of your product idea. That is a failure of capacity planning — or more honestly, a failure to admit that one machine has a ceiling.
Every system design interview and every real production incident eventually lands on the same question: what happens when more people show up than your server can handle? This chapter answers that with three ideas you will use for the rest of your career: server limits, vertical vs horizontal scaling, and load balancing.
No jargon for the sake of jargon. Just the mental model that turns "our site is slow" into "we hit a specific bottleneck, here is the fix."
Server melting during an IPL traffic spike
Yes, one box really can look like that when everyone refreshes the score at the same time.
One server has a ceiling — and you will hit it
Picture a single EC2 instance running your API, a PostgreSQL database, and Redis on the same box. Fine for a college project. Dangerous for anything with real users.
A server is not infinite. It has hard limits on CPU cores, RAM, disk I/O, network bandwidth, open file descriptors, and database connections. Push past any one of them and latency spikes — or the process crashes.
Single server bottleneck under traffic spike
| Resource | What it limits | Typical symptom when exhausted |
|---|---|---|
| CPU | How many requests you can compute per second | High CPU %, slow response times, timeouts |
| RAM | How much data you can hold in memory | OOM kills, swapping, GC pauses |
| Disk I/O | How fast you read/write persistent data | Query queues, write stalls |
| Network | How much data you send/receive per second | Bandwidth saturation, packet drops |
| Connections | How many clients/DB sockets stay open | "Too many connections" errors |
During an IPL final, a fantasy sports app might see 10 lakh concurrent users refreshing scores every 5 seconds. A single 4-core server handling 2,000 requests per second with 200ms average latency will melt. The math is not mysterious — it is arithmetic you can do on a whiteboard.
Rough capacity check:
Max sustainable RPS ≈ (number of cores × 1000) / avg latency in seconds
Example: 4 cores, 200ms (0.2s) per request
≈ (4 × 1000) / 0.2 = 20,000 RPS theoretical ceiling
In practice: cut that by 50–70% for safety → ~6,000–10,000 RPS
Those numbers are not gospel — they are a starting point. A poorly written N+1 query loop can turn a 20,000 RPS server into a 200 RPS embarrassment. But the point stands: one box has a number, and traffic growth eventually crosses it.
Vertical scaling: buy a bigger machine
Vertical scaling (scale up) means throwing more hardware at the same server: 8 cores → 32 cores, 16 GB RAM → 128 GB RAM, HDD → NVMe SSD.
It is the easiest fix. No code changes. No distributed systems headaches. Your PostgreSQL instance on a bigger machine genuinely handles more queries per second — until it does not.
When vertical scaling works:
- Early-stage startups with unpredictable but moderate traffic
- Databases that are hard to shard (strong consistency requirements)
- Batch jobs that need one monster machine, not ten small ones
- You need a fix tonight and cannot redesign architecture
When it stops working:
- Cloud providers cap instance sizes (you cannot buy infinite RAM)
- Downtime during resize — you often reboot the machine
- Single point of failure remains: one fire takes down everything
- Cost grows non-linearly; a 64-core box costs far more than 8× an 8-core box
I have seen teams on Razorpay-scale payment flows try to vertical-scale their way out of Diwali traffic. It bought them two weeks. Then they needed horizontal scaling anyway — with interest.
Horizontal scaling: add more machines
Horizontal scaling (scale out) means running multiple identical servers behind a load balancer. Each server handles a slice of traffic. Add more servers → handle more traffic.
Vertical scaling vs horizontal scaling
| Vertical (scale up) | Horizontal (scale out) | |
|---|---|---|
| How | Bigger single machine | More identical machines |
| Code changes | Usually none | Often requires stateless design |
| Failure mode | One machine dies → total outage | One machine dies → others absorb load |
| Cost curve | Expensive at the top end | Linear-ish if you use commodity hardware |
| Ceiling | Hardware max | Theoretically very large |
Horizontal scaling is how UPI handles 10+ billion transactions a month. Not one giant server — thousands of smaller ones, each doing a fraction of the work.
The catch: your application must be stateless at the server layer.
If user session data lives only in Server A's memory, and the next request lands on Server B, the user gets logged out. That is why production apps store sessions in Redis, not in-process memory.
Bad (sticky sessions required):
User login → session stored in app server RAM
Next request → must hit same server
Good (stateless):
User login → session stored in Redis
Any app server → reads session from Redis
Stateless app servers are the foundation of horizontal scaling. We will go deeper on the data layer — replicas, caching, sharding — in the next chapter.
Load balancing: who gets the next request?
A load balancer sits between clients and your app servers. Every incoming request hits the load balancer first. It picks a healthy backend server and forwards the request.
Load balancer distributing traffic across app servers
Think of it as the traffic cop at a Mumbai junction during rush hour — directing each car to a lane that is actually moving.
Traffic cop load balancer at a Mumbai junction
Round robin with a whistle. The Schoolabe hoarding in the back is optional in production — the health checks are not.
Common load balancing algorithms:
| Algorithm | How it works | Best for |
|---|---|---|
| Round robin | Request 1 → Server A, Request 2 → Server B, repeat | Equal-capacity, stateless servers |
| Least connections | Send to server with fewest active connections | Long-lived connections (WebSockets) |
| Weighted round robin | Servers with more CPU get more traffic | Mixed instance sizes |
| IP hash | Same client IP always hits same server | Sticky sessions (use sparingly) |
| Random | Pick any healthy server | Simple, surprisingly effective at scale |
Where load balancers live:
- Layer 4 (transport): routes by IP and port. Fast. Used by AWS NLB, HAProxy in TCP mode.
- Layer 7 (application): routes by URL path, headers, cookies. Smarter. Used by AWS ALB, Nginx, Envoy.
Layer 4 vs Layer 7 load balancer comparison
Layer 4 only sees the TCP envelope (IP + port) and cannot peek inside the HTTP request. Layer 7 terminates HTTP and can route `/api/payments/ to payment servers and /api/feed/` to feed servers from the same load balancer.
For a typical REST API, Layer 7 is the default choice. You can route /api/payments/* to payment servers and /api/feed/* to feed servers — same load balancer, different backends.
Health checks matter more than the algorithm.
A load balancer that keeps sending traffic to a crashed server is worse than no load balancer at all. Configure health checks: ping /health every 10 seconds, mark server unhealthy after 3 failures, stop routing traffic, alert your on-call engineer.
Putting it together: a realistic scaling story
Let us walk through how a food delivery startup might scale — because everyone in India has an opinion on Swiggy latency.
Stage 0 — MVP (1 server):
One server. Node.js API + PostgreSQL + file uploads on disk. Handles 500 orders/day. Fine.
Stage 1 — Vertical scale (1 bigger server):
Traffic hits 5,000 orders/day. Upgrade from t3.medium to t3.xlarge. Costs 4x. Buys 6 months.
Stage 2 — Horizontal app tier (load balancer + 3 app servers):
Weekend lunch rush causes 30-second page loads. Add an ALB, three stateless API servers, sessions in Redis. Database still on one machine — now that is the bottleneck.
Stage 3 — Data layer scaling (next chapter):
Read replicas, caching, CDN for restaurant images. This is where the real engineering lives.
Notice the pattern: scale the app tier first (cheap, stateless, easy), then attack the data tier (hard, stateful, where consistency trade-offs live).
Common mistakes in scaling conversations
Mistake 1 — scaling before you need to.
A team with 200 daily active users does not need Kubernetes. Premature scaling adds complexity that slows you down. Scale when metrics tell you to, not when a blog post tells you to.
Mistake 2 — scaling the wrong layer.
Adding 10 app servers when your database is the bottleneck just means 10 servers waiting on the same slow Postgres. Profile first. Find the bottleneck. Scale that.
Mistake 3 — ignoring state.
Horizontal scaling without moving sessions, file uploads, and in-memory caches to shared storage is a recipe for random bugs that only happen to 1 in 5 users.
Mistake 4 — no health checks.
A load balancer without health checks is a random request distributor to servers that might be on fire.
Mistake 5 — sticky sessions as the default.
IP-hash sticky sessions break when users switch from WiFi to mobile data. They also prevent even load distribution. Use shared session storage instead.
Interview talking points
When an interviewer asks "how would you scale this system?", start here:
- Estimate load — DAU, requests per user per day, peak-to-average ratio
- Find the bottleneck — CPU-bound? Memory? DB? Network?
- Scale the app tier — stateless servers + load balancer
- Scale the data tier — replicas, cache, sharding (next chapter)
- Measure and repeat — scaling is not a one-time event
Say "I would start with horizontal scaling of stateless app servers behind a load balancer" and you are already ahead of candidates who jump straight to Kafka.
Practice with hands-on labs
Reading about load balancers is not the same as watching one distribute traffic. These labs let you break things safely:
- Scaling Lab — add servers, watch throughput change
- Request Flow Lab — trace a request through load balancer → app → database
Run the scaling lab twice: once with one server, once with three. Write down the RPS difference. That number sticks in interviews better than any definition.
What comes next: data layer scaling
You now know why one server fails, when to scale up vs out, and how load balancers route traffic. The next bottleneck is almost always the database.
The next chapter covers read replicas, sharding, Redis cache-aside, CDN, and the five-stage evolution diagram that maps how real systems grow from one server to global scale.
Continue here: Data Layer Scaling.