Scaling and Load Balancing

Vertical scaling (scale-up): add CPU, RAM, or faster disks to one machine. Simple — no code changes, no coordination overhead. The ceiling is real: a 192-core AWS u-24tb1 costs ~$220/hr and still breaks at ~500k QPS on a single database. Horizontal scaling (scale-out): add more identical nodes. Theoretically unlimited, but requires your application to tolerate the new topology.

Real numbers to anchor intuition: a single c6g.4xlarge EC2 instance handles ~30k HTTP requests/sec. Add 10 of them behind a load balancer and you get ~300k req/sec at ~15% of the cost of one monster box. Scale-out wins on price-performance beyond ~4x vertical growth — which is why every large system eventually horizontalizes.

The hard prerequisite: your nodes must be stateless, or scaling adds more problems than it solves.

A service is stateless if any node can handle any request with zero knowledge of prior interactions. Authentication tokens, request context, and session data live in the request itself (JWT) or in shared external storage (Redis, Postgres) — not in the server's local memory.

A stateful service remembers you. Classic failure mode: user logs in on Node A, next request routes to Node B, Node B has no session → user gets kicked out. Solutions in order of increasing complexity: sticky sessions (bind a client to one node), distributed session stores, or redesign to stateless.

Why it matters for scaling: stateless nodes are interchangeable. You can add 50 of them, kill any 10, and the load balancer adjusts without user impact. Sticky sessions destroy this property — they turn horizontal nodes into snowflakes. The design rule: push state out to Redis/Postgres; keep your app tier stateless.

Layer 4 (transport-layer) LBs route based on IP and TCP/UDP port. They see no HTTP headers, no URLs, no cookies. Ultra-fast (kernel bypass, 10M+ connections/sec on AWS NLB) and TLS-agnostic — great for raw TCP throughput, databases, or protocols the LB shouldn't inspect.

Layer 7 (application-layer) LBs parse HTTP. They can route /api/* to one backend fleet and /static/* to another, insert X-Forwarded-For, terminate TLS, inspect cookies for sticky sessions, and return a 503 before a bad request hits your app. Latency overhead: ~0.5–2 ms. Worth it for most web workloads.

NGINX and HAProxy are the dominant open-source L7 choices; AWS ALB and GCP HTTPS LB are managed equivalents.

Round-robin: requests cycle through nodes in order. Zero memory, works when nodes are homogeneous. Breaks badly if request cost varies wildly (one slow DB query holds a slot).

Least connections: route to the node with the fewest active connections. Better for heterogeneous load — long-running requests don't starve new ones. ~2–5% CPU overhead at the LB.

IP hash / consistent hashing: hash the client IP (or a session key) to a node. Provides soft stickiness without explicit session state at the LB. Consistent hashing with virtual nodes (used in Cassandra, Nginx upstream) minimizes reshuffling when nodes join/leave — only ~1/N keys remap, not all of them.

upstream backend {
    consistent_hash $request_uri;
    server app1:8080;
    server app2:8080;
    server app3:8080;
}

Weighted round-robin: assign weight proportional to node capacity. Useful during blue-green deploys (10% weight on new version = canary).

Traffic path from client to auto-scaled app tier, with health checks and shared state.

A load balancer without health checks routes to dead nodes — guaranteed 5xx storms. Every serious LB implementation runs two kinds:

Active checks: the LB polls GET /healthz every 5–10 seconds. A node is marked unhealthy after N consecutive failures (typically 2–3) and re-enabled after M consecutive successes (typically 2). Tune aggressively — 10-second detection + 20-second recovery = 30 seconds of 5xx traffic.

Passive checks (circuit breaking): the LB watches real request error rates. NGINX upstream supports max_fails=3 fail_timeout=30s. HAProxy tracks rolling error percentages. AWS ALB uses target group health thresholds.

Gotcha: a /healthz endpoint that returns 200 while the app is deadlocked is worse than useless. Make the health check actually exercise a DB query or Redis ping. Budget < 100 ms for the check to complete, so it doesn't itself become a bottleneck.

Autoscaling turns capacity from a fixed cost into a variable one. The two models:

Reactive autoscaling: trigger on CPU > 70% or request queue depth > 500. AWS Auto Scaling Groups, GCP Managed Instance Groups, and Kubernetes HPA all support this. Lag problem: spinning up a new EC2 instance takes 90–120 seconds — if your traffic spike lasts 60 seconds, reactive scaling never fires in time.

Predictive autoscaling: train on historical traffic patterns (Monday 9 AM always spikes) and pre-warm capacity. AWS predictive scaling, GCP Scheduled Autoscaler. Reduces cold-start lag to near zero for predictable workloads.

Scale-in is dangerous: terminate a node mid-request and you drop traffic. Always use connection draining (graceful shutdown) — wait for in-flight requests to complete (typically 30 s timeout) before killing the instance. Kubernetes does this via terminationGracePeriodSeconds.

Here's the irony: you scale your app to 20 nodes, then route all traffic through one load balancer. That LB is now your SPOF.

Active-passive LB pairs: one LB is primary; the standby takes a VIP (virtual IP) via VRRP/keepalived if the primary dies. Failover: 5–15 seconds. Common with bare-metal Nginx or HAProxy.

Active-active LB pairs: both LBs handle traffic simultaneously. DNS round-robins between them. Failure removes one; traffic routes to the survivor with no failover delay.

Managed cloud LBs (AWS ALB, GCP LB) are regionally redundant by design — the SPOF risk shifts to your configuration. But a misconfigured security group that blocks the health check port has the same effect as a dead LB.

Design rule: anything in the critical path of 100% of your requests must be HA. Draw the traffic flow end-to-end; circle every single hop; ask "what breaks if this dies?"