Load Balancing

A load balancer sits between clients and a pool of backend servers, deciding which server handles each incoming request. Its three jobs:

1. Distribute work so no single server is overloaded while others idle.
2. Hide failures: if a backend dies, the LB stops sending it traffic.
3. Decouple addressing — clients only know one endpoint, even if backends come and go.

The payoff is horizontal scaling. Add a server to the pool, it absorbs a slice of traffic immediately. Lose a server, traffic redistributes. Without a load balancer, every backend change requires touching every client.

Load balancers operate at one of two networking layers, and the difference matters:

• Layer 4 (transport): decides based on IP + port. Fast, cheap, protocol-agnostic. Can't inspect HTTP — it just shuttles TCP/UDP packets.
• Layer 7 (application): parses HTTP. Can route by URL path, headers, cookies, content type. Can terminate TLS, modify requests, do compression.

L4 is what you use for raw throughput (databases, custom protocols). L7 is what you use for web traffic where routing by /api/... vs /static/... matters. Most modern apps end up with both: an L4 in front of an L7 fleet, or just an L7 (NGINX, HAProxy, Envoy, AWS ALB).

How does the LB pick a backend? Common algorithms:

• Round robin: rotate through servers in order. Simple, fair, ignores actual load.
• Least connections: send to whichever backend has fewest active connections. Better when requests have variable duration.
• Least response time: factor in measured backend latency. Self-correcting; punishes slow nodes.
• IP hash / consistent hashing: hash the client IP to pick a backend. Same client → same backend. Useful for caches and stateful workloads.
• Weighted variants: give bigger servers a higher slice (e.g. "this 16-core box gets 3x weight").

Default to least-connections for most web traffic. Use consistent hashing when locality matters (cache affinity).

One front door, many backends, health-checked.

A load balancer is only as good as its health check. Two flavours:

• Active: the LB pings each backend periodically (GET /health every 5s). Misses force removal.
• Passive: the LB watches the responses to real traffic; a sudden spike of 5xx pulls the backend.

Good practice:

• The health endpoint should test the actual work path (DB connectivity, dependencies), not just return 200.
• Mark a backend unhealthy on 2-3 consecutive misses, not 1 — transient blips happen.
• Mark it healthy again only on multiple consecutive successes — flapping nodes are worse than dead ones.
• Include the LB's own health in your observability — "the LB thinks server X is down" is the answer to many incidents.

By default an LB sends each request to whichever backend is currently best. But some apps store session state in memory on a single server. If the next request from the same user hits a different backend, the session is gone.

Sticky sessions (session affinity) solve this by binding a user to one backend, usually via a cookie the LB injects or by hashing the client IP.

It works, but it has costs: uneven load (whales stick to one server), no graceful failover (if their pinned server dies, the session is gone anyway), harder rolling deploys. The clean fix is to move session state to a shared store (Redis, the DB) so any backend can handle any request. Sticky sessions are the workaround; stateless backends are the goal.

What goes wrong, and what you do:

• A backend dies: health checks pull it. Other backends absorb its traffic. If you were running at >50% capacity per node, the survivors get crushed — provision headroom (typically run at <70%).
• The LB itself dies: deploy LBs in pairs/clusters with a VIP or DNS failover. AWS ALB, GCP HTTPS LB, Cloudflare are clustered for you.
• Backends are slow, not dead: health checks pass, latency spikes. Configure circuit breakers so the LB stops sending traffic to slow nodes before they cascade.
• Thundering herd on a restart: a node restarts and gets full prod traffic immediately. Use slow-start (Envoy, ALB support it) so traffic ramps up over 30–60s while caches warm.

Counter-intuitively, plenty of services don't need an LB:

• Serverless platforms (Lambda, Cloud Run, Vercel) load-balance for you — sending more requests just spawns more instances.
• Single-instance services (cron jobs, internal admin panels) don't benefit until they outgrow one box.
• Backend-to-backend RPC is often better served by client-side load balancing (gRPC + service discovery, where the client maintains its own pool) — avoids the extra network hop.

The useful question: am I scaling out, or just one big instance? If it's the latter, skip the LB until it's not.