Caching Strategies

Caches exist at every layer of the stack, each with different latency, capacity, and invalidation complexity:

A cache hit at the CDN layer costs you ~10 ms and zero origin compute. A cache miss that hits a cold Postgres query costs 50–200 ms and a DB CPU spike. For a service handling 50k QPS, going from 80% to 95% CDN hit ratio saves ~750 req/s of origin load — often the difference between scaling up and not.

The rule: cache as close to the client as possible, invalidate as close to the data source as possible.

The most common pattern. The application is the cache manager: on a miss, it loads from the DB, writes to cache, then returns the result.

def get_user(user_id: str) -> dict:
    key = f"user:{user_id}"
    user = redis.get(key)
    if user is None:                       # cache miss
        user = db.query(
            "SELECT * FROM users WHERE id = %s", user_id
        )
        redis.setex(key, 300, json.dumps(user))  # TTL = 5 min
    return json.loads(user)

Pros: only caches data that's actually requested; DB is the source of truth; cache failure is non-fatal (fall through to DB). Cons: first request after a miss pays full DB latency (the cold-start penalty); stale data if the DB is updated outside this path. Cache-aside is the right default for read-heavy, write-occasional data like user profiles, product catalogs, and configuration.

Read-through: the cache itself fetches from the DB on a miss — your app never talks to the DB directly. The cache library or a caching proxy (e.g., PgBouncer + pgpool) handles it. Simpler app code; the same cold-start penalty applies.

Write-through: every write goes to cache AND DB synchronously before returning success. Cache is always warm; zero stale reads. Cost: every write is 2x slower (cache write + DB write).

Write-back (write-behind): writes land in the cache first; DB is updated asynchronously. Write latency drops dramatically — great for counters, view counts, analytics events. The danger: if the cache node dies before flushing, writes are lost. Only safe with persistent cache (Redis AOF/RDB) and explicit flush policies.

Summary: cache-aside for reads, write-through for safety-critical writes, write-back for high-frequency non-critical writes (e.g., like counts on a social feed).

When cache memory fills, the eviction policy decides what to drop:

LRU (Least Recently Used): evict the item not accessed in the longest time. Excellent for temporal locality — recently used data is likely to be used again. Default for most Redis use cases (maxmemory-policy allkeys-lru).

LFU (Least Frequently Used): evict the item accessed the fewest times. Better for skewed workloads where a small hot set is accessed constantly (top 100 products out of 1M). Redis allkeys-lfu tracks access frequency with a probabilistic counter.

TTL (time-to-live): every key expires after a fixed duration. Not an eviction policy per se, but the primary mechanism for staleness control. Combine with LRU: keys expire lazily OR get evicted when memory fills.

Gotcha: setting TTL too short causes high miss rates and DB hammering. Too long causes stale reads. For user-facing data that changes rarely, 5–15 minutes is the practical sweet spot. For financial data, 0–10 seconds.

A popular key expires. Simultaneously, 2,000 requests arrive, all miss the cache, and all fire DB queries for the same row. Your DB gets 2,000 identical queries in 10 ms — a cache stampede (also called thundering herd).

Four mitigations:

1. Mutex / lock on miss: the first goroutine/thread acquires a distributed lock (Redis SET NX PX), fetches from DB, populates cache. Others wait. Simple but adds latency for waiters.
2. Probabilistic early expiration: before the TTL hits zero, start refreshing stochastically — keys that are about to expire get re-fetched early with probability proportional to how close they are to expiry. No lock needed.
3. Stale-while-revalidate: serve the stale value immediately while refreshing in the background. HTTP Cache-Control: stale-while-revalidate=60 does this at the CDN layer automatically.
4. Jitter on TTL: instead of TTL = 300, use TTL = 300 + random(0, 60). Spreads expiry across a 60-second window, preventing correlated misses.

For Redis with thousands of keys expiring per second, options 3 and 4 are the lowest-overhead solutions.

Phil Karlton's famous quip: "There are only two hard things in computer science: cache invalidation and naming things." It's hard because stale reads cause subtle, hard-to-reproduce bugs.

Common strategies:

TTL expiry: simplest. Accept up to TTL seconds of stale data. Right choice when consistency isn't critical.

Event-driven invalidation: when the DB row changes, publish an event (Postgres NOTIFY, Debezium CDC) and delete or update the cache key. Consistent, but adds infra complexity. Shopify uses this pattern at scale.

Write-through invalidation: the application deletes the cache key on every DB write (DEL user:42). Simple, correct. Next read repopulates via cache-aside.

Gotcha — delete vs update: updating the cache on a write seems correct but races with concurrent readers. Two writes can race such that the slower one overwrites the faster one's cache entry. Delete the key instead; let the next read repopulate atomically.

For distributed caches: avoid updating; always delete and let the next read win.

Hit ratio = cache hits / total requests. This single number determines whether your cache is earning its keep.

Suppose your service handles 10,000 QPS. Your Postgres can handle 500 queries/sec comfortably before latency climbs.

• At 95% hit ratio: 500 QPS reach the DB. Fine.
• At 90% hit ratio: 1,000 QPS reach the DB. DB at 2x capacity — latency starts climbing.
• At 80% hit ratio: 2,000 QPS reach the DB. 4x capacity — cascading failure.

Improving hit ratio:

• Increase cache size (more keys fit, fewer evictions).
• Increase TTL (keys live longer between misses).
• Better key design (coarser granularity: cache the full user object, not individual fields).
• Warm the cache on startup (pre-load top-N keys from DB before opening traffic).

Track hit ratio per key pattern in production (Redis INFO stats gives keyspace_hits and keyspace_misses). A sudden drop in hit ratio is often the first signal of a traffic anomaly or a cache configuration regression.

CDNs are distributed caches at the network edge. Getting CDN caching right is free performance — but misconfiguration is expensive.

Key HTTP headers:

# Cache this response for 1 hour at CDN; allow stale for 60s while refreshing
location /api/products {
    add_header Cache-Control "public, max-age=3600, stale-while-revalidate=60";
}

# Never cache — user-specific, authenticated
location /api/account {
    add_header Cache-Control "private, no-store";
}

public: CDN may cache. private: only the browser may cache (CDN must not). no-store: cache nothing. s-maxage: CDN-specific TTL overriding max-age.

Gotcha: if your app sets a Set-Cookie header on a cached response, most CDNs (Cloudflare, CloudFront) will bypass the cache entirely by default. Strip cookies from cacheable paths at the CDN edge or use Vary: Cookie deliberately. A misconfigured cookie header on a product listing page can tank your CDN hit ratio from 90% to 5% overnight.