Caching Strategies

A cache is a fast store sitting in front of a slow one. The trade you're making: memory for time, accepting that the cached copy can lag the source.

What you should expect:

• 10–1000x lower latency on hits.
• 10–100x lower cost per read at scale (a cache hit doesn't touch your DB).
• A new failure mode: stale data. Cached copies can disagree with the source of truth.

"There are only two hard things in Computer Science: cache invalidation and naming things." — Phil Karlton. Naming is jokingly the harder one; invalidation is the one that actually keeps people awake.

The most common pattern. The application code talks to both the cache and the database. The cache itself knows nothing about the DB.

def get_user(user_id):
    user = cache.get(f"user:{user_id}")
    if user is not None:
        return user
    user = db.fetch("SELECT * FROM users WHERE id = %s", user_id)
    cache.set(f"user:{user_id}", user, ttl=300)
    return user

Pros: simple, cache can hold arbitrary keys, DB stays the source of truth. Cons: every miss is a DB hit; writes have to invalidate the cache manually (cache.delete after db.update), and forgetting that delete is a common bug.

Read-through: the application talks only to the cache. The cache talks to the DB on a miss. Same caller code as cache-aside, but the responsibility moves to a cache layer that knows your data model.

Write-through: writes go to the cache first, which synchronously writes to the DB. The cache is never out of sync.

App.write(key, value)
  └─> Cache.put(key, value)        # 1. update cache
       └─> DB.write(key, value)     # 2. then DB, atomically

Pros: no stale data, application code is simpler. Cons: every write pays cache + DB latency. Both stores must be available, or writes fail.

In practice: read-through is what managed caches (DAX for DynamoDB, some Redis layers) automate. Write-through is rare in raw form; dual-writes with a write-aside is more common — and famously hard to get right.

Write-behind (write-back) lets writes hit the cache and return immediately. The cache batches and asynchronously writes to the DB later.

Used when:

• Write throughput exceeds DB write capacity.
• A small acknowledged-but-not-yet-durable window is acceptable.
• You can afford an out-of-process buffer (Kafka, a queue) to make the deferred write reliable.

The risk: data loss on cache crash. The user thinks the write succeeded; the cache hadn't flushed it; node dies. Disabled by default for anything money-shaped. Use it for telemetry, hot counters, like buttons — places where dropping 0.01% of writes is tolerable for 100x write throughput.

Where the write hop sits decides who can lose data when.

When data changes, you need to refresh the cache. Three approaches:

1. Explicit invalidation: on every write, cache.delete(key). Simple, prone to bugs if any code path forgets to invalidate.
2. TTL-based: set a time-to-live; cache entries expire automatically. Always works; staleness ≤ TTL. Picks TTL is a balance between freshness and hit rate.
3. Version keys: include a version in the cache key (user:42:v17). When data changes, bump the version. Old entries age out naturally. Great for content distribution; no manual deletes needed.

Most real systems use a mix: short TTL as the safety net + explicit invalidation on writes for promptness.

A popular cache entry expires. Twenty thousand requests hit the cache at once. All twenty thousand miss. All twenty thousand pile onto the database simultaneously. The DB folds.

Mitigations:

• Stale-while-revalidate: serve the old value while a background job refreshes it. Users see slightly stale data; the DB sees one refresh, not 20K.
• Probabilistic early expiration: refresh the cache before the TTL ends, with growing probability as expiration approaches. Spreads load.
• Single-flight (request coalescing): when N requests miss the same key, only one talks to the DB; the others wait for its answer. Trivial to add in front of any read-through cache.
• Background refresh: hot keys are refreshed on a schedule, never expire under load.

Default to single-flight + small jitter on TTL. Most stampede problems disappear.

Caching is not free. Don't cache when:

• Per-user, per-request data. Yes, you can cache it. But the hit rate is ~0% and you're paying memory for no win.
• Data that must be exactly current: financial balances, inventory in flash sales, anything where a stale read causes real harm.
• Writes that must be durable on ack: tournaments, ID generation, money. Use a proper transactional store.
• Things you can recompute trivially: caching len(items) when items is already in memory is just confusion.

The cost of caching is design overhead and stale-data bugs. Apply it where the read pattern is repeated, the source is slow or expensive, and slight staleness is tolerable. That's where caching pays off; everywhere else, it's overhead.