Caching for LLM systems: three layers, in order of leverage

LLM calls are slow and expensive. A frontier model running on a typical 4K-input/1K-output request takes a couple of seconds and costs a couple of cents. Multiply by your user count and request frequency and the bill gets uncomfortable fast.

The good news: most LLM workloads are full of repetition. The same system prompt on every call. The same FAQ asked by different users. The same RAG retrievals appearing across many sessions. Three different caching layers each target a different kind of repetition. Stacked correctly, they routinely cut 50–90% of your inference cost without users noticing.

The dumbest, cheapest cache: hash the inputs (system prompt + user message + parameters), store the response, return it on the next identical request.

import hashlib
import json

def cache_key(messages, system, model, temperature):
    payload = json.dumps({"messages": messages, "system": system, "model": model, "t": temperature}, sort_keys=True)
    return hashlib.sha256(payload.encode()).hexdigest()

key = cache_key(messages, system_prompt, "claude-opus-4-7", 0)
if (cached := redis.get(key)):
    return cached
response = client.messages.create(...)
redis.setex(key, 3600, response)

Works great for FAQs, deterministic prompts, and anything whose answer doesn't depend on time. Useless for highly varied user inputs — hit rates fall off a cliff.

Frontier APIs let you mark a prefix of your prompt as cacheable — system prompt, tool definitions, large reference documents. The provider stores the model's intermediate computation (KV cache state) and reuses it for any subsequent request that starts with the same prefix.

With Anthropic:

response = client.messages.create(
    model="claude-opus-4-7",
    system=[
        {"type": "text", "text": LARGE_SYSTEM_PROMPT, "cache_control": {"type": "ephemeral"}}
    ],
    messages=[...],
)

Typical wins: 5–10x cheaper input on the cached portion, 30–50% faster first token. Cache survives ~5 minutes (extendable to an hour with a paid tier). Works even when the response would be different — the model still runs, just from a checkpoint.

This is the single biggest LLM cost optimisation available today. If you have a stable system prompt, you should be using it.

What if user inputs aren't exact matches but similar? "What's your refund policy?" and "Can I get my money back?" deserve the same answer.

Semantic cache embeds the incoming query, searches an embedding store for similar past queries, returns the cached response if similarity exceeds a threshold.

q_emb = embed(user_query)
hit = vector_store.search(q_emb, top_k=1)
if hit and hit.score > 0.92:
    return hit.response
response = client.messages.create(...)
vector_store.add(q_emb, response, ttl=86400)

Powerful but dangerous. The wrong threshold sends users semantically-close-but-actually-different answers ("How do I cancel my subscription" answered with "How do I cancel my order"). Tune the threshold high (≥0.92), evaluate on real data, and always include a confidence-driven fallback to recompute when uncertain.

Each layer absorbs different repetition.

Cases where caching subtly produces wrong results:

• Personalisation: "What's my account balance?" — same words, different user. Bake the user ID into the cache key.
• Time-sensitive answers: "Is the office open right now?" Cache TTL must be short.
• Tool-using agents: the response depends on what tools returned, which depends on real-world state. Caching the whole conversation is almost always wrong; cache only the LLM-step computations.
• Streaming: cached responses don't stream tokens. Users notice if the UX flips from gradual to instant.
• Stochastic outputs (temperature > 0): users expect variety in creative tasks. A cached identical response feels broken.

The pattern: anything that depends on context outside the prompt is unsafe to cache without that context in the key.

All three layers need a freshness policy:

• Response cache: TTL based on how often the underlying answer changes. FAQs: hours-to-days. Real-time data: minutes or no cache at all.
• Prefix cache: provider-managed, typically 5 minutes. You can extend with a heartbeat (occasional dummy request) for sustained workloads.
• Semantic cache: TTL plus manual invalidation. When you update the underlying source of truth (knowledge base, policies), clear semantic entries that match.

A common mistake: aggressive long TTL plus no invalidation path. The cache is right at first, then quietly serves outdated answers for weeks. Build invalidation when you build the cache, not after the bug report.

If you're starting from scratch with no caching at all:

1. Enable prefix caching today. Almost free, biggest single win. Stabilise your system prompt so the prefix is identical across requests; mark it cacheable; you're done.
2. Add response caching for repetitive endpoints. FAQ-shaped endpoints, batch processing of similar items, anything where the same input keeps reappearing.
3. Measure your hit rates. Log cache hits/misses by layer. Without measurement you're guessing.
4. Only then evaluate semantic caching. It has the highest correctness risk and the lowest hit rate of the three. Add it once you can prove (with eval data) it returns the right answer above 95% of the time.

Most teams skip step 3 and add semantic caching first because it sounds clever. The win-per-engineering-hour is upside down.