Context windows: tokens, limits, and "lost in the middle"

The context window is the maximum number of tokens the model can see in a single forward pass — input tokens plus the tokens it generates. Beyond that limit, the model literally cannot attend to your text; it doesn't exist as far as the math is concerned.

A token isn't a word. In English, 1 token ≈ 0.75 words. "Hello, world!" is 3 tokens. A 1,000-word essay is ~1,300 tokens. Code, JSON, and non-Latin scripts tokenize less efficiently — JSON often runs 1 token per 2–3 characters because of all the structural punctuation.

For sizing prompts in your head: ~750 words ≈ 1K tokens, a typical novel ≈ 100K tokens.

Vendors advertise increasingly large windows: 200K, 1M, even 2M tokens. Three important caveats:

1. The limit is a cap, not a sweet spot. Most models degrade in accuracy well before the advertised maximum. "Supports 1M" ≠ "performs well at 1M".
2. Cost scales linearly. A 200K-token prompt costs 200x a 1K prompt. A few of these blow your budget fast.
3. Latency scales worse than linearly. First-token latency on a 200K input is often 5–15 seconds before the model even starts generating. Attention is roughly $O(n^2)$ in sequence length.

Use big windows when the task needs the full content. For most uses, smaller + retrieval beats stuffing-everything-in.

A well-documented phenomenon: when relevant information sits in the middle of a long context, models retrieve it less accurately than if the same information is at the start or end.

In practice this means:

• Put critical instructions at the top of the prompt.
• Put critical retrieval results at the end, just before the user question.
• If you have to cite many documents, rank them — the most relevant should be first or last, not buried.

The effect varies by model (newer frontier models are noticeably better) but doesn't disappear. Even with 1M tokens of capacity, an attention pattern that overweights start/end is the prior.

Recall is highest at the ends, weakest in the middle.

Behind the scenes, models cache the keys and values from attention layers as they process tokens. This KV cache is what makes the second token cheaper than the first.

At the API level, vendors expose prefix caching (Anthropic) or prompt caching (others). If two requests share the same prefix — system prompt + tool definitions + few-shot examples — the model reuses cached computation.

Typical wins:

• 5–10x input cost reduction on the cached portion
• 30–50% first-token latency improvement
• Cache lives for several minutes; identical-prefix requests within that window hit

Design prompts with caching in mind: stable system prompt → stable tool defs → stable examples → varying user input at the end.

In long-running agents, the conversation history grows every turn. Three strategies to keep it bounded:

1. Sliding window: only include the last N turns. Loses old context but is dead simple.
2. Summarisation: when context grows past a threshold, ask the model to summarise the early part and replace it with the summary. Saves space; introduces drift.
3. Retrieval-over-history: store the conversation in a vector DB, retrieve only relevant chunks for each new turn. Most expensive to implement; preserves the most signal.

Claude Code, ChatGPT's long-conversation handling, and Cursor's agent mode all use variants of (2) — auto-compaction. The art is detecting which parts of history are still load-bearing.

If your context need is "answer questions over a 500-page manual", you have a choice:

• Stuff it all in: feed the whole manual on every call. Works in a 1M-token window. Costs roughly 200K tokens × $price per call.
• Retrieve: embed once, retrieve top-k relevant chunks per question. Costs ~3K tokens per call.

For 100 questions, retrieval is ~100x cheaper and often more accurate (the model isn't distracted by 199,500 irrelevant tokens).

Long context wins when the task needs holistic understanding — "summarise the whole document", "find inconsistencies across chapters". Retrieval wins for lookup-shaped questions, which is the vast majority of real-world Q&A.

Don't guess. Estimate.

import anthropic
client = anthropic.Anthropic()

result = client.messages.count_tokens(
    model="claude-opus-4-7",
    messages=[{"role": "user", "content": long_doc_here}],
)
print(result.input_tokens)

Most providers expose a count_tokens or equivalent. Use it before sending anything large — it's free and prevents "oh, that was 180K tokens" surprises.

For rough mental math: copy your input into the OpenAI tokenizer or [Anthropic's playground]. Tokenizers vary slightly between models but counts are within ~10% across the frontier.