Context Engineering for Long-Running Agents

An agent's context window is its working memory. Every tool call result, every prior message, every retrieved chunk occupies tokens that cost latency and money — and once the window fills, the model either truncates silently or halts entirely. For a 10-turn chat session this is a non-issue. For a multi-hour coding agent or a research loop that calls 50 tools, context pressure is the primary engineering challenge.

The naive approach — just dump everything into the prompt — breaks down around 30-40% of window capacity: attention dilutes, retrieval accuracy drops, and costs compound. Context engineering is the discipline of deciding what to keep, what to compress, and what to fetch on demand at each step of the agent loop.

At each agent step the usable context is divided between fixed overhead (system prompt, tool schemas) and dynamic payload (history, results). Once the dynamic payload crosses a threshold, the loop must decide how to reclaim space without losing information the agent needs to stay on task.

Before you can manage a budget you need to measure it. Most provider SDKs expose usage.input_tokens on each response. A simple budget class tracks spend and fires callbacks when thresholds are crossed:

class ContextBudget:
    def __init__(self, max_tokens: int, compact_at: float = 0.75):
        self.max = max_tokens
        self.compact_threshold = int(max_tokens * compact_at)
        self.used = 0

    def update(self, input_tokens: int):
        self.used = input_tokens  # updated each turn
        return self.used >= self.compact_threshold

Set compact_at to ~0.70–0.80 of the model's limit — not 0.95. You want headroom for the compaction call itself. For Claude Sonnet 4.5 (200 k tokens) a reasonable budget is 140 k before you compact, leaving 60 k for the summary prompt response and next-turn history.

The most reliable compaction strategy is a rolling summary: when the budget fires, you call the model with the last N messages and ask it to produce a dense summary paragraph, then replace those messages with a single [CHECKPOINT] assistant turn.

SUMMARY_PROMPT = """
You are compressing agent history. Produce a concise summary that preserves:
- Every decision made and its rationale
- All open tasks and their current state
- Key facts discovered (filenames, IDs, URLs, numbers)
- What was tried and failed (so the agent doesn't retry)
Do NOT include pleasantries or meta-commentary. Be dense.
"""

def checkpoint(messages: list[dict], llm) -> list[dict]:
    to_compress = messages[1:]  # keep system prompt at [0]
    summary = llm.complete(SUMMARY_PROMPT + "\n\n" + format(to_compress))
    return [messages[0], {"role": "assistant", "content": f"[CHECKPOINT]\n{summary}"}]

Keep the system prompt immutable — it is your agent's identity and should never be folded into a summary.

A scratchpad is a mutable key-value store the agent explicitly updates via a tool call. Unlike the rolling summary (which the orchestrator generates externally), scratchpad entries are agent-authored — the model decides what facts are worth saving.

# Tool definition
{"name": "update_scratchpad",
 "description": "Store a named fact you'll need later. Overwrites existing key.",
 "parameters": {"key": {"type": "string"}, "value": {"type": "string"}}}

# Orchestrator renders it at the top of each turn
def render_scratchpad(pad: dict) -> str:
    if not pad:
        return ""
    lines = "\n".join(f"- {k}: {v}" for k, v in pad.items())
    return f"### Agent scratchpad\n{lines}\n"

Scratchpad beats raw history for things like "the PR number is 4231" or "the user wants CSV not JSON" — short facts that would otherwise get buried. The downside: if the model forgets to update the scratchpad, that fact is lost at the next checkpoint. Pair scratchpad with checkpoints for resilience.

Not all knowledge should live in-context at all times. Retrieval Augmented Generation (RAG) lets you store documents in a vector index and pull only the top-K chunks that are relevant to the current agent step.

from anthropic import Anthropic
from your_index import VectorIndex

index = VectorIndex("codebase-embeddings")
client = Anthropic()

def inject_relevant_context(query: str, messages: list) -> list:
    chunks = index.search(query, top_k=5, max_tokens=4000)
    rag_block = "\n---\n".join(c.text for c in chunks)
    # Inject as a user turn immediately before the assistant's reply
    return messages + [{"role": "user",
        "content": f"Relevant context:\n{rag_block}\n\nNow proceed."}]

For long-running agents, re-query the index per step, not once at the start. The relevant context shifts as the agent progresses. Query with the agent's current sub-task description, not the original user request.

Heads up: injecting stale RAG results mid-task can mislead the agent more than having no context at all. Always timestamp chunks and filter by recency when documents change.

Anthropic's prompt caching lets you mark a prefix of the prompt as cacheable. On cache hit, the provider skips re-processing that prefix — cutting latency by ~85% and cost by ~90% for the cached portion. For agents, the system prompt + tool schemas are the natural cache prefix.

response = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=4096,
    system=[
        {
            "type": "text",
            "text": SYSTEM_PROMPT + "\n\n" + json.dumps(TOOL_SCHEMAS),
            "cache_control": {"type": "ephemeral"}  # cache this prefix
        }
    ],
    messages=conversation_history,
)
print(response.usage.cache_read_input_tokens)  # verify hit

The cache TTL is 5 minutes (ephemeral). For agents with steps longer than 5 minutes, the cache expires between turns — re-send the cache_control marker each turn to keep refreshing it. The cache is keyed on the exact byte sequence of the prefix, so never mutate the system prompt mid-session.

In-context memory is fast but expensive and ephemeral. External stores are cheap but require a retrieval step. The craft is knowing which facts are worth the retrieval latency versus keeping them warm in the window.

Compaction has a cost — the summary call itself uses tokens and adds latency. Compact too eagerly and you lose nuance; compact too late and you hit the hard limit mid-task. Good triggers balance three signals:

1. Token threshold — compact at 70–80% of the model's context limit (most reliable).
2. Step count — compact every N steps regardless of token use; useful when each step is large and unpredictable.
3. Task boundary — compact when the agent completes a named sub-task ("file written", "PR created"). Summaries at natural boundaries are more coherent than mid-sentence cuts.

Avoid compacting during:

• A multi-turn tool-call chain that hasn't resolved yet (the partial state is incoherent to summarize).
• Immediately after receiving a large tool result you haven't acted on — the summary may drop critical details.

A practical rule: compact at the start of the next turn, not the end of the current one. That way the previous turn's result is already in the history to be summarized.

A single rolling summary works for tasks under ~200 steps. For longer tasks — overnight research agents, week-long background workers — you need hierarchical summarization: summaries of summaries.

Level 0: raw messages       →  checkpointed every 20 turns
Level 1: checkpoint summary →  re-summarized every 5 checkpoints (100 turns)
Level 2: epoch summary      →  re-summarized every 5 epochs (500 turns)

Store all levels externally (a database, not in-context). At each turn, reconstruct context from the most recent level-2 epoch, the most recent level-1 checkpoints since that epoch, and the raw messages since the last checkpoint. The agent sees a coherent narrative at every scale.

This mirrors how humans remember projects: rough milestone memory stretching back months, detailed memory for the last few days, vivid recall for the last hour.

Putting it all together — a sketch of a production coding agent:

while not task_complete:
    # 1. Inject relevant context from codebase index
    relevant = index.search(current_subtask, top_k=4)
    inject_chunks(messages, relevant)

    # 2. Render scratchpad at top of user turn
    messages[-1]["content"] = render_scratchpad(pad) + messages[-1]["content"]

    # 3. Call model with cached system prefix
    resp = client.messages.create(
        model="claude-sonnet-4-5",
        system=[{"type": "text", "text": SYSTEM, "cache_control": {"type": "ephemeral"}}],
        messages=messages,
        tools=TOOLS,
        max_tokens=8192,
    )

    # 4. Process tool calls, update scratchpad
    handle_tool_calls(resp, pad, messages)

    # 5. Check budget; compact if needed
    if budget.update(resp.usage.input_tokens):
        messages = checkpoint(messages, client)

Notice that steps 1–3 happen every turn — retrieval is not cached between turns. The scratchpad render is cheap (usually <200 tokens) and ensures the model always sees its own notes prominently.

Lossless summary illusion — summaries always lose information. Never summarize tool call raw outputs that you might need to quote verbatim (API responses, error stack traces, legal text). Store those externally and retrieve them by reference.

Cache invalidation surprises — changing even one byte of a cached prefix busts the cache. Don't version your system prompt per-agent-run; keep it stable across sessions.

RAG hallucination amplification — if the vector index contains outdated or contradictory chunks, the agent will confidently act on stale data. Add metadata filters (date, file hash) and always log retrieved_sources to the scratchpad for auditability.

Context thrashing — injecting the same 4 k chunk every single step wastes budget. Track which chunks you've injected this session and skip re-injection unless the query shifts significantly (cosine similarity < 0.85 from last injection query).

Over-compacting — if your agent never has more than 1 k tokens of history to summarize, you're compacting too aggressively. A one-sentence summary loses too much. Aim for summaries that compress 5–20 k tokens into 500–1000 tokens.