Loop engineering in production: state, errors, and budgets

Once your loop runs past a few iterations, the prompt is mostly prior turns. Three ways to manage that:

• Full history — keep every turn verbatim. Highest fidelity, fastest to bloat. Fine for short loops (≤5 turns).
• Sliding window — keep the last K turns plus the system prompt. Loses early context. Use when the task is local (debugging one file).
• Scratchpad — keep one mutable summary the model rewrites each turn. Best for long multi-step work; trades fidelity for focus.

Most production loops use a hybrid: system prompt + user request + last K turns + a scratchpad of decisions. Don't pick one philosophy; pick what each piece of state earns its keep at.

Compaction is the act of replacing old turns with a shorter representation that preserves what later iterations actually need. The bare minimum to keep:

• The user's original request.
• Decisions the model has made ("chose Python over Bash").
• Tool results that are still being referenced.

What to drop or summarize first:

• Verbose tool output — a directory listing from turn 2 doesn't matter on turn 12.
• Failed attempts — "tried X, got error Y" collapses to "X failed (Y)".
• Repeated reasoning — three turns of "let me think" → one summary line.

A simple rule: when prompt size crosses 50% of the model's context window, summarize the oldest third. Re-run that rule each turn.

Old turns become one summary; recent turns stay verbatim; new turn appends.

Tool calls fail in three distinguishable ways. The dispatcher should classify and respond, not just try/except: pass.

def dispatch(call):
    if not validate_args(call):
        return ToolError("malformed_args", "missing 'path'")
    try:
        return TOOLS[call.name](**call.args)
    except TransientError as e:
        return ToolError("transient", str(e), retry=True)
    except ToolFailure as e:
        return ToolError("unrecoverable", str(e))

• Malformed args — feed the validator's message back so the model fixes the call.
• Transient — 429s, network blips. Retry inside the dispatcher with backoff; the model doesn't need to see it.
• Unrecoverable — "file does not exist." Surface to the model so it replans.

Hiding all three behind a generic "tool failed" string is how loops thrash for ten iterations on the same fixable typo.

Retries can live in two places: inside the dispatcher (around one tool call), or around the whole loop (re-run the agent on failure). Pick one, not both blindly. Combining them silently multiplies attempts — three dispatcher retries × three loop retries = nine calls per failed tool, and the model still sees nothing useful.

Good layering:

• Dispatcher retries — only for transient errors (429, 5xx, timeout). Cap at three with exponential backoff.
• Model-visible retries — let the model see the unrecoverable error and try a different approach. This is the loop's job, not the dispatcher's.
• Outer retries — only at the very edge, around end-to-end runs that hit a fatal harness bug. Rare.

If you can't draw the retry pyramid for your loop on a napkin, you have a bug waiting to happen on the next outage.

Iteration cap alone isn't a budget. Stack four caps; the loop exits on whichever hits first:

The dollar cap is the one teams forget — and it's the only one that survives a tokenizer change, a model price hike, or a misconfigured cap. Track cumulative usage * price on every API response. Exit immediately on cross. Log which cap fired; that telemetry is how you tune the other three.

Bare ReAct decides one action at a time. Plan-act-reflect adds two extra moves: an opening plan turn (the model lists steps before calling any tool), and a reflect turn after each action ("did that get me closer? what's next?").

When it pays off:

• Multi-step branching tasks (debug → fix → re-test → refactor). The plan keeps the loop coherent across ten iterations.
• Tasks the model misjudges in one shot. Reflection catches "I thought the test passed but I read the wrong output."

When it's pure noise:

• Single-tool answers — the plan is one bullet, the reflection is empty.
• Tasks under three iterations — overhead outweighs benefit.

Reach for plan-act-reflect when your bare-ReAct loops average more than five iterations; otherwise stay simple.

When a tool reports an unrecoverable error ("test failed: AssertionError on line 42"), feed the failure back to the model and let it retry. This is the self-correction loop — the agent's superpower for code, math, and structured-output tasks where you can mechanically check the answer.

Three rules:

• Bound it. Cap self-corrections at two or three. If the model can't fix it in three tries, the next try won't help either.
• Give the model the actual error. Stack traces, validator messages, type-check output — verbatim, not paraphrased.
• Don't conflate with dispatcher retries. A 429 isn't a self-correction opportunity; an AssertionError is. Route accordingly.

Self-correction is what turns a 70% pass rate into a 90% pass rate on coding benchmarks. It's also where unbounded loops most often hide.